Build Foundation and enable its test suite

[MaxDesiatov]

We haven't verified how much of the Foundation module works on WASI. At a glance there don't seem to be any obvious blockers that prevent something like FoundationXML from working, but it's clear that FoundationNetworking would have to look quite differently. This needs to be verified by enabling the test suite and checking which of the tests fail.

[MaxDesiatov]

Looks like without #658 and #647 a very significant chunk of CoreFoundation and Foundation needs to be disabled. This is exacerbated by a lack of proper filesystem APIs. After spending a lot of time on it, I don't think I'm very close to fixing this, even simple APIs like CFString, NSString, CFDictionary etc use blocks which require BlocksRuntime, but these simple types are used almost everywhere in the codebase of Foundation. I think important types like Date and Data can probably be reimplemented separately as stubs, while I'm not sure we'll see enough demand for NSString, NSDictionary, NSDate, NSData etc as those are legacy types and should be avoided anyway.

This probably can be revisited after WASI supports threading and dynamic linking properly.

[MaxDesiatov]

I've got simple things like print("\(Date().timeIntervalSince1970)") working, but something more complex, e.g. date formatters and time zones crash with out of bounds error.

The stacktrace I got with wasminspect looks like this with code as simple as print("\(Date())"):

0: _swift_release_dealloc
1: bool swift::RefCounts<swift::RefCountBitsT<(swift::RefCountInlinedness)1> >::doDecrementSlow<(swift::PerformDeinit)1>(swift::RefCountBitsT<(swift::RefCountInlinedness)1>, unsigned int)
2: bool swift::RefCounts<swift::RefCountBitsT<(swift::RefCountInlinedness)1> >::doDecrement<(swift::PerformDeinit)1>(unsigned int)
3: swift_release
4: _CFRelease
5: CFRelease
6: __CFTimeZoneCreateSystem
7: CFTimeZoneCopySystem
8: CFTimeZoneCopyDefault
9: __CreateCFDateFormatter
10: CFDateFormatterCreate
11: Foundation.NSDate.description.getter : Swift.String
12: Foundation.Date.description.getter : Swift.String
13: protocol witness for Swift.CustomStringConvertible.description.getter : Swift.String in conformance Foundation.Date : Swift.CustomStringConvertible in Foundation
14: Swift.DefaultStringInterpolation.appendInterpolation<A where A: Swift.CustomStringConvertible>(A) -> ()
15: main
16: __original_main
17: _start

[kateinoigakukun]

@MaxDesiatov Awesome! I think the out of bounds error is due to thin ICU data.

Could you try to build ICU disabling ICU_DATA_FILTER_FILE for temporary?

https://github.com/swiftwasm/icu4c-wasi/pull/1/files

[MaxDesiatov]

Do I need to rebuild the whole SDK with this ICU? Or do you think that swapping ICU .a libraries in the existing toolchain/SDK installation and relinking with those would work?

[kateinoigakukun]

swapping ICU .a libraries in the existing toolchain/SDK installation and relinking with those would work?

Yes, it would work because libicudata only has data section.

[MaxDesiatov]

That didn't work unfortunately, I've copied .a files from an artifact in this PR swiftwasm/icu4c-wasi#2

It doesn't even seem to be linking with libicudata.a:

/Users/maxd/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-05-11-a/usr/bin/swiftc 
-sdk /Users/maxd/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-05-11-a/usr/share/wasi-sysroot 
-I /Users/maxd/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-05-11-a/usr/lib/swift/wasi/wasm32 
-Xlinker -lCoreFoundation -Xlinker -lBlocksRuntime -Xlinker -licui18n -Xlinker -luuid 
-L /Users/maxd/Documents/life-game-with-swiftwasm/LifeGame/.build/wasm32-unknown-wasi/debug 
-o /Users/maxd/Documents/life-game-with-swiftwasm/LifeGame/.build/wasm32-unknown-wasi/debug/LifeGameWeb
 -module-name LifeGameWeb 
-emit-executable @/Users/maxd/Documents/life-game-with-swiftwasm/LifeGame/.build/wasm32-unknown-wasi/debug/LifeGameWeb.product/Objects.LinkFileList 
-target wasm32-unknown-wasi 
-L /Users/maxd/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-05-11-a/usr/lib

% wasmer .build/debug/LifeGameWeb                                    
Error: error: Caught exception of type "MemoryOutOfBounds".

% ../../wasminspect/target/debug/wasminspect .build/debug/LifeGameWeb
(wasminspect) run
Function exec failure Memory(AccessOutOfBounds(Some(18446744073709551612), 29622272))
(wasminspect) thread backtrace
0: _swift_release_dealloc
1: bool swift::RefCounts<swift::RefCountBitsT<(swift::RefCountInlinedness)1> >::doDecrementSlow<(swift::PerformDeinit)1>(swift::RefCountBitsT<(swift::RefCountInlinedness)1>, unsigned int)
2: bool swift::RefCounts<swift::RefCountBitsT<(swift::RefCountInlinedness)1> >::doDecrement<(swift::PerformDeinit)1>(unsigned int)
3: swift_release
4: _CFRelease
5: CFRelease
6: __CFTimeZoneCreateSystem
7: CFTimeZoneCopySystem
8: CFTimeZoneCopyDefault
9: __CreateCFDateFormatter
10: CFDateFormatterCreate
11: Foundation.NSDate.description.getter : Swift.String
12: Foundation.Date.description.getter : Swift.String
13: protocol witness for Swift.CustomStringConvertible.description.getter : Swift.String in conformance Foundation.Date : Swift.CustomStringConvertible in Foundation
14: Swift.DefaultStringInterpolation.appendInterpolation<A where A: Swift.CustomStringConvertible>(A) -> ()
15: main
16: __original_main
17: _start

The only thing I have in Sources/LifeGameWeb/main.swift is this:

import Foundation

print("\(Date())")

I also build with swift build -v --destination destination.json where destination.json looks like this:

{
  "version": 1,
  "target": "wasm32-unknown-wasi",
  "sdk": "/Users/maxd/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-05-11-a/usr/share/wasi-sysroot",
  "toolchain-bin-dir": "/Users/maxd/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-05-11-a/usr/bin/",
  "extra-cc-flags": [],
  "extra-swiftc-flags": [
    "-I", "/Users/maxd/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-05-11-a/usr/lib/swift/wasi/wasm32",
    "-Xlinker", "-lCoreFoundation",
    "-Xlinker", "-lBlocksRuntime",
    "-Xlinker", "-licui18n",
    "-Xlinker", "-luuid"
  ],
  "extra-cpp-flags": []
}

I've manually copied Foundation and CoreFoundation libraries to the toolchain directories like ~/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-05-11-a/usr/lib/swift/ and ~/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-05-11-a/usr/lib/swift_static/.

But you probably shouldn't spend time on reproducing it in this state, there are too many manual steps here. I'll try to submit a PR soon that copies Foundation into our installable distribution so that this issue could be reproduced more easily.