x/mobile: add bitcode support

[ashutosh-android]

I am able to successfully generate and use .framework file using gomobile bind command, however generated lib doesn't have bitcode enabled
Is there a way to enable bitcode in generated binary or are there are any plan to provide support in near future ?

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8.3 darwin/amd64
gomobile version +44a54e9

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

MacOS

What did you do?

Generated .framework lib from go source code using 'gomobile bind'

What did you expect to see?

bitcode should be enabled in generated framework

What did you see instead?

generated framework doesn't have bitcode enabled

[ashutosh-android]

Can you please update on this. This is a blocker item for one of my project. It will be great if I get some visibility on release plan for this requirement. Thanks.

[timcooijmans]

I'm having the same problem: Currently the Go compiler doesn't support compiling to BitCode intermediate representation. I think something has to be done for Gomobile to have a future. It's expected that Apple will mandate BitCode for iOS-apps in the near future.

However there seems to be some work going on on Google-side: https://go.googlesource.com/gollvm/

You could also try https://dragonegg.llvm.org/ but I have no idea if this is going to work.

[ashutosh-android]

Any update on this ?

[eliasnaur]

I don't think there is any update other than gollvm is progressing. May I ask why this issue is blocking your project?

[ashutosh-android]

@eliasnaur thanks for looking into this query.
I am working on a mobile project where we will be distributing our service as library project. We are writing alot of business logic in go code so that we can reuse them for both android and iOS using go-mobile. Since framework generated by go-mobile doesn't have bitcode enabled so any developer who will be integrating our solution will not be able to enable bitcode for there application. Considering how apple is strongly recommending developers to enable bitcode it will impact adoption our product in a big way.

I hope this answers your question. Let me know if you have any suggestions/query. Thanks again.

[matti777]

Just ran into this. Yes, bitcode support would be a must. Any updates?

[jpop32]

Yes, lack of bitcode support on iOS is a showstopper for library projects, and a big liability for others. We can safely assume that 'strong recommendation' to use bitcode is going to turn into 'requirement' sooner or later.

[adam-p]

Since it hasn't been mentioned here yet: Apple TV apps must support Bitcode. Not optional. Search for "bitcode" here: https://developer.apple.com/tvos/submit/

This has caused us problems when trying to support a client wanting to use our library on Apple TV.

[zboralski]

Just ran into this.

[drewpitchford]

I'll throw my hat in this ring too. We have iOS/Android sdks that use a go-mobile framework for some common encryption/decryption. Lack of bitcode is a major issue for us. I am attempting to replace the go-mobile framework in the iOS project with a js framework that does the same stuff, but it'd be nice to be able to keep the go framework

[pavlo]

It is a showstopper for us as well

[biscottigelato]

WWDC is coming soon, hopefully Bitcode will not become a requirement... The lack of any response from anyone in the know about how one an get Bitcode going for golang/gomobile is definitely concerning.

[ianlancetaylor]

As far as I know the only way to get the required Bitcode is to run LLVM. There is an active project to add Go support to LLVM; you can follow it at https://go.googlesource.com/gollvm/ . Until that project is complete and working, there will be no way to get Bitcode.

That is essentially what @eliasnaur said on March 16. While I understand that the pace of these things is frustrating, I don't think it's entirely fair to say that there has been no response. Go is an open source project; if you want this to happen faster, try building GoLLVM, report bugs, and send in fixes.

[ashutosh-android]

Hi,

Checking to see if there is any new development on this ?

Thanks,
Ashutosh

[ncastagnoli]

Hi,

To add to the need, if that's important.

I'm working on a project that started cloud based, and elected to write the code in "go." It turns out we also need to run the code on the client, both iOS and Android. But, our partners are demanding bit-code compliance. We are at a decision point to decide whether to rewrite the entire set of code so we can have a single code base. Curious about the thinking regarding bit code compliance for "go."

Is it a project? If so, what is the time-frame? Go is perfect for our application, incidentally.

[ashutosh-android]

Hi,

Checking again on this. Is there any roadmap for this support ?

Thanks,
Ashutosh

[eliasnaur]

Please refrain from asking about updates on this issue; fixing it is a lot of work and only benefits niche platforms for Go: tvOS and watchOS. @ianlancetaylor suggested you try gollvm; have you done that? If so, what problems did you run into?

[ianlancetaylor]

https://golang.org/wiki/NoPlusOne

[matti777]

Ahem, what do you mean tvOS and watchOS? What about iOS? :) Not sure if that is a 'niche' environment for gomobile bind..

[eliasnaur]

Only watchOS and tvOS require bitcode; iOS does not.

[drewpitchford]

But we don’t know that will be the case forever. In fact, I’d say it’s highly likely to not be the case. Drew

…

Sent from my iPhone XS Max

On Dec 18, 2018, at 10:48 AM, Elias Naur ***@***.***> wrote: Only watchOS and tvOS require bitcode; iOS does not. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[eliasnaur]

Whatever the likelyhood, let's worry about it when it happens and not waste precious development efforts on hypotheticals. Who knows, perhaps the very existence of non-bitcode gomobile apps will delay Apple's decision.

[matti777]

iOS "requires" bitcode in the way that if you are making a library with golang / gomobile bind, the library you produce will force the iOS app developer to disable bitcode in their app; and we iOS devs really do not like that. :) Also of course there is a point to having bitcode support in the first place and that is now lost. third, who knows when iSteve will decide bitcode support will become mandatory ..

[ianlancetaylor]

Everyone agrees that it would be good if we could compile Go into bitcode. As mentioned above, the only way that is ever going to happen is if you use the GoLLVM compiler. So if you want Go compiled into bitcode, please help with GoLLVM.

[tmm1]

The Kotlin/Native team has found a work-around to full bitcode support: they add a __LLVM,__asm marker (via JetBrains/kotlin-native#2186) which tells Xcode the object was created with a bitcode-aware compiler, and that the object was generated from assembly and thus has no bitcode equivalent. (More details about the bitcode format on https://jonasdevlieghere.com/embedded-bitcode/).

See discussion on the kotlin-native repo: JetBrains/kotlin-native#1202 (comment)

The technique was discovered on an earlier PR, which attempted to emit bitcode but was unable to do so reliably without using Apple's version of LLVM: JetBrains/kotlin-native#1564 (comment)

Would it be possible to add a similar empty __LLVM,__asm section to the generated gomobile .framework?

[randall77]

It would not be hard to add an empty section to the binary.

I worry that this is essentially a way to hack around Apple's policy. As such, they could revoke this exception at any time and then we're back to square one.

[tmm1]

@eliasnaur I'm curious what impact your whitelist change in b4f3b8a has?

[eliasnaur]

Very little; it allows the tvOS and watchOS flags in #cgo directives. I noticed the missing flags while working on this.

[tmm1]

Very little; it allows the tvOS and watchOS flags in #cgo directives. I noticed the missing flags while working on this.

Aha, okay. It was working for me already via the CGO_CFLAGS env vars so I was curious what the change enabled.

Maybe a GODARWIN would be simpler.

Another idea would be to add some new flags to the go linker, so the user could do something like -ldflags="-machoplatform=ios -machominver=11.0.0"

If we can settle on a good approach here, I'm happy to start putting together some CLs.

Another thing to keep in mind is that the LC_VERSION_MIN_* have been replaced with LC_BUILD_VERSION for the next macOS release, and that this new load command will likely be required to build marzipan apps which work on both iOS and macOS. However since all existing macOS/iOS/tvOS/watchOS releases still use the older variant, we will need to keep supporting that for the foreseeable future.

For reference, here are the two load command definitions:

    struct version_min_command {
      uint32_t cmd;       // LC_VERSION_MIN_MACOSX or
                          // LC_VERSION_MIN_IPHONEOS
      uint32_t cmdsize;   // sizeof(struct version_min_command)
      uint32_t version;   // X.Y.Z is encoded in nibbles xxxx.yy.zz
      uint32_t sdk;       // X.Y.Z is encoded in nibbles xxxx.yy.zz
    };

    struct build_tool_version {
      uint32_t tool;      // enum for the tool
      uint32_t version;   // version of the tool
    };

    struct build_version_command {
      uint32_t cmd;       // LC_BUILD_VERSION
      uint32_t cmdsize;   // sizeof(struct build_version_command) +
                          // ntools * sizeof(struct build_tool_version)
      uint32_t platform;  // platform
      uint32_t minos;     // X.Y.Z is encoded in nibbles xxxx.yy.zz
      uint32_t sdk;       // X.Y.Z is encoded in nibbles xxxx.yy.zz
      uint32_t ntools;    // number of tool entries following this
    };

One advantage to the linker flag approach is that we could add more flags like -machosdkver etc as the need arises. Whereas GODARWIN could only support at-most platform and minver (via GODARWIN=ios or GODARWIN=ios11.0.0).

[gopherbot]

Change https://golang.org/cl/168321 mentions this issue: cmd/link/internal/ld: copy LC_VERSION_MIN_* commands to Go object file

[gopherbot]

Change https://golang.org/cl/168320 mentions this issue: cmd/link: enable bitcode build for iOS/tvOS/watchOS

[eliasnaur]

You can check out my current progress at the two CLs posted above. I got LC_VERSION_MIN_* and LC_BUILD_VERSION copying to work, and only need tests.

[gopherbot]

Change https://golang.org/cl/168459 mentions this issue: cmd/link/internal/ld: extract Mach-O load command parsing

[liyimeng]

How this should be used? Just update my gomobile package? I did so, but Xcode still fail and complains the framework is not bitcode enabled.

[liyimeng]

go.o does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[tmm1]

You would need to compile golang from master and use that.

[liyimeng]

aha, thanks! I though run
go get -u golang.org/x/mobile/cmd/... is enough. Let me give it another try.

[tmm1]

I've been using the new golang patches on master for a few days and have been able to run my golang code in the simulator and on tvOS/iOS devices. (Note that I am still using go directly to generate lib.a as outlined above instead of leveraging gomobile).

Now I'm trying to submit an app to the store and running into some weird issues. When submitting to the App Store, I get an email saying processing was stopped:

App Store Connect: Processing stopped for MyApp
While processing your iOS app errors occurred in the app thinning process, and your app couldn’t be thinned. If your app contains bitcode, bitcode processing may have failed. Because of these errors, this build of your app will not be able to be submitted for review or placed on the App Store. For information that may help resolve this issue, see Tech Note 2432.

I tried the debugging steps linked in the Tech Note to "rebuild from bitcode" in Xcode, and am getting a strange error about dSYMs missing in IDEDistributionPipeline.log:

ipatool failed with an exception: #<CmdSpec::NonZeroExcitException: Command exited with pid 71395 exit 1:
/Applications/Xcode.app/Contents/Developer/usr/bin/symbols -failIfMissingDsym -symbolsPackageDir /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/ipatool20190324-69852-vd6ut4/thinned-out/arm64/Payload/MyApp.app /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/ipatool20190324-69852-vd6ut4/thinned-out/arm64/Payload/MyApp.app/MyApp
Stdout:
    ERROR: Did not find dSYM for MyApp
    /var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/ipatool20190324-69852-vd6ut4/thinned-out/arm64/Payload/MyApp.app/MyApp [arm64, 0.001634 seconds]:

Stderr:
    error: Could not load symbol information for 22D519D5-0C78-3905-8681-93FE91325191.  Symbols file will contain no data.

>
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:248:in `run'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2473:in `block in CompileOrStripBitcodeInBundle'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2396:in `each'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2396:in `CompileOrStripBitcodeInBundle'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2628:in `ProcessIPA'
    /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:3445:in `<main>'

I noticed this:

https://github.com/golang/mobile/blob/6621de06e1e9d52c38a8c542770bdbeac9ecfef6/cmd/gomobile/build_iosapp.go#L74-L75

So I tried again with -ldflags=-w, and that worked! I assumed that wasn't necessary anymore after bcdbd58 but I guess it is still required.

I am now able to submit builds up to App Store:

App Store Connect: Version 2019 (3.25.619) for MyApp has completed processing.