Make libsyntax available on crates.io

[nrc]

This has previously been attempted (see #41732 for example) with the goal of making libsyntax a stable crate. That turned out to be difficult (although it might be easier now since some work has been done in this area). It would be useful to be able to use libsyntax via crates.io, and if it is nightly-only, I think this should not be too difficult.

Ideally, a bot would publish once per week or even every day.

The motivation here is so that rustfmt can use the crates.io version of libsyntax and then it would not be broken (which in turn breaks the RLS) every time there is a breaking change to libsyntax (which is fairly often, it turns out).

cc @alexcrichton @Mark-Simulacrum

[CodeSandwich]

If API got compiler agnostic and somewhat stable, could libsyntax be used in procedural macros as Syn alternative? I imagine, that it would make macro's compatibility with user code forever on par with rustc (at least in version chosen as dependency), no more Syn chasing syntax changes?

[nrc]

If API got compiler agnostic and somewhat stable

This is a far more ambitious goal and would take a lot more effort. I'm not entirely sure if it is even possible either - the goals are very different (stable API vs always evolving API and 'just a parser' vs an entire compiler frontend).

[topecongiro]

We have rustc-ap-syntax crate.

[nrc]

This is basically done now, but I think @alexcrichton had some smaller things he thought could be addressed.

[alexcrichton]

Yes I've set this up to be automatically published via this script which is just running locally on my machine. There's a few issues I think we'll want to change though:

• First the call to __build_diagnostic_array! needs to work outside the context of the build system. Right now it requires a few env vars to be defined but we should either lift that restriction (make it a noop if they're not defined) or conditionally only compile it in for rustc itself.
• Next libsyntax links to the std_unicode crate which has functions governed under the rustc_private feature gate. That means libsyntax doesn't actually mention the feature gate because in rustc it's compiled with -Z force-unstable-if-unmarked. What we should do, however, is reserve the feature gate rustc_private exclusively for the compiler and rename the features in std_unicode.

After that the auto-publish script purely only has to edit the Cargo.toml which it'll always have to do anyway. I don't think we'll get away from that difficulty in publishing for quite some time. The next steps would be to compile libsyntax on stable (at least conditionally). That's farther out though and less necessary for immediate steps.

[Mark-Simulacrum]

I believe both of the issue above have been resolved (unicode_internals is the feature gate, and the environment variable is no longer needed) so I'm going to close this as we can likely eliminate the hacks in the autopublisher at any point pretty much (I've gone ahead and filed alexcrichton/rustc-auto-publish#10).