remaster: a reckoning of weird ideas (+ add gRPC runtime service)

[vito]

This is a doozie of a PR. I'll summarize each change into sections. This is a case where one thing led to another, to another, to another, and I'd rather just preserve the history as-is than butcher it too much with rebasing.

The north star here was to support a gRPC service interface for remote runtime usage for Bass Loop. This led me down a Protobuf adventure that involved a few intermediate iterations where I went back and forth (e.g. making bass.lock protobuf based, then not, then back again).

This path was a bit of a reckoning: nearly every tentative decision I made in the past where I thought "this may blow up in my face someday" subsequently did. I'm happy with the end result: now that I had a real use case driving evidence for each of these changes, I felt much more comfortable with the trade-offs.

no more duck-typing JSON

Previous versions of Bass relied on "duck typing" JSON structures to re-hydrate them into complex Bass values - chiefly thunks, thunk paths, host paths, and other types of paths. As in, "if a JSON object looks like a thunk, i.e. has all the necessary thunk fields, it's a thunk!" This approach worked pretty well but it was obviously open to ambiguity and false positives - if someone, for whatever reason, wanted to emit a regular JSON object in the same encoded shape as a Bass value, they were out of luck.

This is now removed. Instead, thunks, thunk paths, and other complex types encode using Protobufs. Complex values now have a well-defined schema. JSON values are no longer magically converted into rich Bass types based on their shape. They may still encode to JSON (using Protobuf's JSON encoding), but can only decode into a Bass value when the other end knows exactly what type they want. This is how bass --export works, for example: it takes a JSON object and tries to unmarshal it either into a Thunk (image export) or a ThunkPath (artifact export), rather than reading a Bass value and switching on its type.

Huzzah!

.bass is no longer implicitly added to the command path during (load)

When initially adding support for (use) and (load) I made the decision to use this syntax for loading the foo.bass module, a sibling to the current module:

(use (*dir*/foo))

The motivation was that (use (*dir*/foo.bass)) felt a little less elegant. Comparing to Ruby for example, you say require "foo" not require "foo.rb". Incidentally, you do say load "foo.rb", but I see use as closer to require than load in that it is the go-to function for loading dependent modules.

The weird thing was if you took that same (*dir*/foo) and passed it to (run) instead, Bass wouldn't append .bass to the path. This forced a divide: use .bass for libraries, and don't use it for scripts. This matches my expectations - I prefer scripts to not have a file extension so they can be refactored into other languages without breaking all the call sites.

However it meant that the thunk (*dir*/foo) could not know what file it was referring to until it was used. This is problematic in the case of (fs/foo) where fs is a Go fs.FS backed filesystem and we need to encode it into a message, which is necessary for calculating the thunk's name (a base64-encoded sha256 checksum of the thunk).

So: the magic is gone. When loading a Bass module, you just give the exact filename.

This is a slight departure from Ruby, but really (use) was never anything like require. In Ruby, require takes the given string and traverses your library load path until it finds what you're looking for. In Bass, thunks are always fully qualified. In that sense I think it's much clearer to see exact filenames.

Take this for example:

; load dependencies
(use (.git (linux/alpine/git))
     (*dir*/bass/bass.bass)
     (git:github/vito/bass-loop/ref/main/bass/github.bass))

The second module definitely has a bit of stutter, but some of that is owed to the fact that this is the helper module for Bass itself. Moving briskly past that to the last module, it becomes much clearer that we're simply specifying a file within the repository rather than using some sort of advanced package loading scheme. It also looks much different from the first module, which takes an image reference as an argument ((linux/alpine/git)).

All in all I'm happy with this change as it makes module loading much easier to understand without context, and means (*foo*/bar) always says what it means.

bass.lock is now protobuf-encoded

Building on the introduction of Protobufs, files written using the memo system are now Protobuf-encoded, rather than JSON-encoded. This is a backwards-incompatible change; existing lock files will need to be re-created. I don't want or expect this sort of breaking change to happen again in the future, for obvious reasons (it breaks loading modules!). Now that we're using Protobuf there are well-defined ways for maintaining backwards compatibility - not that it's a silver bullet.

the Bass runtime is no longer a runtime

This is just a internal refactor with no external impact. I had some vague ideas which provide some context on this change, but in the end I just changed the Bass runtime to bass.Session and removed Load from the bass.Runtime interface.

scopes now encode and decode from JSON in deterministic order

A small quality-of-life change; rather than following lexicographical key order (Go default behavior), JSON encoding/decoding now matches Scope semantics in respecting key definition order.

scopes no longer underscorize <-> hyphenate keys

Previously, {:foo-bar 1} would encode to {"foo_bar": 1}, and vice versa. This was always an obvious YOLO. The motivation was to make using Concourse resource-like interfaces feel more "native" because Lisp uses hyphens as a symbol separator. This subsequently blew up in my face on multiple occasions, and is now removed. So if you've got any places pattern-matching JSON keys using hyphens, you'll need to change those to underscores.

concatenating thunks/values uses lists rather than duck typing {:str [vals]}

Previously, if you wanted to pass a CLI flag like --foo=./thunk/path you had to wrap it in a structure like {:str ["--foo=" path]} - exported by stdlib as (str-thunk). Now you just pass the list value instead, and lists are given special treatment in args/env because those have to resolve to strings anyway.

add gRPC runtime client/server implementation

Previously, Bass Loop used its own built-in runtimes to communicate to SSH-forwarded services. This meant the runtime code was running server-side, which had a few downsides:

1. Because the Buildkit runtime injects a precompiled shim executable into every thunk, this involved a ~2.5MB network transfer for each call.
2. Users could only use runtimes that Bass Loop knew about ahead of time.
3. Runtime code ran separately from the runner's machine, meaning it couldn't e.g. use the OS X keychain for Docker Hub authentication.

With gRPC, Bass Loop can just be a dumb server that talks to a fully remote runtime, solving all of these problems!

---

All in all, the above changes bring Bass much closer to a stable release - I had a feeling a lot of these problems would surface someday, and they finally did!