-
Notifications
You must be signed in to change notification settings - Fork 12
Qi Meeting Jan 19 2024
Qi Meeting Jan 19 2024
Adjacent meetings: Previous | Up | Next
We triaged many outstanding issues to assess priority and relevance, discussed coding style conventions, the modular compiler architecture and extensibility by users, integrating the new benchmarking suite into CI workflows, and Racket platform issues like expander and macro stepper functionality.
We officially released Qi 4 last week. We had to take a few shortcuts on the way to release and also had identified many avenues of exploration along the way. There were a lot of issues to revisit and prioritize.
In a recent PR we noticed that some contribution to the diff was simply from indenting changes due to editors being differently configured. This brought up style conventions for Qi and ways to automate them.
Historically, we've taken a pretty laid back approach to style. To encourage contributors to maintain code they contribute, it seems to make sense to let people use whatever style they are comfortable with. It also doesn't seem too hard to adjust one's style to maintain a piece of code written by someone else when necessary.
This has worked well for our purposes so far, but there are a few issues to balance:
- Different editor configurations could cause gratuitous diffs due to indentation
- New contributors might find it easier to work with the code if there were at least some conventions
We felt that we could take some steps to gain the benefits of conventions without imposing too much stricture:
- Standardize on indentation, especially for Racket platform interfaces like syntax-parse utilities. We could accomplish this by including indentation config for editors like Emacs in the Qi repo or on the wiki, which could be used as "dir locals". Of course, we would want to include config for other editors like DrRacket and Vim, as well.
- Standardize on easy conventions like Racket's square bracket convention for forms like
condand in pattern-based macros. - Write a style guide mentioning these indentation rules, which could give us a starting point in case we want to define other conventions in the future.
At present, we do not expect Qi to be strict about such style conventions. It seems more important to have requirements around clarity and documentation rather than emphasizing one particular style.
We reviewed the proof-of-concept modular compiler architecture and agreed that it was a good starting point for prototyping deforestation of racket/list. We'd like the interface to adding a new compiler pass to simply be define-pass instead of requiring a separate registration step in register-pass, but as it needs to be done in the syntax phase, it may not be so straightforward.
With the new architecture, it's possible to benchmark the code with and without certain compiler passes, for instance, with deforestation against without it.
The point of deforesting racket/list isn't just to extend deforestation to more APIs, but also to generalize the interface further. Once it is sufficiently general, it would be easier to apply it even more broadly to other data structures and settings. The form this generalization may take would include extending syntax classes with new attributes.
The ability for "users" to define custom compiler passes wouldn't be only for bespoke data types, but even for purposes unrelated to optimization, such as debugging. We already have some good debugging tools and can add a lot more functionality using macros, but doing it at the compiler level would allow us to essentially just "flip a switch" (i.e. require the new pass) and have existing code be annotated with debugging information or have its semantics modified to have effects (such as printing) that are useful for debugging purposes. This would not require us to change the source code, so in this respect, it's more powerful, but also, of course, harder to implement, requiring knowledge of some compiler internals.
Of course, the current modular architecture is only intended for "internal" use of Qi developers rather than for general users, so such specialized applications seem viable. For general user-facing extensibility (including for bespoke datatypes) we would likely look to options like language composition.
One concern is that doing it this way may mean that we would not bundle deforestation with the main Qi library, as some of these syntax classes may conflict with more general versions that would be needed for racket/list deforestation, and syntax classes are not composable. We would need to work out a way to resolve such conflicts, for instance, either subsequent requires would override any passes with lower priority (indicated simply using numbers in the current interface), or we could introduce a "tag" so that for any set of passes sharing the same tag, only the one with the highest priority would be retained, i.e. passes with the same tag would be mutually exclusive.
Still, it may be acceptable to expect users to do an additional require to get such list-based optimizations. In any case, we would want to abstract the compiler from the user's perspective, so that, for instance, such requires would look like (require qi/list) (to get list-oriented functionality, including optimizations), rather than something like (require qi/pass/deforest).
One of the great things about the Racket ecosystem is the space-age technologies that power Racket as a platform for languages. This includes syntax-parse, arguably the most powerful macro system available for any language, Racket's macro expander, which organizes expansion into distinct phases, Syntax Spec, which enables the stratified DSL architecture that allows Qi to have an optimizing compiler, and the rich suite of tools built around exploration of syntax, including the Macro Stepper.
As Qi uses Syntax Spec which is at the cutting edge of Racket's macro technology, along with binding spaces, which is a very new feature in Racket that was introduced for use in the Rhombus project, we've found in recent times that some platform tools in the Racket ecosystem do not yet offer full support for these new patterns and paradigms. We talked about some of these features that we missed, and felt that we should begin conversations with other members of the community so that they are aware of these emerging needs.
Some features that could be added to Syntax Spec that would be useful in Qi include:
- Preserving access to source syntax through expansion so that compiler errors can implicate the appropriate source expressions
- Grammar-aware syntax tree traversal, so that compiler optimizations can be performed on subtrees in the syntax that are full, legitimate DSL syntax. We felt that while this would be very useful, it may also be a fairly nontrivial effort.
There are many things in the Macro Stepper that are hardwired to use of the Racket macro expander. But with the addition of Syntax Spec languages, such hosted languages may also have stages of expansion and compilation before they delegate to the Racket expander, and it would be extremely helpful to be able to have visibility into each of these stages just as if they were steps of ordinary (Racket) macro expansion.
Some time ago, we used the emit-local-step API to report before -> after syntax transformations, but unfortunately, this API does not provide a way to capture the context of expansion. Consequently, the Macro Stepper does not convey many clues about the hierarchy of these syntax transformations.
It sounded like one option here could be to identify whether there may already be internal APIs that allow us to provide this tree-structured context, and if so, to encourage Ryan and other core team members to standardize and publicly provide these APIs.
Another thing we noted is that bindings when viewed in the Macro Stepper or Syntax Explorer do not indicate the binding space they are defined in. This too could be a useful feature.
Qi already uses both of these features, so it could serve as a good testing ground for the addition of these features.
The new benchmarking suite is still in an early stage of development, but it is already very capable. We felt it would be valuable to incorporate it into our development workflows at this stage to have a simple feedback loop in place.
Some developments we are interested in exploring down the line include:
- decoupling the benchmarking framework from the programs being benchmarked
- the programs should live in their respective source repos, not in the benchmarking suite repo
- a general interface needs to be defined that would allow it to be used with libraries other than Qi
But in its current form it's almost usable in our CI workflows. We got it to work using the sample profile, and then modified the workflow scripts to run a more rigorous profile. That resulted in the benchmarks report being pushed to Qi's GitHub Pages site.
The next steps to getting this into our development workflows include:
- Writing a simple CLI to allow setting the profile via CLI arguments
- Adding an info.rkt file to make it a Racket package (even if not published to the Package Index), allowing raco to manage its dependencies and simplifying installation in CI workflows
We went through all of the TODOs we'd been carrying over in these meeting notes and decided what to do about each of them -- mostly, Sid will create issues for any of them that are still relevant and clear any from the "Next Steps" that aren't actually something that we're immediately working on.
(Some of these are carried over from last time)
- Incorporate the new benchmark suite into the SDK (including Makefile targets) and CI workflows
- Review whether we can continue optimizing a single syntax node at many levels
- Investigate ways to extend the compiler (e.g. language composition or compiler macros) and use that to implement
qi/list(deforestingracket/list) as a proof of concept. - Preserve the source syntax through expansion in Syntax Spec.
Dominik, Sid
Home | Developer's Guide | Calendar | Events | Projects | Meeting Notes