Deforest All the Things #179
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Deforest all variants of cad*r: - car - cadr - caddr - cadddr - caddddr - cadddddr Deforest (using the same underlying implementation) list-ref as well.
- split syntax matching from syntax production - improve naming of syntax classes - remove unused template variables
- preliminary splitting of the compiler into separate modules for separate passes - update tests to reflect new paths
- rename compiler "passes" subdirectory to "compiler" - strip the passes modules file name pass- prefix
…s fixes and tests for them.
…ntax and do not provide them.
- scribblings for qi/list module - scribble the new literals for matching in deforestation pass - ensure for-label bindings in the generated documentation - new bindings.rkt module
…ler meeting on 2024-06-21. - add detailed explanation for inline-consing syntax - use Racket's conventions for parentheses - add description of fsp-, fst-, and fsc- prefixes - move define-and-register-deforest-pass and related to separate module, add comments
…on (CPS) implementation.
countvajhula
force-pushed
the
deforest-all-the-things
branch
from
52e6923
to
ca711e5
Compare
This form is intended to express any deforestable expression, allowing the core language to express deforestation semantics, which, formerly, we were not able to do within the language and thus resorted to matching, and optimizing, host language syntax, leading to a "host" of problems. This new form is groundwork to enable compiler optimizations being defined purely on the core language, thus representing a clean boundary, or contract, between Qi and the host (Racket). The initial implementation here just introduces the form, and code generation for `filter` specifically, as a proof of concept for the more generic and extensible planned implementation. See the meeting notes for more, e.g.: https://github.com/drym-org/qi/wiki/Qi-Meeting-Jun-21-2024#implementing-it
The function positions in deforestable operations are Racket expr positions, but we want them to be Qi floe positions instead. This modifies the code generation step to recursively invoke codegen on these nested floe positions.
Based on recent discussions, as a general maxim: Our core language should be rich enough to express desired optimizations. Initially, as this wasn't the case, we were performing deforestation by matching host language forms. This of course meant that we were constrained to Racket syntax in such functional operations. Now that we are broadening our core language to express deforestation, in keeping with the above maxim, we would prefer to support Qi syntax in function positions in these operations. Towards this goal, this new syntax for the `#%deforestable` core form introduces support for `floe` positions. Right now, it simply segregates arguments into `expr` and `floe` positions so that these are appropriately expanded. The code generation still matches the name of the functional list transformation (e.g. `map`, `filter`) and "hardcodes" the known invocation of the corresponding underlying operation. Eventually we hope to make deforestation user-extensible to arbitrary functional list (at least) operations. At that stage, we wouldn't have this kind of standard information that we could leverage during code generation, so we will need to modify the syntax of `#%deforestable` to encode enough information to be able to perform appropriate code generation for arbitrary user-defined operations. We are not there yet :)
As the deforestation pass generates escaped Racket, we need to compile any higher-order `floe` positions in the fusable list operations at this stage, since the regular code generation step at the end of compilation would not operate on these resultant escaped expressions.
Now that these semantics tests are simply testing the behavior of newly defined Qi forms rather than ensuring low level rewriting of host language syntax, they can have ordinary unit tests validating their semantics, just like any other built-in Qi macros.
We now deforest via the `#%deforestable` core form and don't need host language (yet provided by Qi) bindings for this purpose anymore.
Use Qi equivalents as lambda is no longer valid in this position (at least until/unless #177 is merged).
We use Qi's `range` now. Also, add a Makefile target to build the SDK to reproduce such build errors in future.
Coverage was reporting this case uncovered
In the provisional syntax of Qi's `range`, we expect the range to be specified syntactically, as it compiles to a lambda accepting no arguments.
- use left-threading in most tests - one test using right-threading to validate deforestation is invariant to threading direction - use `range` with syntactically specified arguments; remove tests using templates - consolidate `deforest-pass` tests since we no longer have a separate test suite for individual applications of the deforestation rewrite rule (should we?)
When a nested form has a different chirality (threading direction)
than a containing form, normalization would not collapse them, but
deforestation may not care about the difference.
Possible approaches:
A. Introduce normalization rules designed to detect
when change of chirality is irrelevant.
B. Look for patterns in the deforestation pass involving
differing threading directions
Probably (A) is the right approach, and we could introduce a set of
chirality normalization rules that "trim" forms on either end of a
nested form which could be collapsed into the containing form. This
would include anything that isn't a host language function
application (which is the only case where chirality matters).
Actually, thinking again, chirality is already represented in the core
language simply as the presence of a blanket template in a function
application form, and nested threading is already collapsed by
normalization, so, I'm not sure anymore why this test is
failing ¯\_(ツ)_/¯
Deforest within the core language
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary of Changes
Comprehensive deforestation of list operations within Qi (integration branch).
This introduces a
qi/listcollection that includes comprehensive functional list operations resembling the ones inracket/list, but which exhibit Qi syntax even in higher order positions, and are deforested (i.e. they avoid building intermediate collections on the way to constructing the result).There is already basic deforestation on the main branch as of Qi 4. This PR improves the architecture of deforestation so that it is:
racket/listas a representative goal)racket/listis deforested, that will give us an idea of what the core operations really are which can express all of the others, and what a good interface to these core operations would look like that could be exposed to users for deforesting custom list operations.As we continue on this work, it will hopefully also give us some idea of whether and how deforestation could be extended to datatypes more general than lists (for instance, vectors and even Qi's core values).
Public Domain Dedication
(Why: The freely released, copyright-free work in this repository represents an investment in a better way of doing things called attribution-based economics. Attribution-based economics is based on the simple idea that we gain more by giving more, not by holding on to things that, truly, we could only create because we, in our turn, received from others. As it turns out, an economic system based on attribution -- where those who give more are more empowered -- is significantly more efficient than capitalism while also being stable and fair (unlike capitalism, on both counts), giving it transformative power to elevate the human condition and address the problems that face us today along with a host of others that have been intractable since the beginning. You can help make this a reality by releasing your work in the same way -- freely into the public domain in the simple hope of providing value. Learn more about attribution-based economics at drym.org, tell your friends, do your part.)