From 44c5cba0dbf8cc0df0bea42aee819fed163d75a8 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 14 Mar 2024 12:59:33 -0700 Subject: [PATCH 01/16] doc: Effect locality --- qi-doc/scribblings/field-guide.scrbl | 2 + qi-doc/scribblings/principles.scrbl | 109 +++++++++++++++++++++++++++ 2 files changed, 111 insertions(+) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 53a1726c..db7916db 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -423,6 +423,8 @@ So, to reiterate, while the behavior of @emph{pure} Qi flows will be the same as If you'd like to use Racket's order of effects in any flow, @seclink["Using_Racket_to_Define_Flows"]{write the flow in Racket} by using a wrapping @racket[esc]. +See @secref["Effect_Locality"] for more insights into Qi's handling of effects. + @section{Effectively Using Feedback Loops} @racket[feedback] is Qi's most powerful looping form, useful for arbitrary recursion. As it encourages quite a different way of thinking than Racket's usual looping forms do, here are some tips on "grokking" it. diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index 09afcdde..4e3f4c83 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -3,12 +3,20 @@ scribble-abbrevs/manual scribble/example racket/sandbox + scribble/core scribble-math @for-label[qi racket]] @(use-mathjax) +@; based on scribble-abbrevs/latex +@(define (definition term . defn*) + (make-paragraph plain + (list + (bold "Definition") + (cons (element #f (list " (" (deftech term) "). ")) defn*)))) + @title{Principles of Qi} After many patient hours meticulously crafting Qi flows, you may find that you seek a deeper understanding; insight into guiding principles and inner workings, so that you can hone your skills on firmer ground. @@ -59,6 +67,107 @@ Consider this example: There are six @tech{flows} here, in all: the entire one, each component of the thread, and each component of the tee junction. +@section{Effect Locality} + +Qi programs provide weaker guarantees on @seclink["Order_of_Effects"]{order of effects} than do otherwise equivalent Racket programs. + +For instance, this Qi flow: + +@codeblock{ + (~>> (filter my-odd?) (map my-sqr)) +} + +is roughly equivalent to this Racket expression: + +@codeblock{ + (lambda (vs) + (map my-sqr + (filter my-odd? vs))) +} + +But if @racket[my-odd?] and @racket[my-sqr] exhibit any side effects, such as printing their inputs to the screen, then the behavior of these two expressions is not quite the same. Racket guarantees that @emph{all} of the @racket[my-odd?] effects will occur before @emph{any} of the @racket[my-sqr] effects. Qi provides a more minimal (and ultimately very simple) guarantee that can be summarized as (1) effects are only defined in association with function invocations, and (2) they will occur at the same time as the function invocation with which they are associated. We call this property "effect locality." + +To understand what this means, we will need to develop some concepts. + +@subsection{Functional and Effective} + +First, regarding the definability of effects, as far as Qi is concerned, they are only well-defined in connection with some @tech{flow}, and are not independently conceivable. + +@definition["Associated effect"]{If a flow @${f} either includes an effect @${e} in its primitive definition or has one declared using @racket[effect], then @${e} is said to be an effect "on" @${f}. We denote an arbitrary effect on @${f} by @${ε(f)}.} + +In the case where we use the @racket[effect] form on its own as in @racket[(effect displayln)], the implicit associated function is the identity flow, @racket[_]. + +@subsection{Upstream and Downstream} + +We already saw how a flow can be thought of as a @seclink["Flows_as_Graphs"]{directed graph}. This naturally suggests that some flows are upstream (or downstream) of others in terms of this directionality. Let's define this relation more precisely, as it will be useful to us. + +@definition["Upstream and downstream"]{A flow invocation @${f} is @deftech{upstream} of another invocation @${g} if the output of @${f} is @emph{necessary} to determining the input of @${g}. Conversely, @${g} is @deftech{downstream} of @${f}. We will denote this relation @${f < g}.} + +Note that this definition relates flow @emph{invocations} rather than @tech{flows} themselves. For now, we need not worry about this distinction, but we will soon see why it matters. + +In terms of the @tech{paths} that could be traced over a flow, the ordering implied by @racket[~>] naturally shows us many members of this relation: flows that come later in the sequence in a @racket[~>] form are downstream of those that come earlier, because the output of earlier flows is needed to determine the input to later flows. + +In the above example, @racket[filter] and @racket[map] are obviously ordered by @racket[~>] in this way, so that @racket[(filter my-odd?)] is upstream of @racket[(map my-sqr)]. But it's not so obvious how @racket[my-odd?] and @racket[my-sqr] should be treated. These are employed "internally" by the higher-order flows @racket[filter] and @racket[map], and are not directly ordered by the @racket[~>] form. Should @racket[my-odd?] be considered to be upstream of @racket[my-sqr] here? + +This is where the distinction between flows and flow invocations comes into play. In fact, not all invocations of @racket[my-odd?] are upstream of any particular invocation of @racket[my-sqr]. Rather, specific invocations of @racket[my-sqr] that use values computed by individual invocations of @racket[my-odd?] are downstream of those invocations, and notably, these invocations involve the individual elements of the input list rather than the entire list, so that the computational dependency expressed by this relation is as fine-grained as possible. + +For instance, for an input list @racket[(list 1 2 3)], @racket[(my-odd? 1)] is @tech{upstream} of @racket[(my-sqr 1)], and likewise, @racket[(my-odd? 3)] is @tech{upstream} of @racket[(my-sqr 3)]. + +That brings us to the guarantee that Qi provides in this case (and in general). + +@subsection{Qi's Guarantee on Effects} + +@definition["Effect locality"]{For @tech{flow} invocations @${f} and @${g} and corresponding effects @${ε(f)} and @${ε(g)}, + +@$${f < g ⇒ ε(f) < ε(g)} + +where @${<} on the left denotes the relation of being upstream, and @${<} on the right denotes one effect happening before another. +} + +This has a few implications of note. + +@subsubsection{The @racket[effect] Form} + +First, for the @racket[effect] form, this implies that Qi considers an effectful flow @${f} performing an effect @${e} to be indistinguishable from a flow @racket[(effect e f′)], where @${f′} is @${f} without @${e} (and therefore pure), and effects declared in this way will never be separated from the associated flows. Thus, Qi @seclink["Separate_Effects_from_Other_Computations"]{encourages writing pure functions} while preserving the intuitive association of effects with functions. + +@subsubsection{The @racket[esc] Form} + +Qi will not optimize a flow that is wrapped with @racket[esc]. Thus, such flows will exhibit @seclink["Racket_vs_Qi"]{Racket's order of effects} (which, as we'll discuss below, also satisfies the requirements of locality). + +For more on how @racket[esc] is handled, see @secref["Using_Racket_to_Define_Flows"]. + +@subsubsection{Truncating Effects} + +Next, for a flow like this one: + +@racketblock[ + (~>> (filter my-odd?) (map my-sqr) car) +] + +… when it is invoked with a large input list, Qi in fact only processes the very first value of the list, since it determines, at the end, that no further elements are needed in order to generate the final result. This means that all effects on would-be subsequent invocations of @racket[my-odd?] and @racket[my-sqr] would simply not be performed. Yet, locality is preserved here, since the @techlink[#:key "effect locality"]{defining implication} holds for every flow invocation that actually happens. Locality is about effects being guided by the @emph{necessity} of @techlink[#:key "associated effect"]{associated} computations to the final result. + +@subsubsection{A Natural Order of Effects} + +By being as fine-grained as possible in expressing computational dependencies, and in tying the execution of effects to such computational dependencies, @tech{effect locality} is in some sense the minimum well-formed guarantee on effects, and a natural one for functional languages to provide. + +@subsection{Racket vs Qi} + +In the earlier example, with an input list @racket[(list 1 2 3)], Racket's order of effects follows the invocation order: + +@racketblock[ + (my-odd? 1) (my-odd? 2) (my-odd? 3) (my-sqr 1) (my-sqr 3) +] + +Qi's order of effects is: + +@racketblock[ + (my-odd? 1) (my-sqr 1) (my-odd? 2) (my-odd? 3) (my-sqr 3) +] + +Either of these orders @emph{satisfies} @tech{effect locality}, but as we saw earlier, Racket guarantees something more than this minimum, or, in mathematical terms, Racket's guarantees on effects are @emph{stronger} than Qi's. + +In principle, this allows Qi to offer @seclink["Don_t_Stop_Me_Now"]{faster performance} in some cases. + @section{Flowy Logic} Qi's design is inspired by buddhist śūnyatā logic. To understand it holistically would require a history lesson to put the sunyata development in context, and that would be quite a digression. But in essence, sunyata is about transcension of context or viewpoint. A viewpoint is identifiable with a logical span of possibilities (@emph{catuṣkoṭi}) in terms of which assertions may be made. Sunyata is the rejection of @emph{all} of the available logical possibilities, thus transcending the very framing of the problem (this is signified by the word @emph{mu} in Zen). This kind of transcension could suggest alternative points of view, but more precisely, does not indicate a point of view (which isn't the same as being ambivalent or even agnostic). This idea has implications not just for formal logical systems but also for everyday experience and profound metaphysical questions alike. From 0326ee03383308e4c20d45e97ae71bee0bb34385 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 14 Mar 2024 15:53:17 -0700 Subject: [PATCH 02/16] doc: use LaTeX syntax (cr) --- qi-doc/scribblings/principles.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index 4e3f4c83..f62dff6a 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -119,7 +119,7 @@ That brings us to the guarantee that Qi provides in this case (and in general). @definition["Effect locality"]{For @tech{flow} invocations @${f} and @${g} and corresponding effects @${ε(f)} and @${ε(g)}, -@$${f < g ⇒ ε(f) < ε(g)} +@$${f \lt g \implies \epsilon(f) \lt \epsilon(g)} where @${<} on the left denotes the relation of being upstream, and @${<} on the right denotes one effect happening before another. } From 9d837054a8e4f68fe93cb81531f133cba62fd164 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 14 Mar 2024 15:53:35 -0700 Subject: [PATCH 03/16] doc: reproduce referenced example for convenience (cr) --- qi-doc/scribblings/principles.scrbl | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index f62dff6a..a8e72bc2 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -152,7 +152,19 @@ By being as fine-grained as possible in expressing computational dependencies, a @subsection{Racket vs Qi} -In the earlier example, with an input list @racket[(list 1 2 3)], Racket's order of effects follows the invocation order: +In the earlier example, reproduced here for convenience: + +@racketblock[ + (lambda (vs) + (map my-sqr + (filter my-odd? vs))) +] + +@racketblock[ + (~>> (filter my-odd?) (map my-sqr)) +] + +… with an input list @racket[(list 1 2 3)], Racket's order of effects follows the invocation order: @racketblock[ (my-odd? 1) (my-odd? 2) (my-odd? 3) (my-sqr 1) (my-sqr 3) From 9f47b724c9dc5332160eef15d5f0a49860fc24ac Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 14 Mar 2024 16:00:52 -0700 Subject: [PATCH 04/16] doc: Schrodinger's probe, on how use of `probe` can affect effects --- qi-doc/scribblings/field-guide.scrbl | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index db7916db..749b97bb 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -99,10 +99,12 @@ To use it, first wrap the entire expression @emph{invoking} the flow with a @rac @defform[(probe flo)] @defidform[readout] )]{ - @racket[probe] simply marks a @tech{flow} invocation for debugging, and does not change its functionality. Then, when evaluation encounters the first occurrence of @racket[readout] within @racket[flo], the values at that point are immediately returned as the value of the entire @racket[flo]. This is done via a @tech/reference{continuation}, so that you may precede it with whatever flows you like that might help you understand what's happening at that point, and you don't have to worry about it affecting downstream flows during the process of debugging since those flows would simply never be hit. Additionally, readouts may be placed @emph{anywhere} within the flow, and not necessarily on the main stream -- it will always return the values observed at the specific point where you place the readout. + @racket[probe] on its own simply marks a @tech{flow} invocation for debugging, and does not change its functionality. Then, when evaluation encounters the first occurrence of @racket[readout] within @racket[flo], the values at that point are immediately returned as the value of the entire @racket[flo]. This is done via a @tech/reference{continuation}, so that you may precede it with whatever flows you like that might help you understand what's happening at that point, and you don't have to worry about it affecting downstream flows during the process of debugging since those flows would simply never be hit. Additionally, readouts may be placed @emph{anywhere} within the flow, and not necessarily on the main stream -- it will always return the values observed at the specific point where you place the readout. Note that @racket[probe] is a Racket (rather than Qi) form, and it must wrap a flow @emph{invocation} rather than a flow @emph{definition}. The @racket[readout], on the other hand, is a Qi expression and must be placed somewhere within the flow @emph{definition}. + Finally, it's important to know that placing a readout represents a change to the program, and it could mean that the running program you are observing is subtly different from the original one. See @secref["Schrodinger_s_Probe"] to understand this phenomenon. + @racketblock[ (~> (5) sqr (* 2) add1) (probe (~> (5) readout sqr (* 2) add1)) @@ -425,6 +427,29 @@ If you'd like to use Racket's order of effects in any flow, @seclink["Using_Rack See @secref["Effect_Locality"] for more insights into Qi's handling of effects. +@subsubsection{Schrodinger's Probe} + +Another curious thing to watch out for is that use of the @seclink["Using_a_Probe"]{probe debugger} can affect the @seclink["Order_of_Effects"]{order of effects}, as it could suppress optimizations that would be otherwise be performed if the @tech{flow} were unobserved. + +Consider this example: + +@racketblock[ +(define-flow foo + (~> (pass (effect E₁ odd?))) readout (>< (effect E₂ sqr))) + +(probe (foo 1 2 3)) +] + +Here, with the @racket[readout], all the effects E₁ would occur first, followed by all of the E₂ effects. Without the @racket[readout], the flow would be deforested by the compiler to: + +@racketblock[ + (>< (if (effect E₁ odd?) (effect E₂ sqr) ⏚)) +] + +… and the effects E₁ and E₂ would be interleaved. Either order is consistent with @seclink["Effect_Locality"]{locality} but they are @emph{different}. Indeed, the @racket[readout] in this case would not even represent a valid point in the a priori optimized program. + +So it's important to bear in mind that one cannot observe a flow using @racket[probe] without changing the program being observed, a change which in some cases has no observable impact, and which in other cases is significant. But now that you understand this phenomenon, you can develop intuition for the nature of such changes, and how best to use the tool to find the answers you are looking for. + @section{Effectively Using Feedback Loops} @racket[feedback] is Qi's most powerful looping form, useful for arbitrary recursion. As it encourages quite a different way of thinking than Racket's usual looping forms do, here are some tips on "grokking" it. From 0c7094802a7c491b3975aab9b3438e3e8ae3b266 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Fri, 15 Mar 2024 16:42:30 -0700 Subject: [PATCH 05/16] doc: another implication of effect locality --- qi-doc/scribblings/principles.scrbl | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index a8e72bc2..fbf5f233 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -144,7 +144,26 @@ Next, for a flow like this one: (~>> (filter my-odd?) (map my-sqr) car) ] -… when it is invoked with a large input list, Qi in fact only processes the very first value of the list, since it determines, at the end, that no further elements are needed in order to generate the final result. This means that all effects on would-be subsequent invocations of @racket[my-odd?] and @racket[my-sqr] would simply not be performed. Yet, locality is preserved here, since the @techlink[#:key "effect locality"]{defining implication} holds for every flow invocation that actually happens. Locality is about effects being guided by the @emph{necessity} of @techlink[#:key "associated effect"]{associated} computations to the final result. +… when it is invoked with a large input list, Qi in fact @seclink["Don_t_Stop_Me_Now"]{only processes the very first value} of the list, since it determines, at the end, that no further elements are needed in order to generate the final result. This means that all effects on would-be subsequent invocations of @racket[my-odd?] and @racket[my-sqr] would simply not be performed. Yet, @techlink[#:key "effect locality"]{locality} is preserved here, since the @techlink[#:key "effect locality"]{defining implication} holds for every flow invocation that actually happens. Locality is about effects being guided by the @emph{necessity} of @techlink[#:key "associated effect"]{associated} computations to the final result. + +@subsubsection{Independent Effects} + +For a nonlinear flow like this one: + +@racketblock[ + +(~>> (filter my-odd?) + (-< (map my-sqr __) + (map my-add1 __)) + (map my-*) + (foldl + 0)) +] + +… as invocations of neither @racket[my-sqr] nor @racket[my-add1] are upstream of the other, there is no guarantee on the order of effects either, so that the effects associated with @racket[my-sqr] and @racket[my-add1] are in fact independent of one another. + +The effects on @racket[my-sqr] may happen first, or those on @racket[my-add1] may happen first, or they may be interleaved. + +But both of these effects would occur before the one on the corresponding @racket[my-*] invocation, since this is downstream of them both. @subsubsection{A Natural Order of Effects} From 9cdadca2a9c7c6c43950d75439c55a9951a09370 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 18 Mar 2024 23:57:36 -0700 Subject: [PATCH 06/16] doc: add `theorem` Scribble form --- qi-doc/scribblings/principles.scrbl | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index fbf5f233..d6b482de 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -15,7 +15,15 @@ (make-paragraph plain (list (bold "Definition") - (cons (element #f (list " (" (deftech term) "). ")) defn*)))) + (element #f (list " (" (deftech term) "). ")) + defn*))) + +@(define (theorem . thm*) + (make-paragraph plain + (list + (bold "Theorem") + (element #f (list ". ")) + thm*))) @title{Principles of Qi} From 77464a2fdedbe4d39a3efd1ce2a6d66bc638d7e5 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Mon, 18 Mar 2024 23:58:20 -0700 Subject: [PATCH 07/16] doc: distinguish effects from inputs and outputs Also begin to link them as `tech` concepts --- qi-doc/scribblings/principles.scrbl | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index d6b482de..962a211a 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -37,7 +37,7 @@ A @deftech{flow} is either made up of flows, or is a native (e.g. Racket) @seclink["lambda" #:doc '(lib "scribblings/guide/guide.scrbl")]{function}. Flows may be composed using a number of combinators that could yield either linear or nonlinear composite flows. - A flow in general accepts @code{m} inputs and yields @code{n} outputs, for arbitrary non-negative integers @code{m} and @code{n}. We say that such a flow is @code{m × n}. + A flow in general accepts @code{m} @deftech{inputs} and yields @code{n} @deftech{outputs}, for arbitrary non-negative integers @code{m} and @code{n}. We say that such a flow is @code{m × n}. The semantics of a flow is function invocation -- simply invoke a flow with inputs (i.e. ordinary arguments) to obtain the outputs. @@ -105,6 +105,8 @@ First, regarding the definability of effects, as far as Qi is concerned, they ar In the case where we use the @racket[effect] form on its own as in @racket[(effect displayln)], the implicit associated function is the identity flow, @racket[_]. +Note that @tech{effects} are a distinct concept from program @tech{inputs} and @tech{outputs}. + @subsection{Upstream and Downstream} We already saw how a flow can be thought of as a @seclink["Flows_as_Graphs"]{directed graph}. This naturally suggests that some flows are upstream (or downstream) of others in terms of this directionality. Let's define this relation more precisely, as it will be useful to us. From 303ffe56c9b7db573202f368555592a2023b01da Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 19 Mar 2024 00:09:33 -0700 Subject: [PATCH 08/16] Introduce "well ordering" of effects And distinguish it from effect locality. Also introduce a few more concepts like "pure projection" that may be useful in characterizing effects. --- qi-doc/scribblings/principles.scrbl | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index 962a211a..c7fa3118 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -127,13 +127,31 @@ That brings us to the guarantee that Qi provides in this case (and in general). @subsection{Qi's Guarantee on Effects} -@definition["Effect locality"]{For @tech{flow} invocations @${f} and @${g} and corresponding effects @${ε(f)} and @${ε(g)}, +@definition["Well-ordering"]{For @tech{flow} invocations @${f} and @${g} and corresponding effects @${ε(f)} and @${ε(g)}, @$${f \lt g \implies \epsilon(f) \lt \epsilon(g)} -where @${<} on the left denotes the relation of being upstream, and @${<} on the right denotes one effect happening before another. +where @${<} on the left denotes the relation of being upstream, and @${<} on the right denotes one effect happening before another. Such effects are said to be @emph{well-ordered}. } +Well-ordering is defined in relation to a source program encoding the intended meaning of the flow, which serves as the point of reference for program translations. Qi guarantees that effects will remain well-ordered through any such translations of the source program that are undertaken during @seclink["It_s_Languages_All_the_Way_Down"]{optimization}. As we will soon see, this guarantee assumes, and prescribes, that effects employed in flows be @tech{local}. + +@definition["Effect locality"]{@tech{Effects} in a flow F are said to be @deftech{local} if the @tech{output} of F is invariant under all @tech{well-orderings} of effects. Specifically, if a @techlink[#:key "well-ordering"]{well-ordered} program translation causes a program to produce different @tech{output}, then the program contains @deftech{nonlocal} effects.} + +For example, effects that mutate shared state serving as the input to other flows are often nonlocal in this way. The section on @secref["Order_of_Effects"] elaborates on this example. + +We will discuss the practicalities of these in more detail shortly, but first, it's worth noting that although well-ordered effects seem natural for flows, the property does not necessarily hold under arbitrary program translations without an explicit compiler guarantee (as Qi provides). We can see this in terms of the underlying pure flow that is free of effects. + +@definition["Pure projection"]{The pure projection of a flow @${f} is @${f} with all effects removed. We'll denote this @${π(f)}. For flows @${f_{1}} and @${f_{2}}, @${π(f_{1})} is @emph{equivalent} to @${π(f_{2})} if they produce the same @tech{output} given the same @tech{input}.} + +@theorem{For a @tech{flow} @${f}, not every flow @${f′} such that @${π(f′)} is equivalent to @${π(f)} preserves @tech{well-ordering} of effects in relation to @${f}.} + +For instance, the compiler could accumulate all effects and execute them in an arbitrary order at the end of execution of the flow. For at least some subset of local effects (say, effects that simply print their inputs), the output remains the same even if the effects are not well-ordered. + +Locality of effects does not imply well-ordering of effects under program translation, nor vice versa – these are independent. + +In sum, Qi guarantees that the @tech{output} of execution of the compiled program is the same as that of the source program, assuming @tech{effects} are @tech{local}, and further, it guarantees that the effects will be @techlink[#:key "well-ordering"]{well-ordered} in relation to the source program. + This has a few implications of note. @subsubsection{The @racket[effect] Form} @@ -154,7 +172,7 @@ Next, for a flow like this one: (~>> (filter my-odd?) (map my-sqr) car) ] -… when it is invoked with a large input list, Qi in fact @seclink["Don_t_Stop_Me_Now"]{only processes the very first value} of the list, since it determines, at the end, that no further elements are needed in order to generate the final result. This means that all effects on would-be subsequent invocations of @racket[my-odd?] and @racket[my-sqr] would simply not be performed. Yet, @techlink[#:key "effect locality"]{locality} is preserved here, since the @techlink[#:key "effect locality"]{defining implication} holds for every flow invocation that actually happens. Locality is about effects being guided by the @emph{necessity} of @techlink[#:key "associated effect"]{associated} computations to the final result. +… when it is invoked with a large input list, Qi in fact @seclink["Don_t_Stop_Me_Now"]{only processes the very first value} of the list, since it determines, at the end, that no further elements are needed in order to generate the final result. This means that all effects on would-be subsequent invocations of @racket[my-odd?] and @racket[my-sqr] would simply not be performed. Yet, @techlink[#:key "well-ordering"]{well-ordering} is preserved here, since the @techlink[#:key "well-ordering"]{defining implication} holds for every flow invocation that actually happens. Well-ordering is about effects being guided by the @emph{necessity} of @techlink[#:key "associated effect"]{associated} computations to the final result. @subsubsection{Independent Effects} @@ -177,7 +195,7 @@ But both of these effects would occur before the one on the corresponding @racke @subsubsection{A Natural Order of Effects} -By being as fine-grained as possible in expressing computational dependencies, and in tying the execution of effects to such computational dependencies, @tech{effect locality} is in some sense the minimum well-formed guarantee on effects, and a natural one for functional languages to provide. +By being as fine-grained as possible in expressing computational dependencies, and in tying the execution of effects to such computational dependencies, @tech{well-ordering} is in some sense the minimum well-formed guarantee on effects, and a natural one for functional languages to provide. @subsection{Racket vs Qi} @@ -205,7 +223,7 @@ Qi's order of effects is: (my-odd? 1) (my-sqr 1) (my-odd? 2) (my-odd? 3) (my-sqr 3) ] -Either of these orders @emph{satisfies} @tech{effect locality}, but as we saw earlier, Racket guarantees something more than this minimum, or, in mathematical terms, Racket's guarantees on effects are @emph{stronger} than Qi's. +Either of these orders @emph{satisfies} @tech{well-ordering}, but as we saw earlier, Racket guarantees something more than this minimum, or, in mathematical terms, Racket's guarantees on effects are @emph{stronger} than Qi's. In principle, this allows Qi to offer @seclink["Don_t_Stop_Me_Now"]{faster performance} in some cases. From bb4519083c23a2ab5d09df012c8f9ac14f8d49d6 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 19 Mar 2024 00:11:48 -0700 Subject: [PATCH 09/16] doc: clarify independent effects a bit --- qi-doc/scribblings/principles.scrbl | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index c7fa3118..ccdf7f0f 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -187,9 +187,7 @@ For a nonlinear flow like this one: (foldl + 0)) ] -… as invocations of neither @racket[my-sqr] nor @racket[my-add1] are upstream of the other, there is no guarantee on the order of effects either, so that the effects associated with @racket[my-sqr] and @racket[my-add1] are in fact independent of one another. - -The effects on @racket[my-sqr] may happen first, or those on @racket[my-add1] may happen first, or they may be interleaved. +… as invocations of neither @racket[my-sqr] nor @racket[my-add1] are @tech{upstream} of the other, there is no guarantee on the mutual order of effects either. For instance, the effects @techlink[#:key "associated effect"]{on} @racket[my-sqr] may happen first, or those on @racket[my-add1] may happen first, or they may be interleaved. But both of these effects would occur before the one on the corresponding @racket[my-*] invocation, since this is downstream of them both. From 3c65153659bd265ea67665da9b5b3e54250a819f Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 19 Mar 2024 00:19:04 -0700 Subject: [PATCH 10/16] doc: include Michael's example of how effects can change outputs --- qi-doc/scribblings/field-guide.scrbl | 36 ++++++++++++++++++++++------ 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 749b97bb..e9ddc3a5 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -31,9 +31,9 @@ A journeyman of one's craft -- a woodworker, electrician, or a plumber, say -- a @subsection{Separate Effects from Other Computations} -In functional programming, "effects" refer to anything a function does that is not captured in its inputs and outputs. This could include things like printing to the screen, writing to a file, or mutating a global variable. +In functional programming, @deftech{effects} refer to anything a function does that is not captured in its @tech{inputs} and @tech{outputs}. This could include things like printing to the screen, writing to a file, or mutating a global variable. -In general, pure functions (that is, functions free of such effects) are easier to understand and easier to reuse, and favoring their use is considered good functional style. But of course, it's necessary for your code to actually do things besides compute values, too! There are many ways in which you might combine effects and pure functions, from mixing them freely, as you might in Racket, to extracting them completely using monads, as you might in Haskell. Qi encourages using pure functions side by side with what we could call "pure effects." +In general, pure functions (that is, functions free of such effects) are easier to understand and easier to reuse, and favoring their use is considered good functional style. But of course, it's necessary for your code to actually do things besides compute values, too! There are many ways in which you might combine effects and pure functions, from mixing them freely, as you might in Racket, to extracting them completely using monads, as you might in Haskell. Qi encourages using pure functions side by side with what we could call @deftech{pure effects}. If you have a function with ordinary inputs and outputs that also performs an effect, then, to adopt this style, decouple the effect from the rest of the function (@seclink["Use_Small_Building_Blocks"]{splitting it into smaller functions}, as necessary) and then invoke it via an explicit use of the @racket[effect] form, thus neatly separating the functional computation from the effect. @@ -413,17 +413,39 @@ So in general, use mutable values with caution. Such values can be useful as sid @subsubsection{Order of Effects} - Qi @tech{flows} may exhibit a different order of effects (in the @seclink["Separate_Effects_from_Other_Computations"]{functional programming sense}) than equivalent Racket functions. +In general, the behavior of @emph{pure} Qi @tech{flows} (in the @seclink["Separate_Effects_from_Other_Computations"]{functional programming sense}) is the same as that of equivalent Racket expressions, but effectful flows may exhibit a different order of effects. Consider the Racket expression: @racket[(map sqr (filter odd? (list 1 2 3 4 5)))]. As this invokes @racket[odd?] on all of the elements of the input list, followed by @racket[sqr] on all of the elements of the intermediate list, if we imagine that @racket[odd?] and @racket[sqr] print their inputs as a side effect before producing their results, then executing this program would print the numbers in the sequence @racket[1,2,3,4,5,1,3,5]. -The equivalent Qi flow is @racket[(~> ((list 1 2 3 4 5)) (filter odd?) (map sqr))]. As this sequence is @seclink["Don_t_Stop_Me_Now"]{deforested by Qi's compiler} to avoid multiple passes over the data and the memory overhead of intermediate representations, it invokes the functions in sequence @emph{on each element} rather than @emph{on all of the elements of each list in turn}. The printed sequence with Qi would be @racket[1,1,2,3,3,4,5,5]. +The equivalent Qi flow is @racket[(~>> ((list 1 2 3 4 5)) (filter odd?) (map sqr))]. As this sequence is @seclink["Don_t_Stop_Me_Now"]{deforested by Qi's compiler} to avoid multiple passes over the data and the memory overhead of intermediate representations, it invokes the functions in sequence @emph{on each element} rather than @emph{on all of the elements of each list in turn}. The printed sequence with Qi would be @racket[1,1,2,3,3,4,5,5]. -Yet, either implementation produces the same output: @racket[(list 1 9 25)]. +Yet, in this case, either implementation produces the same output: @racket[(list 1 9 25)]. Often, as we see here, exhibiting a different order of effects does not make a difference to the @tech{output} of the program. -So, to reiterate, while the behavior of @emph{pure} Qi flows will be the same as that of equivalent Racket expressions, effectful flows may exhibit a different order of effects. In the case where the output of such effectful flows is dependent on those effects (such as relying on a mutable global variable), these flows could even produce different output than otherwise equivalent (from the perspective of inputs and outputs, disregarding effects) Racket code. +But in the case where the output of such effectful flows is dependent on those effects (such as by incorporating mutable state), these flows could produce different output than otherwise equivalent Racket code, as this next example shows. -If you'd like to use Racket's order of effects in any flow, @seclink["Using_Racket_to_Define_Flows"]{write the flow in Racket} by using a wrapping @racket[esc]. +@racketblock[ + (define add-count + (let ([v 0]) + (lambda (arg) + (set! v (+ v 1)) + (+ arg v)))) + + (~>> ((list 1 2 3)) (filter odd?) (map add-count) (map add-count)) +] + +Here, the unoptimized program would be equivalent to: + +@racketblock[ + ((lambda (lst) + (map add-count + (map add-count + (filter odd? lst)))) + (list 1 2 3)) +] + +… which produces the output @racket[(list 5 9)]. + +The optimized program deforests the sequence of functional operations, interleaving the effects (as discussed above), producing a different result, @racket[(list 4 10)]. See @secref["Effect_Locality"] for more insights into Qi's handling of effects. From ea290a9a4d22870a9ec26b4950b685ce391ad1e5 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 19 Mar 2024 00:20:51 -0700 Subject: [PATCH 11/16] doc: designing effects to be local --- qi-doc/scribblings/field-guide.scrbl | 4 +++- qi-doc/scribblings/principles.scrbl | 10 ++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index e9ddc3a5..6f773f84 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -447,7 +447,9 @@ Here, the unoptimized program would be equivalent to: The optimized program deforests the sequence of functional operations, interleaving the effects (as discussed above), producing a different result, @racket[(list 4 10)]. -See @secref["Effect_Locality"] for more insights into Qi's handling of effects. +From the perspective of Qi, such programs are poorly defined, and it is better to @seclink["Designing_Effects"]{design effects} to avoid such nonlocal interactions. If you'd like to employ such effects all the same, it would be advisable to write the program in Racket, encapsulating such behavior in a flow that could be used at a higher level without nonlocal effects, for instance @seclink["Using_Racket_to_Define_Flows"]{by using a wrapping @racket[esc]}. A flow specified with @racket[esc] follows Racket's order of effects since it is a Racket program. + +See @secref["Effect_Locality"] for more insights into Qi's handling of effects and its implications for the design of effects in your flows. @subsubsection{Schrodinger's Probe} diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index ccdf7f0f..d1842858 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -191,6 +191,16 @@ For a nonlinear flow like this one: But both of these effects would occur before the one on the corresponding @racket[my-*] invocation, since this is downstream of them both. +@subsubsection{Designing Effects} + +The guarantee of well-ordering of effects provided by the compiler represents a prescription for the design of @tech{effects} by users. + +As @secref["Order_of_Effects"] elaborates on, it's possible that the @tech{output} of effectful flows will differ from that of seemingly equivalent Racket programs. In fact, it's precisely in the cases where the program contains @tech{nonlocal} effects that the output could differ. + +From Qi's perspective, such effects are poorly defined as they are either too broadly scoped or likely have the scope of their operation diffused over multiple function invocations in a way that is not neatly captured by their composition. + +In general, Qi encourages designing your programs so that effects are @tech{local}. + @subsubsection{A Natural Order of Effects} By being as fine-grained as possible in expressing computational dependencies, and in tying the execution of effects to such computational dependencies, @tech{well-ordering} is in some sense the minimum well-formed guarantee on effects, and a natural one for functional languages to provide. From 0745045f5414e1ab0eb31bcf8083d0ac3d6ae6ca Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 19 Mar 2024 00:41:25 -0700 Subject: [PATCH 12/16] doc: revise explanation of Schrodinger's probe --- qi-doc/scribblings/field-guide.scrbl | 43 ++++++++++++++++++++++++---- 1 file changed, 37 insertions(+), 6 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 6f773f84..8866b266 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -453,26 +453,57 @@ See @secref["Effect_Locality"] for more insights into Qi's handling of effects a @subsubsection{Schrodinger's Probe} -Another curious thing to watch out for is that use of the @seclink["Using_a_Probe"]{probe debugger} can affect the @seclink["Order_of_Effects"]{order of effects}, as it could suppress optimizations that would be otherwise be performed if the @tech{flow} were unobserved. +Another curious thing to watch out for is that use of the @seclink["Using_a_Probe"]{probe debugger} can affect the @seclink["Order_of_Effects"]{order of effects}, as it could suppress optimizations that would otherwise be performed if the @tech{flow} were unobserved. Consider this example: @racketblock[ (define-flow foo - (~> (pass (effect E₁ odd?))) readout (>< (effect E₂ sqr))) + (~> (pass odd?) (>< sqr))) +] + +This program would be optimized by the Qi compiler to: + +@racketblock[ + (>< (if odd? sqr ⏚)) +] -(probe (foo 1 2 3)) +If we placed a readout here: + +@racketblock[ +(define-flow foo + (~> (pass odd?) readout (>< sqr))) ] -Here, with the @racket[readout], all the effects E₁ would occur first, followed by all of the E₂ effects. Without the @racket[readout], the flow would be deforested by the compiler to: +… then for an input list @racket[(list 1 2 3)], the readout would show @racket[(list 1 3)]. But in the optimized program above, the readout would not even represent a valid point in the program (where should it be placed?). Thus, the readout is showing values that are flowing in the original program rather than the one that would actually have been executed in the absence of the readout. + +The optimization here (by requirement) does not change the meaning of the program in the absence of effects. If there were effects present, however, then the situation gets even more spooky. + +The first program would look something like this: + +@racketblock[ +(define-flow foo + (~> (pass (effect E₁ odd?))) (>< (effect E₂ sqr))) +] + +… where all the effects E₁ would happen before any of the effects E₂. And the second program would look like: @racketblock[ (>< (if (effect E₁ odd?) (effect E₂ sqr) ⏚)) ] -… and the effects E₁ and E₂ would be interleaved. Either order is consistent with @seclink["Effect_Locality"]{locality} but they are @emph{different}. Indeed, the @racket[readout] in this case would not even represent a valid point in the a priori optimized program. +… where the effects E₁ and E₂ would be interleaved. Though it changes the order of effects, the optimization is possible because it preserves @tech{well-ordering}. The second program here represents what will actually be executed when the first program is written. + +But what happens when we place a @racket[readout] in the source program, this time? + +@racketblock[ +(define-flow foo + (~> (pass (effect E₁ odd?))) readout (>< (effect E₂ sqr))) +] + +Here, with the @racket[readout], the program would once again not be optimized, and thus, all the effects E₁ would occur first before the values are read out. Without the @racket[readout], the flow would be @seclink["Don_t_Stop_Me_Now"]{deforested} by the compiler, and as we have just seen, the effects E₁ and E₂ would be interleaved, so that the effects observed in the presence of the readout are different from what would be observed without it. -So it's important to bear in mind that one cannot observe a flow using @racket[probe] without changing the program being observed, a change which in some cases has no observable impact, and which in other cases is significant. But now that you understand this phenomenon, you can develop intuition for the nature of such changes, and how best to use the tool to find the answers you are looking for. +So it's important to bear in mind that one cannot observe a flow using @racket[probe] without changing the program being observed, a change which in some cases has no observable impact, and which in other cases (i.e. when there are effects involved) could be significant. But now that you understand this phenomenon, you can develop intuition for the nature of such changes, and how best to use the tool to find the answers you are looking for. @section{Effectively Using Feedback Loops} From 0326599c172b1a36eb448080f7be3ad4f3c128ec Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 19 Mar 2024 13:14:11 -0700 Subject: [PATCH 13/16] doc: misc minor improvements --- qi-doc/scribblings/field-guide.scrbl | 2 +- qi-doc/scribblings/principles.scrbl | 18 +++++++++--------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 8866b266..6f30e2f4 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -413,7 +413,7 @@ So in general, use mutable values with caution. Such values can be useful as sid @subsubsection{Order of Effects} -In general, the behavior of @emph{pure} Qi @tech{flows} (in the @seclink["Separate_Effects_from_Other_Computations"]{functional programming sense}) is the same as that of equivalent Racket expressions, but effectful flows may exhibit a different order of effects. +In general, the behavior of @emph{pure} (in the @seclink["Separate_Effects_from_Other_Computations"]{functional programming sense}) Qi @tech{flows} is the same as that of equivalent Racket expressions, but effectful flows may exhibit a different order of effects. Consider the Racket expression: @racket[(map sqr (filter odd? (list 1 2 3 4 5)))]. As this invokes @racket[odd?] on all of the elements of the input list, followed by @racket[sqr] on all of the elements of the intermediate list, if we imagine that @racket[odd?] and @racket[sqr] print their inputs as a side effect before producing their results, then executing this program would print the numbers in the sequence @racket[1,2,3,4,5,1,3,5]. diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index d1842858..51db0e95 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -37,7 +37,7 @@ A @deftech{flow} is either made up of flows, or is a native (e.g. Racket) @seclink["lambda" #:doc '(lib "scribblings/guide/guide.scrbl")]{function}. Flows may be composed using a number of combinators that could yield either linear or nonlinear composite flows. - A flow in general accepts @code{m} @deftech{inputs} and yields @code{n} @deftech{outputs}, for arbitrary non-negative integers @code{m} and @code{n}. We say that such a flow is @code{m × n}. + A flow in general accepts @code{m} @deftech{inputs} and yields @code{n} @deftech{outputs}, for arbitrary non-negative integers @code{m} and @code{n}. We say that such a flow is @code{m × n}. Inputs and outputs are ordinary @tech/reference{values}. The semantics of a flow is function invocation -- simply invoke a flow with inputs (i.e. ordinary arguments) to obtain the outputs. @@ -57,7 +57,7 @@ @section{Values are Not Collections} - The things that flow are values. Individual values may happen to be collections such as lists, but the values that are flowing are not, together, a collection of any kind. + The things that flow are @tech/reference{values}. Individual values may happen to be collections such as @tech/guide{lists}, but the values that are flowing are not, together, a collection of any kind. To understand this with an example: when we employ a tee junction in a @tech{flow}, colloquially, we might say that the junction "divides the flow into two," which might suggest that there are now two flows. But in fact, there is just one flow that divides @emph{values} down two separate flows which are part of its makeup. More precisely, @racket[-<] composes two flows to yield a single composite flow. Like any flow, this composite flow accepts values and produces values, not collections of values. There is no way to differentiate, at the output end, which values came from the first channel of the junction and which ones came from the second, since downstream flows have no idea about the structure of upstream flows and only see the values they receive. @@ -121,7 +121,7 @@ In the above example, @racket[filter] and @racket[map] are obviously ordered by This is where the distinction between flows and flow invocations comes into play. In fact, not all invocations of @racket[my-odd?] are upstream of any particular invocation of @racket[my-sqr]. Rather, specific invocations of @racket[my-sqr] that use values computed by individual invocations of @racket[my-odd?] are downstream of those invocations, and notably, these invocations involve the individual elements of the input list rather than the entire list, so that the computational dependency expressed by this relation is as fine-grained as possible. -For instance, for an input list @racket[(list 1 2 3)], @racket[(my-odd? 1)] is @tech{upstream} of @racket[(my-sqr 1)], and likewise, @racket[(my-odd? 3)] is @tech{upstream} of @racket[(my-sqr 3)]. +For instance, for an input list @racket[(list 1 2 3)], @racket[(my-odd? 1)] is @tech{upstream} of @racket[(my-sqr 1)], and likewise, @racket[(my-odd? 3)] is @tech{upstream} of @racket[(my-sqr 3)], but @racket[(my-odd? 3)] is not upstream of @racket[(my-sqr 1)], and @racket[(my-odd? 2)] isn't upstream of anything. That brings us to the guarantee that Qi provides in this case (and in general). @@ -150,7 +150,7 @@ For instance, the compiler could accumulate all effects and execute them in an a Locality of effects does not imply well-ordering of effects under program translation, nor vice versa – these are independent. -In sum, Qi guarantees that the @tech{output} of execution of the compiled program is the same as that of the source program, assuming @tech{effects} are @tech{local}, and further, it guarantees that the effects will be @techlink[#:key "well-ordering"]{well-ordered} in relation to the source program. +In sum, @emph{Qi guarantees that the @tech{output} of execution of the compiled program is the same as that of the source program, assuming @tech{effects} are @tech{local}, and further, it guarantees that the effects will be @techlink[#:key "well-ordering"]{well-ordered} in relation to the source program.} This has a few implications of note. @@ -172,7 +172,7 @@ Next, for a flow like this one: (~>> (filter my-odd?) (map my-sqr) car) ] -… when it is invoked with a large input list, Qi in fact @seclink["Don_t_Stop_Me_Now"]{only processes the very first value} of the list, since it determines, at the end, that no further elements are needed in order to generate the final result. This means that all effects on would-be subsequent invocations of @racket[my-odd?] and @racket[my-sqr] would simply not be performed. Yet, @techlink[#:key "well-ordering"]{well-ordering} is preserved here, since the @techlink[#:key "well-ordering"]{defining implication} holds for every flow invocation that actually happens. Well-ordering is about effects being guided by the @emph{necessity} of @techlink[#:key "associated effect"]{associated} computations to the final result. +… when it is invoked with a large input list, Qi in fact @seclink["Don_t_Stop_Me_Now"]{only processes the very first value} of the list, since it determines, at the end, that no further elements are needed in order to generate the final result. This means that all effects on would-be subsequent invocations of @racket[my-odd?] and @racket[my-sqr] would simply not be performed. Yet, @tech{well-ordering} is preserved here, since the @techlink[#:key "well-ordering"]{defining implication} holds for every flow invocation that actually happens. Well-ordering is about effects being guided by the @emph{necessity} of @techlink[#:key "associated effect"]{associated} computations to the final result. @subsubsection{Independent Effects} @@ -187,13 +187,13 @@ For a nonlinear flow like this one: (foldl + 0)) ] -… as invocations of neither @racket[my-sqr] nor @racket[my-add1] are @tech{upstream} of the other, there is no guarantee on the mutual order of effects either. For instance, the effects @techlink[#:key "associated effect"]{on} @racket[my-sqr] may happen first, or those on @racket[my-add1] may happen first, or they may be interleaved. +… as invocations of neither @racket[my-sqr] nor @racket[my-add1] are @tech{upstream} of the other, there is no guarantee on the mutual order of effects either. For instance, the effects @techlink[#:key "associated effect"]{on} @racket[my-sqr] may happen first, or those @techlink[#:key "associated effect"]{on} @racket[my-add1] may happen first, or they may be interleaved. -But both of these effects would occur before the one on the corresponding @racket[my-*] invocation, since this is downstream of them both. +But both of these effects would occur before the one on the corresponding @racket[my-*] invocation, since this is @tech{downstream} of them both. @subsubsection{Designing Effects} -The guarantee of well-ordering of effects provided by the compiler represents a prescription for the design of @tech{effects} by users. +The guarantee of @tech{well-ordering} of effects provided by the compiler represents a prescription for the design of @tech{effects} by users. As @secref["Order_of_Effects"] elaborates on, it's possible that the @tech{output} of effectful flows will differ from that of seemingly equivalent Racket programs. In fact, it's precisely in the cases where the program contains @tech{nonlocal} effects that the output could differ. @@ -305,7 +305,7 @@ Therefore, any theoretical results about arrows should generally apply to Qi as @section{It's Languages All the Way Down} -Qi is a language implemented on top of another language, Racket, by means of a macro called @racket[flow]. All of the other macros that serve as Qi's @seclink["Embedding_a_Hosted_Language"]{embedding} into Racket, such as (the Racket macros) @racket[~>] and @racket[switch], expand to a use of @racket[flow]. +Qi is a language implemented on top of another language, Racket, by means of a @tech/reference{macro} called @racket[flow]. All of the other macros that serve as Qi's @seclink["Embedding_a_Hosted_Language"]{embedding} into Racket, such as (the Racket macros) @racket[~>] and @racket[switch], expand to a use of @racket[flow]. The @racket[flow] form accepts Qi syntax and (like any @tech/reference{macro}) produces Racket syntax. It does this in two stages: From 2816fa300e404499085a6ad6c06ebd4f6a2af313 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Tue, 19 Mar 2024 13:14:33 -0700 Subject: [PATCH 14/16] doc: minor improvements in Schrodinger's Probe --- qi-doc/scribblings/field-guide.scrbl | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/qi-doc/scribblings/field-guide.scrbl b/qi-doc/scribblings/field-guide.scrbl index 6f30e2f4..d0212f6c 100644 --- a/qi-doc/scribblings/field-guide.scrbl +++ b/qi-doc/scribblings/field-guide.scrbl @@ -477,7 +477,9 @@ If we placed a readout here: … then for an input list @racket[(list 1 2 3)], the readout would show @racket[(list 1 3)]. But in the optimized program above, the readout would not even represent a valid point in the program (where should it be placed?). Thus, the readout is showing values that are flowing in the original program rather than the one that would actually have been executed in the absence of the readout. -The optimization here (by requirement) does not change the meaning of the program in the absence of effects. If there were effects present, however, then the situation gets even more spooky. +Yet, the optimization (@seclink["Qi_s_Guarantee_on_Effects"]{by requirement}) does not change the meaning of the program in the absence of effects, and so the actual @tech{output} of the program is consistent with the intermediate values that are read out. + +If there are @tech{effects} present, however, then the situation gets more spooky. The first program would look something like this: @@ -492,7 +494,7 @@ The first program would look something like this: (>< (if (effect E₁ odd?) (effect E₂ sqr) ⏚)) ] -… where the effects E₁ and E₂ would be interleaved. Though it changes the order of effects, the optimization is possible because it preserves @tech{well-ordering}. The second program here represents what will actually be executed when the first program is written. +… where the effects E₁ and E₂ would be interleaved. Though it changes the order of effects, the optimization is still valid because it preserves @tech{well-ordering}. The second program here represents what will actually be executed when the first program is written. But what happens when we place a @racket[readout] in the source program, this time? @@ -501,7 +503,7 @@ But what happens when we place a @racket[readout] in the source program, this ti (~> (pass (effect E₁ odd?))) readout (>< (effect E₂ sqr))) ] -Here, with the @racket[readout], the program would once again not be optimized, and thus, all the effects E₁ would occur first before the values are read out. Without the @racket[readout], the flow would be @seclink["Don_t_Stop_Me_Now"]{deforested} by the compiler, and as we have just seen, the effects E₁ and E₂ would be interleaved, so that the effects observed in the presence of the readout are different from what would be observed without it. +Here, with the @racket[readout], the program once again would not be optimized, and thus, all the effects E₁ would occur first before the values are read out. Without the @racket[readout], the flow would be optimized, as we have just seen, and the effects E₁ and E₂ would be interleaved, so that the effects observed in the presence of the readout are different from what would be observed without it! So it's important to bear in mind that one cannot observe a flow using @racket[probe] without changing the program being observed, a change which in some cases has no observable impact, and which in other cases (i.e. when there are effects involved) could be significant. But now that you understand this phenomenon, you can develop intuition for the nature of such changes, and how best to use the tool to find the answers you are looking for. From ee9e55b7c7a94664853b6e355e45b60e239e3e76 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Thu, 21 Mar 2024 11:43:48 -0700 Subject: [PATCH 15/16] A comment on the upstream relation --- qi-doc/scribblings/principles.scrbl | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index 51db0e95..03e98948 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -113,6 +113,11 @@ We already saw how a flow can be thought of as a @seclink["Flows_as_Graphs"]{dir @definition["Upstream and downstream"]{A flow invocation @${f} is @deftech{upstream} of another invocation @${g} if the output of @${f} is @emph{necessary} to determining the input of @${g}. Conversely, @${g} is @deftech{downstream} of @${f}. We will denote this relation @${f < g}.} +@; This relation, like the usual strict order relation on numbers, is +@; irreflexive (f isn't upstream of itself, not f < f), +@; asymmetric (f < g ⇒ not g < f) +@; and transitive (f < g and g < h ⇒ f < h) + Note that this definition relates flow @emph{invocations} rather than @tech{flows} themselves. For now, we need not worry about this distinction, but we will soon see why it matters. In terms of the @tech{paths} that could be traced over a flow, the ordering implied by @racket[~>] naturally shows us many members of this relation: flows that come later in the sequence in a @racket[~>] form are downstream of those that come earlier, because the output of earlier flows is needed to determine the input to later flows. From 465a0f7f45d77417fc561cfe89baf9867ccd66b7 Mon Sep 17 00:00:00 2001 From: Siddhartha Date: Fri, 5 Apr 2024 21:22:19 -0700 Subject: [PATCH 16/16] minor improvement: section title --- qi-doc/scribblings/principles.scrbl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/qi-doc/scribblings/principles.scrbl b/qi-doc/scribblings/principles.scrbl index 03e98948..60e7a6d5 100644 --- a/qi-doc/scribblings/principles.scrbl +++ b/qi-doc/scribblings/principles.scrbl @@ -97,7 +97,7 @@ But if @racket[my-odd?] and @racket[my-sqr] exhibit any side effects, such as pr To understand what this means, we will need to develop some concepts. -@subsection{Functional and Effective} +@subsection{Functional Effects} First, regarding the definability of effects, as far as Qi is concerned, they are only well-defined in connection with some @tech{flow}, and are not independently conceivable.