Regarding null

[mattdailis]

I'm opening up this discussion about how null values are/should be treated in Aerie's ValueSchemas and SerializedValues.

This discussion was prompted by #1346 , where the sequencing server would crash when an processing an optional struct value such as { "present": false, "value": null }, with value schema (in typescript type shorthand): { "present": boolean, "value": { "foo": "bar } }.

This discussion shouldn't be a blocker for #1347 ; avoiding the sequencing server crash is definitely desirable, and urgent.

The purpose of this discussion is to explore the use cases and pros/cons of different ways of handling null, and to establish what the status quo is, as well as what a desirable future could be.

[mattdailis]

Collection of comments prior to opening this discussion:

From @mattdailis regarding #1347:

I'm curious about implications for generated code - let's say that an Activity takes an Optional parameter, which I imagine turns into a { "present": boolean, "value": number } type in the generated TS code

Does this mean that the expansion logic needs to handle the case where value is null? Should it also handle the case where present is null?

I think I'm a bit concerned about the implications of "any value can be null", since to me that means "downstream tools must handle any value being null" 🤔 This leads to a question of "is null a value, or does it represent the lack of a value?"

All that said, I'm in favor of pursuing your suggested fix in the short term, since I think the broader question of "what's the best way to represent Optionals in ValueSchemas" can be answered independently of making the sequencing server more robust

Ah, I realize that your specific examples have to do with struct and series - would we run into the same issue with an Optional? I suspect we may want to handle null at the top, before splitting into the cases

From @cartermak:

@mattdailis Great questions that I've pondered a bit already.

My current philosophical stance is that any field in a serialized value ought to be nullable (at any "level" of the value schema, if that makes sense, including "parent" structs/series). A few justifications I have for that position:

1. That's generally true in the way the corresponding structures are implemented in the model code (e.g., a complex nested record type in the mission model that maps to a nested set of struct in the value schema can generally be null).

2. If a given value is null, then that is either an expected possibility, in which case any logic which interacts with that value ought to handle the null, or it is unexpected and it's OK (even preferable) for anything that interacts with that value to break.

3. The UI can generally handle this already. The Clipper activities which cause the original errors in command expansion are displayed fine in the UI (the UI seems to just propagate blank values to each "child" in the value schema).

With that said, some specific answers based on that philosophy:

I'm curious about implications for generated code - let's say that an Activity takes an Optional parameter, which I imagine turns into a { "present": boolean, "value": number } type in the generated TS code
Does this mean that the expansion logic needs to handle the case where value is null? Should it also handle the case where present is null?

If it's meaningful that the value is null, then the expansion logic should handle that case. If it's not expected, then I think it's OK for the expansion logic to assume the value is not null and let any errors that may precipitate occur. I also don't think the onus is generally on the expansion logic to perform extensive error checking: the activity type/expansion rule interface ought to be a relatively "safe" one, and it already provides native mechanisms to check parameter values.

I think I'm a bit concerned about the implications of "any value can be null", since to me that means "downstream tools must handle any value being null" 🤔 This leads to a question of "is null a value, or does it represent the lack of a value?"

I haven't developed this thought as much, but I think there ought to be a distinction between a null value, which might occur for any number of programming-related reasons, versus some notion of an optional/"null represents the lack of a value," which should be more explicitly captured. For example, null can be a useful technical solution when parameters are coupled, as in the case of the present and value field's of Clipper's optionals. Alex and I tossed around the idea that the "right" answer might be for optional to get its own container type in the value schema definition alongside struct and series (with corresponding UI behavior, of course).

Ah, I realize that your specific examples have to do with struct and series - would we run into the same issue with an Optional? I suspect we may want to handle null at the top, before splitting into the cases

I don't think we'd run into the same issue with Optional<String> if only the value is null, but I haven't tested it. I believe the sequencing activity deserialization will already propagate a null value for any of the "primitive" value schema data types (i.e., not struct or series). If the whole optional struct were null, I think we would run into this issue.

[mattdailis]

If it's meaningful that the value is null, then the expansion logic should handle that case. If it's not expected, then I think it's OK for the expansion logic to assume the value is not null and let any errors that may precipitate occur.

There are a couple of avenues of inquiry that I think are relevant here: 1) what kinds of static guarantees do we wish to provide about the structure of data? and 2) what do we do about clients that use the value schema without the presence of a value? What should they do about the nullability of all values?

1. Static vs dynamic errors

Regarding (1) I'm referring to how much we want to try to leverage the typescript type system to ensure the type-safety of command expansion logic. Our stance could be that unit testing is sufficient (that's the usual approach taken by dynamic language proponents), but if we could rely on the type system to prove certain safety properties to us, that would reduce the burden on testing. (I'd love to hear from you whether you're finding the fact that expansions are defined in typescript to be valuable, or if you see it as getting in the way of just letting you write javascript)

Now: there are limitations to what type systems can express, and there are tradeoffs to ease-of-use if your type system is too restrictive. The question under consideration is whether it would be valuable to leverage typescript's | null type. Options are:

• (a) Do not use the | null type, and allow misuse of null to fail at runtime (status quo)
• (b) Add | null to every type, and force expansion logic to handle the possible nullness of everything (extreme)
• (c) Encode nullability in the ValueSchema, and use | null only where the values might be null (ideal, imho, but maybe idealistic)

One more consideration around the possible benefits of static type checking has to do with mission model updates. If a previously non-nullable type becomes nullable as a result of a new mission version, it may be desirable to catch that during expansion set creation.

2. Generic data consumers

For avenue (2), I want to consider that the stance we take on nullability of values extends beyond command expansion, to things like the UI input form, or timeline renderer.

I agree on some level with the end-to-end argument. There is the data producer, and the data consumer - the two need to agree on the shape of the data. When these are one-to-one, and owned by the same person, that is fairly straightforward.

There are cases where a data consumer and data producer may be written without knowledge of each other - the language they use to communicate with each other is "value schema".

The most fleshed out example of this is the UI parameter-editing form. I don't believe there is currently a way to enter null as a value for any parameter. A question that arises now is: should there be? An extreme interpretation would be to put a checkbox next to every value, allowing the user to mark it as "null". There would be no way to indicate to the UI that particular value can't be null 🤔 Though, if we did go that route, I think that would reduce the need for Optionals 🤷

Other cases could be gathering statistics on a numeric profile - what should the "average" be for a profile declared as type integer that contains null values? Or for constraint checking - what should Discrete.Resource("/instrument/a/isON").or(Discrete.Resource("/instrument/b/isON") return when either profile has a null value? Is a null value equivalent to the constraint concept of a "gap"? Answers could be

• The UI or constraint checking application should crash - the user should have known better than to try to use that resource for that purpose (this is the Java mindset)
• The UI and constraint checking application should assume that null means 0, or false (this is the JavaScript mindset)
• The UI and constraint checking application should make assumptions, but log warnings (okay... but people ignore warnings)

Unfortunately there is no cross-language consistency when it comes to what null means, so we can't lean too hard on precedent (or, put another way, we get to choose from very many precedents!)

• Java says that null is a special value, and that most (but not all) types can be null. This stems from implementation details of primitives vs Object types, and I see it more as a leaky abstraction than a language feature
• JavaScript says that null is a "nullish" value, along with undefined and NaN, and it will do its very best to not crash, but may do some unexpected things. (We can try to build notions of "truthy", "falsy", and "nullish" into Aerie - but they come with their own complexity baggage)
• SQL says that null represents an "unknown" value - null in and of itself is not a value. (Kind of like a "gap" in Aerie constraints). Functions like COUNT and MAX will skip over nulls, and null is not considered equal to null, and the question of "does this set contain value X" becomes unknowable if the set contains a null value, etc...

Aerie's general philosophy around null so far has been to try to avoid it, where possible. Now, I acknowledge that it is useful in certain scenarios, but I think those are the minority of scenarios, and most of the time it's much more valuable to be able to assume that a value is not null.

I want to emphasize that I write all this not because I think you don't know these things - I'm just trying to think through the design space together. Thanks for taking the time to think through this with me 👍

[dyst5422]

Regarding 1, definitely see value in (c) as it provides a mechanism for ensuring our interfaces rather than relying on documentation and developer intent. Part of the beauty of our codegen is we can metaprogram out a bunch of this, so we have a lot of power in closing these interface gaps.

Regarding 2, we are talking about what kind of semantics to we want to use and support. Like you mentioned, null has different potential semantic meanings and we essentially need to decide what the semantic meaning is in the context of our value schemas. The constraining requirements is that what we decide to use is expressible in all parts of our system including the JSON interface we use to interchange data between the parts of the system.

To elaborate, we can have keys that are non-existent in JSON that is different from a key that is existent and null and also different from a value-type specific additive identity (numbers often use 0, strings would be the empty string '', arrays would be the empty array []).

My anecdotal experience has taught me that there is value to being able to differentiate from "this thing doesn't exist, or doesn't exist yet" and "this thing exists and is explicitly naught", especially when it comes to UIs and APIs.

My position is that we should exercise the mechanisms at our disposal to the maximum practical ability. Reduction in cognitive overhead for users and developers by exercising these tools has a largely immeasurable value, but I generally believe it to be significant in long term maintainability. Our type systems across the system are generally expressive enough to improve on our current behavior if we take the time and care to use them well.

[mattdailis]

We had a chat with @dandelany and @goetzrrGit , and we all agreed on the following two things:

• We want to make sure that Aerie has a consistent story about null as it is used currently before making significant changes
• We think it would be valuable for null to be represented in the value schema somehow

@dandelany also commented that it could be valuable to go further and differentiate between null and undefined (what @dyst5422 described as "this thing exists and is explicitly naught" and "this thing doesn't exist, or doesn't exist yet" respectively)

I started jotting down notes from our discussions, but I realized that I think we're solving two separate problems:

• Representing | null in ValueSchema
• Representing Optional<T> in ValueSchema

The latter is the driving use case, but the former also deserves consideration. It may be possible, but not necessary, for one solution to address both problems.

Notes in collapsible for now, until I have some more time to flesh them out:

Notes

We explored some ideas for how exactly it could be represented in the value schema. (i) focuses on encoding null, while (ii) focuses on removing the need for null

• (i) Add a "nullable" property to every value schema node type.
  • Example schema { "kind": "string", "nullable": true }
  • Example values: "Hello world", null
• (ii) Add a new kind of value schema, called a "tagged union", which can be used to encode the optionality of a value, but can also represent a large number of varying data types. E.g.
  • Example schema: { "kind": "tagged_union", "variants": [ {"empty": { "kind": "struct", "items": [] }}, { "present": { "kind": "string" } }] }
  • Example values: {"empty": {}}, { "present": "Hello world" }

An advantage of Option (i) is that the value schema concise and it directly represents null - i.e. values are allowed to be null. Option (ii) is general, and addresses a broader set of use cases, but its ValueSchema is verbose, and its "empty" values are larger ({"empty": {}} vs null)

@Twisol had a few comments as well (paraphrasing):

• The fact that SerializedValue allows null but ValueSchema doesn't is indicative of a gap in ValueSchema
• Activity Mappers treat null and missing equally at the top level to indicate that a parameter value has not been provided. They don't, however, automatically handle nulls deeper in a structure or series

@Twisol suggested a hybrid approach to make (ii) more concise, by treating null as a special kind of tag, which has no corresponding value.

• (iii) Add a new kind of value schema, called a "tagged union", which can be used to encode the optionality of a value, but can also represent a large number of varying data types. E.g.
  • Example schema: { "kind": "tagged_union", "variants": [ null, { "present": { "kind": "string" } }] }
  • Example values: null, { "present": "Hello world" }

An advantage of (iii) is that it allows the "empty" case to simply by null, but it allows for unambiguous encoding of nested optionals Optional<Optional<T>>. Optional.of(Optional.empty) becomes { "present": null }, while Optional.empty() is null.