feat: add .attrs to highlevel objects

[agoose77]

This will partially fix #1391 by adding .attrs, a dictionary of metadata that is associated with the top-level array, with the following semantics:

• Operations with multiple arrays take the set-union of properties, first wins
• Keys beginning with a special prefix (@) will be removed at serialisation time.

Whilst touching some of this code, I did some work on type hints and refactoring.

[agoose77]

Design question (1) — should array.attrs always return a dict, or can it return None?

Return None

(invert the below for "always return a dict")

• Convenient token for "no metadata", allowing optimisations (including non-Awkward cases)
• Harder to set metadata in a non-functional style (unless metadata already set)

[jpivarski]

The _parameters are internally None or dict for performance, but the external parameters property always returns a dict for uniformity. In fact, the property creates the dict so that it can be mutably updated from that point onward.

I think this is a good compromise, and it would follow the principle of least surprise if attrs does the same thing. In fact, I think this is what behaviors does, too.

You were only asking about how it looks from the outside, so I'm voting "always return a dict." But since most users won't be using attrs (that's the situation we're starting from before this first implementation), so using None internally minimizes the performance impact of adding a new feature across all ak.Arrays.

If we really need a has_attrs: bool (more likely a has_attr(name)), then we can add it.

---

Oh, unlike parameters, I think we should not conflate key-not-found with value-is-None. We should allow users to consider None a meaningful value for an attribute, distinct from not having that attribute. attrs is the users' space, much more so than parameters.

[agoose77]

The _parameters are internally None or dict for performance, but the external parameters property always returns a dict for uniformity. In fact, the property creates the dict so that it can be mutably updated from that point onward.
...
I think this is a good compromise, and it would follow the principle of least surprise if attrs does the same thing. In fact, I think this is what behaviors does, too.

Actually, Array.behavior returns None | dict, making it possible to predict whether the array will use the global behavior lookup.

attrs doesn't have that benefit; there's only a single namespace to look at, so this motivation for exposing None disappears. I think it's therefore OK to hide the detail from users.

---

Oh, unlike parameters, I think we should not conflate key-not-found with value-is-None. We should allow users to consider None a meaningful value for an attribute, distinct from not having that attribute. attrs is the users' space, much more so than parameters.

Agreed.

[agoose77]

This PR indicates that it would be nice to have something like

ctx = HighLevelCtx()

layouts = ctx.finalise([
    ak.to_layout(ctx.maybe_highlevel(a)) for a in arrays
])

backend = ctx.backend
behavior = ctx.behavior

Rather than the existing stateless functional API that revisits inputs multiple times. This function would ensure all layouts have the proper "final" backend.

As such, it's on-hold for PR #2763 and another PR.

[agoose77]

@jpivarski this is a big PR.

I've split it into commits that should be a bit more manageable.

I still need to do a pass to confirm that the various policy settings used by ak.to_layout are sensible, but I'd benefit from the main review to confirm that the (internal) API changes are a step in the right direction.

The purpose of the HighLevelContext addition to this PR was make the general idiom of pulling behaviors off of high-level object(s) and applying them to the results is less error prone. The use of ctx.wrap_layout leads to an Exception if the context hasn't been explicitly finalised, making it harder to pass the wrong behavior argument around.

There's a test failure pertaining to the choice to permit records, that I'll revisit.

[agoose77]

Do we have a formal policy regarding which functions should prohibit/support record objects? e.g. ak.is_none on a record.

[jpivarski]

It is a big PR with a lot of differences, although many of them come from adding the single extra attrs argument to lots of functions, which in many cases turns them from single-line to multi-line due to black.

The context object is a big difference. I see that this context object is private, for our own use, and I'm not certain that it's necessary for keeping track of two pieces of data (behavior and attrs), but it's fine.

(In your comment, I thought you were referring to a public context object, so that people can say,

with ak.Context(behavior, attrs):
    ak.something...

although that doesn't make a lot of sense: one would want the behavior and attrs to be permanently glued to specific objects and not others. But I guess you were talking about a private context.)

I'm in favor of the direction that this PR is heading.

[jpivarski]

This is a huge work, and tests/test_2757_attrs_metadata.py is very extensive, parameterizing over so many functions to make sure that attrs are correctly passed through all of them.

My only request is about Numba: see below. If you have questions about that, I can help during our meeting tomorrow.