distsql: support ARRAY_AGG in distsql

[RaduBerinde]

We currently don't support arrays in DistSQL streams. The main thing that's missing is a way to encode them; this will have to be implemented to support arrays in table columns (see #15818). Once that's done we should enable support in DistSQL as well.

CC @asubiotto

[RaduBerinde]

Part of this work we should also implement ARRAY_AGG in distsql.

[cuongdo]

Given that we currently only support arrays for virtual schemas, is this currently affecting anything? Or is this a natural part of our work to support ARRAYs?

[RaduBerinde]

We support arrays in select expressions (and there is ARRAY_AGG).

[a6802739]

@RaduBerinde could I have a try for this? And If you could give me some reference, I would appreciate it very much.

[RaduBerinde]

Hi @a6802739 - to work on this we need array encoding code to be in place. This is only in the RFC stage at this point (#16172).

[a6802739]

@RaduBerinde Thank you, I will wait until it's ready.

[cuongdo]

@RaduBerinde can you reassign?

[RaduBerinde]

@asubiotto I'll put this on your plate for now. We may need to move to 1.2 (not sure what is the current state of array encoding for table columns).

[jordanlewis]

cc @justinj

[rjnn]

@cuongdo This is not going to make it into 1.1, @asubiotto's other big priority is getting external storage for HashJoiners and there isn't time for both big projects to make it in in 1.1, so lets move it to 1.2 unless we can get more hands on deck between now and the 1.1 feature freeze.

[RaduBerinde]

Fixed in #18243.

[RaduBerinde]

Part of this work we should also implement ARRAY_AGG in distsql.

This is still to be done.

[vivekmenezes]

assigning this issue to abhishek. Can't assign at the moment because he's still not a contributor

[abhimadan]

I think that adding ARRAY_AGG support should only involve removing the explicit check for it in the physical planner, and adding it to the AggregatorSpec_Func proto enum. However, when I tried to do this, I ran into the CLI error pq: unsupported result type: anyelement.

After some investigating, I found that the aggregator processor calls tree.Builtin.FixedType(), which in turn calls returnTypeToFixedType(). This function relies on the fact that its ReturnTyper argument returns a valid type when passed an empty expression list. However, ARRAY_AGG does not follow this assumption (see here). We can actually see this bug manifest in the user docs (see here), where ARRAY_AGG returns anyelement instead of an array.

I'm getting a PR ready to fix this issue.

[abhimadan]

Fixed by #21475.