[YSQL] Segfault on index scan with scankey flattening

[jasonyb]

Jira Link: DB-7311

Description

Case 1: flattening (via row filter) + yb_hash_code filter

create table t (i int, j int, k int, l int);
create index i on t (l, k, j);
/*+IndexScan(t i)*/ select * from t where yb_hash_code(l) > 5 and row(k, j) > row(1, 2);

2023-07-19 17:06:34.903 PDT [508432] WARNING:  server process (PID 508504) was terminated by signal 11: Segmentation fault
2023-07-19 17:06:34.903 PDT [508432] DETAIL:  Failed process was running: /*+IndexScan(t i)*/ select * from t where yb_hash_code(l) > 5 and row(k, j) > row(1, 2);

Created the repro because this logic looked suspicious:

    for (int i = 0; i < nkeys; ++i)
    {
        ScanKey key = &keys[i];
        /*
         * Keys for hash code search should be placed after regular keys.
         * For this purpose they are written into keys array from
         * right to left.
         * We also flatten out row keys
         */
        ybScan->keys[YbIsHashCodeSearch(key)
            ? (nkeys - (++ybScan->nhash_keys))
            : ybScan->nkeys++] = key;

        if (YbIsRowHeader(&keys[i]))
        {
            ScanKey current = (ScanKey) keys[i].sk_argument;
            do
            {
                ybScan->keys[ybScan->nkeys++] = current;
            }
            while (((current++)->sk_flags & SK_ROW_END) == 0);
        }
    }

It looks like it assumes nkeys of the original keys is the total size of ybScan->keys, but due to flattening of header keys, it could end up putting more than nkeys keys in ybScan->keys. The logic writes forward and backward, so this likely results in an overlap in case both hash code search and row filter is used.

Case 2: flattening overflows buffer

The following check also is not proper because it doesn't account for flattening, so it can lead to buffer overflow:

if (nkeys > YB_MAX_SCAN_KEYS)

Repro:

bin/ysqlsh -c "create table t (i int); create index on t (i asc); select i from t where row($(for _ in {1..100}; do echo -n i,; done)i) > row($(for _ in {1..100}; do echo -n 0,; done)0);"

2023-07-19 17:35:14.079 PDT [510322] WARNING:  server process (PID 510587) was terminated by signal 11: Segmentation fault
2023-07-19 17:35:14.079 PDT [510322] DETAIL:  Failed process was running: create table t (i int); create index on t (i asc); select i from t where row(i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i) > row(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);

---

Why was flattening done in the first place? It seems to make the logic more complicated, so consider going back to preserving the original format.

(This was recent master commit 94ba9c7, alma8 fastdebug gcc11.)

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.