feat: add pre caveat to store/list and upload/list

[travis]

Per storacha/w3ui#143 we'd like to give clients a way to get the previous page of results rather than the next page.

This change introduces a pre caveat which, when set to true, will return the page of results preceding cursor.

I considered calling this reverse or rev but I think that implies that results will be returned sorted in reverse, which is not the intent.

[alanshaw]

@travis if I'm understanding this correctly, this would enable you to page in either direction, but would prevent switching direction without storing some state (the cursor for the current page).

I think I had imagined this as being a change to the return value, where you'd get 2 cursors previous and next instead of one cursor. Then you could use that to get the previous or next page without having to store any state.

[travis]

ahh, I think that would be a better API for this yes - I'm going to close this for now and investigate that direction, thanks!

[travis]

ok I did some investigation into this today and as far as I can tell, returning a prev cursor as proposed above would require doing an extra DynamoDB query here:

https://github.com/web3-storage/w3infra/blob/main/upload-api/tables/upload.js#L150

something like:

      const prevCmd = new QueryCommand({
        TableName: tableName,
        Limit: options.size || 20,
        KeyConditions: {
          space: {
            ComparisonOperator: 'EQ',
            AttributeValueList: [{ S: space }],
          },
        },
        ExclusiveStartKey: exclusiveStartKey,
        AttributesToGet: ['root', 'shards'],
        ScanIndexForward: false
      })
      const response = await dynamoDb.send(prevCmd)
      const lastKey = response.LastEvaluatedKey && unmarshall(response.LastEvaluatedKey)
      const prevCursor = lastKey ? lastKey.root : undefined

(may need to tweak some things to avoid off-by-one errors)

I don't think it's possible to avoid doing two queries here, unless there's some way to scan forward and backward in a single DynamoDB query, so my question is: is adding an extra database query to every upload/list worth it to provide a UX convenience? I genuinely don't have good intuition for this at the moment, so would love your thoughts @alanshaw.

I'm going to hold off digging deeper on this until we can hash this out at a design level...

[travis]

OK - chatting out of band we decided this probably is the right way to go, since returning a cursor to the previous page would require an extra DynamoDB query. While I'm not a huge fan of driving protocol design with the constraints of a particular implementation, I do think that this is likely to be an issue in just about any database we implemented this with, and is probably appropriate in this case.

Given that, I think this PR is ready for review!

[alanshaw]

Code changes look good but maybe we should name this parameter differently? prev implies there's some kind of time element here - i.e. it was previously retrieved...but that's not necessarily the case.

Maybe this is weird but what about just pre?: boolean or precursor?: boolean.

[gobengo]

prev implies there's some kind of time element here

IMO prev implies ordering but not necessary along the time dimension, and it makes sense here too in the same way it makes sense in:

• html spec link rel 'prev'

  The prev keyword indicates that the document is part of a sequence, and that the link is leading to the document that is the previous logical document in the sequence.

  • which is in IANA link relation registry https://www.iana.org/assignments/link-relations/link-relations.xhtml
• also used for ordered paging (but not necessarily by time) in AS2 which iirc was informed by ldp-paging doing the same

I don't care very much what we call it, mostly jumping in to point out how other specs have distinguished (or not) between separating ordering along some dimension vs ordering along (one) time dimension

[travis]

yep, fwiw it also doesn't feel time-related to me, and I think the existence of the html link rel makes it feel pretty intuitive.

though I suppose that's the name of a link relation, and this is a boolean that determines whether the index should be read forward or backward...

@alanshaw how would you feel about about rev: boolean, short for "reverse" - this option is specifically designed to communicate that the index should be read in reverse, so that feels pretty intuitive, plus: three letters (good for inclusion in a UCAN) and just one letter off of prev...

[travis]

oh lol, reviewing this I realized I made the case against calling it rev initially, and the implications about sort order do still seem relevant...

I'm down to call it pre! short (ie, good for inclusion in UCANs) and vague enough that it doesn't feel like it implies much, but directional enough that it should read reasonably in code.

I'll get this PR updated.

[alanshaw]

@gobengo noted, thank you. Although, the things you mentioned (rel, AS2 etc.) are links sent by the server to the previous document in the sequence but here we're using it as an option (a boolean) from the client to enable a specific behaviour in the server. i.e. that we want the page of results preceeding the cursor we've sent. So I think this is a different enough situation to warrant a divergence, and as @travis pointed out, folks can think of pre as "previous" if that works better for them.

Ideally we'd have what I was originally advocating for - which would be previous and next, sent by the server...but we decided against due to limitations of our tech choices 🙈.