Return sorted/grouped data (idea dump)

[drewvolz]

Thin client vs fat client. I think there's a middle ground in here somewhere and we're going to find it.

A side conversation / extension / related set of ideas to:

• New data fetching strategy #2735

---

• Part 1 (thin client): what we have implemented today in v1 of the caching server vs proposed data server endpoint additions
• Part 2 (thick client): persistently store data using X

---

Part 1 — thin client

Reduce the amount of work the ccc-client has to do as its goal is to render data efficiently, quickly, and safely. This is only worthwhile if it makes sense to return grouped/sorted data from the caching server (so if we do more parsing of data for the clients on ccc-server, we could save a few rendering cycles, skip some memoization, and do less work overall on clients devices)

This provides more flexibility to change the data without necessarily changing the data contract, but depending on how you look at it this may also be a downside.

v1 (today's implementation)

• returns source content from CDN as-is
• we will always(?) need to support this for backwards compatibility

useFetch(API('/dictionary'), {
    ...
}

// return 
{
  "data": [
      {"word": "...", "definition": "..."},
      {"word": "...", "definition": "..."}
  ]
}

v2

proposed endpoint

useFetch(API('/v2/dictionary'), {
    ...
}

proposed endpoint with query parameters

useFetch(API('/v2/dictionary'), {
    {sortBy: ..., groupBy: ...}
}

proposed header

useFetch(API('/dictionary'), {
    headers: {
        accept: 'application/json',
        <custom-sort-and-group-parameters>
    },
}

both v2 endpoints would return formatted/grouped/sorted data by request

{
  "data": {
    "A": [
      {"word": "...", "definition": "..."},
      {"word": "...", "definition": "..."}
    ],
    "B": [
      {"word": "...", "definition": "..."}
    ]
  }
}

Part 2 — thick client

Alternatively, make the thick client even thicker.

I think one of the painpoints here is choosing where to store app data since it is not currently persisted. There are arguments for redis, sqlite, coredata (for apple clients).

simple sqlite storage

- KV store
- Url to data

key: url, value: serialized json?

we could:
- bundle that and then update it in place web successful requests,
- copy your sqlite file from the bundle to the Documents folder,
- then read/write to that file.

if file exists; return; else; copy; end

shipping with postgresql (note from hawken)

• ship the pg server binary,
• start it as a daemon on a local socket, and talk to it that way.

something else

What else would should we be considering?

[drewvolz]

More copy/pasted notes:

Okay, so if you have redux (-ish) you don't need the SQLite hooks I think.

You know what data is being mutated, so if redux stores a list of tables, and the actions say what tables they affect, then you've got the info you need right there. You could even make the state be like

state = { tables: { hours: {}, contacts: {} } }

and then have the action replace the appropriate empty objects with new ones, so that redux re-renders the components automatically

so, to recap

Step 1

Each part of the app gets a useMyData hook, like useBuildingHours.

let { data, isPending, refresh, isInitial, error } = useBuildingHours()

Today, that Just wraps the fetch call / local data lookup.

In the future, it talks to the local database.

Step 2

One local database file per app data type, I think, so we can attribute sizes? Maybe maybe not.

The root app initializes a database singleton at startup, and stores it in context.

useBuildingHours changes to query the local database, instead of just returning the in-memory data, and its "refresh" method takes responsibility for normalizing the response into the appropriate tables.

Step 3

Make database wrappers for each module. Maybe? How nice is it to have CRUD operations for each module… probably nice. Anyway. Step 3 is "expand operations from R to CRUD"