Stream an object to reconstruct it on the frontend

[davidfou]

Hello 👋

First of all, thanks for the hard work done on this library. Understanding how to use it properly can take time, though it is worth the investment.

With @justinberiot, we wanted to use it for a particular use case. In our context, the JSON is not large (the complete data can fit in memory without any issue). The complexity is much more on the fact it can take a lot of time to get it fully, and it can be a deeply nested structure.

Our objective is to have a frontend which builds incrementally whenever some information is available. We managed to do so thanks to your library. Our code is available on the repo davidfou/stream-json-object.

Code snippet

// Available at https://github.com/davidfou/stream-json-object/blob/main/gif/index.mjs
import Stream from "node:stream";
import timers from "node:timers/promises";
import util from "node:util";
import StreamJSON from "stream-json";
import fetch from "node-fetch";
import _ from "lodash";

const updatePath = (path) => {
  const lastPathValue = path[path.length - 1];
  if (typeof lastPathValue === "number") {
    return [...path.slice(0, -1), lastPathValue + 1];
  }
  return path;
};

await Stream.promises.pipeline(
  async function* () {
    const response = await fetch(
      "https://raw.githubusercontent.com/davidfou/stream-json-object/main/demo/solar_system.json"
    );
    if (!response.ok) {
      throw new Error("Oups");
    }
    yield* response.body;
  },
  StreamJSON.parser({
    streamKeys: false,
    streamValues: false,
  }),
  // Simulate some latency
  async function* (source) {
    for await (const chunk of source) {
      await timers.setTimeout(50);
      yield chunk;
    }
  },
  async function* (source) {
    let path = [];
    for await (const chunk of source) {
      const lastPathValue = path[path.length - 1];
      switch (chunk.name) {
        case "startArray":
          path = [...path, 0];
          break;
        case "endArray":
          path = path.slice(0, -1);
          if (lastPathValue === 0) {
            yield { key: path, value: [] };
          }
          path = updatePath(path);
          break;
        case "startObject":
          path = [...path, null];
          break;
        case "endObject":
          path = path.slice(0, -1);
          if (lastPathValue === null) {
            yield { key: path, value: {} };
          }
          path = updatePath(path);
          break;
        case "keyValue":
          path = [...path.slice(0, -1), chunk.value];
          break;
        default:
          yield {
            key: path,
            value:
              chunk.name === "numberValue"
                ? parseFloat(chunk.value)
                : chunk.value,
          };
          path = updatePath(path);
      }
    }
  },
  // events sent by a backend and reconstruction of the object on the frontend
  async function* (source) {
    let out = null;
    let i = 0;
    for await (const chunk of source) {
      if (out === null) {
        out = typeof chunk.key[0] === "number" ? [] : {};
      }
      out = _.set(out, chunk.key, chunk.value);
      i += 1;
      console.clear();
      console.log(
        util.inspect(out, { depth: null, colors: true, breakLength: 171 })
      );
    }

    console.log("object recreated in %i steps", i);
  }
);

Here is what we wonder about:

• The library is excellent at parsing some large JSON files. Though, it took us some time to build the code above. Could what we've made be valuable for your library?
• Do you have an opinion about the usage of the library? Would it be possible to make something similar and more maintainable by using the library more appropriately?
• Is another library more adequate for our problem?
• Any feedback is welcome 🙏

[sneko]

In my case I have response body that cannot fit into Node.js string, so I wanted to use the stream response.body.getReader() with a "stream-to-json" library. It seems stream-json is one of the last ones maintained so I chose it.

But it's not obvious at all how to make it parsing the whole stream to build the proper object, without using filter or so. Because my object is nested (can be 100+ levels), I cannot rely on array or object helpers presented in the README.md.

Did I miss something to do "passthrough parsing" with stream-json like https://www.npmjs.com/package/bfj could do (even if archived for a few months)?

It feels weird to me to reimplement https://github.com/davidfou/stream-json-object/blob/main/src/streamObjectTransformer.mjs to have the rebuild of the object. Would feel like I'm the only one doing this?

cc @davidfou @uhop

[uhop]

@davidfou If I understood correctly the code in the repo produces key-value pairs, which are more complex that the ones produced by StreamArray and StreamObject. Essentially, it is a flat structure value-wise with full paths instead of keys. Did I miss anything?

stream-json has such code in FilterBase — see options.filter function. It is possible to expose it at the level of StreamObject.

As to usefulness of such component: I try not to judge the usefulness because it is in the eye of the beholder. Typically I consider the following criteria:

• Is it generic enough?
• Is it taxing? For example, does it require a lot of resources? (CPU, memory)
• Does it affect users in some way if they don't need it (e.g., they have to load it anyway)?
• Is it hard to support?

Adding a component that uses a value path as a key fits the bill. If there is a request for that, I would add it.

Thoughts? Comments?

PS: If I am way off, please correct me so I re-evaluate my position.

[davidfou]

Regarding your first sentence, yep, you got it write. We destructured and restructured it without any consideration of the key name or depth. In our use-case, the size of the final JSON wasn't an issue. We wanted to send data to the frontend as soon as possible (the JSON was generated with ChatGPT).

[uhop]

@sneko What is "passthrough parsing"? Could you describe the functionality you need, or find missing? Thank you in advance.

[sneko]

I have fetch response body as a stringified JSON:

{
    "document": {
        "id": "1",
        ...
        "children": [
            {
                "id": "1:1",
                ...
                "children": [
                    ...
                ]
            },
            {
                "id": "1:2",
                ...
                "children": [
                    ...
                ]
            },
            ...
        ]
    }
}

It's a tree that can have a huge amount of nested levels. It always fits into memory on a computer with a few GB of memory but if I do response.json() I get the error:

Error: Cannot create a string longer than 0x1fffffe8 characters
    at TextDecoder.decode (node:internal/encoding:449:16)
    at utf8DecodeBytes (node:internal/deps/undici/undici:2973:34)
    at parseJSONFromBytes (node:internal/deps/undici/undici:4306:25)
    at successSteps (node:internal/deps/undici/undici:4288:27)
    at fullyReadBody (node:internal/deps/undici/undici:2724:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async consumeBody (node:internal/deps/undici/undici:4297:7)
    at getResponseBody (/xxxxxxxx/core/request.ts:217:13)
    at <anonymous> (/xxxxxxxx/core/request.ts:322:26) {
  code: 'ERR_STRING_TOO_LONG'
}

So my idea was instead to get the content stream with response.body that does not require a intermediate step where it stores the whole content into a string. While looking at stream-json I didn't see a simple way to have a "passthrough" behavior that would mimic the exact JSON.parse() behavior (meaning: parsing a stream).

From what I see, I have to implement all mappers to rebuild the object: startArray, startObject.... Am I wrong?

[uhop]

I suggest to take a look at Assembler, which is a streaming implementation of JSON.parse(). The page has a sample code. If it doesn't fit your requirements, please let me know and I'll whip up a more suitable example.

PS: Yes, it is possible to do the opposite. See:

• Disassembler
• Stringer

[sneko]

@uhop thank you for your answer. Using Assembler is doing what I want.

I don't get why I missed it when reading the README.md, maybe a mention of JSON.parse() / JSON.stringify() next to their "stream implementations" inside the README could help inattentive people like me 😄 .

Just to give more insights:

• My JSON file is 600MB
• Using bfj I never succeeded in parsing it, running out of memory by default, and if optimizing with node --optimize_for_size --max_old_space_size=10000 --gc_interval=100 script.js it appears after 23 minutes the object was not parsed yet so I gave up (note that the memory usage was really high)
• With stream-json, without tweaking Node.js options, the entire file was parsed within 1min30 and it's using 3GB of memory at the end

Since I'm doing comparaison between 2 huge trees, using streams "1 by 1" was not an option to me. But your library helps me parsing the whole from a request stream, which unblocks me a technical limitation. Thank you for that!

[uhop]

Is another library more adequate for our problem?

I started stream-json way before many other libraries. The initial commit in this repository was made 11 years ago yet the project itself is much older — I wrote several non-public projects in the same vein before stream-json. There were some other JSON-parsing packages back then, and I tried them for my projects and unfortunately they all failed:

• Some assumed size limits on numbers and/or strings.
• Some were slow for my tasks.
• Some chose to emulate streams to speed things up doing so badly.
  • For example, resume/pause was not implemented properly if it all.
• Some had bugs, which were critical for my projects.
  • For example, some streaming libraries were not actually streaming: they tried to read everything in memory, which worked just fine for tests, but not for my data.

So I implemented and published stream-json and never looked back. You guys asking a wrong person for recommendations on other libraries. ;-)