200k documents search leads to maximum call stack exceeded

[user472]

Describe the bug
When searching with Lyra for 200k documents in it it throws the error Maximum call stack exceeded. This is error is thrown exactly starting from 100,188 documents matching the search query.

To Reproduce

Execute the code below.

import { create, insertBatch, search } from '@lyrasearch/lyra';

async function main() {
  const db = await create({
    schema: {
      title: 'string',
    },
  });

  // the search works exactly until a document count of 100187
  const data = Array.from({ length: 200_000 }).map(() => ({ title: 'tttt' }));

  await insertBatch(db, data);

  const res = await search(db, {
    term: 'tttt',
    properties: ['title'],
  });

  console.log('hits count', res.hits.length);
}

main();

Expected behavior
Search result should be returned by lyra without throwing an error.

Desktop (please complete the following information):

• OS: macOS Ventura 13.2.1
• ts-node version 10.9.1 and tauri version 1.2.3

[micheleriva]

Hey there,
there might be many reasons why you're running out of call stack space.

1. We have a PR open that avoids many recursive calls, which may lead to this error specifically from time to time (feat: Rewritten with components architecture. #305)
2. It also depends on your dataset size and available RAM; Lyra is in-memory, so needs to load everything in-memory to operate

[user472]

Hello, thanks for looking into this.
2. I can reproduce this issue with the shared code snippet. I don't think I run out of memory for storing 200k strings in it.

1. That sounds more like the thing that causes the issue. Good to hear that there is already a solution on it's way.

[user472]

Hello
Since the above PR got merged I though I give it another shot. I executed the code mentioned in this ticket again.

Orama version: 1.0.0-beta.4
Node: 18.10.0

The error is still thrown.

[micheleriva]

@user472 on what machine are you running Orama? How much RAM do you allocate to the process? Are you in a container?

I tested locally with 1M docs on a MacBook Pro M1 Pro (16GB RAM) and I had no problems at all, it might depend on the environment. 200k strings are not that much, unless they're like billions of characters each

[user472]

@micheleriva I am on a MacBook Pro M1 Max (32 GB RAM) and also running it locally without any restrictions on how much RAM it is allowed to use. The string that I store for each document is still tttt so this should also not be an issue.

Which version of node are you running?