Additional SQLite connection optimizations

[daun]

(Hi 👋 Fantastic piece of software you've created, thanks a lot for open-sourcing it!)

I've been toying a bit with optimizing sqlite connections on production sites and thought these might make sense to include by default. The good thing is that Loupe is usually the only client of its particular database, so it can make a few assumptions about how to optimize the connection. Not sure if you agree with all of these suggestions, but it'd be great to get a discussion started.

• Add a few optimizations to reduce disk I/O
• Add a timeout to avoid errors if db is locked by another instance
• Enable incremental vacuum to auto-reduce database size
• Tested this on a few sites and I'm seeing about a 10% increase in speed (as measured by processingTimeMs)

[Toflar]

Hey man, thanks a lot for the props, the PR and taking the time to contribute - love it!
Would you mind targeting this against develop (future version 0.8), please?
Regarding the bench: I've included 2 scripts in the bin folder, can you share your results pre and post PR? 😎

[Toflar]

FYI: Indexing will take about 3 minutes :) You can adjust the config.php in the bin directory to use the movies_1000.json for quicker local tests before running the big one :)

[daun]

@Toflar Sure! Will do the benchmarks when I find a minute.

[Toflar]

I will bench on my machine too and let you know. I've just created another PR that improved indexing for me, would love to have your feedback on that too 😊 #103

[Toflar]

Oh man, the boost on my system is huge!! #103 brings the time from about 220s to 188s and together with your PR we are now at 137s!

[daun]

Sounds fantastic! I'm having trouble getting the benchmarks to run, but it looks like the results on your machine are speaking for themselves :)

[Toflar]

Sure, please just rebase on develop, then I'll happily merge it :-)

[daun]

On it :)

[daun]

Rebased. One last thing — could your try experimenting a bit with the page_size option? I've used a very modest increase from 4KB to 8KB, but I've seen very aggressive recommendations around. Maybe setting it to 16KB increases performance a bit more?

'PRAGMA page_size = 16384;',

[daun]

This might hurt performance when reading/searching, though, that's why I've been a bit conservative with page size.

[Toflar]

Does not have any effect for me :)

[Toflar]

Thanks a lot Philipp!