Segmentation fault when running ingestion loop locally.

[jeroenleenarts]

This is what I see in my output.

I followed the steps outlined in "LOCAL_DEVELOPMENT_SETUP.md"

➜  SwiftPackageIndex-Server git:(main) ./scripts/ingest-loop.sh
[0/0] Build complete!
[ INFO ] Ingesting (limit: 100) ... [component: ingest]
./scripts/ingest-loop.sh: line 28: 35947 Segmentation fault: 11  swift run Run ingest -l 100
➜  SwiftPackageIndex-Server git:(main)

On the locally served page I see "Indexing 4,449 packages and 0 versions." So something has happened. Querying the docker DB also confirms this.

Impressed with the entire Docker based setup by the way. :)

[jeroenleenarts]

I just noticed this. If I reset the DB the ./scripts/ingest-loop.sh does not crash, but there is also nothing to ingest. If I do the following with a clean DB (right after migrate), I get the Segmentation fault.

➜  SwiftPackageIndex-Server git:(main) make reconcile
swift run Run reconcile
[0/0] Build complete!
[ INFO ] Reconciling ... [component: reconcile]
[ INFO ] done. [component: reconcile]
[ WARNING ] Environment variable not set: METRICS_PUSHGATEWAY_URL [component: reconcile]
➜  SwiftPackageIndex-Server git:(main) ./scripts/ingest-loop.sh
[0/0] Build complete!
[ INFO ] Ingesting (limit: 100) ... [component: ingest]
./scripts/ingest-loop.sh: line 28: 37741 Segmentation fault: 11  swift run Run ingest -l 100
➜  SwiftPackageIndex-Server git:(main)

[jeroenleenarts]

Looks like it is because I am missing a .env file.

I have a .env.development file.

[daveverwer]

Looks like it is because I am missing a .env file.

Because of the METRICS_PUSHGATEWAY_URL warning? That's safe to ignore.

[daveverwer]

Hi Jeroen!

The segfault is interesting and the good/bad news is that I can reproduce it!

Note for @finestructure: with a clean database (db-down and db-up) then migrating, running a reconcile, and then the ingest loop script crashed for me immediately with the same segfault.

That scenario is fairly rare though, so I'm not super surprised there is a hidden issue there.

Are you trying to get a populated database so you can fix/add something? If so, I can get a dump of the production database to you. We have a Discord where I can pass you a zip file. Trying to reconstruct the data from the reconciled JSON file is possible, but it's a painful task, as you found out.

Ping me on Discord and I'll get you a recent database dump.