Related articles with OpenAI embeddings on SQLite vectors

[rossta]

Articles now display related articles.

In this PR, we explore the using of "embeddings" for similarity search. Embeddings are "a measure of the relatedness of text strings" [as described in the OpenAI docs](OpenAI’s text embeddings measure the relatedness of text strings). We’re using OpenAI to generate embeddings for each article, storing embeddings as vectors in SQLite using the sqlite-vec extension, and querying for related articles based on embedding "distance" (similarity) across articles.

Helpful resources

• Docs for sqlite-vec: https://alexgarcia.xyz/sqlite-vec/
• OpenAI Embeddings guide: https://platform.openai.com/docs/guides/embeddings/
• Recent Aaron Francis video on SQLite vectors https://www.youtube.com/watch?v=w0FTWcuyliA

Though Aaron‘s video uses PHP and libsql (a SQLite fork that has its own vector support), the concepts are similar and serve as inspiration to what we accomplish here.

This builds on #273 where we implemented Page topics including content analysis with OpenAI to associate articles with topics.

[codecov]

Codecov Report

Attention: Patch coverage is 97.70115% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
app/models/page/sitepressed.rb	96.15%	1 Missing ⚠️
app/models/types/vector.rb	87.50%	1 Missing ⚠️

Files with missing lines	Coverage Δ
app/jobs/pages/batch_embedding_job.rb	100.00% <100.00%> (ø)
app/jobs/pages/batch_upsert_pages_job.rb	100.00% <100.00%> (ø)
app/jobs/pages/embedding_job.rb	100.00% <100.00%> (ø)
app/models/page.rb	100.00% <100.00%> (+2.32%)	⬆️
app/models/page/similarity.rb	100.00% <100.00%> (ø)
app/models/page_embedding.rb	100.00% <100.00%> (ø)
app/views/components/pages/timestamp.rb	100.00% <100.00%> (ø)
app/views/components/pages/topics.rb	85.71% <100.00%> (-0.96%)	⬇️
lib/tasks/pages/embeddings.rake	100.00% <100.00%> (ø)
app/models/page/sitepressed.rb	96.15% <96.15%> (ø)
... and 1 more

... and 1 file with indirect coverage changes