Index Process

The Ayahs index process works as illustarted below:

• Ayahs CSV file is uploaded to Azure Blob storage files container
• The parser Azure Function:
  • Triggered by the new CSV file upload
  • Parses the CSV file into configurable chunks
  • Enqueues each chuk to Azure storage ayah-items queue

The parser Function enqueues all the chunks in parallel and returns to caller. The main idea is not to block the Function processing so that it may timeout and causes scabality issues as one Function can only run on a single VM. By chunking and enqueuing, we engage another Function to process each chunk to make the process a lot more scalable. Azure Durable Functions handle this pattern (i.e. Fan out) very eloqunetly but, unfortunately, they (Durable Functions) are not supported in .NET5.

• The indexer Azure Function:
  • Triggered on each enqueued chunk
  • Unpacks each chunks and enriches each Ayah by auto-translating
  • Pushes ayah documents to the Azure Search Service upon completion