Creating synthetic memories with webclient sometimes results in no summary.

[AndyEdmonds]

I'm using the following code. memory is a MemoryWebClient.

               var context = new RequestContext();
               context.SetArg("custom_summary_prompt_str", PolicyTypePrompt);
               context.SetArg("custom_summary_target_token_size_int", 8000);
               await memory!.ImportWebPageAsync(link, documentId: id, index: PolicyTypeIndex, steps: Constants.PipelineOnlySummary, context: context);
               int waitCount = 0;
               while (waitCount < waitTimeOut) // wait for the summarization to be ready
               {
                   if (await memory!.IsDocumentReadyAsync(documentId: id!, index: PolicyTypeIndex))
                   {
                       break;
                   }
                   Task.WaitAll(Task.Delay(2000));
                   waitCount++;
               }
               if (waitCount < waitTimeOut )
               {
                   var res = await memory.SearchSummariesAsync(filter: MemoryFilters.ByDocument(id!), index: PolicyTypeIndex);
                   if (res.Count > 0 && res[0].Partitions.Count > 0)
                   {
                       return res[0].Partitions[0].Text;
                   }
               }

I use the above to get some analyses of the page - generally a pdf - passed in as "link". So having waited for the document to process, I sometimes find that res is empty. The pipeline has completed, there are no faults, but there's no summary. What could cause this? How can I fix this?

[dluc]

hi @AndyEdmonds did you see any error in the service logs?

I can't see from the code the value of waitTimeOut, is the code waiting enough time?

Any chance that this happens only with some web pages or some web site blocking the download? Are the PDF protected, or perhaps very big to download?

If you can share some links where the problem occurs, I can run some tests.

[AndyEdmonds]

Thanks @dluc, waitTimeOut is set to 60, so plenty of time. It may well be that these pdfs/web pages are inaccessible, but nothing is returned by the web client to say that part of the pipeline failed - it's still marked as complete.

An example that fails is https://www.congress.gov/bill/117th-congress/senate-bill/2551/text
Another is: https://www.dni.gov/files/ODNI/documents/AIM-Strategy.pdf

I'm using the standard Kernel Memory Service on Azure. I can DM the prompt I'm using - it's business sensitive.

[AndyEdmonds]

Further to the above, the kernel memory service Application Insights log shows occasional 429s from Azure OpenAI, which I assume the kernel memory service handles, and 403s for the urls above. As you can try for yourself, both urls are freely accessible. Is there something about the calls, for instance the accepted return types, that are causing this in the kernel memory service? One would expect the service to be as catholic as possible.

[AndyEdmonds]

To investigate further I moved from the web client and used Kernel Memory locally. I created a custom web scraper, copying the code of the original web scraper, and have experimented with adding headers. Postman works with both these sites, but so far I can't get the web scraper to read them. Any ideas?