feat: carry curation rules and synonyms to new collection after scraper runs #66
Conversation
tharropoulos
force-pushed
the
feature/curation-synonym-carrying
branch
from
6e024ff
to
6c1719e
Compare
tharropoulos
force-pushed
the
feature/curation-synonym-carrying
branch
from
b8d139c
to
f86b210
Compare
- Add default typesense server configuration variables to a .env file during pipeline run
- Change the volume mapping for the Typesense Docker container in the GitHub Actions workflow. The data directory is now correctly mapped to `/tmp/typesense`.` - Sleep for 10 seconds before sending the request to the Typesense server.
| pass | ||
|
|
||
| self.typesense_client.aliases.upsert(self.alias_name, {'collection_name': self.collection_name_tmp}) | ||
| self.typesense_client.aliases.upsert( |
There was a problem hiding this comment.
@tharropoulos Could we only update the alias and point it to the new collection, once all the synonyms and curation rules have been transferred over?
Because as it currently stands, there will be a brief moment when searches will be sent to the new collection, without synonyms and curation rules copied over yet.
There was a problem hiding this comment.
This creates side effects indeed, resolved in 41aa511. Still kept the deletion of the old collection being executed last in the workflow.
- Move transfer operations after alias upsert to ensure smooth transition. - Delete old collection after successful transfer.
- Replace static ChromeDriver path with WebDriver Manager for automatic driver installation. - Add webdriver-manager to Pipfile dependencies.
- Simplify the `itertext` method by consolidating the yield statements (addresses a pylint warning).
|
Just saw that |
- After updating, pylint returned errors for duplicate code on tests, so marking the flag as ignored resolves the issue
|
This is now available in v0.10.0 |
Change Summary
Updates the scraper workflow to ensure that user-defined synonyms and curation rules are preserved across runs. Specifically, it adds functionality to copy these rules from the old collection to the new collection before the old collection is deleted. This change helps maintain user configurations without disruption during the scraping process.
Changes include:
PR Checklist