-
-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: carry curation rules and synonyms to new collection after scraper runs #66
feat: carry curation rules and synonyms to new collection after scraper runs #66
Conversation
6e024ff
to
6c1719e
Compare
b8d139c
to
f86b210
Compare
- Add default typesense server configuration variables to a .env file during pipeline run
- Change the volume mapping for the Typesense Docker container in the GitHub Actions workflow. The data directory is now correctly mapped to `/tmp/typesense`.` - Sleep for 10 seconds before sending the request to the Typesense server.
pass | ||
|
||
self.typesense_client.aliases.upsert(self.alias_name, {'collection_name': self.collection_name_tmp}) | ||
self.typesense_client.aliases.upsert( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tharropoulos Could we only update the alias and point it to the new collection, once all the synonyms and curation rules have been transferred over?
Because as it currently stands, there will be a brief moment when searches will be sent to the new collection, without synonyms and curation rules copied over yet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This creates side effects indeed, resolved in 41aa511. Still kept the deletion of the old collection being executed last in the workflow.
- Move transfer operations after alias upsert to ensure smooth transition. - Delete old collection after successful transfer.
- Replace static ChromeDriver path with WebDriver Manager for automatic driver installation. - Add webdriver-manager to Pipfile dependencies.
- Simplify the `itertext` method by consolidating the yield statements (addresses a pylint warning).
Just saw that |
- After updating, pylint returned errors for duplicate code on tests, so marking the flag as ignored resolves the issue
This is now available in v0.10.0 |
Change Summary
Updates the scraper workflow to ensure that user-defined synonyms and curation rules are preserved across runs. Specifically, it adds functionality to copy these rules from the old collection to the new collection before the old collection is deleted. This change helps maintain user configurations without disruption during the scraping process.
Changes include:
PR Checklist