Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add GitHub action that automatically creates the derivative databases #81

Merged
merged 11 commits into from
Mar 3, 2023

Conversation

rickstaa
Copy link
Collaborator

@rickstaa rickstaa commented Mar 3, 2023

This PR adds the '.github/workflows/create_db_derivatives.yaml'GitHub action that is used to automatically create the derivative databases (i.e. 'category', 'images' and 'extended') based on the 'plane-alert-db.csv', 'plane_images.txt' and 'blacklist.txt' files.

Files explained

  • .github/workflows/create_db_derivatives.yaml: A GitHub action file that runs every time somebody modifies the plane-alert-db.csv, plane_images.txt and blacklist.txt files.
  • scripts/create_db_derivatives.py: A file that creates the derivative databases based on the plane-alert-db.csv, plane_images.txt and blacklist.txt files.
  • scripts/create_images_reference.py: A small little script that I used to create the new plane_images.txt file. This file can be removed when we are sure that this file is correct.
  • plane_images.txt: This is the new planepix.txt file. I kept the file extension as txt to distinguish it from the main database and the derivative databases.
  • blacklist.txt: Contains planes that had to be removed because of the new Twitter rules.
  • plane-alert-db-extended: The `plane-alert-db.csv' database with the blacklist items added to it.
  • plane-alert-db-extended-images: Same as above but now with the image links added.

See the action in action

How to test

Before switching to this automatic workflow, we have to ensure that the generated databases are equal to the ones already in the repository. I compared them using VScodes compare feature, and the implementation looks correct to me. However, please double-check.

  1. Go into the scripts folder.
  2. Install the python dependencies.
  3. Add the planepix.txt file from the _bin folder.
  4. Overwrite the plane-alert-db-images.csv file with the one from the _bin folder.
  5. Run the create_images_reference.py script to generate the database of the new image.
  6. Run the create_db_derivatives.py script to generate the derivative databases.
  7. Compare the generated databases.

Things that were not yet automated

I did not know what was in the following databases:

  • badgers-best-images.csv
  • badgers-best.csv
  • plane-alert-wip.csv
  • plane-alert-pia.csv
  • plane-alert-search-terms-to-do.csv
  • plane-alert-wip.csv

@rickstaa rickstaa merged commit ce6b92c into main Mar 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant