Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add GitHub action that automatically creates the derivative databases #76

Merged
merged 2 commits into from
Mar 2, 2023
Merged

ci: add GitHub action that automatically creates the derivative databases #76

merged 2 commits into from
Mar 2, 2023

Conversation

rickstaa
Copy link
Collaborator

@rickstaa rickstaa commented Mar 2, 2023

This PR adds the '.github/workflows/create_db_derivatives.yaml'GitHub action that is used to automatically create the derivative databases (i.e. 'category', 'images' and 'extended') based on the 'plane-alert-db.csv', 'plane_images.txt' and 'blacklist.txt' files.

Files explained

  • .github/workflows/create_db_derivatives.yaml: A GitHub action file that runs every time somebody modifies the plane-alert-db.csv, plane_images.txt and blacklist.txt files.
  • scripts/create_db_derivatives.py: A file that creates the derivative databases based on the plane-alert-db.csv, plane_images.txt and blacklist.txt files.
  • scripts/create_images_reference.py: A small little script that I used to create the new plane_images.txt file. This file can be removed when we are sure that this file is correct.
  • plane_images: This is the new planepix.txt file. I kept the file extension as txt to distinguish it from the main database and the derivative databases.
  • blacklist.txt: Contains planes that had to be removed because of the new Twitter rules.
  • plane-alert-db-extended: The `plane-alert-db.csv' database with the blacklist items added to it.
  • plane-alert-db-extended-images: Same as above but now with the image links added.

See the action in action

How to test

Before switching to this automatic workflow, we have to ensure that the generated databases are equal to the ones already in the repository. I compared them using VScodes compare feature, and the implementation looks correct to me. However, please double-check.

  1. Go into the scripts folder.
  2. Install the python dependencies.
  3. Run the create_images_reference.py script to generate the database of the new image.
  4. Run the create_db_derivatives.py script to generate the derivative databases.
  5. Compare the generated databases.

Things that could cause problems

My script currently replaces N/a with np.nan. I don't know how this field is read in the https://github.com/kx1t/docker-planefence repository, but otherwise, we have to add those values back in.

Things that were not yet automated

I did not know what was in the following databases:

  • badgers-best-images.csv
  • badgers-best.csv
  • plane-alert-wip.csv
  • plane-alert-pia.csv
  • plane-alert-search-terms-to-do.csv
  • plane-alert-wip.csv

rickstaa and others added 2 commits March 2, 2023 19:00
…ases

This commit adds the '.github/workflows/create_db_derivatives.yaml' GitHub
action that is used to automatically create the derivative databases
(i.e. 'category', 'images' and 'extended') based on the
'plane-alert-db.csv', 'plane_images.txt' and 'blacklist.txt' files.
name: Create and commit category and image database files

on:
push:
Copy link
Collaborator Author

@rickstaa rickstaa Mar 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should change this to only run on pull requests? Now it runs both on the PR and on the main branch.

@kx1t kx1t merged commit 3ee1938 into sdr-enthusiasts:main Mar 2, 2023
@Phaeton Phaeton mentioned this pull request Apr 2, 2023
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants