-
Notifications
You must be signed in to change notification settings - Fork 392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Hugging Face integration tests #904
Conversation
Solves #901 This PR adds a github workflow that runs integration tests with Hugging Face. These tests cover: - using a pretrained tokenizer - using a pretrained BERT model - storing model artifacts on the HF model hub The tests can be triggered manually at https://github.com/skorch-dev/skorch/actions and additionally run once per week on Sundays at 9:00 AM UTC.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if we can really test if the workflow works before merging, any ideas?
With a secret, I think the only way to check is to run the GitHub Action on a fork and link the results here.
That's a good idea, I'll try it out. |
- Fix typo: requirement.txt => requirements.txt - Better documentation - Fix error message
@thomasjpfan So I ran this on my fork of the repo and at least the manuel trigger works: https://github.com/BenjaminBossan/inferno/actions/workflows/test-hf-integration.yml As for the cron schedule, it didn't trigger, even though I tried different schedules. There are a few possible reasons for that:
Anyway, I'd be willing to go ahead, worst case the cron schedule doesn't trigger, we'd still have the manual job. Another question: Do we want to add a badge for this test to the README? Edit Now it was triggered on schedule: https://github.com/BenjaminBossan/inferno/actions/runs/3220073700 So 1. is the most likely reason. I'll decrease the frequency of the scheduled workflow on my fork now. |
That is the simplest solution to keep track of the scheduled job. The alternative solution is to auto-open an issue if the scheduled job fails. An example of a self contained version of this is in pydata/xarray. |
I added a badge. This is what it should look like: https://github.com/BenjaminBossan/inferno/actions/workflows/test-hf-integration.yml/badge.svg Other than that, I think we can also set the repo to send us notifications when an action fails, so auto-opening an issue for a feature which is not core to skorch seems to be overkill to me. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
A reference to the badge was added in #904 but for it to show, there needs to be an entry using that reference.
Just to note, the job successfully run via the scheduler: https://github.com/skorch-dev/skorch/actions/runs/3258933674. However, I couldn't figure out at what time it ran, so I'm not sure if the configured time was indeed used. |
Solves #901
This PR adds a github workflow that runs integration tests with Hugging Face. These tests cover:
The tests can be triggered manually here and additionally run once per week on Sundays at 9:00 AM UTC.
I'm not sure if we can really test if the workflow works before merging, any ideas?