Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

Moving InnerEye's Azure code to hi-ml package #548

Merged
merged 68 commits into from
Aug 26, 2021
Merged

Moving InnerEye's Azure code to hi-ml package #548

merged 68 commits into from
Aug 26, 2021

Conversation

ant0nsc
Copy link
Contributor

@ant0nsc ant0nsc commented Jul 30, 2021

Closes Issue 62 on the hi-ml repository.

@dumbledad dumbledad marked this pull request as draft August 20, 2021 10:33
@dumbledad dumbledad self-assigned this Aug 20, 2021
Tests/AfterTraining/test_after_training.py Show resolved Hide resolved
environment.yml Outdated Show resolved Hide resolved
@dumbledad
Copy link
Contributor

dumbledad commented Aug 24, 2021

I have a problem with tests timing out. For example look at the last run of our InnerEye PR build on ADO for this PR: https://innereye.visualstudio.com/InnerEye/_build/results?buildId=43337&view=results The error is

##[error]The operation was canceled
##[error]The job running on agent Azure Pipelines 4 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

In the past we have seen timeout errors when the test environment fails to get the AML credentials (e.g. for the Service Principal) but that is not the case here, since the logs with the timeout (here) show the successful elevation before the timeout occurs.

Further evidence that this is not a problem restricted to this PR can be seen in this run: https://innereye.visualstudio.com/InnerEye/_build/results?buildId=43339&view=results The same tests are timing out if I run the our InnerEye build pipeline in ADO on our main branch!

@javier-alvarez & @mebristo, any ideas?

===== EDIT =====

The test Shruthi pinpointed (and removed) fixed the hanging runs AML bug for us.

@dumbledad dumbledad requested a review from mebristo August 24, 2021 14:05
@dumbledad dumbledad marked this pull request as ready for review August 24, 2021 14:05
InnerEye/Scripts/submit_for_inference.py Outdated Show resolved Hide resolved
InnerEye/ML/runner.py Outdated Show resolved Hide resolved
Tests/ML/test_download_upload.py Outdated Show resolved Hide resolved
Tests/AfterTraining/test_after_training.py Outdated Show resolved Hide resolved
Tests/ML/test_download_upload.py Show resolved Hide resolved
@dumbledad dumbledad enabled auto-merge (squash) August 25, 2021 19:36
@dumbledad dumbledad merged commit b35399f into main Aug 26, 2021
@dumbledad dumbledad deleted the antonsc/himl branch August 26, 2021 08:17
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch InnerEye to using hi-ml as a package
4 participants