-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flexible inputs in template repository for Algorithm containers #309
Comments
To do this we probably need to rework/rethink how the Algorithm class works. It also relates to #246, where we would like to adhere to the input/output algorithm interfaces here: https://grand-challenge.org/algorithms/interfaces/ . It might be a good idea to allow a user to specify expected interfaces for the inputs and outputs, so we know where to look for the inputs and where to put the respective outputs. e.g.:
This way the Algorithm class will know where to look/expect input files, it will know what default validation to use. Also, outputs will be written to default locations. We probably lose the ability to process multiple input files in a single run, but for grand-challenge this should be alright. |
Rather than asking all the questions we can expose the interface on the grand challenge side, ask the user for their API key, and then say, "What algorithm/challenge" do you want to create an algorithm for? Get all the interfaces for that algorithm, then create the template. We should reduce the number of options they have to fill out as most of them are unused. You can process multiple inputs by changing the base from |
I see, so you would like people to use grand-challenge to configuring/designing the interfaces to the algorithm/evaluation and subsequently query the interfaces for that algorithm (using gc-api) inside evalutils.
That's true, are you referring to all the algorithm-related settings, that are currently unused? E.g. required-number of CPUs etc... Should we make a separate issue for this?
That's great! For now, we can move forward to support the single case I guess, and later add support for multiple inputs. |
No need for a separate issue, can be tackled as part of a clean up here when they're redefined. |
Templating will be removed from this repo as that has moved to https://github.com/DIAGNijmegen/rse-grand-challenge-forge. |
I was looking at the code for creating an algorithm container. The
Algorithm
class has a_load_input_image
function which is constrained to load only one input image. Perhaps we could add a new function that accepts flexible inputs?evalutils/evalutils/evalutils.py
Lines 201 to 217 in dbfabbd
I need an update for my own algorithm container, but I can also imagine that other high-profile challenges will need evalutils to handle flexible inputs for algorithm containers. @jmsmkn, @silvandeleemput if you have any specific ideas for this, I can make a pull request for this feature.
The text was updated successfully, but these errors were encountered: