save and update app evaluations #18
Conversation
There was a problem hiding this comment.
Thanks for the code. I have left some minor comments in the code.
I think if it works, let's use it right now as is, but in the short run we need to heavily refactor it.
I don't think the api route should interact directly with mongodb. We need to add an interface between so that we don't get screwed in the short run
I also don't think the current schema would work in the longer run, when we have different datasets. Probably we need something more complicated then.
But I think the goal right now is to get something out of the door, get some feedback, and if we feel we are going in the right direction, we get go back and build the evaluation data management / api in a more robust manner.
I chatted with chatGPT a little bit about the schema, and here is the naming conventions they mentioned
Absolutely, renaming the "Model" entity to "Variant" makes sense given the context. So here is the updated nomenclature:
Project: This is the highest-level entity, which encompasses a series of comparisons between different variants with the same number and type of inputs and outputs.
Variant: An entity that contains the details of each specific machine learning model variant, including metadata like variant name, creation date, creator, and so forth.
Input: A representation of the inputs that each variant in a specific project requires.
Output: A representation of the outputs that each variant in a specific project generates.
Dataset: This is the input data for the variants. It should include a unique identifier, along with the actual data.
Comparison: This entity represents a specific instance where two variants are compared. It will include information about which variants were compared, the dataset used, and the results of the comparison.
Prediction: This entity represents the output from each variant for each row in the dataset during a comparison.
Evaluation: This entity represents the user's assessment of which variant performed better for each row in the dataset.
Variant_Input: This is a many-to-many relationship entity between Variants and Inputs. It represents the specific inputs a variant requires.
Variant_Output: This is a many-to-many relationship entity between Variants and Outputs. It represents the specific outputs a variant produces.
Variant_Project: This is a many-to-many relationship entity between Projects and Variants. It represents the variants included in a project.
| @@ -0,0 +1,7 @@ | |||
| from motor.motor_asyncio import AsyncIOMotorClient | |||
|
|
|||
| client = AsyncIOMotorClient("mongodb://username:password@mongo:27017") | |||
There was a problem hiding this comment.
I would add these in docker-compose. or at least add a #todo in the code so that we don't forget to do it later
|
|
||
| router = APIRouter() | ||
|
|
||
| @router.post("/", response_model=AppEvaluationsExperiment) |
There was a problem hiding this comment.
why use / not /create
I would expect / to list the list of evaluationexperiments
| else: | ||
| raise HTTPException(status_code=500, detail="Failed to create app_evaluation_entry") | ||
|
|
||
| @router.post("/{app_evaluation_experiment_id}/app_evaluation_entry", response_model=AppEvaluationEntry) |
There was a problem hiding this comment.
i would go with /{}/add_entry
if there is no verb, I would expect to get something
save and update app evaluations
No description provided.