​ Problem with remote evaluation worker

[JannikSchneider12]

Hello everyone,

I am trying to set up a remote evaluation challenge and facing issues. When I submit a file in the web interface, the fetching as well as the evaluation script are working but at the end, I am getting this error:

Error: 400 Client Error: Bad Request for url: https://eval.ai/api/jobs/challenge/2389/update_submission/

I also printed out the data that is passed to the update_submission_data function:

{‘challenge_phase’: 4723, ‘submission’: 477497, ‘stdout’: ‘’, ‘stderr’: ‘’, ‘submission_status’: ‘FINISHED’, ‘result’: [{‘train_split’: {‘final_train_loss’: 0.00010908210573358123, ‘final_val_loss’: 0.0009243950532566357, ‘Metric3’: 123, ‘Total’: 123}}, {‘test_split’: {‘final_train_loss’: 0.00010908210573358123, ‘final_val_loss’: 0.0009243950532566357, ‘Metric3’: 123, ‘Total’: 123}}], ‘metadata’: ‘’}

Can someone help me whats wrong?

Thanks for your time and help

[PredictiveManish]

Does anyone working on this? If not interested to work on this issue. Please assign it to me.

[JannikSchneider12]

Just to give an update. So I changed the structure of my results dictionary and now it runs without errors. However while the result files for the submissions are now present, the leaderboard remains empty, so I am not sure whether something with my format is still wrong. One result file in the web interface looks e.g. like this:

[{'final_train_loss': 6.025793465431145e-05, 'final_val_loss': 0.00013760417953432937, 'Metric3': 123, 'Total': 123}]

And this is my leaderboard setting in the config:

leaderboard:

• id: 1
  schema:
  {
  "labels": ["final_train_loss","final_val_loss","Metric3","Total"],
  "default_order_by": "final_val_loss",
  "metadata": {
  "final_val_loss": {
  "sort_ascending": True,
  "description": "The last validation loss from training the linear probing layer",
  }
  }
  }