Explorer new dataset format

[PhilippeMoussalli]

PR that changes the explorer to match the new dataset format.

There is some weird refersh error that is occurring which might get resolved/ will be easier to debug after removing the partitions.
Let's focus on first getting this PR merged since it might fix the issue

[RobbeSneyders]

Thanks @PhilippeMoussalli!

Ran into some issues when testing:

• When trying to select a pipeline run on the first page, I get the following error:

  File "/home/robbe/.cache/pypoetry/virtualenvs/fondant-n6_n8sMX-py3.10/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 534, in _run_script
      exec(code, module.__dict__)
  File "/home/robbe/workspace/fondant/data_explorer/app/main.py", line 189, in <module>
      app.setup_app_page()
  File "/home/robbe/workspace/fondant/data_explorer/app/main.py", line 165, in setup_app_page
      selected_run_update_date = selected_run_info["Last Updated"].to_dict()[0]
  KeyError: 0
  

  It does work when it's first loaded.

• The dataset explorer flickers a couple times when first loaded, and after anything I do on the page
  

• If I refresh on any page except for the overview page, the page becomes narrow:
  

• I don't have images in my dataset, which leads to an error in the image viewer:
  

• I also get an error on the numeric analysis page. I don't have numeric columns, but I don't think that's the reason:
  

[PhilippeMoussalli]

Thanks @PhilippeMoussalli!

Ran into some issues when testing:

Yes I ran into a few of those as well, I introduced new updates to fix them:

- Flickering
Was due to the fact that the pandas dataframe kept reloading everytime and merging, fixed this by caching the function that loads it. We're using the mapping fields dict as the cache key in this case (not possible to cache dask dataframe object)

The rest of the small issues were also handled (pipeline visualization, image gallery, numerical analysis, fixed screen size). I fixed other issues as well to better navigate the datasets. Let me know if you run into any more issues

[RobbeSneyders]

Thanks, seems to work without issues now.

One downside to the global pagination is that all the built-in filtering, sorting, etc. now only works on the 20 loaded rows. Not sure if this is an issue.

[PhilippeMoussalli]

Thanks, seems to work without issues now.

One downside to the global pagination is that all the built-in filtering, sorting, etc. now only works on the 20 loaded rows. Not sure if this is an issue.

Yeah I think before it was also enabled per individual loaded partitions. The global search should partially solve this.

We can decide to completely omit the build it in filtering if having two search functionalities might seem confusing