Redesign base path file structure for caching and data exploration

[PhilippeMoussalli]

The current structure of the base path is as follows:

base_path
├── subset_1
│   └── <pipeline_name>-<pipeline_id>
│       └── data.0.parquet
├── subset_2
│   └── <pipeline_name>-<pipeline_id>
│       └── data.0.parquet
├── component_1
│   └── manifest_<hash_key_1>.json
└── component_2
    └── manifest_<hash_key_2>.json

Note:

• hash key is still not included in the main branch
• Keep in mind that we now have a unified structure between the local and remote runner

We need to have a required file structure that makes it easy to:

1. Find all the corresponding components of a given pipeline_name and pipeline id
   • This is needed for the data explorer where the user might select a pipeline name -> pipeline_id (drop down based on pipeline name) -> component and gets a list of all possible subsets to explore given the component's manifest
2. Final all the manifests for a certain component name (see Implement caching workflow #325)
   • This is needed for the caching mechanism where we check for a cached pipeline run within a certain component folder based on the hash key appended to the manifest name.

The existing file structure only enables us to accomplish (2). For the data explore, the user has to now manually give the manifest path which is not very intuitive. It can also get very cluttered with an increase number of subsets and components.

In order to enables both, I would propose to move to the following file structure:

base_path/
├── pipelines/
│   └── <pipeline_name>/
│       └── <pipeline_id>/
│           ├── component_1/
│           │   ├── manifest_<hash_key_1>.json
│           │   └── subsets/
│           │       ├── <subset_1>/
│           │       │   └── part.0.parquet
│           │       └── <subset_2>/
│           │           └── part.0.parquet
│           └── component_2/
│               └── manifest_<hash_key_2>.json
└── manifest_references/
    ├── component_1/
    │   └── references.json
    └── component_2/
        └── references.json

The pipelines directory enables (1) and the manifest_references directory enables (2). references.json is a file that contains references to all the manifests of a certain component starting from the base path it can be updated.

manifest_references/component_1/references.json

{ "manifest_<hash_key_1>": "pipelines/pipeline_name/pipeline_id/component_1/manifest_<hash_key_1>.json", 
 "manifest_<hash_key_2>": "pipelines/pipeline_name/pipeline_id/component_2/manifest_<hash_key_2>.json"
}

An alternative solution could be to just write all the manifests as well under manifest_references but that would lead to duplicated manifests.

The subset directory is also restructured to avoid storing subsets across different pipelines in the same directory and makes the base path less cluttered.

[RobbeSneyders]

I don't see why we can't solve one of these requirements using a glob pattern?

I agree that the structure can be improved, but I would keep it simple:

- pipeline name
  |- pipeline run
     |- component 1
     |  |- manifest.json
     |  |- subset 1
     |     |- *.parquet
     |  |- subset 2
     |     |- *.parquet
     |- component 2
        |- ...

Then you can find all the components for a given run of a pipeline (1) as pipeline_name/pipeline_run/* and all the manifests of a component of a pipeline (2) as pipeline_name/*/component_name/manifest.json

[PhilippeMoussalli]

I don't see why we can't solve one of these requirements using a glob pattern?

I agree that the structure can be improved, but I would keep it simple:

- pipeline name
  |- pipeline run
     |- component 1
     |  |- manifest.json
     |  |- subset 1
     |     |- *.parquet
     |  |- subset 2
     |     |- *.parquet
     |- component 2
        |- ...

Then you can find all the components for a given run of a pipeline (1) as pipeline_name/pipeline_run/* and all the manifests of a component of a pipeline (2) as pipeline_name/*/component_name/manifest.json

I guess I just wasn't aware of glob patterns 😂
That simplifies things quite a bit indeed

[PhilippeMoussalli]

@RobbeSneyders It seems like fs.glob() is very slow when working with blobs.

I tried the following two glob patterns

from fondant.filesystem import get_filesystem


glob_path = "gs://soy-audio-379412_kfp-artifacts/custom_artifact/commoncrawl-pipeline/*/common_crawl_download_component/manifest_6de9482e66e860779942466ba7b0d215.json"

glob_path = "gs://soy-audio-379412_kfp-artifacts/custom_artifact/commoncrawl-pipeline/commoncrawl-pipeline-20230829150955/*/manifest_61c839281b6c80e65c68f414b09dd260.json"

fs = get_filesystem(glob_path)

print(fs.glob(glob_path))

And they all seem to run quite slow. For reference this is the pipeline where you created almost 10k partitions for the subsets. I would expect the search to stop at the manifest level but it seems to be also searching through the subsets directories.

I think it's related to this:

In contrast, gs://bucket/*abc.txt asks the server for the complete list of objects in the bucket root, and then filters for those objects whose name ends with abc.txt. 

Might be better in that case to implement the initial suggestion above. Wdyt?

[RobbeSneyders]

That indeed seems slow 🐌

I would propose this as an iteration on your proposal above:

base_path/
├── pipelines/
    └── <pipeline_name>/
        ├── <pipeline_id>/
        │   ├── component_1/
        │   │   ├── manifest.json
        │   │   └── subsets/
        │   │       ├── <subset_1>/
        │   │       │   └── part.0.parquet
        │   │       └── <subset_2>/
        │   │           └── part.0.parquet
        │   └── component_2/
        │       └── manifest.json
        ├── cache/   
            ├── <hash_key1>
            ├── <hash_key2>
            ├── <hash_key3>

Where each <hash_keyx> file just contains the path to the relevant manifest.

Since we cache per pipeline name and the hash key should be unique across components, I think this will do, and it's very explicit.

[PhilippeMoussalli]

Alright, one other important consideration is concerning the way we currently work with the dev tag. I think we should still overwrite the manifest in cache/hash_key in case it's already exists and caching is set to False. Mainly to circumvent the way we work with dev tags. This explains it a bit better

[RobbeSneyders]

Yes, agree. The cache parameter should only define if the component should check if it can be cached, not if it needs to write cache information for the future.