Skip to content

Latest commit

 

History

History
80 lines (68 loc) · 4.04 KB

README.md

File metadata and controls

80 lines (68 loc) · 4.04 KB
Raw

Just a place for experimentation -- don't judge!

The idea for manifests layout is to be consistent with our /blobs/ on S3 where we provide a few levels of "declutter" subfolders from the first letters of the zarr UUID, e.g.

128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d

but unlike in case of blobs where the target is a blob file, in our case it is a folder under which we collect manifests as identified by the corresponding zarr checksum: e.g.

128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d/6ddc4625befef8d6f9796835648162be-509--710206390.json

which provides a full listing for the zarr, including the ETags per file, and sizes. But for the zarr access, I think only versionid is needed, so we have also a

128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d/6ddc4625befef8d6f9796835648162be-509--710206390.versionid.json

which has only versionId per each sub path.

Internally both files follow the same schema, and which fields are listed are listed in the header in the field fields which is either a list when multiple or a string when single value (saving space!), e.g.

yoh@typhon:~/proj/dandi/zarr-manifests$ head -n 25 zarr-manifests-v2-sorted/128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d/6ddc4625befef8d6f9796835648162be-509--710206390.json 
{
 "schemaVersion": 2,
 "fields": ["versionId","lastModified","size","ETag"],
 "statistics": {
  "entries": 509,
  "depth": 5,
  "totalSize": 710206390,
  "lastModified": "2022-06-27T23:09:39+00:00",
  "zarrChecksum": "6ddc4625befef8d6f9796835648162be-509--710206390"
 },
 "entries": {
  ".zattrs": ["VwOSu7IVLAQcQHcqOesmlrEDm2sL_Tfs","2022-06-27T23:07:47+00:00",8312,"cb32b88f6488d55818aba94746bcc19a"],
  ".zgroup": ["7obAY5BUNOdI1Uch3RoI4oHuGXhW4h0R","2022-06-27T23:07:47+00:00",24,"e20297935e73dd0154104d4ea53040ab"],
  ".zmetadata": ["Vfe0W0v4zkydmzyXkUMjm2Xr7.rIvfZQ","2022-06-27T23:07:47+00:00",15191,"4f505878fbb943a9793516cf084e07ad"],
  "0": {
   ".zarray": ["Ou6TnKwWPmEJrL.0utCWLPxgfr_lA0I1","2022-06-27T23:07:48+00:00",446,"5477ec3da352681e5ba6f6ea550ef740"],
   "0": {
    "0": {
     "13": {
      "8": {
       "100": ["lqNZ6OQ6lKd2QRW8ekWOiVfdZhiicWsh","2022-06-27T23:09:11+00:00",1793451,"7b5af4c6c28047c83dd86e4814bc0272"],
       "101": ["_i9cZBerb4mB9D8IFbPHo8nrefWcbq0p","2022-06-27T23:09:28+00:00",1799564,"50b6cfb69609319da9bf900a21d5f25c"],
       "103": ["DeraqBNPhVssSggCCTPjmgbu6XxOOupZ","2022-06-27T23:09:22+00:00",1811922,"c88b90dd6d4995fbb32b9a131d1b4ba0"],
       "104": ["J6ejQa4K9cEAjNbkKkZip8.zdK5XFI37","2022-06-27T23:09:20+00:00",1819642,"18d9f3602cb03bfaee7a6f4845db4f90"],
       "106": ["2yX_EGgu8ZxYExf.V.c_kc.jf4ijT8An","2022-06-27T23:09:33+00:00",1837884,"3006f1300734e195b505692e9c301235"],

for the full one so we have everything desired e.g. for webdav to provide listing and access, and compute/validate the zarrChecksum. And the .versionid.json one looks like

yoh@typhon:~/proj/dandi/zarr-manifests$ head -n 25 zarr-manifests-v2-sorted/128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d/6ddc4625befef8d6f9796835648162be-509--710206390.versionid.json 
{
 "schemaVersion": 2,
 "fields": "versionId",
 "statistics": {
  "entries": 509,
  "depth": 5,
  "totalSize": 710206390,
  "lastModified": "2022-06-27T23:09:39+00:00",
  "zarrChecksum": "6ddc4625befef8d6f9796835648162be-509--710206390"
 },
 "entries": {
  ".zattrs": "VwOSu7IVLAQcQHcqOesmlrEDm2sL_Tfs",
  ".zgroup": "7obAY5BUNOdI1Uch3RoI4oHuGXhW4h0R",
  ".zmetadata": "Vfe0W0v4zkydmzyXkUMjm2Xr7.rIvfZQ",
  "0": {
   ".zarray": "Ou6TnKwWPmEJrL.0utCWLPxgfr_lA0I1",
   "0": {
    "0": {
     "13": {
      "8": {
       "100": "lqNZ6OQ6lKd2QRW8ekWOiVfdZhiicWsh",
       "101": "_i9cZBerb4mB9D8IFbPHo8nrefWcbq0p",
       "103": "DeraqBNPhVssSggCCTPjmgbu6XxOOupZ",
       "104": "J6ejQa4K9cEAjNbkKkZip8.zdK5XFI37",
       "106": "2yX_EGgu8ZxYExf.V.c_kc.jf4ijT8An",

which results in over twice smaller file but capable only to provide access to current version of zarr.