Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Table spec proposal #64

Closed
wants to merge 35 commits into from
Closed

Table spec proposal #64

Changes from 3 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
4559ce5
add tables spec draft
kevinyamauchi Oct 1, 2021
c9148a7
X must be a single dtype
kevinyamauchi Oct 1, 2021
d3330a2
add points spec draft
kevinyamauchi Oct 1, 2021
957555f
add spec for axis names
kevinyamauchi Oct 20, 2021
f8f2fd0
remove points spec
kevinyamauchi Jun 19, 2022
83c9491
Merge branch 'main' into add-tables-points
kevinyamauchi Jun 19, 2022
8d2d64e
add layers
kevinyamauchi Jun 19, 2022
d715323
add varm/obsm
kevinyamauchi Jun 19, 2022
98be5fa
add obsp/varp
kevinyamauchi Jun 19, 2022
bc72166
add missing .zarray
kevinyamauchi Jun 19, 2022
f4a13c6
add @type
kevinyamauchi Jun 19, 2022
ffdd046
Apply suggestions from code review
kevinyamauchi Jun 20, 2022
0d5636e
update uns
kevinyamauchi Jun 20, 2022
1d59ca4
add parent
kevinyamauchi Jun 20, 2022
d91da64
MAY -> MUST for anndata encoding
kevinyamauchi Jun 20, 2022
4ad6a90
indices, indptr dtype
kevinyamauchi Jun 20, 2022
fac5855
csc matrix
kevinyamauchi Jun 20, 2022
e3f779c
update table metadata
kevinyamauchi Jun 20, 2022
75fb5e1
Update index.bs
kevinyamauchi Jun 20, 2022
00e85c1
Update latest/index.bs
kevinyamauchi Jun 20, 2022
5d0b5b4
Update latest/index.bs
kevinyamauchi Jun 20, 2022
fe2cfad
update var table
kevinyamauchi Jun 29, 2022
692464f
update path spec
kevinyamauchi Oct 8, 2022
cc83a82
typo
kevinyamauchi Oct 8, 2022
b2de201
Apply suggestions from @ivirshup
kevinyamauchi Oct 28, 2022
283d21b
clarify allowed locations
kevinyamauchi Oct 28, 2022
bf81797
remove intensity image metadata
kevinyamauchi Oct 28, 2022
4b46882
remove reference to image id
kevinyamauchi Nov 15, 2022
f3e960b
add tables metadata
kevinyamauchi Jan 8, 2023
ad48296
add list of keys for subgroups
kevinyamauchi Jan 8, 2023
55fb1a8
add tables
giovp Feb 22, 2023
337971b
rename table
giovp Feb 22, 2023
e9f707b
add my_table consistently
giovp Feb 22, 2023
22844a4
remove comment
giovp Feb 23, 2023
ea8a622
Merge pull request #1 from giovp/add-tables-points
kevinyamauchi Feb 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions latest/index.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,108 @@ multiple levels of resolutions and optionally associated labels.
│ ... # are supported.
└── n
```
Tables {#table-layout}
----------------------
The following describes the expected layout for tabular data. OME-NGFF tables are a compatible subset of the [AnnData model](https://github.com/theislab/anndata). OME-NGFF tables implement the X (central dense matrix), the obs table (annotations on the rows of X) and the var table (annotations on the columns of X).
kevinyamauchi marked this conversation as resolved.
Show resolved Hide resolved

```
. # Root folder, potentially in S3,
│ # with a flat list of images by image ID.
will-moore marked this conversation as resolved.
Show resolved Hide resolved
│
└── 123.zarr
└── table # The table group is a container which holds a table that is compatible with a subset of AnnData.
kevinyamauchi marked this conversation as resolved.
Show resolved Hide resolved
│ # The table group MAY be in the root of the zarr file.
├── .zgroup # The table group MAY be in an image, labels, or points group.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is really cool to see, and it'd be really important to have this flexibility of having tables both in root and in image/labels group.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
├── .zgroup # The table group MAY be in an image, labels, or points group.
├── .zgroup # The table group MAY be in an image, labels, or points group.

Points haven't been defined yet.

Also, I'm not completley sure what this means given the bioformats2raw layout and the fact we can now put the tables group on its own. Maybe we should remove this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I removed the references to points and changed it to say that the tables group may be in root or another group.

├── .zattrs
│
├── X # You MAY add an zarr array `X`.
│ │ # `X` MUST not be a complex type (i.e., MUST be a single type)
│ │ # `X` MAY be chunked as the user desires.
│ ├── .zarray
│ ├── 0.0
│ │ ...
│ └── n.m
│
├── obs # You MAY add an obs group container. The obs group holds a table of annotations on the rows in X.
│ │ # The rows in obs MUST be index-matched to the rows in X.
│ ├── .zgroup
│ │
│ ├── .zattrs # `.zattrs` MUST contain `"_index"`, which is the name of the column in obs to be used as the index.
│ │ # `.zattrs` MUST contain `"column-order"`, which is a list of the order of the non-_index columns.
│ │ # `.zattrs` MAY contain `"encoding-type"`, which is set to `"dataframe"` by AnnData.
│ │ # `.zattrs` MAY contain `"encoding-version"`, which is set to `"0.1.0"` by AnnData.
│ │
│ └── col_0 # Each column in the obs table is a 1D zarr array. The rows can be chunked as the user desires.
kevinyamauchi marked this conversation as resolved.
Show resolved Hide resolved
│ ├── .zarray # However, the obs columns SHOULD be chunked in the same way as the rows in X (if present).
│ │
│ └─ 0
└── var # You MAY add a var group container. The var group holds a table of annotations on the columns in X.
│ # The rows in var MUST be index-matched to the columns in X (if present).
|
├── .zattrs # `.zattrs` MUST contain `"_index"`, which is the name of the column in obs to be used as the index.
│ # `.zattrs` MUST contain `"column-order"`, which is a list of the order of the non-_index columns.
│ # `.zattrs` MAY contain `"encoding-type"`, which is set to `"dataframe"` by AnnData.
│ # `.zattrs` MAY contain `"encoding-version"`, which is set to `"0.1.0"` by AnnData.
│
└── col_0 # Each column in the var table is a 1D zarr array. The rows can be chunked as the user desires.
├── .zarray # However, the var columns SHOULD be chunked in the same way as the columns in X.
│
└─ 0
```

Points {#points-layout}
-----------------------
The following describes the expected layout for point annotations in OME-NGFF. Point annotations are tables with some specified columns and layout.

```
. # Root folder, potentially in S3,
│ # with a flat list of images by image ID.
│
└── 123.zarr
└── points # The points group is a container which holds a table that is compatible with a subset of AnnData.
│ # The points group MAY be in the root of the zarr file.
├── .zgroup # The table group MAY be in an image or labels group.
├── .zattrs
│
├── X # You MUST add an zarr array `X`.
│ │ # `X` MUST contain the coordinates for the points.
│ │ # `X` MAY contain data in addition to the coordinates.
│ │ # `X` MAY be chunked as the user desires.
│ ├── .zarray
│ ├── 0.0
│ │ ...
│ └── n.m
│
├── obs # You MAY add an obs group container. The obs group holds a table of annotations for the points in X.
│ │ # The rows in obs MUST be index-matched to the rows in X.
│ │
│ ├── .zgroup
│ │
│ ├── .zattrs # `.zattrs` MUST contain `"_index"`, which is the name of the column in obs to be used as the index.
│ │ # `.zattrs` MUST contain `"column-order"`, which is a list of the order of the non-_index columns.
│ │ # `.zattrs` MAY contain `"encoding-type"`, which is set to `"dataframe"` by AnnData.
│ │ # `.zattrs` MAY contain `"encoding-version"`, which is set to `"0.1.0"` by AnnData.
│ │
│ └── col_0 # Each column in the obs table is a 1D zarr array. The rows can be chunked as the user desires.
│ ├── .zarray # However, the obs columns SHOULD be chunked in the same way as the rows in X.
│ │
│ └─ 0
└── var # You MUST add a var group container. The var group holds a table of annotations on the columns in X.
| # The var group MUST contain a column called `column_name` that annotates what each column in X contains.
kevinyamauchi marked this conversation as resolved.
Show resolved Hide resolved
| # The `column_name` column must annotate the columns in X containing coordinates with 0_p, 1_p, ..., n_p.
│ # The rows in var MUST be index-matched to the columns in X.
|
├── .zattrs # `.zattrs` MUST contain `"_index"`, which is the name of the column in obs to be used as the index.
│ # `.zattrs` MUST contain `"column-order"`, which is a list of the order of the non-_index columns.
│ # `.zattrs` MAY contain `"encoding-type"`, which is set to `"dataframe"` by AnnData.
│ # `.zattrs` MAY contain `"encoding-version"`, which is set to `"0.1.0"` by AnnData.
│
└── col_0 # Each column in the var table is a 1D zarr array. The rows can be chunked as the user desires.
├── .zarray # However, the var columns SHOULD be chunked in the same way as the columns in X.
│
└─ 0
```


High-content screening {#hcs-layout}
------------------------------------
Expand Down