Merge branch 'main' into issue62

cbyrohl · Aug 12, 2023 · 0d9f78c · 0d9f78c
2 parents 9eb0f1d + 9e148e0
commit 0d9f78c
Show file tree

Hide file tree

Showing 27 changed files with 2,171 additions and 1,276 deletions.
diff --git a/.github/actions/get-testdata-all/action.yml b/.github/actions/get-testdata-all/action.yml
@@ -47,3 +47,29 @@ runs:
         outputfolder: ${{ inputs.outputfolder }}
         filename: minimal_TNG-Cluster_snapshot_z0.hdf5
         url: https://heibox.uni-heidelberg.de/f/fe582fbe745445d7a448/?dl=1
+
+    - uses: ./.github/actions/get-testdata
+      with:
+        outputfolder: ${{ inputs.outputfolder }}
+        filename: minimal_LGal21_z3.hdf5
+        url: https://heibox.uni-heidelberg.de/f/6a24e9db7b8a449d94be/?dl=1
+    - uses: ./.github/actions/get-testdata
+      with:
+        outputfolder: ${{ inputs.outputfolder }}
+        filename: minimal_TNG100-3_snapshot_z0.hdf5
+        url: https://heibox.uni-heidelberg.de/f/6eb403619c534e52bb82/?dl=1
+    - uses: ./.github/actions/get-testdata
+      with:
+        outputfolder: ${{ inputs.outputfolder }}
+        filename: minimal_TNG100-3_group_z0.hdf5
+        url: https://heibox.uni-heidelberg.de/f/699142c6855341d1bb34/?dl=1
+    - uses: ./.github/actions/get-testdata
+      with:
+        outputfolder: ${{ inputs.outputfolder }}
+        filename: minimal_TNG100-2_snapshot_z0.hdf5
+        url: https://heibox.uni-heidelberg.de/f/e3c9c3f591574b969557/?dl=1
+    - uses: ./.github/actions/get-testdata
+      with:
+        outputfolder: ${{ inputs.outputfolder }}
+        filename: minimal_TNG100-2_group_z0.hdf5
+        url: https://heibox.uni-heidelberg.de/f/ed8a1b34a9fe40b1b493/?dl=1
diff --git a/README.md b/README.md
@@ -39,6 +39,13 @@ _scida_ is free and open source software.
 If you encounter any problems,
 please [file an issue](https://github.com/cbyrohl/scida/issues/new) along with a detailed description.
 
+## Contributors
+In alphabetical order:
+
+- @ayromlou
+- @cbyrohl
+- @dnelson86
+
 ## Acknowledgements
 
 The project structure was adapted from [Wolt](https://github.com/woltapp/wolt-python-package-cookiecutter) and [Hypermodern Python](https://github.com/cjolowicz/cookiecutter-hypermodern-python) cookiecutter templates.
diff --git a/docs/customs/lgalaxies.md b/docs/customs/lgalaxies.md
@@ -0,0 +1,10 @@
+# LGalaxies
+
+Access via individual datasets are supported, e.g.:
+
+```pycon
+>>> from scida import load
+>>> load("LGal_Ayromlou2021_snap58.hdf5")
+```
+
+while access to the series at once (i.e. loading all data for all snapshots in a folder) is **not supported**.
diff --git a/docs/halocatalogs.md b/docs/halocatalogs.md
@@ -14,6 +14,7 @@ ds = load("TNG50-4_snapshot") # (1)!
 
 The dataset itself passed to load does not possess information about the FoF/Subfind outputs as they are commonly saved in a separate folder or hdf5 file. For typical folder structures of GADGET/AREPO style simulations, an attempt is made to automatically discover and add such information. The path to the catalog can otherwise explicitly be passed to *load()* via the *catalog=...* keyword.
 
+## Accessing halo/galaxy catalog information
 
 Groups and subhalo information is added into the dataset with the data containers *Group* and *Subhalo*. For example, we can obtain the masses of each group as:
 
@@ -22,16 +23,39 @@ Groups and subhalo information is added into the dataset with the data container
 group_mass = ds.data["Group"]["GroupMass"]
 ```
 
+## Accessing particle-level halo/galaxy information
+
 In addition to these two data containers, new information is added to all other containers about their belonging to a given group and subhalo.
 
 ``` py
-groupid = ds.data["PartType0"]["GroupID"]
+groupid = ds.data["PartType0"]["GroupID"] #(1)!
 subhaloid = ds.data["PartType0"]["SubhaloID"]
+localsubhaloid = ds.data["PartType0"]["LocalSubhaloID"]
 ```
 
-In above example, we fetch the virtual dask arrays holding the group and subhalo belonging of each *PartType0* particle. The *groupid* field returns the integer position into the group catalog that given particle belongs to. For the *subhaloid* the subhalo id within a given *groupid* is given, *where subhaloid==0* marks the belonging to the central subhalo in a halo.
+1. This information is also available for the other particle types.
+
+In above example, we fetch the virtual dask arrays holding information about the halo and subhalo association for each particle.
+
+`GroupID`
+
+:   The group ID of the group the particle belongs to. This is the index into the group catalog.
+
+`SubhaloID`
+
+:   The subhalo ID of the subhalo the particle belongs to. This is the index into the subhalo catalog.
+
+`LocalSubhaloID`
 
-This operation allows us to efficiently query the belonging of given particles. So, for example, we can compute the group IDs of the particles 1000-1099 by running
+:  This is the Subhalo ID relative to the central subhalo of a given group. For the central subhalo, this is 0.
+   Satellites accordingly start at index 1.
+
+Particles that are not associated with a group or subhalo that are queried for such ID
+will return `ds.misc['unboundID']'`. This is currently set to 9223372036854775807, but might change to -1 in the future.
+
+
+This operation allows us to efficiently query the belonging of given particles.
+So, for example, we can compute the group IDs of the gas particles 1000-1099 by running
 
 ``` py
 groupid[1000:1100].compute()
@@ -49,11 +73,21 @@ data = ds.return_data(haloID=42)
 
 *data* will have the same structure as *ds.data* but restricted to particles of a given group.
 
-### Operations on particle data for all groups
+### Applying to all groups in parallel
 
 In many cases, we do not want the particle data of an individual group, but we want to calculate some reduced statistic from the bound particles of each group. For this, we provide the *grouped* functionality. In the following we give a range of examples of its use.
 
 
+???+ warning
+
+    Executing the following commands can be demanding on compute resources and memory.
+    Usually, one wants to restrict the groups to run on. You can either specify "nmax"
+    to limit the maximum halo id to evaluate up to. This is usually desired in any case
+    as halos are ordered (in descending order) by their mass. For more fine-grained control,
+    you can also pass a list of halo IDs to evaluate via the "idxlist" keyword.
+    These keywords should be passed to the "evaluate" call.
+
+
 #### Baryon mass
 Let's say we want to calculate the baryon mass for each halo from the particles.
 

diff --git a/docs/supported_data.md b/docs/supported_data.md
@@ -3,16 +3,17 @@
 The following table shows a selection of supported datasets. The table is not exhaustive, but should give an idea of the range of supported datasets.
 If you want to use a dataset that is not listed here, read on [here](dataset_structure.md) and consider opening an issue or contact us directly.
 
-| Name                                                  | Support                               | Description                                         |
-|-------------------------------------------------------|---------------------------------------|-----------------------------------------------------|
-| [TNG](https://www.tng-project.org/)                   | :material-check-all:                  | Cosmological galaxy formation *simulations*         |
-| [Illustris](https://www.illustris-project.org/)       | :material-check-all:                  | Cosmological galaxy formation *simulations*         |
-| [FIRE2](https://wetzel.ucdavis.edu/fire-simulations/) | :material-check-all:                  | Cosmological zoom-in galaxy formation *simulations* |
-| [EAGLE](https://icc.dur.ac.uk/Eagle/)                 | :material-check-all:                  | Cosmological galaxy formation *simulations*         |
-| [SIMBA](http://simba.roe.ac.uk/)                      | :material-check-all:                  | Cosmological galaxy formation *simulations*         |
-| TNG-Cluster                                           | :material-check-all:                  | Cosmological zoom-in galaxy formation *simulations* |
-| [AURIGA](https://wwwmpa.mpa-garching.mpg.de/auriga/)  | :material-check-all:                  | Cosmological zoom-in galaxy formation *simulations* |
-| [Gaia](https://www.cosmos.esa.int/web/gaia/dr3)       | :material-database-check-outline:[^1] | *Observations* of a billion nearby stars            |
+| Name                                                  | Support                               | Description                                                                                                     |
+|-------------------------------------------------------|---------------------------------------|-----------------------------------------------------------------------------------------------------------------|
+| [TNG](https://www.tng-project.org/)                   | :material-check-all:                  | Cosmological galaxy formation *simulations*                                                                     |
+| [Illustris](https://www.illustris-project.org/)       | :material-check-all:                  | Cosmological galaxy formation *simulations*                                                                     |
+| [FIRE2](https://wetzel.ucdavis.edu/fire-simulations/) | :material-check-all:                  | Cosmological zoom-in galaxy formation *simulations*                                                             |
+| [EAGLE](https://icc.dur.ac.uk/Eagle/)                 | :material-check-all:                  | Cosmological galaxy formation *simulations*                                                                     |
+| [SIMBA](http://simba.roe.ac.uk/)                      | :material-check-all:                  | Cosmological galaxy formation *simulations*                                                                     |
+| TNG-Cluster                                           | :material-check-all:                  | Cosmological zoom-in galaxy formation *simulations*                                                             |
+| [AURIGA](https://wwwmpa.mpa-garching.mpg.de/auriga/)  | :material-check-all:                  | Cosmological zoom-in galaxy formation *simulations*                                                             |
+| [Gaia](https://www.cosmos.esa.int/web/gaia/dr3)       | :material-database-check-outline:[^1] | *Observations* of a billion nearby stars                                                                        |
+| [LGalaxies](customs/lgalaxies.md)                     | :material-check-all:                  | Semi-analytical model for [Millenium](https://wwwmpa.mpa-garching.mpg.de/galform/virgo/millennium/) simulations |
 
 
 A :material-check-all: checkmark indicates support out-of-the-box, a :material-check: checkmark indicates support by creating a suitable configuration file.

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -26,6 +26,9 @@ nav:
     - 'Derived fields': derived_fields.md
     - 'Data series': series.md
     - 'Large datasets': largedatasets.md
+  - 'Advanced Topics':
+    - 'Arepo Simulations':
+      - 'Halo Catalogs': halocatalogs.md
   # - 'Cookbook':
   #   - 'Units': notebooks/cookbook/units.ipynb
   - 'FAQ': faq.md