Loom is
.loom
, an efficient file format for large omics datasetsloompy
, a Python library for creating and working with data in.loom
format.loom-viewer
, a web app for browsing large single-cell RNA-seq datasets
This repository is the loom-viewer part. Read more about .loom and loompy here.
You can choose between installation through PyPi, or build loom-viewer
from source.
- Install the Python 3.x version of Anaconda.
Note: loompy and loom-viewer only work with Python 3
- Open a terminal.
Windows users: open Programs > Anaconda 3 > Anaconda Promp
.
- Install Loom with the following command:
pip install loompy
pip install loom-viewer
Tip: Loom is under development and updated often. To ensure you have the latest version, do this:
pip install -U loom-viewer
- Install the Python 3.x version of Anaconda.
Note: loompy and loom-viewer only work with Python 3
- If you haven't already, install
loompy
, either from source or through PyPI:
pip install loompy
- Install depedencies:
- node.js
- npm ( ubuntu:
sudo apt-get install npm
) - webpack ( ubuntu:
sudo apt-get install webpack
)
- Clone this repository with git
git clone https://github.com/linnarsson-lab/loom-viewer.git
- Navigate to the newly created
loom-viewer
folder, and install all required node packages:
npm install
- Run the build script. On Mac/Linux:
./build
On Windows:
build
To build the production version, type:
./build prod
Tip: when developing, you only modified the Python code, and want to skip building the client side, type:
./build_egg
After installation, you should have access to the loom
CLI from the terminal ("Anaconda Prompt" on Windows). Test if everything works correctly by typing:
loom version
To learn more about the loom tool, type:
loom --help
For help with individual commands, just type:
loom [command] --help
To open the Loom Viewer locally, type in loom
without any extra flags:
loom
This will start a local server, and automatically open localhost:8003
in your default browser. Since this is the first time running the tool, it will fail to find any datasets, showing an empty list and an error message about fetch being broken on Safari (we are working on that one).
Close the server for now (CTRL+C on Linux/Windows, CMD+C on Mac).
To view our Loom files, we have to go through the following steps:
- Store the Loom files in a directory where the viewer can find it
a. (optional) To see heatmap tiles, generate them with
loom tile <loom filename>
b. (optional) To quickly access gene expression data, pre-expand the rows withloom expand -r <loom filename>
- Start the loom server, or if it is already running, refresh the page
The first time you run loom
, a loom-datasets
folder will be created in your home folder. This is where the loom-viewer
server looks for Loom files by default.
You can explicitly point the loom CLI to a different path with loom --dataset-path <your path here>
The dataset folder is organised in projects. These are sub-folders in the root dataset folder. Folders represent individual projects. The Loom files are stored in these project folders: (all other files are ignored)
~/
├─ loom-datasets/
. ├─ # Project 1 (folder)
│ ├─ # dataset1.loom (loom file)
│ └─ # dataset2.loom (loom file)
└─ # Project 2 (folder)
├─ # dataset3.loom (loom file)
└─ # dataset4.loom (loom file)
Either create a Loom file with loompy
, or download one from a loom viewer hosted on-line.
For this example, we will use the Cortex.loom
data set, containing data from one of our papers. First, go to Linnarsson Lab's loom.linnarssonlab.org, and search for Cortex.loom
:
If you want, you can open it now and browse it on-line!
On the right you can see a cloud icon that links to the original loom file. Right-click and save the file to an appropriately labeled project folder in the loom-datasets
folder:
~/
├─ loom-datasets/
. ├─ Linnarsson Lab
. └─ Cortex.loom
Once we start a local loom server (see below) and open localhost:8003
, the resulting view should be a single project, Linnarsson Lab
, with a single loom file, Cortex.loom
. Opening it should give the same results as the on-line versoin.
By default, the server generates and caches the metadata, attributes, and individual rows and columns as they are requested (see below).
However, to view a heat map of the entire data matrix, the associated image tiles must be pre-generated from the Loom file (the reason we do not do this on-the-fly is that generating tiles requires iterating over all the data in a Loom file. For bigger files is so slow that it would freeze the system if done while running a server).
Note: try to avoid running a server at the same time you generate tiles, as this can lead to unexpected HDF5 behaviour due to multiple programs accessing the same file at once.
Tile generation is done through the loom tile
command. For a list of possible flags, type loom tile --help
:
usage: loom tile [-h] [--project [PROJECT [PROJECT ...]]] [-A] [-t]
[file [file ...]]
positional arguments:
file Loom file(s) to expand. Expands all files matching the
provided file names. To avoid this, use an absolute
path to specify a single file.
optional arguments:
-h, --help show this help message and exit
--project [PROJECT [PROJECT ...]]
Project(s) for which to expand all files.
-A, --all Expand all loom files.
-t, --truncate Remove previously expanded tiles if present (False by
default)
Note that the loom tile
command will automatically search all projects for all matching file names, and generate tiles for each. Using unique file names is encouraged!
Examples:
# Generate tiles for all files named `Cortex.loom` or `Cortex_allgenes.loom
loom tile Cortex.loom Cortex_allgenes.loom
# Generate tiles for `/home/me/loom-datasets/Published/Cortex.loom`
# on Windows, this would be something like:
# loom tile C:\Users\Job van der Zwan\loom-datasets\Published\Cortex.loom
loom tile /home/me/loom-datasets/Published/Cortex.loom
# Generate tiles for all files in the `Published` project:
loom tile --project Published
Note: the tiles are not automatically updated when changing data in a loom file, meaning the viewer may show the old data! To fix this, re-do this step with the added -t flag:
# -t for "truncate", telling loom to overwrite the old files
loom tile -t [filenames and project folders]
Because generation can be slow for larger Loom files, the command checks if the relevant <filename>.loom.tile
subfolder exists, skipping generation if it does. This means that if the tile command is aborted without finishing, the remaining tiles will not be added when running loom tile
again! To override this, run: loom tile -t <filename.loom>
. Alternatively, delete the subfolder in question.
Similar to pre-generating image tiles, is possible to pre-generate all metadata, like attributes and gene rows, for a Loom file. This step used to be mandatory, but now the server generates the required JSOn on the fly.
There are two situations in which this is still relevant.
First, if you change data in a loom file, previously generated JSON files for that data becomes outdated. Currently the loom server cannot detect this change, so the JSON files have to manually be updated.
Second, because opening and closing an HDF5 connection gets slower as file-size increases, this may still be useful when serving very large loom files on-line.
For either of these scenarios we use the loom expand
command.
usage: loom expand [-h] [--project [PROJECT [PROJECT ...]]] [-A] [-C] [-t]
[-m] [-a] [-r] [-c]
[file [file ...]]
positional arguments:
file Loom file(s) to expand. Expands all files matching the
provided file names. To avoid this, use an absolute
path to specify a single file. When combined with
--clear it clears all expanded files instead.
optional arguments:
-h, --help show this help message and exit
--project [PROJECT [PROJECT ...]]
Project(s) for which to expand all files (or clear
expansion with --clear).
-A, --all Expand all loom files (or clear expansion with
--clear).
-C, --clear Remove previously expanded files.
-t, --truncate Replace previously expanded files if present (False by
default). Only does something in combination with
expansion (-m, -a, -r or -c).
-m, --metadata Expand metadata (False by default)
-a, --attributes Expand attributes (False by default)
-r, --rows Expand rows (False by default)
-c, --cols Expand columns (False by default)
It is nearly identical to the loom tile
command, except that we also need to specify whether we want to generate JSON files for metadata, attributes, rows and/or columns.
Examples:
# Generate JSON for metadata, attributes and rows
# for all files named `Cortex.loom` or `Cortex_allgenes.loom,
# and overwrite ay previous JSON files encountered
loom expand -mart Cortex.loom Cortex_allgenes.loom
# Generate attributes JSON files for /home/me/loom-datasets/Published/Cortex.loom
# on Windows, this would be something like:
# loom expand -a C:\Users\Job van der Zwan\loom-datasets\Published\Cortex.loom
loom expand -a /home/me/loom-datasets/Published/Cortex.loom
# Generate row JSON files for all files in the `Published` project:
loom expans -r --project Published
The loom server cannot find any loom files, but the website mistakenly thinks it failed to fetch the list of loom files (which (for now) is guaranteed to happen on Safari anyway, hence the warning message).
See above for where and how to store your loom files so that the server can find them.
If this did not solve your problem, open an issue.
If you have not done so yet, generate the tiles for the heatmap first with loom tile <filename.loom>
. See above for more details.
If you have done so but something went wrong, you can try again with loom tile -t <filename.loom>
.
It is likely that you are missing the required image library. This can happen if you use MiniConda, which has fewer default packages. Try installing it with conda install pillow
.
There are two reasons for this:
First, h5py can not safely cope with multiple people accessing the same Loom file simultanously. This is not a problem when exploring loom files locally (unless you open the same loom file in multiple tabs), but when using the loom-viewer
as a server to set up a website for sharing loom-files, two or more people opening the same loom files will cause the server to crash.
Second, loading the genes from the loom file and converting it to either JSON or a PNG is a fairly slow process. The viewer is mainly intended to explore and share finished loom files that do not change, making it wasteful to do this work multiple times. Serving pre-generated static files is thousands of times faster.
This limits the usefulness as an off-line viewer somewhat for now, but we have plans to make the server capable of detecting when you changed something in a loom file, and automatically update the data in question. This would make this process easier, but for now you have to manually trigger an update to the generated JSON files and image tiles when changing data in the a loom file.