create_metadata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Notebook for storing metadata related to cell types, conditions, and colors to keep them consistent throughout the notebooks. This notebook was updated many times throughout the analysis, and so if starting from raw input files, it might not run immediately, but all the files imported here are eventually created in other notebooks. To reproduce results, I would recommend using the `meta.pkl` provided on GEO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle as pkl\n",
    "import scanpy as sc\n",
    "import matplotlib as mpl\n",
    "import itertools as it\n",
    "import os\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "mountpoint = '/data/clue_test/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Path to the most recent concat object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_concat = mountpoint + 'prod/comb/h5ads/concat_6.h5ad'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Path to eQTL results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_eqtl = mountpoint + 'prod/eqtl/mateqtl/vals/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Metadata Dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I use a `pkl` called `meta.pkl` in various notebooks throughout the processing. Here is where I create it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta = dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cts`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "concat = sc.read_h5ad(path_concat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct_types = [i for i in concat.obs.columns if i.startswith('ct')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta['cts'] = dict()\n",
    "for ct_type in ct_types:\n",
    "    meta['cts'][ct_type] = concat.obs[ct_type].cat.categories.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DESeq2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta['cts']['de_set1'] = ['B_Naive', 'pDC', 'T4_Naive', 'HSC', 'T4_EM', 'NK', 'T_Tox', 'B_Mem', 'M_cDC', 'T8_Naive']\n",
    "meta['cts']['de_set2'] = ['ncM', 'cM', 'cDC'] # for all conditions excpet PMA/I"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `ct` colors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct_colors = dict()\n",
    "for ct_type in ct_types:\n",
    "    ct_colors[ct_type] = dict()\n",
    "    ct_colors[ct_type]['hex'] = dict(zip(concat.obs[ct_type].cat.categories, concat.uns[ct_type + '_colors']))\n",
    "    ct_colors[ct_type]['rgb'] = dict(zip(concat.obs[ct_type].cat.categories, list(map(mpl.colors.hex2color, concat.uns[ct_type + '_colors']))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta['ct_colors'] = ct_colors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `conds`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta['conds'] = dict()\n",
    "meta['conds']['conds_all'] = ['0', 'A', 'B', 'C', 'G', 'P' ,'R']\n",
    "meta['conds']['conds_filt'] = ['A', 'B', 'C', 'G', 'P' ,'R']\n",
    "meta['conds']['conds_focus'] = ['A', 'B', 'C', 'G' ,'R']\n",
    "meta['conds']['conds_stims'] = ['A', 'B', 'G' ,'R', 'P']\n",
    "meta['conds']['stims_focus'] = ['A', 'B', 'G' ,'R']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cond_colors`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "cond_colors = dict()\n",
    "conds_all = meta['conds']['conds_all'] # \n",
    "cond_colors['hex'] = dict(zip(conds_all, sc.pl.palettes.default_20))\n",
    "cond_colors['rgb'] = dict(zip(conds_all, list(map(mpl.colors.hex2color, sc.pl.palettes.default_20))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta['cond_colors'] = cond_colors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `eQTL`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_files = [fn for fn in os.listdir(path_eqtl) if fn.startswith('mateqtl_')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "cond_cts = list()\n",
    "for output_file in output_files:\n",
    "    cond_cts.append(tuple(output_file.replace('mateqtl_', '').replace('_all_cis.csv', '').split('_', maxsplit=1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta['eqtl'] = dict()\n",
    "meta['eqtl']['cond_cts'] = pd.DataFrame(cond_cts, columns=['cond', 'ct'], dtype='category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ATAC validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `${mountpoint}/amo/atac/peak_calling.ipynb`, I select a few cell types and conditions on which to run the ATAC-eQTL enrichment based on cell numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "atac_cond_cts = it.product(['C', 'G', 'B'], ['T4_Naive', 'T8_Naive', 'T4_EM', 'T8_Mem', 'NK_CD16+', 'cM', 'B_Naive', 'B_Mem'])\n",
    "meta['eqtl']['cond_cts_atac'] = pd.DataFrame(atac_cond_cts, columns=['cond', 'ct'], dtype='category')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with open(mountpoint + 'meta.pkl', 'wb') as file:\n",
    "#     pkl.dump(meta, file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(mountpoint + 'meta.pkl', 'rb') as file:\n",
    "    meta = pkl.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "clue_test",
   "language": "python",
   "name": "clue_test"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}