nuclei_prediction/upload2cave.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Upload soma table to CAVE\n",
    "To make an efficient and user-friendly table structure, we decided to make 1. a soma table with all the information, and 2. cell-specific tables (i.e., neuron and glia) that have the same information by referencing the soma table. With this approach, the CAVE server only needs to update the information in `somas_dec2022`, but users can just run `client.materialize.query_table('neuron_somas_dec2022')` or `client.materialize.query_table(client.info.get_datastack_info()['soma_table'])` to get updated `pt_root_id` of neurons.\n",
    "![How soma table is organized on CAVE](./img/soma_table_CAVE.png)\n",
    "\n",
    "## 0. Prepare CAVE environment\n",
    "You need to install CAVEclient and initialize it first. If this is your first time to use CAVE, check [here](https://globalv1.daf-apis.com/info/) about which dataset you have access. You should also take a look at [CAVE's official document](https://caveclient.readthedocs.io/en/latest/index.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "import pandas as pd\n",
    "from tqdm import tqdm\n",
    "from caveclient import CAVEclient\n",
    "from fanc.upload import xyz_StringSeries2List\n",
    "from fanc import ngl_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "datastack_name = 'fanc_production_mar2021'\n",
    "client = CAVEclient(datastack_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Create (empty) tables\n",
    "Uploading data to CAVE usually takes two steps: 1. make an empty table, and 2. upload annotations to the table you have just created. Here, we first made three tables: `somas_dec2022` stores all the coordinates of neuronal and glial nuclei, `neuron_somas_dec2022` stores which ids belong to neurons, and `glia_somas_dec2022` stores which ids belong to glias."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false\n",
    "\n",
    "client.annotation.create_table(table_name='somas_dec2022',\n",
    "                               schema_name='nucleus_detection',\n",
    "                               description = 'Information about all the somas identified by the nucleus detection pipeline. We segmented nuclei using 3D U-net (DeepEM) and excluded objects smaller than the size dimension of (1,376 nm, 1,376 nm, 1,800 nm) for (x, y, z). id : a 17-digit nucleus id, pt_position (x,y,z) : a center coordinate, volume (um^3) : a volume, bb (x,y,z) : a bounding box. [created/uploaded/managed by Sumiya Kuroda - s.kuroda@ucl.ac.uk]',\n",
    "                               flat_segmentation_source=ngl_info.nuclei[\"path\"],\n",
    "                               write_permission=\"GROUP\",\n",
    "                               voxel_resolution =[4.3, 4.3, 45.0])\n",
    "client.annotation.create_table(table_name='neuron_somas_dec2022',\n",
    "                               schema_name='simple_reference',\n",
    "                               description = 'This table points to neuronal somas in somas_dec2022. target_id: a nucleus id of a neuron. [created/uploaded/managed by Sumiya Kuroda - s.kuroda@ucl.ac.uk]',\n",
    "                               reference_table = 'somas_dec2022',\n",
    "                               track_target_id_updates=True,\n",
    "                               flat_segmentation_source=ngl_info.nuclei[\"path\"],\n",
    "                               write_permission=\"GROUP\",\n",
    "                               voxel_resolution =[4.3, 4.3, 45.0])\n",
    "client.annotation.create_table(table_name='glia_somas_dec2022',\n",
    "                               schema_name='simple_reference',\n",
    "                               description = 'This table points to glial somas in somas_dec2022. target_id: a nucleus id of a glial cell. [created/uploaded/managed by Sumiya Kuroda - s.kuroda@ucl.ac.uk]',\n",
    "                               reference_table = 'somas_dec2022',\n",
    "                               track_target_id_updates=True,\n",
    "                               flat_segmentation_source=ngl_info.nuclei[\"path\"],\n",
    "                               write_permission=\"GROUP\",\n",
    "                               voxel_resolution =[4.3, 4.3, 45.0])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Load data and format tables\n",
    "We also have to format the tables before uploading to CAVE. You can check what columns are required by CAVE, by running `client.schema.schema_definition('nucleus_detection')` or [here](https://globalv1.daf-apis.com/schema/views/type/nucleus_detection/view)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"../Output/soma_info_Aug2021ver5_1_new_column_of_center_for_CAVE.csv\", header=0) # original detection result\n",
    "newID_neuron = pd.read_csv(\"../Output/nucID_mapping_neuron_20221207.csv\", header=0) # new nucleus ids for each label\n",
    "newID_glia = pd.read_csv(\"../Output/nucID_mapping_glia_20221219.csv\", header=0)\n",
    "newID_fp = pd.read_csv(\"../Output/nucID_mapping_fp_20221219.csv\", header=0)\n",
    "newID_dup_neuron = pd.read_csv(\"../Output/nucID_mapping_dup_neuron.csv\", header=0)\n",
    "newID_dup_glia = pd.read_csv(\"../Output/nucID_mapping_dup_glia_20221219.csv\", header=0)\n",
    "newIDs = [newID_neuron, newID_glia, newID_fp, newID_dup_neuron, newID_dup_glia]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def which_label(id, label, dfs):\n",
    "    label_list = list(label.values())\n",
    "    for i in range(len(label)):\n",
    "        if sum(np.isin(dfs[i][\"old_nucID\"], np.array(id, dtype=np.uint64))) == 1:\n",
    "            return label_list[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 17076/17076 [01:08<00:00, 248.79it/s]\n"
     ]
    }
   ],
   "source": [
    "df[\"new_nucID\"] = 0\n",
    "df[\"vol_um\"] = 0\n",
    "df[\"label\"] = \"\"\n",
    "voxel2um = (0.0043*2**4)*(0.0043*2**4)*0.045 \n",
    "\n",
    "newID_all = pd.concat(newIDs)\n",
    "label = {'Neurons': \"neurons\", 'Glia': \"glias\", 'False positive': \"false_positives\", 'Duplicated neuron': \"dup_neurons\", 'Duplicated glia': \"dup_glias\"}\n",
    "\n",
    "for i in tqdm(range(len(df))):\n",
    "    # update nucleus ids\n",
    "    nucID = df.iloc[i][\"nucID\"]\n",
    "    idx = np.isin(newID_all[\"old_nucID\"], np.array(nucID, dtype=np.uint64))\n",
    "    df.at[i, \"new_nucID\"] = newID_all[idx][\"new_nucID\"].values[0]\n",
    "\n",
    "    # calculate volume in um^3\n",
    "    voxel_size = df.iloc[i][\"voxel_size\"]\n",
    "    df.at[i, \"vol_um\"] = voxel_size * voxel2um\n",
    "\n",
    "    # label whether neuron or glia or false positive or duplicated neuron\n",
    "    df.at[i, \"label\"] = which_label(nucID, label, newIDs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14628\n",
      "2019\n",
      "412\n",
      "15\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "print(len(df[df[\"label\"]==list(label.values())[0]]))\n",
    "print(len(df[df[\"label\"]==list(label.values())[1]]))\n",
    "print(len(df[df[\"label\"]==list(label.values())[2]]))\n",
    "print(len(df[df[\"label\"]==list(label.values())[3]]))\n",
    "print(len(df[df[\"label\"]==list(label.values())[4]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "nucleus_df = df.reindex(columns=['new_nucID', 'center_for_CAVE', 'vol_um', 'bbx_min', 'bbx_max', 'label'])\n",
    "nucleus_df = nucleus_df.rename(columns={'new_nucID': 'id', 'center_for_CAVE': 'pt_position', 'vol_um': 'volume', 'bbx_min': 'bb_start_position', 'bbx_max': 'bb_end_position'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "nucleus_df['pt_position'] = xyz_StringSeries2List(nucleus_df['pt_position'])\n",
    "nucleus_df['bb_start_position'] = xyz_StringSeries2List(nucleus_df['bb_start_position'])\n",
    "nucleus_df['bb_end_position'] = xyz_StringSeries2List(nucleus_df['bb_end_position'])\n",
    "nucleus_df_s = nucleus_df.sort_values('volume', ascending=True).reset_index(drop=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Upload annotations\n",
    "After formatting the tables, we uploaded them into the empty tables we just created. You can directly upload them, but `stage_annotations` is a more safe option because it will warn you if your table is incorrectly formatted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we first uploaded the soma table\n",
    "tmp = nucleus_df_s[nucleus_df_s['label'].isin([\"neurons\", \"glias\"])].reset_index(drop=True)\n",
    "df_upload = tmp.reindex(columns=['id', 'pt_position', 'volume', 'bb_start_position', 'bb_end_position'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>pt_position</th>\n",
       "      <th>volume</th>\n",
       "      <th>bb_start_position</th>\n",
       "      <th>bb_end_position</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>72481181429400700</td>\n",
       "      <td>[25528, 84220, 2199]</td>\n",
       "      <td>1.589229</td>\n",
       "      <td>[25136, 84000, 2167]</td>\n",
       "      <td>[25920, 84544, 2210]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>72481250081768945</td>\n",
       "      <td>[27260, 86824, 2002]</td>\n",
       "      <td>2.807190</td>\n",
       "      <td>[27008, 86544, 1982]</td>\n",
       "      <td>[27408, 87104, 2023]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72481319002571550</td>\n",
       "      <td>[28028, 92796, 2662]</td>\n",
       "      <td>3.072168</td>\n",
       "      <td>[27696, 92448, 2634]</td>\n",
       "      <td>[28512, 93248, 2700]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>72623705288605712</td>\n",
       "      <td>[35088, 188960, 791]</td>\n",
       "      <td>3.302213</td>\n",
       "      <td>[34832, 188656, 771]</td>\n",
       "      <td>[35344, 189264, 812]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>72481181362296023</td>\n",
       "      <td>[26584, 83696, 2034]</td>\n",
       "      <td>3.782539</td>\n",
       "      <td>[26176, 83344, 2013]</td>\n",
       "      <td>[26992, 84048, 2060]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16642</th>\n",
       "      <td>72412118355280021</td>\n",
       "      <td>[23512, 160472, 2125]</td>\n",
       "      <td>401.849957</td>\n",
       "      <td>[22016, 159408, 2033]</td>\n",
       "      <td>[25008, 161536, 2217]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16643</th>\n",
       "      <td>72623018630709815</td>\n",
       "      <td>[34288, 149640, 3050]</td>\n",
       "      <td>412.453123</td>\n",
       "      <td>[33072, 148528, 2949]</td>\n",
       "      <td>[35504, 150752, 3151]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16644</th>\n",
       "      <td>73044681272852922</td>\n",
       "      <td>[60672, 116472, 2665]</td>\n",
       "      <td>431.741133</td>\n",
       "      <td>[59408, 115472, 2550]</td>\n",
       "      <td>[61936, 117472, 2781]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16645</th>\n",
       "      <td>72833712814817408</td>\n",
       "      <td>[47520, 124320, 3907]</td>\n",
       "      <td>484.838331</td>\n",
       "      <td>[46112, 123248, 3791]</td>\n",
       "      <td>[48928, 125392, 4023]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16646</th>\n",
       "      <td>73115256175462047</td>\n",
       "      <td>[63544, 126192, 2665]</td>\n",
       "      <td>568.930070</td>\n",
       "      <td>[62192, 124832, 2538]</td>\n",
       "      <td>[64896, 127552, 2792]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>16647 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      id            pt_position      volume  \\\n",
       "0      72481181429400700   [25528, 84220, 2199]    1.589229   \n",
       "1      72481250081768945   [27260, 86824, 2002]    2.807190   \n",
       "2      72481319002571550   [28028, 92796, 2662]    3.072168   \n",
       "3      72623705288605712   [35088, 188960, 791]    3.302213   \n",
       "4      72481181362296023   [26584, 83696, 2034]    3.782539   \n",
       "...                  ...                    ...         ...   \n",
       "16642  72412118355280021  [23512, 160472, 2125]  401.849957   \n",
       "16643  72623018630709815  [34288, 149640, 3050]  412.453123   \n",
       "16644  73044681272852922  [60672, 116472, 2665]  431.741133   \n",
       "16645  72833712814817408  [47520, 124320, 3907]  484.838331   \n",
       "16646  73115256175462047  [63544, 126192, 2665]  568.930070   \n",
       "\n",
       "           bb_start_position        bb_end_position  \n",
       "0       [25136, 84000, 2167]   [25920, 84544, 2210]  \n",
       "1       [27008, 86544, 1982]   [27408, 87104, 2023]  \n",
       "2       [27696, 92448, 2634]   [28512, 93248, 2700]  \n",
       "3       [34832, 188656, 771]   [35344, 189264, 812]  \n",
       "4       [26176, 83344, 2013]   [26992, 84048, 2060]  \n",
       "...                      ...                    ...  \n",
       "16642  [22016, 159408, 2033]  [25008, 161536, 2217]  \n",
       "16643  [33072, 148528, 2949]  [35504, 150752, 3151]  \n",
       "16644  [59408, 115472, 2550]  [61936, 117472, 2781]  \n",
       "16645  [46112, 123248, 3791]  [48928, 125392, 4023]  \n",
       "16646  [62192, 124832, 2538]  [64896, 127552, 2792]  \n",
       "\n",
       "[16647 rows x 5 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_upload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false\n",
    "\n",
    "k=10000 # CAVE only accepts up to 10,000 annotations\n",
    "minidfs = [df_upload.loc[i:i+k-1, :] for i in range(0, len(df_upload), k)]\n",
    "for dftmp in minidfs:\n",
    "    stage = client.annotation.stage_annotations(\"somas_dec2022\", id_field=True)\n",
    "    stage.add_dataframe(dftmp)\n",
    "    client.annotation.upload_staged_annotations(stage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# neuron table\n",
    "tmp = nucleus_df_s[nucleus_df_s['label'].isin([\"neurons\"])].reset_index(drop=True)\n",
    "df_upload = tmp.reindex(columns=['id'])\n",
    "df_upload['target_id'] = df_upload['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>target_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>72481181429400700</td>\n",
       "      <td>72481181429400700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>72481250081768945</td>\n",
       "      <td>72481250081768945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72481319002571550</td>\n",
       "      <td>72481319002571550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>72623705288605712</td>\n",
       "      <td>72623705288605712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>72481181362296023</td>\n",
       "      <td>72481181362296023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14623</th>\n",
       "      <td>72412118355280021</td>\n",
       "      <td>72412118355280021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14624</th>\n",
       "      <td>72623018630709815</td>\n",
       "      <td>72623018630709815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14625</th>\n",
       "      <td>73044681272852922</td>\n",
       "      <td>73044681272852922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14626</th>\n",
       "      <td>72833712814817408</td>\n",
       "      <td>72833712814817408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14627</th>\n",
       "      <td>73115256175462047</td>\n",
       "      <td>73115256175462047</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14628 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      id          target_id\n",
       "0      72481181429400700  72481181429400700\n",
       "1      72481250081768945  72481250081768945\n",
       "2      72481319002571550  72481319002571550\n",
       "3      72623705288605712  72623705288605712\n",
       "4      72481181362296023  72481181362296023\n",
       "...                  ...                ...\n",
       "14623  72412118355280021  72412118355280021\n",
       "14624  72623018630709815  72623018630709815\n",
       "14625  73044681272852922  73044681272852922\n",
       "14626  72833712814817408  72833712814817408\n",
       "14627  73115256175462047  73115256175462047\n",
       "\n",
       "[14628 rows x 2 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_upload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false\n",
    "\n",
    "k=10000\n",
    "minidfs = [df_upload.loc[i:i+k-1, :] for i in range(0, len(df_upload), k)]\n",
    "for dftmp in minidfs:\n",
    "    stage = client.annotation.stage_annotations(\"neuron_somas_dec2022\", id_field=True)\n",
    "    stage.add_dataframe(dftmp)\n",
    "    client.annotation.upload_staged_annotations(stage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# glia table\n",
    "tmp = nucleus_df_s[nucleus_df_s['label'].isin([\"glias\"])].reset_index(drop=True)\n",
    "df_upload = tmp.reindex(columns=['id'])\n",
    "df_upload['target_id'] = df_upload['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>target_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>72904562259788371</td>\n",
       "      <td>72904562259788371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>73327186840388111</td>\n",
       "      <td>73327186840388111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>72341886848730335</td>\n",
       "      <td>72341886848730335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>72342161793745692</td>\n",
       "      <td>72342161793745692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>72834605496926408</td>\n",
       "      <td>72834605496926408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>73186037169389815</td>\n",
       "      <td>73186037169389815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015</th>\n",
       "      <td>72412049501587544</td>\n",
       "      <td>72412049501587544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016</th>\n",
       "      <td>73114912443859674</td>\n",
       "      <td>73114912443859674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017</th>\n",
       "      <td>73115393345978782</td>\n",
       "      <td>73115393345978782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018</th>\n",
       "      <td>73114912242532399</td>\n",
       "      <td>73114912242532399</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2019 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     id          target_id\n",
       "0     72904562259788371  72904562259788371\n",
       "1     73327186840388111  73327186840388111\n",
       "2     72341886848730335  72341886848730335\n",
       "3     72342161793745692  72342161793745692\n",
       "4     72834605496926408  72834605496926408\n",
       "...                 ...                ...\n",
       "2014  73186037169389815  73186037169389815\n",
       "2015  72412049501587544  72412049501587544\n",
       "2016  73114912443859674  73114912443859674\n",
       "2017  73115393345978782  73115393345978782\n",
       "2018  73114912242532399  73114912242532399\n",
       "\n",
       "[2019 rows x 2 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_upload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%script false\n",
    "\n",
    "k=10000\n",
    "minidfs = [df_upload.loc[i:i+k-1, :] for i in range(0, len(df_upload), k)]\n",
    "for dftmp in minidfs:\n",
    "    stage = client.annotation.stage_annotations(\"glia_somas_dec2022\", id_field=True)\n",
    "    stage.add_dataframe(dftmp)\n",
    "    client.annotation.upload_staged_annotations(stage)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.13 ('nuclei': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "3e8f183bd7944e152f2cac16c10b6a857d859db4a3e89a573de1818f219d7cd5"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}