Download all .fastq files in a project/Download all .fastq files in a project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Downloading all `.fastq`s in a project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this vignette, we'll demonstrate how to identify bundles associated with a given project, and how to download all `.fastq` files in those bundles.\n",
    "\n",
    "This vignette assumes some familiarity with the HCA API and command line tools. If you need to brush up, try paging through [the other vignettes](https://github.com/HumanCellAtlas/data-consumer-vignettes/) - the [Download Any BAM File](https://github.com/HumanCellAtlas/data-consumer-vignettes/tree/master/Download%20Any%20BAM%20File) vignette will be particularly relevant. Make sure that you've installed the HCA CLI and that you're logged into it before starting!\n",
    "\n",
    "## Method 1: Using the Data Explorer\n",
    "\n",
    "The fastest way to get all `.fastq`s associated with a project is to find the project in the HCA Data Explorer, then download a file manifest for the `.fastq`s associated with the project, then use `hca dss download-manifest` to download the files you selected. See [this guide](https://data.humancellatlas.org/guides/quick-start-guide) for details.\n",
    "\n",
    "## Method 2: Using the HCA DSS REST API, the long way\n",
    "\n",
    "We can also retrieve all `.fastq`s associated with a project with the DSS API. First, you'll want to find a project that you like on the [HCA Data Explorer](https://data.humancellatlas.org/explore/). (You'll probably want to filter for `.fastq` files to make sure that whatever you're downloading has the kind of data that you're looking for.)\n",
    "\n",
    "Once you find a project you like, open the project in the Data Explorer and then copy its URL. The UUID is at the end of the URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'74b6d569-3b11-42ef-b6b1-a0454522b4a0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = 'https://data.humancellatlas.org/explore/projects/74b6d569-3b11-42ef-b6b1-a0454522b4a0'\n",
    "_, project_uuid = url.rsplit('/', maxsplit=1)\n",
    "project_uuid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(This tutorial works a lot better if you choose a project that you know has `.fastq`s in it - see Method 1).\n",
    "\n",
    "Given a project UUID, we can make a `POST /search` request to the Data Store to search for all bundles that belong to the given project. `POST /search` requests accept ElasticSearch queries. In a bundle, the `files.project_json.provenance.document_id` field contains the project UUID. That means we want a query that looks like this:\n",
    "\n",
    "    {\n",
    "      \"query\": {\n",
    "        \"bool\": {\n",
    "          \"must\": {\n",
    "            \"match\": {\n",
    "              \"files.project_json.provenance.document_id\": project_uuid\n",
    "            }\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "\n",
    "We can use the HCA API to search for bundles matching this query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 5459 bundles in this project. Wow!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'bundle_fqid': 'fffcea5e-2e6c-4ca1-9aa9-c23b90b2e8b8.2019-05-16T211813.059000Z',\n",
       " 'bundle_url': 'https://dss.data.humancellatlas.org/v1/bundles/fffcea5e-2e6c-4ca1-9aa9-c23b90b2e8b8?version=2019-05-16T211813.059000Z&replica=aws',\n",
       " 'search_score': None}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import hca.dss\n",
    "client = hca.dss.DSSClient()\n",
    "query = {'query':\n",
    "            {'bool':\n",
    "                {'must':\n",
    "                    {'match': {'files.project_json.provenance.document_id': project_uuid}}\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "results = client.post_search(replica='aws', es_query=query)\n",
    "bundle_uuid, bundle_version = results['results'][0]['bundle_fqid'].split('.', maxsplit=1)\n",
    "print(\"There are %d bundles in this project. Wow!\" % results['total_hits'])\n",
    "results['results'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can actually improve this code - `post_search` is a paginated endpoint, and might only return a subset of all results with each request. We *could* control how many results a `post_search` request returns with the `per_page` argument, but if we use the `post_search.iterate` method, the API Python bindings will handle pagination for us transparently:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fffcea5e-2e6c-4ca1-9aa9-c23b90b2e8b8 2019-05-16T211813.059000Z\n",
      "ffc1714d-caef-41fe-9b68-a6db5efd4c69 2019-05-16T211813.060000Z\n",
      "ffa728d1-3767-46a0-ad94-adefd36630c9 2019-05-16T211813.105000Z\n",
      "ff982627-9abf-4d66-bc45-f9298ffb9311 2019-05-16T211813.111000Z\n",
      "ff8020e9-5bce-491b-b5f3-821ab25fb167 2019-05-16T211813.082000Z\n",
      "ff781a9d-e4f1-4a4f-acee-8bf28d8cd684 2019-05-16T211813.068000Z\n",
      "ff74ec2f-21b7-4949-8c55-42315afa8295 2019-05-16T211813.074000Z\n",
      "ff5e0cfc-bb2a-4f20-bade-897f1626fa8d 2019-05-16T211813.074000Z\n",
      "ff5b1de9-fb65-4451-95dc-7da7d11bd493 2019-05-16T211813.105000Z\n",
      "ff5a4036-08cf-4abc-98a7-572eaaa452cc 2019-05-16T211813.057000Z\n"
     ]
    }
   ],
   "source": [
    "results = client.post_search.iterate(replica='aws', es_query=query)\n",
    "# Let's print just the first ten results for brevity. We can 'slice'\n",
    "# the first ten results out of `results` using itertools.islice\n",
    "import itertools\n",
    "for result in itertools.islice(results, 10):\n",
    "    bundle_uuid, bundle_version = result['bundle_fqid'].split('.', maxsplit=1)\n",
    "    print(bundle_uuid, bundle_version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice! That's a lot of bundles...\n",
    "\n",
    "Now that we've identified all the bundles belonging to the project we're interested in, we can make a `GET /bundle/{uuid}` request to list all the files in a given bundle, and from there, identify all `.fastq` files in each bundle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bundle': {'creator_uid': 8008,\n",
       "  'files': [{'content-type': 'application/json; dcp-type=\"metadata/biomaterial\"',\n",
       "    'crc32c': 'f04827fa',\n",
       "    'indexed': True,\n",
       "    'name': 'cell_suspension_0.json',\n",
       "    's3_etag': '458bf0395db347c39bfaf267b0154568',\n",
       "    'sha1': 'd5ba2ba27848c3b41d42c6e275aa4e3ad7cc6515',\n",
       "    'sha256': '0e50c74847558a3123546b6e9d7eaf6766d504f42e6a7b389dba0369876af33d',\n",
       "    'size': 1366,\n",
       "    'uuid': 'd980eed4-9ac6-4343-81fc-647b0d88239f',\n",
       "    'version': '2019-05-16T162211.753000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/biomaterial\"',\n",
       "    'crc32c': 'd3fc2fae',\n",
       "    'indexed': True,\n",
       "    'name': 'specimen_from_organism_0.json',\n",
       "    's3_etag': '5c2033719c75743ed71058d7becf31ca',\n",
       "    'sha1': '53b104f207ac57ab1f4e07cf6d5634ca856852e7',\n",
       "    'sha256': 'cf9325255cc7bc9763844bbd1f50c7616f907985524dffd972d849f430d8bb07',\n",
       "    'size': 1687,\n",
       "    'uuid': '324191c3-db02-4671-9c51-1fdca8e6e88f',\n",
       "    'version': '2019-05-16T161926.507000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/biomaterial\"',\n",
       "    'crc32c': '9625a22d',\n",
       "    'indexed': True,\n",
       "    'name': 'donor_organism_0.json',\n",
       "    's3_etag': '102c0cf9d76a03752e7f79d7125492af',\n",
       "    'sha1': '7dd3a024d330d32a580354f2054d45d8ce05c129',\n",
       "    'sha256': 'cbd02327b590f1db5580766cb54efea98bc5e93236d434507365ad36af7b349f',\n",
       "    'size': 1442,\n",
       "    'uuid': '3c60e613-5ace-44e0-a07c-4c14bc28a2b3',\n",
       "    'version': '2019-05-16T161920.184000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/file\"',\n",
       "    'crc32c': '86e5a5cc',\n",
       "    'indexed': True,\n",
       "    'name': 'sequence_file_0.json',\n",
       "    's3_etag': '3d51e4559b3e4fed8dbf29d921b55f54',\n",
       "    'sha1': 'a40aecd4e97551f1e6ccf5643d9f0797898543a7',\n",
       "    'sha256': '35ed3f53c4dcdeac32fc9159b41144a705fbd8eef23dca12edebdff78fe0fdfd',\n",
       "    'size': 685,\n",
       "    'uuid': '87057262-cc5e-453e-bdae-48284146201b',\n",
       "    'version': '2019-05-16T175146.186000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/file\"',\n",
       "    'crc32c': '97ce552e',\n",
       "    'indexed': True,\n",
       "    'name': 'sequence_file_1.json',\n",
       "    's3_etag': '4c17f17f87d1979d323a2408a37b4fd2',\n",
       "    'sha1': 'b3222676caae0cb71184ce5319895738d55b641e',\n",
       "    'sha256': 'c075b3efa38dd71af6ecdde90e097f88b484830e3ff4aa54b7497ece20257ac7',\n",
       "    'size': 685,\n",
       "    'uuid': '8a1c4348-0fb2-4567-b694-eaef973ed39c',\n",
       "    'version': '2019-05-16T175148.065000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/file\"',\n",
       "    'crc32c': 'e78f94e2',\n",
       "    'indexed': True,\n",
       "    'name': 'sequence_file_2.json',\n",
       "    's3_etag': '9f747c94434d25bdca44d42eab91bd11',\n",
       "    'sha1': '9820a1f9b19e3db53a88bfc785fea293d2bc1e69',\n",
       "    'sha256': 'c884cd92cecbbf9a6dc2abbdcff4b84abfade41504e00f3d6207175769dced96',\n",
       "    'size': 686,\n",
       "    'uuid': 'b0a967a2-e69f-4a71-9e94-c78943afb1b0',\n",
       "    'version': '2019-05-16T175200.121000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/project\"',\n",
       "    'crc32c': '9647ec22',\n",
       "    'indexed': True,\n",
       "    'name': 'project_0.json',\n",
       "    's3_etag': '789513ea661a6950c53c825c57dd499a',\n",
       "    'sha1': 'afe181a2581b712d660a9a8022b4e9fd0e9a11c1',\n",
       "    'sha256': '7fa59ec6b3ca57e32467e340c82592c42343764cfe29012950a98538729328d3',\n",
       "    'size': 5204,\n",
       "    'uuid': '74b6d569-3b11-42ef-b6b1-a0454522b4a0',\n",
       "    'version': '2019-05-16T161920.125000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/protocol\"',\n",
       "    'crc32c': '9e68fec0',\n",
       "    'indexed': True,\n",
       "    'name': 'library_preparation_protocol_0.json',\n",
       "    's3_etag': '4d9bf42a40d617723d504443e04b0b47',\n",
       "    'sha1': '046b7cce2329addfafa6bdc1e5844567784b0ab5',\n",
       "    'sha256': 'ab299d4334ff7a6bdd84786f18fbc4acfcf2a5bdb68f80fa53d493119039205f',\n",
       "    'size': 1222,\n",
       "    'uuid': 'c2dacc52-da61-49a6-ac4f-6a684ae45d4f',\n",
       "    'version': '2019-05-16T161630.693000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/protocol\"',\n",
       "    'crc32c': 'bca4462b',\n",
       "    'indexed': True,\n",
       "    'name': 'sequencing_protocol_0.json',\n",
       "    's3_etag': '16628e132df43f10fdde8e0df676f659',\n",
       "    'sha1': 'b0b63c2577fe75be0ea26f5c2445b2ee931637a4',\n",
       "    'sha256': '2ad9e10bf7d2e662b34fbf9dac67629e308664f987a605601de9546846fc3b21',\n",
       "    'size': 967,\n",
       "    'uuid': '95dec35a-6ddd-4595-ae12-0f952fe3416d',\n",
       "    'version': '2019-05-16T161631.087000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/protocol\"',\n",
       "    'crc32c': '55fdc35e',\n",
       "    'indexed': True,\n",
       "    'name': 'dissociation_protocol_0.json',\n",
       "    's3_etag': 'd110e06c9cb826c0df9417ea582cfb41',\n",
       "    'sha1': '19cb3a4d6b7665b651203e567b65cb356fce1c7d',\n",
       "    'sha256': '8faec66817969ae6f847b0c649e7328af6085d88e71b32b0e3a8284df4cd88f7',\n",
       "    'size': 1178,\n",
       "    'uuid': '08b0be6b-23dd-40ce-8d2d-800ebc3afd15',\n",
       "    'version': '2019-05-16T161630.582000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/process\"',\n",
       "    'crc32c': '189dd58d',\n",
       "    'indexed': True,\n",
       "    'name': 'process_0.json',\n",
       "    's3_etag': '4168eaa059fb198ee610a35cb97a025a',\n",
       "    'sha1': '190e3403fd3077d73577505a4be2bfec5d4d840d',\n",
       "    'sha256': '8efce6c81dbb470c03e34a216957d318335b14ede82c064f5dc98a0d326a0c6f',\n",
       "    'size': 408,\n",
       "    'uuid': 'adb89502-180e-4dcf-ad8d-5d4dbad1f7e8',\n",
       "    'version': '2019-05-16T163124.481000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/process\"',\n",
       "    'crc32c': 'a2089808',\n",
       "    'indexed': True,\n",
       "    'name': 'process_1.json',\n",
       "    's3_etag': 'f18afaa7978f81980a8a1641df46c76d',\n",
       "    'sha1': 'df09cc36a792bb57e91dc03b15e6d0db006884cd',\n",
       "    'sha256': '093c719f4a51558d94b7aedef062b9cd4628f09f79d87d434de14c39070fafd2',\n",
       "    'size': 377,\n",
       "    'uuid': '506fab03-177b-4095-a900-7b73af0a23d3',\n",
       "    'version': '2019-05-16T163104.286000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/process\"',\n",
       "    'crc32c': '7165b9c5',\n",
       "    'indexed': True,\n",
       "    'name': 'process_2.json',\n",
       "    's3_etag': 'bac86dabca0f941e46f42e782a33f1d3',\n",
       "    'sha1': '7933122a8b63402045d71f99a58e620e7ec2dd55',\n",
       "    'sha256': 'f5e5d17140d646626237acf34fdb2b69033e6b69a870e869f94173560a4cb8f4',\n",
       "    'size': 376,\n",
       "    'uuid': '84574386-5993-4997-9a3f-ba9d033d48f3',\n",
       "    'version': '2019-05-16T163103.532000Z'},\n",
       "   {'content-type': 'application/json; dcp-type=\"metadata/links\"',\n",
       "    'crc32c': 'c73b7fcf',\n",
       "    'indexed': True,\n",
       "    'name': 'links.json',\n",
       "    's3_etag': '68b1108e9a51b1e7b81a75ec7d311fa3',\n",
       "    'sha1': 'b5dd161a6606665dc2b13626d536db2b7f44d9bc',\n",
       "    'sha256': '402671047c82b9d6c19c619ec5618ec22a4711edb279753fec9e0f87d4271fa7',\n",
       "    'size': 1968,\n",
       "    'uuid': '84390049-73af-4051-b033-e7f975d68e4f',\n",
       "    'version': '2019-05-17T002145.360957Z'},\n",
       "   {'content-type': 'application/gzip; dcp-type=data',\n",
       "    'crc32c': 'a4315fae',\n",
       "    'indexed': False,\n",
       "    'name': 'E18_20160930_Neurons_Sample_41_S039_L004_I1_002.fastq.gz',\n",
       "    's3_etag': '4ee97bf8ecf486bed6a82ba26aa0119a',\n",
       "    'sha1': 'f331531b105e7a3e02798de723bdb2edf1f552c9',\n",
       "    'sha256': 'd13e8d17d264bf5430cbf6400e933edba53376e35b0d18a47a1d306580b670aa',\n",
       "    'size': 47016964,\n",
       "    'uuid': '563500d1-4831-48f7-bfe1-6a649aefe419',\n",
       "    'version': '2019-05-17T002145.782835Z'},\n",
       "   {'content-type': 'application/gzip; dcp-type=data',\n",
       "    'crc32c': 'ed642943',\n",
       "    'indexed': False,\n",
       "    'name': 'E18_20160930_Neurons_Sample_41_S039_L004_R1_002.fastq.gz',\n",
       "    's3_etag': 'e7255450e9598a491c42cf1aed037857-2',\n",
       "    'sha1': '7ac961ad94e99d7d7b1e4b8dae51480404c5ab98',\n",
       "    'sha256': 'ff3d272b26735d5e0d785416c03bb8e92ac5f3b8dae17899b225c66ac3b426ab',\n",
       "    'size': 112680375,\n",
       "    'uuid': 'bfdb5791-7b9a-4dd3-8972-3a8bef887953',\n",
       "    'version': '2019-05-17T002146.005586Z'},\n",
       "   {'content-type': 'application/gzip; dcp-type=data',\n",
       "    'crc32c': 'c2963377',\n",
       "    'indexed': False,\n",
       "    'name': 'E18_20160930_Neurons_Sample_41_S039_L004_R2_002.fastq.gz',\n",
       "    's3_etag': '1be46d40fd29c06eaeb920103509de85-8',\n",
       "    'sha1': '3cabd21a19470cb863bf64c3595c3e86ae658ade',\n",
       "    'sha256': '3a4dfd8135dfc57bb261d1b89c8dbf8ecf9f06b01c9b71991cecd28ee16056e7',\n",
       "    'size': 507444197,\n",
       "    'uuid': 'c5c6f7dc-ccfc-45a2-96c5-fb0bfe2ae10a',\n",
       "    'version': '2019-05-17T002146.210252Z'}],\n",
       "  'uuid': 'ff5a4036-08cf-4abc-98a7-572eaaa452cc',\n",
       "  'version': '2019-05-16T211813.057000Z'}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.get_bundle(uuid=bundle_uuid, replica='aws')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write a function `find_fastqs_in_bundle` that, given a bundle UUID, will return the UUIDs of `.fastq` files in that bundle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_fastqs_in_bundle(bundle_uuid, version):\n",
    "    bundle = client.get_bundle(uuid=bundle_uuid, version=version, replica='aws')\n",
    "    fastq_files = []\n",
    "    for file_ in bundle['bundle']['files']:\n",
    "        if file_['name'].endswith('.fastq.gz'):\n",
    "            fastq_files.append(file_)\n",
    "    return fastq_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experienced Pythonistas will recognize that we can make a small improvement to `find_fastqs_in_bundle` that is more concise and a little faster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_fastqs_in_bundle(bundle_uuid,version ):\n",
    "    bundle = client.get_bundle(uuid=bundle_uuid, version=version, replica='aws')\n",
    "    return filter(lambda x: x['name'].endswith('.fastq.gz'), bundle['bundle']['files'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's try using `find_fastqs_in_bundle` with all of the bundles we found at the beginning of this tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bundle fffcea5e-2e6c-4ca1-9aa9-c23b90b2e8b8.2019-05-16T211813.059000Z has E18_20161004_Neurons_Sample_14_S083_L005_I1_002.fastq.gz\n",
      "Bundle fffcea5e-2e6c-4ca1-9aa9-c23b90b2e8b8.2019-05-16T211813.059000Z has E18_20161004_Neurons_Sample_14_S083_L005_R1_002.fastq.gz\n",
      "Bundle fffcea5e-2e6c-4ca1-9aa9-c23b90b2e8b8.2019-05-16T211813.059000Z has E18_20161004_Neurons_Sample_14_S083_L005_R2_002.fastq.gz\n",
      "Bundle ffc1714d-caef-41fe-9b68-a6db5efd4c69.2019-05-16T211813.060000Z has E18_20161004_Neurons_Sample_57_S126_L008_I1_002.fastq.gz\n",
      "Bundle ffc1714d-caef-41fe-9b68-a6db5efd4c69.2019-05-16T211813.060000Z has E18_20161004_Neurons_Sample_57_S126_L008_R1_002.fastq.gz\n",
      "Bundle ffc1714d-caef-41fe-9b68-a6db5efd4c69.2019-05-16T211813.060000Z has E18_20161004_Neurons_Sample_57_S126_L008_R2_002.fastq.gz\n",
      "Bundle ffa728d1-3767-46a0-ad94-adefd36630c9.2019-05-16T211813.105000Z has E18_20160930_Neurons_Sample_61_S058_L002_I1_010.fastq.gz\n",
      "Bundle ffa728d1-3767-46a0-ad94-adefd36630c9.2019-05-16T211813.105000Z has E18_20160930_Neurons_Sample_61_S058_L002_R1_010.fastq.gz\n",
      "Bundle ffa728d1-3767-46a0-ad94-adefd36630c9.2019-05-16T211813.105000Z has E18_20160930_Neurons_Sample_61_S058_L002_R2_010.fastq.gz\n",
      "Bundle ff982627-9abf-4d66-bc45-f9298ffb9311.2019-05-16T211813.111000Z has E18_20161004_Neurons_Sample_52_S090_L005_I1_011.fastq.gz\n",
      "Bundle ff982627-9abf-4d66-bc45-f9298ffb9311.2019-05-16T211813.111000Z has E18_20161004_Neurons_Sample_52_S090_L005_R1_011.fastq.gz\n",
      "Bundle ff982627-9abf-4d66-bc45-f9298ffb9311.2019-05-16T211813.111000Z has E18_20161004_Neurons_Sample_52_S090_L005_R2_011.fastq.gz\n",
      "Bundle ff8020e9-5bce-491b-b5f3-821ab25fb167.2019-05-16T211813.082000Z has E18_20160930_Neurons_Sample_27_S025_L005_I1_007.fastq.gz\n",
      "Bundle ff8020e9-5bce-491b-b5f3-821ab25fb167.2019-05-16T211813.082000Z has E18_20160930_Neurons_Sample_27_S025_L005_R1_007.fastq.gz\n",
      "Bundle ff8020e9-5bce-491b-b5f3-821ab25fb167.2019-05-16T211813.082000Z has E18_20160930_Neurons_Sample_27_S025_L005_R2_007.fastq.gz\n",
      "Bundle ff781a9d-e4f1-4a4f-acee-8bf28d8cd684.2019-05-16T211813.068000Z has E18_20161004_Neurons_Sample_55_S115_L003_I1_003.fastq.gz\n",
      "Bundle ff781a9d-e4f1-4a4f-acee-8bf28d8cd684.2019-05-16T211813.068000Z has E18_20161004_Neurons_Sample_55_S115_L003_R1_003.fastq.gz\n",
      "Bundle ff781a9d-e4f1-4a4f-acee-8bf28d8cd684.2019-05-16T211813.068000Z has E18_20161004_Neurons_Sample_55_S115_L003_R2_003.fastq.gz\n",
      "Bundle ff74ec2f-21b7-4949-8c55-42315afa8295.2019-05-16T211813.074000Z has E18_20161004_Neurons_Sample_17_S016_L004_I1_005.fastq.gz\n",
      "Bundle ff74ec2f-21b7-4949-8c55-42315afa8295.2019-05-16T211813.074000Z has E18_20161004_Neurons_Sample_17_S016_L004_R1_005.fastq.gz\n",
      "Bundle ff74ec2f-21b7-4949-8c55-42315afa8295.2019-05-16T211813.074000Z has E18_20161004_Neurons_Sample_17_S016_L004_R2_005.fastq.gz\n",
      "Bundle ff5e0cfc-bb2a-4f20-bade-897f1626fa8d.2019-05-16T211813.074000Z has E18_20161004_Neurons_Sample_23_S022_L003_I1_005.fastq.gz\n",
      "Bundle ff5e0cfc-bb2a-4f20-bade-897f1626fa8d.2019-05-16T211813.074000Z has E18_20161004_Neurons_Sample_23_S022_L003_R1_005.fastq.gz\n",
      "Bundle ff5e0cfc-bb2a-4f20-bade-897f1626fa8d.2019-05-16T211813.074000Z has E18_20161004_Neurons_Sample_23_S022_L003_R2_005.fastq.gz\n",
      "Bundle ff5b1de9-fb65-4451-95dc-7da7d11bd493.2019-05-16T211813.105000Z has E18_20160930_Neurons_Sample_61_S058_L004_I1_010.fastq.gz\n",
      "Bundle ff5b1de9-fb65-4451-95dc-7da7d11bd493.2019-05-16T211813.105000Z has E18_20160930_Neurons_Sample_61_S058_L004_R1_010.fastq.gz\n",
      "Bundle ff5b1de9-fb65-4451-95dc-7da7d11bd493.2019-05-16T211813.105000Z has E18_20160930_Neurons_Sample_61_S058_L004_R2_010.fastq.gz\n",
      "Bundle ff5a4036-08cf-4abc-98a7-572eaaa452cc.2019-05-16T211813.057000Z has E18_20160930_Neurons_Sample_41_S039_L004_I1_002.fastq.gz\n",
      "Bundle ff5a4036-08cf-4abc-98a7-572eaaa452cc.2019-05-16T211813.057000Z has E18_20160930_Neurons_Sample_41_S039_L004_R1_002.fastq.gz\n",
      "Bundle ff5a4036-08cf-4abc-98a7-572eaaa452cc.2019-05-16T211813.057000Z has E18_20160930_Neurons_Sample_41_S039_L004_R2_002.fastq.gz\n"
     ]
    }
   ],
   "source": [
    "results = client.post_search.iterate(replica='aws', es_query=query)\n",
    "for result in itertools.islice(results, 10):\n",
    "    bundle_uuid, bundle_version = result['bundle_fqid'].split('.', maxsplit=1)\n",
    "    for fastq in find_fastqs_in_bundle(bundle_uuid, bundle_version):\n",
    "        print(f\"Bundle {result['bundle_fqid']} has {fastq['name']}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}