POIs API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Global Parameters & Imports for below functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import urllib.request\n",
    "import json\n",
    "import html\n",
    "import re\n",
    "from haversine import haversine, haversine_vector\n",
    "import heapq\n",
    "import pandas as pd\n",
    "from dateutil import parser\n",
    "\n",
    "limit = 1\n",
    "radius = 6\n",
    "lat = 52.635875 #uk - norfolk\n",
    "lng = 1.301 #uk - norfolk\n",
    "# lat = 52.50003299 #germany - berlin\n",
    "# lng = 13.3913285 #germany - berlin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notes about the Overall Design Schema adopted in the POI Scraper Project:\n",
    "A consistent design schema was adopted in this project with the aim of making all code as similar and safe as possible, allowing easy understanding of the projects inner workings for newcomers, while making altering prior code easier. This Schema is detailed below:\n",
    "## New Heap code structure that deals with all cases of missing and out of order days:\n",
    "* This is the standardised structure, introduced in **v9**, used throughout all store functions that use an \"integer based\" system for keeping store opening hours data. This term encompassesmeans all systems where the opening hours are not stored in a dictionary with the keys explicitly identifying/being the days themselves. For example any system where the days are stored as an array would count as an \"integer based\" system, or one where the days were stored in a dictionary with the keys being 0-n with the day then stored as a field inside this dictionary (e.g. Co-op, or slightly less obviously, Sainsbury's). However a system such as Tesco's where the keys of the dictionary are the days themselves, a heap system is unneccessary and therefore a simple loop containing a try, except clause suffices (these such systems are not counter as \"integer based\" systems). At the core of this structure, the code for processing and producing the dictionaries that form the opening hours array is placed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'openingHours' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-1-b2ae53f9616c>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mdayDoneSet\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mset\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mdaysHeap\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[1;32mfor\u001b[0m \u001b[0mdayObject\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mopeningHours\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m         \u001b[0mday\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mintermediaryDayKeys\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mdayObject\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m\"day\"\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m         \u001b[1;31m#if statement to deal with days being included twice (implemnted due to a bug in rewe system)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'openingHours' is not defined"
     ]
    }
   ],
   "source": [
    "# Copyable Heap structure code:\n",
    "dayDoneSet = set()\n",
    "daysHeap = []\n",
    "for dayObject in openingHours:\n",
    "        day = intermediaryDayKeys[dayObject[\"day\"]]\n",
    "        #if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "        if day not in dayDoneSet:\n",
    "            dayDoneSet.add(day)\n",
    "            #keyHours processing code\n",
    "            #INSERT HOUR PROCESSING CODE HERE\n",
    "        \n",
    "    \n",
    "\n",
    "            heapq.heappush(daysHeap, [day,keyHours]) #(REPLACE hoursArray.append(keyHours) with this statement)\n",
    "#Check for any missing days and ensure all days are in order for insertion into our database\n",
    "for missingDay in set(dayKeys.keys()).difference(dayDoneSet):\n",
    "    closedDayHours = {\n",
    "            \"day\": dayKeys[missingDay],\n",
    "            \"open\": False\n",
    "    }\n",
    "    heapq.heappush(daysHeap, [missingDay, closedDayHours])\n",
    "if len(daysHeap) != 7:\n",
    "    return False\n",
    "hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "return hoursArray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional notes about design schema throughout:\n",
    "In addition to that stated above, in an API system that returns (or alternatively is able to return) all stores rather than the stores close to a pair of coordinates, the standardised approach adopted in this project has been to set up a database of preprepared opening hours arrays, stored along with the coordinates of the stores themselves. This database is then called by the get_X_data(lat, lng) function, where the haversine_vector function is called to efficiently compute the distances between all stores and the desired point. The minimum index is then retrieved and the opening hours of that indexed store are returned (see EDEKA or Netto's Sections in the below document for examples of both parts of this process).\n",
    "\n",
    "# Documentation for all Stores:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# REWE's API (Germany)\n",
    "## Base URL:\n",
    "https://www.rewe.de/market/content/marketsearch\n",
    "## Parameters:\n",
    "**Note:** Parameters are not usually supplied in a traditional fashion, they are supplied as \"Form Data\" (under the Chrome network inspector). However, after testing, the API does seem to return results if the same parameters are supplied as traditional queries?\n",
    "\n",
    "\n",
    "**WARNING:** If the API call is made from the python requests module, it will return 403 Forbidden as its status code. After much work it was discovered that a GET request made through the urllib module using the headers shown below (specifically \"user-agent\") is succesful (200 OK status code).\n",
    "* searchString: Literally searches the store names and addresses for **STRING MATCH** so if post code is 12099 giving 12098 will give no results for example.\n",
    "* **unknown** latitude and longitude option?? Tried: lat lng, lat lon, latitude longitude, coordinates. I believe their system is extremely simplistic and has no form of search beyond searchString, or their premade city tabs.\n",
    "* page = 0 (which batch of pageSize stores to return)\n",
    "* city (optional)\n",
    "* pageSize = number of stores to return (up to 500, beyond there only 500 stores will be returned, and page system breaks(?))\n",
    "\n",
    "\n",
    "## Sample API Call:\n",
    "https://www.rewe.de/market/content/marketsearch?searchString=REWE&pageSize=500&page=0\n",
    "## Sample API Call Python Code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "import json\n",
    "API_URL = \"https://www.rewe.de/market/content/marketsearch\"\n",
    "headers = {\n",
    "    \"User-Agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36\",\n",
    "    \"accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\",\n",
    "    \"accept-language\": \"en-US,en;q=0.9\",\n",
    "    \"sec-fetch-dest\": \"document\",\n",
    "    \"sec-fetch-mode\": \"navigate\",\n",
    "    \"sec-fetch-site\": \"none\",\n",
    "    \"sec-fetch-user\": \"?1\",\n",
    "    \"upgrade-insecure-requests\": \"1\"\n",
    "}\n",
    "params = {  'searchString' : \"REWE\",\n",
    "            'pageSize' : 500,\n",
    "            'page' : 0}\n",
    "req_url = API_URL + '?' + '&'.join(k + '=' + str(v) for k, v in params.items())\n",
    "resp = urllib.request.urlopen(urllib.request.Request(url=req_url, data=None, headers=headers))\n",
    "data = resp.read().decode('utf-8')\n",
    "res = json.loads(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Response:\n",
    "**Note:** Two sample responses are expanded (the second one partially) in order to exacerbate that stores may have multiple \"opening hours day ranges\", and thus this must be accounted for.\n",
    "\n",
    "**WARNING:** Occasionally some stores return bugged store opening hours (always the same stores) where the day ranges overlap and often disagree for hours of individual days. \n",
    "**E.g.**:\n",
    "{'days': 'Do', 'hours': '08:00 - 22:00'}, {'days': 'Fr-Sa', 'hours': '08:00 - 20:00'}, {'days': 'Mo-Sa', 'hours': '08:00 - 20:00'}\n",
    "This is why the dayDoneSet were introduced in the set_up_rewe_database() function.\n",
    "\n",
    "**WARNING:** Some stores return empty lists for opening hours, there is a comment in the code where this check is performed, any code dealing with these stores e.g. flagging them as closed may be performed there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-1-23983c798c4c>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-1-23983c798c4c>\"\u001b[1;36m, line \u001b[1;32m1\u001b[0m\n\u001b[1;33m    **Fetch:**\u001b[0m\n\u001b[1;37m    ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "markets: [{marketManager: \"\", phone: null, advertisingCounty: null, regionShort: \"WE\",…},…]\n",
    "    [0 … 99]\n",
    "        0: {marketManager: \"\", phone: null, advertisingCounty: null, regionShort: \"WE\",…}\n",
    "            address: {street: \"Saarwerdenstraße\", houseNumber: \"6\", postalCode: \"41541\", city: \"Dormagen - Zons\",…}\n",
    "                city: \"Dormagen - Zons\"\n",
    "                houseNumber: \"6\"\n",
    "                postalCode: \"41541\"\n",
    "                state: \"Nordrhein-Westfalen\"\n",
    "                street: \"Saarwerdenstraße\"\n",
    "                streetWithNumber: \"Saarwerdenstraße 6\"\n",
    "            advertisingCounty: null\n",
    "            closedFrom: null\n",
    "            closedUntil: null\n",
    "            company: {name: \"REWE Bernd Uderhardt oHG\", street: \"Saarwerdenstr. 6\", zipCode: \"41541\", city: \"Dormagen-Zons\"}\n",
    "                city: \"Dormagen-Zons\"\n",
    "                name: \"REWE Bernd Uderhardt oHG\"\n",
    "                street: \"Saarwerdenstr. 6\"\n",
    "                zipCode: \"41541\"\n",
    "            companyName: \"REWE Bernd Uderhardt oHG\"\n",
    "            dortmund: false\n",
    "            externalLink: null\n",
    "            firstOpeningDate: \"2004-12-01\"\n",
    "            geoLocation: {latitude: 51.123861, longitude: 6.84568}\n",
    "                latitude: 51.123861\n",
    "                longitude: 6.84568\n",
    "            holiday: null\n",
    "            holidayDate: null\n",
    "            id: 1765235\n",
    "            marketManager: \"\"\n",
    "            nahkauf: false\n",
    "            openOnSunday: false\n",
    "            opened: true\n",
    "            openedUntil: {hour: 22, minute: 0, second: 0, nano: 0}\n",
    "                hour: 22\n",
    "                minute: 0\n",
    "                nano: 0\n",
    "                second: 0\n",
    "            openingHours: {condensed: [{days: \"Mo-Sa\", hours: \"07:00 - 22:00\"}],…}\n",
    "                condensed: [{days: \"Mo-Sa\", hours: \"07:00 - 22:00\"}]\n",
    "                    0: {days: \"Mo-Sa\", hours: \"07:00 - 22:00\"}\n",
    "                        days: \"Mo-Sa\"\n",
    "                        hours: \"07:00 - 22:00\"\n",
    "                dayAndTimeRanges: [{startDay: \"MONDAY\", endDay: \"SATURDAY\", startTime: {hour: 7, minute: 0, second: 0, nano: 0},…}]\n",
    "                    0: {startDay: \"MONDAY\", endDay: \"SATURDAY\", startTime: {hour: 7, minute: 0, second: 0, nano: 0},…}\n",
    "                        closes: \"22:00\"\n",
    "                        endDay: \"SATURDAY\"\n",
    "                        endTime: {hour: 22, minute: 0, second: 0, nano: 0}\n",
    "                            hour: 22\n",
    "                            minute: 0\n",
    "                            nano: 0\n",
    "                            second: 0\n",
    "                        hour: 22\n",
    "                        minute: 0\n",
    "                        nano: 0\n",
    "                        second: 0\n",
    "                        opens: \"07:00\"\n",
    "                        seoRepresentation: \"Mo-Sa 07:00-22:00\"\n",
    "                        startDay: \"MONDAY\"\n",
    "                        startTime: {hour: 7, minute: 0, second: 0, nano: 0}\n",
    "                            hour: 7\n",
    "                            minute: 0\n",
    "                            nano: 0\n",
    "                            second: 0\n",
    "            phone: null\n",
    "            pickupMarket: true\n",
    "            regionShort: \"WE\"\n",
    "            specialEnd: null\n",
    "            specialOpening: null\n",
    "            specialOpeningHours: null\n",
    "            specialStart: null\n",
    "            state: \"Nordrhein-Westfalen\"\n",
    "            stateShort: \"NW\"\n",
    "            temporaryClosed: false\n",
    "            type: {name: \"REWE\", id: null}\n",
    "            id: null\n",
    "            name: \"REWE\"\n",
    "            wawi: \"43655399\"\n",
    "        1: {marketManager: \"\", phone: \"02631-946281\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        2: {marketManager: \"\", phone: \"089-30000521\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        3: {marketManager: \"\", phone: \"0211-86283696\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        4: {marketManager: \"\", phone: \"08331-9851762\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        5: {marketManager: \"\", phone: \"0721-83185594\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        6: {marketManager: \"\", phone: \"06442-95190\", advertisingCounty: null, regionShort: \"MI\",…}\n",
    "        7: {marketManager: \"\", phone: \"0531-2502474\", advertisingCounty: null, regionShort: \"NO\",…}\n",
    "        8: {marketManager: \"\", phone: \"07072-6009502\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        9: {marketManager: \"\", phone: \"0681-98229365\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        10: {marketManager: \"\", phone: \"04151-866759\", advertisingCounty: null, regionShort: \"NO\",…}\n",
    "        11: {marketManager: \"\", phone: \"0821-52138832\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        12: {marketManager: \"\", phone: \"0221-9359560\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        13: {marketManager: \"\", phone: \"036020-769723\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        14: {marketManager: \"\", phone: \"07031-744914\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        15: {marketManager: \"\", phone: \"06032-93760\", advertisingCounty: null, regionShort: \"MI\",…}\n",
    "        16: {marketManager: \"\", phone: \"08094-907956\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        17: {marketManager: \"\", phone: \"02401/896400\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        18: {marketManager: \"\", phone: \"02131-1249939\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        19: {marketManager: \"\", phone: \"09972-9019010\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        20: {marketManager: \"\", phone: \"08841-3022\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        21: {marketManager: \"\", phone: \"089-96209706\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        22: {marketManager: \"\", phone: \"0821-6084537\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        23: {marketManager: \"\", phone: \"02452-98872-0\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        24: {marketManager: \"\", phone: \"069-90749571\", advertisingCounty: null, regionShort: \"MI\",…}\n",
    "        25: {marketManager: \"\", phone: \"06092-97250\", advertisingCounty: null, regionShort: \"MI\",…}\n",
    "        26: {marketManager: \"\", phone: \"0214-8404797\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        27: {marketManager: \"\", phone: \"02722-9261-0\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        28: {marketManager: \"\", phone: \"06131-6278995\", advertisingCounty: null, regionShort: \"MI\",…}\n",
    "        29: {marketManager: \"\", phone: \"0511-804201\", advertisingCounty: null, regionShort: \"NO\",…}\n",
    "        30: {marketManager: \"\", phone: \"08142-29962\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        31: {marketManager: \"\", phone: \"0911-97901923\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        32: {marketManager: \"\", phone: \"03928-7699822\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        33: {marketManager: \"\", phone: \"089-89398183\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        34: {marketManager: \"\", phone: \"07129-928630\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        35: {marketManager: \"\", phone: \"036041-309889\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        36: {marketManager: \"\", phone: \"0621-46082209\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        37: {marketManager: \"\", phone: \"089-99300731\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        38: {marketManager: \"Niklas Gerlach\", phone: \"0421/84919860\", advertisingCounty: null, regionShort: \"NO\",…}\n",
    "            address: {street: \"Bahnhofsplatz\", houseNumber: \"42\", postalCode: \"28195\", city: \"Bremen, Stadt\",…}\n",
    "            advertisingCounty: null\n",
    "            closedFrom: null\n",
    "            closedUntil: null\n",
    "            company: {name: \"REWE Niklas Gerlach oHG\", street: \"Bahnhofsplatz 42\", zipCode: \"28195\", city: \"Bremen\"}\n",
    "            companyName: \"REWE Niklas Gerlach oHG\"\n",
    "            dortmund: false\n",
    "            externalLink: null\n",
    "            firstOpeningDate: \"2019-05-16\"\n",
    "            geoLocation: {latitude: 53.081348, longitude: 8.812994}\n",
    "            holiday: null\n",
    "            holidayDate: null\n",
    "            id: 541755\n",
    "            marketManager: \"Niklas Gerlach\"\n",
    "            nahkauf: false\n",
    "            openOnSunday: false\n",
    "            opened: true\n",
    "            openedUntil: {hour: 23, minute: 59, second: 0, nano: 0}\n",
    "            openingHours: {condensed: [{days: \"Mo-Fr\", hours: \"06:00 - 24:00\"}, {days: \"Sa\", hours: \"06:00 - 23:30\"}],…}\n",
    "                condensed: [{days: \"Mo-Fr\", hours: \"06:00 - 24:00\"}, {days: \"Sa\", hours: \"06:00 - 23:30\"}]\n",
    "                    0: {days: \"Mo-Fr\", hours: \"06:00 - 24:00\"}\n",
    "                        days: \"Mo-Fr\"\n",
    "                        hours: \"06:00 - 24:00\"\n",
    "                    1: {days: \"Sa\", hours: \"06:00 - 23:30\"}\n",
    "                        days: \"Sa\"\n",
    "                        hours: \"06:00 - 23:30\"\n",
    "                dayAndTimeRanges: [{startDay: \"MONDAY\", endDay: \"FRIDAY\", startTime: {hour: 6, minute: 0, second: 0, nano: 0},…},…]\n",
    "                    0: {startDay: \"MONDAY\", endDay: \"FRIDAY\", startTime: {hour: 6, minute: 0, second: 0, nano: 0},…}\n",
    "                        closes: \"23:59\"\n",
    "                        endDay: \"FRIDAY\"\n",
    "                        endTime: {hour: 23, minute: 59, second: 0, nano: 0}\n",
    "                        opens: \"06:00\"\n",
    "                        seoRepresentation: \"Mo-Fr 06:00-23:59\"\n",
    "                        startDay: \"MONDAY\"\n",
    "                        startTime: {hour: 6, minute: 0, second: 0, nano: 0}\n",
    "                    1: {startDay: \"SATURDAY\", endDay: null, startTime: {hour: 6, minute: 0, second: 0, nano: 0},…}\n",
    "                        closes: \"23:30\"\n",
    "                        endDay: null\n",
    "                        endTime: {hour: 23, minute: 30, second: 0, nano: 0}\n",
    "                        opens: \"06:00\"\n",
    "                        seoRepresentation: \"Sa 06:00-23:30\"\n",
    "                        startDay: \"SATURDAY\"\n",
    "                        startTime: {hour: 6, minute: 0, second: 0, nano: 0}\n",
    "            phone: \"0421/84919860\"\n",
    "            pickupMarket: true\n",
    "            regionShort: \"NO\"\n",
    "            specialEnd: null\n",
    "            specialOpening: null\n",
    "            specialOpeningHours: null\n",
    "            specialStart: null\n",
    "            state: \"Bremen\"\n",
    "            stateShort: \"HB\"\n",
    "            temporaryClosed: false\n",
    "            type: {name: \"REWE\", id: null}\n",
    "            wawi: \"41653792\"\n",
    "        39: {marketManager: \"\", phone: \"02236-3945206\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        40: {marketManager: \"\", phone: \"0711-99718392\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        41: {marketManager: \"\", phone: \"07425-3275109\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        42: {marketManager: \"\", phone: \"05303/9797939\", advertisingCounty: null, regionShort: \"NO\",…}\n",
    "        43: {marketManager: \"\", phone: \"069-13028727\", advertisingCounty: null, regionShort: \"MI\",…}\n",
    "        44: {marketManager: \"\", phone: \"08231-340221\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        45: {marketManager: \"\", phone: \"06821-8691052\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        46: {marketManager: \"\", phone: \"0711-9973658\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        47: {marketManager: \"\", phone: \"07307-955001\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        48: {marketManager: \"\", phone: \"030-56298753\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        49: {marketManager: \"\", phone: \"0331-74001370\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        50: {marketManager: \"\", phone: \"089-91059059\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        51: {marketManager: \"\", phone: \"0941-46021047\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        52: {marketManager: \"\", phone: \"06486-900600\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        53: {marketManager: \"\", phone: \"06236-462809\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        54: {marketManager: \"\", phone: \"02065-892138\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        55: {marketManager: \"\", phone: \"07731-984576\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        56: {marketManager: \"David Hegemann\", phone: \"0211 213108\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        57: {marketManager: \"\", phone: \"030-31504210\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        58: {marketManager: \"\", phone: \"030-28390336\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        59: {marketManager: \"\", phone: \"06222-70351\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        60: {marketManager: \"\", phone: \"030-52279324\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        61: {marketManager: \"\", phone: \"07842-994114\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        62: {marketManager: \"\", phone: \"05162-900583\", advertisingCounty: null, regionShort: \"NO\",…}\n",
    "        63: {marketManager: \"\", phone: \"0221-9320092\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        64: {marketManager: \"\", phone: \"07151-5027884\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        65: {marketManager: \"\", phone: \"07522-9155829\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        66: {marketManager: \"\", phone: \"07151-2051475\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        67: {marketManager: \"\", phone: \"08333-923938\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        68: {marketManager: \"\", phone: \"0511-5799402\", advertisingCounty: null, regionShort: \"NO\",…}\n",
    "        69: {marketManager: \"\", phone: \"05481/997173\", advertisingCounty: null, regionShort: \"NO\",…}\n",
    "        70: {marketManager: \"\", phone: \"07644-923407\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        71: {marketManager: \"\", phone: \"07305-927144\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        72: {marketManager: \"Vanessa Richter-Helms\", phone: \"04453-989620\", advertisingCounty: null,…}\n",
    "        73: {marketManager: \"\", phone: \"0221-9541363\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        74: {marketManager: \"\", phone: \"02443-315490\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        75: {marketManager: \"\", phone: \"0201-8405411\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        76: {marketManager: \"\", phone: \"039954-2797030\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        77: {marketManager: \"\", phone: \"03381-211538\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        78: {marketManager: \"\", phone: \"07042-8171934\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        79: {marketManager: \"\", phone: \"02739/1547\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        80: {marketManager: \"\", phone: \"030-61709080\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        81: {marketManager: \"\", phone: \"0385-5557430\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        82: {marketManager: \"\", phone: \"0381-8651690\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        83: {marketManager: \"\", phone: \"0221-557492\", advertisingCounty: null, regionShort: \"WE\",…}\n",
    "        84: {marketManager: \"Albion Karaxha\", phone: \"07545-1727\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        85: {marketManager: \"\", phone: \"03327-668970\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        86: {marketManager: \"\", phone: \"03831-66671130\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        87: {marketManager: \"\", phone: \"039601-3240130\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        88: {marketManager: \"\", phone: \"030-51062792\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        89: {marketManager: \"\", phone: \"06226-990814\", advertisingCounty: null, regionShort: \"SW\",…}\n",
    "        90: {marketManager: \"\", phone: \"0331-2373520\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        91: {marketManager: \"\", phone: \"0395-430640\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        92: {marketManager: \"\", phone: \"039753-32330\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        93: {marketManager: \"\", phone: \"089-66002219\", advertisingCounty: null, regionShort: \"SU\",…}\n",
    "        94: {marketManager: \"\", phone: \"0385-39576330\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        95: {marketManager: \"\", phone: \"030-78990490\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        96: {marketManager: \"\", phone: \"030-42016551\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        97: {marketManager: \"\", phone: \"030-54712219\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        98: {marketManager: \"\", phone: \"030-83202307\", advertisingCounty: null, regionShort: \"OS\",…}\n",
    "        99: {marketManager: \"\", phone: \"0531-5160190\", advertisingCounty: null, regionShort: \"NO\",…}\n",
    "    [100 … 199]\n",
    "    [200 … 299]\n",
    "    [300 … 399]\n",
    "    [400 … 498]\n",
    "nextPage: 4\n",
    "prevPage: 2\n",
    "total: 3236"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code for setting up a database filled with all of REWE's opening hours, formatted to our liking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_up_rewe_database():\n",
    "    API_URL = \"https://www.rewe.de/market/content/marketsearch\"\n",
    "    headers = { \"User-Agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36\",\n",
    "                \"accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\",\n",
    "                \"accept-language\": \"en-US,en;q=0.9\",\n",
    "                \"sec-fetch-dest\": \"document\",\n",
    "                \"sec-fetch-mode\": \"navigate\",\n",
    "                \"sec-fetch-site\": \"none\",\n",
    "                \"sec-fetch-user\": \"?1\",\n",
    "                \"upgrade-insecure-requests\": \"1\" \n",
    "              }\n",
    "    pageStoreCount = 1\n",
    "    page = 0\n",
    "    reweArray = []\n",
    "    intermediaryDayKeys = { 'MONDAY' : 0, 'TUESDAY' : 1, 'WEDNESDAY' : 2, 'THURSDAY' : 3, 'FRIDAY' : 4, 'SATURDAY' : 5, 'SUNDAY' : 6, None : None}\n",
    "    dayKeys = { 0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday', 5 : 'Saturday', 6 : 'Sunday'}\n",
    "    while pageStoreCount > 0:\n",
    "        params = {  'searchString' : \"REWE\",\n",
    "                    'pageSize' : 500,\n",
    "                    'page' : page}\n",
    "        try:\n",
    "            req_url = API_URL + '?' + '&'.join(k + '=' + str(v) for k, v in params.items())\n",
    "            rq = urllib.request.urlopen(urllib.request.Request(url=req_url, data=None, headers=headers))\n",
    "            if rq.status != 200:\n",
    "                return False\n",
    "            data = rq.read().decode('utf-8')\n",
    "            res = json.loads(data)\n",
    "            if res[\"total\"] == 0:\n",
    "                return False\n",
    "            res = res[\"markets\"]\n",
    "            pageStoreCount = len(res)\n",
    "        except:\n",
    "            return False\n",
    "        # print(res[347][\"openingHours\"][\"dayAndTimeRanges\"])\n",
    "        for index in range(len(res)):\n",
    "            dayDoneSet = set()\n",
    "            daysHeap = []\n",
    "            openingHours = res[index][\"openingHours\"][\"dayAndTimeRanges\"]\n",
    "            for dayRange in range(len(openingHours)):\n",
    "                startDay = intermediaryDayKeys[openingHours[dayRange][\"startDay\"]]\n",
    "                endDay = intermediaryDayKeys[openingHours[dayRange][\"endDay\"]]\n",
    "                opens = openingHours[dayRange][\"opens\"]\n",
    "                closes = openingHours[dayRange][\"closes\"]\n",
    "                actualHours = [{'open' : opens, 'close' : closes}]\n",
    "                #deal with 1 day ranges\n",
    "                if endDay == None:\n",
    "                    endDay = startDay\n",
    "                #deal with wrap around windows e.g. sunday to tuesday\n",
    "                if startDay > endDay:\n",
    "                    startDay += -7\n",
    "                for day in range(startDay, endDay+1):\n",
    "                    if day < 0:\n",
    "                        day += 7\n",
    "                    #if statement to deal with incorrect responses where multiple ranges cover the same day (implemnted due to a bug in their system)\n",
    "                    if day not in dayDoneSet:\n",
    "                        dayDoneSet.add(day)\n",
    "                        keyHours = {'day' : dayKeys[day],\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours\n",
    "                        }\n",
    "                        heapq.heappush(daysHeap, [day,keyHours])\n",
    "            # simple heap method to check for any missing days (method which could be deployed to all other API functions quite easily)\n",
    "            for missingDay in set(dayKeys.keys()).difference(dayDoneSet):\n",
    "                closedDayHours = {\n",
    "                    \"day\": dayKeys[missingDay],\n",
    "                    \"open\": False\n",
    "                }\n",
    "                heapq.heappush(daysHeap, [missingDay, closedDayHours])\n",
    "            if len(daysHeap) != 7:\n",
    "                return False\n",
    "            hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "            coords = res[index][\"geoLocation\"]\n",
    "            reweArray.append([(float(coords[\"latitude\"]),float(coords[\"longitude\"])), hoursArray])\n",
    "        page += 1\n",
    "    reweDB = pd.DataFrame(reweArray, columns=['Coordinates', 'OpeningHours'])\n",
    "    reweDB.to_csv(\"REWE.csv\")\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code for retrieving REWE's opening hours data from our database (via Haversine):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_rewe_data(lat, lng):\n",
    "    desiredCoords = [lat, lng]\n",
    "    reweDB = pd.read_csv(\"REWE.csv\", index_col=\"Unnamed: 0\", converters={'Coordinates': eval, 'OpeningHours' : eval})\n",
    "    reweDB[\"Distances\"] = haversine_vector([desiredCoords]*len(reweDB[\"Coordinates\"]), list(reweDB[\"Coordinates\"]))\n",
    "    closestStoreIndex = reweDB[\"Distances\"].idxmin()\n",
    "    return reweDB[\"OpeningHours\"][closestStoreIndex]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Netto's API\n",
    "**Note:** The function for this system also accepts lat and lng as length 1 arrays (simulated functioning overriding). In which case the storeDistance to the returned store is also returned ( in the form <code>[storeDisance, hoursArray]</code>), to allow the multi store get_netto_brands_data(lat, lng) function to call the different Netto functions and compare the distances of the stores returned.\n",
    "\n",
    "**Note:** Not to be confused with Netto Marken-Discount, which is an entirely different brand with 4100 stores vs 342.\n",
    "\n",
    "**Note:** This API returns **all** stores in Germany **only** upon calling. (For example does not return the 500+ stores located in Denmark)\n",
    "\n",
    "**Note:** To retrieve the opening hours of a Netto store's for a given pair of coordinates and you are unsure of which of the two Companies own it, it is recommand to use the get_netto_brands_data(lat, lng) function instead, as this will return the hours of whichever store is closest from the union of the two sets of branches. (for example if using Fuzzywuzzy string matches you could match on Netto rather than distinguish the two confusingly named store brands)\n",
    "\n",
    "\n",
    "## Base URL:\n",
    "https://netto.de/umbraco/api/StoresData/StoresV2\n",
    "\n",
    "\n",
    "## Sample API Call:\n",
    "https://netto.de/umbraco/api/StoresData/StoresV2\n",
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[0 … 99]\n",
    "    0: {,…}\n",
    "        address: {city: \"Demmin\", street: \"Jarmener Strasse 57-58\", zip: \"17109\", country: \"DE\", extra: null}\n",
    "            city: \"Demmin\"\n",
    "            country: \"DE\"\n",
    "            extra: null\n",
    "            street: \"Jarmener Strasse 57-58\"\n",
    "            zip: \"17109\"\n",
    "        attributes: {}\n",
    "        coordinates: [13.0500393, 53.90632]\n",
    "            0: 13.0500393\n",
    "            1: 53.90632\n",
    "        hours: [,…]\n",
    "            0: {close: \"2020-07-22T20:00:00\", open: \"2020-07-22T07:00:00\", closed: false, date: \"2020-07-22T00:00:00\",…}\n",
    "                close: \"2020-07-22T20:00:00\"\n",
    "                closed: false\n",
    "                customerFlow: [0, 0, 0, 0, 0, 0, 0, 0.19, 0.28, 0.38, 0.5, 0.39, 0.29, 0.28, 0.33, 0.33, 0.43, 0.4, 0.31, 0.26, 0, 0,…]\n",
    "                    0: 0\n",
    "                    1: 0\n",
    "                    2: 0\n",
    "                    3: 0\n",
    "                    4: 0\n",
    "                    5: 0\n",
    "                    6: 0\n",
    "                    7: 0.19\n",
    "                    8: 0.28\n",
    "                    9: 0.38\n",
    "                    10: 0.5\n",
    "                    11: 0.39\n",
    "                    12: 0.29\n",
    "                    13: 0.28\n",
    "                    14: 0.33\n",
    "                    15: 0.33\n",
    "                    16: 0.43\n",
    "                    17: 0.4\n",
    "                    18: 0.31\n",
    "                    19: 0.26\n",
    "                    20: 0\n",
    "                    21: 0\n",
    "                    22: 0\n",
    "                    23: 0\n",
    "                date: \"2020-07-22T00:00:00\"\n",
    "                open: \"2020-07-22T07:00:00\"\n",
    "                type: \"store\"\n",
    "            1: {close: \"2020-07-23T20:00:00\", open: \"2020-07-23T07:00:00\", closed: false, date: \"2020-07-23T00:00:00\",…}\n",
    "                close: \"2020-07-23T20:00:00\"\n",
    "                closed: false\n",
    "                customerFlow: [0, 0, 0, 0, 0, 0, 0, 0.19, 0.3, 0.54, 0.58, 0.44, 0.36, 0.2, 0.28, 0.31, 0.43, 0.43, 0.34, 0.23, 0, 0,…]\n",
    "                    0: 0\n",
    "                    1: 0\n",
    "                    2: 0\n",
    "                    3: 0\n",
    "                    4: 0\n",
    "                    5: 0\n",
    "                    6: 0\n",
    "                    7: 0.19\n",
    "                    8: 0.3\n",
    "                    9: 0.54\n",
    "                    10: 0.58\n",
    "                    11: 0.44\n",
    "                    12: 0.36\n",
    "                    13: 0.2\n",
    "                    14: 0.28\n",
    "                    15: 0.31\n",
    "                    16: 0.43\n",
    "                    17: 0.43\n",
    "                    18: 0.34\n",
    "                    19: 0.23\n",
    "                    20: 0\n",
    "                    21: 0\n",
    "                    22: 0\n",
    "                    23: 0\n",
    "                date: \"2020-07-23T00:00:00\"\n",
    "                open: \"2020-07-23T07:00:00\"\n",
    "                type: \"store\"\n",
    "            2: {close: \"2020-07-24T20:00:00\", open: \"2020-07-24T07:00:00\", closed: false, date: \"2020-07-24T00:00:00\",…}\n",
    "                close: \"2020-07-24T20:00:00\"\n",
    "                closed: false\n",
    "                customerFlow: [0, 0, 0, 0, 0, 0, 0, 0.18, 0.29, 0.51, 0.57, 0.54, 0.34, 0.33, 0.29, 0.45, 0.43, 0.43, 0.35, 0.28, 0,…]\n",
    "                    0: 0\n",
    "                    1: 0\n",
    "                    2: 0\n",
    "                    3: 0\n",
    "                    4: 0\n",
    "                    5: 0\n",
    "                    6: 0\n",
    "                    7: 0.18\n",
    "                    8: 0.29\n",
    "                    9: 0.51\n",
    "                    10: 0.57\n",
    "                    11: 0.54\n",
    "                    12: 0.34\n",
    "                    13: 0.33\n",
    "                    14: 0.29\n",
    "                    15: 0.45\n",
    "                    16: 0.43\n",
    "                    17: 0.43\n",
    "                    18: 0.35\n",
    "                    19: 0.28\n",
    "                    20: 0\n",
    "                    21: 0\n",
    "                    22: 0\n",
    "                    23: 0\n",
    "                date: \"2020-07-24T00:00:00\"\n",
    "                open: \"2020-07-24T07:00:00\"\n",
    "                type: \"store\"\n",
    "            3: {close: \"2020-07-25T20:00:00\", open: \"2020-07-25T07:00:00\", closed: false, date: \"2020-07-25T00:00:00\",…}\n",
    "                close: \"2020-07-25T20:00:00\"\n",
    "                closed: false\n",
    "                customerFlow: [0, 0, 0, 0, 0, 0, 0, 0.2, 0.25, 0.44, 0.55, 0.39, 0.32, 0.31, 0.35, 0.31, 0.36, 0.38, 0.4, 0.31, 0, 0,…]\n",
    "                    0: 0\n",
    "                    1: 0\n",
    "                    2: 0\n",
    "                    3: 0\n",
    "                    4: 0\n",
    "                    5: 0\n",
    "                    6: 0\n",
    "                    7: 0.2\n",
    "                    8: 0.25\n",
    "                    9: 0.44\n",
    "                    10: 0.55\n",
    "                    11: 0.39\n",
    "                    12: 0.32\n",
    "                    13: 0.31\n",
    "                    14: 0.35\n",
    "                    15: 0.31\n",
    "                    16: 0.36\n",
    "                    17: 0.38\n",
    "                    18: 0.4\n",
    "                    19: 0.31\n",
    "                    20: 0\n",
    "                    21: 0\n",
    "                    22: 0\n",
    "                    23: 0\n",
    "                date: \"2020-07-25T00:00:00\"\n",
    "                open: \"2020-07-25T07:00:00\"\n",
    "                type: \"store\"\n",
    "            4: {close: null, open: null, closed: true, date: \"2020-07-26T00:00:00\", type: \"store\",…}\n",
    "                close: null\n",
    "                closed: true\n",
    "                customerFlow: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
    "                    0: 0\n",
    "                    1: 0\n",
    "                    2: 0\n",
    "                    3: 0\n",
    "                    4: 0\n",
    "                    5: 0\n",
    "                    6: 0\n",
    "                    7: 0\n",
    "                    8: 0\n",
    "                    9: 0\n",
    "                    10: 0\n",
    "                    11: 0\n",
    "                    12: 0\n",
    "                    13: 0\n",
    "                    14: 0\n",
    "                    15: 0\n",
    "                    16: 0\n",
    "                    17: 0\n",
    "                    18: 0\n",
    "                    19: 0\n",
    "                    20: 0\n",
    "                    21: 0\n",
    "                    22: 0\n",
    "                    23: 0\n",
    "                date: \"2020-07-26T00:00:00\"\n",
    "                open: null\n",
    "                type: \"store\"\n",
    "            5: {close: \"2020-07-27T20:00:00\", open: \"2020-07-27T07:00:00\", closed: false, date: \"2020-07-27T00:00:00\",…}\n",
    "                close: \"2020-07-27T20:00:00\"\n",
    "                closed: false\n",
    "                customerFlow: [0, 0, 0, 0, 0, 0, 0, 0.31, 0.35, 0.51, 0.54, 0.42, 0.36, 0.33, 0.37, 0.49, 0.52, 0.48, 0.35, 0.23, 0,…]\n",
    "                    0: 0\n",
    "                    1: 0\n",
    "                    2: 0\n",
    "                    3: 0\n",
    "                    4: 0\n",
    "                    5: 0\n",
    "                    6: 0\n",
    "                    7: 0.31\n",
    "                    8: 0.35\n",
    "                    9: 0.51\n",
    "                    10: 0.54\n",
    "                    11: 0.42\n",
    "                    12: 0.36\n",
    "                    13: 0.33\n",
    "                    14: 0.37\n",
    "                    15: 0.49\n",
    "                    16: 0.52\n",
    "                    17: 0.48\n",
    "                    18: 0.35\n",
    "                    19: 0.23\n",
    "                    20: 0\n",
    "                    21: 0\n",
    "                    22: 0\n",
    "                    23: 0\n",
    "                date: \"2020-07-27T00:00:00\"\n",
    "                open: \"2020-07-27T07:00:00\"\n",
    "                type: \"store\"\n",
    "            6: {close: \"2020-07-28T20:00:00\", open: \"2020-07-28T07:00:00\", closed: false, date: \"2020-07-28T00:00:00\",…}\n",
    "                close: \"2020-07-28T20:00:00\"\n",
    "                closed: false\n",
    "                customerFlow: [0, 0, 0, 0, 0, 0, 0, 0.17, 0.32, 0.48, 0.54, 0.4, 0.36, 0.31, 0.41, 0.36, 0.44, 0.44, 0.36, 0.18, 0,…]\n",
    "                    0: 0\n",
    "                    1: 0\n",
    "                    2: 0\n",
    "                    3: 0\n",
    "                    4: 0\n",
    "                    5: 0\n",
    "                    6: 0\n",
    "                    7: 0.17\n",
    "                    8: 0.32\n",
    "                    9: 0.48\n",
    "                    10: 0.54\n",
    "                    11: 0.4\n",
    "                    12: 0.36\n",
    "                    13: 0.31\n",
    "                    14: 0.41\n",
    "                    15: 0.36\n",
    "                    16: 0.44\n",
    "                    17: 0.44\n",
    "                    18: 0.36\n",
    "                    19: 0.18\n",
    "                    20: 0\n",
    "                    21: 0\n",
    "                    22: 0\n",
    "                    23: 0\n",
    "                date: \"2020-07-28T00:00:00\"\n",
    "                open: \"2020-07-28T07:00:00\"\n",
    "                type: \"store\"\n",
    "        id: \"c9d0231d-d88f-4aa8-9e62-2272342d3e63\"\n",
    "        name: \"Netto Demmin\"\n",
    "        type: \"Point\"\n",
    "        url: \"/geschaefte/demmin/jarmener-strasse-57-58/c9d0231dd88f4aa89e622272342d3e63\"\n",
    "    1: {address: {city: \"Malchin\", street: \"Teichstrasse 8\", zip: \"17139\", country: \"DE\", extra: null},…}\n",
    "    2: {,…}\n",
    "    3: {address: {city: \"Dargun\", street: \"Schloßstraße 73\", zip: \"17159\", country: \"DE\", extra: null},…}\n",
    "    4: {address: {city: \"Berlin\", street: \"Pistoriusstrasse 13\", zip: \"13086\", country: \"DE\", extra: null},…}\n",
    "    5: {,…}\n",
    "    6: {,…}\n",
    "    7: {,…}\n",
    "    8: {,…}\n",
    "    9: {address: {city: \"Koserow\", street: \"Hauptstraße 87\", zip: \"17459\", country: \"DE\", extra: null},…}\n",
    "    10: {,…}\n",
    "    11: {,…}\n",
    "    12: {,…}\n",
    "    13: {address: {city: \"Mahlow\", street: \"Ibsenstrasse 69\", zip: \"15831\", country: \"DE\", extra: null},…}\n",
    "    14: {address: {city: \"Friedland\", street: \"Riemannstrasse 34\", zip: \"17098\", country: \"DE\", extra: null},…}\n",
    "    15: {,…}\n",
    "    16: {address: {city: \"Jarmen\", street: \"Müssentiner Weg 1\", zip: \"17126\", country: \"DE\", extra: null},…}\n",
    "    17: {,…}\n",
    "    18: {,…}\n",
    "    19: {,…}\n",
    "    20: {address: {city: \"Anklam\", street: \"Lübecker Straße 19\", zip: \"17389\", country: \"DE\", extra: null},…}\n",
    "    21: {,…}\n",
    "    22: {,…}\n",
    "    23: {address: {city: \"Penzlin\", street: \"Bahnhofplatz 5\", zip: \"17217\", country: \"DE\", extra: null},…}\n",
    "    24: {address: {city: \"Lübbenau\", street: \"Bahnhofstraße 21\", zip: \"03222\", country: \"DE\", extra: null},…}\n",
    "    25: {address: {city: \"Beeskow\", street: \"Luchstraße 32\", zip: \"15848\", country: \"DE\", extra: null},…}\n",
    "    26: {,…}\n",
    "    27: {,…}\n",
    "    28: {,…}\n",
    "    29: {address: {city: \"Berlin\", street: \"Kiefholzstraße 420\", zip: \"12435\", country: \"DE\", extra: null},…}\n",
    "    30: {,…}\n",
    "    31: {address: {city: \"Parchim\", street: \"Neuhofer Weiche 1\", zip: \"19370\", country: \"DE\", extra: null},…}\n",
    "    32: {address: {city: \"Strasburg\", street: \"Lindenstrasse 5\", zip: \"17335\", country: \"DE\", extra: null},…}\n",
    "    33: {address: {city: \"Parchim\", street: \"Westring 39\", zip: \"19370\", country: \"DE\", extra: null},…}\n",
    "    34: {,…}\n",
    "    35: {address: {city: \"Berlin\", street: \"Glienicker Weg 126\", zip: \"12489\", country: \"DE\", extra: null},…}\n",
    "    36: {,…}\n",
    "    37: {address: {city: \"Brüel\", street: \"Am Mühlenberg 1a\", zip: \"19412\", country: \"DE\", extra: null},…}\n",
    "    38: {,…}\n",
    "    39: {address: {city: \"Barth\", street: \"Hölzern-Kreuz-Weg 13b\", zip: \"18356\", country: \"DE\", extra: null},…}\n",
    "    40: {,…}\n",
    "    41: {address: {city: \"Rostock\", street: \"Zum Laakkanal 3\", zip: \"18109\", country: \"DE\", extra: null},…}\n",
    "    42: {,…}\n",
    "    43: {address: {city: \"Rostock\", street: \"Brahestraße 44\", zip: \"18059\", country: \"DE\", extra: null},…}\n",
    "    44: {address: {city: \"Rostock\", street: \"Ziolkowskistraße 1\", zip: \"18059\", country: \"DE\", extra: null},…}\n",
    "    45: {,…}\n",
    "    46: {,…}\n",
    "    47: {,…}\n",
    "    48: {address: {city: \"Cottbus\", street: \"Leuthener Straße 7\", zip: \"03050\", country: \"DE\", extra: null},…}\n",
    "    49: {address: {city: \"Velten\", street: \"Lindenstrasse 8\", zip: \"16727\", country: \"DE\", extra: null},…}\n",
    "    50: {,…}\n",
    "    51: {,…}\n",
    "    52: {address: {city: \"Eggersdorf\", street: \"Am Fuchsbau 3\", zip: \"15345\", country: \"DE\", extra: null},…}\n",
    "    53: {address: {city: \"Binz\", street: \"Proraer Chaussee 2\", zip: \"18609\", country: \"DE\", extra: null},…}\n",
    "    54: {,…}\n",
    "    55: {,…}\n",
    "    56: {,…}\n",
    "    57: {address: {city: \"Templin\", street: \"Lychener Straße 23a\", zip: \"17268\", country: \"DE\", extra: null},…}\n",
    "    58: {,…}\n",
    "    59: {address: {city: \"Wittenberge\", street: \"Turmstraße 13\", zip: \"19322\", country: \"DE\", extra: null},…}\n",
    "    60: {address: {city: \"Neuenhagen\", street: \"Lindenstrasse 61\", zip: \"15366\", country: \"DE\", extra: null},…}\n",
    "    61: {,…}\n",
    "    62: {address: {city: \"Berlin\", street: \"Alt Heiligensee 8\", zip: \"13503\", country: \"DE\", extra: null},…}\n",
    "    63: {address: {city: \"Berlin\", street: \"Am Gehrensee 2\", zip: \"13057\", country: \"DE\", extra: null},…}\n",
    "    64: {,…}\n",
    "    65: {address: {city: \"Berlin\", street: \"Ribnitzer Strasse 24\", zip: \"13051\", country: \"DE\", extra: null},…}\n",
    "    66: {address: {city: \"Berlin\", street: \"Eschengraben 47\", zip: \"13189\", country: \"DE\", extra: null},…}\n",
    "    67: {,…}\n",
    "    68: {,…}\n",
    "    69: {,…}\n",
    "    70: {,…}\n",
    "    71: {,…}\n",
    "    72: {,…}\n",
    "    73: {address: {city: \"Wittstock\", street: \"Polthierstraße 14\", zip: \"16909\", country: \"DE\", extra: null},…}\n",
    "    74: {address: {city: \"Salzwedel\", street: \"Bahnhofstraße 2\", zip: \"29410\", country: \"DE\", extra: null},…}\n",
    "    75: {,…}\n",
    "    76: {,…}\n",
    "    77: {,…}\n",
    "    78: {address: {city: \"Hagenow\", street: \"Goethestrasse 6\", zip: \"19230\", country: \"DE\", extra: null},…}\n",
    "    79: {,…}\n",
    "    80: {,…}\n",
    "    81: {,…}\n",
    "    82: {,…}\n",
    "    83: {address: {city: \"Pasewalk\", street: \"Bahnhofstraße 17\", zip: \"17309\", country: \"DE\", extra: null},…}\n",
    "    84: {,…}\n",
    "    85: {,…}\n",
    "    86: {,…}\n",
    "    87: {address: {city: \"Burg\", street: \"Friedensstrasse 49\", zip: \"39288\", country: \"DE\", extra: null},…}\n",
    "    88: {,…}\n",
    "    89: {,…}\n",
    "    90: {,…}\n",
    "    91: {,…}\n",
    "    92: {,…}\n",
    "    93: {address: {city: \"Nauen\", street: \"Dammstraße 7a\", zip: \"14641\", country: \"DE\", extra: null},…}\n",
    "    94: {,…}\n",
    "    95: {,…}\n",
    "    96: {address: {city: \"Löcknitz\", street: \"Chaussee Straße 37\", zip: \"17321\", country: \"DE\", extra: null},…}\n",
    "    97: {,…}\n",
    "    98: {,…}\n",
    "    99: {address: {city: \"Stendal\", street: \"Stadtseeallee 31\", zip: \"39576\", country: \"DE\", extra: null},…}\n",
    "    [100 … 199]\n",
    "    [200 … 299]\n",
    "    [300 … 341]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for setting up a database filled with all of Netto's opening hours, formatted to our liking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_up_netto_database():\n",
    "    API_URL = \"https://netto.de/umbraco/api/StoresData/StoresV2\"\n",
    "    nettoArray = []\n",
    "    dayKeys = { 0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday', 5 : 'Saturday', 6 : 'Sunday'}\n",
    "    try:\n",
    "        rq = requests.get(\"https://netto.de/umbraco/api/StoresData/StoresV2\")\n",
    "        res = rq.json()\n",
    "        if len(res) == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "    for index in range(len(res)):\n",
    "        dayDoneSet = set()\n",
    "        daysHeap = []\n",
    "        openingHours = res[index][\"hours\"]\n",
    "        for dayIndex in range(len(openingHours)):\n",
    "            #potential weakness if a bugged date is retrieved and the datetime parser raises an exception?\n",
    "            day = parser.parse(openingHours[dayIndex][\"date\"]).weekday()\n",
    "            if day not in dayDoneSet:\n",
    "                try:\n",
    "                    dayDoneSet.add(day)\n",
    "                    #keyHours processing code\n",
    "                    if openingHours[dayIndex][\"closed\"] == False:\n",
    "                        opens = parser.parse(openingHours[dayIndex][\"open\"]).strftime(\"%H:%M\")\n",
    "                        closes = parser.parse(openingHours[dayIndex][\"close\"]).strftime(\"%H:%M\")\n",
    "                        actualHours = {'open' : opens,\n",
    "                                'close' : closes}\n",
    "                        keyHours = {'day' : dayKeys[day],\n",
    "                        'open' : True,\n",
    "                        'hours' : [actualHours]\n",
    "                        }\n",
    "                        heapq.heappush(daysHeap, [day,keyHours])\n",
    "                    else:\n",
    "                        closedDayHours = {  'day' : dayKeys[day],\n",
    "                        'open' : False  }\n",
    "                        heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "                except:\n",
    "                    closedDayHours = {  'day' : dayKeys[day],\n",
    "                        'open' : False  }\n",
    "                    heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "        #Check for any missing days and ensure all days are in order for insertion into our database\n",
    "        for missingDay in set(dayKeys.keys()).difference(dayDoneSet):\n",
    "            closedDayHours = {\n",
    "                \"day\": dayKeys[missingDay],\n",
    "                \"open\": False\n",
    "            }\n",
    "            heapq.heappush(daysHeap, [missingDay, closedDayHours])\n",
    "        if len(daysHeap) != 7:\n",
    "            return False\n",
    "        hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "        coords = res[index][\"coordinates\"]\n",
    "        nettoArray.append([(float(coords[1]),float(coords[0])), hoursArray])\n",
    "    nettoDB = pd.DataFrame(nettoArray, columns=['Coordinates', 'OpeningHours'])\n",
    "    nettoDB.to_csv(\"Netto.csv\")\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Netto's opening hours data from our database (via Haversine):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_netto_data(lat, lng):\n",
    "    #pass lat and lng as arrays of length 1 if you want the storeDistance to be returned too\n",
    "    if isinstance(lat, list) :\n",
    "        returnDistanceToo = True\n",
    "        lat = lat[0]\n",
    "        lng = lng[0]\n",
    "    else:\n",
    "        returnDistanceToo = False\n",
    "    desiredCoords = [lat, lng]\n",
    "    nettoDB = pd.read_csv(\"Netto.csv\", index_col=\"Unnamed: 0\", converters={'Coordinates': eval, 'OpeningHours' : eval})\n",
    "    nettoDB[\"Distances\"] = haversine_vector([desiredCoords]*len(nettoDB[\"Coordinates\"]), list(nettoDB[\"Coordinates\"]))\n",
    "    closestStoreIndex = nettoDB[\"Distances\"].idxmin()\n",
    "    if returnDistanceToo:\n",
    "        return nettoDB[\"Distances\"][closestStoreIndex], nettoDB[\"OpeningHours\"][closestStoreIndex]\n",
    "    else:\n",
    "        return nettoDB[\"OpeningHours\"][closestStoreIndex]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Netto Marken-Discount's API (Germany)\n",
    "**Note:** The function for this system also accepts lat and lng as length 1 arrays (simulated functioning overriding). In which case the storeDistance to the returned store is also returned ( in the form <code>[storeDisance, hoursArray]</code>), to allow the multi store get_netto_brands_data(lat, lng) function to call the different Netto functions and compare the distances of the stores returned.\n",
    "\n",
    "\n",
    "**Note:** To retrieve the opening hours of a Netto store's for a given pair of coordinates and you are unsure of which of the two Companies own it, it is recommand to use the get_netto_brands_data(lat, lng) function instead, as this will return the hours of whichever store is closest from the union of the two sets of branches. (for example if using Fuzzywuzzy string matches you could match on Netto rather than distinguish the two confusingly named store brands)\n",
    "## Base URL:\n",
    "https://www.netto-online.de/INTERSHOP/web/WFS/Plus-NettoDE-Site/de_DE/-/EUR/ViewNettoStoreFinder-GetStoreItems\n",
    "## Parameters:\n",
    "These four parameters form the boundary (South, North, West, East) of the map viewport within which to return stores(website submits these via Form Data):\n",
    "* s = lower latitude boundary\n",
    "* n = upper latitude boundary\n",
    "* w = lower longitude boundary\n",
    "* e = upper longitude boundary\n",
    "\n",
    "**Note:** There is a maximum size allowed on the website.\n",
    "\n",
    "## Sample API Call:\n",
    "https://www.netto-online.de/INTERSHOP/web/WFS/Plus-NettoDE-Site/de_DE/-/EUR/ViewNettoStoreFinder-GetStoreItems?w=13.3&e=13.5&s=52.4&n=52.6\n",
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[{store_id: \"5544\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…},…]\n",
    "    0: {store_id: \"5544\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "        alternative_stores_id: []\n",
    "        city: \"Berlin-Treptow\"\n",
    "        coord_latitude: \"52.479225\"\n",
    "        coord_longitude: \"13.481276\"\n",
    "        has_bs: true\n",
    "        is_city: false\n",
    "        is_closed: false\n",
    "        is_delivery: \"0\"\n",
    "        is_pickup: false\n",
    "        is_user_store: false\n",
    "        meat_market_type: \"SELF-SERVICE\"\n",
    "        post_code: \"12435\"\n",
    "        region_id: \"37\"\n",
    "        state: \"Berlin\"\n",
    "        store_extra_info_long: \"\"\n",
    "        store_id: \"5544\"\n",
    "        store_name: \"Netto Filiale\"\n",
    "        store_opening: \"Mo.-Sa.: 7.00 - 22.00 Uhr<br />So.: geschlossen<br />\"\n",
    "        store_type_id: \"1\"\n",
    "        street: \"Dammweg 56\"\n",
    "    1: {store_id: \"5341\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    2: {store_id: \"5358\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    3: {store_id: \"5008\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    4: {store_id: \"5026\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    5: {store_id: \"5032\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    6: {store_id: \"5040\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    7: {store_id: \"5115\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    8: {store_id: \"5142\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    9: {store_id: \"5228\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    10: {store_id: \"5250\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    11: {store_id: \"7779\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    12: {store_id: \"7783\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: true,…}\n",
    "    13: {store_id: \"7785\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    14: {store_id: \"7788\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: true,…}\n",
    "    15: {store_id: \"7829\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    16: {store_id: \"7428\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    17: {store_id: \"7429\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    18: {store_id: \"7446\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    19: {store_id: \"7448\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    20: {store_id: \"7455\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}\n",
    "    21: {store_id: \"7456\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: true,…}\n",
    "    22: {store_id: \"7457\", is_user_store: false, region_id: \"37\", store_type_id: \"1\", is_city: false,…}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Netto Marken-Discount's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_netto_marken_discount_data(lat, lng):\n",
    "    #pass lat and lng as arrays of length 1 if you want the storeDistance to be returned too\n",
    "    if isinstance(lat, list) :\n",
    "        returnDistanceToo = True\n",
    "        lat = lat[0]\n",
    "        lng = lng[0]\n",
    "    else:\n",
    "        returnDistanceToo = False\n",
    "    API_URL = \"https://www.netto-online.de/INTERSHOP/web/WFS/Plus-NettoDE-Site/de_DE/-/EUR/ViewNettoStoreFinder-GetStoreItems\"\n",
    "        #The conversion rates below have been estimated through experimentation, and should approximately preserve radius (although the geometry will be a cross between a square and a circle)\n",
    "    params = {  's' : float(lat) - radius / 70,\n",
    "                'n' : float(lat) + radius / 70,\n",
    "                'w' : float(lng) - radius / 110,\n",
    "                'e' : float(lng) + radius / 110\n",
    "             }\n",
    "    rq = requests.get(API_URL, params=params)\n",
    "    if rq.status_code != 200:\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()\n",
    "        if len(res) == 0:\n",
    "            print(2)\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "    storeHeap = []\n",
    "    for index in range(len(res)):\n",
    "        storeLat, storeLng = float(res[index][\"coord_latitude\"]), float(res[index][\"coord_longitude\"])\n",
    "        #compute distance between the two points using the haversine function\n",
    "        storeDistance = haversine((lat, lng),(storeLat, storeLng))\n",
    "        heapq.heappush(storeHeap, [storeDistance,res[index]])\n",
    "    res = heapq.nsmallest(limit, storeHeap)\n",
    "\n",
    "    i = 0\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!!\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[0]['name'] or res[0]['other_name']:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    intermediaryDayKeys = { 'Mo.' : 0, 'Di.' : 1, 'Mi.' : 2, 'Do.' : 3, 'Fr.' : 4, 'Sa.' : 5, 'So.' : 6, None : None}\n",
    "    dayKeys = { 0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday', 5 : 'Saturday', 6 : 'Sunday'}\n",
    "    dayDoneSet = set()\n",
    "    daysHeap = []\n",
    "    storeDistance = res[i][0]\n",
    "    openingHours = res[i][1][\"store_opening\"]\n",
    "    dayRanges = openingHours.split(\"<br />\")[:-1]\n",
    "    for dayRange in range(len(dayRanges)):\n",
    "        daysAndTimes = dayRanges[dayRange].split(\":\")\n",
    "        days = daysAndTimes[0]\n",
    "        days = days.split(\"-\")\n",
    "        startDay = intermediaryDayKeys[days[0]]\n",
    "        if len(days) == 1:\n",
    "            endDay = startDay\n",
    "        elif len(days) == 2:\n",
    "            endDay = intermediaryDayKeys[days[1]]\n",
    "        else:\n",
    "            return False\n",
    "        #deal with wrap around windows e.g. sunday to tuesday\n",
    "        if startDay > endDay:\n",
    "            startDay += -7\n",
    "        times = daysAndTimes[1]\n",
    "        if \"geschlossen\" in times:\n",
    "            for day in range(startDay, endDay+1):\n",
    "                if day < 0:\n",
    "                    day += 7\n",
    "                #if statement to deal with incorrect responses where multiple ranges cover the same day (implemnted due to a bug in their system)\n",
    "                if day not in dayDoneSet:\n",
    "                    dayDoneSet.add(day)\n",
    "                    closedDayHours = {  'day' : dayKeys[day],\n",
    "                                'open' : False  }\n",
    "                    heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "        else:\n",
    "            try:\n",
    "                times = times.replace(\".\",\":\")\n",
    "                times = times.split()\n",
    "                if len(times) != 4:\n",
    "                    #means this method is non exhaustive (e.g. multiple time slots) and must be amended \n",
    "                    return False\n",
    "                opens = times[0]\n",
    "                closes = times[2]\n",
    "                if len(opens) == 4:\n",
    "                    opens = \"0\" + opens\n",
    "                if len(closes) == 4:\n",
    "                    closes = \"0\" + closes\n",
    "                if len(opens) != 5 or len(closes) != 5:\n",
    "                    #means this method is non exhaustive (e.g. doesnt catch store closure) and must be amended \n",
    "                    return False\n",
    "                actualHours = [{'open' : opens, 'close' : closes}]\n",
    "                for day in range(startDay, endDay+1):\n",
    "                    if day < 0:\n",
    "                        day += 7\n",
    "                    #if statement to deal with incorrect responses where multiple ranges cover the same day (implemnted due to a bug in their system)\n",
    "                    if day not in dayDoneSet:\n",
    "                        dayDoneSet.add(day)\n",
    "                        keyHours = {'day' : dayKeys[day],\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours\n",
    "                        }\n",
    "                        heapq.heappush(daysHeap, [day,keyHours])\n",
    "            except:\n",
    "                for day in range(startDay, endDay+1):\n",
    "                    if day < 0:\n",
    "                        day += 7\n",
    "                    #if statement to deal with incorrect responses where multiple ranges cover the same day (implemnted due to a bug in their system)\n",
    "                    if day not in dayDoneSet:\n",
    "                        dayDoneSet.add(day)\n",
    "                        closedDayHours = {  'day' : dayKeys[day],\n",
    "                                    'open' : False  }\n",
    "                        heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "    # simple heap method to check for any missing days (method which could be deployed to all other API functions quite easily)\n",
    "    for missingDay in set(dayKeys.keys()).difference(dayDoneSet):\n",
    "        closedDayHours = {\n",
    "            \"day\": dayKeys[missingDay],\n",
    "            \"open\": False\n",
    "        }\n",
    "        heapq.heappush(daysHeap, [missingDay, closedDayHours])\n",
    "    if len(daysHeap) != 7:\n",
    "        return False\n",
    "    hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "    if returnDistanceToo:\n",
    "        return storeDistance, hoursArray\n",
    "    else:\n",
    "        if storeDistance > radius:\n",
    "            return False\n",
    "        return hoursArray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving opening hours data of whichever store is closest out of the two Netto Brands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_netto_brands_data(lat, lng):\n",
    "    netto = get_netto_data([lat], [lng])\n",
    "    nettoMD = get_netto_marken_discount_data([lat], [lng])\n",
    "    if netto[0] < nettoMD[0]:\n",
    "        return netto[1]\n",
    "    else:\n",
    "        return nettoMD[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EDEKA's API (Germany)\n",
    "## Base URL:\n",
    "https://www.edeka.de/api/marketsearch/markets\n",
    "## Parameters:\n",
    "* searchstring = address to search around (the parameter the website uses)          \n",
    "**OR**\n",
    "* coordinates = hidden parameter which can be used to supply coordinates in the following form \"lat={Latitude}&lon={Longitude}\"\n",
    "\n",
    "## Sample API Call:\n",
    "https://www.edeka.de/api/marketsearch/markets?coordinates=lat%3D52.628684799999995%26lon%3D1.3369343999999999\n",
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{totalCount: 52, limit: 10, offset: 0,…}\n",
    "limit: 10\n",
    "markets: [{id: 8001594, name: \"EDEKA Friedrichstraße\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"},…},…]\n",
    "    0: {id: 8001594, name: \"EDEKA Friedrichstraße\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"},…}\n",
    "        businessHours: {monday: {weekday: \"MONDAY\", from: \"07:00\", to: \"22:00\"},…}\n",
    "            friday: {weekday: \"FRIDAY\", from: \"07:00\", to: \"22:00\"}\n",
    "                from: \"07:00\"\n",
    "                to: \"22:00\"\n",
    "                weekday: \"FRIDAY\"\n",
    "            monday: {weekday: \"MONDAY\", from: \"07:00\", to: \"22:00\"}\n",
    "                from: \"07:00\"\n",
    "                to: \"22:00\"\n",
    "                weekday: \"MONDAY\"\n",
    "            saturday: {weekday: \"SATURDAY\", from: \"07:00\", to: \"22:00\"}\n",
    "                from: \"07:00\"\n",
    "                to: \"22:00\"\n",
    "                weekday: \"SATURDAY\"\n",
    "            thursday: {weekday: \"THURSDAY\", from: \"07:00\", to: \"22:00\"}\n",
    "                from: \"07:00\"\n",
    "                to: \"22:00\"\n",
    "                weekday: \"THURSDAY\"\n",
    "            tuesday: {weekday: \"TUESDAY\", from: \"07:00\", to: \"22:00\"}\n",
    "                from: \"07:00\"\n",
    "                to: \"22:00\"\n",
    "                weekday: \"TUESDAY\"\n",
    "            wednesday: {weekday: \"WEDNESDAY\", from: \"07:00\", to: \"22:00\"}\n",
    "                from: \"07:00\"\n",
    "                to: \"22:00\"\n",
    "                weekday: \"WEDNESDAY\"\n",
    "        contact: {,…}\n",
    "            address: {city: {name: \"Berlin\", zipCode: \"10969\"}, street: \"Friedrichstr. 246\", federalState: \"Berlin\"}\n",
    "                city: {name: \"Berlin\", zipCode: \"10969\"}\n",
    "                    name: \"Berlin\"\n",
    "                    zipCode: \"10969\"\n",
    "                federalState: \"Berlin\"\n",
    "                street: \"Friedrichstr. 246\"\n",
    "            emailAddress: \"edeka.friedrichstrasse@minden.edeka.de\"\n",
    "            facsimileNumber: \"+493025295872\"\n",
    "            phoneNumber: \"+493025295870\"\n",
    "        coordinates: {lat: \"52.50003299999999\", lon: \"13.3913285\"}\n",
    "            lat: \"52.50003299999999\"\n",
    "            lon: \"13.3913285\"\n",
    "        distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"}\n",
    "            name: \"EDEKA\"\n",
    "            type: \"EDEKA\"\n",
    "        id: 8001594\n",
    "        imprint: {companyName: \"EDEKA Minden-Hannover Stiftung & Co. KG\",…}\n",
    "            address: {city: {name: \"Minden\", zipCode: \"32427\"}, street: \"Wittelsbacherallee 61\"}\n",
    "                city: {name: \"Minden\", zipCode: \"32427\"}\n",
    "                    name: \"Minden\"\n",
    "                    zipCode: \"32427\"\n",
    "                street: \"Wittelsbacherallee 61\"\n",
    "            companyName: \"EDEKA Minden-Hannover Stiftung & Co. KG\"\n",
    "        name: \"EDEKA Friedrichstraße\"\n",
    "        nationalOffers: true\n",
    "        services: [{id: \"AUSBILDENDER_BETRIEB\", name: \"Ausbildender Betrieb\", abbreviation: \"a\"},…]\n",
    "            0: {id: \"AUSBILDENDER_BETRIEB\", name: \"Ausbildender Betrieb\", abbreviation: \"a\"}\n",
    "                abbreviation: \"a\"\n",
    "                id: \"AUSBILDENDER_BETRIEB\"\n",
    "                name: \"Ausbildender Betrieb\"\n",
    "            1: {id: \"BACKSTATION\", name: \"Backstation\", abbreviation: \"bs\"}\n",
    "                abbreviation: \"bs\"\n",
    "                id: \"BACKSTATION\"\n",
    "                name: \"Backstation\"\n",
    "            2: {id: \"DEUTSCHLAND_CARD\", name: \"DeutschlandCard\", abbreviation: \"d\"}\n",
    "                abbreviation: \"d\"\n",
    "                id: \"DEUTSCHLAND_CARD\"\n",
    "                name: \"DeutschlandCard\"\n",
    "            3: {id: \"EDEKA_MOBIL\", name: \"EDEKA mobil\", abbreviation: \"em\"}\n",
    "                abbreviation: \"em\"\n",
    "                id: \"EDEKA_MOBIL\"\n",
    "                name: \"EDEKA mobil\"\n",
    "            4: {id: \"ERNAEHRUNGSSERVICE\", name: \"Ernährungsservice\", abbreviation: \"e\"}\n",
    "                abbreviation: \"e\"\n",
    "                id: \"ERNAEHRUNGSSERVICE\"\n",
    "                name: \"Ernährungsservice\"\n",
    "            5: {id: \"FLEISCH_WURSTTHEKE\", name: \"Fleisch-& Wursttheke\", abbreviation: \"fw\"}\n",
    "                abbreviation: \"fw\"\n",
    "                id: \"FLEISCH_WURSTTHEKE\"\n",
    "                name: \"Fleisch-& Wursttheke\"\n",
    "            6: {id: \"GLUTENFREI\", name: \"Glutenfrei\", abbreviation: \"gf\"}\n",
    "                abbreviation: \"gf\"\n",
    "                id: \"GLUTENFREI\"\n",
    "                name: \"Glutenfrei\"\n",
    "            7: {id: \"HANDYAUFLADUNG\", name: \"Handy-Aufladung\", abbreviation: \"ha\"}\n",
    "                abbreviation: \"ha\"\n",
    "                id: \"HANDYAUFLADUNG\"\n",
    "                name: \"Handy-Aufladung\"\n",
    "            8: {id: \"HAUSHALTSWAREN\", name: \"Haushaltswaren\", abbreviation: \"nf\"}\n",
    "                abbreviation: \"nf\"\n",
    "                id: \"HAUSHALTSWAREN\"\n",
    "                name: \"Haushaltswaren\"\n",
    "            9: {id: \"KREDITKARTE_AKZEPTIERT\", name: \"Kreditkarte akzeptiert\", abbreviation: \"ka\"}\n",
    "                abbreviation: \"ka\"\n",
    "                id: \"KREDITKARTE_AKZEPTIERT\"\n",
    "                name: \"Kreditkarte akzeptiert\"\n",
    "            10: {id: \"KAESETHEKE\", name: \"Käsetheke\", abbreviation: \"k\"}\n",
    "                abbreviation: \"k\"\n",
    "                id: \"KAESETHEKE\"\n",
    "                name: \"Käsetheke\"\n",
    "            11: {id: \"LAKTOSEFREI\", name: \"Laktosefrei\", abbreviation: \"lf\"}\n",
    "                abbreviation: \"lf\"\n",
    "                id: \"LAKTOSEFREI\"\n",
    "                name: \"Laktosefrei\"\n",
    "            12: {id: \"MOBILE_HANDY_COUPONS\", name: \"Mobile Handy-Coupons\", abbreviation: \"mhc\"}\n",
    "                abbreviation: \"mhc\"\n",
    "                id: \"MOBILE_HANDY_COUPONS\"\n",
    "                name: \"Mobile Handy-Coupons\"\n",
    "            13: {id: \"MOBILES_BEZAHLEN\", name: \"Mobiles Bezahlen per Handy\", abbreviation: \"mb\"}\n",
    "                abbreviation: \"mb\"\n",
    "                id: \"MOBILES_BEZAHLEN\"\n",
    "                name: \"Mobiles Bezahlen per Handy\"\n",
    "            14: {id: \"VEGAN\", name: \"Vegan\", abbreviation: \"vega\"}\n",
    "                abbreviation: \"vega\"\n",
    "                id: \"VEGAN\"\n",
    "                name: \"Vegan\"\n",
    "            15: {id: \"VEGETARISCH\", name: \"Vegetarisch\", abbreviation: \"vege\"}\n",
    "                abbreviation: \"vege\"\n",
    "                id: \"VEGETARISCH\"\n",
    "                name: \"Vegetarisch\"\n",
    "        url: \"https://www.edeka.de/eh/minden-hannover/edeka-friedrichstraße-friedrichstr.-246/index.jsp\"\n",
    "        validFrom: \"09.06.2017\"\n",
    "    1: {id: 8001540, name: \"EDEKA Ritterstraße\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"},…}\n",
    "    2: {id: 10001148, name: \"EDEKA Stresemannstr.\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"},…}\n",
    "    3: {id: 128443, name: \"EDEKA Ungefroren\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"},…}\n",
    "    4: {id: 10001525, name: \"EDEKA Berlin Yorckstraße\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"},…}\n",
    "    5: {id: 8001596, name: \"EDEKA Leipziger Platz\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"},…}\n",
    "    6: {id: 8001532, name: \"EDEKA Annenstraße\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"},…}\n",
    "    7: {id: 8001539, name: \"EDEKA Bergmannstraße\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"},…}\n",
    "    8: {id: 20666, name: \"EDEKA Siebert\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"}, contact: {,…},…}\n",
    "    9: {id: 8000841, name: \"EDEKA Brehm\", distributionChannel: {name: \"EDEKA\", type: \"EDEKA\"}, contact: {,…},…}\n",
    "offset: 0\n",
    "totalCount: 52\n",
    "_links: {self: {rel: \"self\",…}, next: {rel: \"next\",…}}\n",
    "next: {rel: \"next\",…}\n",
    "href: \"/api/marketsearch/markets?coordinates=lat%3D52.503705%26lon%3D13.395646&page=1&size=10\"\n",
    "rel: \"next\"\n",
    "self: {rel: \"self\",…}\n",
    "href: \"/api/marketsearch/markets?coordinates=lat%3D52.503705%26lon%3D13.395646&page=0&size=10\"\n",
    "rel: \"self\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving EDEKA's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_edeka_data(lat, lng):\n",
    "    API_URL = \"https://www.edeka.de/api/marketsearch/markets\"\n",
    "    params = {  'coordinates' : \"lat={0}&lon={1}\".format(lat, lng)}\n",
    "    rq = requests.get(API_URL, params=params)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.json())\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()\n",
    "        if res[\"totalCount\"] == 0 or len(res['markets']) == 0:\n",
    "            return False\n",
    "        res = res[\"markets\"]\n",
    "    except:\n",
    "        return False\n",
    "    i = 0\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (e.g. res[0][\"name\"])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[i][\"name\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break \n",
    "    storeLat, storeLng = float(res[i][\"coordinates\"][\"lat\"]), float(res[i][\"coordinates\"][\"lon\"])\n",
    "    #compute distance between the two points using the haversine function\n",
    "    storeDistance = haversine((lat, lng),(storeLat, storeLng))\n",
    "    if storeDistance > radius:\n",
    "        return False\n",
    "    openingHours = res[i][\"businessHours\"]\n",
    "    dayKeys = { 'monday' : 'Monday', 'tuesday' : 'Tuesday', 'wednesday' : 'Wednesday', 'thursday' : 'Thursday', 'friday' : 'Friday', 'saturday' : 'Saturday', 'sunday' : 'Sunday'}\n",
    "    hoursArray = []\n",
    "    for key in ['monday','tuesday','wednesday','thursday','friday','saturday','sunday']:\n",
    "        try:\n",
    "            actualHours = { 'open' : openingHours[key]['from'],\n",
    "                            'close' : openingHours[key]['to']}\n",
    "            keyHours = {'day' : dayKeys[key],\n",
    "                    'open' : True,\n",
    "                    'hours' : [actualHours]\n",
    "                    }\n",
    "        except:\n",
    "            keyHours = {'day' : dayKeys[key],\n",
    "                    'open' : False}\n",
    "        hoursArray.append(keyHours)\n",
    "    return hoursArray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sainsbury's API:\n",
    "## Documentation:\n",
    "### Base URL:\n",
    "https://stores.sainsburys.co.uk/api/v1/stores/\n",
    "### Parameters:\n",
    "* fields: Not sure what that means. Default value: <code>fields=slfe-list-2.21</code>\n",
    "* api_client_id: Not sure either. Default value: <code>api_client_id=slfe</code>\n",
    "* lat: Latitude of search area. Example: <code>lat=51.5073509</code>\n",
    "* lon: Longitude of search area. Example: <code>lon=-0.1277583</code>\n",
    "* limit: Stores displayed per page. Default value: <code>limit=25</code>\n",
    "* store_type: Type of stores displayed. Can contain *main* for superstores and/or *local* for local stores. Example: <code>stores_type=main,local</code>\n",
    "* sort: How results are sorted. Only found *by_distance* so far. Default value: <code>sort=by_distance</code>\n",
    "* within: Max distance for which results should be shown, Measured in Miles (**tested**). Seems to take a value between 0 and 500. Example: <code>within=15</code>\n",
    "* page: Page of paginated results. Example: <code>page=1</code>\n",
    "\n",
    "### Sample API Call:\n",
    "https://stores.sainsburys.co.uk/api/v1/stores/?fields=slfe-list-2.21&api_client_id=slfe&lat=52.6476395&lon=1.2932064&limit=25&store_type=main%2Clocal&sort=by_distance&within=15&page=1\n",
    "\n",
    "### Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "page_meta: {limit: 25, offset: 0, total: 6}\n",
    "results: [{additional_data: {main_store_branded_as_local: null, restaurant_branded_as_fresh_kitchen: null},…},…]\n",
    "    0: {additional_data: {main_store_branded_as_local: null, restaurant_branded_as_fresh_kitchen: null},…}\n",
    "        additional_data: {main_store_branded_as_local: null, restaurant_branded_as_fresh_kitchen: null}\n",
    "            main_store_branded_as_local: null\n",
    "            restaurant_branded_as_fresh_kitchen: null\n",
    "        closed: null\n",
    "        code: \"4359\"\n",
    "        contact: {address1: \"9 - 13 Stephens Street\", address2: \"\", city: \"Norwich\", country: \"England\",…}\n",
    "            address1: \"9 - 13 Stephens Street\"\n",
    "            address2: \"\"\n",
    "            city: \"Norwich\"\n",
    "            country: \"England\"\n",
    "            country_code: \"GB\"\n",
    "            country_name: \"United Kingdom\"\n",
    "            county: \"\"\n",
    "            fax_number: \"\"\n",
    "            manager: \"Vicky Hampson\"\n",
    "            post_code: \"NR1 3QN\"\n",
    "            state: \"\"\n",
    "            telephone: \"01603764089\"\n",
    "        distance: 1.5303873646684123\n",
    "        district_code: \"4359\"\n",
    "        exception_times: []\n",
    "        location: {lat: \"52.62549\", lon: \"1.29324\"}\n",
    "            lat: \"52.62549\"\n",
    "            lon: \"1.29324\"\n",
    "        name: \"Nwch St Stpns St Loc\"\n",
    "        open: {current_date: \"2020-07-18\", current_time: \"13:10\", is_open: true, open_until: \"23:00\",…}\n",
    "            current_date: \"2020-07-18\"\n",
    "            current_time: \"13:10\"\n",
    "            is_open: true\n",
    "            open_until: \"23:00\"\n",
    "            today: {24hrs: false, end_time: \"23:00\", exception: false, start_time: \"07:00\",…}\n",
    "                24hrs: false\n",
    "                end_time: \"23:00\"\n",
    "                exception: false\n",
    "                start_time: \"07:00\"\n",
    "                times: [{end_time: \"23:00\", start_time: \"07:00\"}]\n",
    "                    0: {end_time: \"23:00\", start_time: \"07:00\"}\n",
    "                        end_time: \"23:00\"\n",
    "                        start_time: \"07:00\"\n",
    "            tomorrow: {24hrs: false, end_time: \"23:00\", exception: false, start_time: \"07:00\",…}\n",
    "                24hrs: false\n",
    "                end_time: \"23:00\"\n",
    "                exception: false\n",
    "                start_time: \"07:00\"\n",
    "                times: [{end_time: \"23:00\", start_time: \"07:00\"}]\n",
    "                    0: {end_time: \"23:00\", start_time: \"07:00\"}\n",
    "                        end_time: \"23:00\"\n",
    "                        start_time: \"07:00\"\n",
    "        opening_times: [{day: 0, end_time: \"23:00\", start_time: \"07:00\", times: [{end_time: \"23:00\", start_time: \"07:00\"}]},…]\n",
    "            0: {day: 0, end_time: \"23:00\", start_time: \"07:00\", times: [{end_time: \"23:00\", start_time: \"07:00\"}]}\n",
    "                day: 0\n",
    "                end_time: \"23:00\"\n",
    "                start_time: \"07:00\"\n",
    "                times: [{end_time: \"23:00\", start_time: \"07:00\"}]\n",
    "                    0: {end_time: \"23:00\", start_time: \"07:00\"}\n",
    "                        end_time: \"23:00\"\n",
    "                        start_time: \"07:00\"\n",
    "            1: {day: 1, end_time: \"23:00\", start_time: \"07:00\", times: [{end_time: \"23:00\", start_time: \"07:00\"}]}\n",
    "                day: 1\n",
    "                end_time: \"23:00\"\n",
    "                start_time: \"07:00\"\n",
    "                times: [{end_time: \"23:00\", start_time: \"07:00\"}]\n",
    "                    0: {end_time: \"23:00\", start_time: \"07:00\"}\n",
    "                        end_time: \"23:00\"\n",
    "                        start_time: \"07:00\"\n",
    "            2: {day: 2, end_time: \"23:00\", start_time: \"07:00\", times: [{end_time: \"23:00\", start_time: \"07:00\"}]}\n",
    "                day: 2\n",
    "                end_time: \"23:00\"\n",
    "                start_time: \"07:00\"\n",
    "                times: [{end_time: \"23:00\", start_time: \"07:00\"}]\n",
    "                    0: {end_time: \"23:00\", start_time: \"07:00\"}\n",
    "                        end_time: \"23:00\"\n",
    "                        start_time: \"07:00\"\n",
    "            3: {day: 3, end_time: \"23:00\", start_time: \"07:00\", times: [{end_time: \"23:00\", start_time: \"07:00\"}]}\n",
    "                day: 3\n",
    "                end_time: \"23:00\"\n",
    "                start_time: \"07:00\"\n",
    "                times: [{end_time: \"23:00\", start_time: \"07:00\"}]\n",
    "                    0: {end_time: \"23:00\", start_time: \"07:00\"}\n",
    "                        end_time: \"23:00\"\n",
    "                        start_time: \"07:00\"\n",
    "            4: {day: 4, end_time: \"23:00\", start_time: \"07:00\", times: [{end_time: \"23:00\", start_time: \"07:00\"}]}\n",
    "                day: 4\n",
    "                end_time: \"23:00\"\n",
    "                start_time: \"07:00\"\n",
    "                times: [{end_time: \"23:00\", start_time: \"07:00\"}]\n",
    "                    0: {end_time: \"23:00\", start_time: \"07:00\"}\n",
    "                        end_time: \"23:00\"\n",
    "                        start_time: \"07:00\"\n",
    "            5: {day: 5, end_time: \"23:00\", start_time: \"07:00\", times: [{end_time: \"23:00\", start_time: \"07:00\"}]}\n",
    "                day: 5\n",
    "                end_time: \"23:00\"\n",
    "                start_time: \"07:00\"\n",
    "                times: [{end_time: \"23:00\", start_time: \"07:00\"}]\n",
    "                    0: {end_time: \"23:00\", start_time: \"07:00\"}\n",
    "                        end_time: \"23:00\"\n",
    "                        start_time: \"07:00\"\n",
    "            6: {day: 6, end_time: \"23:00\", start_time: \"07:00\", times: [{end_time: \"23:00\", start_time: \"07:00\"}]}\n",
    "                day: 6\n",
    "                end_time: \"23:00\"\n",
    "                start_time: \"07:00\"\n",
    "                times: [{end_time: \"23:00\", start_time: \"07:00\"}]\n",
    "                    0: {end_time: \"23:00\", start_time: \"07:00\"}\n",
    "                        end_time: \"23:00\"\n",
    "                        start_time: \"07:00\"\n",
    "        opening_times_message: null\n",
    "        originator_id: null\n",
    "        other_name: \"Norwich St Stephens Street Local\"\n",
    "        store_type: \"local\"\n",
    "    1: {additional_data: {main_store_branded_as_local: null, restaurant_branded_as_fresh_kitchen: null},…}\n",
    "    2: {additional_data: {main_store_branded_as_local: null, restaurant_branded_as_fresh_kitchen: null},…}\n",
    "    3: {additional_data: {main_store_branded_as_local: null, restaurant_branded_as_fresh_kitchen: null},…}\n",
    "    4: {additional_data: {main_store_branded_as_local: null, restaurant_branded_as_fresh_kitchen: null},…}\n",
    "    5: {additional_data: {main_store_branded_as_local: null, restaurant_branded_as_fresh_kitchen: null},…}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Sainsbury's opening times data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_sainsburys_data(lat,lng):\n",
    "    API_URL = \"https://stores.sainsburys.co.uk/api/v1/stores/\"\n",
    "    params = {\n",
    "        'fields': 'slfe-list-2.21',\n",
    "        'api_client_id': 'slfe',\n",
    "        'store_type': 'main,local',\n",
    "        'sort': 'by_distance',\n",
    "        'within': radius,\n",
    "        'limit': limit,\n",
    "        'page': '1',\n",
    "        'lat': lat,\n",
    "        'lon': lng,\n",
    "    }\n",
    "    rq = requests.get(API_URL, params=params)\n",
    "    if rq.status_code != 200:\n",
    "        # print(rq.json())\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()\n",
    "        if res[\"page_meta\"][\"total\"] == 0:\n",
    "            return False\n",
    "        res = res[\"results\"]\n",
    "    except:\n",
    "        return False\n",
    "    dayKeys = { 0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday', 5 : 'Saturday', 6 : 'Sunday'}\n",
    "    i=0\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (res[0]['name'] or res[0]['other_name'])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[0]['name'] or res[0]['other_name']:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    if res[i][\"distance\"] > radius * 0.621371:\n",
    "        return False\n",
    "    openingHours = res[i]['opening_times']\n",
    "    dayDoneSet = set()\n",
    "    daysHeap = []\n",
    "    for index in range(len(openingHours)):\n",
    "        key = openingHours[index]['day']\n",
    "        #if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "        if key not in dayDoneSet:\n",
    "            dayDoneSet.add(key)\n",
    "            #keyHours processing code\n",
    "            actualHours = []\n",
    "            for timeSlotNumber in range(len(openingHours[index]['times'])):\n",
    "                try:\n",
    "                    opens = openingHours[index]['times'][timeSlotNumber]['start_time']\n",
    "                    closes = openingHours[index]['times'][timeSlotNumber]['end_time']\n",
    "                    actualHoursDict = { 'open' : opens,\n",
    "                                        'close' : closes}\n",
    "                    actualHours.append(actualHoursDict)\n",
    "                except:\n",
    "                    #If dictionary keys start_time or end_time dont exist, assume store is closed on that day\n",
    "                    closedDayHours = {  'day' : dayKeys[key],\n",
    "                                    'open' : False  }\n",
    "                    heapq.heappush(daysHeap, [key,closedDayHours])\n",
    "                    break\n",
    "            keyHours = {'day' : dayKeys[key],\n",
    "                    'open' : True,\n",
    "                    'hours' : actualHours\n",
    "                    }\n",
    "            heapq.heappush(daysHeap, [key,keyHours])\n",
    "    #Check for any missing days and ensure all days are in order for insertion into our database\n",
    "    for missingDay in set(dayKeys.keys()).difference(dayDoneSet):\n",
    "        closedDayHours = {\n",
    "            \"day\": dayKeys[missingDay],\n",
    "            \"open\": False\n",
    "        }\n",
    "        heapq.heappush(daysHeap, [missingDay, closedDayHours])\n",
    "    if len(daysHeap) != 7:\n",
    "        return False\n",
    "    hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "    return hoursArray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ASDA's API\n",
    "It is of note that ASDA's system returns a potentially extrememely large quanitity of different qualitative data regarding the store in question, although not all of this data is returned for each store's \"response\" field. Although, the full schema of possible information for the API to return can be found under \"schema\" of the API call.\n",
    "\n",
    "Notes:\n",
    "Has Click and Collect info.\n",
    "Diesel price is potentially something that could be displayed (especially if this is displayed on stores like BP Garage etc..).\n",
    "## Base URL:\n",
    "https://storelocator.asda.com/index.html\n",
    "## Parameters:\n",
    "* q = appears to send off to an external (Google?) API in order to translate whatever form of address you input. Therefore latitude and longitude may be used as so: \"latitude,longitude\"\n",
    "* qn = generally seems to mimic what is input into q when used through the website, however when \"My Location\" button is used, q = lat,lon and qn = \"My Location\"    **(optional)**\n",
    "* l = language (\"en\" for english)  **(optional)**\n",
    "\n",
    "## Headers:\n",
    "* accept = \"application/json\" (otherwise will return html (?))\n",
    "\n",
    "## Sample API Call:\n",
    "https://storelocator.asda.com/index.html?q=52.635875,%201.306051&qp=52.635875,%201.306051&l=en\n",
    "## Sample Response:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseUrl: \"\"\n",
    "businessId: 2479739\n",
    "errors: null\n",
    "featureId: \"Search\"\n",
    "isHttps: true\n",
    "isPrimaryLocale: true\n",
    "isStaging: false\n",
    "locale: \"en\"\n",
    "queryParams: {l: [\"en\"], q: [\"52.635875, 1.306051\"], qp: [\"52.635875, 1.306051\"]}\n",
    "response: {count: 2, entities: [{derivedData: {address: {countryName: \"United Kingdom\"}, uber: {,…}},…},…],…}\n",
    "    count: 2\n",
    "    entities: [{derivedData: {address: {countryName: \"United Kingdom\"}, uber: {,…}},…},…]\n",
    "        0: {derivedData: {address: {countryName: \"United Kingdom\"}, uber: {,…}},…}\n",
    "            derivedData: {address: {countryName: \"United Kingdom\"}, uber: {,…}}\n",
    "                address: {countryName: \"United Kingdom\"}\n",
    "                    countryName: \"United Kingdom\"\n",
    "                uber: {,…}\n",
    "                    url: \"https://m.uber.com/ul/?action=setPickup&client_id=KXQcwoj2Zb8ymDzKgVgbIaDE5iAE_TAj&dropoff%5Bformatted_address%5D=Hall%20Road%2C%20null%2C%20Norwich%2C%20null&dropoff%5Blatitude%5D=52.60555&dropoff%5Blongitude%5D=1.28634&dropoff%5Bnickname%5D=Asda%20Norwich%20Hall%20Road%20Superstore&pickup=my_location\"\n",
    "                    wrappedUrl: \"http://a.gotoloc.com/uber/F3EkTqKGX2A\"\n",
    "            distance: {distanceKilometers: 3.4855743361404476, distanceMiles: 2.1658354808794438, id: \"4383\"}\n",
    "                distanceKilometers: 3.4855743361404476\n",
    "                distanceMiles: 2.1658354808794438\n",
    "                id: \"4383\"\n",
    "            listings: {,…}\n",
    "                googleMyBusiness: {placeId: \"ChIJKZs0vAHk2UcRkaLqm7RF3hk\", url: \"https://maps.google.com/maps?cid=1864003937790239377\"}\n",
    "                    placeId: \"ChIJKZs0vAHk2UcRkaLqm7RF3hk\"\n",
    "                    url: \"https://maps.google.com/maps?cid=1864003937790239377\"\n",
    "            profile: {,…}\n",
    "                address: {city: \"Norwich\", countryCode: \"GB\", extraDescription: null, line1: \"Hall Road\", line2: null,…}\n",
    "                    city: \"Norwich\"\n",
    "                    countryCode: \"GB\"\n",
    "                    extraDescription: null\n",
    "                    line1: \"Hall Road\"\n",
    "                    line2: null\n",
    "                    line3: null\n",
    "                    postalCode: \"NR4 6DP\"\n",
    "                    region: null\n",
    "                    sublocality: null\n",
    "                c_24HourStore: false\n",
    "                c_24HourStoreWithSundayClosure: false\n",
    "                c_aboutSectionCTA: {text: \"ASDA.com\", url: \"https://www.asda.com/\"}\n",
    "                    text: \"ASDA.com\"\n",
    "                    url: \"https://www.asda.com/\"\n",
    "                c_aboutSectionDescription: \"Asda Norwich Hall Road is part of the Asda group; one of Britain’s leading grocery retailers. Asda offers great prices and quality products helping customers save money & live better. Visit us in-store and get groceries delivered to anywhere in the UK, or collect from your local store or petrol station at a convenient time for you.\"\n",
    "                c_aboutSectionTitle: \"About ASDA - Norwich Hall Road\"\n",
    "                c_alertBannerBackgroundColor: \"#5e9b1a\"\n",
    "                c_alertBannerDatePosted: {day: 6, month: 5, year: 2020}\n",
    "                    day: 6\n",
    "                    month: 5\n",
    "                    year: 2020\n",
    "                c_alertBannerText: \"We’re so grateful for the amazing work of our NHS workers, Care Home Workers and Carers, which is why we continue to offer priority access to our stores for these customers <b>between 8am and 9am each Monday, Wednesday and Friday morning—as well as an exclusive hour of browsing on Sunday morning before our stores open.</b>↵↵Please try and avoid coming to our stores at these times to let these vital workers get what they need, and we ask these customers to bring their work ID with them to help our colleagues spot you.\"\n",
    "                c_alertBannerTitle: \"<b>Priority Shopping</b>\"\n",
    "                c_appleName: \"Asda Superstore\"\n",
    "                c_asdaLiving: false\n",
    "                c_babyChanging: true\n",
    "                c_bakery: true\n",
    "                c_bakeryCounter: false\n",
    "                c_barclaysBank: false\n",
    "                c_butcherCounter: false\n",
    "                c_cafe: true\n",
    "                c_carPark: true\n",
    "                c_carparkType: [\"FREE\"]\n",
    "                    0: \"FREE\"\n",
    "                c_cashMachine: true\n",
    "                c_changingPlacesFacility: false\n",
    "                c_clickAndCollect: true\n",
    "                c_clickAndCollectCardTitle: \"Click & Collect\"\n",
    "                c_clickAndCollectLockersText: \"External Collection Point\"\n",
    "                c_clickAndCollectText: \"Click to book your groceries collection at a time to suit you\"\n",
    "                c_communityChamionNominationText1: \"How to nominate a group for Green Token Giving\"\n",
    "                c_communityChamionNominationText2: \"Email or speak to your Community Champion in store to nominate a cause for Green token giving scheme.\"\n",
    "                c_communityChampion: true\n",
    "                c_communityChampionContactText: \"For further information contact your community champion at Community_SouthNorwich@asda.co.uk\"\n",
    "                c_communityChampionNominationLink: \"http://www.asda.com\"\n",
    "                c_communityChampionsCTA: {text: \"Read More\", url: \"https://www.asda.com/community-champions\"}\n",
    "                    text: \"Read More\"\n",
    "                    url: \"https://www.asda.com/community-champions\"\n",
    "                c_communityChampionsImage: {image: {height: 119,…}}\n",
    "                    image: {height: 119,…}\n",
    "                        height: 119\n",
    "                    thumbnails: [{height: 48, url: \"//dynl.mktgcdn.com/p/_d5C6bX8hYCMiZGCbHmiVVqBslZRmeue99ETKUi91tI/196x48.png\",…}]\n",
    "                        0: {height: 48, url: \"//dynl.mktgcdn.com/p/_d5C6bX8hYCMiZGCbHmiVVqBslZRmeue99ETKUi91tI/196x48.png\",…}\n",
    "                            height: 48\n",
    "                            url: \"//dynl.mktgcdn.com/p/_d5C6bX8hYCMiZGCbHmiVVqBslZRmeue99ETKUi91tI/196x48.png\"\n",
    "                            width: 196\n",
    "                    url: \"//dynl.mktgcdn.com/p/_d5C6bX8hYCMiZGCbHmiVVqBslZRmeue99ETKUi91tI/480x119.png\"\n",
    "                    width: 480\n",
    "                c_communityChampionsText: {,…}\n",
    "                    text(list): [\"Local grants\", \"Green token giving\", \"Product donation\", \"Using Asda facilities\", \"Volunteering\"]\n",
    "                        0: \"Local grants\"\n",
    "                        1: \"Green token giving\"\n",
    "                        2: \"Product donation\"\n",
    "                        3: \"Using Asda facilities\"\n",
    "                        4: \"Volunteering\"\n",
    "                    text(multi-line): \"Are you a charity or not for profit organisation looking for support from Asda? Then contact your Local Community Champion to discuss:\"\n",
    "                c_communityRoom: false\n",
    "                c_contactSectionTitle: \"Contact\"\n",
    "                c_costaCoffee: true\n",
    "                c_customerWC: true\n",
    "                c_deliCounter: false\n",
    "                c_directoryCounty: \"East Of England\"\n",
    "                c_disabledFacilities: true\n",
    "                c_division: \"South\"\n",
    "                c_electricCarCharging: true\n",
    "                c_electricVehicleChargingPoint: true\n",
    "                c_electricalGoods: true\n",
    "                c_ethnicFoods: false\n",
    "                c_expressDiner: true\n",
    "                c_expressPizzaCounter: false\n",
    "                c_expressRotisserie: false\n",
    "                c_facilitiesList: [\"Car Park\", \"Cash Machine\", \"Customer WC\", \"Disabled Facilities\", \"Baby Changing\",…]\n",
    "                    0: \"Car Park\"\n",
    "                    1: \"Cash Machine\"\n",
    "                    2: \"Customer WC\"\n",
    "                    3: \"Disabled Facilities\"\n",
    "                    4: \"Baby Changing\"\n",
    "                    5: \"Electric Vehicle Charging Point\"\n",
    "                    6: \"Helium Balloons\"\n",
    "                c_facilitiesTitle: \"Facilities\"\n",
    "                c_featuredServiceCardCafe: {hours: {holidayHours: [],…}, photo: {image: {height: 768,…}},…}\n",
    "                    hours: {holidayHours: [],…}\n",
    "                        holidayHours: []\n",
    "                        normalHours: [{day: \"MONDAY\", intervals: [], isClosed: true}, {day: \"TUESDAY\", intervals: [], isClosed: true},…]\n",
    "                            0: {day: \"MONDAY\", intervals: [], isClosed: true}\n",
    "                                day: \"MONDAY\"\n",
    "                                intervals: []\n",
    "                                isClosed: true\n",
    "                            1: {day: \"TUESDAY\", intervals: [], isClosed: true}\n",
    "                                day: \"TUESDAY\"\n",
    "                                intervals: []\n",
    "                                isClosed: true\n",
    "                            2: {day: \"WEDNESDAY\", intervals: [], isClosed: true}\n",
    "                                day: \"WEDNESDAY\"\n",
    "                                intervals: []\n",
    "                                isClosed: true\n",
    "                            3: {day: \"THURSDAY\", intervals: [], isClosed: true}\n",
    "                                day: \"THURSDAY\"\n",
    "                                intervals: []\n",
    "                                isClosed: true\n",
    "                            4: {day: \"FRIDAY\", intervals: [], isClosed: true}\n",
    "                                day: \"FRIDAY\"\n",
    "                                intervals: []\n",
    "                                isClosed: true\n",
    "                            5: {day: \"SATURDAY\", intervals: [], isClosed: true}\n",
    "                                day: \"SATURDAY\"\n",
    "                                intervals: []\n",
    "                                isClosed: true\n",
    "                            6: {day: \"SUNDAY\", intervals: [], isClosed: true}\n",
    "                                day: \"SUNDAY\"\n",
    "                                intervals: []\n",
    "                                isClosed: true\n",
    "                photo: {image: {height: 768,…}}\n",
    "                    image: {height: 768,…}\n",
    "                        height: 768\n",
    "                        sourceUrl: \"http://a.mktgcdn.com/p/FKZfwMNvVM2oQ1EvQIPTq8dLEkWU1_sFy592GblCSAA/1152x768.png\"\n",
    "                        thumbnails: [,…]\n",
    "                            0: {height: 412, url: \"//dynl.mktgcdn.com/p/FKZfwMNvVM2oQ1EvQIPTq8dLEkWU1_sFy592GblCSAA/619x412.png\",…}\n",
    "                                height: 412\n",
    "                                url: \"//dynl.mktgcdn.com/p/FKZfwMNvVM2oQ1EvQIPTq8dLEkWU1_sFy592GblCSAA/619x412.png\"\n",
    "                                width: 619\n",
    "                            1: {height: 400, url: \"//dynl.mktgcdn.com/p/FKZfwMNvVM2oQ1EvQIPTq8dLEkWU1_sFy592GblCSAA/600x400.png\",…}\n",
    "                                height: 400\n",
    "                                url: \"//dynl.mktgcdn.com/p/FKZfwMNvVM2oQ1EvQIPTq8dLEkWU1_sFy592GblCSAA/600x400.png\"\n",
    "                                width: 600\n",
    "                            2: {height: 130, url: \"//dynl.mktgcdn.com/p/FKZfwMNvVM2oQ1EvQIPTq8dLEkWU1_sFy592GblCSAA/196x130.png\",…}\n",
    "                                height: 130\n",
    "                                url: \"//dynl.mktgcdn.com/p/FKZfwMNvVM2oQ1EvQIPTq8dLEkWU1_sFy592GblCSAA/196x130.png\"\n",
    "                                width: 196\n",
    "                        url: \"//dynl.mktgcdn.com/p/FKZfwMNvVM2oQ1EvQIPTq8dLEkWU1_sFy592GblCSAA/1152x768.png\"\n",
    "                        width: 1152\n",
    "                    text(multi-line): \"Our Asda Cafés have your dining experience covered! We’ve got great food at low prices - just what you’d expect from Asda. What’s more, kids Eat Free Monday-Friday after 3pm when an adult spends just £4, and Pensioners get 10% off all café items on a Wednesday.\"\n",
    "                    text(single-line): \"Cafe\"\n",
    "                c_firstChoiceTravel: false\n",
    "                c_fishCounter: false\n",
    "                c_foodToGo: true\n",
    "                c_format: \"Superstore\"\n",
    "                c_fuelPrice: {diesel: \"112.70\", effectiveDate: {day: 10, month: 7, year: 2020}, unleaded: \"107.70\"}\n",
    "                    diesel: \"112.70\"\n",
    "                    effectiveDate: {day: 10, month: 7, year: 2020}\n",
    "                        day: 10\n",
    "                        month: 7\n",
    "                        year: 2020\n",
    "                    unleaded: \"107.70\"\n",
    "                c_george: true\n",
    "                c_georgeEssentials: false\n",
    "                c_groceryClickAndCollect: true\n",
    "                c_groceryClickAndCollectImage: {image: {height: 1080,…}}\n",
    "                    image: {height: 1080,…}\n",
    "                        height: 1080\n",
    "                        sourceUrl: \"https://p20.zdusercontent.com/attachment/58910/gDBXr12Pgnpbuo5pjdnenBscP?token=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..elcmPhwVI236Uq70DHK-0g.4wX3LoIZBvmeZ8XufZRN95wu_eNx_hQGuZ7Uq7jfwpkhi-7orwPfNKuv8tc5_KFS-UJ3tuiwVRT8HwqIet2PXeT6jXE4w5nHIF7KhoHVBGMu42yauAljxhppLBJb_oFPt89qxHaO5CQNFkaLZsfzV6-i2F26FmglJs0n_34LF4mHOQvASophHAA8SSmIa5rlGUeqrcFd3F1jGjGqq_8UWyOojF9xDgoXqfrZNaqlJCSwC89l7UkLsqtXNAWBLWDswsP7GNYySLzUk5_ut5aAg_DwHDOw2kDiUBJRo35OJuU.MrcEow8y-L3a-zUT-0rR4g\"\n",
    "                        thumbnails: [,…]\n",
    "                            0: {height: 1068, url: \"//dynl.mktgcdn.com/p/fyIydzdnhOol40zGaC_WdaSwnBCoZyi2MLLWGzafnVw/1900x1068.jpg\",…}\n",
    "                                height: 1068\n",
    "                                url: \"//dynl.mktgcdn.com/p/fyIydzdnhOol40zGaC_WdaSwnBCoZyi2MLLWGzafnVw/1900x1068.jpg\"\n",
    "                                width: 1900\n",
    "                            1: {height: 348, url: \"//dynl.mktgcdn.com/p/fyIydzdnhOol40zGaC_WdaSwnBCoZyi2MLLWGzafnVw/619x348.jpg\",…}\n",
    "                                height: 348\n",
    "                                url: \"//dynl.mktgcdn.com/p/fyIydzdnhOol40zGaC_WdaSwnBCoZyi2MLLWGzafnVw/619x348.jpg\"\n",
    "                                width: 619\n",
    "                            2: {height: 337, url: \"//dynl.mktgcdn.com/p/fyIydzdnhOol40zGaC_WdaSwnBCoZyi2MLLWGzafnVw/600x337.jpg\",…}\n",
    "                                height: 337\n",
    "                                url: \"//dynl.mktgcdn.com/p/fyIydzdnhOol40zGaC_WdaSwnBCoZyi2MLLWGzafnVw/600x337.jpg\"\n",
    "                                width: 600\n",
    "                            3: {height: 110, url: \"//dynl.mktgcdn.com/p/fyIydzdnhOol40zGaC_WdaSwnBCoZyi2MLLWGzafnVw/196x110.jpg\",…}\n",
    "                                height: 110\n",
    "                                url: \"//dynl.mktgcdn.com/p/fyIydzdnhOol40zGaC_WdaSwnBCoZyi2MLLWGzafnVw/196x110.jpg\"\n",
    "                                width: 196\n",
    "                        url: \"//dynl.mktgcdn.com/p/fyIydzdnhOol40zGaC_WdaSwnBCoZyi2MLLWGzafnVw/1920x1080.jpg\"\n",
    "                        width: 1920\n",
    "                c_halal: false\n",
    "                c_homeShopping: true\n",
    "                c_hotChicken: false\n",
    "                c_hoursSectionTitle: \"Store Hours\"\n",
    "                c_instantPhotoPrint: true\n",
    "                c_johnsonsDryCleaners: false\n",
    "                c_linkedEntities: [\"4383_C\"]\n",
    "                    0: \"4383_C\"\n",
    "                c_maxSpeilman: false\n",
    "                c_mcdonalds: false\n",
    "                c_mcgeesButchers: false\n",
    "                c_mobilePhoneConnectionCentre: false\n",
    "                c_nearbyStoresSectionTitle: \"Nearby Stores\"\n",
    "                c_opticians: false\n",
    "                c_organic: false\n",
    "                c_pagesURL: \"https://storelocator.asda.com/east-of-england/norwich/hall-road\"\n",
    "                c_paypoint: false\n",
    "                c_petrolStation: true\n",
    "                c_pharmacy: false\n",
    "                c_photoMe: false\n",
    "                c_pizzaCounter: true\n",
    "                c_postOffice: false\n",
    "                c_promoSectionImage: {clickthroughUrl: \"https://www.asdagoodliving.co.uk/food/get-your-cook-on\",…}\n",
    "                clickthroughUrl: \"https://www.asdagoodliving.co.uk/food/get-your-cook-on\"\n",
    "                    image: {height: 666, thumbnails: [,…],…}\n",
    "                    height: 666\n",
    "                    thumbnails: [,…]\n",
    "                        0: {height: 246, url: \"//dynl.mktgcdn.com/p/Oca1qYoHmBBMTt250kCtsbUyx5Uf-eUuJYqVLHHUp0w/619x246.jpg\",…}\n",
    "                            height: 246\n",
    "                            url: \"//dynl.mktgcdn.com/p/Oca1qYoHmBBMTt250kCtsbUyx5Uf-eUuJYqVLHHUp0w/619x246.jpg\"\n",
    "                            width: 619\n",
    "                        1: {height: 239, url: \"//dynl.mktgcdn.com/p/Oca1qYoHmBBMTt250kCtsbUyx5Uf-eUuJYqVLHHUp0w/600x239.jpg\",…}\n",
    "                            height: 239\n",
    "                            url: \"//dynl.mktgcdn.com/p/Oca1qYoHmBBMTt250kCtsbUyx5Uf-eUuJYqVLHHUp0w/600x239.jpg\"\n",
    "                            width: 600\n",
    "                        2: {height: 78, url: \"//dynl.mktgcdn.com/p/Oca1qYoHmBBMTt250kCtsbUyx5Uf-eUuJYqVLHHUp0w/196x78.jpg\",…}\n",
    "                            height: 78\n",
    "                            url: \"//dynl.mktgcdn.com/p/Oca1qYoHmBBMTt250kCtsbUyx5Uf-eUuJYqVLHHUp0w/196x78.jpg\"\n",
    "                            width: 196\n",
    "                    url: \"//dynl.mktgcdn.com/p/Oca1qYoHmBBMTt250kCtsbUyx5Uf-eUuJYqVLHHUp0w/1670x666.jpg\"\n",
    "                    width: 1670\n",
    "                c_recycling: false\n",
    "                c_region: \"Anglia\"\n",
    "                c_scanAndGo: true\n",
    "                c_serviceDetailCardCoinstar: {hours: null, photoField(single): null, text(multi-line): null, text(single-line)1: \"Coinstar\",…}\n",
    "                    hours: null\n",
    "                    photoField(single): null\n",
    "                    text(multi-line): null\n",
    "                    text(single-line)1: \"Coinstar\"\n",
    "                    text(single-line)2: null\n",
    "                    url: null\n",
    "                c_serviceDetailCardCosta: {hours: null, photoField(single): null, text(multi-line): null, text(single-line)1: \"Costa Coffee\",…}\n",
    "                    hours: null\n",
    "                    photoField(single): null\n",
    "                    text(multi-line): null\n",
    "                    text(single-line)1: \"Costa Coffee\"\n",
    "                    text(single-line)2: null\n",
    "                    url: null\n",
    "                c_serviceDetailCardGeorge: {hours: null, photoField(single): null,…}\n",
    "                    hours: null\n",
    "                    photoField(single): null\n",
    "                    text(multi-line): \"Visit George to discover clothing, home accessories and footwear for the whole family, including school uniforms, women’s dresses and accessories.\"\n",
    "                    text(single-line)1: \"George\"\n",
    "                    text(single-line)2: \"Shop now\"\n",
    "                    url: \"http://www.george.com\"\n",
    "                c_serviceDetailCardGeorgeClickAndCollect: {hours: {holidayHours: [],…}, photoField(single): null,…}\n",
    "                    hours: {holidayHours: [],…}\n",
    "                        holidayHours: []\n",
    "                        normalHours: [{day: \"MONDAY\", intervals: [{end: 2200, start: 800}], isClosed: false},…]\n",
    "                            0: {day: \"MONDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                                day: \"MONDAY\"\n",
    "                                intervals: [{end: 2200, start: 800}]\n",
    "                                    0: {end: 2200, start: 800}\n",
    "                                        end: 2200\n",
    "                                        start: 800\n",
    "                                isClosed: false\n",
    "                            1: {day: \"TUESDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                                day: \"TUESDAY\"\n",
    "                                intervals: [{end: 2200, start: 800}]\n",
    "                                    0: {end: 2200, start: 800}\n",
    "                                        end: 2200\n",
    "                                        start: 800\n",
    "                                isClosed: false\n",
    "                            2: {day: \"WEDNESDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                                day: \"WEDNESDAY\"\n",
    "                                intervals: [{end: 2200, start: 800}]\n",
    "                                    0: {end: 2200, start: 800}\n",
    "                                        end: 2200\n",
    "                                        start: 800\n",
    "                                isClosed: false\n",
    "                            3: {day: \"THURSDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                                day: \"THURSDAY\"\n",
    "                                intervals: [{end: 2200, start: 800}]\n",
    "                                    0: {end: 2200, start: 800}\n",
    "                                        end: 2200\n",
    "                                        start: 800\n",
    "                                isClosed: false\n",
    "                            4: {day: \"FRIDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                                day: \"FRIDAY\"\n",
    "                                intervals: [{end: 2200, start: 800}]\n",
    "                                    0: {end: 2200, start: 800}\n",
    "                                        end: 2200\n",
    "                                        start: 800\n",
    "                                isClosed: false\n",
    "                            5: {day: \"SATURDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                                day: \"SATURDAY\"\n",
    "                                intervals: [{end: 2200, start: 800}]\n",
    "                                    0: {end: 2200, start: 800}\n",
    "                                        end: 2200\n",
    "                                        start: 800\n",
    "                                isClosed: false\n",
    "                            6: {day: \"SUNDAY\", intervals: [{end: 1600, start: 1000}], isClosed: false}\n",
    "                                day: \"SUNDAY\"\n",
    "                                intervals: [{end: 1600, start: 1000}]\n",
    "                                    0: {end: 1600, start: 1000}\n",
    "                                        end: 1600\n",
    "                                        start: 1000\n",
    "                                isClosed: false\n",
    "                    photoField(single): null\n",
    "                    text(multi-line): \"TO ASSIST OUR STORE COLLEAGUES AT THIS BUSY TIME, THIS SERVICE IS NOT CURRENTLY AVAILABLE\"\n",
    "                    text(single-line)1: \"George Click & Collect\"\n",
    "                    text(single-line)2: null\n",
    "                    url: null\n",
    "                c_serviceDetailCardPetrolFillingStation: {hours: null, photoField(single): null, text(multi-line): null,…}\n",
    "                    hours: null\n",
    "                    photoField(single): null\n",
    "                    text(multi-line): null\n",
    "                    text(single-line)1: \"Petrol Filling Station\"\n",
    "                    text(single-line)2: null\n",
    "                    url: null\n",
    "                c_serviceDetailCardRugDoctor: {hours: null, photoField(single): null, text(multi-line): null, text(single-line)1: \"Rug Doctor\",…}\n",
    "                    hours: null\n",
    "                    photoField(single): null\n",
    "                    text(multi-line): null\n",
    "                    text(single-line)1: \"Rug Doctor\"\n",
    "                    text(single-line)2: null\n",
    "                    url: null\n",
    "                c_serviceDetailCardScanAndGo: {hours: {holidayHours: [],…}, photoField(single): null,…}\n",
    "                    hours: {holidayHours: [],…}\n",
    "                        holidayHours: []\n",
    "                        normalHours: [{day: \"MONDAY\", intervals: [{end: 2200, start: 800}], isClosed: false},…]\n",
    "                            0: {day: \"MONDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                            day: \"MONDAY\"\n",
    "                            intervals: [{end: 2200, start: 800}]\n",
    "                                0: {end: 2200, start: 800}\n",
    "                                    end: 2200\n",
    "                                    start: 800\n",
    "                            isClosed: false\n",
    "                        1: {day: \"TUESDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                            day: \"TUESDAY\"\n",
    "                            intervals: [{end: 2200, start: 800}]\n",
    "                                0: {end: 2200, start: 800}\n",
    "                                    end: 2200\n",
    "                                    start: 800\n",
    "                            isClosed: false\n",
    "                        2: {day: \"WEDNESDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                            day: \"WEDNESDAY\"\n",
    "                            intervals: [{end: 2200, start: 800}]\n",
    "                                0: {end: 2200, start: 800}\n",
    "                                    end: 2200\n",
    "                                    start: 800\n",
    "                            isClosed: false\n",
    "                        3: {day: \"THURSDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                            day: \"THURSDAY\"\n",
    "                            intervals: [{end: 2200, start: 800}]\n",
    "                                0: {end: 2200, start: 800}\n",
    "                                    end: 2200\n",
    "                                    start: 800\n",
    "                            isClosed: false\n",
    "                        4: {day: \"FRIDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                            day: \"FRIDAY\"\n",
    "                            intervals: [{end: 2200, start: 800}]\n",
    "                                0: {end: 2200, start: 800}\n",
    "                                    end: 2200\n",
    "                                    start: 800\n",
    "                            isClosed: false\n",
    "                        5: {day: \"SATURDAY\", intervals: [{end: 2200, start: 800}], isClosed: false}\n",
    "                            day: \"SATURDAY\"\n",
    "                            intervals: [{end: 2200, start: 800}]\n",
    "                                0: {end: 2200, start: 800}\n",
    "                                    end: 2200\n",
    "                                    start: 800\n",
    "                            isClosed: false\n",
    "                        6: {day: \"SUNDAY\", intervals: [{end: 1600, start: 1000}], isClosed: false}\n",
    "                            day: \"SUNDAY\"\n",
    "                            intervals: [{end: 1600, start: 1000}]\n",
    "                                0: {end: 1600, start: 1000}\n",
    "                                    end: 1600\n",
    "                                    start: 1000\n",
    "                            isClosed: false\n",
    "                    photoField(single): null\n",
    "                    text(multi-line): \"You can use both our Scan & Go handsets and Mobile App in this store.\"\n",
    "                    text(single-line)1: \"Scan & Go\"\n",
    "                    text(single-line)2: \"Read More\"\n",
    "                    url: \"https://www.asda.com/about/instore/scan-and-go\"\n",
    "                c_serviceDetailCardToYou: {hours: null, photoField(single): null,…}\n",
    "                    hours: null\n",
    "                    photoField(single): null\n",
    "                    text(multi-line): \"Click/Collect/Return at Asda stores with over a 100 of your favourite retailers.\"\n",
    "                    text(single-line)1: \"toyou\"\n",
    "                    text(single-line)2: \"Read More\"\n",
    "                    url: \"https://www.asda.com/toyou/?cmpid=dmc-_-ahc-_-vanityurl-_-generic-_-toyou&utm_source=vanityurl&utm_medium=dmc&utm_term=toyou&utm_content=generic&utm_campaign=ahc\"\n",
    "                c_serviceDetailCardsJustEatKitchen: {hours: null, photoField(single): null,…}\n",
    "                    hours: null\n",
    "                    photoField(single): null\n",
    "                    text(multi-line): \"Hot food for collection & delivery with Just Eat\"\n",
    "                    text(single-line)1: \"Just Eat Kitchen\"\n",
    "                    text(single-line)2: null\n",
    "                    url: null\n",
    "                c_serviceDetailsTitle: \"Services\"\n",
    "                c_storeLocatorFacilitiesFilter: [\"ToYou\", \"Electric Car Charging\", \"Cash Machine\"]\n",
    "                    0: \"ToYou\"\n",
    "                    1: \"Electric Car Charging\"\n",
    "                    2: \"Cash Machine\"\n",
    "                c_storeLocatorServicesFilter: [\"Scan and Go\", \"Cafe\", \"Grocery Click & Collect\", \"Just Eat Kitchen\"]\n",
    "                    0: \"Scan and Go\"\n",
    "                    1: \"Cafe\"\n",
    "                    2: \"Grocery Click & Collect\"\n",
    "                    3: \"Just Eat Kitchen\"\n",
    "                c_storeType: [\"Superstore\"]\n",
    "                    0: \"Superstore\"\n",
    "                c_subway: false\n",
    "                c_takeawayHotPizza: false\n",
    "                c_technologyCentre: false\n",
    "                c_thomasCook: false\n",
    "                c_timpsons: false\n",
    "                c_travelMoney: false\n",
    "                c_travelMoneyClickAndCollect: false\n",
    "                c_trolleySectionDescription: \"Have you found an Asda trolley in your community that has been taken from our store? Let us know exactly where it is and we’ll come and collect it and return it back to Asda.\"\n",
    "                c_trolleywise: true\n",
    "                c_tuiTravel: false\n",
    "                c_uKRegion: [\"England\"]\n",
    "                    0: \"England\"\n",
    "                c_whatsAtThisStoreList: [\"Rotisserie\", \"Pizza Counter\", \"Food To Go\", \"Instant Photo Print\", \"Bakery\", \"Express Diner\",…]\n",
    "                    0: \"Rotisserie\"\n",
    "                    1: \"Pizza Counter\"\n",
    "                    2: \"Food To Go\"\n",
    "                    3: \"Instant Photo Print\"\n",
    "                    4: \"Bakery\"\n",
    "                    5: \"Express Diner\"\n",
    "                    6: \"Helium Balloons\"\n",
    "                c_whatsAtThisStoreTitle: \"What's at this store\"\n",
    "                cityCoordinate: {lat: 52.63529968261719, long: 1.294010043144226}\n",
    "                    lat: 52.63529968261719\n",
    "                    long: 1.294010043144226\n",
    "                description: \"Asda Norwich Hall Road Superstore is part of the Asda group; one of Britain’s leading grocery retailers. Asda offers great prices and quality products helping customers save money & live better. Visit us in-store and get groceries delivered to anywhere in the UK, or collect from your local store or petrol station at a convenient time for you.\"\n",
    "                displayCoordinate: {lat: 52.606659, long: 1.288596}\n",
    "                    lat: 52.606659\n",
    "                    long: 1.288596\n",
    "                facebookCoverPhoto: {image: {height: 273,…}}\n",
    "                    image: {height: 273,…}\n",
    "                        height: 273\n",
    "                        sourceUrl: \"https://scontent-bos3-1.xx.fbcdn.net/v/t1.0-9/s720x720/96379594_3804414072963302_8747344537808011264_n.png?_nc_cat=1&_nc_sid=dd9801&_nc_ohc=nKEKKHjU_hkAX-q129p&_nc_ht=scontent-bos3-1.xx&oh=ab10e811c6c006d6288d6351d4e576de&oe=5EE042E9\"\n",
    "                        url: \"//dynl.mktgcdn.com/p/2aa3EsjnMcrzNB-76pyJ-NkPNOEPPc4tHnk2EYUXytA/720x273.png\"\n",
    "                        width: 720\n",
    "                facebookName: \"Asda\"\n",
    "                facebookPageUrl: \"https://www.facebook.com/457762304577879\"\n",
    "                facebookParentPageId: \"141700922567987\"\n",
    "                facebookProfilePhoto: {image: {height: 200,…}}\n",
    "                    image: {height: 200,…}\n",
    "                        height: 200\n",
    "                        sourceUrl: \"https://scontent-bos3-1.xx.fbcdn.net/v/t31.0-1/p200x200/17359452_1454805254590874_1187074472493250060_o.png?_nc_cat=1&_nc_sid=0c64ff&_nc_ohc=24w1xzbZDC4AX-30l-y&_nc_ht=scontent-bos3-1.xx&oh=4cff6b231ae9659e4873db25995cf638&oe=5EE05BEA\"\n",
    "                        url: \"//dynl.mktgcdn.com/p/RB41DA38HIa7bG3QCUXHyDXHx0mdAt8i80htQFuQqos/200x200.png\"\n",
    "                        width: 200\n",
    "                featuredMessage: {description: \"Visit us in-store or online\", url: \"https://www.asda.com/\"}\n",
    "                    description: \"Visit us in-store or online\"\n",
    "                    url: \"https://www.asda.com/\"\n",
    "                geocodedCoordinate: {lat: 52.6065825, long: 1.2876707}\n",
    "                    lat: 52.6065825\n",
    "                    long: 1.2876707\n",
    "                geomodifier: \"Norwich Hall Road\"\n",
    "                googleAccountId: \"117489345900231741701\"\n",
    "                googleAttributes: {}\n",
    "                googleCoverPhoto: {,…}\n",
    "                    image: {height: 720, sourceUrl: \"https://yext.box.com/shared/static/0oq6t16l8qtmkvbedwbwqt90lm3cv98q.jpg\",…}\n",
    "                        height: 720\n",
    "                        sourceUrl: \"https://yext.box.com/shared/static/0oq6t16l8qtmkvbedwbwqt90lm3cv98q.jpg\"\n",
    "                        thumbnails: [,…]\n",
    "                            0: {height: 348, url: \"//dynl.mktgcdn.com/p/Oyr6H_jd6d6J9STzXi32HJoiX89URA7GxW1eOl31th0/619x348.jpg\",…}\n",
    "                                height: 348\n",
    "                                url: \"//dynl.mktgcdn.com/p/Oyr6H_jd6d6J9STzXi32HJoiX89URA7GxW1eOl31th0/619x348.jpg\"\n",
    "                                width: 619\n",
    "                            1: {height: 337, url: \"//dynl.mktgcdn.com/p/Oyr6H_jd6d6J9STzXi32HJoiX89URA7GxW1eOl31th0/600x337.jpg\",…}\n",
    "                                height: 337\n",
    "                                url: \"//dynl.mktgcdn.com/p/Oyr6H_jd6d6J9STzXi32HJoiX89URA7GxW1eOl31th0/600x337.jpg\"\n",
    "                                width: 600\n",
    "                            2: {height: 110, url: \"//dynl.mktgcdn.com/p/Oyr6H_jd6d6J9STzXi32HJoiX89URA7GxW1eOl31th0/196x110.jpg\",…}\n",
    "                                height: 110\n",
    "                                url: \"//dynl.mktgcdn.com/p/Oyr6H_jd6d6J9STzXi32HJoiX89URA7GxW1eOl31th0/196x110.jpg\"\n",
    "                                width: 196\n",
    "                            url: \"//dynl.mktgcdn.com/p/Oyr6H_jd6d6J9STzXi32HJoiX89URA7GxW1eOl31th0/1280x720.jpg\"\n",
    "                            width: 1280\n",
    "                googlePlaceId: \"ChIJKZs0vAHk2UcRkaLqm7RF3hk\"\n",
    "                hours: {holidayHours: [],…}\n",
    "                    holidayHours: []\n",
    "                    normalHours: [{day: \"MONDAY\", intervals: [{end: 2200, start: 700}], isClosed: false},…]\n",
    "                        0: {day: \"MONDAY\", intervals: [{end: 2200, start: 700}], isClosed: false}\n",
    "                            day: \"MONDAY\"\n",
    "                            intervals: [{end: 2200, start: 700}]\n",
    "                                0: {end: 2200, start: 700}\n",
    "                                    end: 2200\n",
    "                                    start: 700\n",
    "                            isClosed: false\n",
    "                        1: {day: \"TUESDAY\", intervals: [{end: 2200, start: 700}], isClosed: false}\n",
    "                            day: \"TUESDAY\"\n",
    "                            intervals: [{end: 2200, start: 700}]\n",
    "                                0: {end: 2200, start: 700}\n",
    "                                    end: 2200\n",
    "                                    start: 700\n",
    "                            isClosed: false\n",
    "                        2: {day: \"WEDNESDAY\", intervals: [{end: 2200, start: 700}], isClosed: false}\n",
    "                            day: \"WEDNESDAY\"\n",
    "                            intervals: [{end: 2200, start: 700}]\n",
    "                                0: {end: 2200, start: 700}\n",
    "                                    end: 2200\n",
    "                                    start: 700\n",
    "                            isClosed: false\n",
    "                        3: {day: \"THURSDAY\", intervals: [{end: 2200, start: 700}], isClosed: false}\n",
    "                            day: \"THURSDAY\"\n",
    "                            intervals: [{end: 2200, start: 700}]\n",
    "                                0: {end: 2200, start: 700}\n",
    "                                    end: 2200\n",
    "                                    start: 700\n",
    "                            isClosed: false\n",
    "                        4: {day: \"FRIDAY\", intervals: [{end: 2200, start: 700}], isClosed: false}\n",
    "                            day: \"FRIDAY\"\n",
    "                            intervals: [{end: 2200, start: 700}]\n",
    "                                0: {end: 2200, start: 700}\n",
    "                                    end: 2200\n",
    "                                    start: 700\n",
    "                            isClosed: false\n",
    "                        5: {day: \"SATURDAY\", intervals: [{end: 2200, start: 700}], isClosed: false}\n",
    "                            day: \"SATURDAY\"\n",
    "                            intervals: [{end: 2200, start: 700}]\n",
    "                                0: {end: 2200, start: 700}\n",
    "                                    end: 2200\n",
    "                                    start: 700\n",
    "                            isClosed: false\n",
    "                        6: {day: \"SUNDAY\", intervals: [{end: 1600, start: 1000}], isClosed: false}\n",
    "                            day: \"SUNDAY\"\n",
    "                            intervals: [{end: 1600, start: 1000}]\n",
    "                                0: {end: 1600, start: 1000}\n",
    "                                    end: 1600\n",
    "                                    start: 1000\n",
    "                            isClosed: false\n",
    "                isoRegionCode: \"NFK\"\n",
    "                logo: {image: {height: 400, thumbnails: [,…],…}}\n",
    "                    image: {height: 400, thumbnails: [,…],…}\n",
    "                        height: 400\n",
    "                        thumbnails: [,…]\n",
    "                        0: {height: 150, url: \"//dynl.mktgcdn.com/p/uxpSIwyZRALdFsUMpGERiKVVeUVlEaMMTBvKbuOZB-E/150x150.png\",…}\n",
    "                            height: 150\n",
    "                            url: \"//dynl.mktgcdn.com/p/uxpSIwyZRALdFsUMpGERiKVVeUVlEaMMTBvKbuOZB-E/150x150.png\"\n",
    "                            width: 150\n",
    "                        1: {height: 94, url: \"//dynl.mktgcdn.com/p/uxpSIwyZRALdFsUMpGERiKVVeUVlEaMMTBvKbuOZB-E/94x94.png\",…}\n",
    "                            height: 94\n",
    "                            url: \"//dynl.mktgcdn.com/p/uxpSIwyZRALdFsUMpGERiKVVeUVlEaMMTBvKbuOZB-E/94x94.png\"\n",
    "                            width: 94\n",
    "                        url: \"//dynl.mktgcdn.com/p/uxpSIwyZRALdFsUMpGERiKVVeUVlEaMMTBvKbuOZB-E/400x400.png\"\n",
    "                        width: 400\n",
    "                mainPhone: {countryCode: \"GB\", display: \"01603 964246\", number: \"+441603964246\"}\n",
    "                    countryCode: \"GB\"\n",
    "                    display: \"01603 964246\"\n",
    "                    number: \"+441603964246\"\n",
    "                matchingSearchIds: [\"4656\"]\n",
    "                    0: \"4656\"\n",
    "                meta: {accountId: \"3332574490752748386\", countryCode: \"GB\", entityType: \"location\", folderId: \"298473\",…}\n",
    "                    accountId: \"3332574490752748386\"\n",
    "                    countryCode: \"GB\"\n",
    "                    entityType: \"location\"\n",
    "                    folderId: \"298473\"\n",
    "                    id: \"4383\"\n",
    "                    labels: [\"78691\", \"78687\", \"78686\", \"78690\", \"82562\", \"84251\", \"95837\"]\n",
    "                        0: \"78691\"\n",
    "                        1: \"78687\"\n",
    "                        2: \"78686\"\n",
    "                        3: \"78690\"\n",
    "                        4: \"82562\"\n",
    "                        5: \"84251\"\n",
    "                        6: \"95837\"\n",
    "                    language: \"en_GB\"\n",
    "                    schemaTypes: [\"GroceryStore\", \"AutomatedTeller\", \"GasStation\", \"ClothingStore\"]\n",
    "                        0: \"GroceryStore\"\n",
    "                        1: \"AutomatedTeller\"\n",
    "                        2: \"GasStation\"\n",
    "                        3: \"ClothingStore\"\n",
    "                    timestamp: \"2020-07-13T01:47:47\"\n",
    "                    uid: \"6wOk6E\"\n",
    "                    utcOffsets: [{offset: 3600, start: 1585443600}, {offset: 0, start: 1603587600}, {offset: 3600, start: 1616893200}]\n",
    "                        0: {offset: 3600, start: 1585443600}\n",
    "                            offset: 3600\n",
    "                            start: 1585443600\n",
    "                        1: {offset: 0, start: 1603587600}\n",
    "                            offset: 0\n",
    "                            start: 1603587600\n",
    "                        2: {offset: 3600, start: 1616893200}\n",
    "                            offset: 3600\n",
    "                            start: 1616893200\n",
    "                    yextId: 13618640\n",
    "                name: \"Asda Norwich Hall Road Superstore\"\n",
    "                timezone: \"Europe/London\"\n",
    "                websiteUrl: \"http://storelocator.asda.com/store/norwich-hall-road\"\n",
    "                yextDisplayCoordinate: {lat: 52.6065825, long: 1.2876707}\n",
    "                lat: 52.6065825\n",
    "                long: 1.2876707\n",
    "                yextRoutableCoordinate: {lat: 52.60555, long: 1.28634}\n",
    "                lat: 52.60555\n",
    "                long: 1.28634\n",
    "            url: \"east-of-england/norwich/hall-road\"\n",
    "        1: {derivedData: {address: {countryName: \"United Kingdom\"}, uber: {,…}},…}\n",
    "    geo: {address: {city: \"Norwich\", country: \"GB\", postalCode: \"NR3 1TP\", region: \"England\"},…}\n",
    "schema: {additionalHoursText: {archived: false, displayName: \"Additional Hours Text\",…},…}\n",
    "searchId: \"entitysearch\"\n",
    "siteAttributes: {footer: {legal: \"© 2019 ASDA. All Rights Reserved.\", link1: \"Privacy\", link2: \"Terms of Service\",…},…}\n",
    "siteDomain: \"storelocator.asda.com\"\n",
    "siteId: 1812\n",
    "siteInternalHostName: \"storelocator.asda.com\"\n",
    "versions: {en: \"index.html\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retreiving ASDA's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_asda_data(lat,lng):\n",
    "    import json\n",
    "    API_URL = \"https://storelocator.asda.com/index.html\"\n",
    "    params = { 'q' : \"{0},{1}\".format(lat,lng)}\n",
    "    headers = {\n",
    "    \"accept\": \"application/json\",\n",
    "  }\n",
    "    rq = requests.get(API_URL,params=params, headers = headers)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.json())\n",
    "        return False\n",
    "    # res = json.dumps(rq.text)\n",
    "    try:\n",
    "        res = rq.json()[\"response\"]\n",
    "        if res[\"count\"] == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "    i = 0 \n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (res[\"entities\"][0][\"name\"])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[\"entities\"][i][\"name\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    if res[\"entities\"][i][\"distance\"][\"distanceKilometers\"] > radius:\n",
    "        return False\n",
    "    openingHours = res[\"entities\"][i][\"profile\"][\"hours\"][\"normalHours\"]\n",
    "    hoursArray = []\n",
    "    intermediaryDayKeys = { 'MONDAY' : 0, 'TUESDAY' : 1, 'WEDNESDAY' : 2, 'THURSDAY' : 3, 'FRIDAY' : 4, 'SATURDAY' : 5, 'SUNDAY' : 6}\n",
    "    dayKeys = { 0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday', 5 : 'Saturday', 6 : 'Sunday'}\n",
    "    dayDoneSet = set()\n",
    "    daysHeap = []\n",
    "    for index in range(len(openingHours)):\n",
    "            day = intermediaryDayKeys[openingHours[index][\"day\"]]\n",
    "            #if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "            if day not in dayDoneSet:\n",
    "                dayDoneSet.add(day)\n",
    "                if openingHours[index][\"isClosed\"] != False:\n",
    "                    closedDayHours = {  'day' : dayKeys[day],\n",
    "                                        'open' : False  }\n",
    "                    heapq.heappush(daysHeap, [day,closedDayHours])\n",
    "                else:\n",
    "                    actualHours = []\n",
    "                    for timeSlotNumber in range(len(openingHours[index]['intervals'])):\n",
    "                        try:\n",
    "                            actualHoursDict = {}\n",
    "                            start = str(openingHours[index]['intervals'][timeSlotNumber]['start'])\n",
    "                            if start == \"0\":\n",
    "                                start = \"00:00\"\n",
    "                            elif len(start) == 3:\n",
    "                                start = \"0{0}:{1}\".format(start[0],start[1:])\n",
    "                            else:\n",
    "                                start = \"{0}:{1}\".format(start[:2],start[2:])\n",
    "                            end = str(openingHours[index]['intervals'][timeSlotNumber]['end'])\n",
    "                            if end == \"0\":\n",
    "                                end = \"00:00\"\n",
    "                            elif len(end) == 3:\n",
    "                                end = \"0{0}:{1}\".format(end[0],end[1:])\n",
    "                            else:\n",
    "                                end = \"{0}:{1}\".format(end[:2],end[2:])\n",
    "                            actualHoursDict['open'] = start\n",
    "                            actualHoursDict['close'] = end\n",
    "                            actualHours.append(actualHoursDict)\n",
    "                        except:\n",
    "                            #If dictionary keys start_time or end_time dont exist, assume store is closed on that day\n",
    "                            closedDayHours = {  'day' : dayKeys[day],\n",
    "                                            'open' : False  }\n",
    "                            heapq.heappush(daysHeap, [day,closedDayHours])\n",
    "                            break\n",
    "                    keyHours = {'day' : dayKeys[day],\n",
    "                            'open' : True,\n",
    "                            'hours' : actualHours\n",
    "                            }\n",
    "                    heapq.heappush(daysHeap, [day,keyHours])\n",
    "\n",
    "    #Check for any missing days and ensure all days are in order for insertion into our database     \n",
    "    for missingDay in set(dayKeys.keys()).difference(dayDoneSet):\n",
    "        closedDayHours = {\n",
    "            \"day\": dayKeys[missingDay],\n",
    "            \"open\": False\n",
    "        }\n",
    "        heapq.heappush(daysHeap, [missingDay, closedDayHours])\n",
    "    if len(daysHeap) != 7:\n",
    "        return False\n",
    "    hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "    return hoursArray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tesco's API\n",
    "\n",
    "## Parameters:\n",
    "* offset : 0 \n",
    "* limit : number of stores to show (int)\n",
    "* sort : \"near:%22{LATITUDE},{LONGITUDE}%22\"\n",
    "* filter : \"category:Store%20AND%20isoCountryCode:x-uk\"\n",
    "* fields : what to return (subset of <code>\"name,geo,openingHours,altIds.branchNumber,contact,facilities\"</code>)\n",
    "\n",
    "## Headers:\n",
    "* x-appkey : \"store-locator-web-cde\"\n",
    "\n",
    " \n",
    "## Sample API Call directly from website:\n",
    "<code>rq = requests.get(\"https://api.tesco.com/tescolocation/v3/locations/search?offset=0&limit=10&sort=near:%2252.62682342529297,1.3366719484329224%22&filter=category:Store%20AND%20isoCountryCode:x-uk&fields=name,geo,openingHours,altIds.branchNumber,contact,facilities\", headers={\"x-appkey\": \"store-locator-web-cde\"})\n",
    "</code>\n",
    "\n",
    "RESULTS:\n",
    "* of interest is: rq.json()['results'][0]['location']['openingHours'][0]['standardOpeningHours']\n",
    "* and : rq.json()['results'][0]['location']['openingHours'][0]['exceptions']\n",
    "* However, exceptions provides you a historical(?) list of dictionaries of all dates that have had exceptions in the past(?), in a seemingly completely unsorted order, making using this information for anything other than extrapolation (**NOT IMPLEMENTED**) of 'special dates'' opening times (e.g. christmas) somewhat unfeasible.\n",
    "\n",
    "### Sample response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{,…}\n",
    "    results: [{location: {id: \"2ad4c578-962b-42cd-8541-0f75c1952680\", name: \"Norwich Plumstead Rd Express\",…},…},…]\n",
    "        0: {location: {id: \"2ad4c578-962b-42cd-8541-0f75c1952680\", name: \"Norwich Plumstead Rd Express\",…},…}\n",
    "            distanceFrom: {unit: \"miles\", value: 0.8}\n",
    "                unit: \"miles\"\n",
    "                value: 0.8\n",
    "            location: {id: \"2ad4c578-962b-42cd-8541-0f75c1952680\", name: \"Norwich Plumstead Rd Express\",…}\n",
    "                altIds: {branchNumber: 5284}\n",
    "                    branchNumber: 5284\n",
    "                contact: {address: {lines: [{lineNumber: 1, text: \"197 Plumstead Rd\"}], town: \"Norwich\", postcode: \"NR1 4AB\"},…}\n",
    "                    address: {lines: [{lineNumber: 1, text: \"197 Plumstead Rd\"}], town: \"Norwich\", postcode: \"NR1 4AB\"}\n",
    "                        lines: [{lineNumber: 1, text: \"197 Plumstead Rd\"}]\n",
    "                            0: {lineNumber: 1, text: \"197 Plumstead Rd\"}\n",
    "                                lineNumber: 1\n",
    "                                text: \"197 Plumstead Rd\"\n",
    "                            postcode: \"NR1 4AB\"\n",
    "                            town: \"Norwich\"\n",
    "                    phoneNumbers: [{alias: \"Default\", number: \"0345 674 6492\"}]\n",
    "                        0: {alias: \"Default\", number: \"0345 674 6492\"}\n",
    "                            alias: \"Default\"\n",
    "                            number: \"0345 674 6492\"\n",
    "                facilities: [{name: \"FREE_FROM\", tags: [\"public\", \"food_range\"], description: \"Free From\"},…]\n",
    "                0: {name: \"FREE_FROM\", tags: [\"public\", \"food_range\"], description: \"Free From\"}\n",
    "                    description: \"Free From\"\n",
    "                    name: \"FREE_FROM\"\n",
    "                    tags: [\"public\", \"food_range\"]\n",
    "                        0: \"public\"\n",
    "                        1: \"food_range\"\n",
    "                1: {name: \"ATM\", tags: [\"public\"], description: \"ATM\"}\n",
    "                    description: \"ATM\"\n",
    "                    name: \"ATM\"\n",
    "                    tags: [\"public\"]\n",
    "                        0: \"public\"\n",
    "                2: {name: \"GIFT_CARDS\", tags: [\"public\"], description: \"Sells Gift Cards\"}\n",
    "                    description: \"Sells Gift Cards\"\n",
    "                    name: \"GIFT_CARDS\"\n",
    "                    tags: [\"public\"]\n",
    "                        0: \"public\"\n",
    "                3: {name: \"TESCO_PAY_PLUS\", tags: [\"public\"], description: \"Tesco Pay+\"}\n",
    "                    description: \"Tesco Pay+\"\n",
    "                    name: \"TESCO_PAY_PLUS\"\n",
    "                    tags: [\"public\"]\n",
    "                        0: \"public\"\n",
    "                4: {name: \"ASSISTANCE_DOGS\", tags: [\"enabling\", \"public\"],…}\n",
    "                    description: \"Assistance dogs are welcome in our store.\"\n",
    "                    name: \"ASSISTANCE_DOGS\"\n",
    "                    tags: [\"enabling\", \"public\"]\n",
    "                        0: \"enabling\"\n",
    "                        1: \"public\"\n",
    "                5: {name: \"PAYPOINT\", tags: [\"public\"], description: \"PayPoint\"}\n",
    "                    description: \"PayPoint\"\n",
    "                    name: \"PAYPOINT\"\n",
    "                    tags: [\"public\"]\n",
    "                        0: \"public\"\n",
    "                geo: {coordinates: {longitude: 1.32715, latitude: 52.63697}}\n",
    "                    coordinates: {longitude: 1.32715, latitude: 52.63697}\n",
    "                        latitude: 52.63697\n",
    "                        longitude: 1.32715\n",
    "                id: \"2ad4c578-962b-42cd-8541-0f75c1952680\"\n",
    "                name: \"Norwich Plumstead Rd Express\"\n",
    "                openingHours: [{type: \"Trading\",…}]\n",
    "                    0: {type: \"Trading\",…}\n",
    "                        exceptions: [{date: \"2017-12-25\", hours: {isOpen: \"false\"}},…]\n",
    "                            0: {date: \"2017-12-25\", hours: {isOpen: \"false\"}}\n",
    "                                date: \"2017-12-25\"\n",
    "                                hours: {isOpen: \"false\"}\n",
    "                                    isOpen: \"false\"\n",
    "                            1: {date: \"2020-03-21\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2020-03-21\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            2: {date: \"2017-12-26\", hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}}\n",
    "                                date: \"2017-12-26\"\n",
    "                                hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0800\"\n",
    "                            3: {date: \"2020-03-22\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2020-03-22\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            4: {date: \"2020-03-18\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2020-03-18\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            5: {date: \"2019-12-31\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2019-12-31\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            6: {date: \"2018-12-24\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2018-12-24\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            7: {date: \"2020-03-23\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2020-03-23\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            8: {date: \"2019-12-25\", hours: {isOpen: \"false\"}}\n",
    "                                date: \"2019-12-25\"\n",
    "                                hours: {isOpen: \"false\"}\n",
    "                                    isOpen: \"false\"\n",
    "                            9: {date: \"2020-03-19\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2020-03-19\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            10: {date: \"2018-01-01\", hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}}\n",
    "                                date: \"2018-01-01\"\n",
    "                                hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0800\"\n",
    "                            11: {date: \"2018-12-31\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2018-12-31\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            12: {date: \"2019-12-26\", hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}}\n",
    "                                date: \"2019-12-26\"\n",
    "                                hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0800\"\n",
    "                            13: {date: \"2020-01-01\", hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}}\n",
    "                                date: \"2020-01-01\"\n",
    "                                hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0800\"\n",
    "                            14: {date: \"2018-12-25\", hours: {isOpen: \"false\"}}\n",
    "                                date: \"2018-12-25\"\n",
    "                                hours: {isOpen: \"false\"}\n",
    "                                    isOpen: \"false\"\n",
    "                            15: {date: \"2017-12-24\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2017-12-24\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            16: {date: \"2019-12-24\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2019-12-24\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            17: {date: \"2017-12-31\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2017-12-31\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            18: {date: \"2019-01-01\", hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}}\n",
    "                                date: \"2019-01-01\"\n",
    "                                hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0800\"\n",
    "                            19: {date: \"2020-03-20\", hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}}\n",
    "                                date: \"2020-03-20\"\n",
    "                                hours: {isOpen: \"true\", open: \"0600\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0600\"\n",
    "                            20: {date: \"2018-12-26\", hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}}\n",
    "                                date: \"2018-12-26\"\n",
    "                                hours: {isOpen: \"true\", open: \"0800\", close: \"2200\"}\n",
    "                                    close: \"2200\"\n",
    "                                    isOpen: \"true\"\n",
    "                                    open: \"0800\"\n",
    "                        standardOpeningHours: {mo: {isOpen: \"true\", open: \"0600\", close: \"2300\"}, tu: {isOpen: \"true\", open: \"0600\", close: \"2300\"},…}\n",
    "                            fr: {isOpen: \"true\", open: \"0600\", close: \"2300\"}\n",
    "                                close: \"2300\"\n",
    "                                isOpen: \"true\"\n",
    "                                open: \"0600\"\n",
    "                            mo: {isOpen: \"true\", open: \"0600\", close: \"2300\"}\n",
    "                                close: \"2300\"\n",
    "                                isOpen: \"true\"\n",
    "                                open: \"0600\"\n",
    "                            sa: {isOpen: \"true\", open: \"0600\", close: \"2300\"}\n",
    "                                close: \"2300\"\n",
    "                                isOpen: \"true\"\n",
    "                                open: \"0600\"\n",
    "                            su: {isOpen: \"true\", open: \"0600\", close: \"2300\"}\n",
    "                                close: \"2300\"\n",
    "                                isOpen: \"true\"\n",
    "                                open: \"0600\"\n",
    "                            th: {isOpen: \"true\", open: \"0600\", close: \"2300\"}\n",
    "                                close: \"2300\"\n",
    "                                isOpen: \"true\"\n",
    "                                open: \"0600\"\n",
    "                            tu: {isOpen: \"true\", open: \"0600\", close: \"2300\"}\n",
    "                                close: \"2300\"\n",
    "                                isOpen: \"true\"\n",
    "                                open: \"0600\"\n",
    "                            we: {isOpen: \"true\", open: \"0600\", close: \"2300\"}\n",
    "                                close: \"2300\"\n",
    "                                isOpen: \"true\"\n",
    "                                open: \"0600\"\n",
    "                        type: \"Trading\"\n",
    "        1: {location: {id: \"122a682a-34df-45f3-92d7-0cd3c4326067\", name: \"Norwich Wales Rd Express\",…},…}\n",
    "        2: {location: {id: \"8c7b5bce-0139-4831-8f58-7280d88f6ed4\", name: \"Norwich Westlegate Express\",…},…}\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " # Code for retrieving Tesco's opening times data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tesco_data(lat, lng):\n",
    "    API_URL = \"https://api.tesco.com/tescolocation/v3/locations/search\"\n",
    "    params = {  'offset' : 0,\n",
    "                'limit' : limit,\n",
    "                'sort' : 'near:\"{0},{1}\"'.format(lat,lng),\n",
    "                'filter' : \"category:Store AND isoCountryCode:x-uk\",\n",
    "                'fields' : \"name,geo,openingHours\"\n",
    "                #known fields: \"name,geo,openingHours,altIds.branchNumber,contact,facilities\"\n",
    "             }\n",
    "    headers = {\"x-appkey\": \"store-locator-web-cde\"}\n",
    "    rq = requests.get(API_URL, params=params, headers=headers)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.json())\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()['results']\n",
    "        if len(res) == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "    i=0\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (res[0]['location']['name'])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[i]['location']['name']:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    if res[i][\"distanceFrom\"][\"value\"] > 0.621371*radius:\n",
    "        return False\n",
    "    openingHours = res[i]['location']['openingHours'][0]['standardOpeningHours']\n",
    "    dayKeys = { 'mo' : 'Monday', 'tu' : 'Tuesday', 'we' : 'Wednesday', 'th' : 'Thursday', 'fr' : 'Friday', 'sa' : 'Saturday', 'su' : 'Sunday'}\n",
    "    hoursArray = []\n",
    "    for key in ['mo','tu','we','th','fr','sa','su']:\n",
    "        try:\n",
    "            if openingHours[key]['isOpen'] == 'true':\n",
    "                actualHours = {'open' : openingHours[key]['open'][:2] + \":\" + openingHours[key]['open'][2:],\n",
    "                            'close' : openingHours[key]['close'][:2] + \":\" + openingHours[key]['close'][2:]}\n",
    "                keyHours = {'day' : dayKeys[key],\n",
    "                        'open' : True,\n",
    "                        'hours' : [actualHours]\n",
    "                        }\n",
    "            else:\n",
    "                keyHours = {'day' : dayKeys[key],\n",
    "                        'open' : False}\n",
    "        except:\n",
    "            keyHours = {'day' : dayKeys[key],\n",
    "                        'open' : False}\n",
    "\n",
    "        hoursArray.append(keyHours)\n",
    "    return hoursArray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Morrison's API\n",
    "## Base URL:\n",
    "https://api.morrisons.com/location/v2//stores\n",
    "## Parameters:\n",
    "* distance = radius within which to search for store, measured in km (**tested**), website uses 50000 and returns at least 21 miles radius, so one assumes meters, so 50km radius.\n",
    "* lat = latitude\n",
    "* lon = longitude\n",
    "* limit = number of stores to display\n",
    "* offset = (?) wesbite uses 0\n",
    "* storeformat = \"supermarket\"\n",
    "\n",
    "## Headers:\n",
    "* apikey = one i used on first day: \"kxBdM2chFwZjNvG2PwnSn3sj6C53dLEY\" (appears to still work 2 days later, but still same location (**Note: further testing required**))\n",
    "\n",
    "## Sample API Call:\n",
    "<code>https://api.morrisons.com/location/v2//stores?apikey=kxBdM2chFwZjNvG2PwnSn3sj6C53dLEY&distance=50000&lat=52.62682342529297&limit=10&lon=1.3366719484329224&offset=0&storeformat=supermarket</code>\n",
    "### Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-1-3db5c2fcad70>, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-1-3db5c2fcad70>\"\u001b[1;36m, line \u001b[1;32m1\u001b[0m\n\u001b[1;33m    <code>{,…}\u001b[0m\n\u001b[1;37m    ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "{,…}\n",
    "    metadata: {limit: 10, count: 10, offset: 0, total: 12}\n",
    "    stores: [{name: 115, storeName: \"Norwich\", storeFormat: \"supermarket\", category: \"Supermarket\",…},…]\n",
    "        0: {name: 115, storeName: \"Norwich\", storeFormat: \"supermarket\", category: \"Supermarket\",…}\n",
    "            address: {addressLine1: \"4 Albion Way\", addressLine2: \"Riverside\", county: \"Norfolk\", city: \"Norwich\",…}\n",
    "                addressLine1: \"4 Albion Way\"\n",
    "                addressLine2: \"Riverside\"\n",
    "                city: \"Norwich\"\n",
    "                country: \"United Kingdom\"\n",
    "                county: \"Norfolk\"\n",
    "                postcode: \"NR1 1WR\"\n",
    "            category: \"Supermarket\"\n",
    "            distance: 2065\n",
    "            location: {latitude: 52.623662, longitude: 1.30663}\n",
    "                latitude: 52.623662\n",
    "                longitude: 1.30663\n",
    "            name: 115\n",
    "            openingTimes: {mon: {open: \"07:00:00\", close: \"22:00:00\"}, tue: {open: \"07:00:00\", close: \"22:00:00\"},…}\n",
    "                fri: {open: \"07:00:00\", close: \"22:00:00\"}\n",
    "                    close: \"22:00:00\"\n",
    "                    open: \"07:00:00\"\n",
    "                mon: {open: \"07:00:00\", close: \"22:00:00\"}\n",
    "                    close: \"22:00:00\"\n",
    "                    open: \"07:00:00\"\n",
    "                sat: {open: \"07:00:00\", close: \"22:00:00\"}\n",
    "                    close: \"22:00:00\"\n",
    "                    open: \"07:00:00\"\n",
    "                sun: {open: \"10:00:00\", close: \"16:00:00\"}\n",
    "                    close: \"16:00:00\"\n",
    "                    open: \"10:00:00\"\n",
    "                thu: {open: \"07:00:00\", close: \"22:00:00\"}\n",
    "                    close: \"22:00:00\"\n",
    "                    open: \"07:00:00\"\n",
    "                tue: {open: \"07:00:00\", close: \"22:00:00\"}\n",
    "                    close: \"22:00:00\"\n",
    "                    open: \"07:00:00\"\n",
    "                wed: {open: \"07:00:00\", close: \"22:00:00\"}\n",
    "                    close: \"22:00:00\"\n",
    "                    open: \"07:00:00\"\n",
    "            region: \"East England\"\n",
    "            satnav: {latitude: 52.623662, longitude: 1.30663}\n",
    "                latitude: 52.623662\n",
    "                longitude: 1.30663\n",
    "            storeFormat: \"supermarket\"\n",
    "            storeName: \"Norwich\"\n",
    "            telephone: \"01603 663999\"\n",
    "        1: {name: 385, storeName: \"Catton\", storeFormat: \"supermarket\", category: \"Supermarket\",…}\n",
    "        2: {name: 655, storeName: \"Wymondham\", storeFormat: \"supermarket\", category: \"Supermarket\",…}\n",
    "        3: {name: 275, storeName: \"Beccles\", storeFormat: \"supermarket\", category: \"Supermarket\",…}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Morrison's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_morrisons_data(lat, lng):\n",
    "    apikey = \"kxBdM2chFwZjNvG2PwnSn3sj6C53dLEY\"\n",
    "    API_URL = \"https://api.morrisons.com/location/v2//stores\"\n",
    "    params = { 'apikey' : apikey,\n",
    "                'distance' : radius*1000,\n",
    "                'lat' : lat,\n",
    "                'lon' : lng,\n",
    "                'limit' : limit,\n",
    "                'offset' : 0,\n",
    "                'storeformat' : \"supermarket\"\n",
    "             }\n",
    "    rq = requests.get(API_URL, params=params)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.json())\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()[\"stores\"]\n",
    "        if len(res) == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "    i=0\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (e.g. \"Morrisons\" + res[0][\"storename\"])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != \"Morrisons\" + res[i][\"storename\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    if res[i][\"distance\"] > radius * 1000:\n",
    "        return False \n",
    "    openingHours = res[i]['openingTimes']\n",
    "    dayKeys = { 'mon' : 'Monday', 'tue' : 'Tuesday', 'wed' : 'Wednesday', 'thu' : 'Thursday', 'fri' : 'Friday', 'sat' : 'Saturday', 'sun' : 'Sunday'}\n",
    "    hoursArray = []\n",
    "    for key in ['mon','tue','wed','thu','fri','sat','sun']:\n",
    "        try:\n",
    "            actualHours = {'open' : openingHours[key]['open'][:5],\n",
    "                        'close' : openingHours[key]['close'][:5]}\n",
    "            keyHours = {'day' : dayKeys[key],\n",
    "                    'open' : True,\n",
    "                    'hours' : [actualHours]\n",
    "                    }\n",
    "        except:\n",
    "            keyHours = {'day' : dayKeys[key],\n",
    "                        'open' : False}\n",
    "\n",
    "        hoursArray.append(keyHours)\n",
    "    return hoursArray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Waitrose's \"API\":\n",
    "Waitrose's system works very differently from the majority of other supermarkets'.\n",
    "Waitrose first calls their NearestBranchesCmd API, to obtain a list of the 6 nearest branches.\n",
    "Waitrose then calls their StandardWorkingHoursView API for the closest 3 of those stores, which returns an html encoded table(?) of the opening hours of that branch (see examples below).\n",
    "## NearestBranchesCmd API:\n",
    "### Base URL:\n",
    "https://www.waitrose.com/shop/NearestBranchesCmd\n",
    "### Parameters:\n",
    "* latitude = latitude\n",
    "* longitude = longitude\n",
    "* fromMultipleBranch = True (Note: True/False doesnt seem to make a difference) (optional)\n",
    "* _ = 1594910860791 (on the first day i used it, perhaps this is some form of API key??) (optional)\n",
    "### Sample API Call:\n",
    "https://www.waitrose.com/shop/NearestBranchesCmd?latitude=52.6054482&longitude=1.2412015&fromMultipleBranch=true&_=1594910860791\n",
    "https://www.waitrose.com/shop/NearestBranchesCmd?latitude=52.6470831&longitude=1.3433364&fromMultipleBranch=true&_=1594911360288\n",
    "### Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{success: true, branchList: [{branchId: 222, branchName: \"Norwich\", phoneNumber: \"01603 458114\",…},…]}\n",
    "    branchList: [{branchId: 222, branchName: \"Norwich\", phoneNumber: \"01603 458114\",…},…]\n",
    "        0: {branchId: 222, branchName: \"Norwich\", phoneNumber: \"01603 458114\",…}\n",
    "            addressLine1: \"Eaton Centre, Church Lane\"\n",
    "            branchDesc: \"Standard Branch\"\n",
    "            branchId: 222\n",
    "            branchName: \"Norwich\"\n",
    "            city: \"Eaton\"\n",
    "            isEntertaining: false\n",
    "            isGrocery: false\n",
    "            latitude: 52.60794\n",
    "            longitude: 1.25119\n",
    "            phoneNumber: \"01603 458114\"\n",
    "            postCode: \"NR4 6NU\"\n",
    "        1: {branchId: 662, branchName: \"Wymondham\", phoneNumber: \"01953 607772\", addressLine1: \"Norwich Road\",…}\n",
    "        2: {branchId: 622, branchName: \"North Walsham\", phoneNumber: \"01692 503109\", addressLine1: \"Cromer Road\",…}\n",
    "        3: {branchId: 686, branchName: \"Swaffham\", phoneNumber: \"01760 336307\", addressLine1: \"Castle Acre Road\",…}\n",
    "        4: {branchId: 483, branchName: \"Saxmundham\", phoneNumber: \"01728 603004\", addressLine1: \"Church Street\",…}\n",
    "        5: {branchId: 140, branchName: \"Bury St Edmunds\", phoneNumber: \"01284 725222\",…}\n",
    "    success: true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## StandardWorkingHours API:\n",
    "### Base URL:\n",
    "https://www.waitrose.com/shop/StandardWorkingHoursView\n",
    "### Parameters:\n",
    "* branchId = id of branch (retrieved from the NearestBranchesCmd API)\n",
    "* storeId = id of store (retrieved from the NearestBranchesCmd API)\n",
    "* linkType = \"standard\" (not sure what other options could be)\n",
    "* _ = 1594911360644 (on first try, once again unsure of what this is or does, but does seem to go up over quite quickly time, so perhaps a global counter?)\n",
    "\n",
    "### Sample API Call:\n",
    "https://www.waitrose.com/shop/StandardWorkingHoursView?branchId=222&storeId=10317&linkType=standard&_=1594911360644\n",
    "### Sample Response:\n",
    "#### Note:\n",
    "Edit this cell in order to see exact response received (the < pre > and < /pre > tags have been added in for formatting and were not part of the original reponse).\n",
    "<pre>\n",
    "\n",
    "<table>\n",
    "<tbody>\n",
    "\t\n",
    "\t\t\t <tr>\n",
    "\t\t\t \n",
    "\t\t\t <td class=\"day\">\n",
    "\t\t\t Sunday:\n",
    "\t\t\t </td>\n",
    " \t\t      \n",
    "\t\t\t  <td class=\"time\">\n",
    "\t\t\t    10:00 - 16:00\n",
    "\t\t\t  </td>\n",
    "\t\t\t  \n",
    "\t\t</tr>\n",
    "\t\n",
    "\t\t\t <tr>\n",
    "\t\t\t \n",
    "\t\t\t <td class=\"day\">\n",
    "\t\t\t Monday:\n",
    "\t\t\t </td>\n",
    " \t\t      \n",
    "\t\t\t  <td class=\"time\">\n",
    "\t\t\t    07:00 - 20:00\n",
    "\t\t\t  </td>\n",
    "\t\t\t  \n",
    "\t\t</tr>\n",
    "\t\n",
    "\t\t\t <tr>\n",
    "\t\t\t \n",
    "\t\t\t <td class=\"day\">\n",
    "\t\t\t Tuesday:\n",
    "\t\t\t </td>\n",
    " \t\t      \n",
    "\t\t\t  <td class=\"time\">\n",
    "\t\t\t    07:00 - 20:00\n",
    "\t\t\t  </td>\n",
    "\t\t\t  \n",
    "\t\t</tr>\n",
    "\t\n",
    "\t\t\t <tr>\n",
    "\t\t\t \n",
    "\t\t\t <td class=\"day\">\n",
    "\t\t\t Wednesday:\n",
    "\t\t\t </td>\n",
    " \t\t      \n",
    "\t\t\t  <td class=\"time\">\n",
    "\t\t\t    07:00 - 20:00\n",
    "\t\t\t  </td>\n",
    "\t\t\t  \n",
    "\t\t</tr>\n",
    "\t\n",
    "\t\t\t <tr>\n",
    "\t\t\t \n",
    "\t\t\t <td class=\"day\">\n",
    "\t\t\t Thursday:\n",
    "\t\t\t </td>\n",
    " \t\t      \n",
    "\t\t\t  <td class=\"time\">\n",
    "\t\t\t    07:00 - 20:00\n",
    "\t\t\t  </td>\n",
    "\t\t\t  \n",
    "\t\t</tr>\n",
    "\t\n",
    "\t\t\t <tr>\n",
    "\t\t\t \n",
    "\t\t\t <td class=\"day\">\n",
    "\t\t\t Friday:\n",
    "\t\t\t </td>\n",
    " \t\t      \n",
    "\t\t\t  <td class=\"time\">\n",
    "\t\t\t    07:00 - 20:00\n",
    "\t\t\t  </td>\n",
    "\t\t\t  \n",
    "\t\t</tr>\n",
    "\t\n",
    "\t\t\t <tr>\n",
    "\t\t\t \n",
    "\t\t\t <td class=\"day\">\n",
    "\t\t\t Saturday:\n",
    "\t\t\t </td>\n",
    " \t\t      \n",
    "\t\t\t  <td class=\"time\">\n",
    "\t\t\t    07:00 - 20:00\n",
    "\t\t\t  </td>\n",
    "\t\t\t  \n",
    "\t\t</tr>\n",
    "\t\n",
    "</tbody>\n",
    "</table>\n",
    "\n",
    "</pre>\n",
    "\n",
    "### Version obtained in plaintext via opening in Chrome (although in browser this does display with newlines):\n",
    "Sunday:\t10:00 - 16:00\n",
    "Monday:\t07:00 - 20:00\n",
    "Tuesday:\t07:00 - 20:00\n",
    "Wednesday:\t07:00 - 20:00\n",
    "Thursday:\t07:00 - 20:00\n",
    "Friday:\t07:00 - 20:00\n",
    "Saturday:\t07:00 - 20:00\n",
    "\n",
    "# Code for retreiving Waitrose's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_waitrose_data(lat, lng):\n",
    "    API_URL_locations = \"https://www.waitrose.com/shop/NearestBranchesCmd\"\n",
    "    API_URL_hours = \"https://www.waitrose.com/shop/StandardWorkingHoursView\"\n",
    "    params_locations = {'latitude' : lat,\n",
    "                        'longitude' : lng,\n",
    "                        }\n",
    "    rq = requests.get(API_URL_locations, params=params_locations)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.json())\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()\n",
    "        if not res['success'] or len(res['branchList']) == 0:\n",
    "            return False\n",
    "        res = res[\"branchList\"]\n",
    "    except:\n",
    "        return False\n",
    "    i=0\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (e.g. \"Waitrose\" + res[0][\"branchName\"])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != \"Waitrose\" + res[i][\"branchName\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    storeLat, storeLng = res[i][\"latitude\"], res[i][\"longitude\"]\n",
    "    #compute distance between the two points using the haversine function\n",
    "    storeDistance = haversine((lat, lng),(storeLat, storeLng))\n",
    "    if storeDistance > radius:\n",
    "        return False\n",
    "    branchId = res[i]['branchId']\n",
    "    params_hours = { 'branchId' : branchId}\n",
    "    rq = requests.get(API_URL_hours, params=params_hours)\n",
    "    hoursStringHTML = rq.text\n",
    "    cleanr = re.compile('<.*?>|\\s')\n",
    "    hoursString = re.sub(cleanr, '', hoursStringHTML)\n",
    "    #This section is to ensure if the shop is closed, correct times are displayed\n",
    "    sundayIndex = hoursString.find(\"Sunday:\")\n",
    "    mondayIndex = hoursString.find(\"Monday:\")\n",
    "    tuesdayIndex = hoursString.find(\"Tuesday:\")\n",
    "    wednesdayIndex = hoursString.find(\"Wednesday:\")\n",
    "    thursdayIndex = hoursString.find(\"Thursday:\")\n",
    "    fridayIndex = hoursString.find(\"Friday:\")\n",
    "    saturdayIndex = hoursString.find(\"Saturday:\")\n",
    "\n",
    "    hoursArray = []\n",
    "    #equivalent of the usual for loop V\n",
    "    if mondayIndex == -1:\n",
    "        keyHours = {'day' : \"Monday\",\n",
    "                        'open' : False}\n",
    "        hoursArray.append(keyHours)\n",
    "    else:\n",
    "        try:\n",
    "            day = hoursString[mondayIndex+7:tuesdayIndex]\n",
    "            if day.find(\"los\") != -1: #check if the world close or Close or closed or Closed or Closure etc.. appears\n",
    "                keyHours = {'day' : \"Monday\",\n",
    "                        'open' : False}\n",
    "                hoursArray.append(keyHours)\n",
    "            firstDigitIndex = [x.isdigit() for x in day].index(True)\n",
    "            actualHours = { 'open' : day[firstDigitIndex:firstDigitIndex+5],\n",
    "                            'close' : day[firstDigitIndex+6:firstDigitIndex+11]}\n",
    "            keyHours = {'day' : \"Monday\",\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours}\n",
    "            hoursArray.append(keyHours)\n",
    "        except:\n",
    "            keyHours = {'day' : \"Monday\",\n",
    "                        'open' : False}\n",
    "            hoursArray.append(keyHours)\n",
    "            \n",
    "\n",
    "    if tuesdayIndex == -1:\n",
    "        keyHours = {'day' : \"Tuesday\",\n",
    "                        'open' : False}\n",
    "        hoursArray.append(keyHours)\n",
    "    else:\n",
    "        try:\n",
    "            day = hoursString[tuesdayIndex+8:wednesdayIndex]\n",
    "            if day.find(\"los\") != -1: #check if the world close or Close or closed or Closed or Closure etc.. appears\n",
    "                keyHours = {'day' : \"Tuesday\",\n",
    "                        'open' : False}\n",
    "                hoursArray.append(keyHours)\n",
    "            firstDigitIndex = [x.isdigit() for x in day].index(True)\n",
    "            actualHours = { 'open' : day[firstDigitIndex:firstDigitIndex+5],\n",
    "                            'close' : day[firstDigitIndex+6:firstDigitIndex+11]}\n",
    "            keyHours = {'day' : \"Tuesday\",\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours}\n",
    "            hoursArray.append(keyHours)\n",
    "        except:\n",
    "            keyHours = {'day' : \"Tuesday\",\n",
    "                        'open' : False}\n",
    "            hoursArray.append(keyHours)\n",
    "    if wednesdayIndex == -1:\n",
    "        keyHours = {'day' : \"Wednesday\",\n",
    "                        'open' : False}\n",
    "        hoursArray.append(keyHours)\n",
    "    else:\n",
    "        try:\n",
    "            day = hoursString[wednesdayIndex+10:thursdayIndex]\n",
    "            if day.find(\"los\") != -1: #check if the world close or Close or closed or Closed or Closure etc.. appears\n",
    "                keyHours = {'day' : \"Wednesday\",\n",
    "                        'open' : False}\n",
    "                hoursArray.append(keyHours)\n",
    "            firstDigitIndex = [x.isdigit() for x in day].index(True)\n",
    "            actualHours = { 'open' : day[firstDigitIndex:firstDigitIndex+5],\n",
    "                            'close' : day[firstDigitIndex+6:firstDigitIndex+11]}\n",
    "            keyHours = {'day' : \"Wednesday\",\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours}\n",
    "            hoursArray.append(keyHours)\n",
    "        except:\n",
    "            keyHours = {'day' : \"Wednesday\",\n",
    "                        'open' : False}\n",
    "            hoursArray.append(keyHours)\n",
    "        \n",
    "    if thursdayIndex == -1:\n",
    "        keyHours = {'day' : \"Thursday\",\n",
    "                        'open' : False}\n",
    "        hoursArray.append(keyHours)\n",
    "    else:\n",
    "        try:\n",
    "            day = hoursString[thursdayIndex+9:fridayIndex]\n",
    "            if day.find(\"los\") != -1: #check if the world close or Close or closed or Closed or Closure etc.. appears\n",
    "                keyHours = {'day' : \"Thursday\",\n",
    "                        'open' : False}\n",
    "                hoursArray.append(keyHours)\n",
    "            firstDigitIndex = [x.isdigit() for x in day].index(True)\n",
    "            actualHours = { 'open' : day[firstDigitIndex:firstDigitIndex+5],\n",
    "                            'close' : day[firstDigitIndex+6:firstDigitIndex+11]}\n",
    "            keyHours = {'day' : \"Thursday\",\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours}\n",
    "            hoursArray.append(keyHours)\n",
    "        except:\n",
    "            keyHours = {'day' : \"Thursday\",\n",
    "                        'open' : False}\n",
    "            hoursArray.append(keyHours)\n",
    "    if fridayIndex == -1:\n",
    "        keyHours = {'day' : \"Friday\",\n",
    "                        'open' : False}\n",
    "        hoursArray.append(keyHours)\n",
    "    else:\n",
    "        try:\n",
    "            day = hoursString[fridayIndex+7:saturdayIndex]\n",
    "            if day.find(\"los\") != -1: #check if the world close or Close or closed or Closed or Closure etc.. appears\n",
    "                keyHours = {'day' : \"Friday\",\n",
    "                        'open' : False}\n",
    "                hoursArray.append(keyHours)\n",
    "            firstDigitIndex = [x.isdigit() for x in day].index(True)\n",
    "            actualHours = { 'open' : day[firstDigitIndex:firstDigitIndex+5],\n",
    "                            'close' : day[firstDigitIndex+6:firstDigitIndex+11]}\n",
    "            keyHours = {'day' : \"Friday\",\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours}\n",
    "            hoursArray.append(keyHours)\n",
    "        except:\n",
    "            keyHours = {'day' : \"Friday\",\n",
    "                        'open' : False}\n",
    "            hoursArray.append(keyHours)\n",
    "    if saturdayIndex == -1:\n",
    "        keyHours = {'day' : \"Saturday\",\n",
    "                        'open' : False}\n",
    "        hoursArray.append(keyHours)\n",
    "    else:\n",
    "        try:\n",
    "            day = hoursString[saturdayIndex+9:]\n",
    "            if day.find(\"los\") != -1: #check if the world close or Close or closed or Closed or Closure etc.. appears\n",
    "                keyHours = {'day' : \"Saturday\",\n",
    "                        'open' : False}\n",
    "                hoursArray.append(keyHours)\n",
    "            firstDigitIndex = [x.isdigit() for x in day].index(True)\n",
    "            actualHours = { 'open' : day[firstDigitIndex:firstDigitIndex+5],\n",
    "                            'close' : day[firstDigitIndex+6:firstDigitIndex+11]}\n",
    "            keyHours = {'day' : \"Saturday\",\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours}\n",
    "            hoursArray.append(keyHours)\n",
    "        except:\n",
    "            keyHours = {'day' : \"Saturday\",\n",
    "                        'open' : False}\n",
    "            hoursArray.append(keyHours)\n",
    "    if sundayIndex == -1:\n",
    "        keyHours = {'day' : \"Sunday\",\n",
    "                        'open' : False}\n",
    "        hoursArray.append(keyHours)\n",
    "    else:\n",
    "        try:\n",
    "            day = hoursString[sundayIndex+7:mondayIndex]\n",
    "            if day.find(\"los\") != -1: #check if the world close or Close or closed or Closed or Closure etc.. appears\n",
    "                keyHours = {'day' : \"Sunday\",\n",
    "                        'open' : False}\n",
    "                hoursArray.append(keyHours)\n",
    "            firstDigitIndex = [x.isdigit() for x in day].index(True)\n",
    "            actualHours = { 'open' : day[firstDigitIndex:firstDigitIndex+5],\n",
    "                            'close' : day[firstDigitIndex+6:firstDigitIndex+11]}\n",
    "            keyHours = {'day' : \"Sunday\",\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours}\n",
    "            hoursArray.append(keyHours)\n",
    "        except:\n",
    "            keyHours = {'day' : \"Sunday\",\n",
    "                        'open' : False}\n",
    "            hoursArray.append(keyHours)\n",
    "    return hoursArray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aldi's API:\n",
    "## Base URL:\n",
    "https://www.aldi.co.uk/api/store-finder/search\n",
    "## Parameters:\n",
    "* q = postcode (optional)\n",
    "* address = Road, Postcode (optional)\n",
    "* latitude = latidude\n",
    "* longitude = longitude\n",
    "\n",
    "## Example API call:\n",
    "https://www.aldi.co.uk/api/store-finder/search?q=nr70en&address=Yarmouth+Rd%2C+Norwich+NR7+0EN&latitude=52.6266168&longitude=1.3366618\n",
    "## Example response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{query: \"nr70en\", address: \"Yarmouth Rd, Norwich NR7 0EN\",…}\n",
    "    address: \"Yarmouth Rd, Norwich NR7 0EN\"\n",
    "    bounds: {n: \"52.64952\", e: \"1.4334135999999997\", s: \"52.6037136\", w: \"1.23991\"}\n",
    "        e: \"1.4334135999999997\"\n",
    "        n: \"52.64952\"\n",
    "        s: \"52.6037136\"\n",
    "        w: \"1.23991\"\n",
    "    center: {lat: \"52.6266168\", lng: \"1.3366618\"}\n",
    "        lat: \"52.6266168\"\n",
    "        lng: \"1.3366618\"\n",
    "    formattedCount: \"Nearest <strong>30 stores</strong> found for <strong>Yarmouth Rd, Norwich NR7 0EN</strong>\"\n",
    "    pagination: {next: 1,…}\n",
    "        next: 1\n",
    "        pages: [{page: 0, prettyPage: 1, currentPage: true, center: {lat: \"52.6266168\", lng: \"1.3366618\"},…},…]\n",
    "            0: {page: 0, prettyPage: 1, currentPage: true, center: {lat: \"52.6266168\", lng: \"1.3366618\"},…}\n",
    "                address: \"Yarmouth Rd, Norwich NR7 0EN\"\n",
    "                center: {lat: \"52.6266168\", lng: \"1.3366618\"}\n",
    "                    lat: \"52.6266168\"\n",
    "                    lng: \"1.3366618\"\n",
    "                currentPage: true\n",
    "                page: 0\n",
    "                prettyPage: 1\n",
    "                query: \"nr70en\"\n",
    "            1: {page: 1, prettyPage: 2, center: {lat: \"52.6266168\", lng: \"1.3366618\"}, query: \"nr70en\",…}\n",
    "            2: {page: 2, prettyPage: 3, center: {lat: \"52.6266168\", lng: \"1.3366618\"}, query: \"nr70en\",…}\n",
    "            3: {page: 3, prettyPage: 4, center: {lat: \"52.6266168\", lng: \"1.3366618\"}, query: \"nr70en\",…}\n",
    "            4: {page: 4, prettyPage: 5, center: {lat: \"52.6266168\", lng: \"1.3366618\"}, query: \"nr70en\",…}\n",
    "            5: {page: 5, prettyPage: 6, center: {lat: \"52.6266168\", lng: \"1.3366618\"}, query: \"nr70en\",…}\n",
    "    query: \"nr70en\"\n",
    "    results: [,…]\n",
    "        0: {code: \"s-uk-E0566\", name: \"ALDI - 174-178 Plumstead Road\", distance: \"0.8 Miles\", description: \"\",…}\n",
    "            address: [\"Norwich\", \"NR1 4JZ\"]\n",
    "                0: \"Norwich\"\n",
    "                1: \"NR1 4JZ\"\n",
    "            code: \"s-uk-E0566\"\n",
    "            description: \"\"\n",
    "            distance: \"0.8 Miles\"\n",
    "            features: [\"parking\"]\n",
    "                0: \"parking\"\n",
    "            isFirstStore: true\n",
    "            latlng: {lat: 52.6366, lng: 1.32667}\n",
    "                lat: 52.6366\n",
    "                lng: 1.32667\n",
    "            name: \"ALDI - 174-178 Plumstead Road\"\n",
    "            openingTimes: [{day: \"Mon\", hours: \"08:00&nbsp;&ndash;&nbsp;22:00\", closed: false, special: false},…]\n",
    "                0: {day: \"Mon\", hours: \"08:00&nbsp;&ndash;&nbsp;22:00\", closed: false, special: false}\n",
    "                    closed: false\n",
    "                    day: \"Mon\"\n",
    "                    hours: \"08:00&nbsp;&ndash;&nbsp;22:00\"\n",
    "                    special: false\n",
    "                1: {day: \"Tue\", hours: \"08:00&nbsp;&ndash;&nbsp;22:00\", closed: false, special: false}\n",
    "                    closed: false\n",
    "                    day: \"Tue\"\n",
    "                    hours: \"08:00&nbsp;&ndash;&nbsp;22:00\"\n",
    "                    special: false\n",
    "                2: {day: \"Wed\", hours: \"08:00&nbsp;&ndash;&nbsp;22:00\", closed: false, special: false}\n",
    "                    closed: false\n",
    "                    day: \"Wed\"\n",
    "                    hours: \"08:00&nbsp;&ndash;&nbsp;22:00\"\n",
    "                    special: false\n",
    "                3: {day: \"Thu\", hours: \"08:00&nbsp;&ndash;&nbsp;22:00\", closed: false, special: false}\n",
    "                    closed: false\n",
    "                    day: \"Thu\"\n",
    "                    hours: \"08:00&nbsp;&ndash;&nbsp;22:00\"\n",
    "                    special: false\n",
    "                4: {day: \"Fri\", hours: \"08:00&nbsp;&ndash;&nbsp;22:00\", closed: false, special: false}\n",
    "                    closed: false\n",
    "                    day: \"Fri\"\n",
    "                    hours: \"08:00&nbsp;&ndash;&nbsp;22:00\"\n",
    "                    special: false\n",
    "                5: {day: \"Sat\", hours: \"08:00&nbsp;&ndash;&nbsp;22:00\", closed: false, special: false}\n",
    "                    closed: false\n",
    "                    day: \"Sat\"\n",
    "                    hours: \"08:00&nbsp;&ndash;&nbsp;22:00\"\n",
    "                    special: false\n",
    "                6: {day: \"Sun\", hours: \"10:00&nbsp;&ndash;&nbsp;16:00\", closed: false, special: false}\n",
    "                    closed: false\n",
    "                    day: \"Sun\"\n",
    "                    hours: \"10:00&nbsp;&ndash;&nbsp;16:00\"\n",
    "                    special: false\n",
    "            selectedPage: \"0\"\n",
    "        1: {code: \"s-uk-E0487\", name: \"ALDI - 463 Sprowston Road\", distance: \"2 Miles\", description: \"\",…}\n",
    "        2: {code: \"s-uk-E1022\", name: \"ALDI - Unit 5, Hall Road\", distance: \"2.3 Miles\", description: \"\",…}\n",
    "        3: {code: \"s-uk-E1091\", name: \"ALDI - Unit1, 36 - 50 Drayton Road\", distance: \"2.6 Miles\",…}\n",
    "        4: {code: \"s-uk-E0377\", name: \"ALDI - 1 Larkman Lane\", distance: \"4.1 Miles\", description: \"\",…}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retreiving Aldi's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_aldi_data(lat, lng):\n",
    "    API_URL = \"https://www.aldi.co.uk/api/store-finder/search\"\n",
    "    params = {  'latitude' : lat,\n",
    "                'longitude' : lng,\n",
    "                'fromMultipleBranch' : False\n",
    "             }\n",
    "    rq = requests.get(API_URL, params=params)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.json())\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()[\"results\"]\n",
    "        if len(res) == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "    i=0\n",
    "    intermediaryDayKeys = { 'Mon' : 0, 'Tue' : 1, 'Wed' : 2, 'Thu' : 3, 'Fri' : 4, 'Sat' : 5, 'Sun' : 6}\n",
    "    dayKeys = { 0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday', 5 : 'Saturday', 6 : 'Sunday'}\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (e.g. res[0][\"name\"])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[i][\"name\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    distance = float(res[i][\"distance\"].split(\" \")[0]) * 1.60934\n",
    "    if distance > radius:\n",
    "        return False\n",
    "    openingHours = res[i][\"openingTimes\"]\n",
    "    hoursArray = []\n",
    "    dayDoneSet = set()\n",
    "    daysHeap = []\n",
    "    for index in range(len(openingHours)):\n",
    "            day = intermediaryDayKeys[openingHours[index][\"day\"]]\n",
    "            #if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "            if day not in dayDoneSet:\n",
    "                dayDoneSet.add(day)\n",
    "                #keyHours processing code\n",
    "                key = intermediaryDayKeys[openingHours[index]['day']]  #turn key into int\n",
    "                actualHours = []\n",
    "                if openingHours[index]['closed'] == False:\n",
    "                    hoursString = html.unescape(openingHours[index]['hours'])\n",
    "                    while len(hoursString) > 11: #incase there are multiple time slots encoded within the string (format is unknown hence the strange code)\n",
    "                        timeSlot = { 'open' : hoursString[:5],\n",
    "                                        'close' : hoursString[8:13]}\n",
    "                        actualHours.append(timeSlot)\n",
    "                        hoursString = hoursString[13:]\n",
    "                        if len(hoursString) > 0:\n",
    "                            try:\n",
    "                                firstDigitIndex = [x.isdigit() for x in hoursString].index(True)\n",
    "                                hoursString = hoursString[firstDigitIndex:]\n",
    "                            except:\n",
    "                                break\n",
    "                    keyHours = {'day' : dayKeys[key],\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours\n",
    "                        }\n",
    "                else:\n",
    "                    keyHours = {'day' : dayKeys[key],\n",
    "                        'open' : False\n",
    "                        }\n",
    "                heapq.heappush(daysHeap, [key, keyHours])\n",
    "    #Check for any missing days and ensure all days are in order for insertion into our database\n",
    "    for missingDay in set(dayKeys.keys()).difference(dayDoneSet):\n",
    "        closedDayHours = {\n",
    "            \"day\": dayKeys[missingDay],\n",
    "            \"open\": False\n",
    "        }\n",
    "        heapq.heappush(daysHeap, [missingDay, closedDayHours])\n",
    "    if len(daysHeap) != 7:\n",
    "        return False\n",
    "    hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "    return hoursArray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Co-op's API\n",
    "## Base URL:\n",
    "https://api.coop.co.uk/locationservices/finder/food/\n",
    "## Parameters:\n",
    "* location = \"Latitude,Longitude\"\n",
    "* distance = maximum distance to look for stores within (m)\n",
    "* min_distance = minimum distance to look for stores within (m)\n",
    "* min_results = amount of results that must (at least) be returned, if not radius is increased until a store is found(?) ( **WARNING:** if this occurs, a bug in their API occurs and day indexing in the response is reversed with sunday being 0 and monday 6, therefore setting min_results = 0 is recommended.\n",
    "(**Note:** from version 9 of this project onwards, after the redesign of various functions using heaps instead, this no longer causes an issue (as heaps were implemented specifically to deal with out of order days better while maintaining the array format of hoursArray)))\n",
    "* format = \"json\"\n",
    "## Sample API Call:\n",
    "https://api.coop.co.uk/locationservices/finder/food/?location=52.6470831%2C1.3433364&distance=3000&min_distance=0&min_results=1&format=json\n",
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count: 4\n",
    "next: null\n",
    "previous: null\n",
    "results: [{name: \"Heartsease\", location_type: \"Food\", society: \"East of England Co-operative Society\",…},…]\n",
    "    0: {name: \"Heartsease\", location_type: \"Food\", society: \"East of England Co-operative Society\",…}\n",
    "        amazon_counter_shortname: \"\"\n",
    "        anacode: \"5000128514185\"\n",
    "        cedar: \"\"\n",
    "        county: \"Norfolk\"\n",
    "        distance: {m: 810}\n",
    "            m: 810\n",
    "        email: null\n",
    "        facilities: []\n",
    "        hubnumber: null\n",
    "        image: null\n",
    "        location_type: \"Food\"\n",
    "        name: \"Heartsease\"\n",
    "        open_status: \"Open\"\n",
    "        open_status_label: \"Open until 10pm\"\n",
    "        opening_hours: [{name: \"Monday\", label: \"7am to 10pm\", type: \"hours\", opens: \"07:00\", closes: \"22:00\"},…]\n",
    "            0: {name: \"Monday\", label: \"7am to 10pm\", type: \"hours\", opens: \"07:00\", closes: \"22:00\"}\n",
    "                closes: \"22:00\"\n",
    "                label: \"7am to 10pm\"\n",
    "                name: \"Monday\"\n",
    "                opens: \"07:00\"\n",
    "                type: \"hours\"\n",
    "            1: {name: \"Tuesday\", label: \"7am to 10pm\", type: \"hours\", opens: \"07:00\", closes: \"22:00\"}\n",
    "                closes: \"22:00\"\n",
    "                label: \"7am to 10pm\"\n",
    "                name: \"Tuesday\"\n",
    "                opens: \"07:00\"\n",
    "                type: \"hours\"\n",
    "            2: {name: \"Wednesday\", label: \"7am to 10pm\", type: \"hours\", opens: \"07:00\", closes: \"22:00\"}\n",
    "                closes: \"22:00\"\n",
    "                label: \"7am to 10pm\"\n",
    "                name: \"Wednesday\"\n",
    "                opens: \"07:00\"\n",
    "                type: \"hours\"\n",
    "            3: {name: \"Thursday\", label: \"7am to 10pm\", type: \"hours\", opens: \"07:00\", closes: \"22:00\"}\n",
    "                closes: \"22:00\"\n",
    "                label: \"7am to 10pm\"\n",
    "                name: \"Thursday\"\n",
    "                opens: \"07:00\"\n",
    "                type: \"hours\"\n",
    "            4: {name: \"Friday\", label: \"7am to 10pm\", type: \"hours\", opens: \"07:00\", closes: \"22:00\"}\n",
    "                closes: \"22:00\"\n",
    "                label: \"7am to 10pm\"\n",
    "                name: \"Friday\"\n",
    "                opens: \"07:00\"\n",
    "                type: \"hours\"\n",
    "            5: {name: \"Saturday\", label: \"7am to 10pm\", type: \"hours\", opens: \"07:00\", closes: \"22:00\"}\n",
    "                closes: \"22:00\"\n",
    "                label: \"7am to 10pm\"\n",
    "                name: \"Saturday\"\n",
    "                opens: \"07:00\"\n",
    "                type: \"hours\"\n",
    "            6: {name: \"Sunday\", label: \"7am to 10pm\", type: \"hours\", opens: \"07:00\", closes: \"22:00\"}\n",
    "                closes: \"22:00\"\n",
    "                label: \"7am to 10pm\"\n",
    "                name: \"Sunday\"\n",
    "                opens: \"07:00\"\n",
    "                type: \"hours\"\n",
    "        phone: \"01603 433127\"\n",
    "        position: {x: 1.33706, y: 52.64088, srid: 4326}\n",
    "            srid: 4326\n",
    "            x: 1.33706\n",
    "            y: 52.64088\n",
    "        postcode: \"NR7 9DX\"\n",
    "        public: true\n",
    "        seasonal_hours: null\n",
    "        society: \"East of England Co-operative Society\"\n",
    "        street_address: \"Witard Road\"\n",
    "        street_address2: \"Heartease Estate\"\n",
    "        street_address3: \"\"\n",
    "        temporarily_closed: false\n",
    "        town: \"Norwich\"\n",
    "        url: \"/food/store/NR7-9DX/witard-road\"\n",
    "        website: \"\"\n",
    "1: {name: \"Thorpe St Andrew\", location_type: \"Food\", society: \"East of England Co-operative Society\",…}\n",
    "2: {name: \"Cannerby Lane\", location_type: \"Food\", society: \"East of England Co-operative Society\",…}\n",
    "3: {name: \"Norwich - Sprowston Road\", location_type: \"Food\", society: \"The Co-operative Group\",…}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Co-op's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_coop_data(lat, lng):\n",
    "    API_URL = \"https://api.coop.co.uk/locationservices/finder/food/\"\n",
    "    params = {  'location' : \"{0},{1}\".format(lat,lng),\n",
    "                'distance' : radius*1000,\n",
    "                'min_distance' : \"0\",\n",
    "                'min_results' : \"0\",\n",
    "                'format' : \"json\"}\n",
    "    rq = requests.get(API_URL, params=params)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.text)\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()\n",
    "        if res['count'] == 0:\n",
    "            return False\n",
    "        res = res[\"results\"]\n",
    "    except:\n",
    "        return False\n",
    "    i=0\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (e.g. res[0][\"name\"])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[i][\"name\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    if res[i][\"distance\"][\"m\"] > radius * 1000:\n",
    "        return False\n",
    "    openingHours = res[i][\"opening_hours\"]\n",
    "    intermediaryDayKeys = { 'Monday' : 0, 'Tuesday' : 1, 'Wednesday' : 2, 'Thursday' : 3, 'Friday' : 4, 'Saturday' : 5, 'Sunday' : 6}\n",
    "    dayKeys = { 0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday', 5 : 'Saturday', 6 : 'Sunday'}\n",
    "    dayDoneSet = set()\n",
    "    daysHeap = []\n",
    "    for index in range(len(openingHours)):\n",
    "            key = intermediaryDayKeys[openingHours[index]['name']]  #turn key into int\n",
    "            #if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "            if key not in dayDoneSet:\n",
    "                dayDoneSet.add(key)\n",
    "                #keyHours processing code\n",
    "                actualHours = []\n",
    "                try:\n",
    "                    if openingHours[index][\"label\"].find(\"los\") != -1:\n",
    "                        #incase the world closed/closure/Closed/Closure is in the label\n",
    "                        closedDayHours = {  'day' : dayKeys[key],\n",
    "                                    'open' : False  }\n",
    "                        heapq.heappush(daysHeap, [key, closedDayHours])\n",
    "                    else:\n",
    "                        actualHoursDict = {}\n",
    "                        actualHoursDict['open'] = openingHours[index]['opens']\n",
    "                        actualHoursDict['close'] = openingHours[index]['closes']\n",
    "                        actualHours = [actualHoursDict]\n",
    "                        keyHours = {'day' : dayKeys[key],\n",
    "                            'open' : True,\n",
    "                            'hours' : actualHours\n",
    "                            }\n",
    "                        heapq.heappush(daysHeap, [key, keyHours])\n",
    "                except:\n",
    "                    #If dictionary keys start_time or end_time dont exist, assume store is closed on that day\n",
    "                    closedDayHours = {  'day' : dayKeys[key],\n",
    "                                    'open' : False  }\n",
    "                    heapq.heappush(daysHeap, [key, closedDayHours])\n",
    "    # Check for any missing days and ensure all days are in order for insertion into our database\n",
    "    for missingDay in set(dayKeys.keys()).difference(dayDoneSet):\n",
    "        closedDayHours = {\n",
    "            \"day\": dayKeys[missingDay],\n",
    "            \"open\": False\n",
    "        }\n",
    "        heapq.heappush(daysHeap, [missingDay, closedDayHours])\n",
    "    if len(daysHeap) != 7:\n",
    "        return False\n",
    "    hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "    return hoursArray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Marks and Spencer's API\n",
    "## Base URL:\n",
    "https://api.marksandspencer.com/v1/stores\n",
    "\n",
    "## Parameters:\n",
    "* apikey = \"aVCi8dmPbHgHrdCv9gNt6rusFK98VokK\" on the first day (**required**)\n",
    "* jsonp = Appears to put a plaintext prefix before the json response. Website uses \"angular.callbacks._1\" (optional)\n",
    "* latlong = \"Latitude,Longitude\"\n",
    "* limit = number of stores to return\n",
    "* radius = radius within which to look for stores in (miles or km)\n",
    "\n",
    "## Sample API Call:\n",
    "https://api.marksandspencer.com/v1/stores?apikey=aVCi8dmPbHgHrdCv9gNt6rusFK98VokK&jsonp=angular.callbacks._1&latlong=52.64497375488281,1.2892669439315796&limit=10&radius=40\n",
    "## Sample Response:\n",
    "**Note:** the 0'th store is a BP Garage, and the 2nd store is a more usual M&S store, therefore the 2 results are both displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "angular.callbacks._1({results: [{id: 1977, name: \"MOUSEHOLD BP\", storeType: \"franchise\",…},…], count: 10});\n",
    "    count: 10\n",
    "    results: [{id: 1977, name: \"MOUSEHOLD BP\", storeType: \"franchise\",…},…]\n",
    "        0: {id: 1977, name: \"MOUSEHOLD BP\", storeType: \"franchise\",…}\n",
    "            address: {addressLine1: \"MOUSEHOLD BP\", addressLine2: \"58 Mousehold Lane\", city: \"NORWICH\",…}\n",
    "                addressLine1: \"MOUSEHOLD BP\"\n",
    "                addressLine2: \"58 Mousehold Lane\"\n",
    "                city: \"NORWICH\"\n",
    "                country: \"United Kingdom\"\n",
    "                isoTwoCountryCode: \"GB\"\n",
    "                postalCode: \"NR7 8HA\"\n",
    "            coordinates: {latitude: 52.648823, longitude: 1.31477}\n",
    "                latitude: 52.648823\n",
    "                longitude: 1.31477\n",
    "            coreOpeningHours: [{day: \"Monday\", open: \"00:00\", close: \"24:00\"}, {day: \"Tuesday\", open: \"00:00\", close: \"24:00\"},…]\n",
    "                0: {day: \"Monday\", open: \"00:00\", close: \"24:00\"}\n",
    "                    close: \"24:00\"\n",
    "                    day: \"Monday\"\n",
    "                    open: \"00:00\"\n",
    "                1: {day: \"Tuesday\", open: \"00:00\", close: \"24:00\"}\n",
    "                    close: \"24:00\"\n",
    "                    day: \"Tuesday\"\n",
    "                    open: \"00:00\"\n",
    "                2: {day: \"Wednesday\", open: \"00:00\", close: \"24:00\"}\n",
    "                    close: \"24:00\"\n",
    "                    day: \"Wednesday\"\n",
    "                    open: \"00:00\"\n",
    "                3: {day: \"Thursday\", open: \"00:00\", close: \"24:00\"}\n",
    "                    close: \"24:00\"\n",
    "                    day: \"Thursday\"\n",
    "                    open: \"00:00\"\n",
    "                4: {day: \"Friday\", open: \"00:00\", close: \"24:00\"}\n",
    "                    close: \"24:00\"\n",
    "                    day: \"Friday\"\n",
    "                    open: \"00:00\"\n",
    "                5: {day: \"Saturday\", open: \"00:00\", close: \"24:00\"}\n",
    "                    close: \"24:00\"\n",
    "                    day: \"Saturday\"\n",
    "                    open: \"00:00\"\n",
    "                6: {day: \"Sunday\", open: \"00:00\", close: \"24:00\"}\n",
    "                    close: \"24:00\"\n",
    "                    day: \"Sunday\"\n",
    "                    open: \"00:00\"\n",
    "            departments: []\n",
    "            distance: 1778.482835104\n",
    "            facilities: []\n",
    "            id: 1977\n",
    "            isChristmasOrderAccepted: false\n",
    "            isFoodCollectionSupported: false\n",
    "            isGmCollectionSupported: false\n",
    "            name: \"MOUSEHOLD BP\"\n",
    "            phone: \"01603 408611\"\n",
    "            services: []\n",
    "            specialOpeningHours: []\n",
    "            storeType: \"franchise\"\n",
    "        1: {id: 2642, name: \"NORWICH\", storeType: \"mands\", coordinates: {latitude: 52.62612, longitude: 1.29328},…}\n",
    "        2: {id: 1297, name: \"NORWICH SWEET BRIAR SIMPLY FOOD\", storeType: \"mands\",…}\n",
    "            address: {addressLine1: \"NORWICH SWEET BRIAR SIMPLY FOOD\", addressLine2: \"UNIT 1A SWEET BRIAR NORWICH RETAIL\",…}\n",
    "                addressLine1: \"NORWICH SWEET BRIAR SIMPLY FOOD\"\n",
    "                addressLine2: \"UNIT 1A SWEET BRIAR NORWICH RETAIL\"\n",
    "                city: \"NORWICH\"\n",
    "                country: \"United Kingdom\"\n",
    "                county: \"NK\"\n",
    "                isoTwoCountryCode: \"GB\"\n",
    "                postalCode: \"NR6 5DH\"\n",
    "            coordinates: {latitude: 52.651409, longitude: 1.259467}\n",
    "                latitude: 52.651409\n",
    "                longitude: 1.259467\n",
    "            coreOpeningHours: [{day: \"Monday\", open: \"08:00\", close: \"21:00\"}, {day: \"Tuesday\", open: \"08:00\", close: \"21:00\"},…]\n",
    "                0: {day: \"Monday\", open: \"08:00\", close: \"21:00\"}\n",
    "                    close: \"21:00\"\n",
    "                    day: \"Monday\"\n",
    "                    open: \"08:00\"\n",
    "                1: {day: \"Tuesday\", open: \"08:00\", close: \"21:00\"}\n",
    "                    close: \"21:00\"\n",
    "                    day: \"Tuesday\"\n",
    "                    open: \"08:00\"\n",
    "                2: {day: \"Wednesday\", open: \"08:00\", close: \"21:00\"}\n",
    "                    close: \"21:00\"\n",
    "                    day: \"Wednesday\"\n",
    "                    open: \"08:00\"\n",
    "                3: {day: \"Thursday\", open: \"08:00\", close: \"21:00\"}\n",
    "                    close: \"21:00\"\n",
    "                    day: \"Thursday\"\n",
    "                    open: \"08:00\"\n",
    "                4: {day: \"Friday\", open: \"08:00\", close: \"21:00\"}\n",
    "                    close: \"21:00\"\n",
    "                    day: \"Friday\"\n",
    "                    open: \"08:00\"\n",
    "                5: {day: \"Saturday\", open: \"08:00\", close: \"21:00\"}\n",
    "                    close: \"21:00\"\n",
    "                    day: \"Saturday\"\n",
    "                    open: \"08:00\"\n",
    "                6: {day: \"Sunday\", open: \"10:30\", close: \"16:30\"}\n",
    "                    close: \"16:30\"\n",
    "                    day: \"Sunday\"\n",
    "                    open: \"10:30\"\n",
    "            departments: [{id: \"DPT_BKRY\", name: \"Bakery\"}, {id: \"DPT_CARD\", name: \"Cards & wrap\"},…]\n",
    "                0: {id: \"DPT_BKRY\", name: \"Bakery\"}\n",
    "                    id: \"DPT_BKRY\"\n",
    "                    name: \"Bakery\"\n",
    "                1: {id: \"DPT_CARD\", name: \"Cards & wrap\"}\n",
    "                    id: \"DPT_CARD\"\n",
    "                    name: \"Cards & wrap\"\n",
    "                2: {id: \"DPT_FWPLNT\", name: \"Flowers & plants\"}\n",
    "                    id: \"DPT_FWPLNT\"\n",
    "                    name: \"Flowers & plants\"\n",
    "                3: {id: \"DPT_FD\", name: \"Food\"}\n",
    "                    id: \"DPT_FD\"\n",
    "                    name: \"Food\"\n",
    "                4: {id: \"DPT_FDMV\", name: \"Food on the move\"}\n",
    "                    id: \"DPT_FDMV\"\n",
    "                    name: \"Food on the move\"\n",
    "                5: {id: \"DPT_FDORDCLLCDSK\", name: \"Food order collection desk\"}\n",
    "                    id: \"DPT_FDORDCLLCDSK\"\n",
    "                    name: \"Food order collection desk\"\n",
    "                6: {id: \"DPT_WN\", name: \"Wine\"}\n",
    "                    id: \"DPT_WN\"\n",
    "                    name: \"Wine\"\n",
    "            distance: 2140.261181148\n",
    "            facilities: [{id: \"FCT_BBYCHNG\", name: \"Baby changing facilities\"}, {id: \"FCT_CARPRK\", name: \"Car parking\"},…]\n",
    "                0: {id: \"FCT_BBYCHNG\", name: \"Baby changing facilities\"}\n",
    "                    id: \"FCT_BBYCHNG\"\n",
    "                    name: \"Baby changing facilities\"\n",
    "                1: {id: \"FCT_CARPRK\", name: \"Car parking\"}\n",
    "                    id: \"FCT_CARPRK\"\n",
    "                    name: \"Car parking\"\n",
    "                2: {id: \"FCT_TOLT\", name: \"Toilets\"}\n",
    "                    id: \"FCT_TOLT\"\n",
    "                    name: \"Toilets\"\n",
    "            id: 1297\n",
    "            isChristmasOrderAccepted: true\n",
    "            isFoodCollectionSupported: true\n",
    "            isGmCollectionSupported: true\n",
    "            name: \"NORWICH SWEET BRIAR SIMPLY FOOD\"\n",
    "            phone: \"01603604398\"\n",
    "            services: [{id: \"SVC_CLHCD\", name: \"Clothing & Home collection desk\"},…]\n",
    "                0: {id: \"SVC_CLHCD\", name: \"Clothing & Home collection desk\"}\n",
    "                    id: \"SVC_CLHCD\"\n",
    "                    name: \"Clothing & Home collection desk\"\n",
    "                1: {id: \"SVC_CLHO\", name: \"Clothing & Home ordering\"}\n",
    "                    id: \"SVC_CLHO\"\n",
    "                    name: \"Clothing & Home ordering\"\n",
    "                2: {id: \"SVC_FPOD\", name: \"Flower & Plant ordering\"}\n",
    "                    id: \"SVC_FPOD\"\n",
    "                    name: \"Flower & Plant ordering\"\n",
    "                3: {id: \"SVC_FOOD\", name: \"Food ordering\"}\n",
    "                    id: \"SVC_FOOD\"\n",
    "                    name: \"Food ordering\"\n",
    "                4: {id: \"SVC_WIFI\", name: \"Free Wi-Fi\"}\n",
    "                    id: \"SVC_WIFI\"\n",
    "                    name: \"Free Wi-Fi\"\n",
    "                5: {id: \"SVC_HPOL\", name: \"Hamper ordering\"}\n",
    "                    id: \"SVC_HPOL\"\n",
    "                    name: \"Hamper ordering\"\n",
    "                6: {id: \"SVC_CAFE\", name: \"M&S Cafe\"}\n",
    "                    id: \"SVC_CAFE\"\n",
    "                    name: \"M&S Cafe\"\n",
    "                7: {id: \"SVC_PRCO\", name: \"Personalised Cake ordering\"}\n",
    "                    id: \"SVC_PRCO\"\n",
    "                    name: \"Personalised Cake ordering\"\n",
    "                8: {id: \"SVC_TAXFRSHP\", name: \"Tax free shopping\"}\n",
    "                    id: \"SVC_TAXFRSHP\"\n",
    "                    name: \"Tax free shopping\"\n",
    "                9: {id: \"SVC_WCOD\", name: \"Wedding Cake ordering\"}\n",
    "                    id: \"SVC_WCOD\"\n",
    "                    name: \"Wedding Cake ordering\"\n",
    "                10: {id: \"SVC_WINE\", name: \"Wine ordering\"}\n",
    "                    id: \"SVC_WINE\"\n",
    "                    name: \"Wine ordering\"\n",
    "            specialOpeningHours: [{name: \"Good Friday\", open: \"2020-04-10T08:00:00Z\", close: \"2020-04-10T20:00:00Z\", state: \"open\"},…]\n",
    "                0: {name: \"Good Friday\", open: \"2020-04-10T08:00:00Z\", close: \"2020-04-10T20:00:00Z\", state: \"open\"}\n",
    "                    close: \"2020-04-10T20:00:00Z\"\n",
    "                    name: \"Good Friday\"\n",
    "                    open: \"2020-04-10T08:00:00Z\"\n",
    "                    state: \"open\"\n",
    "                1: {name: \"Easter Sunday\", open: \"2020-04-12T00:00:00Z\", close: \"2020-04-12T00:00:00Z\", state: \"closed\"}\n",
    "                    close: \"2020-04-12T00:00:00Z\"\n",
    "                    name: \"Easter Sunday\"\n",
    "                    open: \"2020-04-12T00:00:00Z\"\n",
    "                    state: \"closed\"\n",
    "                2: {name: \"Easter Monday\", open: \"2020-04-13T08:00:00Z\", close: \"2020-04-13T18:00:00Z\", state: \"open\"}\n",
    "                    close: \"2020-04-13T18:00:00Z\"\n",
    "                    name: \"Easter Monday\"\n",
    "                    open: \"2020-04-13T08:00:00Z\"\n",
    "                    state: \"open\"\n",
    "                3: {name: \"Bank Holiday\", open: \"2020-05-08T08:00:00Z\", close: \"2020-05-08T20:00:00Z\", state: \"open\"}\n",
    "                    close: \"2020-05-08T20:00:00Z\"\n",
    "                    name: \"Bank Holiday\"\n",
    "                    open: \"2020-05-08T08:00:00Z\"\n",
    "                    state: \"open\"\n",
    "                4: {name: \"Bank Holiday\", open: \"2020-05-25T08:00:00Z\", close: \"2020-05-25T20:00:00Z\", state: \"open\"}\n",
    "                    close: \"2020-05-25T20:00:00Z\"\n",
    "                    name: \"Bank Holiday\"\n",
    "                    open: \"2020-05-25T08:00:00Z\"\n",
    "                    state: \"open\"\n",
    "                5: {name: null, open: \"2020-08-31T09:00:00Z\", close: \"2020-08-31T18:00:00Z\", state: \"open\"}\n",
    "                    close: \"2020-08-31T18:00:00Z\"\n",
    "                    name: null\n",
    "                    open: \"2020-08-31T09:00:00Z\"\n",
    "                    state: \"open\"\n",
    "            storeType: \"mands\"\n",
    "        3: {id: 904, name: \"Norwich NHS Hospital\", storeType: \"mands\",…}\n",
    "        4: {id: 1460, name: \"NORWICH LONGWATER FOODHALL\", storeType: \"mands\",…}\n",
    "        5: {id: 2155, name: \"AYLSHAM BP\", storeType: \"franchise\",…}\n",
    "        6: {id: 1346, name: \"ACLE NORWICH BP\", storeType: \"franchise\",…}\n",
    "        7: {id: 1444, name: \"BESTHORPE BP\", storeType: \"franchise\",…}\n",
    "        8: {id: 1970, name: \"LYNN HILL BP\", storeType: \"franchise\",…}\n",
    "        9: {id: 1412, name: \"GREAT YARMOUTH GAPTON HALL FOODHALL\", storeType: \"mands\",…}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Marks and Spencer's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_marks_and_spencers_data(lat, lng):\n",
    "    API_URL = \"https://api.marksandspencer.com/v1/stores\"\n",
    "    params = {  'apikey' : \"aVCi8dmPbHgHrdCv9gNt6rusFK98VokK\",\n",
    "                'latlong' : \"{0},{1}\".format(lat,lng),\n",
    "                'limit' : limit,\n",
    "                'radius' : radius}\n",
    "    rq = requests.get(API_URL, params=params)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.text)\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()\n",
    "        if res['count'] == 0:\n",
    "            return False\n",
    "        res = res[\"results\"]\n",
    "    except:\n",
    "        return False\n",
    "    i=0\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (e.g. res[0][\"name\"])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[i][\"name\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    if res[i][\"distance\"] > radius * 1000:\n",
    "        return False\n",
    "    openingHours = res[i][\"coreOpeningHours\"]\n",
    "    intermediaryDayKeys = { 'Monday' : 0, 'Tuesday' : 1, 'Wednesday' : 2, 'Thursday' : 3, 'Friday' : 4, 'Saturday' : 5, 'Sunday' : 6}\n",
    "    dayKeys = { 0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday', 5 : 'Saturday', 6 : 'Sunday'}\n",
    "    dayDoneSet = set()\n",
    "    daysHeap = []\n",
    "    for index in range(len(openingHours)):\n",
    "            key = intermediaryDayKeys[openingHours[index]['day']]  #turn key into int\n",
    "            #if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "            if key not in dayDoneSet:\n",
    "                dayDoneSet.add(key)\n",
    "            #keyHours processing code\n",
    "                try:\n",
    "                    actualHoursDict = {}\n",
    "                    actualHoursDict['open'] = openingHours[index]['open']\n",
    "                    actualHoursDict['close'] = openingHours[index]['close']\n",
    "                    actualHours = [actualHoursDict]\n",
    "                    keyHours = {'day' : dayKeys[key],\n",
    "                        'open' : True,\n",
    "                        'hours' : actualHours\n",
    "                        }\n",
    "                    heapq.heappush(daysHeap, [key,keyHours])\n",
    "                except:\n",
    "                    #If dictionary keys start_time or end_time dont exist, assume store is closed on that day\n",
    "                    closedDayHours = {  'day' : dayKeys[key],\n",
    "                                    'open' : False  }\n",
    "                    heapq.heappush(daysHeap, [key,closedDayHours])\n",
    "    # Check for any missing days and ensure all days are in order for insertion into our database\n",
    "    for missingDay in set(dayKeys.keys()).difference(dayDoneSet):\n",
    "        closedDayHours = {\n",
    "            \"day\": dayKeys[missingDay],\n",
    "            \"open\": False\n",
    "        }\n",
    "        heapq.heappush(daysHeap, [missingDay, closedDayHours])\n",
    "    if len(daysHeap) != 7:\n",
    "        return False\n",
    "    hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "    return hoursArray\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Iceland's API\n",
    "## Base URL:\n",
    "https://www.iceland.co.uk/on/demandware.store/Sites-icelandfoodsuk-Site/default/Stores-GetNearestStores\n",
    "## Parameters:\n",
    "* latitude = Latitude\n",
    "* longitude = Longitude\n",
    "* countryCode = \"GB\" for UK\n",
    "* distanceUnit = \"mi\" for miles\n",
    "* maxdistance = distance within which to search for stores (measured in distanceUnit)\n",
    "\n",
    "**Note:** The returned response is not sorted by distance from the given coordinates, this must be done client-side (in this case within the below function) and it is therefore recommended to pass a small \"maxdistance\" parameter to the API.\n",
    "\n",
    "## Sample API Call:\n",
    "https://www.iceland.co.uk/on/demandware.store/Sites-icelandfoodsuk-Site/default/Stores-GetNearestStores?latitude=52.6476395&longitude=1.2932064&countryCode=GB&distanceUnit=mi&maxdistance=50\n",
    "\n",
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{stores: {,…}}\n",
    "stores: {,…}\n",
    "16: {name: \"GORLESTON\", storeSecondaryName: \"\", address1: \"138d High Street\", address2: \"Gorleston On Sea\",…}\n",
    "    address1: \"138d High Street\"\n",
    "    address2: \"Gorleston On Sea\"\n",
    "    city: \"Great Yarmouth\"\n",
    "    countryCode: \"\"\n",
    "    email: \"\"\n",
    "    fax: \"\"\n",
    "    image: \"\"\n",
    "    latitude: \"52.5782\"\n",
    "    longitude: \"1.72711\"\n",
    "    name: \"GORLESTON\"\n",
    "    phone: \"01493441088\"\n",
    "    postalCode: \"NR31 6QX\"\n",
    "    stateCode: \"\"\n",
    "    storeEvents: \"\"\n",
    "    storeFeatures: \"↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵<div class=\"store-details\">↵<div class=\"store-info-title\">Store Details<span>:</span></div>↵<ul>↵↵↵↵↵↵↵↵↵↵↵<li class=\"default\">↵↵Online delivery available↵</li>↵↵↵↵↵↵↵↵↵↵↵↵<li class=\"default\">↵↵Home delivery available↵</li>↵↵↵↵↵↵↵↵↵↵↵↵<li class=\"default\">↵↵Alcohol available↵</li>↵↵↵↵↵↵↵↵↵↵↵↵<li class=\"default\">↵↵Onsite carpark↵</li>↵↵↵</ul>↵</div>↵↵\"\n",
    "    storeHours: \"<Store-Hours><div class=\"mon\">Monday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"tue\">Tuesday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"wed\">Wednesday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"thur\">Thursday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"fri\">Friday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"sat\">Saturday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"Sun\">Sunday<div class=\"store-opening-hours\">10:00AM <span>-</span> 4:00PM</div></div></Store-Hours>\"\n",
    "    storeSecondaryName: \"\"\n",
    "317: {name: \"THETFORD\", storeSecondaryName: \"\", address1: \"Unit 1 The Burrells\",…}\n",
    "    address1: \"Unit 1 The Burrells\"\n",
    "    address2: \"Minstergate Street\"\n",
    "    city: \"Thetford\"\n",
    "    countryCode: \"\"\n",
    "    email: \"\"\n",
    "    fax: \"\"\n",
    "    image: \"\"\n",
    "    latitude: \"52.415\"\n",
    "    longitude: \"0.745208\"\n",
    "    name: \"THETFORD\"\n",
    "    phone: \"01842750746\"\n",
    "    postalCode: \"IP24 1BN\"\n",
    "    stateCode: \"\"\n",
    "    storeEvents: \"\"\n",
    "    storeFeatures: \"↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵↵<div class=\"store-details\">↵<div class=\"store-info-title\">Store Details<span>:</span></div>↵<ul>↵↵↵↵↵↵↵↵↵↵↵<li class=\"default\">↵↵Online delivery available↵</li>↵↵↵↵↵↵↵↵↵↵↵↵<li class=\"default\">↵↵Home delivery available↵</li>↵↵↵↵↵↵↵↵↵↵↵↵<li class=\"default\">↵↵Alcohol available↵</li>↵↵↵↵↵↵↵↵↵↵↵↵<li class=\"default\">↵↵Onsite carpark↵</li>↵↵↵</ul>↵</div>↵↵\"\n",
    "    storeHours: \"<Store-Hours><div class=\"mon\">Monday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"tue\">Tuesday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"wed\">Wednesday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"thur\">Thursday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"fri\">Friday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"sat\">Saturday<div class=\"store-opening-hours\">9:00AM <span>-</span> 6:00PM</div></div><div class=\"Sun\">Sunday<div class=\"store-opening-hours\">10:00AM <span>-</span> 4:00PM</div></div></Store-Hours>\"\n",
    "    storeSecondaryName: \"\"\n",
    "351: {name: \"DEREHAM\", storeSecondaryName: \"\", address1: \"10 Nelson Place\", address2: \"East Dereham\",…}\n",
    "387: {name: \"DOWNHAM MARKET\", storeSecondaryName: \"\", address1: \"6 Prince of Wales Court\",…}\n",
    "439: {name: \"FELIXSTOWE\", storeSecondaryName: \"\", address1: \"56/60 Hamilton Road\", address2: \"Felixstowe\",…}\n",
    "508: {name: \"GT YARMOUTH SOUTH RD\", storeSecondaryName: \"\", address1: \"90/92 South Rd\",…}\n",
    "588: {name: \"NORWICH (2)\", storeSecondaryName: \"\", address1: \"Unit 1/3 Anglia Square\", address2: \"Norwich\",…}\n",
    "752: {name: \"LOWESTOFT\", storeSecondaryName: \"\", address1: \"15 The Britten Centre\", address2: \"Lowestoft\",…}\n",
    "768: {name: \"NEWMARKET\", storeSecondaryName: \"\", address1: \"1 Crown Walk\", address2: \"Exeter Road\",…}\n",
    "776: {name: \"NORWICH (3)\", storeSecondaryName: \"\", address1: \"67/73 St Stephens Street\",…}\n",
    "945: {name: \"BURY ST EDMUNDS\", storeSecondaryName: \"\", address1: \"5/6 Cornhill\",…}\n",
    "992: {name: \"ELY\", storeSecondaryName: \"\", address1: \"11 The Cloisters\", address2: \"Brays Lane Ely\",…}\n",
    "1246: {name: \"SUDBURY\", storeSecondaryName: \"\", address1: \"94 North Street\", address2: \"Sudbury\",…}\n",
    "1247: {name: \"SWAFFHAM\", storeSecondaryName: \"\", address1: \"77/81 Market Place\", address2: \"Swaffham\",…}\n",
    "1263: {name: \"DOVERCOURT\", storeSecondaryName: \"\", address1: \"246/250 High Street\", address2: \"Dovercourt\",…}\n",
    "1265: {name: \"CROMER\", storeSecondaryName: \"\", address1: \"57 Church Street\", address2: \"Cromer\",…}\n",
    "1377: {name: \"IPSWICH TOWER RAMPARTS\", storeSecondaryName: \"\", address1: \"Unit 34\",…}\n",
    "1410: {name: \"BECCLES\", storeSecondaryName: \"\", address1: \"4B/4C Taylors Square\", address2: \"Newgate\",…}\n",
    "1459: {name: \"WISBECH 2\", storeSecondaryName: \"\", address1: \"Unit 2 Cromwell Ret Park\",…}\n",
    "1507: {name: \"LOWESTOFT FWH\", storeSecondaryName: \"\", address1: \"Unit 6B\",…}\n",
    "1516: {name: \"GT YARMOUTH FWH\", storeSecondaryName: \"\", address1: \"Unit C1\",…}\n",
    "1645: {name: \"KINGS LYNN FWH\", storeSecondaryName: \"\", address1: \"UNIT A\", address2: \"HARDWICK RETAIL PARK\",…}\n",
    "1654: {name: \"IPSWICH EURO FWH\", storeSecondaryName: \"\", address1: \"UNIT 18 EURO RETAIL PARK\",…}\n",
    "1659: {name: \"IPSWICH SUFFOLK FWH\", storeSecondaryName: \"\", address1: \"SUFFOLK RETAIL PARK\",…}\n",
    "1690: {name: \"THETFORD FWH\", storeSecondaryName: \"\", address1: \"UNIT D BRECKLAND RET.PK.\",…}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Iceland's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_iceland_data(lat, lng):\n",
    "    API_URL = \"https://www.iceland.co.uk/on/demandware.store/Sites-icelandfoodsuk-Site/default/Stores-GetNearestStores\"\n",
    "    params = {  'latitude' : lat,\n",
    "                'longitude' : lng,\n",
    "                'countryCode' : \"GB\",\n",
    "                'distanceUnit' : \"km\", #NOTE might need to change to \"mi\" for api to function?\n",
    "                'maxdistance' : radius}\n",
    "    rq = requests.get(API_URL, params=params)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.json())\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()\n",
    "        if len(res['stores']) == 0:\n",
    "            return False\n",
    "        res = res[\"stores\"]\n",
    "    except:\n",
    "        return False\n",
    "    storeHeap = []\n",
    "    for index in res:\n",
    "        storeLat, storeLng = float(res[index][\"latitude\"]), float(res[index][\"longitude\"])\n",
    "        #compute distance between the two points using the haversine function\n",
    "        storeDistance = haversine((lat, lng),(storeLat, storeLng))\n",
    "        heapq.heappush(storeHeap, [storeDistance,res[index]])\n",
    "    res = heapq.nsmallest(limit, storeHeap)\n",
    "    i = 0\n",
    "    #INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (e.g. res[0][1][\"name\"])\n",
    "    #e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[i][1][\"name\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break \n",
    "    if res[i][0] > radius:\n",
    "        return False\n",
    "    openingHours = res[i][1][\"storeHours\"]\n",
    "    cleanr = re.compile('<.*?>|\\s|-')\n",
    "    hoursString = re.sub(cleanr, '', openingHours)\n",
    "    openingHoursArray = hoursString.split(\"day\")[1:]\n",
    "    dayKeys = { 0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday', 5 : 'Saturday', 6 : 'Sunday'}\n",
    "    hoursArray = []\n",
    "    for index in range(len(openingHoursArray)):\n",
    "        openingHoursArray[index] = openingHoursArray[index].split(\"M\")[:2]\n",
    "        for index1 in [0,1]:\n",
    "            try:\n",
    "                word = openingHoursArray[index][index1]\n",
    "                wordSplit = word.split(\":\")\n",
    "                if word[-1] == 'A':\n",
    "                    if wordSplit[0] == \"12\":\n",
    "                        word = \"00:{0}\".format(wordSplit[1][:-1])\n",
    "                    elif len(wordSplit[0]) == 1:\n",
    "                        word = \"0\" + word[:-1]\n",
    "                    elif len(wordSplit[0]) == 2:\n",
    "                        word = word[:-1]\n",
    "                    else: \n",
    "                        word = False\n",
    "                elif word[-1] == 'P':\n",
    "                    if wordSplit[0] == \"12\":\n",
    "                        word = word[:-1]\n",
    "                    elif len(wordSplit[0]) < 3:\n",
    "                        word = str( int(wordSplit[0]) + 12) + \":\" + wordSplit[1][:-1]\n",
    "                    else:\n",
    "                        word = False\n",
    "                openingHoursArray[index][index1] = word\n",
    "            except:\n",
    "                openingHoursArray[index][index1] = False\n",
    "        if not openingHoursArray[index][0] or not openingHoursArray[index][1]:\n",
    "            closedDayHours = {  'day' : dayKeys[index],\n",
    "                                'open' : False  }\n",
    "            hoursArray.append(closedDayHours)\n",
    "        else:\n",
    "            actualHoursDict = {}\n",
    "            actualHoursDict['open'] = openingHoursArray[index][0]\n",
    "            actualHoursDict['close'] = openingHoursArray[index][1]\n",
    "            actualHours = [actualHoursDict]\n",
    "            keyHours = {'day' : dayKeys[index],\n",
    "                'open' : True,\n",
    "                'hours' : actualHours\n",
    "                }\n",
    "            hoursArray.append(keyHours)\n",
    "    return hoursArray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kaufland's API\n",
    "## Base URL: \n",
    "https://www.kaufland.de/.klstorefinder.json\n",
    "\n",
    "## Sample API Call: \n",
    "https://www.kaufland.de/.klstorefinder.json\n",
    "\n",
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[,…]\n",
    "[0 … 99]\n",
    "    0: {n: \"DE1310\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"51.739458\", lng: \"7.878134\",…}\n",
    "    1: {n: \"DE1390\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"51.320667\",…}\n",
    "        cn: \"Kaufland Gevelsberg\"\n",
    "        f: {from: \"2019-02-03\", to: \"2020-10-29\"}\n",
    "        lat: \"51.320667\"\n",
    "        lng: \"7.339364\"\n",
    "        n: \"DE1390\"\n",
    "        p: \"02332/92170\"\n",
    "        pc: \"58285\"\n",
    "        slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\"\n",
    "        sn: \"Großer Markt 7\"\n",
    "        sod: [\"2020-08-01|00:00|00:00\", \"2020-08-02|00:00|00:00\", \"2020-08-03|00:00|00:00\",…]\n",
    "            0: \"2020-08-01|00:00|00:00\"\n",
    "            1: \"2020-08-02|00:00|00:00\"\n",
    "            2: \"2020-08-03|00:00|00:00\"\n",
    "            3: \"2020-08-04|00:00|00:00\"\n",
    "            4: \"2020-08-05|00:00|00:00\"\n",
    "            5: \"2020-08-06|00:00|00:00\"\n",
    "            6: \"2020-08-07|00:00|00:00\"\n",
    "        t: \"Gevelsberg\"\n",
    "        wod: [\"Monday|07:00|22:00\", \"Tuesday|07:00|22:00\", \"Wednesday|07:00|22:00\", \"Thursday|07:00|22:00\",…]\n",
    "            0: \"Monday|07:00|22:00\"\n",
    "            1: \"Tuesday|07:00|22:00\"\n",
    "            2: \"Wednesday|07:00|22:00\"\n",
    "            3: \"Thursday|07:00|22:00\"\n",
    "            4: \"Friday|07:00|22:00\"\n",
    "            5: \"Saturday|07:00|22:00\"\n",
    "    2: {n: \"DE1353\", slf: \"\", lat: \"48.724494\", lng: \"9.352354\", cn: \"Kaufland Esslingen am Neckar\",…}\n",
    "        cn: \"Kaufland Esslingen am Neckar\"\n",
    "        lat: \"48.724494\"\n",
    "        lng: \"9.352354\"\n",
    "        n: \"DE1353\"\n",
    "        p: \"0711/9829470\"\n",
    "        pc: \"73730\"\n",
    "        slf: \"\"\n",
    "        sn: \"Alleenstraße 32\"\n",
    "        t: \"Esslingen am Neckar OT Zell\"\n",
    "        wod: [\"Monday|07:00|22:00\", \"Tuesday|07:00|22:00\", \"Wednesday|07:00|22:00\", \"Thursday|07:00|22:00\",…]\n",
    "            0: \"Monday|07:00|22:00\"\n",
    "            1: \"Tuesday|07:00|22:00\"\n",
    "            2: \"Wednesday|07:00|22:00\"\n",
    "            3: \"Thursday|07:00|22:00\"\n",
    "            4: \"Friday|07:00|22:00\"\n",
    "            5: \"Saturday|07:00|22:00\"\n",
    "    3: {n: \"DE1360\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"51.816106\", lng: \"12.217098\",…}\n",
    "    4: {n: \"DE1443\", slf: \"CheeseService,SausageService,SnackStand,Barrier-freeWC,Elevators\",…}\n",
    "    5: {n: \"DE1323\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"48.737462\",…}\n",
    "    6: {n: \"DE1300\", slf: \"Barrier-freeWC\", lat: \"52.066022\", lng: \"6.965075\", cn: \"Kaufland Ahaus\",…}\n",
    "    7: {n: \"DE1303\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"50.999046\",…}\n",
    "    8: {n: \"DE1330\", slf: \"MeatService,CheeseService,SausageService,FishService,Barrier-freeWC\",…}\n",
    "    9: {n: \"DE1340\", slf: \"MeatService,CheeseService,SausageService,Barrier-freeWC\", lat: \"51.692556\",…}\n",
    "    10: {n: \"DE1350\",…}\n",
    "    11: {n: \"DE1370\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"51.251556\", lng: \"6.327598\",…}\n",
    "    12: {n: \"DE1380\", slf: \"\", lat: \"51.843504\", lng: \"6.251638\", cn: \"Kaufland Emmerich am Rhein\",…}\n",
    "    13: {n: \"DE1410\", slf: \"Barrier-freeWC,Elevators\", lat: \"51.673095\", lng: \"6.159703\", cn: \"Kaufland Goch\",…}\n",
    "    14: {n: \"DE1413\", slf: \"CheeseService,SausageService\", lat: \"53.793752\", lng: \"9.980712\",…}\n",
    "    15: {n: \"DE1420\", slf: \"MeatService,CheeseService,SausageService,FishService,Barrier-freeWC\",…}\n",
    "    16: {n: \"DE1430\", slf: \"Barrier-freeWC\", lat: \"51.665747\", lng: \"7.828612\", cn: \"Kaufland Hamm-Mitte\",…}\n",
    "    17: {n: \"DE1423\", slf: \"MeatService,CheeseService,SausageService,Barrier-freeWC\", lat: \"52.845247\",…}\n",
    "    18: {n: \"DE1373\",…}\n",
    "    19: {n: \"DE1470\", slf: \"\", lat: \"51.695413\", lng: \"13.261168\", cn: \"Kaufland Herzberg (Elster)\",…}\n",
    "    20: {n: \"DE1520\", slf: \"\", lat: \"52.158894\", lng: \"12.784625\", cn: \"Kaufland Linthe\", p: \"033844/75040\",…}\n",
    "    21: {n: \"DE1523\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"50.557219\", lng: \"10.413219\",…}\n",
    "    22: {n: \"DE1620\", slf: \"CheeseService,SausageService\", lat: \"51.380789\", lng: \"12.487861\",…}\n",
    "    23: {n: \"DE1660\", slf: \"CheeseService,SausageService\", lat: \"52.599673\", lng: \"11.836157\",…}\n",
    "    24: {n: \"DE1700\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"52.423064\",…}\n",
    "    25: {n: \"DE1720\", slf: \"CheeseService,SausageService,EVChargingStation_normal,Barrier-freeWC,Elevators\",…}\n",
    "    26: {n: \"DE1740\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"51.966072\", lng: \"12.085523\",…}\n",
    "    27: {n: \"DE1780\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"52.532952\",…}\n",
    "    28: {n: \"DE1850\", slf: \"CheeseService,SausageService,FuelStation,CarPark,Barrier-freeWC,Elevators\",…}\n",
    "    29: {n: \"DE1940\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"51.119260\",…}\n",
    "    30: {n: \"DE1960\", slf: \"CheeseService,SausageService,EVChargingStation_normal,Barrier-freeWC,Elevators\",…}\n",
    "    31: {n: \"DE1980\",…}\n",
    "    32: {n: \"DE2000\", slf: \"CheeseService,SausageService,Barrier-freeWC,ParkingLevel,Elevators\",…}\n",
    "    33: {n: \"DE2030\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"50.988265\", lng: \"11.053433\",…}\n",
    "    34: {n: \"DE2040\", slf: \"MeatService,CheeseService,SausageService,Barrier-freeWC,Elevators\",…}\n",
    "    35: {n: \"DE2080\", slf: \"CheeseService,SausageService,FuelStation,Barrier-freeWC\", lat: \"51.618117\",…}\n",
    "    36: {n: \"DE2090\",…}\n",
    "    37: {n: \"DE1453\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"49.918632\", lng: \"10.807095\",…}\n",
    "    38: {n: \"DE1463\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"48.684191\", lng: \"12.692460\",…}\n",
    "    39: {n: \"DE1493\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"47.966123\", lng: \"12.610198\",…}\n",
    "    40: {n: \"DE1680\",…}\n",
    "    41: {n: \"DE1730\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"48.327128\", lng: \"9.897468\",…}\n",
    "    42: {n: \"DE1810\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"49.749339\", lng: \"11.538843\",…}\n",
    "    43: {n: \"DE1820\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"48.285903\", lng: \"11.560747\",…}\n",
    "    44: {n: \"DE2010\",…}\n",
    "    45: {n: \"DE2020\",…}\n",
    "    46: {n: \"DE2050\", slf: \"CheeseService,SausageService,Elevators\", lat: \"48.128019\", lng: \"9.802567\",…}\n",
    "    47: {n: \"DE2100\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"48.578541\",…}\n",
    "    48: {n: \"DE2130\", slf: \"CheeseService,SausageService,SnackStand\", lat: \"49.333760\", lng: \"8.438317\",…}\n",
    "    49: {n: \"DE2140\",…}\n",
    "    50: {n: \"DE2160\",…}\n",
    "    51: {n: \"DE2170\", slf: \"CheeseService,SausageService,CarPark,Barrier-freeWC\", lat: \"49.140541\",…}\n",
    "    52: {n: \"DE2180\", slf: \"MeatService,CheeseService,SausageService,FishService,Barrier-freeWC,Elevators\",…}\n",
    "    53: {n: \"DE2200\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"48.707676\", lng: \"8.996997\",…}\n",
    "    54: {n: \"DE2210\",…}\n",
    "    55: {n: \"DE2220\", slf: \"Elevators\", lat: \"49.404656\", lng: \"8.681305\", cn: \"Kaufland Heidelberg-Weststadt\",…}\n",
    "    56: {n: \"DE2240\", slf: \"CheeseService,SausageService,FuelStation\", lat: \"48.713841\", lng: \"10.161344\",…}\n",
    "    57: {n: \"DE2250\",…}\n",
    "    58: {n: \"DE1860\",…}\n",
    "    59: {n: \"DE1870\", slf: \"CheeseService,SausageService,SnackStand,Barrier-freeWC\", lat: \"49.368523\",…}\n",
    "    60: {n: \"DE1880\", slf: \"CheeseService,SausageService,FishService,SnackStand,Barrier-freeWC\",…}\n",
    "    61: {n: \"DE1890\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"49.437500\", lng: \"6.871225\",…}\n",
    "    62: {n: \"DE1920\", slf: \"MeatService,CheeseService,SausageService,FishService,Barrier-freeWC\",…}\n",
    "    63: {n: \"DE1450\", slf: \"CheeseService,SausageService,FuelStation,Barrier-freeWC\", lat: \"52.216256\",…}\n",
    "    64: {n: \"DE1480\", slf: \"CheeseService,SausageService,FishService\", lat: \"51.364187\", lng: \"7.584131\",…}\n",
    "    65: {n: \"DE1490\", slf: \"CheeseService,SausageService\", lat: \"51.578523\", lng: \"7.669285\",…}\n",
    "    66: {n: \"DE1500\", slf: \"MeatService,CheeseService,SausageService,Barrier-freeWC\", lat: \"50.801054\",…}\n",
    "    67: {n: \"DE1510\", slf: \"CheeseService,SausageService\", lat: \"51.802547\", lng: \"6.168352\",…}\n",
    "    68: {n: \"DE1530\", slf: \"MeatService,CheeseService,SausageService,Barrier-freeWC\", lat: \"51.448711\",…}\n",
    "    69: {n: \"DE1550\", slf: \"MeatService,CheeseService,SausageService,Barrier-freeWC,ParkingLevel,Elevators\",…}\n",
    "    70: {n: \"DE1560\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"51.743499\", lng: \"8.726285\",…}\n",
    "    71: {n: \"DE1563\", slf: \"MeatService,CheeseService,SausageService,Barrier-freeWC,Elevators\",…}\n",
    "    72: {n: \"DE1570\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"51.450688\", lng: \"7.563034\",…}\n",
    "    73: {n: \"DE1580\", slf: \"CheeseService,SausageService,FuelStation\", lat: \"50.864133\", lng: \"8.038579\",…}\n",
    "    74: {n: \"DE1583\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"51.819561\", lng: \"9.851755\",…}\n",
    "    75: {n: \"DE1600\", slf: \"MeatService,CheeseService,SausageService,FishService,SnackStand,Barrier-freeWC\",…}\n",
    "    76: {n: \"DE1630\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"51.549403\",…}\n",
    "    77: {n: \"DE1640\", slf: \"MeatService,CheeseService,SausageService,Barrier-freeWC\", lat: \"50.555222\",…}\n",
    "    78: {n: \"DE1650\", slf: \"CheeseService,SausageService\", lat: \"51.594239\", lng: \"7.135616\",…}\n",
    "    79: {n: \"DE1663\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"51.715920\",…}\n",
    "    80: {n: \"DE1690\",…}\n",
    "    81: {n: \"DE1710\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"51.456500\",…}\n",
    "    82: {n: \"DE1750\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"51.615518\",…}\n",
    "    83: {n: \"DE1760\", slf: \"CheeseService,SausageService\", lat: \"53.601028\", lng: \"9.975191\",…}\n",
    "    84: {n: \"DE1770\", slf: \"CheeseService,SausageService,EVChargingStation_normal,Barrier-freeWC,Elevators\",…}\n",
    "    85: {n: \"DE1800\",…}\n",
    "    86: {n: \"DE1830\", slf: \"Barrier-freeWC\", lat: \"52.408623\", lng: \"9.711369\",…}\n",
    "    87: {n: \"DE1840\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"52.181482\", lng: \"7.854389\",…}\n",
    "    88: {n: \"DE1910\", slf: \"CheeseService,SausageService,CarPark,Barrier-freeWC,Elevators\", lat: \"49.977415\",…}\n",
    "    89: {n: \"DE1930\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"51.654702\", lng: \"7.096771\",…}\n",
    "    90: {n: \"DE1950\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"51.516387\",…}\n",
    "    91: {n: \"DE1970\",…}\n",
    "    92: {n: \"DE1990\",…}\n",
    "    93: {n: \"DE1573\", slf: \"MeatService,CheeseService,SausageService,Barrier-freeWC,ParkingLevel,Elevators\",…}\n",
    "    94: {n: \"DE2083\", slf: \"CheeseService,SausageService,Barrier-freeWC\", lat: \"50.588318\", lng: \"11.813559\",…}\n",
    "    95: {n: \"DE1843\", slf: \"MeatService,CheeseService,SausageService,CarPark\", lat: \"52.637229\",…}\n",
    "    96: {n: \"DE1633\", slf: \"MeatService,CheeseService,SausageService,FishService,Barrier-freeWC\",…}\n",
    "    97: {n: \"DE1693\", slf: \"CheeseService,SausageService,Barrier-freeWC,Elevators\", lat: \"49.249192\",…}\n",
    "    98: {n: \"DE1713\", slf: \"MeatService,CheeseService,SausageService,EVChargingStation_normal,Barrier-freeWC\",…}\n",
    "    99: {n: \"DE2253\", slf: \"CheeseService,SausageService\", lat: \"50.008864\", lng: \"8.435044\",…}\n",
    "[100 … 199]\n",
    "[200 … 299]\n",
    "[300 … 399]\n",
    "[400 … 499]\n",
    "[500 … 599]\n",
    "[600 … 668]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code for setting up a database filled with all of Kaufland's opening hours, formatted to our liking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_up_kaufland_database():\n",
    "    API_URL = \"https://www.kaufland.de/.klstorefinder.json\"\n",
    "    kauflandArray = []\n",
    "    dayKeys = {\n",
    "        0: \"Monday\",\n",
    "        1: \"Tuesday\",\n",
    "        2: \"Wednesday\",\n",
    "        3: \"Thursday\",\n",
    "        4: \"Friday\",\n",
    "        5: \"Saturday\",\n",
    "        6: \"Sunday\",\n",
    "    }\n",
    "    intermediaryDayKeys = {\n",
    "        \"Monday\": 0,\n",
    "        \"Tuesday\": 1,\n",
    "        \"Wednesday\": 2,\n",
    "        \"Thursday\": 3,\n",
    "        \"Friday\": 4,\n",
    "        \"Saturday\": 5,\n",
    "        \"Sunday\": 6,\n",
    "        None: None,\n",
    "    }\n",
    "    try:\n",
    "        rq = requests.get(API_URL)\n",
    "        res = rq.json()\n",
    "        if len(res) == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "\n",
    "    for index in range(len(res)):\n",
    "        dayDoneSet = set()\n",
    "        daysHeap = []\n",
    "        openingHours = res[index][\"wod\"]\n",
    "        dayCounter = 0\n",
    "        for dayIndex in range(len(openingHours)):\n",
    "            day = dayIndex\n",
    "            \n",
    "            #if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "            if dayIndex not in dayDoneSet:\n",
    "                try:\n",
    "                    dayDoneSet.add(\n",
    "                        intermediaryDayKeys[openingHours[dayIndex].split(\"|\")[0]]\n",
    "                    )\n",
    "                    # keyHours processing code\n",
    "                    opens = openingHours[dayIndex].split(\"|\")[1]\n",
    "                    closes = openingHours[dayIndex].split(\"|\")[2]\n",
    "                    if opens == closes:\n",
    "                        closedDayHours = {\n",
    "                            \"day\": openingHours[dayIndex].split(\"|\")[0],\n",
    "                            \"open\": False,\n",
    "                        }\n",
    "                        heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "                    actualHours = {\"open\": opens, \"close\": closes}\n",
    "                    keyHours = {\n",
    "                        \"day\": openingHours[dayIndex].split(\"|\")[0],\n",
    "                        \"open\": True,\n",
    "                        \"hours\": [actualHours],\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [day, keyHours])\n",
    "\n",
    "                except:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": openingHours[dayIndex].split(\"|\")[0],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "\n",
    "        # Check for any missing days and ensure all days are in order for insertion into our database\n",
    "        for day in daysHeap:\n",
    "            if day[1][\"day\"] != dayKeys[day[0]]:\n",
    "                day[0] = intermediaryDayKeys[day[1][\"day\"]]\n",
    "\n",
    "        if len(daysHeap) < 7:\n",
    "            for dayDone in dayKeys.keys():\n",
    "                if dayDone not in dayDoneSet:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": dayKeys[dayDone],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [dayDone, closedDayHours])\n",
    "\n",
    "        hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "        latitude = res[index][\"lat\"]\n",
    "        longitude = res[index][\"lng\"]\n",
    "        kauflandArray.append([(float(latitude), float(longitude)), hoursArray])\n",
    "    kauflandDB = pd.DataFrame(kauflandArray, columns=[\"Coordinates\", \"OpeningHours\"])\n",
    "    kauflandDB.to_csv(\"Kaufland.csv\")\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Kaufland's opening hours data from our database (via Haversine):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_kaufland_data(lat, lng):\n",
    "    # pass lat and lng as arrays of length 1 if you want the storeDistance to be returned too\n",
    "    if isinstance(lat, list):\n",
    "        returnDistanceToo = True\n",
    "        lat = lat[0]\n",
    "        lng = lng[0]\n",
    "    else:\n",
    "        returnDistanceToo = False\n",
    "    desiredCoords = [lat, lng]\n",
    "    kauflandDB = pd.read_csv(\n",
    "        \"Kaufland.csv\",\n",
    "        index_col=\"Unnamed: 0\",\n",
    "        converters={\"Coordinates\": eval, \"OpeningHours\": eval},\n",
    "    )\n",
    "    kauflandDB[\"Distances\"] = haversine_vector(\n",
    "        [desiredCoords] * len(kauflandDB[\"Coordinates\"]),\n",
    "        list(kauflandDB[\"Coordinates\"]),\n",
    "    )\n",
    "    closestStoreIndex = kauflandDB[\"Distances\"].idxmin()\n",
    "    if returnDistanceToo:\n",
    "        return (\n",
    "            kauflandDB[\"Distances\"][closestStoreIndex],\n",
    "            kauflandDB[\"OpeningHours\"][closestStoreIndex],\n",
    "        )\n",
    "    else:\n",
    "        return kauflandDB[\"OpeningHours\"][closestStoreIndex]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Migros Supermarkt's API\n",
    "\n",
    "## Base URL:\n",
    "https://web-api.migros.ch/widgets/stores\n",
    "\n",
    "## Parameters:\n",
    "* key = 'loh7Diephiengaiv' (Required api key)\n",
    "* filters[markets][0][0] = 'super' (Required filter to get Supermarkets)\n",
    "* filters[markets][0][2] = 'voi' (Required filter to get VOI)\n",
    "* filters[markets][0][3] = 'mp' (Required filter to get Migros-Partner)\n",
    "* limit = '737' (Specified to get the number of markets. Given 737 because only got 737 supermarteks by applying filter on the website)\n",
    "\n",
    "\n",
    "## Sample API Call:\n",
    "https://web-api.migros.ch/widgets/stores?key=loh7Diephiengaiv&filters[markets][0][0]=super&filters[markets][0][2]=voi&filters[markets][0][3]=mp&limit=737\n",
    "\n",
    "\n",
    "## Sample API Call Python Code:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "headers = {\n",
    "    'Accept-Language': 'de',\n",
    "    'Origin': 'https://filialen.migros.ch',\n",
    "}\n",
    "\n",
    "params = (\n",
    "    ('key', 'loh7Diephiengaiv'),\n",
    "    ('filters[markets][0][0]', 'super'),\n",
    "    ('filters[markets][0][2]', 'voi'),\n",
    "    ('filters[markets][0][3]', 'mp'),\n",
    "    ('limit', '736'),\n",
    ")\n",
    "\n",
    "response = requests.get('https://web-api.migros.ch/widgets/stores', headers=headers, params=params)\n",
    "response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Response:\n",
    "\n",
    "Below sample response is of only 1 store as to reduce the length of the response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{\n",
    "    \"stores\": [\n",
    "        {\n",
    "            \"markets\": [\n",
    "                {\n",
    "                    \"closing_date\": null,\n",
    "                    \"media\": [],\n",
    "                    \"type\": \"super\",\n",
    "                    \"sap_id\": \"4762\",\n",
    "                    \"logo\": {\n",
    "                        \"small\": \"//image.migros.ch/filialen/50x50/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo.png\",\n",
    "                        \"stack\": \"//image.migros.ch/filialen/{stack}/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo.png\",\n",
    "                        \"original\": \"//image.migros.ch/filialen/original/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo.png\",\n",
    "                        \"large\": \"//image.migros.ch/filialen/600x400/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo.png\",\n",
    "                        \"custom\": \"//image.migros.ch/filialen/custom/resize-width-{width}-height-{height}/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo.png\",\n",
    "                        \"medium\": \"//image.migros.ch/filialen/200x200/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo.png\"\n",
    "                    },\n",
    "                    \"logo_minimal\": {\n",
    "                        \"small\": \"//image.migros.ch/filialen/50x50/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-minimal.png\",\n",
    "                        \"stack\": \"//image.migros.ch/filialen/{stack}/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-minimal.png\",\n",
    "                        \"original\": \"//image.migros.ch/filialen/original/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-minimal.png\",\n",
    "                        \"large\": \"//image.migros.ch/filialen/600x400/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-minimal.png\",\n",
    "                        \"custom\": \"//image.migros.ch/filialen/custom/resize-width-{width}-height-{height}/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-minimal.png\",\n",
    "                        \"medium\": \"//image.migros.ch/filialen/200x200/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-minimal.png\"\n",
    "                    },\n",
    "                    \"id\": \"0090320_super\",\n",
    "                    \"slug\": \"migros-supermarkt-aachtal\",\n",
    "                    \"all_types\": [\n",
    "                        \"super\"\n",
    "                    ],\n",
    "                    \"email\": \"fbk-sm-Aachtal@gmos.ch\",\n",
    "                    \"opening_date\": \"2017-02-02\",\n",
    "                    \"prefixed_phone\": \"+41 58 712 59 30\",\n",
    "                    \"special_day_schedules\": [\n",
    "                        {\n",
    "                            \"just_message\": false,\n",
    "                            \"date_to\": \"2020-12-25\",\n",
    "                            \"time_close1\": null,\n",
    "                            \"message\": \"\",\n",
    "                            \"time_close2\": null,\n",
    "                            \"holiday\": \"Weihnachten\",\n",
    "                            \"time_open2\": null,\n",
    "                            \"time_open1\": null,\n",
    "                            \"open\": false,\n",
    "                            \"date_from\": \"2020-12-25\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"just_message\": false,\n",
    "                            \"date_to\": \"2020-12-26\",\n",
    "                            \"time_close1\": null,\n",
    "                            \"message\": \"\",\n",
    "                            \"time_close2\": null,\n",
    "                            \"holiday\": \"Stephanstag\",\n",
    "                            \"time_open2\": null,\n",
    "                            \"time_open1\": null,\n",
    "                            \"open\": false,\n",
    "                            \"date_from\": \"2020-12-26\"\n",
    "                        }\n",
    "                    ],\n",
    "                    \"logo_square\": {\n",
    "                        \"small\": \"//image.migros.ch/filialen/50-sqr/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-square.png\",\n",
    "                        \"stack\": \"//image.migros.ch/filialen/{stack}/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-square.png\",\n",
    "                        \"original\": \"//image.migros.ch/filialen/original-sqr/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-square.png\",\n",
    "                        \"large\": \"//image.migros.ch/filialen/600-sqr/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-square.png\",\n",
    "                        \"custom\": \"//image.migros.ch/filialen/custom-sqr/v-size-{size}/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-square.png\",\n",
    "                        \"medium\": \"//image.migros.ch/filialen/200-sqr/5222e5d31bd3a03018ae0eba14ea06b4aa55b848/super-super-de-logo-square.png\"\n",
    "                    },\n",
    "                    \"assortments\": [\n",
    "                        {\n",
    "                            \"name\": \"Lebensmittel\",\n",
    "                            \"id\": \"food\"\n",
    "                        }\n",
    "                    ],\n",
    "                    \"services\": [\n",
    "                        {\n",
    "                            \"sub_type\": \"subito-selfcheckout\",\n",
    "                            \"type\": \"selfcheckout\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"sub_type\": \"subito-selfscanning\",\n",
    "                            \"type\": \"selfscanning\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"sub_type\": \"subito\",\n",
    "                            \"name\": \"Self-Scanning / Self-Checkout\",\n",
    "                            \"link\": \"https://www.migros.ch/de/zahlungsmoeglichkeiten/subito.html\",\n",
    "                            \"logo\": {\n",
    "                                \"small\": \"//image.migros.ch/filialen/50x50/15cbe934303e781fa4a8107cb962944a24a3ee4e/self-scanning-self-checkout-logo.png\",\n",
    "                                \"stack\": \"//image.migros.ch/filialen/{stack}/15cbe934303e781fa4a8107cb962944a24a3ee4e/self-scanning-self-checkout-logo.png\",\n",
    "                                \"original\": \"//image.migros.ch/filialen/original/15cbe934303e781fa4a8107cb962944a24a3ee4e/self-scanning-self-checkout-logo.png\",\n",
    "                                \"large\": \"//image.migros.ch/filialen/600x400/15cbe934303e781fa4a8107cb962944a24a3ee4e/self-scanning-self-checkout-logo.png\",\n",
    "                                \"custom\": \"//image.migros.ch/filialen/custom/resize-width-{width}-height-{height}/15cbe934303e781fa4a8107cb962944a24a3ee4e/self-scanning-self-checkout-logo.png\",\n",
    "                                \"medium\": \"//image.migros.ch/filialen/200x200/15cbe934303e781fa4a8107cb962944a24a3ee4e/self-scanning-self-checkout-logo.png\"\n",
    "                            },\n",
    "                            \"subito_types\": [\n",
    "                                \"selfcheckout\",\n",
    "                                \"selfscanning\"\n",
    "                            ],\n",
    "                            \"text\": \"In dieser Filiale kann man mit Self-Scanning / Self-Checkout selber einscannen, einpacken und danach selbständig und bargeldlos bezahlen.\",\n",
    "                            \"type\": \"subito\",\n",
    "                            \"link_text\": \"Mehr zu Subito Self-Scanning / Self-Checkout\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"boss_numbers\": null,\n",
    "                            \"code\": \"mig_online\",\n",
    "                            \"sub_type\": \"market-service-mig_online\",\n",
    "                            \"name\": \"Abholort für online Theke\",\n",
    "                            \"link\": \"https://theke.migros.ch/de\",\n",
    "                            \"logo\": {\n",
    "                                \"small\": null,\n",
    "                                \"stack\": null,\n",
    "                                \"original\": null,\n",
    "                                \"large\": null,\n",
    "                                \"custom\": null,\n",
    "                                \"medium\": null\n",
    "                            },\n",
    "                            \"text\": \"Frisch, saisonal und einfach bequem: Bestellen Sie Fleisch online und holen Sie es in Ihrer Wunschfiliale ab.\",\n",
    "                            \"type\": \"market-service\",\n",
    "                            \"title\": \"Abholort für online Theke\",\n",
    "                            \"link_text\": \"Zum Angebot\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"boss_numbers\": null,\n",
    "                            \"code\": \"mig_bread\",\n",
    "                            \"sub_type\": \"market-service-mig_bread\",\n",
    "                            \"name\": \"Torten Variationen\",\n",
    "                            \"link\": \"\",\n",
    "                            \"logo\": {\n",
    "                                \"small\": null,\n",
    "                                \"stack\": null,\n",
    "                                \"original\": null,\n",
    "                                \"large\": null,\n",
    "                                \"custom\": null,\n",
    "                                \"medium\": null\n",
    "                            },\n",
    "                            \"text\": \"Hier können Sie individuelle dekorierte Torten vorbestellen und abholen.\",\n",
    "                            \"type\": \"market-service\",\n",
    "                            \"title\": \"Torten Variationen\",\n",
    "                            \"link_text\": \"\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"boss_numbers\": null,\n",
    "                            \"code\": \"mig_gift\",\n",
    "                            \"sub_type\": \"market-service-mig_gift\",\n",
    "                            \"name\": \"Geschenkkörbe\",\n",
    "                            \"link\": \"\",\n",
    "                            \"logo\": {\n",
    "                                \"small\": null,\n",
    "                                \"stack\": null,\n",
    "                                \"original\": null,\n",
    "                                \"large\": null,\n",
    "                                \"custom\": null,\n",
    "                                \"medium\": null\n",
    "                            },\n",
    "                            \"text\": \"Sie lieben Geschenkkörbe? Wir stellen Ihren Wunschkorb mit Sortimentsprodukten zusammen – ganz individuell und für jeden Anlass.\",\n",
    "                            \"type\": \"market-service\",\n",
    "                            \"title\": \"Geschenkkörbe\",\n",
    "                            \"link_text\": \"\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"sub_type\": \"srv\",\n",
    "                            \"hint\": \"\",\n",
    "                            \"link\": \"https://www.migros.ch/de/kontakt/ersatzteile-reparaturen.html\",\n",
    "                            \"text\": \"In dieser Filiale können Sie Reparaturen in Auftrag geben, sowie Ersatzteile und Zubehör bestellen und abholen.\",\n",
    "                            \"type\": \"srv\",\n",
    "                            \"title\": \"Reparaturen / Ersatzteile und Zubehör\",\n",
    "                            \"link_text\": \"Mehr zu Reparaturen und Ersatzteile \"\n",
    "                        },\n",
    "                        {\n",
    "                            \"sub_type\": \"health-center\",\n",
    "                            \"type\": \"health-center\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"code\": \"bargeld_kasse\",\n",
    "                            \"sub_type\": \"bank-service-bargeld_kasse\",\n",
    "                            \"name\": \"Kostenloser Bargeldbezug mit der Maestro-Karte\",\n",
    "                            \"link\": \"\",\n",
    "                            \"text\": \"Wenn Sie Bargeld brauchen, können Sie es an den Bancomaten der Migros Bank abheben. Oder in mehreren hundert Geschäften der Migros Gruppe – unkompliziert und ohne Fremdbankgebühren.\",\n",
    "                            \"type\": \"bank-service\",\n",
    "                            \"title\": \"\",\n",
    "                            \"link_text\": \"\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"sub_type\": \"post-service-point\",\n",
    "                            \"link\": \"https://www.migros.ch/de/services/pickmup/zusatzservices-post.html\",\n",
    "                            \"logo\": {\n",
    "                                \"small\": \"//image.migros.ch/filialen/50x50/d7e5735451cac67e8d571d49dc112df05eaf1a3f/pakete-abholen-und-aufgeben-logo.png\",\n",
    "                                \"stack\": \"//image.migros.ch/filialen/{stack}/d7e5735451cac67e8d571d49dc112df05eaf1a3f/pakete-abholen-und-aufgeben-logo.png\",\n",
    "                                \"original\": \"//image.migros.ch/filialen/original/d7e5735451cac67e8d571d49dc112df05eaf1a3f/pakete-abholen-und-aufgeben-logo.png\",\n",
    "                                \"large\": \"//image.migros.ch/filialen/600x400/d7e5735451cac67e8d571d49dc112df05eaf1a3f/pakete-abholen-und-aufgeben-logo.png\",\n",
    "                                \"custom\": \"//image.migros.ch/filialen/custom/resize-width-{width}-height-{height}/d7e5735451cac67e8d571d49dc112df05eaf1a3f/pakete-abholen-und-aufgeben-logo.png\",\n",
    "                                \"medium\": \"//image.migros.ch/filialen/200x200/d7e5735451cac67e8d571d49dc112df05eaf1a3f/pakete-abholen-und-aufgeben-logo.png\"\n",
    "                            },\n",
    "                            \"text\": \"In dieser Filiale können Sie Economy-Pakete versenden und via «PickPost» empfangen.\",\n",
    "                            \"type\": \"post-service-point\",\n",
    "                            \"title\": \"Pakete abholen und aufgeben\",\n",
    "                            \"link_text\": \"SO FUNKTIONIERT DER PAKET-SERVICE\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"promoted_group\": \"migros\",\n",
    "                            \"pickmup_type\": \"migrossup\",\n",
    "                            \"pickup_location_id\": \"903\",\n",
    "                            \"link\": \"https://www.pickmup.ch/de.html#so-gehts\",\n",
    "                            \"supported_stores\": [\n",
    "                                \"ackermann\",\n",
    "                                \"amorana\",\n",
    "                                \"bikeworld\",\n",
    "                                \"brack\",\n",
    "                                \"decathlon\",\n",
    "                                \"discountle\",\n",
    "                                \"doit\",\n",
    "                                \"equinet\",\n",
    "                                \"exlibris\",\n",
    "                                \"fischen\",\n",
    "                                \"galaxus\",\n",
    "                                \"hauptner\",\n",
    "                                \"hauptnerja\",\n",
    "                                \"hobbyfarmi\",\n",
    "                                \"interio\",\n",
    "                                \"jelmoli\",\n",
    "                                \"melshop\",\n",
    "                                \"micasa\",\n",
    "                                \"myluckydog\",\n",
    "                                \"orellfuess\",\n",
    "                                \"qoqa\",\n",
    "                                \"quelle\",\n",
    "                                \"quickpac\",\n",
    "                                \"reitsport\",\n",
    "                                \"sportxx\",\n",
    "                                \"yonc\"\n",
    "                            ],\n",
    "                            \"type\": \"pickmup\",\n",
    "                            \"promoted\": true,\n",
    "                            \"title\": \"Dies ist ein PickMup Standort\",\n",
    "                            \"return_option\": true,\n",
    "                            \"sub_type\": \"pickmup\",\n",
    "                            \"packet_sizes\": [\n",
    "                                \"standard\"\n",
    "                            ],\n",
    "                            \"pickmup_type_name\": \"PickMup Migros\",\n",
    "                            \"hint\": \"Melden Sie sich an der Kasse 1, um Ihre Bestellung abzuholen.\",\n",
    "                            \"name\": \"PickMup\",\n",
    "                            \"logo\": {\n",
    "                                \"small\": \"//image.migros.ch/filialen/50x50/505a5c808e744cd82396d8f59790fd73486ef1c0/pickmup-logo.png\",\n",
    "                                \"stack\": \"//image.migros.ch/filialen/{stack}/505a5c808e744cd82396d8f59790fd73486ef1c0/pickmup-logo.png\",\n",
    "                                \"original\": \"//image.migros.ch/filialen/original/505a5c808e744cd82396d8f59790fd73486ef1c0/pickmup-logo.png\",\n",
    "                                \"large\": \"//image.migros.ch/filialen/600x400/505a5c808e744cd82396d8f59790fd73486ef1c0/pickmup-logo.png\",\n",
    "                                \"custom\": \"//image.migros.ch/filialen/custom/resize-width-{width}-height-{height}/505a5c808e744cd82396d8f59790fd73486ef1c0/pickmup-logo.png\",\n",
    "                                \"medium\": \"//image.migros.ch/filialen/200x200/505a5c808e744cd82396d8f59790fd73486ef1c0/pickmup-logo.png\"\n",
    "                            },\n",
    "                            \"text\": \"Online bestellen und abholen, wann und wo Sie möchten.\",\n",
    "                            \"link_text\": \"So funktioniert PickMup\"\n",
    "                        },\n",
    "                        {\n",
    "                            \"size\": 1,\n",
    "                            \"sub_type\": \"alna\",\n",
    "                            \"logo\": {\n",
    "                                \"small\": \"//image.migros.ch/filialen/50x50/35d1b7e8aefbee61880953372375c3352aa46915/alnatura-sortiment-logo.png\",\n",
    "                                \"stack\": \"//image.migros.ch/filialen/{stack}/35d1b7e8aefbee61880953372375c3352aa46915/alnatura-sortiment-logo.png\",\n",
    "                                \"original\": \"//image.migros.ch/filialen/original/35d1b7e8aefbee61880953372375c3352aa46915/alnatura-sortiment-logo.png\",\n",
    "                                \"large\": \"//image.migros.ch/filialen/600x400/35d1b7e8aefbee61880953372375c3352aa46915/alnatura-sortiment-logo.png\",\n",
    "                                \"custom\": \"//image.migros.ch/filialen/custom/resize-width-{width}-height-{height}/35d1b7e8aefbee61880953372375c3352aa46915/alnatura-sortiment-logo.png\",\n",
    "                                \"medium\": \"//image.migros.ch/filialen/200x200/35d1b7e8aefbee61880953372375c3352aa46915/alnatura-sortiment-logo.png\"\n",
    "                            },\n",
    "                            \"text\": \"Diese Filiale führt ein Alnatura-Sortiment.\",\n",
    "                            \"type\": \"alna\",\n",
    "                            \"title\": \"Alnatura-Sortiment\"\n",
    "                        }\n",
    "                    ],\n",
    "                    \"additional_slugs\": [],\n",
    "                    \"cooperative\": {\n",
    "                        \"name\": \"Ostschweiz\",\n",
    "                        \"id\": \"gmos\",\n",
    "                        \"url\": \"https://www.migros.ch/de/genossenschaften/migros-ostschweiz.html\"\n",
    "                    },\n",
    "                    \"facebook_id\": null,\n",
    "                    \"full_name\": \"Migros-Supermarkt - Aachtal\",\n",
    "                    \"localized_slugs\": {\n",
    "                        \"de\": \"migros-supermarkt-aachtal\",\n",
    "                        \"it\": \"supermercato-migros-aachtal\",\n",
    "                        \"fr\": \"supermarche-migros-aachtal\"\n",
    "                    },\n",
    "                    \"cost_center\": \"0090320\",\n",
    "                    \"phone\": \"058 712 59 30\",\n",
    "                    \"hint\": \"\",\n",
    "                    \"name\": \"Migros-Supermarkt\",\n",
    "                    \"weblink\": {\n",
    "                        \"text\": \"www.migros.ch\",\n",
    "                        \"url\": \"https://www.migros.ch/de.html\"\n",
    "                    },\n",
    "                    \"opening_hours\": [\n",
    "                        {\n",
    "                            \"opening_hours\": [\n",
    "                                {\n",
    "                                    \"time_close1\": \"20:00\",\n",
    "                                    \"time_close2\": null,\n",
    "                                    \"time_open2\": null,\n",
    "                                    \"time_open1\": \"07:00\",\n",
    "                                    \"day_of_week\": 1\n",
    "                                },\n",
    "                                {\n",
    "                                    \"time_close1\": \"20:00\",\n",
    "                                    \"time_close2\": null,\n",
    "                                    \"time_open2\": null,\n",
    "                                    \"time_open1\": \"07:00\",\n",
    "                                    \"day_of_week\": 2\n",
    "                                },\n",
    "                                {\n",
    "                                    \"time_close1\": \"20:00\",\n",
    "                                    \"time_close2\": null,\n",
    "                                    \"time_open2\": null,\n",
    "                                    \"time_open1\": \"07:00\",\n",
    "                                    \"day_of_week\": 3\n",
    "                                },\n",
    "                                {\n",
    "                                    \"time_close1\": \"20:00\",\n",
    "                                    \"time_close2\": null,\n",
    "                                    \"time_open2\": null,\n",
    "                                    \"time_open1\": \"07:00\",\n",
    "                                    \"day_of_week\": 4\n",
    "                                },\n",
    "                                {\n",
    "                                    \"time_close1\": \"20:00\",\n",
    "                                    \"time_close2\": null,\n",
    "                                    \"time_open2\": null,\n",
    "                                    \"time_open1\": \"07:00\",\n",
    "                                    \"day_of_week\": 5\n",
    "                                },\n",
    "                                {\n",
    "                                    \"time_close1\": \"18:00\",\n",
    "                                    \"time_close2\": null,\n",
    "                                    \"time_open2\": null,\n",
    "                                    \"time_open1\": \"07:00\",\n",
    "                                    \"day_of_week\": 6\n",
    "                                },\n",
    "                                {\n",
    "                                    \"time_close1\": null,\n",
    "                                    \"time_close2\": null,\n",
    "                                    \"time_open2\": null,\n",
    "                                    \"time_open1\": null,\n",
    "                                    \"day_of_week\": 7\n",
    "                                }\n",
    "                            ],\n",
    "                            \"active\": true,\n",
    "                            \"date_to\": null,\n",
    "                            \"date_from\": \"2019-05-06\"\n",
    "                        }\n",
    "                    ],\n",
    "                    \"sub_types\": [\n",
    "                        \"super\"\n",
    "                    ],\n",
    "                    \"visit_frequencies\": null,\n",
    "                    \"managers\": [],\n",
    "                    \"computed_opening_hours\": []\n",
    "                }\n",
    "            ],\n",
    "            \"localized_slugs\": {\n",
    "                \"de\": \"aachtal\",\n",
    "                \"it\": \"aachtal\",\n",
    "                \"fr\": \"aachtal\"\n",
    "            },\n",
    "            \"phone\": \"058 712 59 30\",\n",
    "            \"additional_ids\": [\n",
    "                \"0090320_903\"\n",
    "            ],\n",
    "            \"name\": \"Aachtal\",\n",
    "            \"logo\": {\n",
    "                \"small\": \"//image.migros.ch/filialen/50x50/b7ed598472b2aa065658425a494316272885c3ff/m-de.png\",\n",
    "                \"stack\": \"//image.migros.ch/filialen/{stack}/b7ed598472b2aa065658425a494316272885c3ff/m-de.png\",\n",
    "                \"original\": \"//image.migros.ch/filialen/original/b7ed598472b2aa065658425a494316272885c3ff/m-de.png\",\n",
    "                \"large\": \"//image.migros.ch/filialen/600x400/b7ed598472b2aa065658425a494316272885c3ff/m-de.png\",\n",
    "                \"custom\": \"//image.migros.ch/filialen/custom/resize-width-{width}-height-{height}/b7ed598472b2aa065658425a494316272885c3ff/m-de.png\",\n",
    "                \"medium\": \"//image.migros.ch/filialen/200x200/b7ed598472b2aa065658425a494316272885c3ff/m-de.png\"\n",
    "            },\n",
    "            \"location\": {\n",
    "                \"geo\": {\n",
    "                    \"lon\": 9.2732473,\n",
    "                    \"lat\": 47.5482982\n",
    "                },\n",
    "                \"zip\": \"8580\",\n",
    "                \"country\": \"CH\",\n",
    "                \"address\": \"Schrofenstrasse 19\",\n",
    "                \"city\": \"Amriswil\",\n",
    "                \"address2\": \"\"\n",
    "            },\n",
    "            \"id\": \"0090320\",\n",
    "            \"type\": \"m\",\n",
    "            \"additional_slugs\": [],\n",
    "            \"slug\": \"aachtal\",\n",
    "            \"cooperative\": {\n",
    "                \"name\": \"Ostschweiz\",\n",
    "                \"id\": \"gmos\",\n",
    "                \"url\": \"https://www.migros.ch/de/genossenschaften/migros-ostschweiz.html\"\n",
    "            }\n",
    "        }\n",
    "    ],\n",
    "    \"total_hits\": 737,\n",
    "    \"aggregations\": {}\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code for setting up a database filled with all of Migros Supermarkt's opening hours, formatted to our liking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_up_migros_database():\n",
    "\n",
    "    API_URL = \"https://web-api.migros.ch/widgets/stores\"\n",
    "\n",
    "    headers = {\n",
    "        \"Accept-Language\": \"de\",\n",
    "        \"Origin\": \"https://filialen.migros.ch\",\n",
    "    }\n",
    "\n",
    "    params = (\n",
    "        (\"key\", \"loh7Diephiengaiv\"),\n",
    "        (\"filters[markets][0][0]\", \"super\"),\n",
    "        (\"filters[markets][0][2]\", \"voi\"),\n",
    "        (\"filters[markets][0][3]\", \"mp\"),\n",
    "        (\"limit\", \"737\"),\n",
    "    )\n",
    "\n",
    "    try:\n",
    "        rq = requests.get(API_URL, headers=headers, params=params)\n",
    "        res = rq.json()\n",
    "        if len(res) == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "\n",
    "    migrosArray = []\n",
    "\n",
    "    dayKeys = {\n",
    "        0: \"Monday\",\n",
    "        1: \"Tuesday\",\n",
    "        2: \"Wednesday\",\n",
    "        3: \"Thursday\",\n",
    "        4: \"Friday\",\n",
    "        5: \"Saturday\",\n",
    "        6: \"Sunday\",\n",
    "    }\n",
    "    intermediaryDayKeys = {\n",
    "        \"Monday\": 0,\n",
    "        \"Tuesday\": 1,\n",
    "        \"Wednesday\": 2,\n",
    "        \"Thursday\": 3,\n",
    "        \"Friday\": 4,\n",
    "        \"Saturday\": 5,\n",
    "        \"Sunday\": 6,\n",
    "        None: None,\n",
    "    }\n",
    "\n",
    "    for index in range(len(res[\"stores\"])):\n",
    "        dayDoneSet = set()\n",
    "        daysHeap = []\n",
    "        openingHours = res[\"stores\"][index][\"markets\"][0][\"opening_hours\"][0][\n",
    "            \"opening_hours\"\n",
    "        ]\n",
    "        dayCounter = 0\n",
    "        for dayIndex in range(len(openingHours)):\n",
    "            if dayIndex not in dayDoneSet:\n",
    "                try:\n",
    "                    day = openingHours[dayIndex][\"day_of_week\"] - 1\n",
    "                    dayDoneSet.add(day)\n",
    "\n",
    "                    # keyHours processing code\n",
    "                    # changes done because some stores opens two times a day.\n",
    "                    opens_firstHalf = openingHours[dayIndex][\"time_open1\"]\n",
    "                    closes_firstHalf = openingHours[dayIndex][\"time_close1\"]\n",
    "                    opens_secondHalf = openingHours[dayIndex][\"time_open2\"]\n",
    "                    closes_secondHalf = openingHours[dayIndex][\"time_close2\"]\n",
    "                    if (\n",
    "                        opens_firstHalf\n",
    "                        == closes_firstHalf\n",
    "                        == opens_secondHalf\n",
    "                        == closes_secondHalf\n",
    "                    ):\n",
    "                        closedDayHours = {\n",
    "                            \"day\": dayKeys[day],\n",
    "                            \"open\": False,\n",
    "                        }\n",
    "                        heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "                    else:\n",
    "                        if opens_secondHalf == closes_secondHalf:\n",
    "                            actualHours = {\n",
    "                                \"open\": opens_firstHalf,\n",
    "                                \"close\": closes_firstHalf,\n",
    "                            }\n",
    "                            keyHours = {\n",
    "                                \"day\": dayKeys[day],\n",
    "                                \"open\": True,\n",
    "                                \"hours\": [actualHours],\n",
    "                            }\n",
    "                            heapq.heappush(daysHeap, [day, keyHours])\n",
    "                        else:\n",
    "                            actualHours_firstHalf = {\n",
    "                                \"open\": opens_firstHalf,\n",
    "                                \"close\": closes_firstHalf,\n",
    "                            }\n",
    "                            actualHours_secondHalf = {\n",
    "                                \"open\": opens_secondHalf,\n",
    "                                \"close\": closes_secondHalf,\n",
    "                            }\n",
    "                            keyHours = {\n",
    "                                \"day\": dayKeys[day],\n",
    "                                \"open\": True,\n",
    "                                \"hours\": [\n",
    "                                    actualHours_firstHalf,\n",
    "                                    actualHours_secondHalf,\n",
    "                                ],\n",
    "                            }\n",
    "                            heapq.heappush(daysHeap, [day, keyHours])\n",
    "\n",
    "                except:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": dayKeys[day],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "\n",
    "        # Check for any missing days and ensure all days are in order for insertion into our database\n",
    "        for day in daysHeap:\n",
    "            if day[1][\"day\"] != dayKeys[day[0]]:\n",
    "                day[0] = intermediaryDayKeys[day[1][\"day\"]]\n",
    "\n",
    "        if len(daysHeap) < 7:\n",
    "            for dayDone in dayKeys.keys():\n",
    "                if dayDone not in dayDoneSet:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": dayKeys[dayDone],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [dayDone, closedDayHours])\n",
    "\n",
    "        hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "        latitude = res[\"stores\"][index][\"location\"][\"geo\"][\"lat\"]\n",
    "        longitude = res[\"stores\"][index][\"location\"][\"geo\"][\"lon\"]\n",
    "        migrosArray.append([(float(latitude), float(longitude)), hoursArray])\n",
    "    migrosDB = pd.DataFrame(migrosArray, columns=[\"Coordinates\", \"OpeningHours\"])\n",
    "    migrosDB.to_csv(\"Migros.csv\")\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Migros Supermarkt's opening hours data from our database (via Haversine):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_migros_data(lat, lng):\n",
    "    # pass lat and lng as arrays of length 1 if you want the storeDistance to be returned too\n",
    "    if isinstance(lat, list):\n",
    "        returnDistanceToo = True\n",
    "        lat = lat[0]\n",
    "        lng = lng[0]\n",
    "    else:\n",
    "        returnDistanceToo = False\n",
    "    desiredCoords = [lat, lng]\n",
    "    migrosDB = pd.read_csv(\n",
    "        \"Migros.csv\",\n",
    "        index_col=\"Unnamed: 0\",\n",
    "        converters={\"Coordinates\": eval, \"OpeningHours\": eval},\n",
    "    )\n",
    "    migrosDB[\"Distances\"] = haversine_vector(\n",
    "        [desiredCoords] * len(migrosDB[\"Coordinates\"]), list(migrosDB[\"Coordinates\"]),\n",
    "    )\n",
    "    closestStoreIndex = migrosDB[\"Distances\"].idxmin()\n",
    "    if returnDistanceToo:\n",
    "        return (\n",
    "            migrosDB[\"Distances\"][closestStoreIndex],\n",
    "            migrosDB[\"OpeningHours\"][closestStoreIndex],\n",
    "        )\n",
    "    else:\n",
    "        return migrosDB[\"OpeningHours\"][closestStoreIndex]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Carrefour's Belgium API\n",
    "## Base URL:\n",
    "https://magasins.carrefour.eu/api/v3/near/locations/by/slug\n",
    "## Parameters:\n",
    "* near = Parameter used to supply coordinates in the following form \"radius=Latitude,Longitude\". Example: <code>near=51.012676,4.114648</code>\n",
    "* size = To limit the number of stores returned. <code> size=3 </code>\n",
    "* radius = Max distance for which results should be shown, measured in Meters (**tested**). Example: <code>radius=20000</code>\n",
    "\n",
    "## Sample API Call:\n",
    "https://magasins.carrefour.eu/api/v3/near/locations/by/slug?near=51.012676,4.114648&size=3&radius=20000\n",
    "\n",
    "## Sample API Call Python Code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "params = (\n",
    "    ('near', '51.012676,4.114648'),\n",
    "    ('size', '3'),\n",
    "    ('radius', '20000'),\n",
    ")\n",
    "\n",
    "response = requests.get('https://magasins.carrefour.eu/api/v3/near/locations/by/slug', params=params, verify=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[\n",
    "    {\n",
    "        \"guid\": \"fa166cb5-5626-4eb6-80ee-c65611a2eb1e\",\n",
    "        \"siteId\": \"67f48d7e-5c97-48a4-a474-b67be09dd6c6\",\n",
    "        \"externalId\": \"1498\",\n",
    "        \"name\": \"Carrefour express Dendermonde\",\n",
    "        \"status\": \"OPEN\",\n",
    "        \"slug\": \"carrefour-express-dendermonde\",\n",
    "        \"brand\": \"Carrefour express\",\n",
    "        \"brandSlug\": \"express\",\n",
    "        \"language\": \"fr\",\n",
    "        \"description\": \"\",\n",
    "        \"locationItemIds\": [\n",
    "            \"service.carrefour.bancontact\",\n",
    "            \"service.carrefour.cfrvisa\",\n",
    "            \"service.carrefour.itunes\",\n",
    "            \"service.carrefour.maestro\",\n",
    "            \"service.carrefour.mastercard\",\n",
    "            \"service.carrefour.mealvouchers\",\n",
    "            \"service.carrefour.nfc\",\n",
    "            \"service.carrefour.prepaid\",\n",
    "            \"service.carrefour.transport\",\n",
    "            \"service.carrefour.visa\",\n",
    "            \"service.carrefour.bike.parking\",\n",
    "            \"service.carrefour.anti.waste\",\n",
    "            \"service.carrefour.biscuits\",\n",
    "            \"service.carrefour.bread\",\n",
    "            \"service.carrefour.chicken\",\n",
    "            \"service.carrefour.flowers\",\n",
    "            \"service.carrefour.juice\",\n",
    "            \"service.carrefour.localprod\",\n",
    "            \"service.carrefour.press\",\n",
    "            \"service.carrefour.stamps\"\n",
    "        ],\n",
    "        \"contact\": {\n",
    "            \"phone\": \"052 34 32 94\",\n",
    "            \"email\": \"maximemoonen@hotmail.com\",\n",
    "            \"url\": \"https://winkels.carrefour.eu/nl/s/carrefour/carrefour-express-dendermonde/1498\"\n",
    "        },\n",
    "        \"address\": {\n",
    "            \"street\": \"OUDE VEST, 32\",\n",
    "            \"locality\": \"DENDERMONDE\",\n",
    "            \"zipCode\": \"9200\",\n",
    "            \"country\": \"BE\",\n",
    "            \"fullAddress\": \"OUDE VEST, 32,9200,DENDERMONDE,BE\",\n",
    "            \"timezone\": \"Europe/Brussels\",\n",
    "            \"latitude\": 51.028634677765645,\n",
    "            \"longitude\": 4.1006972184200094,\n",
    "            \"distance\": 2.0\n",
    "        },\n",
    "        \"businessHours\": [\n",
    "            {\n",
    "                \"startDay\": 3,\n",
    "                \"endDay\": 3,\n",
    "                \"openTime\": \"0600\",\n",
    "                \"closeTime\": \"2000\"\n",
    "            },\n",
    "            {\n",
    "                \"startDay\": 7,\n",
    "                \"endDay\": 7,\n",
    "                \"openTime\": \"0700\",\n",
    "                \"closeTime\": \"1900\"\n",
    "            },\n",
    "            {\n",
    "                \"startDay\": 6,\n",
    "                \"endDay\": 6,\n",
    "                \"openTime\": \"0700\",\n",
    "                \"closeTime\": \"1900\"\n",
    "            },\n",
    "            {\n",
    "                \"startDay\": 4,\n",
    "                \"endDay\": 4,\n",
    "                \"openTime\": \"0600\",\n",
    "                \"closeTime\": \"2000\"\n",
    "            },\n",
    "            {\n",
    "                \"startDay\": 2,\n",
    "                \"endDay\": 2,\n",
    "                \"openTime\": \"0600\",\n",
    "                \"closeTime\": \"2000\"\n",
    "            },\n",
    "            {\n",
    "                \"startDay\": 5,\n",
    "                \"endDay\": 5,\n",
    "                \"openTime\": \"0600\",\n",
    "                \"closeTime\": \"2000\"\n",
    "            }\n",
    "        ],\n",
    "        \"specialHours\": [],\n",
    "        \"locationExtraData\": [],\n",
    "        \"images\": [],\n",
    "        \"services\": {\n",
    "            \"CAT_FINANCE_JEUX_CARTES\": [\n",
    "                {\n",
    "                    \"externalId\": \"BANCONTACT\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.bancontact\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Point de retrait Bancontact\",\n",
    "                        \"longContent\": \"Point de retrait Bancontact\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"CFRVISA\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.cfrvisa\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Carrefour Visa\",\n",
    "                        \"longContent\": \"Carrefour Visa\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"ITUNES\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.itunes\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Carte itunes\",\n",
    "                        \"longContent\": \"Carte itunes\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"MAESTRO\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.maestro\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Maestro\",\n",
    "                        \"longContent\": \"Maestro\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"MASTERCARD\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.mastercard\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Mastercard\",\n",
    "                        \"longContent\": \"Mastercard\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"MEALVOUCHERS\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.mealvouchers\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Chèques-repas\",\n",
    "                        \"longContent\": \"Chèques-repas\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"NFC\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.nfc\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Google Pay\",\n",
    "                        \"longContent\": \"Google Pay\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"PREPAID\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.prepaid\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Carte téléphonique\",\n",
    "                        \"longContent\": \"Carte  téléphonique\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"TRANSPORT\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.transport\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Carte de transport\",\n",
    "                        \"longContent\": \"Carte de transport\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"VISA\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.visa\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Visa\",\n",
    "                        \"longContent\": \"Visa\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_FINANCE_JEUX_CARTES\",\n",
    "                    \"assets\": []\n",
    "                }\n",
    "            ],\n",
    "            \"CAT_IN_CAR\": [\n",
    "                {\n",
    "                    \"externalId\": \"BIKE_PARKING\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.bike.parking\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Parking pour vélo\",\n",
    "                        \"longContent\": \"Parking pour vélo\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_IN_CAR\",\n",
    "                    \"assets\": []\n",
    "                }\n",
    "            ],\n",
    "            \"CAT_SERVICES\": [\n",
    "                {\n",
    "                    \"externalId\": \"ANTI_WASTE\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.anti.waste\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Panier anti-gaspillage\",\n",
    "                        \"longContent\": \"Panier anti-gaspillage\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_SERVICES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"BISCUITS\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.biscuits\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Baguettes/viennoiseries cuites sur place\",\n",
    "                        \"longContent\": \"Baguettes/viennoiseries cuites sur place\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_SERVICES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"BREAD\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.bread\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Pain cuit sur place\",\n",
    "                        \"longContent\": \"Pain cuit sur place\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_SERVICES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"CHICKEN\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.chicken\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Poulet rôti\",\n",
    "                        \"longContent\": \"Poulet rôti\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_SERVICES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"FLOWERS\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.flowers\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Fleurs\",\n",
    "                        \"longContent\": \"Fleurs\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_SERVICES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"JUICE\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.juice\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Jus d'orange frais\",\n",
    "                        \"longContent\": \"Jus d'orange frais\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_SERVICES\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"LOCALPROD\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.localprod\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Produits locaux\",\n",
    "                        \"longContent\": \"Produits locaux\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_SERVICES\",\n",
    "                    \"assets\": []\n",
    "                }\n",
    "            ],\n",
    "            \"CAT_TELEPHONIE_POSTE_PAPIER\": [\n",
    "                {\n",
    "                    \"externalId\": \"PRESS\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.press\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Coin presse\",\n",
    "                        \"longContent\": \"Coin presse\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_TELEPHONIE_POSTE_PAPIER\",\n",
    "                    \"assets\": []\n",
    "                },\n",
    "                {\n",
    "                    \"externalId\": \"STAMPS\",\n",
    "                    \"itemTypeExternalId\": \"service.carrefour.stamps\",\n",
    "                    \"contentFacet\": {\n",
    "                        \"title\": \"Vente de timbres\",\n",
    "                        \"longContent\": \"Vente de timbres\"\n",
    "                    },\n",
    "                    \"validityFacet\": [],\n",
    "                    \"priceFacet\": [],\n",
    "                    \"callToActionFacet\": [],\n",
    "                    \"itemTypeCategoryLabel\": \"CAT_TELEPHONIE_POSTE_PAPIER\",\n",
    "                    \"assets\": []\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Carrefour's Belgium opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_carrefour_data(lat, lng):\n",
    "    API_URL = \"https://magasins.carrefour.eu/api/v3/near/locations/by/slug\"\n",
    "    params = (\n",
    "        (\"near\", \"{},{}\".format(lat, lng)),\n",
    "        (\"size\", \"{}\".format(limit)),\n",
    "        (\"radius\", \"20000\"),\n",
    "    )\n",
    "    rq = requests.get(API_URL, params=params, verify=False)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.json())\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()\n",
    "        if len(res) == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "    i = 0\n",
    "    # INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (e.g. res[0][\"name\"])\n",
    "    # e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[i][\"name\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    storeLat, storeLng = (\n",
    "        float(res[i][\"address\"][\"latitude\"]),\n",
    "        float(res[i][\"address\"][\"longitude\"]),\n",
    "    )\n",
    "    # compute distance between the two points using the haversine function\n",
    "    storeDistance = haversine((lat, lng), (storeLat, storeLng), unit=\"m\")\n",
    "    if storeDistance > radius:\n",
    "        return False\n",
    "    openingHours = res[i][\"businessHours\"]\n",
    "\n",
    "    carrefourArray = []\n",
    "    dayKeys = {\n",
    "        0: \"Monday\",\n",
    "        1: \"Tuesday\",\n",
    "        2: \"Wednesday\",\n",
    "        3: \"Thursday\",\n",
    "        4: \"Friday\",\n",
    "        5: \"Saturday\",\n",
    "        6: \"Sunday\",\n",
    "    }\n",
    "    intermediaryDayKeys = {\n",
    "        \"Monday\": 0,\n",
    "        \"Tuesday\": 1,\n",
    "        \"Wednesday\": 2,\n",
    "        \"Thursday\": 3,\n",
    "        \"Friday\": 4,\n",
    "        \"Saturday\": 5,\n",
    "        \"Sunday\": 6,\n",
    "        None: None,\n",
    "    }\n",
    "\n",
    "    for index in range(len(res)):\n",
    "        dayDoneSet = set()\n",
    "        daysHeap = []\n",
    "        openingHours = res[index][\"businessHours\"]\n",
    "        dayCounter = 0\n",
    "        for dayIndex in range(len(openingHours)):\n",
    "            day = dayIndex\n",
    "            dayDone = openingHours[dayIndex][\"startDay\"] - 1\n",
    "            # if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "            if dayDone not in dayDoneSet:\n",
    "                try:\n",
    "                    dayDoneSet.add(dayDone)\n",
    "                    # keyHours processing code\n",
    "                    opens = datetime.strptime(openingHours[dayIndex][\"openTime\"], \"%H%M\").strftime(\"%H:%M\")\n",
    "                    closes = datetime.strptime(openingHours[dayIndex][\"closeTime\"], \"%H%M\").strftime(\"%H:%M\")\n",
    "                    if opens == closes:\n",
    "                        closedDayHours = {\n",
    "                            \"day\": dayKeys[dayDone],\n",
    "                            \"open\": False,\n",
    "                        }\n",
    "                        heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "                    actualHours = {\"open\": opens, \"close\": closes}\n",
    "                    keyHours = {\n",
    "                        \"day\": dayKeys[dayDone],\n",
    "                        \"open\": True,\n",
    "                        \"hours\": [actualHours],\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [day, keyHours])\n",
    "\n",
    "                except:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": dayKeys[dayDone],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "\n",
    "        # Check for any missing days and ensure all days are in order for insertion into our database\n",
    "        for day in daysHeap:\n",
    "            if day[1][\"day\"] != dayKeys[day[0]]:\n",
    "                day[0] = intermediaryDayKeys[day[1][\"day\"]]\n",
    "\n",
    "        if len(daysHeap) < 7:\n",
    "            for dayDone in dayKeys.keys():\n",
    "                if dayDone not in dayDoneSet:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": dayKeys[dayDone],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [dayDone, closedDayHours])\n",
    "\n",
    "        hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "        latitude = res[index][\"address\"][\"latitude\"]\n",
    "        longitude = res[index][\"address\"][\"longitude\"]\n",
    "        carrefourArray.append([(float(latitude), float(longitude)), hoursArray])\n",
    "    return carrefourArray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mercadona's API\n",
    "## Base URL:\n",
    "https://www.mercadona.com/estaticos/cargas/data.js\n",
    "\n",
    "## Sample API Call:\n",
    "https://www.mercadona.com/estaticos/cargas/data.js\n",
    "\n",
    "## Sample API Call Python Code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "response = requests.get(\"https://www.mercadona.com/estaticos/cargas/data.js\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var dataJson = {\"tiendasFull\":[\n",
    "    {\"id\":883185314342, \"p\":\"ES\", \"pv\":\"A CORUÃ‘A\", \"lc\":\"A CORUÃ‘A\", \"dr\":\"AVDA. DE  OZA, 134 - 138\", \"cp\":\"15006\", \"tf\":\"981303071\", \"in\":\"####C#C#\", \"fi\":\"####C#C#\", \"fs\":\"15/08/20-C#16/08/20-C#23/08/20-C#30/08/20-C#06/09/20-C\", \"pk\":\"S\", \"lt\":43.3534259247151, \"lg\":-8.39317618154519, \"fap\":\"24/11/2011\"},\n",
    "    {\"id\":883185313520, \"p\":\"ES\", \"pv\":\"A CORUÃ‘A\", \"lc\":\"A CORUÃ‘A\", \"dr\":\"C/ RONDA OUTEIRO,  217\", \"cp\":\"15007\", \"tf\":\"981277366\", \"in\":\"####C#C#\", \"fi\":\"####C#C#\", \"fs\":\"15/08/20-C#16/08/20-C#23/08/20-C#30/08/20-C#06/09/20-C\", \"pk\":\"S\", \"lt\":43.3582727534584, \"lg\":-8.41961492642406, \"fap\":\"09/11/2006\"},\n",
    "    {\"id\":883185315232, \"p\":\"ES\", \"pv\":\"A CORUÃ‘A\", \"lc\":\"A CORUÃ‘A\", \"dr\":\"LUGAR DE MONTEPATELO,63-67\", \"cp\":\"15008\", \"tf\":\"981303072\", \"in\":\"1000#1000#1000#1000#C#C#1000\", \"fi\":\"2200#2200#2200#2200#C#C#2200\", \"fs\":\"15/08/20-C#16/08/20-C#23/08/20-C#30/08/20-C#06/09/20-C\", \"pk\":\"S\", \"lt\":43.3431870048226, \"lg\":-8.43089402446753, \"fap\":\"09/02/2012\"},\n",
    "    {\"id\":883185314430, \"p\":\"ES\", \"pv\":\"A CORUÃ‘A\", \"lc\":\"A CORUÃ‘A\", \"dr\":\"RÃšA JOSÃ‰ PASCUAL LÃ“PEZ CORTÃ“N\", \"cp\":\"15008\", \"tf\":\"981172014\", \"in\":\"####C#C#\", \"fi\":\"####C#C#\", \"fs\":\"15/08/20-C#16/08/20-C#23/08/20-C#30/08/20-C#06/09/20-C\", \"pk\":\"S\", \"lt\":43.336367370105, \"lg\":-8.41034099601745, \"fap\":\"20/10/2009\"},\n",
    "    {\"id\":883185315002, \"p\":\"ES\", \"pv\":\"A CORUÃ‘A\", \"lc\":\"A CORUÃ‘A\", \"dr\":\"RÃšA PANADEIRAS, 29\", \"cp\":\"15001\", \"tf\":\"981213681\", \"in\":\"####C#C#\", \"fi\":\"####C#C#\", \"fs\":\"15/08/20-C#16/08/20-C#23/08/20-C#30/08/20-C#06/09/20-C\", \"pk\":\"S\", \"lt\":43.3728825795506, \"lg\":-8.3985813988113, \"fap\":\"13/05/2010\"},\n",
    "    {\"id\":883185313710, \"p\":\"ES\", \"pv\":\"A CORUÃ‘A\", \"lc\":\"AMÃ‰S\", \"dr\":\"AVDA. DE  MAHIA, 75\", \"cp\":\"15864\", \"tf\":\"981883877\", \"in\":\"####C#C#\", \"fi\":\"####C#C#\", \"fs\":\"15/08/20-C#16/08/20-C#23/08/20-C#30/08/20-C#06/09/20-C\", \"pk\":\"S\", \"lt\":42.8604446435338, \"lg\":-8.64989935945505, \"fap\":\"19/04/2007\"},\n",
    "    {\"id\":883185314448, \"p\":\"ES\", \"pv\":\"A CORUÃ‘A\", \"lc\":\"AMÃ‰S\", \"dr\":\"TRAVESÃA DO PORTO-RÃšA DAS PALMEIRAS\", \"cp\":\"15864\", \"tf\":\"981308900\", \"in\":\"####C#C#\", \"fi\":\"####C#C#\", \"fs\":\"15/08/20-C#16/08/20-C#23/08/20-C#30/08/20-C#06/09/20-C\", \"pk\":\"S\", \"lt\":42.8478770063104, \"lg\":-8.58078597553254, \"fap\":\"13/09/2012\"},\n",
    "    {\"id\":883185315462, \"p\":\"ES\", \"pv\":\"A CORUÃ‘A\", \"lc\":\"BETANZOS\", \"dr\":\"AVDA. FRAGA IRIBARNE, 7\", \"cp\":\"15300\", \"tf\":\"981302550\", \"in\":\"####C#C#\", \"fi\":\"####C#C#\", \"fs\":\"15/08/20-C#16/08/20-C#18/08/20-C#23/08/20-C#30/08/20-C\", \"pk\":\"S\", \"lt\":43.2756706814282, \"lg\":-8.21834065012354, \"fap\":\"11/12/2015\"},\n",
    "]};"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Mercadona's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_up_mercadona_database():\n",
    "    API_URL = \"https://www.mercadona.com/estaticos/cargas/data.js\"\n",
    "    mercadonaArray = []\n",
    "    dayKeys = {\n",
    "        0: \"Monday\",\n",
    "        1: \"Tuesday\",\n",
    "        2: \"Wednesday\",\n",
    "        3: \"Thursday\",\n",
    "        4: \"Friday\",\n",
    "        5: \"Saturday\",\n",
    "        6: \"Sunday\",\n",
    "    }\n",
    "    intermediaryDayKeys = {\n",
    "        \"Monday\": 0,\n",
    "        \"Tuesday\": 1,\n",
    "        \"Wednesday\": 2,\n",
    "        \"Thursday\": 3,\n",
    "        \"Friday\": 4,\n",
    "        \"Saturday\": 5,\n",
    "        \"Sunday\": 6,\n",
    "        None: None,\n",
    "    }\n",
    "    try:\n",
    "        rq = requests.get(API_URL)\n",
    "        obj = rq.text[rq.text.find(\"{\") : -5] + \"]}\"\n",
    "        res = json.loads(obj)\n",
    "\n",
    "        if len(res) == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "\n",
    "    for index in range(len(res[\"tiendasFull\"])):\n",
    "        dayDoneSet = set()\n",
    "        daysHeap = []\n",
    "        daysArray = []\n",
    "        closingDayArray = res[\"tiendasFull\"][index][\"fi\"].split(\"#\")\n",
    "        openingDayArray = res[\"tiendasFull\"][index][\"in\"].split(\"#\")\n",
    "\n",
    "        # finding current day in Spain(madrid)\n",
    "        timeZoneEurope = pytz.timezone(\"Europe/Madrid\")\n",
    "        datetimeEurope = datetime.now(timeZoneEurope)\n",
    "        currentday = datetimeEurope.strftime(\"%A\")\n",
    "        dayCounter = intermediaryDayKeys[currentday]\n",
    "\n",
    "        # implemented because time string returned accordni to the current day in Spain(eg : \"##C#C###\")\n",
    "        for val in range(7):\n",
    "            daysArray.append(dayCounter)\n",
    "            dayCounter += 1\n",
    "            if dayCounter == 7:\n",
    "                dayCounter = 0\n",
    "\n",
    "        openingHours = daysArray\n",
    "\n",
    "        for dayIndex in range(len(openingHours)):\n",
    "            day = daysArray[dayIndex]\n",
    "\n",
    "            # if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "            if day not in dayDoneSet:\n",
    "                try:\n",
    "                    dayDoneSet.add(day)\n",
    "                    # keyHours processing code\n",
    "                    opens = openingDayArray[dayIndex]\n",
    "                    closes = closingDayArray[dayIndex]\n",
    "                    if opens == closes != \"\":\n",
    "                        closedDayHours = {\n",
    "                            \"day\": dayKeys[daysArray[dayIndex]],\n",
    "                            \"open\": False,\n",
    "                        }\n",
    "                        heapq.heappush(daysHeap, [dayIndex, closedDayHours])\n",
    "                    else:\n",
    "                        if opens == closes == \"\":\n",
    "                            opens = \"09:00\"\n",
    "                            closes = \"21:30\"\n",
    "                        elif opens == \"\" and closes != \"\":\n",
    "                            opens = \"09:00\"\n",
    "                            closes = datetime.strptime(closes, \"%H%M\").strftime(\"%H:%M\")\n",
    "                        elif opens != \"\" and closes == \"\":\n",
    "                            closes = \"21:30\"\n",
    "                            opens = datetime.strptime(opens, \"%H%M\").strftime(\"%H:%M\")\n",
    "                        else:\n",
    "                            opens = datetime.strptime(opens, \"%H%M\").strftime(\"%H:%M\")\n",
    "                            closes = datetime.strptime(closes, \"%H%M\").strftime(\"%H:%M\")\n",
    "                        actualHours = {\"open\": opens, \"close\": closes}\n",
    "                        keyHours = {\n",
    "                            \"day\": dayKeys[daysArray[dayIndex]],\n",
    "                            \"open\": True,\n",
    "                            \"hours\": [actualHours],\n",
    "                        }\n",
    "                        heapq.heappush(daysHeap, [dayIndex, keyHours])\n",
    "\n",
    "                except:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": dayKeys[daysArray[dayIndex]],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [dayIndex, closedDayHours])\n",
    "\n",
    "        # Check for any missing days and ensure all days are in order for insertion into our database\n",
    "        for day in daysHeap:\n",
    "            if day[1][\"day\"] != dayKeys[day[0]]:\n",
    "                day[0] = intermediaryDayKeys[day[1][\"day\"]]\n",
    "\n",
    "        if len(daysHeap) < 7:\n",
    "            for dayDone in dayKeys.keys():\n",
    "                if dayDone not in dayDoneSet:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": dayKeys[daysArray[dayIndex]],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [dayDone, closedDayHours])\n",
    "\n",
    "        hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "        latitude = res[\"tiendasFull\"][index][\"lt\"]\n",
    "        longitude = res[\"tiendasFull\"][index][\"lg\"]\n",
    "        mercadonaArray.append([(float(latitude), float(longitude)), hoursArray])\n",
    "    mercadonaDB = pd.DataFrame(mercadonaArray, columns=[\"Coordinates\", \"OpeningHours\"])\n",
    "    mercadonaDB.to_csv(\"Mercadona.csv\")\n",
    "    return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code for retrieving Mercadona's opening hours data from our database (via Haversine):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_mercadona_data(lat, lng):\n",
    "    # pass lat and lng as arrays of length 1 if you want the storeDistance to be returned too\n",
    "    if isinstance(lat, list):\n",
    "        returnDistanceToo = True\n",
    "        lat = lat[0]\n",
    "        lng = lng[0]\n",
    "    else:\n",
    "        returnDistanceToo = False\n",
    "    desiredCoords = [lat, lng]\n",
    "    mercadonaDB = pd.read_csv(\n",
    "        \"Mercadona.csv\",\n",
    "        index_col=\"Unnamed: 0\",\n",
    "        converters={\"Coordinates\": eval, \"OpeningHours\": eval},\n",
    "    )\n",
    "    mercadonaDB[\"Distances\"] = haversine_vector(\n",
    "        [desiredCoords] * len(mercadonaDB[\"Coordinates\"]),\n",
    "        list(mercadonaDB[\"Coordinates\"]),\n",
    "    )\n",
    "    closestStoreIndex = mercadonaDB[\"Distances\"].idxmin()\n",
    "    if returnDistanceToo:\n",
    "        return (\n",
    "            mercadonaDB[\"Distances\"][closestStoreIndex],\n",
    "            mercadonaDB[\"OpeningHours\"][closestStoreIndex],\n",
    "        )\n",
    "    else:\n",
    "        return mercadonaDB[\"OpeningHours\"][closestStoreIndex]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Coles's API\n",
    "## Base URL:\n",
    "https://apigw.coles.com.au/digital/colesweb/v1/stores/search\n",
    "## Parameters:\n",
    "* latitude = Parameter used to pass latitude value. Example: <code>latitude=-33.875347</code>\n",
    "* longitude = Parameter used to pass longitude value. Example: <code>longitude=151.222213</code>\n",
    "* brandIds = Parameter used to differentiate between stores (Coles(1) and Coles Express(2)). Example: <code>brandIds=2,1</code>\n",
    "* numberOfStores = Parameter used to pass number of stores. Maximum value can be 15. Example: <code>numberOfStores=15</code>\n",
    "\n",
    "**Note:** All parameters are required.\n",
    "\n",
    "## Sample API Call:\n",
    "https://apigw.coles.com.au/digital/colesweb/v1/stores/search?latitude=-33.875347&longitude=151.222213&brandIds=2,1&numberOfStores=15\n",
    "\n",
    "## Sample API Call Python Code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "params = (\n",
    "    ('latitude', '-33.875347'),\n",
    "    ('longitude', '151.222213'),\n",
    "    ('brandIds', '2,1'),\n",
    "    ('numberOfStores', '15'),\n",
    ")\n",
    "\n",
    "response = requests.get('https://apigw.coles.com.au/digital/colesweb/v1/stores/search', params=params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{\n",
    "    \"stores\": [\n",
    "        {\n",
    "            \"storeId\": \"754\",\n",
    "            \"brandName\": \"Coles Supermarkets\",\n",
    "            \"brandId\": 2,\n",
    "            \"latitude\": -33.875347,\n",
    "            \"longitude\": 151.222213,\n",
    "            \"distance\": 0.0,\n",
    "            \"storeName\": \"Coles Central Kings Cross\",\n",
    "            \"address\": \"88 - 94 Darlinghurst Rd\",\n",
    "            \"suburb\": \"Kings Cross\",\n",
    "            \"state\": \"NSW\",\n",
    "            \"postcode\": \"2011\",\n",
    "            \"phone\": \"(02) 9358 6247\",\n",
    "            \"tradingHoursToday\": {\n",
    "                \"openNow\": \"Closed\",\n",
    "                \"storeTime\": \"Reopens at 7am\"\n",
    "            },\n",
    "            \"tradingHours\": [\n",
    "                {\n",
    "                    \"daysOfWeek\": \"Mon-Sun\",\n",
    "                    \"storeTime\": \"7am-12am\"\n",
    "                }\n",
    "            ],\n",
    "            \"holidayTradingHours\": [],\n",
    "            \"nextDoorStores\": [\n",
    "                {\n",
    "                    \"storeId\": \"3993\",\n",
    "                    \"storeName\": \"Kings Cross\",\n",
    "                    \"brandId\": 3,\n",
    "                    \"brandName\": \"Liquorland\",\n",
    "                    \"tradingHoursToday\": {\n",
    "                        \"openNow\": \"Closed\",\n",
    "                        \"storeTime\": \"reopens at 10am\"\n",
    "                    }\n",
    "                }\n",
    "            ],\n",
    "            \"services\": []\n",
    "        },\n",
    "        {\n",
    "            \"storeId\": \"1585\",\n",
    "            \"brandName\": \"Coles Express\",\n",
    "            \"brandId\": 1,\n",
    "            \"latitude\": -33.869568,\n",
    "            \"longitude\": 151.221675,\n",
    "            \"distance\": 0.64,\n",
    "            \"storeName\": \"Coles Express Woolloomooloo\",\n",
    "            \"address\": \"61-63 Cowper Wharf Rd (cnr Dowling St)\",\n",
    "            \"suburb\": \"Woolloomooloo\",\n",
    "            \"state\": \"NSW\",\n",
    "            \"postcode\": \"2011\",\n",
    "            \"phone\": \"(02) 9326 9547\",\n",
    "            \"tradingHoursToday\": {\n",
    "                \"openNow\": \"Open\",\n",
    "                \"storeTime\": \"Today 24 hours\"\n",
    "            },\n",
    "            \"tradingHours\": [\n",
    "                {\n",
    "                    \"daysOfWeek\": \"Mon-Sun\",\n",
    "                    \"storeTime\": \"24 hours\"\n",
    "                }\n",
    "            ],\n",
    "            \"holidayTradingHours\": [],\n",
    "            \"nextDoorStores\": [],\n",
    "            \"services\": [\n",
    "                {\n",
    "                    \"serviceName\": \"Shell V-Power\",\n",
    "                    \"serviceCategory\": \"Fuels\"\n",
    "                },\n",
    "                {\n",
    "                    \"serviceName\": \"Shell Unleaded 95\",\n",
    "                    \"serviceCategory\": \"Fuels\"\n",
    "                },\n",
    "                {\n",
    "                    \"serviceName\": \"Shell Unleaded Ethanol\",\n",
    "                    \"serviceCategory\": \"Fuels\"\n",
    "                },\n",
    "                {\n",
    "                    \"serviceName\": \"Shell Diesel\",\n",
    "                    \"serviceCategory\": \"Fuels\"\n",
    "                },\n",
    "                {\n",
    "                    \"serviceName\": \"Open 24 hours\",\n",
    "                    \"serviceCategory\": \"Services\"\n",
    "                },\n",
    "                {\n",
    "                    \"serviceName\": \"ATM\",\n",
    "                    \"serviceCategory\": \"Services\"\n",
    "                },\n",
    "                {\n",
    "                    \"serviceName\": \"Hot food\",\n",
    "                    \"serviceCategory\": \"Services\"\n",
    "                },\n",
    "                {\n",
    "                    \"serviceName\": \"Coffee\",\n",
    "                    \"serviceCategory\": \"Services\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"storeId\": \"899\",\n",
    "            \"brandName\": \"Coles Supermarkets\",\n",
    "            \"brandId\": 2,\n",
    "            \"latitude\": -33.879329,\n",
    "            \"longitude\": 151.236607,\n",
    "            \"distance\": 1.31,\n",
    "            \"storeName\": \"Coles Edgecliff\",\n",
    "            \"address\": \"New South Head Rd & Ocean St\",\n",
    "            \"suburb\": \"Edgecliff\",\n",
    "            \"state\": \"NSW\",\n",
    "            \"postcode\": \"2027\",\n",
    "            \"phone\": \"(02) 9328 7978\",\n",
    "            \"tradingHoursToday\": {\n",
    "                \"openNow\": \"Closed\",\n",
    "                \"storeTime\": \"Reopens at 7am\"\n",
    "            },\n",
    "            \"tradingHours\": [\n",
    "                {\n",
    "                    \"daysOfWeek\": \"Mon-Sun\",\n",
    "                    \"storeTime\": \"7am-12am\"\n",
    "                }\n",
    "            ],\n",
    "            \"holidayTradingHours\": [],\n",
    "            \"nextDoorStores\": [\n",
    "                {\n",
    "                    \"storeId\": \"3967\",\n",
    "                    \"storeName\": \"Edgecliff\",\n",
    "                    \"brandId\": 3,\n",
    "                    \"brandName\": \"Liquorland\",\n",
    "                    \"tradingHoursToday\": {\n",
    "                        \"openNow\": \"Closed\",\n",
    "                        \"storeTime\": \"reopens at 10am\"\n",
    "                    }\n",
    "                }\n",
    "            ],\n",
    "            \"services\": []\n",
    "        },\n",
    "        {\n",
    "            \"storeId\": \"710\",\n",
    "            \"brandName\": \"Coles Supermarkets\",\n",
    "            \"brandId\": 2,\n",
    "            \"latitude\": -33.877321,\n",
    "            \"longitude\": 151.206106,\n",
    "            \"distance\": 1.39,\n",
    "            \"storeName\": \"Coles World Square\",\n",
    "            \"address\": \"650 George St\",\n",
    "            \"suburb\": \"Sydney\",\n",
    "            \"state\": \"NSW\",\n",
    "            \"postcode\": \"2000\",\n",
    "            \"phone\": \"(02) 8275 8600\",\n",
    "            \"tradingHoursToday\": {\n",
    "                \"openNow\": \"Closed\",\n",
    "                \"storeTime\": \"Reopens at 6am\"\n",
    "            },\n",
    "            \"tradingHours\": [\n",
    "                {\n",
    "                    \"daysOfWeek\": \"Mon-Sun\",\n",
    "                    \"storeTime\": \"6am-12am\"\n",
    "                }\n",
    "            ],\n",
    "            \"holidayTradingHours\": [],\n",
    "            \"nextDoorStores\": [\n",
    "                {\n",
    "                    \"storeId\": \"3617\",\n",
    "                    \"storeName\": \"Sydney World Square Express\",\n",
    "                    \"brandId\": 3,\n",
    "                    \"brandName\": \"Liquorland\",\n",
    "                    \"tradingHoursToday\": {\n",
    "                        \"openNow\": \"Closed\",\n",
    "                        \"storeTime\": \"reopens at 10am\"\n",
    "                    }\n",
    "                }\n",
    "            ],\n",
    "            \"services\": []\n",
    "        },\n",
    "        {\n",
    "            \"storeId\": \"840\",\n",
    "            \"brandName\": \"Coles Supermarkets\",\n",
    "            \"brandId\": 2,\n",
    "            \"latitude\": -33.865729,\n",
    "            \"longitude\": 151.206146,\n",
    "            \"distance\": 1.74,\n",
    "            \"storeName\": \"Coles Central Wynyard Express\",\n",
    "            \"address\": \"Wynyard St & Carrington St\",\n",
    "            \"suburb\": \"Sydney\",\n",
    "            \"state\": \"NSW\",\n",
    "            \"postcode\": \"2000\",\n",
    "            \"phone\": \"(02) 9299 4769\",\n",
    "            \"tradingHoursToday\": {\n",
    "                \"openNow\": \"Closed\",\n",
    "                \"storeTime\": \"Reopens at 7am\"\n",
    "            },\n",
    "            \"tradingHours\": [\n",
    "                {\n",
    "                    \"daysOfWeek\": \"Mon-Fri\",\n",
    "                    \"storeTime\": \"6am-12am\"\n",
    "                },\n",
    "                {\n",
    "                    \"daysOfWeek\": \"Sat-Sun\",\n",
    "                    \"storeTime\": \"7am-10pm\"\n",
    "                }\n",
    "            ],\n",
    "            \"holidayTradingHours\": [],\n",
    "            \"nextDoorStores\": [],\n",
    "            \"services\": []\n",
    "        }\n",
    "    ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Cole's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_coles_data(lat, lng):\n",
    "    API_URL = \"https://apigw.coles.com.au/digital/colesweb/v1/stores/search\"\n",
    "    params = (\n",
    "        (\"latitude\", \"{}\".format(lat)),\n",
    "        (\"longitude\", \"{}\".format(lng)),\n",
    "        (\"brandIds\", \"2,1\"),\n",
    "        (\"numberOfStores\", \"{}\".format(limit)),\n",
    "    )\n",
    "    rq = requests.get(API_URL, params=params)\n",
    "    if rq.status_code != 200:\n",
    "        print(rq.json())\n",
    "        return False\n",
    "    try:\n",
    "        res = rq.json()\n",
    "        if len(res) == 0:\n",
    "            return False\n",
    "    except:\n",
    "        return False\n",
    "    i = 0\n",
    "    # INSERT POTENTIAL CHECK THAT STORE MATCHES DESIRED STORENAME!! (e.g. res[0][\"name\"])\n",
    "    # e.g.:\n",
    "    # while True:\n",
    "    #     if name != res[i][\"name\"]:\n",
    "    #         i += 1\n",
    "    #     else:\n",
    "    #         break\n",
    "    storeLat, storeLng = (\n",
    "        float(res[\"stores\"][0][\"latitude\"]),\n",
    "        float(res[\"stores\"][0][\"longitude\"]),\n",
    "    )\n",
    "    # compute distance between the two points using the haversine function\n",
    "    storeDistance = haversine((lat, lng), (storeLat, storeLng), unit=\"m\")\n",
    "    if storeDistance > radius:\n",
    "        return False\n",
    "    openingHours = res[\"stores\"][i][\"tradingHours\"]\n",
    "\n",
    "    colesArray = []\n",
    "    dayKeys = {\n",
    "        0: \"Monday\",\n",
    "        1: \"Tuesday\",\n",
    "        2: \"Wednesday\",\n",
    "        3: \"Thursday\",\n",
    "        4: \"Friday\",\n",
    "        5: \"Saturday\",\n",
    "        6: \"Sunday\",\n",
    "    }\n",
    "    intermediaryDayKeys = {\n",
    "        \"Mon\": 0,\n",
    "        \"Tue\": 1,\n",
    "        \"Wed\": 2,\n",
    "        \"Thu\": 3,\n",
    "        \"Fri\": 4,\n",
    "        \"Sat\": 5,\n",
    "        \"Sun\": 6,\n",
    "        None: None,\n",
    "    }\n",
    "\n",
    "    for index in range(len(res[\"stores\"])):\n",
    "        dayDoneSet = set()\n",
    "        daysHeap = []\n",
    "        openingHours = res[\"stores\"][index][\"tradingHours\"]\n",
    "        daysArray = []\n",
    "        if len(openingHours) == 1:\n",
    "            daysArray = [[0, 0], [1, 0], [2, 0], [3, 0], [4, 0], [5, 0], [6, 0]]\n",
    "\n",
    "        else:\n",
    "            dayIndex = 0\n",
    "            for val in range(len(openingHours)):\n",
    "                day = openingHours[val][\"daysOfWeek\"].split(\"-\")\n",
    "                if len(day) < 2:\n",
    "                    dayFirst = day[0]\n",
    "                    dayLast = day[0]\n",
    "                else:\n",
    "                    dayFirst = day[0]\n",
    "                    dayLast = day[1]\n",
    "                for dayVal in range(\n",
    "                    intermediaryDayKeys[dayFirst], intermediaryDayKeys[dayLast] + 1\n",
    "                ):\n",
    "                    daysArray.append([dayVal, dayIndex])\n",
    "                dayIndex += 1\n",
    "\n",
    "        dayCounter = 0\n",
    "        for dayIndex in range(len(daysArray)):\n",
    "            day = dayIndex\n",
    "\n",
    "            dayDone = daysArray[dayIndex][0]\n",
    "            # if statement to deal with days being included twice (implemnted due to a bug in rewe system)\n",
    "            if dayDone not in dayDoneSet:\n",
    "                try:\n",
    "                    dayDoneSet.add(dayDone)\n",
    "                    # keyHours processing code\n",
    "                    timeValue = openingHours[daysArray[dayIndex][1]][\"storeTime\"].split(\n",
    "                        \"-\"\n",
    "                    )\n",
    "                    if len(timeValue) < 2:\n",
    "                        opens = timeValue[0]\n",
    "                        closes = \"null\"\n",
    "                    else:\n",
    "                        if \":\" in timeValue[0] and \":\" not in timeValue[1]:\n",
    "                            opens = datetime.strftime(datetime.strptime(timeValue[0], \"%I:%M%p\"), \"%H:%M\")\n",
    "                            closes = datetime.strftime(datetime.strptime(timeValue[1], \"%I%p\"), \"%H:%M\")\n",
    "                        elif \":\" in timeValue[1] and \":\" not in timeValue[0]:\n",
    "                            opens = datetime.strftime(datetime.strptime(timeValue[0], \"%I%p\"), \"%H:%M\")\n",
    "                            closes = datetime.strftime(datetime.strptime(timeValue[1], \"%I:%M%p\"), \"%H:%M\")\n",
    "                        elif \":\" in timeValue[0] and \":\" in timeValue[1]:\n",
    "                            opens = datetime.strftime(datetime.strptime(timeValue[0], \"%I:%M%p\"), \"%H:%M\")\n",
    "                            closes = datetime.strftime(datetime.strptime(timeValue[1], \"%I:%M%p\"), \"%H:%M\")\n",
    "                        else:\n",
    "                            opens = datetime.strftime(datetime.strptime(timeValue[0], \"%I%p\"), \"%H:%M\")\n",
    "                            closes = datetime.strftime(datetime.strptime(timeValue[1], \"%I%p\"), \"%H:%M\")\n",
    "\n",
    "                    if opens == closes:\n",
    "                        closedDayHours = {\n",
    "                            \"day\": dayKeys[dayDone],\n",
    "                            \"open\": False,\n",
    "                        }\n",
    "                        heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "                    actualHours = {\"open\": opens, \"close\": closes}\n",
    "                    keyHours = {\n",
    "                        \"day\": dayKeys[dayDone],\n",
    "                        \"open\": True,\n",
    "                        \"hours\": [actualHours],\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [day, keyHours])\n",
    "\n",
    "                except:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": dayKeys[dayDone],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [day, closedDayHours])\n",
    "\n",
    "        # Check for any missing days and ensure all days are in order for insertion into our database\n",
    "        for day in daysHeap:\n",
    "            if day[1][\"day\"] != dayKeys[day[0]]:\n",
    "                day[0] = intermediaryDayKeys[day[1][\"day\"]]\n",
    "\n",
    "        if len(daysHeap) < 7:\n",
    "            for dayDone in dayKeys.keys():\n",
    "                if dayDone not in dayDoneSet:\n",
    "                    closedDayHours = {\n",
    "                        \"day\": dayKeys[dayDone],\n",
    "                        \"open\": False,\n",
    "                    }\n",
    "                    heapq.heappush(daysHeap, [dayDone, closedDayHours])\n",
    "\n",
    "        hoursArray = [i[1] for i in heapq.nsmallest(7, daysHeap)]\n",
    "        latitude = res[\"stores\"][index][\"latitude\"]\n",
    "        longitude = res[\"stores\"][index][\"longitude\"]\n",
    "        colesArray.append([(float(latitude), float(longitude)), hoursArray])\n",
    "    return colesArray"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# X's API\n",
    "## Base URL:\n",
    "\n",
    "## Parameters:\n",
    "\n",
    "\n",
    "## Sample API Call:\n",
    "\n",
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving X's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Y's API (Germany)\n",
    "## Base URL:\n",
    "\n",
    "## Parameters:\n",
    "\n",
    "\n",
    "## Sample API Call:\n",
    "\n",
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Y's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Z's API (Germany)\n",
    "## Base URL:\n",
    "\n",
    "## Parameters:\n",
    "\n",
    "\n",
    "## Sample API Call:\n",
    "\n",
    "## Sample Response:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code for retrieving Z's opening hours data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}