mission_to_mars.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Title](Images/mission_to_mars.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mission to Mars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scrape sites to retrieve data. Use Pandas, Beautiful Soup, Splinter, etc. to parse and clean data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NASA Mars News Site (https://mars.nasa.gov/news/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NASA Mars News\n",
    "\n",
    "* Scrape the [NASA Mars News Site](https://mars.nasa.gov/news/) and collect the latest News Title and Paragraph Text. Assign the text to variables that you can reference later.\n",
    "\n",
    "```python\n",
    "# Example:\n",
    "news_title = \"NASA's Next Mars Mission to Investigate Interior of Red Planet\"\n",
    "\n",
    "news_p = \"Preparation of NASA's next spacecraft to Mars, InSight, has ramped up this summer, on course for launch next May from Vandenberg Air Force Base in central California -- the first interplanetary launch in history from America's West Coast.\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import all libraries\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "from bs4 import BeautifulSoup\n",
    "import pymongo\n",
    "from splinter import Browser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# launch chrome driver\n",
    "executable_path = {'executable_path': '/Users/prettyvo/Downloads/chromedriver2'}\n",
    "browser = Browser('chrome', **executable_path, headless=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect browser to url and parse data images of the titles and summaries\n",
    "mars_url = \"https://mars.nasa.gov/news/\"\n",
    "browser.visit(mars_url)\n",
    "html = browser.html\n",
    "mars_news = BeautifulSoup(html, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# titles\n",
    "headlines = mars_news.find_all('div', class_='content_title')\n",
    "titles = [headline.text.strip() for headline in headlines][0]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# summary data\n",
    "details = mars_news.find_all('div', class_='article_teaser_body')\n",
    "summary = [detail.text.strip() for detail in details]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JPL Mars Space Images - Featured Image\n",
    "\n",
    "* Visit the url for JPL Featured Space Image [here](https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars).\n",
    "\n",
    "* Use splinter to navigate the site and find the image url for the current Featured Mars Image and assign the url string to a variable called `featured_image_url`.\n",
    "\n",
    "* Make sure to find the image url to the full size `.jpg` image.\n",
    "\n",
    "* Make sure to save a complete url string for this image.\n",
    "\n",
    "```python\n",
    "# Example:\n",
    "featured_image_url = 'https://www.jpl.nasa.gov/spaceimages/images/largesize/PIA16225_hires.jpg'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect browser to url and parse data for image\n",
    "jpl_url = \"https://www.jpl.nasa.gov/spaceimages/?search=&category=Mars\"\n",
    "browser.visit(jpl_url)\n",
    "html = browser.html\n",
    "mars_jpl_image = BeautifulSoup(html, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.jpl.nasa.gov/spaceimages/images/mediumsize/PIA17009_ip.jpg\n"
     ]
    }
   ],
   "source": [
    "# collect the link for the image\n",
    "images = mars_jpl_image.find_all('div', class_='default floating_text_area ms-layer')\n",
    "img = [image.a['data-fancybox-href'] for image in images]\n",
    "featured_image_url = \"https://www.jpl.nasa.gov\" + img[0]\n",
    "x = print(featured_image_url)\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mars Weather\n",
    "\n",
    "* Visit the Mars Weather twitter account [here](https://twitter.com/marswxreport?lang=en) and scrape the latest Mars weather tweet from the page. Save the tweet text for the weather report as a variable called `mars_weather`.\n",
    "\n",
    "```python\n",
    "# Example:\n",
    "mars_weather = 'Sol 1801 (Aug 30, 2017), Sunny, high -21C/-5F, low -80C/-112F, pressure at 8.82 hPa, daylight 06:09-17:55'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect browser to url and parse data for tweet\n",
    "mars_tweet_url = \"https://twitter.com/marswxreport?lang=en\"\n",
    "browser.visit(mars_tweet_url)\n",
    "html = browser.html\n",
    "mars_twitter = BeautifulSoup(html, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InSight sol 136 (2019-04-14) low -97.3ºC (-143.2ºF) high -16.5ºC (2.4ºF)\n",
      "winds from the WNW at 4.3 m/s (9.6 mph) gusting to 10.8 m/s (24.2 mph)\n",
      "pressure at 7.30 hPapic.twitter.com/OJFvEUR8OR\n"
     ]
    }
   ],
   "source": [
    "tweets = mars_twitter.find_all('p', class_='TweetTextSize TweetTextSize--normal js-tweet-text tweet-text')\n",
    "mars_weather = [tweet.text for tweet in tweets][0]\n",
    "print(mars_weather)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mars Facts\n",
    "\n",
    "* Visit the Mars Facts webpage [here](https://space-facts.com/mars/) and use Pandas to scrape the table containing facts about the planet including Diameter, Mass, etc.\n",
    "\n",
    "* Use Pandas to convert the data to a HTML table string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect browser to url and parse data for tweet\n",
    "mars_facts_url = \"https://space-facts.com/mars/\"\n",
    "browser.visit(mars_facts_url)\n",
    "html = browser.html\n",
    "mars_facts = BeautifulSoup(html, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<table border=\"1\" class=\"dataframe\">\n",
      "  <thead>\n",
      "    <tr style=\"text-align: right;\">\n",
      "      <th></th>\n",
      "      <th>Data</th>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <th>Physical</th>\n",
      "      <th></th>\n",
      "    </tr>\n",
      "  </thead>\n",
      "  <tbody>\n",
      "    <tr>\n",
      "      <th>Equatorial Diameter</th>\n",
      "      <td>6,792 km</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <th>Polar Diameter</th>\n",
      "      <td>6,752 km</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <th>Mass</th>\n",
      "      <td>6.42 x 10^23 kg (10.7% Earth)</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <th>Moons</th>\n",
      "      <td>2 (Phobos &amp; Deimos)</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <th>Orbit Distance</th>\n",
      "      <td>227,943,824 km (1.52 AU)</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <th>Orbit Period</th>\n",
      "      <td>687 days (1.9 years)</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <th>Surface Temperature</th>\n",
      "      <td>-153 to 20 °C</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <th>First Record</th>\n",
      "      <td>2nd millennium BC</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <th>Recorded By</th>\n",
      "      <td>Egyptian astronomers</td>\n",
      "    </tr>\n",
      "  </tbody>\n",
      "</table>\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Data</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Physical</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Equatorial Diameter</th>\n",
       "      <td>6,792 km</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Polar Diameter</th>\n",
       "      <td>6,752 km</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mass</th>\n",
       "      <td>6.42 x 10^23 kg (10.7% Earth)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Moons</th>\n",
       "      <td>2 (Phobos &amp; Deimos)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Orbit Distance</th>\n",
       "      <td>227,943,824 km (1.52 AU)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              Data\n",
       "Physical                                          \n",
       "Equatorial Diameter                       6,792 km\n",
       "Polar Diameter                            6,752 km\n",
       "Mass                 6.42 x 10^23 kg (10.7% Earth)\n",
       "Moons                          2 (Phobos & Deimos)\n",
       "Orbit Distance            227,943,824 km (1.52 AU)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "facts = mars_facts.find_all('table', id='tablepress-mars')\n",
    "table = [fact.text for fact in facts]\n",
    "data = list(filter(None, table[0].split('\\n')))\n",
    "result = []\n",
    "\n",
    "for i in range(0,9):\n",
    "    separated = data[i].split(':')\n",
    "    results = ({\n",
    "        'Physical' : separated[0],\n",
    "        'Data' :separated[1]\n",
    "    })\n",
    "    result.append(results)\n",
    "df = pd.DataFrame(result)\n",
    "df = df[['Physical', 'Data']].set_index(['Physical'])\n",
    "mars_facts_table = print(df.to_html().strip())\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mars Hemispheres\n",
    "\n",
    "* Visit the USGS Astrogeology site [here](https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars) to obtain high resolution images for each of Mar's hemispheres.\n",
    "\n",
    "* You will need to click each of the links to the hemispheres in order to find the image url to the full resolution image.\n",
    "\n",
    "* Save both the image url string for the full resolution hemisphere image, and the Hemisphere title containing the hemisphere name. Use a Python dictionary to store the data using the keys `img_url` and `title`.\n",
    "\n",
    "* Append the dictionary with the image url string and the hemisphere title to a list. This list will contain one dictionary for each hemisphere.\n",
    "\n",
    "```python\n",
    "# Example:\n",
    "hemisphere_image_urls = [\n",
    "    {\"title\": \"Valles Marineris Hemisphere\", \"img_url\": \"...\"},\n",
    "    {\"title\": \"Cerberus Hemisphere\", \"img_url\": \"...\"},\n",
    "    {\"title\": \"Schiaparelli Hemisphere\", \"img_url\": \"...\"},\n",
    "    {\"title\": \"Syrtis Major Hemisphere\", \"img_url\": \"...\"},\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect browser to url and parse data images of the hemisphere\n",
    "mars_hemisphere_url = \"https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars\"\n",
    "browser.visit(mars_hemisphere_url)\n",
    "html = browser.html\n",
    "mars_hemisphere = BeautifulSoup(html, 'html.parser')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MARS HEMISPHERES\n",
    "# connect browser to url and parse data images of the hemisphere\n",
    "mars_hemisphere_url = \"https://astrogeology.usgs.gov/search/results?q=hemisphere+enhanced&k1=target&v1=Mars\"\n",
    "browser.visit(mars_hemisphere_url)\n",
    "html = browser.html\n",
    "mars_hemisphere = BeautifulSoup(html, 'html.parser')\n",
    "hemisphere_imgs = mars_hemisphere.find_all('div', class_='description')\n",
    "img_link = [img.a['href'] for img in hemisphere_imgs]\n",
    "hem_title = [img.a.text for img in hemisphere_imgs]\n",
    "hemisphere_image_urls = []\n",
    "for j in range(0,4):\n",
    "\n",
    "    res_link = 'https://astrogeology.usgs.gov/' + img_link[j]\n",
    "    browser.visit(res_link)\n",
    "    res_hemisphere = BeautifulSoup(browser.html, 'html.parser')\n",
    "    hemispheres = res_hemisphere.find_all('div', class_='downloads')\n",
    "    img_url = [hemisphere.a['href'] for hemisphere in hemispheres]\n",
    "    hemisphere_image_dict = ({\n",
    "        'title': hem_title[j],\n",
    "        'img_url': img_url[0]\n",
    "    })\n",
    "    hemisphere_image_urls.append(hemisphere_image_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'title': 'Cerberus Hemisphere Enhanced', 'img_url': 'http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/cerberus_enhanced.tif/full.jpg'}\n",
      "{'title': 'Schiaparelli Hemisphere Enhanced', 'img_url': 'http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/schiaparelli_enhanced.tif/full.jpg'}\n",
      "{'title': 'Syrtis Major Hemisphere Enhanced', 'img_url': 'http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/syrtis_major_enhanced.tif/full.jpg'}\n",
      "{'title': 'Valles Marineris Hemisphere Enhanced', 'img_url': 'http://astropedia.astrogeology.usgs.gov/download/Mars/Viking/valles_marineris_enhanced.tif/full.jpg'}\n"
     ]
    }
   ],
   "source": [
    "for i in hemisphere_image_urls:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "mars_data = {}\n",
    "# news headline\n",
    "mars_data['news_headline'] = titles\n",
    "mars_data['news_summary'] = summary\n",
    "# JPL featured photo\n",
    "mars_data['featured_image_url'] = featured_image_url\n",
    "# mars facts\n",
    "mars_data['table'] = table\n",
    "#mars weather\n",
    "mars_data['mars_weather'] = mars_weather\n",
    "# mars hemispheres\n",
    "mars_data['hemisphere_image_urls'] = hemisphere_image_urls\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:PythonData]",
   "language": "python",
   "name": "pythondata"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}