Appendix 07 - Gender Inference - Results Testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix 7 - Gender Inference - Results Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: When running the rule based gender classification algorithm on the dataset, we used the version of the raw, unprocessed dataset, as slightly different pre-processing was required to get accurate results from the gender algorithm (tokenisation of the description field, lowercase text).\n",
    "\n",
    "The final rule based algorithm classified our full dataset as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<bound method NDFrame.head of                                   date  \\\n",
      "0       Wed Jan 31 03:30:46 +0000 2018   \n",
      "1       Wed Jan 31 03:30:46 +0000 2018   \n",
      "2       Wed Jan 31 03:30:46 +0000 2018   \n",
      "3       Wed Jan 31 03:30:46 +0000 2018   \n",
      "4       Wed Jan 31 03:30:46 +0000 2018   \n",
      "5       Wed Jan 31 03:30:46 +0000 2018   \n",
      "6       Wed Jan 31 03:30:46 +0000 2018   \n",
      "7       Wed Jan 31 03:30:46 +0000 2018   \n",
      "8       Wed Jan 31 03:30:46 +0000 2018   \n",
      "9       Wed Jan 31 03:30:46 +0000 2018   \n",
      "10      Wed Jan 31 03:30:46 +0000 2018   \n",
      "11      Wed Jan 31 03:30:46 +0000 2018   \n",
      "12      Wed Jan 31 03:30:46 +0000 2018   \n",
      "13      Wed Jan 31 03:30:46 +0000 2018   \n",
      "14      Wed Jan 31 03:30:46 +0000 2018   \n",
      "15      Wed Jan 31 03:30:46 +0000 2018   \n",
      "16      Wed Jan 31 03:30:46 +0000 2018   \n",
      "17      Wed Jan 31 03:30:46 +0000 2018   \n",
      "18      Wed Jan 31 03:30:46 +0000 2018   \n",
      "19      Wed Jan 31 03:30:46 +0000 2018   \n",
      "20      Wed Jan 31 03:30:46 +0000 2018   \n",
      "21      Wed Jan 31 03:30:46 +0000 2018   \n",
      "22      Wed Jan 31 03:30:46 +0000 2018   \n",
      "23      Wed Jan 31 03:30:46 +0000 2018   \n",
      "24      Wed Jan 31 03:30:46 +0000 2018   \n",
      "25      Wed Jan 31 03:30:46 +0000 2018   \n",
      "26      Wed Jan 31 03:30:46 +0000 2018   \n",
      "27      Wed Jan 31 03:30:46 +0000 2018   \n",
      "28      Wed Jan 31 03:30:46 +0000 2018   \n",
      "29      Wed Jan 31 03:30:46 +0000 2018   \n",
      "...                                ...   \n",
      "274490  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274491  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274492  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274493  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274494  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274495  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274496  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274497  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274498  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274499  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274500  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274501  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274502  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274503  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274504  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274505  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274506  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274507  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274508  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274509  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274510  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274511  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274512  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274513  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274514  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274515  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274516  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274517  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274518  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274519  Wed Jan 31 02:10:14 +0000 2018   \n",
      "\n",
      "                                              description  favorite_count  \\\n",
      "0                                                     NaN               0   \n",
      "1                                         Waiting for FDR               1   \n",
      "2                                 ENFP; tweets are my own               1   \n",
      "3       It takes more than embarrassment to shame me t...               2   \n",
      "4       4 1/2 billion years of evolution produced our ...               0   \n",
      "5                      Jeremiah 18:1-6 & Revelation 12:11               3   \n",
      "6       Wife to Jay. Mommy to Zachary Louis. Kitty mam...               1   \n",
      "7       Education is the most important thing in stopp...               0   \n",
      "8       Baltimore native. Nerd for pop, politics, snar...               0   \n",
      "9       https://t.co/FNmWSgyYlM Senior Columnist, husb...               1   \n",
      "10      Your source for original reporting and trusted...             101   \n",
      "11      #MilkyWayMood  #LoveFullTilt  #TheSoulTheEssen...               1   \n",
      "12                                                    NaN               0   \n",
      "13      Love sports, dogs, music and cannabis. HATE ra...               1   \n",
      "14                                                    NaN               0   \n",
      "15      Twitter's best kept secret. Former SIGINT Anal...              33   \n",
      "16      💡design grad student @ucdavis 〰️ electric bass...               3   \n",
      "17      Proud father.  Lucky Husband.  Cheering on my ...               0   \n",
      "18      Founder of Holy Koolaid LLC\\nhttps://t.co/IkaQ...               0   \n",
      "19      Utah's #1 radio station. Traffic and weather e...               2   \n",
      "20      Home of KSL 5 News and Studio 5, and the local...               9   \n",
      "21      Used to be athletic, smart, and ambitious. Now...               1   \n",
      "22      Computer scientist, progressive, christian, an...               1   \n",
      "23      A Patriot who is tired of the corruption in ou...               1   \n",
      "24      🎙Host Jason Rantz Show on KTTH 770AM in Seattl...               0   \n",
      "25      I'm special because I'm me.  Progressive (Dem)...               3   \n",
      "26      Editor for the @FlorhamPEagle, staff writer fo...               0   \n",
      "27      Political & Investigative Reporter for @KCBSNe...               4   \n",
      "28      Enjoys politics, podcasts, post-impressionists...               0   \n",
      "29      Proud to serve the First District of West Virg...               5   \n",
      "...                                                   ...             ...   \n",
      "274490  Blocked by Goldust. It's about the wrestling. ...               0   \n",
      "274491  My words are my own. Except when I'm quoting s...               0   \n",
      "274492       mostly spotify links https://t.co/v4x39TpF6N               2   \n",
      "274493  The Suffolk Journal is the award-winning under...               0   \n",
      "274494  Don't whine, vote. And tell your family & frie...               9   \n",
      "274495  🇱🇷 TRUMPLICAN 🇺🇸MAGA💯 SUPPORTING THE MAGA AGEN...               0   \n",
      "274496                        Married AF. I love my life.               0   \n",
      "274497  23. College student. I’ve been winning all my ...               0   \n",
      "274498  writer. lab rat. purveyor of fine energy drink...               1   \n",
      "274499  Official account of the @NALCAB Policy Team. W...               0   \n",
      "274500  Married to Myra. Chief of Staff, @RepThomasMas...               0   \n",
      "274501  If you know me, you don't need the useless inf...               0   \n",
      "274502  Economic Policy reporter at The New York Times...               4   \n",
      "274503  Award-winning reporter w/the New York Daily Ne...               0   \n",
      "274504  Artist, Self Help, Writer, News, Christian Bel...               0   \n",
      "274505  Egalitarian. Some tweets are ironic, some as s...               1   \n",
      "274506  Terry Kinder: God knows who I am. Tweets are n...               0   \n",
      "274507  the great matters are often as worthless as aﬆ...               0   \n",
      "274508  Lover of all things Beauty & Politics! Wife. E...               5   \n",
      "274509  Constitutional & Fiscal Conservative. \\nProMil...               1   \n",
      "274510  Haitian Air Force F-52 Pilot. \\n!!Les Trous de...               0   \n",
      "274511  Winner of @OneRepublic singing competition. Tr...              37   \n",
      "274512                                                NaN               0   \n",
      "274513  Humor, Hiking, Photography, Sarcastic,  Conser...               0   \n",
      "274514  Twitter- any idiot elected to the highest offi...               0   \n",
      "274515  You can make a lot of sacrifices and keep ever...               0   \n",
      "274516  #NOLA bred. @HowardU educated. @SiriusXM emplo...               0   \n",
      "274517  R.I.P. Nayna Masi-12/9/1969 to 6/12/2015 & Lil...               1   \n",
      "274518                          Enough about me, already.               0   \n",
      "274519  ....Actually, throughout my life, my two great...               0   \n",
      "\n",
      "        followers_count              id_str                        location  \\\n",
      "0                   163  958543051391229952                Freakin' Florida   \n",
      "1                  1419  958543051248603136                        Brooklyn   \n",
      "2                    65  958543051080781824                          Boston   \n",
      "3                  1370  958543050967568384                             NaN   \n",
      "4                   346  958543050510405632                           Texas   \n",
      "5                   231  958543050472640512                       Fargo, ND   \n",
      "6                  1814  958543050468483072  Raleigh NC but a NYer at heart   \n",
      "7                    57  958543050468360192                 Knightstown, IN   \n",
      "8                  1318  958543050338439168                              DC   \n",
      "9                  9396  958543050300710912                     Merrick, NY   \n",
      "10              6386672  958543050199961600                    New York, NY   \n",
      "11                  411  958543050158026752            CAROLINABLUE2DAHEART   \n",
      "12                   10  958543050090909696                             NaN   \n",
      "13                 2667  958543049977548800                          NorCal   \n",
      "14                   29  958543049885454336                             NaN   \n",
      "15                 3791  958543049860071424                             NaN   \n",
      "16                  264  958543049826562048                 California, USA   \n",
      "17                   42  958543049629618176                Mullica Hill, NJ   \n",
      "18                 1815  958543049457590272                             NaN   \n",
      "19                19499  958543049306460160              Salt Lake City, UT   \n",
      "20                63800  958543049285517312            Salt Lake City, Utah   \n",
      "21                  175  958543049277263872                     Chicago, IL   \n",
      "22                 1004  958543049155543040                             NaN   \n",
      "23                 6744  958543049025466368                             NaN   \n",
      "24                 5562  958543048937324544                      Seattle WA   \n",
      "25                   70  958543048547491840                 San Antonio, TX   \n",
      "26                  526  958543048539074560                 New Jersey, USA   \n",
      "27                 7482  958543048509571072       San Francisco, California   \n",
      "28                  920  958543048408879104                 Los Angeles, CA   \n",
      "29                27025  958543048346128384                             NaN   \n",
      "...                 ...                 ...                             ...   \n",
      "274490              283  958522782962286592                         The 716   \n",
      "274491               82  958522782924419072                     Los Angeles   \n",
      "274492               35  958522782903549952                          brophy   \n",
      "274493             2738  958522782828105728                      Boston, MA   \n",
      "274494            13973  958522782811336704                   United States   \n",
      "274495               46  958522782798802944                Somewhere U.S.A.   \n",
      "274496              153  958522782782025728                     Orleans, MA   \n",
      "274497             9077  958522782739963904               Las Vegas, Nevada   \n",
      "274498             6622  958522782660288512                         Seattle   \n",
      "274499               32  958522782480044032                             NaN   \n",
      "274500             6560  958522782362546176            Bloomington, IN, USA   \n",
      "274501              315  958522782215700480              West Lafayette, IN   \n",
      "274502            12834  958522781909553152                   Washington DC   \n",
      "274503             1442  958522781871812608                    Brooklyn, NY   \n",
      "274504            80153  958522781817122816                      California   \n",
      "274505             2417  958522781687312384                  Beside the sea   \n",
      "274506            58752  958522781653786624                     Jackson, MO   \n",
      "274507              704  958522781557170176       Edmonton, Alberta, Canada   \n",
      "274508             8213  958522781460701184                 California, USA   \n",
      "274509             4652  958522781280489472                  Tennessee, USA   \n",
      "274510                0  958522781246816256                           Haiti   \n",
      "274511             4102  958522781162975232                         Chicago   \n",
      "274512               70  958522781158772736                  North Carolina   \n",
      "274513              155  958522780940623872                     Phoenix, AZ   \n",
      "274514               94  958522780789694464              Point Pleasant, NJ   \n",
      "274515              650  958522780764557312        Worcester, Massachusetts   \n",
      "274516              566  958522780747751424                             DMV   \n",
      "274517              190  958522780705787904                Philadelphia, PA   \n",
      "274518              221  958522780571418624                 los angeles, ca   \n",
      "274519               73  958522780466622464            Brisbane, Queensland   \n",
      "\n",
      "        retweet_count  statuses_count  \\\n",
      "0                   0            8723   \n",
      "1                   1           20287   \n",
      "2                   0             373   \n",
      "3                   0           35380   \n",
      "4                   0           10621   \n",
      "5                   0            2210   \n",
      "6                   0           26747   \n",
      "7                   0            2990   \n",
      "8                   0           10412   \n",
      "9                   0           17983   \n",
      "10                 60          161400   \n",
      "11                  0           19042   \n",
      "12                  0               2   \n",
      "13                  0           36898   \n",
      "14                  0             190   \n",
      "15                  5            3943   \n",
      "16                  0            1904   \n",
      "17                  0            7861   \n",
      "18                  0            4424   \n",
      "19                  0           34700   \n",
      "20                  0           66383   \n",
      "21                  0           35013   \n",
      "22                  0           84832   \n",
      "23                  1           35763   \n",
      "24                  0           35332   \n",
      "25                  1            1542   \n",
      "26                  0            5253   \n",
      "27                  1           29816   \n",
      "28                  0            4045   \n",
      "29                  0            4741   \n",
      "...               ...             ...   \n",
      "274490              0            7667   \n",
      "274491              0             858   \n",
      "274492              0             780   \n",
      "274493              0            6704   \n",
      "274494              0           58619   \n",
      "274495              1             726   \n",
      "274496              0            6417   \n",
      "274497              2          194629   \n",
      "274498              0           46820   \n",
      "274499              0              72   \n",
      "274500              0           19160   \n",
      "274501              0            3027   \n",
      "274502              5           10473   \n",
      "274503              0            4875   \n",
      "274504              0           82523   \n",
      "274505              0           38990   \n",
      "274506              0          123387   \n",
      "274507              0            8793   \n",
      "274508              0           73992   \n",
      "274509              0           13365   \n",
      "274510              0              13   \n",
      "274511             14            2573   \n",
      "274512              0            2116   \n",
      "274513              0            1201   \n",
      "274514              0            6827   \n",
      "274515              0            7536   \n",
      "274516              0           18287   \n",
      "274517              0           22544   \n",
      "274518              0             844   \n",
      "274519              0            2608   \n",
      "\n",
      "                                                     text             user  \\\n",
      "0       Retweeted ♻️ Christopher Zullo (@ChrisJZullo):...        Likipedia   \n",
      "1       I wish someone would have told me that I could...          CoreyCW   \n",
      "2       Surprise cameo: Science found its way into the...       craigregis   \n",
      "3       That's odd. He said he likes music. But I dist...        kimwithak   \n",
      "4       He doesn't have any standing to comment on fam...     CraneStation   \n",
      "5       If you didn't watch the State of the Union, yo...       jeremykopp   \n",
      "6       Remember that time that Stephen Miller said th...        Colleen84   \n",
      "7       Ok time to pull the plug (about 30 minut s ago...         wabashcc   \n",
      "8       #DoD has done far better than domestic program...      SaraLDuBois   \n",
      "9       Here's my take on the #SOTU https://t.co/QzJt8...       jakejakeny   \n",
      "10      Pres. Trump calls for the end of chain migrati...          CBSNews   \n",
      "11      The look you give when a know-it-all  can't re...      JAZZYONEMEL   \n",
      "12      Thank you president Trump. You are greatly app...     Jeff56566268   \n",
      "13      @FrankLuntz I just wish he’d STFU, go home to ...     imthe420nerd   \n",
      "14      .@POTUS “Above all else, they are Americans, a...         sccmigop   \n",
      "15      Since I’m not watching the #sotu I’ll guess tr...   deepstaterogue   \n",
      "16                            What an embarrassment #SOTU       AdamMusing   \n",
      "17      Trump must want to keep talking because he doe...        jerkat429   \n",
      "18      I don't think I've ever seen a president smirk...      holykoolaid   \n",
      "19      .@realDonaldTrump: The people dreamed this cou...     kslnewsradio   \n",
      "20      .@realDonaldTrump: The people dreamed this cou...           KSL5TV   \n",
      "21      It’d be a real shame if the plot of Designated...      NotHotTakes   \n",
      "22      #PuertoRico when @realDonaldTrump says \"We lov...      sephius1999   \n",
      "23      The people who are making America Great Again....      DAColdriver   \n",
      "24      Okay, this speech was over 4 minutes ago (minu...       jasonrantz   \n",
      "25      #SOTU Immigrants built this country you FUckin...      annrollins0   \n",
      "26      “It’s the people that are making America great...         Cleespot   \n",
      "27      And...scene. That was the 3rd longest #sotu in...     SovernNation   \n",
      "28      Trump wrapping mostly (and surprisingly) on sc...       katielewDC   \n",
      "29      “It is the people who are making America great...      RepMcKinley   \n",
      "...                                                   ...              ...   \n",
      "274490              Oh man, his ego is LOVING this. #SOTU          LvkeTCB   \n",
      "274491  Let's be honest. This Republican-driven applau...        CandidlyB   \n",
      "274492               watching #sotu because i hate myself     lance_keeble   \n",
      "274493  Many Democratic Congresswomen are boycotting t...   SuffolkJournal   \n",
      "274494  The pundits say Trump has been working hard on...      pittgriffin   \n",
      "274495  ......MY POTUS.... \\n#TuesdayThoughts \\n#MAGA ...     ORIGINAL_ME1   \n",
      "274496  Currently watching a group of demons give Sata...       BremTweets   \n",
      "274497  We really have a reality tv show president. #sotu       J0rdan0re0   \n",
      "274498  There has to be a Huey Lewis &amp; The News co...        RDS3attle   \n",
      "274499  NALCAB applauds those members of Congress who ...     NALCABPolicy   \n",
      "274500      Live #SOTU broadcast  https://t.co/V3X1AiVcDf         jimpfaff   \n",
      "274501     #SOTU finally starting https://t.co/hfA2bFE354     nielsen_karl   \n",
      "274502  The @nytimes #SOTU live chat is here ---&gt; h...       arappeport   \n",
      "274503                                   Here we go #sotu     ChrisCarrega   \n",
      "274504  #SOTU God bless and protect #POTUS #USA #Relea...           HPSelf   \n",
      "274505  The clapping was good, unfortunately it's all ...       DamonMercy   \n",
      "274506        DEMS Behave?  https://t.co/kW1o7AhZKm #SOTU          tkinder   \n",
      "274507       This one's for #SOTU https://t.co/ws6Zz0ingx        pabinkley   \n",
      "274508  You can see Nancy Pelosi trying to calm her pe...     Mrsmmtbeauty   \n",
      "274509  @TheJuanWilliams @FoxNews @FoxNewsOpinion Isn’...        ISaidKnow   \n",
      "274510  Betsy Devos outfit cost more den me house. Oui...    CovfefeLeBrun   \n",
      "274511  They’re not applauding, they’re a$$ kissing. #...    IsabellaCigno   \n",
      "274512  So nice to watch a #SOTU again! #MAGA #SOTU201...          naine11   \n",
      "274513  #SOTU\\nFirst impression is of #MelaniaTrump . ...   SarcasticRusty   \n",
      "274514            This is already an embarrassment. #SOTU  MatthewJDonovan   \n",
      "274515  How can those senators and congressmen stand t...           JaFlo1   \n",
      "274516  POTUS clapping for himself at the #SOTU podium...     Charla4Mayor   \n",
      "274517  It’s so annoying when they get up after every ...        Jatinb811   \n",
      "274518  Will somebody wake me when he gets to the Stor...     PacoMccauley   \n",
      "274519  I'm sure this level of applause happened under...       kelsitonin   \n",
      "\n",
      "        verified  \n",
      "0          False  \n",
      "1          False  \n",
      "2          False  \n",
      "3          False  \n",
      "4          False  \n",
      "5          False  \n",
      "6          False  \n",
      "7          False  \n",
      "8          False  \n",
      "9          False  \n",
      "10          True  \n",
      "11         False  \n",
      "12         False  \n",
      "13         False  \n",
      "14         False  \n",
      "15         False  \n",
      "16         False  \n",
      "17         False  \n",
      "18         False  \n",
      "19          True  \n",
      "20          True  \n",
      "21         False  \n",
      "22         False  \n",
      "23         False  \n",
      "24          True  \n",
      "25         False  \n",
      "26         False  \n",
      "27          True  \n",
      "28         False  \n",
      "29          True  \n",
      "...          ...  \n",
      "274490     False  \n",
      "274491     False  \n",
      "274492     False  \n",
      "274493     False  \n",
      "274494     False  \n",
      "274495     False  \n",
      "274496     False  \n",
      "274497     False  \n",
      "274498     False  \n",
      "274499     False  \n",
      "274500     False  \n",
      "274501     False  \n",
      "274502      True  \n",
      "274503      True  \n",
      "274504     False  \n",
      "274505     False  \n",
      "274506     False  \n",
      "274507     False  \n",
      "274508     False  \n",
      "274509     False  \n",
      "274510     False  \n",
      "274511     False  \n",
      "274512     False  \n",
      "274513     False  \n",
      "274514     False  \n",
      "274515     False  \n",
      "274516     False  \n",
      "274517     False  \n",
      "274518     False  \n",
      "274519     False  \n",
      "\n",
      "[274520 rows x 11 columns]>\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Import the raw dataset\n",
    "tweets = pd.read_excel(\"01sotu_final_corpus.xlsx\")\n",
    "\n",
    "print(tweets.head)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2013, 4)\n",
      "(1944, 4)\n"
     ]
    }
   ],
   "source": [
    "#PREPARATION OF NAME DATA\n",
    "\n",
    "# Import dataset of names and their genders\n",
    "namesfinal = pd.read_excel(\"top2000names2014final.xlsx\")\n",
    "\n",
    "# Sort names dataframe so longest names come first (so that Erica will then match against Erica and not Eric)\n",
    "namesfinal[\"namelength\"] = \"\"\n",
    "for i in range(len(namesfinal)):\n",
    "    namesfinal.at[i, \"namelength\"] = len(namesfinal.at[i, \"name\"])\n",
    "namesfinal = namesfinal.sort_values(by=['namelength'], ascending=False)\n",
    "namesfinal = namesfinal.reset_index(drop=True)\n",
    "print(namesfinal.shape)\n",
    "\n",
    "# Remove any names 3 characters or less\n",
    "namesfinal.drop(namesfinal[namesfinal.namelength < 4].index, inplace=True)\n",
    "print(namesfinal.shape)\n",
    "\n",
    "# Create a column with name converted to lowercase\n",
    "namesfinal[\"name_lower\"] = namesfinal[\"name\"].str.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1000\n",
      "2000\n",
      "3000\n",
      "4000\n",
      "5000\n",
      "6000\n",
      "7000\n",
      "8000\n",
      "9000\n",
      "10000\n",
      "11000\n",
      "12000\n",
      "13000\n",
      "14000\n",
      "15000\n",
      "16000\n",
      "17000\n",
      "18000\n",
      "19000\n",
      "20000\n",
      "21000\n",
      "22000\n",
      "23000\n",
      "24000\n",
      "25000\n",
      "26000\n",
      "27000\n",
      "28000\n",
      "29000\n",
      "30000\n",
      "31000\n",
      "32000\n",
      "33000\n",
      "34000\n",
      "35000\n",
      "36000\n",
      "37000\n",
      "38000\n",
      "39000\n",
      "40000\n",
      "41000\n",
      "42000\n",
      "43000\n",
      "44000\n",
      "45000\n",
      "46000\n",
      "47000\n",
      "48000\n",
      "49000\n",
      "50000\n",
      "51000\n",
      "52000\n",
      "53000\n",
      "54000\n",
      "55000\n",
      "56000\n",
      "57000\n",
      "58000\n",
      "59000\n",
      "60000\n",
      "61000\n",
      "62000\n",
      "63000\n",
      "64000\n",
      "65000\n",
      "66000\n",
      "67000\n",
      "68000\n",
      "69000\n",
      "70000\n",
      "71000\n",
      "72000\n",
      "73000\n",
      "74000\n",
      "75000\n",
      "76000\n",
      "77000\n",
      "78000\n",
      "79000\n",
      "80000\n",
      "81000\n",
      "82000\n",
      "83000\n",
      "84000\n",
      "85000\n",
      "86000\n",
      "87000\n",
      "88000\n",
      "89000\n",
      "90000\n",
      "91000\n",
      "92000\n",
      "93000\n",
      "94000\n",
      "95000\n",
      "96000\n",
      "97000\n",
      "98000\n",
      "99000\n",
      "100000\n",
      "101000\n",
      "102000\n",
      "103000\n",
      "104000\n",
      "105000\n",
      "106000\n",
      "107000\n",
      "108000\n",
      "109000\n",
      "110000\n",
      "111000\n",
      "112000\n",
      "113000\n",
      "114000\n",
      "115000\n",
      "116000\n",
      "117000\n",
      "118000\n",
      "119000\n",
      "120000\n",
      "121000\n",
      "122000\n",
      "123000\n",
      "124000\n",
      "125000\n",
      "126000\n",
      "127000\n",
      "128000\n",
      "129000\n",
      "130000\n",
      "131000\n",
      "132000\n",
      "133000\n",
      "134000\n",
      "135000\n",
      "136000\n",
      "137000\n",
      "138000\n",
      "139000\n",
      "140000\n",
      "141000\n",
      "142000\n",
      "143000\n",
      "144000\n",
      "145000\n",
      "146000\n",
      "147000\n",
      "148000\n",
      "149000\n",
      "150000\n",
      "151000\n",
      "152000\n",
      "153000\n",
      "154000\n",
      "155000\n",
      "156000\n",
      "157000\n",
      "158000\n",
      "159000\n",
      "160000\n",
      "161000\n",
      "162000\n",
      "163000\n",
      "164000\n",
      "165000\n",
      "166000\n",
      "167000\n",
      "168000\n",
      "169000\n",
      "170000\n",
      "171000\n",
      "172000\n",
      "173000\n",
      "174000\n",
      "175000\n",
      "176000\n",
      "177000\n",
      "178000\n",
      "179000\n",
      "180000\n",
      "181000\n",
      "182000\n",
      "183000\n",
      "184000\n",
      "185000\n",
      "186000\n",
      "187000\n",
      "188000\n",
      "189000\n",
      "190000\n",
      "191000\n",
      "192000\n",
      "193000\n",
      "194000\n",
      "195000\n",
      "196000\n",
      "197000\n",
      "198000\n",
      "199000\n",
      "200000\n",
      "201000\n",
      "202000\n",
      "203000\n",
      "204000\n",
      "205000\n",
      "206000\n",
      "207000\n",
      "208000\n",
      "209000\n",
      "210000\n",
      "211000\n",
      "212000\n",
      "213000\n",
      "214000\n",
      "215000\n",
      "216000\n",
      "217000\n",
      "218000\n",
      "219000\n",
      "220000\n",
      "221000\n",
      "222000\n",
      "223000\n",
      "224000\n",
      "225000\n",
      "226000\n",
      "227000\n",
      "228000\n",
      "229000\n",
      "230000\n",
      "231000\n",
      "232000\n",
      "233000\n",
      "234000\n",
      "235000\n",
      "236000\n",
      "237000\n",
      "238000\n",
      "239000\n",
      "240000\n",
      "241000\n",
      "242000\n",
      "243000\n",
      "244000\n",
      "245000\n",
      "246000\n",
      "247000\n",
      "248000\n",
      "249000\n",
      "250000\n",
      "251000\n",
      "252000\n",
      "253000\n",
      "254000\n",
      "255000\n",
      "256000\n",
      "257000\n",
      "258000\n",
      "259000\n",
      "260000\n",
      "261000\n",
      "262000\n",
      "263000\n",
      "264000\n",
      "265000\n",
      "266000\n",
      "267000\n",
      "268000\n",
      "269000\n",
      "270000\n",
      "271000\n",
      "272000\n",
      "273000\n",
      "274000\n"
     ]
    }
   ],
   "source": [
    "#ASSIGNMENT OF GENDER BASED ON USERNAME\n",
    "\n",
    "# Create a column with username converted to lowercase\n",
    "tweets[\"user_lower\"] = tweets[\"user\"].str.lower()\n",
    "\n",
    "# Create 2 columns for the algorithm to make its gender prediction in\n",
    "tweets[\"gender_username_algo\"] = \"\"\n",
    "tweets[\"namematched_algo\"] = \"\"\n",
    "\n",
    "# Run the name matching algorithm\n",
    "for i in range(len(tweets)):\n",
    "    for j in range (len(namesfinal)):\n",
    "        if tweets.at[i, \"gender_username_algo\"]==\"\":\n",
    "            if (namesfinal.at[j, \"name_lower\"] in tweets.at[i, \"user_lower\"]):\n",
    "                tweets.at[i, \"gender_username_algo\"] = namesfinal.at[j, \"gender\"]\n",
    "                tweets.at[i, \"namematched_algo\"] = namesfinal.at[j, \"name\"]\n",
    "            else:\n",
    "                continue\n",
    "        else:\n",
    "            continue\n",
    "    # Just to get a sense of progress while processing\n",
    "    if i%1000==0:\n",
    "        (print(i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#ASSIGNMENT OF GENDER BASED ON DESCRIPTION\n",
    "\n",
    "descr_keywords = pd.read_excel(\"descr_keywords.xlsx\")\n",
    "\n",
    "# Add column for new gender prediction based on description\n",
    "# Create a column with username converted to lowercase, on both datasets\n",
    "tweets[\"gender_description_algo\"] = \"\"\n",
    "\n",
    "# Make description field lowercase\n",
    "tweets[\"description\"] = tweets[\"description\"].str.lower()\n",
    "\n",
    "# Tokenize the description field using natural language toolkit\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "\n",
    "tweets[\"description_tokenized\"] = \"\"\n",
    "tokenizer = RegexpTokenizer(r'\\w+')\n",
    "\n",
    "# Run the tokenizing algorithm\n",
    "for i in range(len(tweets)):\n",
    "    if pd.notnull(tweets.at[i, 'description'])==False :\n",
    "        (tweets.at[i, 'description_tokenized'])=\"\"\n",
    "    else:\n",
    "        (tweets.at[i, 'description_tokenized'])=tokenizer.tokenize((tweets.at[i, 'description']))\n",
    "\n",
    "descr_keywords.columns = ['word', 'gender']\n",
    "\n",
    "# Run the gender assigning algorithm\n",
    "for i in range(len(tweets)):\n",
    "    for j in range(len(descr_keywords)):\n",
    "        if (descr_keywords.at[j, 'word'] in tweets.at[i, 'description_tokenized']) & (tweets.at[i, 'gender_description_algo']==\"\"):\n",
    "            tweets.at[i, 'gender_description_algo'] = descr_keywords.at[j, 'gender']\n",
    "        else:\n",
    "            continue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ASSIGNMENT OF FINAL GENDER VALUE\n",
    "\n",
    "tweets[\"gender_final\"] = \"\"\n",
    "\n",
    "for i in range(len(tweets)):\n",
    "    tweets.at[i, \"gender_final\"] = tweets.at[i, \"gender_description_algo\"] \n",
    "    if tweets.at[i, \"gender_final\"]==\"\":\n",
    "        tweets.at[i, \"gender_final\"]=tweets.at[i, \"gender_username_algo\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Export the tweet subset to excel for checking\n",
    "writer = pd.ExcelWriter('04sotu_with_gender.xlsx')\n",
    "tweets.to_excel(writer,'Sheet1')\n",
    "writer.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "138936\n"
     ]
    }
   ],
   "source": [
    "# Check how many genders have been assigned\n",
    "gendercount=0\n",
    "for i in range(len(tweets)):\n",
    "        if tweets.at[i, 'gender_final'] !=\"\":\n",
    "            gendercount+=1\n",
    "print(gendercount)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(274520, 17)\n"
     ]
    }
   ],
   "source": [
    "print(tweets.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5061052018067901"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "138936/274520"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The gender algorithm assigned 138,936 rows with a gender, out of a total 274,520, giving 51% of our dataset a gender classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluation of accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Export a random sample of 1800 rows to be checked for accuracy of classifications\n",
    "tweets_sample=tweets.sample(1800)\n",
    "\n",
    "writer = pd.ExcelWriter('04sotu_with_gender_sample.xlsx')\n",
    "tweets_sample.to_excel(writer,'Sheet1')\n",
    "writer.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "# Reimport the checked sample\n",
    "tweets_sample1000_withgender_checked = pd.read_excel('04sotu_with_gender_sample.xlsx')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              count\n",
      "gender_final       \n",
      "F               389\n",
      "M               536\n",
      "U               875\n",
      "All            1800 \n",
      "\n",
      "            count\n",
      "humancheck       \n",
      "OK            801\n",
      "notsure        80\n",
      "wrong          44\n",
      "All           925 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Pivots to view aggregated results\n",
    "pivot0=pd.pivot_table(tweets_sample1000_withgender_checked, index=[\"gender_final\"],values=[\"user\"], aggfunc=[len], margins=True)\n",
    "pivot0.columns=['count']\n",
    "print(pivot0, '\\n')\n",
    "\n",
    "pivot1=pd.pivot_table(tweets_sample1000_withgender_checked, index=[\"humancheck\"],values=[\"user\"], aggfunc=[len], margins=True)\n",
    "pivot1.columns=['count']\n",
    "print(pivot1, '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first pivot shows us that the algorithm assigned gender to 925/1800 rows (of which, incidentally: 389 F, 536 M)\n",
    "\n",
    "The second pivot shows us that we have judged 801 (86.6%) of these assignments as 'OK' (the human agrees with the gender assignment), 80 (8.6%) as 'notsure' (the human finds the user's gender ambiguous) and 44 (4.8%) as wrong (the human is sure the gender is in fact the opposite of that assigned by the algorithm)\n",
    "\n",
    "This gives an accuracy rate of 86.6 - 95.2 %"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final step was to copy the column of gender values to the version of the dataset where preprocessing had been carried out. As the two datasets are identical in terms of row number and order of tweets ( as checked below) this was achieved via a single loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(274520, 17)\n",
      "(274520, 19)\n"
     ]
    }
   ],
   "source": [
    "print(tweets.shape)\n",
    "print(tweets_final.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<bound method NDFrame.head of                                   date  \\\n",
      "0       Wed Jan 31 03:30:46 +0000 2018   \n",
      "1       Wed Jan 31 03:30:46 +0000 2018   \n",
      "2       Wed Jan 31 03:30:46 +0000 2018   \n",
      "3       Wed Jan 31 03:30:46 +0000 2018   \n",
      "4       Wed Jan 31 03:30:46 +0000 2018   \n",
      "5       Wed Jan 31 03:30:46 +0000 2018   \n",
      "6       Wed Jan 31 03:30:46 +0000 2018   \n",
      "7       Wed Jan 31 03:30:46 +0000 2018   \n",
      "8       Wed Jan 31 03:30:46 +0000 2018   \n",
      "9       Wed Jan 31 03:30:46 +0000 2018   \n",
      "10      Wed Jan 31 03:30:46 +0000 2018   \n",
      "11      Wed Jan 31 03:30:46 +0000 2018   \n",
      "12      Wed Jan 31 03:30:46 +0000 2018   \n",
      "13      Wed Jan 31 03:30:46 +0000 2018   \n",
      "14      Wed Jan 31 03:30:46 +0000 2018   \n",
      "15      Wed Jan 31 03:30:46 +0000 2018   \n",
      "16      Wed Jan 31 03:30:46 +0000 2018   \n",
      "17      Wed Jan 31 03:30:46 +0000 2018   \n",
      "18      Wed Jan 31 03:30:46 +0000 2018   \n",
      "19      Wed Jan 31 03:30:46 +0000 2018   \n",
      "20      Wed Jan 31 03:30:46 +0000 2018   \n",
      "21      Wed Jan 31 03:30:46 +0000 2018   \n",
      "22      Wed Jan 31 03:30:46 +0000 2018   \n",
      "23      Wed Jan 31 03:30:46 +0000 2018   \n",
      "24      Wed Jan 31 03:30:46 +0000 2018   \n",
      "25      Wed Jan 31 03:30:46 +0000 2018   \n",
      "26      Wed Jan 31 03:30:46 +0000 2018   \n",
      "27      Wed Jan 31 03:30:46 +0000 2018   \n",
      "28      Wed Jan 31 03:30:46 +0000 2018   \n",
      "29      Wed Jan 31 03:30:46 +0000 2018   \n",
      "...                                ...   \n",
      "274490  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274491  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274492  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274493  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274494  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274495  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274496  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274497  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274498  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274499  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274500  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274501  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274502  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274503  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274504  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274505  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274506  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274507  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274508  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274509  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274510  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274511  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274512  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274513  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274514  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274515  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274516  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274517  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274518  Wed Jan 31 02:10:14 +0000 2018   \n",
      "274519  Wed Jan 31 02:10:14 +0000 2018   \n",
      "\n",
      "                                              description  favorite_count  \\\n",
      "0                                                     NaN               0   \n",
      "1                                         WAITING FOR FDR               1   \n",
      "2                                  ENFP TWEETS ARE MY OWN               1   \n",
      "3       IT TAKES MORE THAN EMBARRASSMENT TO SHAME ME T...               2   \n",
      "4       4 1 2 BILLION YEARS OF EVOLUTION PRODUCED OUR ...               0   \n",
      "5                        JEREMIAH 18 1 6 REVELATION 12 11               3   \n",
      "6       WIFE TO JAY MOMMY TO ZACHARY LOUIS KITTY MAMA ...               1   \n",
      "7       EDUCATION IS THE MOST IMPORTANT THING IN STOPP...               0   \n",
      "8       BALTIMORE NATIVE NERD FOR POP POLITICS SNARK H...               0   \n",
      "9                        SENIOR COLUMNIST HUSBAND AND DAD               1   \n",
      "10      YOUR SOURCE FOR ORIGINAL REPORTING AND TRUSTED...             101   \n",
      "11                                                    NaN               1   \n",
      "12                                                    NaN               0   \n",
      "13      LOVE SPORTS DOGS MUSIC AND CANNABIS HATE RACIS...               1   \n",
      "14                                                    NaN               0   \n",
      "15      TWITTER S BEST KEPT SECRET FORMER SIGINT ANALY...              33   \n",
      "16      💡DESIGN GRAD STUDENT 〰️ ELECTRIC BASSOONIST 〰️...               3   \n",
      "17      PROUD FATHER LUCKY HUSBAND CHEERING ON MY SF G...               0   \n",
      "18      FOUNDER OF HOLY KOOLAID LLC DON T DRINK THE KO...               0   \n",
      "19      UTAH S RADIO STATION TRAFFIC AND WEATHER EVERY...               2   \n",
      "20      HOME OF KSL 5 NEWS AND STUDIO 5 AND THE LOCAL ...               9   \n",
      "21      USED TO BE ATHLETIC SMART AND AMBITIOUS NOW I ...               1   \n",
      "22      COMPUTER SCIENTIST PROGRESSIVE CHRISTIAN AND A...               1   \n",
      "23      A PATRIOT WHO IS TIRED OF THE CORRUPTION IN OU...               1   \n",
      "24      🎙HOST JASON RANTZ SHOW ON KTTH 770AM IN SEATTL...               0   \n",
      "25      I M SPECIAL BECAUSE I M ME PROGRESSIVE DEM AME...               3   \n",
      "26      EDITOR FOR THE STAFF WRITER FOR THE AND SOCIAL...               0   \n",
      "27      POLITICAL INVESTIGATIVE REPORTER FOR FICTION A...               4   \n",
      "28      ENJOYS POLITICS PODCASTS POST IMPRESSIONISTS P...               0   \n",
      "29      PROUD TO SERVE THE FIRST DISTRICT OF WEST VIRG...               5   \n",
      "...                                                   ...             ...   \n",
      "274490  BLOCKED BY GOLDUST IT S ABOUT THE WRESTLING CO...               0   \n",
      "274491  MY WORDS ARE MY OWN EXCEPT WHEN I M QUOTING SO...               0   \n",
      "274492                               MOSTLY SPOTIFY LINKS               2   \n",
      "274493  THE SUFFOLK JOURNAL IS THE AWARD WINNING UNDER...               0   \n",
      "274494  DON T WHINE VOTE AND TELL YOUR FAMILY FRIENDS ...               9   \n",
      "274495  🇱🇷 TRUMPLICAN 🇺🇸MAGA💯 SUPPORTING THE MAGA AGEN...               0   \n",
      "274496                          MARRIED AF I LOVE MY LIFE               0   \n",
      "274497  23 COLLEGE STUDENT I VE BEEN WINNING ALL MY LI...               0   \n",
      "274498  WRITER LAB RAT PURVEYOR OF FINE ENERGY DRINKS ...               1   \n",
      "274499  OFFICIAL ACCOUNT OF THE POLICY TEAM WE WORK TO...               0   \n",
      "274500  MARRIED TO MYRA CHIEF OF STAFF FMR RADIO TALK ...               0   \n",
      "274501  IF YOU KNOW ME YOU DON T NEED THE USELESS INFO...               0   \n",
      "274502  ECONOMIC POLICY REPORTER AT THE NEW YORK TIMES...               4   \n",
      "274503  AWARD WINNING REPORTER W THE NEW YORK DAILY NE...               0   \n",
      "274504  ARTIST SELF HELP WRITER NEWS CHRISTIAN BELIEVE...               0   \n",
      "274505  EGALITARIAN SOME TWEETS ARE IRONIC SOME AS SER...               1   \n",
      "274506  TERRY KINDER GOD KNOWS WHO I AM TWEETS ARE NOT...               0   \n",
      "274507  THE GREAT MATTERS ARE OFTEN AS WORTHLESS AS AS...               0   \n",
      "274508  LOVER OF ALL THINGS BEAUTY POLITICS WIFE EX DE...               5   \n",
      "274509  CONSTITUTIONAL FISCAL CONSERVATIVE PROMILITARY...               1   \n",
      "274510  HAITIAN AIR FORCE F 52 PILOT LES TROUS DE MERD...               0   \n",
      "274511  WINNER OF SINGING COMPETITION TRANSGENDER SING...              37   \n",
      "274512                                                NaN               0   \n",
      "274513  HUMOR HIKING PHOTOGRAPHY SARCASTIC CONSERVATIV...               0   \n",
      "274514  TWITTER ANY IDIOT ELECTED TO THE HIGHEST OFFIC...               0   \n",
      "274515  YOU CAN MAKE A LOT OF SACRIFICES AND KEEP EVER...               0   \n",
      "274516  BRED EDUCATED EMPLOYED BE EXTRA OR BE NOTHING ...               0   \n",
      "274517  R I P NAYNA MASI 12 9 1969 TO 6 12 2015 LILABA...               1   \n",
      "274518                            ENOUGH ABOUT ME ALREADY               0   \n",
      "274519  ACTUALLY THROUGHOUT MY LIFE MY TWO GREATEST AS...               0   \n",
      "\n",
      "        followers_count              id_str                        location  \\\n",
      "0                   163  958543051391229952                FREAKIN  FLORIDA   \n",
      "1                  1419  958543051248603136                        BROOKLYN   \n",
      "2                    65  958543051080781824                          BOSTON   \n",
      "3                  1370  958543050967568384                             NaN   \n",
      "4                   346  958543050510405632                           TEXAS   \n",
      "5                   231  958543050472640512                       FARGO  ND   \n",
      "6                  1814  958543050468483072  RALEIGH NC BUT A NYER AT HEART   \n",
      "7                    57  958543050468360192                 KNIGHTSTOWN  IN   \n",
      "8                  1318  958543050338439168                              DC   \n",
      "9                  9396  958543050300710912                     MERRICK  NY   \n",
      "10              6386672  958543050199961600                    NEW YORK  NY   \n",
      "11                  411  958543050158026752            CAROLINABLUE2DAHEART   \n",
      "12                   10  958543050090909696                             NaN   \n",
      "13                 2667  958543049977548800                          NORCAL   \n",
      "14                   29  958543049885454336                             NaN   \n",
      "15                 3791  958543049860071424                             NaN   \n",
      "16                  264  958543049826562048                 CALIFORNIA  USA   \n",
      "17                   42  958543049629618176                MULLICA HILL  NJ   \n",
      "18                 1815  958543049457590272                             NaN   \n",
      "19                19499  958543049306460160              SALT LAKE CITY  UT   \n",
      "20                63800  958543049285517312            SALT LAKE CITY  UTAH   \n",
      "21                  175  958543049277263872                     CHICAGO  IL   \n",
      "22                 1004  958543049155543040                             NaN   \n",
      "23                 6744  958543049025466368                             NaN   \n",
      "24                 5562  958543048937324544                      SEATTLE WA   \n",
      "25                   70  958543048547491840                 SAN ANTONIO  TX   \n",
      "26                  526  958543048539074560                 NEW JERSEY  USA   \n",
      "27                 7482  958543048509571072       SAN FRANCISCO  CALIFORNIA   \n",
      "28                  920  958543048408879104                 LOS ANGELES  CA   \n",
      "29                27025  958543048346128384                             NaN   \n",
      "...                 ...                 ...                             ...   \n",
      "274490              283  958522782962286592                         THE 716   \n",
      "274491               82  958522782924419072                     LOS ANGELES   \n",
      "274492               35  958522782903549952                          BROPHY   \n",
      "274493             2738  958522782828105728                      BOSTON  MA   \n",
      "274494            13973  958522782811336704                   UNITED STATES   \n",
      "274495               46  958522782798802944                SOMEWHERE U S A    \n",
      "274496              153  958522782782025728                     ORLEANS  MA   \n",
      "274497             9077  958522782739963904               LAS VEGAS  NEVADA   \n",
      "274498             6622  958522782660288512                         SEATTLE   \n",
      "274499               32  958522782480044032                             NaN   \n",
      "274500             6560  958522782362546176            BLOOMINGTON  IN  USA   \n",
      "274501              315  958522782215700480              WEST LAFAYETTE  IN   \n",
      "274502            12834  958522781909553152                   WASHINGTON DC   \n",
      "274503             1442  958522781871812608                    BROOKLYN  NY   \n",
      "274504            80153  958522781817122816                      CALIFORNIA   \n",
      "274505             2417  958522781687312384                  BESIDE THE SEA   \n",
      "274506            58752  958522781653786624                     JACKSON  MO   \n",
      "274507              704  958522781557170176       EDMONTON  ALBERTA  CANADA   \n",
      "274508             8213  958522781460701184                 CALIFORNIA  USA   \n",
      "274509             4652  958522781280489472                  TENNESSEE  USA   \n",
      "274510                0  958522781246816256                           HAITI   \n",
      "274511             4102  958522781162975232                         CHICAGO   \n",
      "274512               70  958522781158772736                  NORTH CAROLINA   \n",
      "274513              155  958522780940623872                     PHOENIX  AZ   \n",
      "274514               94  958522780789694464              POINT PLEASANT  NJ   \n",
      "274515              650  958522780764557312        WORCESTER  MASSACHUSETTS   \n",
      "274516              566  958522780747751424                             DMV   \n",
      "274517              190  958522780705787904                PHILADELPHIA  PA   \n",
      "274518              221  958522780571418624                 LOS ANGELES  CA   \n",
      "274519               73  958522780466622464            BRISBANE  QUEENSLAND   \n",
      "\n",
      "        retweet_count  statuses_count  \\\n",
      "0                   0            8723   \n",
      "1                   1           20287   \n",
      "2                   0             373   \n",
      "3                   0           35380   \n",
      "4                   0           10621   \n",
      "5                   0            2210   \n",
      "6                   0           26747   \n",
      "7                   0            2990   \n",
      "8                   0           10412   \n",
      "9                   0           17983   \n",
      "10                 60          161400   \n",
      "11                  0           19042   \n",
      "12                  0               2   \n",
      "13                  0           36898   \n",
      "14                  0             190   \n",
      "15                  5            3943   \n",
      "16                  0            1904   \n",
      "17                  0            7861   \n",
      "18                  0            4424   \n",
      "19                  0           34700   \n",
      "20                  0           66383   \n",
      "21                  0           35013   \n",
      "22                  0           84832   \n",
      "23                  1           35763   \n",
      "24                  0           35332   \n",
      "25                  1            1542   \n",
      "26                  0            5253   \n",
      "27                  1           29816   \n",
      "28                  0            4045   \n",
      "29                  0            4741   \n",
      "...               ...             ...   \n",
      "274490              0            7667   \n",
      "274491              0             858   \n",
      "274492              0             780   \n",
      "274493              0            6704   \n",
      "274494              0           58619   \n",
      "274495              1             726   \n",
      "274496              0            6417   \n",
      "274497              2          194629   \n",
      "274498              0           46820   \n",
      "274499              0              72   \n",
      "274500              0           19160   \n",
      "274501              0            3027   \n",
      "274502              5           10473   \n",
      "274503              0            4875   \n",
      "274504              0           82523   \n",
      "274505              0           38990   \n",
      "274506              0          123387   \n",
      "274507              0            8793   \n",
      "274508              0           73992   \n",
      "274509              0           13365   \n",
      "274510              0              13   \n",
      "274511             14            2573   \n",
      "274512              0            2116   \n",
      "274513              0            1201   \n",
      "274514              0            6827   \n",
      "274515              0            7536   \n",
      "274516              0           18287   \n",
      "274517              0           22544   \n",
      "274518              0             844   \n",
      "274519              0            2608   \n",
      "\n",
      "                                                     text             user  \\\n",
      "0       RETWEETED ♻️ CHRISTOPHER ZULLO WHAT A SUPERFLU...        LIKIPEDIA   \n",
      "1       I WISH SOMEONE WOULD HAVE TOLD ME THAT I COULD...          COREYCW   \n",
      "2       SURPRISE CAMEO SCIENCE FOUND ITS WAY INTO THE ...       CRAIGREGIS   \n",
      "3       THAT S ODD HE SAID HE LIKES MUSIC BUT I DISTIN...        KIMWITHAK   \n",
      "4       HE DOESN T HAVE ANY STANDING TO COMMENT ON FAM...     CRANESTATION   \n",
      "5       IF YOU DIDN T WATCH THE STATE OF THE UNION YOU...       JEREMYKOPP   \n",
      "6       REMEMBER THAT TIME THAT STEPHEN MILLER SAID TH...        COLLEEN84   \n",
      "7           OK TIME TO PULL THE PLUG ABOUT 30 MINUT S AGO         WABASHCC   \n",
      "8       HAS DONE FAR BETTER THAN DOMESTIC PROGRAMS UND...      SARALDUBOIS   \n",
      "9                                   HERE S MY TAKE ON THE       JAKEJAKENY   \n",
      "10      PRES TRUMP CALLS FOR THE END OF CHAIN MIGRATIO...          CBSNEWS   \n",
      "11      THE LOOK YOU GIVE WHEN A KNOW IT ALL CAN T REA...      JAZZYONEMEL   \n",
      "12      THANK YOU PRESIDENT TRUMP YOU ARE GREATLY APPR...     JEFF56566268   \n",
      "13      I JUST WISH HE D STFU GO HOME TO HIS BEDTIME C...     IMTHE420NERD   \n",
      "14      ABOVE ALL ELSE THEY ARE AMERICANS AND THIS CIT...         SCCMIGOP   \n",
      "15      SINCE I M NOT WATCHING THE I LL GUESS TRUMPS T...   DEEPSTATEROGUE   \n",
      "16                                  WHAT AN EMBARRASSMENT       ADAMMUSING   \n",
      "17      TRUMP MUST WANT TO KEEP TALKING BECAUSE HE DOE...        JERKAT429   \n",
      "18      I DON T THINK I VE EVER SEEN A PRESIDENT SMIRK...      HOLYKOOLAID   \n",
      "19      THE PEOPLE DREAMED THIS COUNTRY THE PEOPLE BUI...     KSLNEWSRADIO   \n",
      "20      THE PEOPLE DREAMED THIS COUNTRY THE PEOPLE BUI...           KSL5TV   \n",
      "21      IT D BE A REAL SHAME IF THE PLOT OF DESIGNATED...      NOTHOTTAKES   \n",
      "22                                  WHEN SAYS WE LOVE YOU      SEPHIUS1999   \n",
      "23          THE PEOPLE WHO ARE MAKING AMERICA GREAT AGAIN      DACOLDRIVER   \n",
      "24      OKAY THIS SPEECH WAS OVER 4 MINUTES AGO MINUS ...       JASONRANTZ   \n",
      "25        IMMIGRANTS BUILT THIS COUNTRY YOU FUCKING IDIOT      ANNROLLINS0   \n",
      "26      IT S THE PEOPLE THAT ARE MAKING AMERICA GREAT ...         CLEESPOT   \n",
      "27      AND SCENE THAT WAS THE 3RD LONGEST IN HISTORY ...     SOVERNNATION   \n",
      "28       TRUMP WRAPPING MOSTLY AND SURPRISINGLY ON SCRIPT       KATIELEWDC   \n",
      "29      IT IS THE PEOPLE WHO ARE MAKING AMERICA GREAT ...      REPMCKINLEY   \n",
      "...                                                   ...              ...   \n",
      "274490                      OH MAN HIS EGO IS LOVING THIS          LVKETCB   \n",
      "274491  LET S BE HONEST THIS REPUBLICAN DRIVEN APPLAUS...        CANDIDLYB   \n",
      "274492                     WATCHING BECAUSE I HATE MYSELF     LANCE KEEBLE   \n",
      "274493  MANY DEMOCRATIC CONGRESSWOMEN ARE BOYCOTTING T...   SUFFOLKJOURNAL   \n",
      "274494  THE PUNDITS SAY TRUMP HAS BEEN WORKING HARD ON...      PITTGRIFFIN   \n",
      "274495                                           MY POTUS     ORIGINAL ME1   \n",
      "274496  CURRENTLY WATCHING A GROUP OF DEMONS GIVE SATA...       BREMTWEETS   \n",
      "274497         WE REALLY HAVE A REALITY TV SHOW PRESIDENT       J0RDAN0RE0   \n",
      "274498  THERE HAS TO BE A HUEY LEWIS AMP THE NEWS COVE...        RDS3ATTLE   \n",
      "274499  NALCAB APPLAUDS THOSE MEMBERS OF CONGRESS WHO ...     NALCABPOLICY   \n",
      "274500                                     LIVE BROADCAST         JIMPFAFF   \n",
      "274501                                   FINALLY STARTING     NIELSEN KARL   \n",
      "274502                           THE LIVE CHAT IS HERE GT       ARAPPEPORT   \n",
      "274503                                         HERE WE GO     CHRISCARREGA   \n",
      "274504                              GOD BLESS AND PROTECT           HPSELF   \n",
      "274505  THE CLAPPING WAS GOOD UNFORTUNATELY IT S ALL D...       DAMONMERCY   \n",
      "274506                                        DEMS BEHAVE          TKINDER   \n",
      "274507                                     THIS ONE S FOR        PABINKLEY   \n",
      "274508  YOU CAN SEE NANCY PELOSI TRYING TO CALM HER PE...     MRSMMTBEAUTY   \n",
      "274509  ISN T THE RADICAL LEFT TO BLAME FOR SERIOUSLY ...        ISAIDKNOW   \n",
      "274510      BETSY DEVOS OUTFIT COST MORE DEN ME HOUSE OUI    COVFEFELEBRUN   \n",
      "274511           THEY RE NOT APPLAUDING THEY RE A KISSING    ISABELLACIGNO   \n",
      "274512                           SO NICE TO WATCH A AGAIN          NAINE11   \n",
      "274513  FIRST IMPRESSION IS OF I CAN T IMAGINE MY WIFE...   SARCASTICRUSTY   \n",
      "274514                   THIS IS ALREADY AN EMBARRASSMENT  MATTHEWJDONOVAN   \n",
      "274515  HOW CAN THOSE SENATORS AND CONGRESSMEN STAND T...           JAFLO1   \n",
      "274516  POTUS CLAPPING FOR HIMSELF AT THE PODIUM IS AL...     CHARLA4MAYOR   \n",
      "274517  IT S SO ANNOYING WHEN THEY GET UP AFTER EVERY ...        JATINB811   \n",
      "274518  WILL SOMEBODY WAKE ME WHEN HE GETS TO THE STOR...     PACOMCCAULEY   \n",
      "274519  I M SURE THIS LEVEL OF APPLAUSE HAPPENED UNDER...       KELSITONIN   \n",
      "\n",
      "        verified                               description_hashtags  \\\n",
      "0          False                                                NaN   \n",
      "1          False                                                NaN   \n",
      "2          False                                                NaN   \n",
      "3          False                                                NaN   \n",
      "4          False                                                NaN   \n",
      "5          False                                                NaN   \n",
      "6          False                                         IMWITHHERS   \n",
      "7          False                                                NaN   \n",
      "8          False                                                NaN   \n",
      "9          False                                                NaN   \n",
      "10          True                                                NaN   \n",
      "11         False  MILKYWAYMOOD LOVEFULLTILT THESOULTHEESSENCEOFB...   \n",
      "12         False                                                NaN   \n",
      "13         False                                    THERESISTANCE Ω   \n",
      "14         False                                                NaN   \n",
      "15         False                                   ALTGOV DEEPSTATE   \n",
      "16         False                                                NaN   \n",
      "17         False                                                NaN   \n",
      "18         False                        HOLYKOOLAID SCIENCE SKEPTIC   \n",
      "19          True                                                  1   \n",
      "20          True                                                NaN   \n",
      "21         False                                                NaN   \n",
      "22         False                                                NaN   \n",
      "23         False                                                NaN   \n",
      "24          True                                                NaN   \n",
      "25         False                                                NaN   \n",
      "26         False                                                NaN   \n",
      "27          True                                                NaN   \n",
      "28         False                                       BURSONPERSON   \n",
      "29          True                                                NaN   \n",
      "...          ...                                                ...   \n",
      "274490     False                                                NaN   \n",
      "274491     False                                                NaN   \n",
      "274492     False                                                NaN   \n",
      "274493     False                                                NaN   \n",
      "274494     False                      VOTE2018 RESIST THERESISTANCE   \n",
      "274495     False                                                NaN   \n",
      "274496     False                                                NaN   \n",
      "274497     False                                                NaN   \n",
      "274498     False                                                NaN   \n",
      "274499     False                                                NaN   \n",
      "274500     False                PROLIFE CONSTITUTION CO2 GOLDENRICE   \n",
      "274501     False                                                NaN   \n",
      "274502      True                                                NaN   \n",
      "274503      True                                       STJOHNS NABJ   \n",
      "274504     False                                                NaN   \n",
      "274505     False                                                NaN   \n",
      "274506     False                                                NaN   \n",
      "274507     False                                                NaN   \n",
      "274508     False                                               MAGA   \n",
      "274509     False                                        2A TCOT RNR   \n",
      "274510     False                                                NaN   \n",
      "274511     False                                                NaN   \n",
      "274512     False                                                NaN   \n",
      "274513     False                                                NaN   \n",
      "274514     False                                                NaN   \n",
      "274515     False                                                NaN   \n",
      "274516     False                                        NOLA WHODAT   \n",
      "274517     False                                                NaN   \n",
      "274518     False                                                NaN   \n",
      "274519     False                                                NaN   \n",
      "\n",
      "                                description_mentions text_emojis  \\\n",
      "0                                                NaN         NaN   \n",
      "1                                                NaN         NaN   \n",
      "2                                                NaN         NaN   \n",
      "3                                                NaN         NaN   \n",
      "4                                                NaN         NaN   \n",
      "5                                                NaN         NaN   \n",
      "6                                                NaN         NaN   \n",
      "7                                                NaN           🚽   \n",
      "8       PENTAGONCHOICES AMNESTYUSA NATSECNET MESSAGE         NaN   \n",
      "9                                                NaN         NaN   \n",
      "10                                               NaN         NaN   \n",
      "11                                               NaN         NaN   \n",
      "12                                               NaN         NaN   \n",
      "13                                               NaN         NaN   \n",
      "14                                               NaN         NaN   \n",
      "15                                               NaN         NaN   \n",
      "16                                           UCDAVIS         NaN   \n",
      "17                                               NaN         NaN   \n",
      "18                                               NaN         NaN   \n",
      "19                                               NaN         NaN   \n",
      "20                                               NaN         NaN   \n",
      "21                                               NaN         NaN   \n",
      "22                                               NaN         NaN   \n",
      "23                                               NaN         NaN   \n",
      "24                                               NaN         NaN   \n",
      "25                                               NaN         NaN   \n",
      "26        FLORHAMPEAGLE MADISONEAGLE NJHILLS WILKESU         NaN   \n",
      "27                                          KCBSNEWS         NaN   \n",
      "28                                               NaN         NaN   \n",
      "29                                               NaN         NaN   \n",
      "...                                              ...         ...   \n",
      "274490                                THECHARGINGBUF         NaN   \n",
      "274491                                           NaN         NaN   \n",
      "274492                                           NaN         NaN   \n",
      "274493                                           NaN         NaN   \n",
      "274494                                           NaN         NaN   \n",
      "274495                                           NaN         NaN   \n",
      "274496                                           NaN         NaN   \n",
      "274497                                           NaN         NaN   \n",
      "274498                                           NaN         NaN   \n",
      "274499                                        NALCAB         NaN   \n",
      "274500                REPTHOMASMASSIE ILLY YALIBERTY         NaN   \n",
      "274501                                           NaN         NaN   \n",
      "274502       FT MARJORIEDEANE THEECONOMIST LSEECHIST         NaN   \n",
      "274503                                           NaN         NaN   \n",
      "274504                                           NaN         NaN   \n",
      "274505                                           NaN         NaN   \n",
      "274506                                           NaN         NaN   \n",
      "274507                                           NaN         NaN   \n",
      "274508           HANNITY SEBGORKA CVPAYNE SCARAMUCCI         NaN   \n",
      "274509                                           NaN         NaN   \n",
      "274510                                           NaN         NaN   \n",
      "274511                                   ONEREPUBLIC         NaN   \n",
      "274512                                           NaN         NaN   \n",
      "274513                                           NaN         NaN   \n",
      "274514                                           NaN         NaN   \n",
      "274515                                           NaN         NaN   \n",
      "274516                              HOWARDU SIRIUSXM         NaN   \n",
      "274517                                           NaN         NaN   \n",
      "274518                                           NaN         NaN   \n",
      "274519                                           NaN         NaN   \n",
      "\n",
      "                                            text_hashtags  \\\n",
      "0                                                    SOTU   \n",
      "1                                                     NaN   \n",
      "2                                         STATEOFTHEUNION   \n",
      "3                                                     NaN   \n",
      "4                                                     NaN   \n",
      "5                                                    SOTU   \n",
      "6                                                     NaN   \n",
      "7                                                    SOTU   \n",
      "8                                                     DOD   \n",
      "9                                                    SOTU   \n",
      "10                                                   SOTU   \n",
      "11                                                   SOTU   \n",
      "12                                                   SOTU   \n",
      "13                                                   SOTU   \n",
      "14                                                   SOTU   \n",
      "15                                                   SOTU   \n",
      "16                                                   SOTU   \n",
      "17                                     SOTU STORMYWEATHER   \n",
      "18                                                   SOTU   \n",
      "19                                                    NaN   \n",
      "20                                                    NaN   \n",
      "21                                                   SOTU   \n",
      "22                                        PUERTORICO SOTU   \n",
      "23                                                   SOTU   \n",
      "24                                                   SOTU   \n",
      "25                                                   SOTU   \n",
      "26                                                   SOTU   \n",
      "27                                                   SOTU   \n",
      "28                                                   SOTU   \n",
      "29                                              SOTU MAGA   \n",
      "...                                                   ...   \n",
      "274490                                               SOTU   \n",
      "274491                                               SOTU   \n",
      "274492                                               SOTU   \n",
      "274493                                         METOO SOTU   \n",
      "274494                             SOTU SOTUNIOM SOTU2018   \n",
      "274495  TUESDAYTHOUGHTS MAGA CLOSERNATION RELEASETHEME...   \n",
      "274496                                               SOTU   \n",
      "274497                                               SOTU   \n",
      "274498                                               SOTU   \n",
      "274499                                                NaN   \n",
      "274500                                               SOTU   \n",
      "274501                                               SOTU   \n",
      "274502                                               SOTU   \n",
      "274503                                               SOTU   \n",
      "274504  SOTU POTUS USA RELEASETHEMEMO FBI RUSSIA TRUMP...   \n",
      "274505                                               SOTU   \n",
      "274506                                               SOTU   \n",
      "274507                                               SOTU   \n",
      "274508                                                NaN   \n",
      "274509                                EXTREMEPOLARIZATION   \n",
      "274510                                               SOTU   \n",
      "274511                                               SOTU   \n",
      "274512  SOTU MAGA SOTU2018 STATEOFTHEUNIONADDRESS AMER...   \n",
      "274513                                  SOTU MELANIATRUMP   \n",
      "274514                                               SOTU   \n",
      "274515                                           FOH SOTU   \n",
      "274516                                               SOTU   \n",
      "274517                                               SOTU   \n",
      "274518                                               SOTU   \n",
      "274519                                               SOTU   \n",
      "\n",
      "                                      text_hashtags_split  \\\n",
      "0                                                    SOTU   \n",
      "1                                                     NaN   \n",
      "2                                      STATE OF THE UNION   \n",
      "3                                                     NaN   \n",
      "4                                                     NaN   \n",
      "5                                                    SOTU   \n",
      "6                                                     NaN   \n",
      "7                                                    SOTU   \n",
      "8                                                    DO D   \n",
      "9                                                    SOTU   \n",
      "10                                                   SOTU   \n",
      "11                                                   SOTU   \n",
      "12                                                   SOTU   \n",
      "13                                                   SOTU   \n",
      "14                                                   SOTU   \n",
      "15                                                   SOTU   \n",
      "16                                                   SOTU   \n",
      "17                                   SOTU  STORMY WEATHER   \n",
      "18                                                   SOTU   \n",
      "19                                                    NaN   \n",
      "20                                                    NaN   \n",
      "21                                                   SOTU   \n",
      "22                                       PUERTO RICO SOTU   \n",
      "23                                                   SOTU   \n",
      "24                                                   SOTU   \n",
      "25                                                   SOTU   \n",
      "26                                                   SOTU   \n",
      "27                                                   SOTU   \n",
      "28                                                   SOTU   \n",
      "29                                              SOTU MAGA   \n",
      "...                                                   ...   \n",
      "274490                                               SOTU   \n",
      "274491                                               SOTU   \n",
      "274492                                               SOTU   \n",
      "274493                                        ME TOO SOTU   \n",
      "274494                            SOTU SOT UNIOM SOTU2018   \n",
      "274495  TUESDAY THOUGHTS MAGA  CLOSER NATION  RELEASE ...   \n",
      "274496                                               SOTU   \n",
      "274497                                               SOTU   \n",
      "274498                                               SOTU   \n",
      "274499                                                NaN   \n",
      "274500                                               SOTU   \n",
      "274501                                               SOTU   \n",
      "274502                                               SOTU   \n",
      "274503                                               SOTU   \n",
      "274504  SOTU POTUS USA  RELEASE THE MEMO FBI  RUSSIA  ...   \n",
      "274505                                               SOTU   \n",
      "274506                                               SOTU   \n",
      "274507                                               SOTU   \n",
      "274508                                                NaN   \n",
      "274509                               EXTREME POLARIZATION   \n",
      "274510                                               SOTU   \n",
      "274511                                               SOTU   \n",
      "274512  SOTU MAGA SOTU2018  STATE OF THE UNION ADDRESS...   \n",
      "274513                                SOTU  MELANIA TRUMP   \n",
      "274514                                               SOTU   \n",
      "274515                                           FOH SOTU   \n",
      "274516                                               SOTU   \n",
      "274517                                               SOTU   \n",
      "274518                                               SOTU   \n",
      "274519                                               SOTU   \n",
      "\n",
      "                                 text_mentions state  \n",
      "0                                  CHRISJZULLO    FL  \n",
      "1                                          NaN   NaN  \n",
      "2                                          NaN    MA  \n",
      "3                                          NaN   NaN  \n",
      "4                                          NaN    TX  \n",
      "5                              REALDONALDTRUMP    ND  \n",
      "6                                          NaN    NC  \n",
      "7                                          NaN   NaN  \n",
      "8                                          NaN    DC  \n",
      "9                                          NaN    NY  \n",
      "10                                         NaN    NY  \n",
      "11                                         NaN   NaN  \n",
      "12                                         NaN   NaN  \n",
      "13                                  FRANKLUNTZ   NaN  \n",
      "14                                       POTUS   NaN  \n",
      "15                                         NaN   NaN  \n",
      "16                                         NaN    CA  \n",
      "17                                         NaN    NJ  \n",
      "18                                         NaN   NaN  \n",
      "19                             REALDONALDTRUMP    UT  \n",
      "20                             REALDONALDTRUMP    UT  \n",
      "21                                         NaN    IL  \n",
      "22                             REALDONALDTRUMP   NaN  \n",
      "23                             REALDONALDTRUMP   NaN  \n",
      "24                                         NaN    WA  \n",
      "25                                         NaN    TX  \n",
      "26                             REALDONALDTRUMP    NJ  \n",
      "27                             REALDONALDTRUMP    CA  \n",
      "28                                         NaN    CA  \n",
      "29                       POTUS REALDONALDTRUMP   NaN  \n",
      "...                                        ...   ...  \n",
      "274490                                     NaN   NaN  \n",
      "274491                                     NaN    CA  \n",
      "274492                                     NaN   NaN  \n",
      "274493                                     NaN    MA  \n",
      "274494                                     NaN   USA  \n",
      "274495                        REALDONALDTRUMP…   NaN  \n",
      "274496                                     NaN    MA  \n",
      "274497                                     NaN    NV  \n",
      "274498                                     NaN    WA  \n",
      "274499                                     NaN   NaN  \n",
      "274500                                     NaN   USA  \n",
      "274501                                     NaN    LA  \n",
      "274502                                 NYTIMES    DC  \n",
      "274503                                     NaN    NY  \n",
      "274504                   REALDONALDTRUMP POTUS    CA  \n",
      "274505                                     NaN   NaN  \n",
      "274506                                     NaN    MO  \n",
      "274507                                     NaN   NaN  \n",
      "274508                                     NaN    CA  \n",
      "274509  THEJUANWILLIAMS FOXNEWS FOXNEWSOPINION    TN  \n",
      "274510                                     NaN   NaN  \n",
      "274511                                     NaN    IL  \n",
      "274512                                     NaN    NC  \n",
      "274513                                     NaN    AZ  \n",
      "274514                                     NaN    NJ  \n",
      "274515                                     NaN    MA  \n",
      "274516                                     NaN   NaN  \n",
      "274517                                     NaN    PA  \n",
      "274518                                     NaN    CA  \n",
      "274519                                     NaN   NaN  \n",
      "\n",
      "[274520 rows x 18 columns]>\n"
     ]
    }
   ],
   "source": [
    "# Copy the gender column into the pre-processed dataset\n",
    "tweets_final = pd.read_excel(\"03sotu_with_states.xlsx\")\n",
    "\n",
    "print(tweets_final.head)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "274520\n"
     ]
    }
   ],
   "source": [
    "# Check the ordering of the datasets is identical by matching on id string:\n",
    "\n",
    "tweets_final[\"gender\"] = \"\"\n",
    "\n",
    "count=0;\n",
    "for i in range(len(tweets_final)):\n",
    "    if tweets_final.at[i, \"id_str\"] == tweets.at[i, \"id_str\"]:\n",
    "        count+=1\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The above shows that the order of rows in both datasets is identical, and the gender values can be copied across using the above method\n",
    "for i in range(len(tweets_final)):\n",
    "    tweets_final.at[i, \"gender\"] = tweets.at[i, \"gender_final\"]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Export the final dataset\n",
    "\n",
    "writer = pd.ExcelWriter('04sotu_with_gender_final.xlsx')\n",
    "tweets_final.to_excel(writer,'Sheet1')\n",
    "writer.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}