{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another popular text analysis technique is called topic modeling. The ultimate goal of topic modeling is to find various topics that are present in your corpus. Each document in the corpus will be made up of at least one topic, if not multiple topics.\n",
    "\n",
    "In this notebook, we will be covering the steps on how to do **Latent Dirichlet Allocation (LDA)**, which is one of many topic modeling techniques. It was specifically designed for text data.\n",
    "\n",
    "To use a topic modeling technique, you need to provide (1) a document-term matrix and (2) the number of topics you would like the algorithm to pick up.\n",
    "\n",
    "Once the topic modeling technique is applied, your job as a human is to interpret the results and see if the mix of words in each topic make sense. If they don't make sense, you can try changing up the number of topics, the terms in the document-term matrix, model parameters, or even try a different model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling - Attempt #1 (All Text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's read in our document-term matrix\n",
    "import pandas as pd\n",
    "import pickle\n",
    "\n",
    "data = pd.read_pickle('dtm_stop.pkl')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import the necessary modules for LDA with gensim\n",
    "# Terminal / Anaconda Navigator: conda install -c conda-forge gensim\n",
    "from gensim import matutils, models\n",
    "import scipy.sparse\n",
    "\n",
    "# import logging\n",
    "# logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# One of the required inputs is a term-document matrix\n",
    "tdm = data.transpose()\n",
    "tdm.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We're going to put the term-document matrix into a new gensim format, from df --> sparse matrix --> gensim corpus\n",
    "sparse_counts = scipy.sparse.csr_matrix(tdm)\n",
    "corpus = matutils.Sparse2Corpus(sparse_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Gensim also requires dictionary of the all terms and their respective location in the term-document matrix\n",
    "cv = pickle.load(open(\"cv_stop.pkl\", \"rb\"))\n",
    "id2word = dict((v, k) for k, v in cv.vocabulary_.items())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the corpus (term-document matrix) and id2word (dictionary of location: term), we need to specify two other parameters - the number of topics and the number of passes. Let's start the number of topics at 2, see if the results make sense, and increase the number from there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now that we have the corpus (term-document matrix) and id2word (dictionary of location: term),\n",
    "# we need to specify two other parameters as well - the number of topics and the number of passes\n",
    "lda = models.LdaModel(corpus=corpus, id2word=id2word, num_topics=2, passes=10)\n",
    "lda.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LDA for num_topics = 3\n",
    "lda = models.LdaModel(corpus=corpus, id2word=id2word, num_topics=3, passes=10)\n",
    "lda.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LDA for num_topics = 4\n",
    "lda = models.LdaModel(corpus=corpus, id2word=id2word, num_topics=4, passes=10)\n",
    "lda.print_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These topics aren't looking too great. We've tried modifying our parameters. Let's try modifying our terms list as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling - Attempt #2 (Nouns Only)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One popular trick is to look only at terms that are from one part of speech (only nouns, only adjectives, etc.). Check out the UPenn tag set: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's create a function to pull out nouns from a string of text\n",
    "from nltk import word_tokenize, pos_tag\n",
    "\n",
    "def nouns(text):\n",
    "    '''Given a string of text, tokenize the text and pull out only the nouns.'''\n",
    "    is_noun = lambda pos: pos[:2] == 'NN'\n",
    "    tokenized = word_tokenize(text)\n",
    "    all_nouns = [word for (word, pos) in pos_tag(tokenized) if is_noun(pos)] \n",
    "    return ' '.join(all_nouns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in the cleaned data, before the CountVectorizer step\n",
    "data_clean = pd.read_pickle('data_clean.pkl')\n",
    "data_clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the nouns function to the transcripts to filter only on nouns\n",
    "data_nouns = pd.DataFrame(data_clean.transcript.apply(nouns))\n",
    "data_nouns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new document-term matrix using only nouns\n",
    "from sklearn.feature_extraction import text\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Re-add the additional stop words since we are recreating the document-term matrix\n",
    "add_stop_words = ['like', 'im', 'know', 'just', 'dont', 'thats', 'right', 'people',\n",
    "                  'youre', 'got', 'gonna', 'time', 'think', 'yeah', 'said']\n",
    "stop_words = text.ENGLISH_STOP_WORDS.union(add_stop_words)\n",
    "\n",
    "# Recreate a document-term matrix with only nouns\n",
    "cvn = CountVectorizer(stop_words=stop_words)\n",
    "data_cvn = cvn.fit_transform(data_nouns.transcript)\n",
    "data_dtmn = pd.DataFrame(data_cvn.toarray(), columns=cvn.get_feature_names())\n",
    "data_dtmn.index = data_nouns.index\n",
    "data_dtmn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create the gensim corpus\n",
    "corpusn = matutils.Sparse2Corpus(scipy.sparse.csr_matrix(data_dtmn.transpose()))\n",
    "\n",
    "# Create the vocabulary dictionary\n",
    "id2wordn = dict((v, k) for k, v in cvn.vocabulary_.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's start with 2 topics\n",
    "ldan = models.LdaModel(corpus=corpusn, num_topics=2, id2word=id2wordn, passes=10)\n",
    "ldan.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try topics = 3\n",
    "ldan = models.LdaModel(corpus=corpusn, num_topics=3, id2word=id2wordn, passes=10)\n",
    "ldan.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try 4 topics\n",
    "ldan = models.LdaModel(corpus=corpusn, num_topics=4, id2word=id2wordn, passes=10)\n",
    "ldan.print_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Modeling - Attempt #3 (Nouns and Adjectives)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's create a function to pull out nouns from a string of text\n",
    "def nouns_adj(text):\n",
    "    '''Given a string of text, tokenize the text and pull out only the nouns and adjectives.'''\n",
    "    is_noun_adj = lambda pos: pos[:2] == 'NN' or pos[:2] == 'JJ'\n",
    "    tokenized = word_tokenize(text)\n",
    "    nouns_adj = [word for (word, pos) in pos_tag(tokenized) if is_noun_adj(pos)] \n",
    "    return ' '.join(nouns_adj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the nouns function to the transcripts to filter only on nouns\n",
    "data_nouns_adj = pd.DataFrame(data_clean.transcript.apply(nouns_adj))\n",
    "data_nouns_adj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new document-term matrix using only nouns and adjectives, also remove common words with max_df\n",
    "cvna = CountVectorizer(stop_words=stop_words, max_df=.8)\n",
    "data_cvna = cvna.fit_transform(data_nouns_adj.transcript)\n",
    "data_dtmna = pd.DataFrame(data_cvna.toarray(), columns=cvna.get_feature_names())\n",
    "data_dtmna.index = data_nouns_adj.index\n",
    "data_dtmna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create the gensim corpus\n",
    "corpusna = matutils.Sparse2Corpus(scipy.sparse.csr_matrix(data_dtmna.transpose()))\n",
    "\n",
    "# Create the vocabulary dictionary\n",
    "id2wordna = dict((v, k) for k, v in cvna.vocabulary_.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's start with 2 topics\n",
    "ldana = models.LdaModel(corpus=corpusna, num_topics=2, id2word=id2wordna, passes=10)\n",
    "ldana.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try 3 topics\n",
    "ldana = models.LdaModel(corpus=corpusna, num_topics=3, id2word=id2wordna, passes=10)\n",
    "ldana.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try 4 topics\n",
    "ldana = models.LdaModel(corpus=corpusna, num_topics=4, id2word=id2wordna, passes=10)\n",
    "ldana.print_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify Topics in Each Document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of the 9 topic models we looked at, the nouns and adjectives, 4 topic one made the most sense. So let's pull that down here and run it through some more iterations to get more fine-tuned topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.009*\"joke\" + 0.005*\"mom\" + 0.005*\"parents\" + 0.004*\"hasan\" + 0.004*\"jokes\" + 0.004*\"anthony\" + 0.003*\"nuts\" + 0.003*\"dead\" + 0.003*\"tit\" + 0.003*\"twitter\"'),\n",
       " (1,\n",
       "  '0.005*\"mom\" + 0.005*\"jenny\" + 0.005*\"clinton\" + 0.004*\"friend\" + 0.004*\"parents\" + 0.003*\"husband\" + 0.003*\"cow\" + 0.003*\"ok\" + 0.003*\"wife\" + 0.003*\"john\"'),\n",
       " (2,\n",
       "  '0.005*\"bo\" + 0.005*\"gun\" + 0.005*\"guns\" + 0.005*\"repeat\" + 0.004*\"um\" + 0.004*\"ass\" + 0.004*\"eye\" + 0.004*\"contact\" + 0.003*\"son\" + 0.003*\"class\"'),\n",
       " (3,\n",
       "  '0.006*\"ahah\" + 0.004*\"nigga\" + 0.004*\"gay\" + 0.003*\"dick\" + 0.003*\"door\" + 0.003*\"young\" + 0.003*\"motherfucker\" + 0.003*\"stupid\" + 0.003*\"bitch\" + 0.003*\"mad\"')]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Our final LDA model (for now)\n",
    "ldana = models.LdaModel(corpus=corpusna, num_topics=4, id2word=id2wordna, passes=80)\n",
    "ldana.print_topics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These four topics look pretty decent. Let's settle on these for now.\n",
    "* Topic 0: mom, parents\n",
    "* Topic 1: husband, wife\n",
    "* Topic 2: guns\n",
    "* Topic 3: profanity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 'ali'),\n",
       " (0, 'anthony'),\n",
       " (2, 'bill'),\n",
       " (2, 'bo'),\n",
       " (3, 'dave'),\n",
       " (0, 'hasan'),\n",
       " (2, 'jim'),\n",
       " (3, 'joe'),\n",
       " (1, 'john'),\n",
       " (0, 'louis'),\n",
       " (1, 'mike'),\n",
       " (0, 'ricky')]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's take a look at which topics each transcript contains\n",
    "corpus_transformed = ldana[corpusna]\n",
    "list(zip([a for [(a,b)] in corpus_transformed], data_dtmna.index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "For a first pass of LDA, these kind of make sense to me, so we'll call it a day for now.\n",
    "* Topic 0: mom, parents [Anthony, Hasan, Louis, Ricky]\n",
    "* Topic 1: husband, wife [Ali, John, Mike]\n",
    "* Topic 2: guns [Bill, Bo, Jim]\n",
    "* Topic 3: profanity [Dave, Joe]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Additional Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Try further modifying the parameters of the topic models above and see if you can get better topics.\n",
    "2. Create a new topic model that includes terms from a different [part of speech](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html) and see if you can get better topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}