{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "423050d6-415a-4d17-b0fc-0446357d6bb7",
   "metadata": {},
   "source": [
    "# Livres à censurer\n",
    " \n",
    "Étude de la liste de livres à censurer de Matt Krause, un dinosaure républicain du Texas\n",
    "\n",
    "Source des informations: \n",
    "\n",
    "- [All 850 Books Texas Lawmaker Matt Krause Wants to Ban: An Analysis ](https://bookriot.com/texas-book-ban-list/)\n",
    "- [Liste de listes GoodReads avec tous les titres](https://www.goodreads.com/search?q=krause+ban+list&search%5Bsource%5D=goodreads&search_type=lists&tab=lists)\n",
    "\n",
    "## Importation des données de la liste de Krause"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4385661c-abf0-4fae-b20e-8ae5ce2c4a9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# NLP stuff\n",
    "import spacy\n",
    "import re\n",
    "from spacy import displacy\n",
    "from collections import Counter\n",
    "from string import punctuation\n",
    "\n",
    "# Dataviz stuff\n",
    "from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "\n",
    "# Data Science stuff\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c671c72-ae8c-4a0b-b2d7-915d4a60ce51",
   "metadata": {},
   "source": [
    "## Lecture des données brutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cd0c4cb-c1c9-44c9-ba07-59ddb4092808",
   "metadata": {},
   "outputs": [],
   "source": [
    "krausebooklist = pd.read_csv(\"krausebooklist.csv\", dtype={'Published': int})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "073346e3-24be-47d5-9b22-b5ef9fb34aeb",
   "metadata": {},
   "source": [
    "## Visualisation des années"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82a4e5a6-fb29-4a2e-8a39-a4e72d653791",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(5,5)) \n",
    "sns.histplot(data=krausebooklist, \n",
    "             x=\"Published\", \n",
    "             discrete=False, \n",
    "             binrange=[1965,2025],\n",
    "             binwidth=5)\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2881f4b2-1569-42a6-8d04-50b2e6ee1c59",
   "metadata": {},
   "source": [
    "## Données de GoodReads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53befb5c-cca3-4ec3-92dc-7d399605b1ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "goodreadsbooks = pd.read_json(\"scraped_data/goodreadsbooks.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c7c84ff-7a85-4164-bf58-5b8f946eed21",
   "metadata": {},
   "outputs": [],
   "source": [
    "goodreadsbooks['description_clean'] = \\\n",
    "goodreadsbooks['description'] \\\n",
    ".apply(lambda x: x.replace('\\n',' ') \\\n",
    "       .replace('_',' ') \\\n",
    "       .replace('*',' ') \\\n",
    "       .replace('...more',' ') \\\n",
    "       .replace('—',', ') \\\n",
    "       .replace('\\\\',' ') \\\n",
    "       .replace('”','\"') \\\n",
    "       .replace('“','\"')\n",
    "      ) \\\n",
    ".apply(lambda x: re.sub(\"\\-\\s\",\"-\",x)) \\\n",
    ".apply(lambda x: re.sub(\"\\s+\",\" \",x)) \\\n",
    ".apply(lambda x: re.sub(\"\\.\\s+$\",\". \",x)) \\\n",
    ".apply(lambda x: re.sub(\"\\s*--\\s*\",\", \",x)) \\\n",
    ".apply(lambda x: re.sub(\"\\s+\\.\",'.',x)) \\\n",
    ".apply(lambda x: re.sub(\"[\\\"]{2,}\",'\"',x))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7c475f3-1849-4bc8-b1c3-0bcd19f8e810",
   "metadata": {},
   "source": [
    "### Itération / Exportation\n",
    "\n",
    "Je fais ici des itérations pour alimenter les étapes de remplacement de caractères. j'ouvre le fichier dans Kate et j'active le rechargement automatique. comme ça je peux itérer rapidement avec les deux derniers blocs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7aefbdf-ba81-4047-95d0-f83882052601",
   "metadata": {},
   "outputs": [],
   "source": [
    "goodreadsbooks['description_clean'].to_csv(\"clean_data/goodreads_description_clean.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b804c85c-ab7f-4721-8493-7b7a52aa9a0e",
   "metadata": {},
   "source": [
    "### Chargement des données avec Spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca21ada6-20af-4794-a4c6-6267df3ff3fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"en_core_web_md\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74521c29-d980-4c07-a9d7-8d50b8a7522c",
   "metadata": {},
   "source": [
    "## Détection des mots-clés principaux"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c8c7a09-d047-4f3f-bb3b-5d957d138c57",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs = nlp.pipe(goodreadsbooks['description_clean'], n_process=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d7fca50-1451-48b5-bd80-2ced3d42a05b",
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_tag = ['PROPN', 'ADJ', 'NOUN', 'VERB']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5af3f14c-5c39-418b-91f3-919bbb8f2470",
   "metadata": {},
   "outputs": [],
   "source": [
    "i=0\n",
    "docs_list = []\n",
    "docs_keyword = []\n",
    "for doc in docs:\n",
    "    j=0\n",
    "    keyword = []\n",
    "    docs_list.append(doc)\n",
    "    for token in doc:\n",
    "        if(token.is_stop or token.text in punctuation):\n",
    "            continue\n",
    "        if(token.pos_ in pos_tag):\n",
    "            keyword.append({\n",
    "                'token': j,\n",
    "                'text': token.text,\n",
    "                'pos': token.pos_,\n",
    "                'lemma': token.lemma_,\n",
    "                'tag': token.tag_,\n",
    "                'dep': token.dep_\n",
    "            })\n",
    "        j=j+1\n",
    "    docs_keyword.append({'document':i, 'keywords':keyword})\n",
    "    i=i+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8093b6f1-fc41-4cec-be9c-95648e32ba08",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_keyword_exp = pd.DataFrame(docs_keyword).explode('keywords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf3d1371-e7ef-45b2-aa45-b09d030a3f1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_keyword_exp2 = pd.DataFrame(docs_keyword_exp['keywords'].values.tolist(), index=docs_keyword_exp.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d8bf599-af04-410b-9cc7-8113118a9daa",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_keyword_exp2['document'] = docs_keyword_exp['document']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c08de94-dc78-4f40-a925-d3f3a81b7db5",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_keyword_exp2 = docs_keyword_exp2.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98a58ec3-b4c5-468f-85eb-9a7dcb2b5e06",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_keyword_exp2['lemma'] = docs_keyword_exp2['lemma'].apply(lambda x: x.lower())\n",
    "docs_keyword_exp2['text'] = docs_keyword_exp2['text'].apply(lambda x: x.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47e681aa-645b-49ce-97b0-a264aea863c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "del docs_keyword_exp2['index']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4173fee6-d1ee-4154-ac6a-fcae7872d6cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_keyword_exp2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3561a708-9d03-4aa0-b9ef-23dbf3461e40",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_frequency = docs_keyword_exp2['lemma'].value_counts().to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "166177e5-d990-4f1f-b1a6-784e65108dbf",
   "metadata": {},
   "source": [
    "## Noun Chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d431db5-a1ff-4c2a-915d-ff70065aa5ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs2 = nlp.pipe(goodreadsbooks['description_clean'], n_process=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19d4060f-14a8-4305-b7c0-7dbbd8e4522b",
   "metadata": {},
   "outputs": [],
   "source": [
    "i=0\n",
    "docs_chunks = []\n",
    "docs2_list = []\n",
    "for doc in docs2:\n",
    "    j=0\n",
    "    keyword = []\n",
    "    docs2_list.append(doc)\n",
    "    for chunk in doc.noun_chunks:\n",
    "        if(chunk.root.text in punctuation):\n",
    "            continue\n",
    "        if(chunk.root.pos_ in pos_tag):\n",
    "            keyword.append({\n",
    "                'token': j,\n",
    "                'text': chunk.text,\n",
    "                'root': chunk.root.text,\n",
    "                'pos': chunk.root.pos_,\n",
    "                'lemma': chunk.root.lemma_,\n",
    "                'tag': chunk.root.tag_,\n",
    "                'dep': token.dep_\n",
    "            })\n",
    "        j=j+1\n",
    "    docs_chunks.append({'document':i, 'keywords':keyword})\n",
    "    i=i+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f89d4abe-2963-4c93-b411-c62937839bb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_chunks_exp = pd.DataFrame(docs_chunks).explode('keywords').reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a29efb9-37a3-48c4-8f53-428c267d6ca4",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_chunks_exp2 = pd.DataFrame(docs_chunks_exp['keywords'].values.tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2650d65-e6b7-444c-a29c-3dd7bf8ca98d",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_chunks_exp2['document'] = docs_chunks_exp['document']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b29ed8a9-7a4f-4bde-ab47-8b497ef44e76",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_chunks_exp2 = docs_chunks_exp2.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10e132dc-ff83-4fa2-a4cd-d99cabcea8a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_chunks_exp2['lemma'] = docs_chunks_exp2['lemma'].apply(lambda x: x.lower())\n",
    "docs_chunks_exp2['root'] = docs_chunks_exp2['root'].apply(lambda x: x.lower())\n",
    "docs_chunks_exp2['text'] = docs_chunks_exp2['text'].apply(lambda x: x.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd59bca3-19b3-4a25-abf3-bdbf00aac5b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "del docs_chunks_exp2['index']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fa8e713-d0a5-4d63-b947-30e01b2b9476",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_chunks_exp2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ec40fff-c929-47bb-bd89-24ebd4d82808",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "word_frequency_chunks = docs_chunks_exp2['text'].value_counts().to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb82dc33-19a3-40a3-bbad-18df4fdc32bf",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Word Cloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4134260d-bc74-490f-abc0-045fdcdac8c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "wordcloud = WordCloud(width=512,\n",
    "                      height=512,\n",
    "                      max_font_size=72, \n",
    "                      max_words=1000, \n",
    "                      background_color=\"#ece9dc\") \\\n",
    ".generate_from_frequencies(word_frequency_chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f563a57-89b2-4d7a-a3b8-a689c1f63a6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 10))\n",
    "plt.imshow(wordcloud, interpolation='bilinear')\n",
    "plt.axis(\"off\")\n",
    "plt.savefig(\"out_img/wordcloud_regular.png\", format=\"png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "337641f5-bee1-44b1-8fce-87aa44149609",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_mask = np.array(Image.open(\"mask_img/heart.png\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8cb1b3f-1be7-4e16-9b13-0146a4afb630",
   "metadata": {},
   "outputs": [],
   "source": [
    "wordcloud_mask = WordCloud(mode=\"RGB\",\n",
    "                           max_words=1000, \n",
    "                           mask=my_mask) \\\n",
    ".generate_from_frequencies(word_frequency_chunks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "785bfd5d-a53d-4d37-9412-77193139b575",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_colors = ImageColorGenerator(my_mask)\n",
    "\n",
    "plt.figure(figsize=(10, 10))\n",
    "plt.imshow(wordcloud_book.recolor(color_func=image_colors), interpolation='bilinear')\n",
    "plt.axis(\"off\")\n",
    "plt.savefig(\"out_img/wordcloud_heart.png\", format=\"png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b73bacab-810c-4f89-9e69-a8f3e9e8781f",
   "metadata": {},
   "source": [
    "## Exemples de phrases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50c79290-0b7f-49ce-b499-79dcf801d701",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_sample = docs_list[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3e9e1d9-9794-40d8-85c8-e22142316fc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_sentence = list(doc_sample.sents)[2]\n",
    "my_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dcc3722-ed6c-4c52-8878-89cf0ffcd634",
   "metadata": {},
   "outputs": [],
   "source": [
    "displacy.render(my_sentence, style=\"dep\", \n",
    "                options={'compact':True, \n",
    "                         \"bg\": \"#ece9dc\",\n",
    "                         \"color\": \"#3c3b39\"\n",
    "                        },\n",
    "                page=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc4f48c5-e208-4042-bd40-a455e07328a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "displacy.render(doc_sample, style=\"ent\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2287f1d0-bc6b-4958-9ae7-1e3fbfa522e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence2_1 = list(doc_sample.sents)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abfc6cd7-bff6-4b0a-b956-6a1a4b66d632",
   "metadata": {},
   "outputs": [],
   "source": [
    "your_word=sentence2_1[2].text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2630966-1c7c-48a3-98fe-e8617868021e",
   "metadata": {},
   "outputs": [],
   "source": [
    "your_word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d1405a6-d25f-4bcc-917e-48b3aad70b01",
   "metadata": {},
   "outputs": [],
   "source": [
    "ms = nlp.vocab.vectors.most_similar(\n",
    "    np.asarray([nlp.vocab.vectors[nlp.vocab.strings[your_word]]]), n=10)\n",
    "words = [nlp.vocab.strings[w] for w in ms[0][0]]\n",
    "distances = ms[2]\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bae29e2f-b945-45a8-b925-5d35839e4f0b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
