{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a05c860c",
   "metadata": {},
   "source": [
    "# Split by tokens \n",
    "\n",
    "Language models have a token limit. You should not exceed the token limit. When you split your text into chunks it is therefore a good idea to count the number of tokens. There are many tokenizers. When you count tokens in your text you should use the same tokenizer as used in the language model. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7683b36a",
   "metadata": {},
   "source": [
    "## tiktoken\n",
    "\n",
    ">[tiktoken](https://github.com/openai/tiktoken) is a fast `BPE` tokenizer created by `OpenAI`.\n",
    "\n",
    "\n",
    "We can use it to estimate tokens used. It will probably be more accurate for the OpenAI models.\n",
    "\n",
    "1. How the text is split: by character passed in\n",
    "2. How the chunk size is measured: by `tiktoken` tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c4ef83e-f43a-4658-ad1a-3952e0a5bbe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1ad2d0f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a long document we can split up.\n",
    "with open(\"../../../state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()\n",
    "from langchain.text_splitter import CharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "825f7c0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n",
    "    chunk_size=100, chunk_overlap=0\n",
    ")\n",
    "texts = text_splitter.split_text(state_of_the_union)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ae35d165",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n",
      "\n",
      "Last year COVID-19 kept us apart. This year we are finally together again. \n",
      "\n",
      "Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n",
      "\n",
      "With a duty to one another to the American people to the Constitution.\n"
     ]
    }
   ],
   "source": [
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de5b6a6e",
   "metadata": {},
   "source": [
    "We can also load a tiktoken splitter directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4454c70e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import TokenTextSplitter\n",
    "\n",
    "text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)\n",
    "\n",
    "texts = text_splitter.split_text(state_of_the_union)\n",
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f95f06",
   "metadata": {},
   "source": [
    "## spaCy\n",
    "\n",
    ">[spaCy](https://spacy.io/) is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython.\n",
    "\n",
    "Another alternative to `NLTK` is to use [spaCy tokenizer](https://spacy.io/api/tokenizer).\n",
    "\n",
    "1. How the text is split: by `spaCy` tokenizer\n",
    "2. How the chunk size is measured: by number of characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0b9242f-690c-4819-b35a-bb68187281ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f1de7767",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a long document we can split up.\n",
    "with open(\"../../../state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f4ec9b90",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import SpacyTextSplitter\n",
    "\n",
    "text_splitter = SpacyTextSplitter(chunk_size=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cef2b29e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.\n",
      "\n",
      "Members of Congress and the Cabinet.\n",
      "\n",
      "Justices of the Supreme Court.\n",
      "\n",
      "My fellow Americans.  \n",
      "\n",
      "\n",
      "\n",
      "Last year COVID-19 kept us apart.\n",
      "\n",
      "This year we are finally together again. \n",
      "\n",
      "\n",
      "\n",
      "Tonight, we meet as Democrats Republicans and Independents.\n",
      "\n",
      "But most importantly as Americans. \n",
      "\n",
      "\n",
      "\n",
      "With a duty to one another to the American people to the Constitution. \n",
      "\n",
      "\n",
      "\n",
      "And with an unwavering resolve that freedom will always triumph over tyranny. \n",
      "\n",
      "\n",
      "\n",
      "Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.\n",
      "\n",
      "But he badly miscalculated. \n",
      "\n",
      "\n",
      "\n",
      "He thought he could roll into Ukraine and the world would roll over.\n",
      "\n",
      "Instead he met a wall of strength he never imagined. \n",
      "\n",
      "\n",
      "\n",
      "He met the Ukrainian people. \n",
      "\n",
      "\n",
      "\n",
      "From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.\n"
     ]
    }
   ],
   "source": [
    "texts = text_splitter.split_text(state_of_the_union)\n",
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73dbcdb9",
   "metadata": {},
   "source": [
    "## SentenceTransformers\n",
    "\n",
    "The `SentenceTransformersTokenTextSplitter` is a specialized text splitter for use with the sentence-transformer models. The default behaviour is to split the text into chunks that fit the token window of the sentence transformer model that you would like to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9dd5419e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import SentenceTransformersTokenTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b43e5d54",
   "metadata": {},
   "outputs": [],
   "source": [
    "splitter = SentenceTransformersTokenTextSplitter(chunk_overlap=0)\n",
    "text = \"Lorem \""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1df84cb4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "count_start_and_stop_tokens = 2\n",
    "text_token_count = splitter.count_tokens(text=text) - count_start_and_stop_tokens\n",
    "print(text_token_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d7ad2213",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens in text to split: 514\n"
     ]
    }
   ],
   "source": [
    "token_multiplier = splitter.maximum_tokens_per_chunk // text_token_count + 1\n",
    "\n",
    "# `text_to_split` does not fit in a single chunk\n",
    "text_to_split = text * token_multiplier\n",
    "\n",
    "print(f\"tokens in text to split: {splitter.count_tokens(text=text_to_split)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "818aea04",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lorem\n"
     ]
    }
   ],
   "source": [
    "text_chunks = splitter.split_text(text=text_to_split)\n",
    "\n",
    "print(text_chunks[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea2973ac",
   "metadata": {},
   "source": [
    "## NLTK\n",
    "\n",
    ">[The Natural Language Toolkit](https://en.wikipedia.org/wiki/Natural_Language_Toolkit), or more commonly [NLTK](https://www.nltk.org/), is a suite of libraries and programs for symbolic and statistical natural language processing (NLP) for English written in the Python programming language.\n",
    "\n",
    "Rather than just splitting on \"\\n\\n\", we can use `NLTK` to split based on [NLTK tokenizers](https://www.nltk.org/api/nltk.tokenize.html).\n",
    "\n",
    "1. How the text is split: by `NLTK` tokenizer.\n",
    "2. How the chunk size is measured:by number of characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6af9886-7d53-4aab-84f6-303c4cce7882",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "aed17ddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a long document we can split up.\n",
    "with open(\"../../../state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "20fa9c23",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.text_splitter import NLTKTextSplitter\n",
    "\n",
    "text_splitter = NLTKTextSplitter(chunk_size=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5ea10835",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.\n",
      "\n",
      "Members of Congress and the Cabinet.\n",
      "\n",
      "Justices of the Supreme Court.\n",
      "\n",
      "My fellow Americans.\n",
      "\n",
      "Last year COVID-19 kept us apart.\n",
      "\n",
      "This year we are finally together again.\n",
      "\n",
      "Tonight, we meet as Democrats Republicans and Independents.\n",
      "\n",
      "But most importantly as Americans.\n",
      "\n",
      "With a duty to one another to the American people to the Constitution.\n",
      "\n",
      "And with an unwavering resolve that freedom will always triumph over tyranny.\n",
      "\n",
      "Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.\n",
      "\n",
      "But he badly miscalculated.\n",
      "\n",
      "He thought he could roll into Ukraine and the world would roll over.\n",
      "\n",
      "Instead he met a wall of strength he never imagined.\n",
      "\n",
      "He met the Ukrainian people.\n",
      "\n",
      "From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.\n",
      "\n",
      "Groups of citizens blocking tanks with their bodies.\n"
     ]
    }
   ],
   "source": [
    "texts = text_splitter.split_text(state_of_the_union)\n",
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13dc0983",
   "metadata": {},
   "source": [
    "## Hugging Face tokenizer\n",
    "\n",
    ">[Hugging Face](https://huggingface.co/docs/tokenizers/index) has many tokenizers.\n",
    "\n",
    "We use Hugging Face tokenizer, the [GPT2TokenizerFast](https://huggingface.co/Ransaka/gpt2-tokenizer-fast) to count the text length in tokens.\n",
    "\n",
    "1. How the text is split: by character passed in\n",
    "2. How the chunk size is measured: by number of tokens calculated by the `Hugging Face` tokenizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a8ce51d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import GPT2TokenizerFast\n",
    "\n",
    "tokenizer = GPT2TokenizerFast.from_pretrained(\"gpt2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "388369ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a long document we can split up.\n",
    "with open(\"../../../state_of_the_union.txt\") as f:\n",
    "    state_of_the_union = f.read()\n",
    "from langchain.text_splitter import CharacterTextSplitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ca5e72c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_splitter = CharacterTextSplitter.from_huggingface_tokenizer(\n",
    "    tokenizer, chunk_size=100, chunk_overlap=0\n",
    ")\n",
    "texts = text_splitter.split_text(state_of_the_union)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "37cdfbeb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.  \n",
      "\n",
      "Last year COVID-19 kept us apart. This year we are finally together again. \n",
      "\n",
      "Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n",
      "\n",
      "With a duty to one another to the American people to the Constitution.\n"
     ]
    }
   ],
   "source": [
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a43b0fa6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
