{ "cells": [ { "cell_type": "markdown", "id": "9fc3897d-176f-4729-8fd1-cfb4add53abd", "metadata": {}, "source": [ "## VDMS multi-modal RAG\n", "\n", "Many documents contain a mixture of content types, including text and images. \n", "\n", "Yet, information captured in images is lost in most RAG applications.\n", "\n", "With the emergence of multimodal LLMs, like [GPT-4V](https://openai.com/research/gpt-4v-system-card), it is worth considering how to utilize images in RAG. \n", "\n", "This cookbook highlights: \n", "* Use of [Unstructured](https://unstructured.io/) to parse images, text, and tables from documents (PDFs).\n", "* Use of multimodal embeddings (such as [CLIP](https://openai.com/research/clip)) to embed images and text\n", "* Use of [VDMS](https://github.com/IntelLabs/vdms/blob/master/README.md) as a vector store with support for multi-modal\n", "* Retrieval of both images and text using similarity search\n", "* Passing raw images and text chunks to a multimodal LLM for answer synthesis " ] }, { "cell_type": "markdown", "id": "2498a0a1", "metadata": {}, "source": [ "## Packages\n", "\n", "For `unstructured`, you will also need `poppler` ([installation instructions](https://pdf2image.readthedocs.io/en/latest/installation.html)) and `tesseract` ([installation instructions](https://tesseract-ocr.github.io/tessdoc/Installation.html)) in your system." ] }, { "cell_type": "code", "execution_count": 1, "id": "febbc459-ebba-4c1a-a52b-fed7731593f8", "metadata": {}, "outputs": [], "source": [ "! pip install --quiet -U langchain-vdms langchain-experimental langchain-ollama\n", "\n", "# lock to 0.10.19 due to a persistent bug in more recent versions\n", "! pip install --quiet pdf2image \"unstructured[all-docs]==0.10.19\" \"onnxruntime==1.17.0\" pillow pydantic lxml open_clip_torch" ] }, { "cell_type": "code", "execution_count": 2, "id": "78ac6543", "metadata": {}, "outputs": [], "source": [ "# from dotenv import load_dotenv, find_dotenv\n", "# load_dotenv(find_dotenv(), override=True);" ] }, { "cell_type": "markdown", "id": "e5c8916e", "metadata": {}, "source": [ "## Start VDMS Server\n", "\n", "Let's start a VDMS docker using port 55559 instead of default 55555. \n", "Keep note of the port and hostname as this is needed for the vector store as it uses the VDMS Python client to connect to the server." ] }, { "cell_type": "code", "execution_count": 3, "id": "1e6e2c15", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "a701e5ac3523006e9540b5355e2d872d5d78383eab61562a675d5b9ac21fde65\n" ] } ], "source": [ "! docker run --rm -d -p 55559:55555 --name vdms_rag_nb intellabs/vdms:latest\n", "\n", "# Connect to VDMS Vector Store\n", "from langchain_vdms.vectorstores import VDMS_Client\n", "\n", "vdms_client = VDMS_Client(port=55559)" ] }, { "cell_type": "markdown", "id": "1e94b3fb-8e3e-4736-be0a-ad881626c7bd", "metadata": {}, "source": [ "## Data Loading\n", "\n", "### Partition PDF text and images\n", " \n", "Let's use famous photographs from the PDF version of Library of Congress Magazine in this example.\n", "\n", "We can use `partition_pdf` from [Unstructured](https://unstructured-io.github.io/unstructured/introduction.html#key-concepts) to extract text and images." ] }, { "cell_type": "code", "execution_count": 4, "id": "9646b524-71a7-4b2a-bdc8-0b81f77e968f", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "import requests\n", "\n", "# Folder to store pdf and extracted images\n", "base_datapath = Path(\"./data/multimodal_files\").resolve()\n", "datapath = base_datapath / \"images\"\n", "datapath.mkdir(parents=True, exist_ok=True)\n", "\n", "pdf_url = \"https://www.loc.gov/lcm/pdf/LCM_2020_1112.pdf\"\n", "pdf_path = str(base_datapath / pdf_url.split(\"/\")[-1])\n", "with open(pdf_path, \"wb\") as f:\n", " f.write(requests.get(pdf_url).content)" ] }, { "cell_type": "code", "execution_count": 5, "id": "bc4839c0-8773-4a07-ba59-5364501269b2", "metadata": {}, "outputs": [], "source": [ "# Extract images, tables, and chunk text\n", "from unstructured.partition.pdf import partition_pdf\n", "\n", "raw_pdf_elements = partition_pdf(\n", " filename=pdf_path,\n", " extract_images_in_pdf=True,\n", " infer_table_structure=True,\n", " chunking_strategy=\"by_title\",\n", " max_characters=4000,\n", " new_after_n_chars=3800,\n", " combine_text_under_n_chars=2000,\n", " image_output_dir_path=datapath,\n", ")\n", "\n", "datapath = str(datapath)" ] }, { "cell_type": "code", "execution_count": 6, "id": "969545ad", "metadata": {}, "outputs": [], "source": [ "# Categorize text elements by type\n", "tables = []\n", "texts = []\n", "for element in raw_pdf_elements:\n", " if \"unstructured.documents.elements.Table\" in str(type(element)):\n", " tables.append(str(element))\n", " elif \"unstructured.documents.elements.CompositeElement\" in str(type(element)):\n", " texts.append(str(element))" ] }, { "cell_type": "markdown", "id": "5d8e6349-1547-4cbf-9c6f-491d8610ec10", "metadata": {}, "source": [ "## Multi-modal embeddings with our document\n", "\n", "In this section, we initialize the VDMS vector store for both text and images. For better performance, we use model `ViT-g-14` from [OpenClip multimodal embeddings](https://python.langchain.com/docs/integrations/text_embedding/open_clip).\n", "The images are stored as base64 encoded strings with `vectorstore.add_images`.\n" ] }, { "cell_type": "code", "execution_count": 7, "id": "4bc15842-cb95-4f84-9eb5-656b0282a800", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "from langchain_experimental.open_clip import OpenCLIPEmbeddings\n", "from langchain_vdms import VDMS\n", "\n", "# Create VDMS\n", "vectorstore = VDMS(\n", " client=vdms_client,\n", " collection_name=\"mm_rag_clip_photos\",\n", " embedding=OpenCLIPEmbeddings(model_name=\"ViT-g-14\", checkpoint=\"laion2b_s34b_b88k\"),\n", ")\n", "\n", "# Get image URIs with .jpg extension only\n", "image_uris = sorted(\n", " [\n", " os.path.join(datapath, image_name)\n", " for image_name in os.listdir(datapath)\n", " if image_name.endswith(\".jpg\")\n", " ]\n", ")\n", "\n", "# Add images\n", "if image_uris:\n", " vectorstore.add_images(uris=image_uris)\n", "\n", "# Add documents\n", "if texts:\n", " vectorstore.add_texts(texts=texts)\n", "\n", "# Make retriever\n", "retriever = vectorstore.as_retriever()" ] }, { "cell_type": "markdown", "id": "02a186d0-27e0-4820-8092-63b5349dd25d", "metadata": {}, "source": [ "## RAG\n", "\n", "Here we define helper functions for image results." ] }, { "cell_type": "code", "execution_count": 8, "id": "344f56a8-0dc3-433e-851c-3f7600c7a72b", "metadata": {}, "outputs": [], "source": [ "import base64\n", "from io import BytesIO\n", "\n", "from PIL import Image\n", "\n", "\n", "def resize_base64_image(base64_string, size=(128, 128)):\n", " \"\"\"\n", " Resize an image encoded as a Base64 string.\n", "\n", " Args:\n", " base64_string (str): Base64 string of the original image.\n", " size (tuple): Desired size of the image as (width, height).\n", "\n", " Returns:\n", " str: Base64 string of the resized image.\n", " \"\"\"\n", " # Decode the Base64 string\n", " img_data = base64.b64decode(base64_string)\n", " img = Image.open(BytesIO(img_data))\n", "\n", " # Resize the image\n", " resized_img = img.resize(size, Image.LANCZOS)\n", "\n", " # Save the resized image to a bytes buffer\n", " buffered = BytesIO()\n", " resized_img.save(buffered, format=img.format)\n", "\n", " # Encode the resized image to Base64\n", " return base64.b64encode(buffered.getvalue()).decode(\"utf-8\")\n", "\n", "\n", "def is_base64(s):\n", " \"\"\"Check if a string is Base64 encoded\"\"\"\n", " try:\n", " return base64.b64encode(base64.b64decode(s)) == s.encode()\n", " except Exception:\n", " return False\n", "\n", "\n", "def split_image_text_types(docs):\n", " \"\"\"Split numpy array images and texts\"\"\"\n", " images = []\n", " text = []\n", " for doc in docs:\n", " doc = doc.page_content # Extract Document contents\n", " if is_base64(doc):\n", " # Resize image to avoid OAI server error\n", " images.append(\n", " resize_base64_image(doc, size=(250, 250))\n", " ) # base64 encoded str\n", " else:\n", " text.append(doc)\n", " return {\"images\": images, \"texts\": text}" ] }, { "cell_type": "markdown", "id": "23a2c1d8-fea6-4152-b184-3172dd46c735", "metadata": {}, "source": [ "Currently, we format the inputs using a `RunnableLambda` while we add image support to `ChatPromptTemplates`.\n", "\n", "Our runnable follows the classic RAG flow - \n", "\n", "* We first compute the context (both \"texts\" and \"images\" in this case) and the question (just a RunnablePassthrough here) \n", "* Then we pass this into our prompt template, which is a custom function that formats the message for the llava model. \n", "* And finally we parse the output as a string.\n", "\n", "Here we are using Ollama to serve the Llava model. Please see [Ollama](https://python.langchain.com/docs/integrations/llms/ollama) for setup instructions." ] }, { "cell_type": "code", "execution_count": 9, "id": "4c93fab3-74c4-4f1d-958a-0bc4cdd0797e", "metadata": {}, "outputs": [], "source": [ "from langchain_core.messages import HumanMessage, SystemMessage\n", "from langchain_core.output_parsers import StrOutputParser\n", "from langchain_core.runnables import RunnableLambda, RunnablePassthrough\n", "from langchain_ollama.llms import OllamaLLM\n", "\n", "\n", "def prompt_func(data_dict):\n", " # Joining the context texts into a single string\n", " formatted_texts = \"\\n\".join(data_dict[\"context\"][\"texts\"])\n", " messages = []\n", "\n", " # Adding image(s) to the messages if present\n", " if data_dict[\"context\"][\"images\"]:\n", " image_message = {\n", " \"type\": \"image_url\",\n", " \"image_url\": {\n", " \"url\": f\"data:image/jpeg;base64,{data_dict['context']['images'][0]}\"\n", " },\n", " }\n", " messages.append(image_message)\n", "\n", " # Adding the text message for analysis\n", " text_message = {\n", " \"type\": \"text\",\n", " \"text\": (\n", " \"As an expert art critic and historian, your task is to analyze and interpret images, \"\n", " \"considering their historical and cultural significance. Alongside the images, you will be \"\n", " \"provided with related text to offer context. Both will be retrieved from a vectorstore based \"\n", " \"on user-input keywords. Please use your extensive knowledge and analytical skills to provide a \"\n", " \"comprehensive summary that includes:\\n\"\n", " \"- A detailed description of the visual elements in the image.\\n\"\n", " \"- The historical and cultural context of the image.\\n\"\n", " \"- An interpretation of the image's symbolism and meaning.\\n\"\n", " \"- Connections between the image and the related text.\\n\\n\"\n", " f\"User-provided keywords: {data_dict['question']}\\n\\n\"\n", " \"Text and / or tables:\\n\"\n", " f\"{formatted_texts}\"\n", " ),\n", " }\n", " messages.append(text_message)\n", " return [HumanMessage(content=messages)]\n", "\n", "\n", "def multi_modal_rag_chain(retriever):\n", " \"\"\"Multi-modal RAG chain\"\"\"\n", "\n", " # Multi-modal LLM\n", " llm_model = OllamaLLM(\n", " verbose=True, temperature=0.5, model=\"llava\", base_url=\"http://localhost:11434\"\n", " )\n", "\n", " # RAG pipeline\n", " chain = (\n", " {\n", " \"context\": retriever | RunnableLambda(split_image_text_types),\n", " \"question\": RunnablePassthrough(),\n", " }\n", " | RunnableLambda(prompt_func)\n", " | llm_model\n", " | StrOutputParser()\n", " )\n", "\n", " return chain" ] }, { "cell_type": "markdown", "id": "1566096d-97c2-4ddc-ba4a-6ef88c525e4e", "metadata": {}, "source": [ "## Test retrieval and run RAG\n", "Now let's query for a `woman with children` and retrieve the top results." ] }, { "cell_type": "code", "execution_count": 10, "id": "90121e56-674b-473b-871d-6e4753fd0c45", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "GREAT PHOTOGRAPHS\n", "The subject of the photo, Florence Owens Thompson, a Cherokee from Oklahoma, initially regretted that Lange ever made this photograph. “She was a very strong woman. She was a leader,” her daughter Katherine later said. “I think that's one of the reasons she resented the photo — because it didn't show her in that light.”\n", "\n", "DOROTHEA LANGE. “DESTITUTE PEA PICKERS IN CALIFORNIA. MOTHER OF SEVEN CHILDREN. AGE THIRTY-TWO. NIPOMO, CALIFORNIA.” MARCH 1936. NITRATE NEGATIVE. FARM SECURITY ADMINISTRATION-OFFICE OF WAR INFORMATION COLLECTION. PRINTS AND PHOTOGRAPHS DIVISION.\n", "\n", "—Helena Zinkham\n", "\n", "—Helena Zinkham\n", "\n", "NOVEMBER/DECEMBER 2020 LOC.GOV/LCM\n" ] }, { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "© 2017 LARRY D. MOORE\n", "\n", "contemporary criticism of the less-than- thoughtful circumstances under which Lange photographed Thomson, the picture’s power to engage has not diminished. Artists in other countries have appropriated the image, changing the mother’s features into those of other ethnicities, but keeping her expression and the positions of her clinging children. Long after anyone could help the Thompson family, this picture has resonance in another time of national crisis, unemployment and food shortages.\n", "\n", "A striking, but very different picture is a 1900 portrait of the legendary Hin-mah-too-yah- lat-kekt (Chief Joseph) of the Nez Percé people. The Bureau of American Ethnology in Washington, D.C., regularly arranged for its photographer, De Lancey Gill, to photograph Native American delegations that came to the capital to confer with officials about tribal needs and concerns. Although Gill described Chief Joseph as having “an air of gentleness and quiet reserve,” the delegate skeptically appraises the photographer, which is not surprising given that the United States broke five treaties with Chief Joseph and his father between 1855 and 1885.\n", "\n", "More than a glance, second looks may reveal new knowledge into complex histories.\n", "\n", "Anne Wilkes Tucker is the photography curator emeritus of the Museum of Fine Arts, Houston and curator of the “Not an Ostrich” exhibition.\n", "\n", "28\n", "\n", "28 LIBRARY OF CONGRESS MAGAZINE\n", "\n", "LIBRARY OF CONGRESS MAGAZINE\n", "THEYRE WILLING TO HAVE MEENTERTAIN THEM DURING THE DAY,BUT AS SOON AS IT STARTSGETTING DARK, THEY ALLGO OFF, AND LEAVE ME! \n", "ROSA PARKS: IN HER OWN WORDS\n", "\n", "COMIC ART: 120 YEARS OF PANELS AND PAGES\n", "\n", "SHALL NOT BE DENIED: WOMEN FIGHT FOR THE VOTE\n", "\n", "More information loc.gov/exhibits\n", "Nuestra Sefiora de las Iguanas\n", "\n", "Graciela Iturbide’s 1979 portrait of Zobeida Díaz in the town of Juchitán in southeastern Mexico conveys the strength of women and reflects their important contributions to the economy. Díaz, a merchant, was selling iguanas to cook and eat, carrying them on her head, as is customary.\n", "\n", "GRACIELA ITURBIDE. “NUESTRA SEÑORA DE LAS IGUANAS.” 1979. GELATIN SILVER PRINT. © GRACIELA ITURBIDE, USED BY PERMISSION. PRINTS AND PHOTOGRAPHS DIVISION.\n", "\n", "Iturbide requested permission to take a photograph, but this proved challenging because the iguanas were constantly moving, causing Díaz to laugh. The result, however, was a brilliant portrait that the inhabitants of Juchitán claimed with pride. They have reproduced it on posters and erected a statue honoring Díaz and her iguanas. The photo now appears throughout the world, inspiring supporters of feminism, women’s rights and gender equality.\n", "\n", "—Adam Silvia is a curator in the Prints and Photographs Division.\n", "\n", "6\n", "\n", "6 LIBRARY OF CONGRESS MAGAZINE\n", "\n", "LIBRARY OF CONGRESS MAGAZINE\n", "\n", "‘Migrant Mother’ is Florence Owens Thompson\n", "\n", "The iconic portrait that became the face of the Great Depression is also the most famous photograph in the collections of the Library of Congress.\n", "\n", "The Library holds the original source of the photo — a nitrate negative measuring 4 by 5 inches. Do you see a faint thumb in the bottom right? The photographer, Dorothea Lange, found the thumb distracting and after a few years had the negative altered to make the thumb almost invisible. Lange’s boss at the Farm Security Administration, Roy Stryker, criticized her action because altering a negative undermines the credibility of a documentary photo.\n", "Shrimp Picker\n", "\n", "The photos and evocative captions of Lewis Hine served as source material for National Child Labor Committee reports and exhibits exposing abusive child labor practices in the United States in the first decades of the 20th century.\n", "\n", "LEWIS WICKES HINE. “MANUEL, THE YOUNG SHRIMP-PICKER, FIVE YEARS OLD, AND A MOUNTAIN OF CHILD-LABOR OYSTER SHELLS BEHIND HIM. HE WORKED LAST YEAR. UNDERSTANDS NOT A WORD OF ENGLISH. DUNBAR, LOPEZ, DUKATE COMPANY. LOCATION: BILOXI, MISSISSIPPI.” FEBRUARY 1911. NATIONAL CHILD LABOR COMMITTEE COLLECTION. PRINTS AND PHOTOGRAPHS DIVISION.\n", "\n", "For 15 years, Hine\n", "\n", "crisscrossed the country, documenting the practices of the worst offenders. His effective use of photography made him one of the committee's greatest publicists in the campaign for legislation to ban child labor.\n", "\n", "Hine was a master at taking photos that catch attention and convey a message and, in this photo, he framed Manuel in a setting that drove home the boy’s small size and unsafe environment.\n", "\n", "Captions on photos of other shrimp pickers emphasized their long working hours as well as one hazard of the job: The acid from the shrimp made pickers’ hands sore and “eats the shoes off your feet.”\n", "\n", "Such images alerted viewers to all that workers, their families and the nation sacrificed when children were part of the labor force. The Library holds paper records of the National Child Labor Committee as well as over 5,000 photographs.\n", "\n", "—Barbara Natanson is head of the Reference Section in the Prints and Photographs Division.\n", "\n", "8\n", "\n", "LIBRARY OF CONGRESS MAGAZINE\n", "\n", "LIBRARY OF CONGRESS MAGAZINE\n", "\n", "Intergenerational Portrait\n", "\n", "Raised on the Apsáalooke (Crow) reservation in Montana, photographer Wendy Red Star created her “Apsáalooke Feminist” self-portrait series with her daughter Beatrice. With a dash of wry humor, mother and daughter are their own first-person narrators.\n", "\n", "Red Star explains the significance of their appearance: “The dress has power: You feel strong and regal wearing it. In my art, the elk tooth dress specifically symbolizes Crow womanhood and the matrilineal line connecting me to my ancestors. As a mother, I spend hours searching for the perfect elk tooth dress materials to make a prized dress for my daughter.”\n", "\n", "In a world that struggles with cultural identities, this photograph shows us the power and beauty of blending traditional and contemporary styles.\n", "‘American Gothic’ Product #216040262 Price: $24\n", "\n", "U.S. Capitol at Night Product #216040052 Price: $24\n", "\n", "Good Reading Ahead Product #21606142 Price: $24\n", "\n", "Gordon Parks created an iconic image with this 1942 photograph of cleaning woman Ella Watson.\n", "\n", "Snow blankets the U.S. Capitol in this classic image by Ernest L. Crandall.\n", "\n", "Start your new year out right with a poster promising good reading for months to come.\n", "\n", "▪ Order online: loc.gov/shop ▪ Order by phone: 888.682.3557\n", "\n", "26\n", "\n", "LIBRARY OF CONGRESS MAGAZINE\n", "\n", "LIBRARY OF CONGRESS MAGAZINE\n", "\n", "SUPPORT\n", "\n", "A PICTURE OF PHILANTHROPY Annenberg Foundation Gives $1 Million and a Photographic Collection to the Library.\n", "\n", "A major gift by Wallis Annenberg and the Annenberg Foundation in Los Angeles will support the effort to reimagine the visitor experience at the Library of Congress. The foundation also is donating 1,000 photographic prints from its Annenberg Space for Photography exhibitions to the Library.\n", "\n", "The Library is pursuing a multiyear plan to transform the experience of its nearly 2 million annual visitors, share more of its treasures with the public and show how Library collections connect with visitors’ own creativity and research. The project is part of a strategic plan established by Librarian of Congress Carla Hayden to make the Library more user-centered for Congress, creators and learners of all ages.\n", "\n", "A 2018 exhibition at the Annenberg Space for Photography in Los Angeles featured over 400 photographs from the Library. The Library is planning a future photography exhibition, based on the Annenberg-curated show, along with a documentary film on the Library and its history, produced by the Annenberg Space for Photography.\n", "\n", "“The nation’s library is honored to have the strong support of Wallis Annenberg and the Annenberg Foundation as we enhance the experience for our visitors,” Hayden said. “We know that visitors will find new connections to the Library through the incredible photography collections and countless other treasures held here to document our nation’s history and creativity.”\n", "\n", "To enhance the Library’s holdings, the foundation is giving the Library photographic prints for long-term preservation from 10 other exhibitions hosted at the Annenberg Space for Photography. The Library holds one of the world’s largest photography collections, with about 14 million photos and over 1 million images digitized and available online.\n", "18 LIBRARY OF CONGRESS MAGAZINE\n" ] } ], "source": [ "from IPython.display import HTML, display\n", "\n", "\n", "def plt_img_base64(img_base64):\n", " # Create an HTML img tag with the base64 string as the source\n", " image_html = f''\n", "\n", " # Display the image by rendering the HTML\n", " display(HTML(image_html))\n", "\n", "\n", "query = \"Woman with children\"\n", "docs = retriever.invoke(query, k=10)\n", "\n", "for doc in docs:\n", " if is_base64(doc.page_content):\n", " plt_img_base64(doc.page_content)\n", " else:\n", " print(doc.page_content)" ] }, { "cell_type": "markdown", "id": "15e9b54d", "metadata": {}, "source": [ "Now let's use the `multi_modal_rag_chain` to process the same query and display the response." ] }, { "cell_type": "code", "execution_count": 11, "id": "69fb15fd-76fc-49b4-806d-c4db2990027d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " The image is a black and white photograph by Dorothea Lange titled \"Destitute Pea Pickers in California. Mother of Seven Children. Age Thirty-Two. Nipomo, California.\" It was taken in March 1936 as part of the Farm Security Administration-Office of War Information Collection.\n", "\n", "The photograph features a woman with seven children, who appear to be in a state of poverty and hardship. The woman is seated, looking directly at the camera, while three of her children are standing behind her. They all seem to be dressed in ragged clothing, indicative of their impoverished condition.\n", "\n", "The historical context of this image is related to the Great Depression, which was a period of economic hardship in the United States that lasted from 1929 to 1939. During this time, many people struggled to make ends meet, and poverty was widespread. This photograph captures the plight of one such family during this difficult period.\n", "\n", "The symbolism of the image is multifaceted. The woman's direct gaze at the camera can be seen as a plea for help or an expression of desperation. The ragged clothing of the children serves as a stark reminder of the poverty and hardship experienced by many during this time.\n", "\n", "In terms of connections to the related text, it is mentioned that Florence Owens Thompson, the woman in the photograph, initially regretted having her picture taken. However, she later came to appreciate the importance of the image as a representation of the struggles faced by many during the Great Depression. The mention of Helena Zinkham suggests that she may have played a role in the creation or distribution of this photograph.\n", "\n", "Overall, this image is a powerful depiction of poverty and hardship during the Great Depression, capturing the resilience and struggles of one family amidst difficult times. \n" ] } ], "source": [ "chain = multi_modal_rag_chain(retriever)\n", "response = chain.invoke(query)\n", "print(response)" ] }, { "cell_type": "code", "execution_count": 12, "id": "ec2ea7e6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "vdms_rag_nb\n" ] } ], "source": [ "! docker kill vdms_rag_nb" ] }, { "cell_type": "markdown", "id": "fe4a98ee", "metadata": {}, "source": [] } ], "metadata": { "kernelspec": { "display_name": ".test-venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.10" } }, "nbformat": 4, "nbformat_minor": 5 }