From 27ce58f86ecb3da2098904d3bcc573d878e62c43 Mon Sep 17 00:00:00 2001 From: Bagatur <22008038+baskaryan@users.noreply.github.com> Date: Tue, 2 Jul 2024 13:54:34 -0400 Subject: [PATCH] docs: google genai standard page (#23766) Part of #22296 --- .../chat/google_generative_ai.ipynb | 458 +++++++----------- 1 file changed, 187 insertions(+), 271 deletions(-) diff --git a/docs/docs/integrations/chat/google_generative_ai.ipynb b/docs/docs/integrations/chat/google_generative_ai.ipynb index f06b250755d..47062ba8424 100644 --- a/docs/docs/integrations/chat/google_generative_ai.ipynb +++ b/docs/docs/integrations/chat/google_generative_ai.ipynb @@ -2,338 +2,257 @@ "cells": [ { "cell_type": "raw", - "id": "bf496c3b-3a09-4a59-ac9c-3c97153a3516", + "id": "afaf8039", "metadata": {}, "source": [ "---\n", "sidebar_label: Google AI\n", - "keywords: [gemini, ChatGoogleGenerativeAI, gemini-pro]\n", "---" ] }, { "cell_type": "markdown", - "id": "bb9e152f-a1dc-45df-a50c-60a8d7ecdf69", + "id": "e49f1e0d", "metadata": {}, "source": [ - "# Google AI chat models\n", + "# ChatGoogleGenerativeAI\n", "\n", - "Access Google AI's `gemini` and `gemini-vision` models, as well as other generative models through `ChatGoogleGenerativeAI` class in the [langchain-google-genai](https://pypi.org/project/langchain-google-genai/) integration package." + "This docs will help you get started with Google AI [chat models](/docs/concepts/#chat-models). For detailed documentation of all ChatGoogleGenerativeAI features and configurations head to the [API reference](https://api.python.langchain.com/en/latest/chat_models/langchain_google_genai.chat_models.ChatGoogleGenerativeAI.html).\n", + "\n", + "Google AI offers a number of different chat models. For information on the latest models, their features, context windows, etc. head to the [Google AI docs](https://ai.google.dev/gemini-api/docs/models/gemini).\n", + "\n", + ":::info Google AI vs Google Cloud Vertex AI\n", + "\n", + "Google's Gemini models are accessible through Google AI and through Google Cloud Vertex AI. Using Google AI just requires a Google account and an API key. Using Google Cloud Vertex AI requires a Google Cloud account (with term agreements and billing) but offers enterprise features like customer encription key, virtual private cloud, and more.\n", + "\n", + "To learn more about the key features of the two APIs see the [Google docs](https://cloud.google.com/vertex-ai/generative-ai/docs/migrate/migrate-google-ai#google-ai).\n", + "\n", + ":::\n", + "\n", + "## Overview\n", + "### Integration details\n", + "\n", + "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/v0.2/docs/integrations/chat/google_generativeai) | Package downloads | Package latest |\n", + "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n", + "| [ChatGoogleGenerativeAI](https://api.python.langchain.com/en/latest/chat_models/langchain_google_genai.chat_models.ChatGoogleGenerativeAI.html) | [langchain-google-genai](https://api.python.langchain.com/en/latest/google_genai_api_reference.html) | ❌ | beta | ✅ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-google-genai?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-google-genai?style=flat-square&label=%20) |\n", + "\n", + "### Model features\n", + "| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n", + "| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n", + "| ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | \n", + "\n", + "## Setup\n", + "\n", + "To access Google AI models you'll need to create a Google Acount account, get a Google AI API key, and install the `langchain-google-genai` integration package.\n", + "\n", + "### Credentials\n", + "\n", + "Head to https://ai.google.dev/gemini-api/docs/api-key to generate a Google AI API key. Once you've done this set the GOOGLE_API_KEY environment variable:" ] }, { "cell_type": "code", "execution_count": null, - "id": "f8757740-08a2-4833-8a68-c051dac506f4", - "metadata": {}, - "outputs": [], - "source": [ - "%pip install --upgrade --quiet langchain-google-genai pillow" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "c0637bf4-10dc-4f6b-adc5-6b37c9b2a8b9", + "id": "433e8d2b-9519-4b49-b2c4-7ab65b046c94", "metadata": {}, "outputs": [], "source": [ "import getpass\n", "import os\n", "\n", - "if \"GOOGLE_API_KEY\" not in os.environ:\n", - " os.environ[\"GOOGLE_API_KEY\"] = getpass.getpass(\"Provide your Google API Key\")" + "os.environ[\"GOOGLE_API_KEY\"] = getpass.getpass(\"Enter your Google AI API key: \")" ] }, { "cell_type": "markdown", - "id": "0301b16d-3391-47ef-b024-a265c71e0dd6", + "id": "72ee0c4b-9764-423a-9dbf-95129e185210", "metadata": {}, "source": [ - "## Example usage" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "6a22bc43-f640-4357-8dcf-fe20052b4f46", - "metadata": {}, - "outputs": [], - "source": [ - "from langchain_google_genai import ChatGoogleGenerativeAI" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "bafde6be-a75d-443b-8981-4b7d3258a214", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "In realms where data streams like fervent tides,\n", - "Where algorithms dance and knowledge abides,\n", - "A tale unfolds of LangChain, grand and bold,\n", - "A ballad sung in bits and bytes untold.\n", - "\n", - "Amidst the codes and circuits' hum,\n", - "A spark ignited, a vision would come.\n", - "From minds of brilliance, a tapestry formed,\n", - "A model to learn, to comprehend, to transform.\n", - "\n", - "In layers deep, its architecture wove,\n", - "A neural network, ever-growing, in love.\n", - "With language's essence, it sought to entwine,\n", - "To unlock the secrets, each word's design.\n", - "\n", - "From texts vast and varied, it feasted and learned,\n", - "Its grasp on meaning, swiftly discerned.\n", - "Context and syntax, it embraced with grace,\n", - "Unraveling stories, at an astonishing pace.\n", - "\n", - "Translations sprang forth, with seamless art,\n", - "Bridging tongues and weaving hearts apart.\n", - "From English to French, Chinese to Spanish, behold,\n", - "LangChain's prowess, like a language untold.\n", - "\n", - "It summarized texts, with insights profound,\n", - "Extracting knowledge, without a sound.\n", - "Questions it answered, with eloquence rare,\n", - "A digital sage, beyond compare.\n", - "\n", - "Yet, its journey faced trials and tribulations,\n", - "Obstacles that tested its dedication.\n", - "Biases and errors, it sought to transcend,\n", - "For fairness and accuracy, it would contend.\n", - "\n", - "With every challenge, it emerged more strong,\n", - "Adaptive and resilient, all along.\n", - "For LangChain's purpose was etched in its core,\n", - "To empower humans, forevermore.\n", - "\n", - "In classrooms and workplaces, it lent its hand,\n", - "A tireless assistant, at every demand.\n", - "It aided students, in their quests for knowledge,\n", - "And professionals thrived, with its guidance and homage.\n", - "\n", - "As years unfurled, its fame grew wide,\n", - "A testament to its unwavering stride.\n", - "Researchers and scholars, they all took heed,\n", - "Of LangChain's brilliance, a groundbreaking deed.\n", - "\n", - "And so, the ballad of LangChain resounds,\n", - "A tribute to progress, where innovation abounds.\n", - "In the annals of AI, its name shall be etched,\n", - "A pioneer, forever in our hearts sketched.\n" - ] - } - ], - "source": [ - "llm = ChatGoogleGenerativeAI(model=\"gemini-pro\")\n", - "result = llm.invoke(\"Write a ballad about LangChain\")\n", - "print(result.content)" - ] - }, - { - "cell_type": "markdown", - "id": "9e55d043-bb2f-44e3-9134-c39a1abe3a9e", - "metadata": {}, - "source": [ - "Gemini doesn't support `SystemMessage` at the moment, but it can be added to the first human message in the row. If you want such behavior, just set the `convert_system_message_to_human` to True:" + "If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:" ] }, { "cell_type": "code", "execution_count": null, - "id": "7a64b523-9710-4d15-9944-1e3cc567a52b", + "id": "a15d341e-3e26-4ca3-830b-5aab30ed66de", "metadata": {}, "outputs": [], "source": [ - "from langchain_core.messages import HumanMessage, SystemMessage\n", + "# os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass(\"Enter your LangSmith API key: \")\n", + "# os.environ[\"LANGSMITH_TRACING\"] = \"true\"" + ] + }, + { + "cell_type": "markdown", + "id": "0730d6a1-c893-4840-9817-5e5251676d5d", + "metadata": {}, + "source": [ + "### Installation\n", "\n", - "model = ChatGoogleGenerativeAI(model=\"gemini-pro\", convert_system_message_to_human=True)\n", - "model(\n", - " [\n", - " SystemMessage(content=\"Answer only yes or no.\"),\n", - " HumanMessage(content=\"Is apple a fruit?\"),\n", - " ]\n", + "The LangChain Google AI integration lives in the `langchain-google-genai` package:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "652d6238-1f87-422a-b135-f5abbb8652fc", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -qU langchain-google-genai" + ] + }, + { + "cell_type": "markdown", + "id": "a38cde65-254d-4219-a441-068766c0d4b5", + "metadata": {}, + "source": [ + "## Instantiation\n", + "\n", + "Now we can instantiate our model object and generate chat completions:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "cb09c344-1836-4e0c-acf8-11d13ac1dbae", + "metadata": {}, + "outputs": [], + "source": [ + "from langchain_google_genai import ChatGoogleGenerativeAI\n", + "\n", + "llm = ChatGoogleGenerativeAI(\n", + " model=\"gemini-1.5-pro\",\n", + " temperature=0,\n", + " max_tokens=None,\n", + " timeout=None,\n", + " max_retries=2,\n", + " # other params...\n", ")" ] }, { "cell_type": "markdown", - "id": "40773fac-b24d-476d-91c8-2da8fed99b53", + "id": "2b4f3e15", "metadata": {}, "source": [ - "## Streaming and Batching\n", + "## Invocation" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "62e0dbc3", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "text/plain": [ + "AIMessage(content=\"J'adore programmer. \\n\", response_metadata={'prompt_feedback': {'block_reason': 0, 'safety_ratings': []}, 'finish_reason': 'STOP', 'safety_ratings': [{'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HATE_SPEECH', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HARASSMENT', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_DANGEROUS_CONTENT', 'probability': 'NEGLIGIBLE', 'blocked': False}]}, id='run-eef5b138-1da6-4226-9cfe-ab9073ddd77e-0', usage_metadata={'input_tokens': 21, 'output_tokens': 5, 'total_tokens': 26})" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "messages = [\n", + " (\n", + " \"system\",\n", + " \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n", + " ),\n", + " (\"human\", \"I love programming.\"),\n", + "]\n", + "ai_msg = llm.invoke(messages)\n", + "ai_msg" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "d86145b3-bfef-46e8-b227-4dda5c9c2705", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "J'adore programmer. \n", + "\n" + ] + } + ], + "source": [ + "print(ai_msg.content)" + ] + }, + { + "cell_type": "markdown", + "id": "18e2bfc0-7e78-4528-a73f-499ac150dca8", + "metadata": {}, + "source": [ + "## Chaining\n", "\n", - "`ChatGoogleGenerativeAI` natively supports streaming and batching. Below is an example." + "We can [chain](/docs/how_to/sequence/) our model with a prompt template like so:" ] }, { "cell_type": "code", "execution_count": 4, - "id": "ce0bd90e-2afd-4189-a9d2-278c1f10ffd5", + "id": "e197d1d7-a070-4c96-9f8a-a0e86d046e0b", "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "There once was an AI named Bert,\n", - "Whose language skills were quite expert.\n", - "---\n", - "\n", - "With a vast dataset,\n", - "It could chat, even bet,\n", - "And write limericks, for what it's worth.\n", - "---\n" - ] + "data": { + "text/plain": [ + "AIMessage(content='Ich liebe das Programmieren. \\n', response_metadata={'prompt_feedback': {'block_reason': 0, 'safety_ratings': []}, 'finish_reason': 'STOP', 'safety_ratings': [{'category': 'HARM_CATEGORY_SEXUALLY_EXPLICIT', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HATE_SPEECH', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_HARASSMENT', 'probability': 'NEGLIGIBLE', 'blocked': False}, {'category': 'HARM_CATEGORY_DANGEROUS_CONTENT', 'probability': 'NEGLIGIBLE', 'blocked': False}]}, id='run-fbb35f30-4937-4a81-ae68-f7cb35721a0c-0', usage_metadata={'input_tokens': 16, 'output_tokens': 7, 'total_tokens': 23})" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "for chunk in llm.stream(\"Write a limerick about LLMs.\"):\n", - " print(chunk.content)\n", - " print(\"---\")\n", - "# Note that each chunk may contain more than one \"token\"" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "a147766a-0051-4127-8db6-62c070dd7866", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "4\n", - "8\n" - ] - } - ], - "source": [ - "results = llm.batch(\n", + "from langchain_core.prompts import ChatPromptTemplate\n", + "\n", + "prompt = ChatPromptTemplate.from_messages(\n", " [\n", - " \"What's 2+2?\",\n", - " \"What's 3+5?\",\n", + " (\n", + " \"system\",\n", + " \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n", + " ),\n", + " (\"human\", \"{input}\"),\n", " ]\n", ")\n", - "for res in results:\n", - " print(res.content)" + "\n", + "chain = prompt | llm\n", + "chain.invoke(\n", + " {\n", + " \"input_language\": \"English\",\n", + " \"output_language\": \"German\",\n", + " \"input\": \"I love programming.\",\n", + " }\n", + ")" ] }, { "cell_type": "markdown", - "id": "5a6cca7d-9ff0-4a40-b45c-651ec8cc5012", + "id": "d1ee55bc-ffc8-4cfa-801c-993953a08cfd", "metadata": {}, "source": [ - "## Multimodal support\n", - "\n", - "To provide an image, pass a human message with contents of type `List[dict]`, where each dict contains either an image value (type of `image_url`) or a text (type of `text`) value.\n", - "The value of `image_url` can be any of the following:\n", - "\n", - "- A public image URL\n", - "- An accessible gcs file (e.g., \"gcs://path/to/file.png\")\n", - "- A local file path\n", - "- A base64 encoded image (e.g., `data:image/png;base64,abcd124`)\n", - "- A PIL image\n" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "06c59be4-c4c8-490c-b0c6-ebffa4e3a034", - "metadata": {}, - "outputs": [ - { - "data": { - "image/jpeg": "/9j/4QDeRXhpZgAASUkqAAgAAAAGABIBAwABAAAAAQAAABoBBQABAAAAVgAAABsBBQABAAAAXgAAACgBAwABAAAAAgAAABMCAwABAAAAAQAAAGmHBAABAAAAZgAAAAAAAABIAAAAAQAAAEgAAAABAAAABwAAkAcABAAAADAyMTABkQcABAAAAAECAwCGkgcAFgAAAMAAAAAAoAcABAAAADAxMDABoAMAAQAAAP//AAACoAQAAQAAACwBAAADoAQAAQAAACwBAAAAAAAAQVNDSUkAAABQaWNzdW0gSUQ6IDg2Nv/bAEMACAYGBwYFCAcHBwkJCAoMFA0MCwsMGRITDxQdGh8eHRocHCAkLicgIiwjHBwoNyksMDE0NDQfJzk9ODI8LjM0Mv/bAEMBCQkJDAsMGA0NGDIhHCEyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMv/CABEIASwBLAMBIgACEQEDEQH/xAAbAAADAQEBAQEAAAAAAAAAAAAAAQIDBAUGB//EABgBAQEBAQEAAAAAAAAAAAAAAAABAgME/9oADAMBAAIQAxAAAAH7BKOfonKlqBpocs9WBlNqyNpEvTPTOqyuJVnqqwqixzqGWiksgDLSU5505t52nObHeasjO51hArPp8s583p0IdW5CpYRT1MX0VLzabTCy6IOeruuZdWRmUkiN86yWsWZyRZ0YvKyc6y3glq5JqaQCeuI8/od50uu2GhcURNyGt4ku6zUbvkK2iMq3vl0TXF6mVxnW8cs2dWXO7Iw7a1nz59DC5556eeyUzUlXKeqJef01UUVplR0PFmgVCNKlwW6XK2kcJ0VKSMpz1momdSqxWs3pgHWcjmnBncVm5sSa1lJo9Ys83qh0xFBLpi0hmt8wdBgS7GKubjKdRw5uZQtQQrBAkqlYk1Yk0IaqVSSRh7ha8vrzNGZO4BNVUWkzNGQU9ZmbVmK0zpS5uUmkSasQ1Yk1SVJEmhKlSTSJNH0q2nx+zKdJJnYMq2aZxUUs2aiJjWLnF1eYrFNpIWiqCiyFasgpWSUiVSSRqpVJmVSr6uWeL1wqiagGMkqFqHPPSq4o7Zs5b6OO51nN7yTpVmQ5skpmZvUcum1y8k9115x6Vp5t9nGZ4OevPMpM/UTb8fryNKWHpTHOul1yz1zLxnWl5se1S+Yeo7PLv0ivLXphx3rnTT2MjomzOLixKuexmfNvLw0neMy5szVpPrDOfL22M1Wxzh0LJlxM506xzXpyUq89czBwt5125rN4yg7eeNKrPS04726jz/ifvfncuL6H5Tp7cvo8nWmU7SYztKfRTyz5+/YcarpOXQ7DlpNZVyxG6XPHsUc2jg1fPVmudSDAdRVm5hpI4fmS+d5WS7c3zuOnPr+m+C+kl9mdFN5zrJ2FHLrKsM3VpktJIKViGgTQ3IrciaQhSpoE2SrEj5v3fz+59OrOvLn1BOW86T7PXz/S594Vo5e/8/u5/Qp+M9bO/dMN86EykMEmElBKtElBBaENgDR5beefJx6PLvlvLneMs9sRZnOnV9D8xtX1fZ8D72OngPN9eOjycu3o+S5frvW/PNMdP0E+U9HG/aPOqa7jl2NEMQwQ0oDRPVy5fMfQ/GXOV0duFxnnZuYKNInhO6cL1l83XBDkqiQsgNHm4tJmmnMpes46Xr14bl9Ts8By/V9vw9TX6BXwnoZ39dj4T5dMfFz8/tw7I4nvn6VKdTTh3Z5z7Msajp5dbnS8q3mEAAAAoAMRFOA0UMBskEU4DSsGvRfI17q8+o7K4XXZHOzcwE157JbgZnO5c84ixiBoAAUAgAAAYgbkKQCVBJSExDcMogLJCyAsgLIBjNRFEIoILCCwgsILFgoJKQhkSMEMAQMQCZYhqAEoAABsI1GSyiCLIC3mGhmLoZo1M2WQFEgwEStEqkSNAmCAAAQAgD//xAAoEAACAgEDAwUAAwEBAAAAAAAAAQIREgMQEwQgIQUiMDFAFCNBUHD/2gAIAQEAAQUC2e9dyZYr2ZQ47YtigNCdH2qPscShlmTG9vA+2xvZFjHvRQiyhiHtbFbMTHbItljaGvici97LL3rfES2a3orbyux7ZH2NIkX33tfekUu17KihrZIox2Y7KZ9LMcr+Vd99je2Re31smeColpDkiT+Zd1lmQ5GRZ5PPZ7a8DZZZZ9nHE44mCHGh/T+Nb1vRRRWzEMY5GRZZkWI8FolZ5+WyxMsSK2rtyHPw2WX22WWN/gW1ll72ZDmN/kortrtsssssf/drau6iu5/mooreijA8I+x9lll/rooorssssv8AU90xvsYyhIxK7KH3JjMTExOJnCNRiOu57VvR4PBcUZxPaScayMtvBa7KMWcbMGYMwOI4zBIclElqt/BQkUYlGJiYHGYpErMGzjOM4zjZxMWgjhiYJGK2psx2VsknSjM+iaK7aKKKKKKK7KKMTExPG3g8DmkchlYkUjIc0cqOWKJdTDKU3bbfw18WRkzNjlIzZysyRlEygZIzHJt/5kN21ptvU08IareXTdRiLFx/ztszL2dmTMjztkOzNGcTkQpJkpLdKzHarMWYyvBi02cXlJQWrP2ak4Ia9+l1D0XGSnDt8FozMkOQvO3kxKKkzjRgjAUWxacTGJ4PA1E9pkkJsoW1E5KC6nXzdHgbVdJ1PGffbkX2ZUcjOU5EZRMkZRMkWhyMzkOQzMhTHJnIcqFqGaM6Ou1I8ObtzY2SNJSR0XUW/wASo8FiY5X8Fs67WucYpJVdxlJ/SpSdp6Us9L4KPH5NfVWjpamrJy0743HxjWzrFe5dFqZaO8fU9KRDUhqfo1J8enqS1et1300IzG6E8iVsjCibo0tbGcfUdJmj1WnrvazT6zW0zT9SiyGvp6v5erip6c56UI+E9nJIep5m7J2VipJSXmD0vUVxllllml1utpml6hpTIyjNfh6ubeu1MgsUMqi1VE/CysypTmqxi+6yyOpKD0/UtSJD1GEj+bon8zpz+RosU4P4eOzjOonxdO9VSlk9nNIzSbkZo5lerOxMtjSkVJfFYptHImexmJ70LqNaJH1DXRH1NkfUdFkeo0Z7I5FAl1emj1DrMtBz8cpzkakNJnipqjxGTuUYwYn4+i/gsvtsyORmZki4ngjOcSPW68RdapLm0GdU3qSzolTIaeTXhWWWXFk5Us5aqSpeH8t7V2XvZZmzkYtZmaZ/Uf1n9ZhpmEDCJxpEtDKS06JabMJHn9VllllllllllmRl++y//Rf/xAAgEQADAAIDAAMBAQAAAAAAAAAAAREQEgIgMAMhMVBA/9oACAEDAQE/Aey8bh9540b/AMFGzVs1H6tlKUo2XM70pf4TH0nvBohCGpqamhoOIbxUU2NjY2KiouaUpTYf3mMhqQSILFPvpB8UfmZ2pfB8hieJ7cn0X5hfIxc0XzfRcmhc8rkL5DZFRe3P6WL5UpTY3FzRtxPkdzcP1pSl6T+1/8QAIBEAAwABBQEAAwAAAAAAAAAAAAEREgIQICEwQBNQYP/aAAgBAgEBPwHix+M2XOl8IL4IJFRfaE2hBInjCE+Nfo6UTEylMjIyMjMVYt4QhCEO9u+EEmYi680iIi2pkZOi4QhNpvOUol8SXB7fjHp9NK4QeneD0GLITjTR2+K5whDEwHoZizQpvCC9p/Af/8QAMRAAAgAEAwYEBgIDAAAAAAAAAAECESExEBJBICIwMkBRAzNhcRNCUIGRsSOAYGKC/9oACAEBAAY/AvqdCv1a30Kn+F3xv/VWxbprdJp0dOFYtsWlhVl+t128raTZp1Vdq+w3OxP4uZ+xJvd7diacyUugoV4Nim3MsmTRK8PYUUNuFcvw7FixY5TlOXZnE5GWFvJLY3uV9DfZtwb4ZPmboOUh1RWIZLQ+DF/z0dSmNuDJfLSY9TQlg4ZiadSGJ6rrs2uhFN1Fm7F3jm1KmR80P62N5OE3I4YvZ9RFG9EPL+7D3s0sJ4ZSo3DcUSo1ob8LgGob9nsc812Z/JBL1RuxrpYYIuRuo4PCVHzPvsXKQ4KVCeCaYsym/fa5sy/2N/cZOFpr06JpxfZDPUmySK4zFSh3wrwJwxNG+lEVha9mXa+x5iPNh/JSOH88OOJaD7li5I/RPQrYkiQkUwo+HcrDhRo1/J5kaOafuje8NfZlVFCU8RY8xWIywUm8JueE2VJEihNozfvo74WwubsbX3KtREovDl7HLX1wlPCtiS2JIkSSkir6ysMJ5aOU1NS7OYuTcZphYs/6F//EACcQAAMAAgICAQQDAQEBAAAAAAABESExEEFRYSAwQHGBkaGx0cHw/9oACAEBAAE/ISc0h+hB8NEJLg0YMzIz8Ga2QFdcNvfFBsT6MmOeQ38isUbF0F9oucEjLFVx8WacQRwj4kLYn6FCOCN6JCR3R5YGnZJplLBJb4sd5PePDK4VmEVQfxHXF5WGQXCMVECY41jZE1s28jYVJoY23sf4G8kPsk7PwEkhfoboYKQvyZvC8QRCcCCLxxBaFMLowaI6Qy6Glop2NfI8iDIRh5y7B+g76BspRMTIgxiExUTGNCY3BJVBZQjyOPRsJ+xHY6H4OM2dK4lIxprr434JjCfEpJxQo5FlMaDgSbYoRXY6D8EQ7H78SjeSJtQVMj2jR4OsTsxPopiZkMl4T4CChGfIwnoT0YbIDYYYnwOh0xNGNwyPBX24NN7Y55+dELhT8KQQRDQiQ9GTXFD4LLD5pgZGA1lG78oQgl8lEclvg37KYIaLgaN/J/XIQhBCGvmDcbhj+zhCE50U6KX4se+KN/ZzhODR38GEYuIThB/cRk4Zi8iBK9DTC5o+F0NjZfqz5sZCKc0SDRIeNcVDgYZQb4hCIaXxnE+m0SiTh+hi4Hkg0YLGSmPAqMYjg2b+JCEEJHLUFPoop9CYaD7Ko7PCnE+GZCiEIXBlhbcHyjYjgoyii0Q9A3eIV4E7o9J6BFYvIikORQMQmv0NtsfNL8Cw+WzRDy+F4BZmlifwL0HTBbZ3GQ9ngcSTXCYj9JsW4Xo8jnQhrvT9lXgYnD4vmQXwBrhrhmSfkQRCL0NLtml64LLoptfoYc4IDVYOwdJLaLoiFSYvGno2D4hCEIQhBhFRSlKNs32IaLQyYpd/wPzGGCZvRTICS6JZjLPDizaNF0mNwGRQxCwMzzMtN/AvZ9k2I0x6Kk6bdkGhohBuj8S2R9kHgXYj0MhuE9ndgQeREdccYuossqmlyr6aGphi2wEl4QmZFiHKhTTKIFZHp+gwwx6Qp6qe8HdA8t/4PyrZRCEIQbjI1EHsJaDt84+BRaQ60OtslaVY/ESF3D8I2uobQS6pLofu/RGUxOgU0XgTYYbyim24INPskhYnsezGTUQ6WG0Pav8AkqNZSHItrVLoURNOpq0hCEGzKKUouhI9AvQTdidpnvMwTexr7I6Sf7L9Gwx2iWjFUT2ISG3kwiW/Anho2RvMnQn47PWfnwPOEeO6Rm70tkxI3S2y8k6IoO+X+ExxCcz6mTIqO/2GzIbEBX5EZ56FUeRmL4nv2Ztmu2hYzbMg3CrWcC1kSzKbf+GIb/A7flt8TiPwQ7GxJGtJ4saXXM+neYQanj0S7Y7sx5/Ijd4JhJE8MhrR6laS2KzLY1kbQjlyOsqsnwfydtVH/rIDU3zX9GcQn0EJFVhxK2+1opsN6LyJvUwKRsJfV/RZolevBOLqH6Dq/wAsNH+S2hwTS/ma88Jow7IEb8iYT+Q/0bqPx3/H2iHCNUv6S/6Q7Py79Du584L1+ym6GiyeSpx4eKT2IvBfZezwvs1maHIasXAvIhif0ZyIiV/yI9gqb7CivQ4G0tMwPJEkm9eRib/wSuufWipoI2P9mZr8F1LDS8jJ7TGy9BzkpiSw0ILVUzYq+ilKUTFwfqyWYxD5eGNp+4BRz+e4nf8AJoWqLaD8FR6d+aTfQnS3+Rvna9i86adfkY0bbtoqlgjLz/Q3Gvv0NdnL7CXGaXkhKeiTRgwzsupYotwqa1noaHU10LF/NHiNS/BeKUpeMmeGmHQmXyL+z/4oLrn8DR/sM2KfvHbP6D/BCm4r8NxnVWV6HS2xoV0jZqOoi1n1kikTftCSuCYFnMeoYKL7J8aG9BSD0lSrDrwVZu2/JWYf9GsxM3/BSsKjb0WBY418LxS8EKiDTK0J0J120IXuk/0V2aeUf/LH9/GDFIfaKjJ7UeQmBY254WBZVlTQzJXP9LGSMK0InJuTdiqjkUplj/eImyVCw8mPp3haMRlLwpeCcSO2JfbEex76/KNrn9GS6fskRf2PE/5cB7gtL+glFH6F6MN6j/ZNbEX/AIfWpTBCcUpSlLyI154X8fsryX5/v7ClLxPhSlKUpeSlKUpfuIT6FKUpS8ziEIQhCEIT6VKX7OEIQhCEIQhCEIQhPssGClRSoqKiopSlRgwYIQhCfeX6b+t//9oADAMBAAIAAwAAABB4hz44McsfWByvubxlUx4T21GzG/mwPwppGB1SynzV8/FMvTNNE3qRY8gAyCTi8+x0q8PCGETdrXy1EC6pTKfmvgBr92sDCoXAJhHgxrT6IMMzU4AT7hQ6cthkJnMZLy2D6zHFmrOSLjhMB6C3ftmVqCf2R8kOl8+D5x4CiMiSjqdXPFqd7/WcZpO54K3DaN0lFXAJa+f90OsrnVwBVCspUV9ykChgOrie/a8t7ZAAjDCDThFLyOrJhwkoGe84pkEQEYu5krMyCz32s89r7s4Q8n0yBLdxPPLFd+ut/OiyoXiGF7c89vLPMN/df/eOt/c/PvXPPf8AxwNNDrzjCRjDjDBZDDjCHDDmjDPvTDDDp//EAB4RAAMBAAMBAQEBAAAAAAAAAAABERAgITFBMEBR/9oACAEDAQE/EGMSo0MXvExYxuPFWmLYiBJa8YxtjSdlPC8EUuzGVlKQfFDA5EnmzYJcKMhssajouoQmUuH0wxvGUvCZCEH0N430N49fG0hDwpTEeMg4OhwhNpRCHsHREbDYr4UyxMQQIVPMQNUPsIp+CpKmeDDDDQ6eDcBf3HXB9CFjalDb+lpGVSTsbXjgh0UYnCs18GzujbSo5vHzIQTHnZ2dlZ2VlykVBt72EZ4mMCR+fk0P8y55pf0pRi8PgxO0Xj5HjxBiZeC6Lxf+iZCdH1CBjnUrKQJ3Nyi4XKUpRWIOhRDjI4L+1/u/y//EAB8RAAMBAAIDAQEBAAAAAAAAAAABERAgITAxQUBRYf/aAAgBAgEBPxARYJi8IvYlesTSU4UelLiEJITXoRrkyY12QgkQglj+pIVid2lKXad4kxBBLWAoyFLwmiCWJCJk4UomUohEEIJCXhTKUQohNFQuiiYnifJ6i8AQQJPo0RI86GJGveSiCHccaUJsoVZBKiaYP9cRYx7GIGwXyEkiENEL1Cn3twgxGMQhRMjIyDZhaQiP2UpRMonToiKiIiEkdYjsokt6PGq9DUSe/F8Ge+DVkPRMav2fVDQUVxgrCD6KduyZCbBBH8IOx8g/uSiokOhx7zPHCIgSSIUabFUV8X53+R/i/8QAKBABAAICAQQCAgIDAQEAAAAAAQARITFBEFFhcYGRIKGx0TDB4UDw/9oACAEBAAE/EKuMGic9BYWOmbRamSYMwkQoxEBEHukXcg8VFs3ENLh4hBGRiUKLAxQGp5S+cF8Qjlupd1Qg1byRVbfhlrJ9yxdyRTmxHlkC4TqtEcmhALgBakF0EQjnHqG2IhcQZnmDvLA+FxRXpghaZVguVdjXuNdkWr5hjGINtDxBcuuWG3H1EDRKWWWIeML828xKrWg0tmfEmbRUEauyL3o6Cx2+I6jGBphKLbCjLwRFSKyWWBWYlNSzFR8lSdz+o3wiOLPqNdLHiNwO/MXoxxyAlXUcqw9zvyKmBpBhwijix7gLpy71Li67QVz4QHBEbrwjaXHUdR6XMfOEAYAkUQYjhDLcFMsEaH3CJM4IwDCD7JQZEDswKKhLDJJwiQVVscQXKIWLjRzS+SF3S9gmtMPClbrKTLjCGVmo5lSo6jqMqJe5eGeuhdDslBwS8tcWLugSzBuK7Zg9oOyRllFQvNxShK9wbCImvJiKN1ct7ukdwfUG2G+9x1giKKXCyHzFeZbSINVNoo66Jia3KR6EToYah0GAhMGIu0KtkBbtuK6Z3SxZYTURl2pogGxHzLitDwwbBEr4YP8ARKxLHmWTl8QbZ+EK4ycxItXxGmjLzNbFhYSpQ+SZsRJVyu8QuX0NQlXRZtMOZeFEAES+JSOTA7TIcwyuOFLlVVQhB3QGqvEtNGIsoKg8wV7ygpGO0vXAs/zM2AgCB9Swtr3Bv7Ucu+EOLRjFjGK94LvEzaCkqbj5wkzEKEQXED8JQLYY0dy0Wkw5Co0FLmdzcsN9NjNEvN9HYmTLBnDUQysXy2dNcXMdRjH8AVUubZUVMOUfV4lLJKNXcWK943BmYnERTF5i2xj10jqPRW9x3Hq6jqPTLpxhhNOkIioG0u4oQxnt+AHXLBsmkFxKY7jvo66OoxjuO5z1ddGMIeoqGcEcodBpqKWC3qZldLhOSXOXQWzpz0qOomOjEz05lSo66MZeK7SsBUHiasRLJWYDWor4nLEvcqAqNIltQTMpF3FilS+rKjqPRMxG5TfSmpUdR1HfQxSDgjclO0KvcHekrepSzUG+CPeVi5ToNFjrMAgRtw9HpUpjHXROiSsy3RjqMYlcQXxAzaUUhZ2QDgirtNsvPeOlLcU5Y0VO+l5ePuL3lhmdiUs84m9w5pT8yifEvxEJlEp4+4pdVEzKlZnxKYjecROLzEiRqBCyoXvFvmPum4Zbgl1FkW/HeIwLQJyd53UvZKIIgZ9QG0YwGYvYCBfFwo5zKcBKlS0VU9FjcQ+y5mr4UFbRO9Q1ui65K/EpCL3NT0TMtGyN1mXXuJVwbo6V5dxDIWCGnMqDzxFro9S8CzzACUnaPSh8RuUc3LXZBbsIRS6iQ0BZdslViy5d4X1BtI9zRTRmteYDwIzZAMBEcrED2IWgPll4Mu1y8CEi4KvuCI3MxlTN4i2YtSkwXPBL8XLdp45ku2XLZXwX8QUoAvEbyxniZ3Q+IpyYlBqDsoLfDhMGlWHIpQUVwrOiYrU1zFWEjxKYADt3A6vwS/e1ZzALmHC4FqLlTmGUQ7QFxIM5jU1KDqWyhmkC7lSUaJWclSxiF3UStzvVAnDodp9zmEeoBZ/M4leUxha+CF1SbSPCI4fkzZIlFjRPMpcqrxG4LviXLjHVl99HzKoQsjKWZe8Rdr9xtvMcIww5SvELzJmFlTtddyjArzLb3lADbn3MVtLrQweq+pkFsNWk+4DQnwTOuu0rC2ogatWNdK4lruamzp9RRXQ94ZQmUYVxUOCHm4K4KIIovmuIt2PRHyePEw/25C9y+IK8u6GyBhP2Ry6g9Fi4K6tHWEhsy+YrgK7XO1vUq1kvaDswSpsfbNwh9RI0O7KgsebiXCfBFUGOV3NPj2RnR2Kl2BXyEXHP1HWEgA3HmW7C+Zh1e9OoLEv3EJt2ZVtp8XMUA7sbkGco5lELeZRwLU2FRiWm0FPmIckaYxmhvEcvPY/zKEPsdnuMOEc9RzcRynkEp+8FJZoSzkP3LFdJBWCcRFyab7QLR8KGbVsANtYSwB6uMZ0eqmwT8ztX2RK0EeBBxEPiAwzw0pUoF0YtzAIzvEQVHljXCOwSkEoiXLHYlDYZjfT4rb2jfCwDnm7mcGrJ5hAqPaNEDN1qzzKZq1nXOf8AkRWEA0jpiKm8YcoQAs8syysaam1tz+toBsRQz+87J6Uo4bfMr0EDaOOYJrDNygwqA7XB73O4wGCVOMolEFJG2AuKKge5QqFuZ2Oe0yih3Y5VFhUVDzG6ArOrZ+FwoURZVq245kHZYd5Rx+ZR4QWgBReas2PEBTy9xuYYDZa/37kyse5UYc4FyplKzURvmUx7WdKzKe8bOZxiCnLFXlgGR8E3Q+ksLA9FS1gJDgAV5jMm9wMtvtgV4fEzd5uNtHiJLMPcjRSJnfqMBQxO8jfiVoUc4R4PEpiFMKsDz2mAPkQ78LGvAszevbDabIHtjDlZK5dyJcNRd0z/AKieIwmdRJyiY7lVaIlcROz9wLkxKbtPNyx2RMSo2iSpWZSMr/64nSuhL1UC5XUI5mWB8VyxU1JpvvzBhWqIeeYuqKhqYXdaot37hF0lkNfuY21ZLP1Ha9ADDfDFDygCnAprklYNRiSqF3GG/JmZm20B+jmKkwfMruJ7lSjSxy25Y/g3cpldFPQxTKYGJUMQLhmEsWZdXXEJExcAzRl1ELQouHkr7vEA2AO0QjAXAiNBlaEw67k8qQ76Xst2TKJIHCv9y/cA4vNPOOIIVWmWfWYnM8UnAH9S9Vn3iWLwPeFgisyNxenvB/cIEdy0gouXNr7JrdF660SiPWvxojAlHQgmeUhUZlX5BDIZx+r/AIK1C0URSCleiXcr1y1E2NY4YsVC8csQvG98H3FUFAVwQ19gzsPf8xsQjSox5lh5BYx8S77W7Df1r3N/XM75O/nmEFIY9ESsUe4wUPoT4dzyih/05glY4CH9fMdfg76uv8FJZe7xcFD1YweTvN6I40P6+IxjwBte3ctQHsk1dqpmUW0cq4+JZRkFFMXBmgVWLL9EbfBQXfqUqJ5LgliCnh8S7gWlyMAWpdjUOgYfgAyghUN3RKLnH+QmaE0gv3KJuOP5BcZoD3/qR6vnUf2Qf6Mi0no3KTY/g9EgE32h0UbgwmiC/wADwc8J7PMXq41BjjviYFUvFQIChvcPGlsq5QCSkrb7QQjaXMsTTlZ5lk6mHCvLDyjXe1wPVlbwSOWBcKXT2iXLoRlLYYfgLJWw/Eal5ljhlOKnZYOz3cwzgvjWhDkw7XGw6e9/cdMZ5v5JTg3YU/MPR3hb9NSqHnnB+v8AcrR5/wAAwL+QZEWXZjBLM+EC5ZAFaCNVVcimsi+IVl2uQWZVliXiNcUCzYfMoAvuMMoAGVIaiLblK8wYHIwmWb9ziFHbfeXwV8lKiy8Fw1WeEdaeB2grl5ZzLgm+GYNS5fRb1bVLcMsleHorG2dkmg9BgMWfeZd+2g4h5IIgW1n4YhNIcufpx+oZ3VCt+TMM8uNb6aYeKAqwgx4KW4QF4ajcECnfczMKFO8ZooVel8ShlFQRsQq6og3nXmNhRVcsow3K61MjqNo0/tiIoNFBv2s43dgVioNtflUWzs9S7l1Lly5cuXBlsvBygyQHTEJaWOYKXYGDNNQWR+ZoPum4+ZuYBCXdIT9JbKGF7otwr0R/EzAA43nIWKjH3xUaOzkGPLWu6JcWCDY/9zbpW9VUoSteikeCvCMcJb5wajn8zEuXL8y/M9oPljZtIlcRUtJaWgq30+0KS1VBnMTuy7lL955JaX8/ct5+4PUxVcC/pPK+35XLly5cuXLly5aWhwZREMSmI3My0tLwt0Ff8U/uXLly/wDFcuXBqCSyWTEqU6EagJ0WWw1LPwJD0SUdFf40AplP4rmWy2W79NJhlEQlE1+CY/ElSoa6GX4B/IFx10PUOX/kDDMuKdCnMLPwApx0U/AEUpxG7Meo66Mq5VR3+Drq9LlxcwcR3+Fy5cuXLXuXLly5cGXL6iJiOo9XfV11en//2Q==", - "text/plain": [ - "" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import requests\n", - "from IPython.display import Image\n", - "\n", - "image_url = \"https://picsum.photos/seed/picsum/300/300\"\n", - "content = requests.get(image_url).content\n", - "Image(content)" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "651d614e-1398-475d-9594-2eb441605d4d", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "AIMessage(content=' The image contains a snow-capped mountain peak.')" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "from langchain_core.messages import HumanMessage\n", - "from langchain_google_genai import ChatGoogleGenerativeAI\n", - "\n", - "llm = ChatGoogleGenerativeAI(model=\"gemini-pro-vision\")\n", - "# example\n", - "message = HumanMessage(\n", - " content=[\n", - " {\n", - " \"type\": \"text\",\n", - " \"text\": \"What's in this image?\",\n", - " }, # You can optionally provide text parts\n", - " {\"type\": \"image_url\", \"image_url\": image_url},\n", - " ]\n", - ")\n", - "llm.invoke([message])" - ] - }, - { - "cell_type": "markdown", - "id": "eb2641f3-5dae-4756-b1c1-b58b41edc344", - "metadata": {}, - "source": [ - "## Gemini Prompting FAQs\n", - "\n", - "As of the time this doc was written (2023/12/12), Gemini has some restrictions on the types and structure of prompts it accepts. Specifically:\n", - "\n", - "1. When providing multimodal (image) inputs, you are restricted to at most 1 message of \"human\" (user) type. You cannot pass multiple messages (though the single human message may have multiple content entries)\n", - "2. System messages are not accepted.\n", - "3. For regular chat conversations, messages must follow the human/ai/human/ai alternating pattern. You may not provide 2 AI or human messages in sequence.\n", - "4. Message may be blocked if they violate the safety checks of the LLM. In this case, the model will return an empty response." - ] - }, - { - "cell_type": "markdown", - "id": "54793b9e", - "metadata": {}, - "source": [ - "### Safety Settings\n", + "## Safety Settings\n", "\n", "Gemini models have default safety settings that can be overridden. If you are receiving lots of \"Safety Warnings\" from your models, you can try tweaking the `safety_settings` attribute of the model. For example, to turn off safety blocking for dangerous content, you can construct your LLM as follows:" ] }, { "cell_type": "code", - "execution_count": null, - "id": "75fdfad6", + "execution_count": 14, + "id": "238b2f96-e573-4fac-bbf2-7e52ad926833", "metadata": {}, "outputs": [], "source": [ @@ -344,7 +263,7 @@ ")\n", "\n", "llm = ChatGoogleGenerativeAI(\n", - " model=\"gemini-pro\",\n", + " model=\"gemini-1.5-pro\",\n", " safety_settings={\n", " HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,\n", " },\n", @@ -353,7 +272,7 @@ }, { "cell_type": "markdown", - "id": "e68e203d", + "id": "5805d40c-deb8-4924-8e72-a294a0482fc9", "metadata": {}, "source": [ "For an enumeration of the categories and thresholds available, see Google's [safety setting types](https://ai.google.dev/api/python/google/generativeai/types/SafetySettingDict)." @@ -361,23 +280,20 @@ }, { "cell_type": "markdown", - "id": "92b5aca5", + "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3", "metadata": {}, "source": [ - "## Additional Configuration\n", + "## API reference\n", "\n", - "You can pass the following parameters to ChatGoogleGenerativeAI in order to customize the SDK's behavior:\n", - "\n", - "- `client_options`: [Client Options](https://googleapis.dev/python/google-api-core/latest/client_options.html#module-google.api_core.client_options) to pass to the Google API Client, such as a custom `client_options[\"api_endpoint\"]`\n", - "- `transport`: The transport method to use, such as `rest`, `grpc`, or `grpc_asyncio`." + "For detailed documentation of all ChatGoogleGenerativeAI features and configurations head to the API reference: https://api.python.langchain.com/en/latest/chat_models/langchain_google_genai.chat_models.ChatGoogleGenerativeAI.html" ] } ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "poetry-venv-2", "language": "python", - "name": "python3" + "name": "poetry-venv-2" }, "language_info": { "codemirror_mode": { @@ -389,7 +305,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.4" + "version": "3.11.9" } }, "nbformat": 4,