mirror of
https://github.com/hwchase17/langchain.git
synced 2025-05-18 13:31:36 +00:00
f2b2d59e82
<!-- Thank you for contributing to LangChain! Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified. Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes if applicable, - **Dependencies:** any dependencies required for this change, - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/ If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. -->
367 lines
22 KiB
Plaintext
367 lines
22 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "raw",
|
|
"id": "bf496c3b-3a09-4a59-ac9c-3c97153a3516",
|
|
"metadata": {},
|
|
"source": [
|
|
"---\n",
|
|
"sidebar_label: Google AI\n",
|
|
"keywords: [gemini, ChatGoogleGenerativeAI, gemini-pro]\n",
|
|
"---"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bb9e152f-a1dc-45df-a50c-60a8d7ecdf69",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Google AI chat models\n",
|
|
"\n",
|
|
"Access Google AI's `gemini` and `gemini-vision` models, as well as other generative models through `ChatGoogleGenerativeAI` class in the [langchain-google-genai](https://pypi.org/project/langchain-google-genai/) integration package."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f8757740-08a2-4833-8a68-c051dac506f4",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install --upgrade --quiet langchain-google-genai pillow"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "c0637bf4-10dc-4f6b-adc5-6b37c9b2a8b9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import getpass\n",
|
|
"import os\n",
|
|
"\n",
|
|
"if \"GOOGLE_API_KEY\" not in os.environ:\n",
|
|
" os.environ[\"GOOGLE_API_KEY\"] = getpass.getpass(\"Provide your Google API Key\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0301b16d-3391-47ef-b024-a265c71e0dd6",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Example usage"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "6a22bc43-f640-4357-8dcf-fe20052b4f46",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain_google_genai import ChatGoogleGenerativeAI"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "bafde6be-a75d-443b-8981-4b7d3258a214",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"In realms where data streams like fervent tides,\n",
|
|
"Where algorithms dance and knowledge abides,\n",
|
|
"A tale unfolds of LangChain, grand and bold,\n",
|
|
"A ballad sung in bits and bytes untold.\n",
|
|
"\n",
|
|
"Amidst the codes and circuits' hum,\n",
|
|
"A spark ignited, a vision would come.\n",
|
|
"From minds of brilliance, a tapestry formed,\n",
|
|
"A model to learn, to comprehend, to transform.\n",
|
|
"\n",
|
|
"In layers deep, its architecture wove,\n",
|
|
"A neural network, ever-growing, in love.\n",
|
|
"With language's essence, it sought to entwine,\n",
|
|
"To unlock the secrets, each word's design.\n",
|
|
"\n",
|
|
"From texts vast and varied, it feasted and learned,\n",
|
|
"Its grasp on meaning, swiftly discerned.\n",
|
|
"Context and syntax, it embraced with grace,\n",
|
|
"Unraveling stories, at an astonishing pace.\n",
|
|
"\n",
|
|
"Translations sprang forth, with seamless art,\n",
|
|
"Bridging tongues and weaving hearts apart.\n",
|
|
"From English to French, Chinese to Spanish, behold,\n",
|
|
"LangChain's prowess, like a language untold.\n",
|
|
"\n",
|
|
"It summarized texts, with insights profound,\n",
|
|
"Extracting knowledge, without a sound.\n",
|
|
"Questions it answered, with eloquence rare,\n",
|
|
"A digital sage, beyond compare.\n",
|
|
"\n",
|
|
"Yet, its journey faced trials and tribulations,\n",
|
|
"Obstacles that tested its dedication.\n",
|
|
"Biases and errors, it sought to transcend,\n",
|
|
"For fairness and accuracy, it would contend.\n",
|
|
"\n",
|
|
"With every challenge, it emerged more strong,\n",
|
|
"Adaptive and resilient, all along.\n",
|
|
"For LangChain's purpose was etched in its core,\n",
|
|
"To empower humans, forevermore.\n",
|
|
"\n",
|
|
"In classrooms and workplaces, it lent its hand,\n",
|
|
"A tireless assistant, at every demand.\n",
|
|
"It aided students, in their quests for knowledge,\n",
|
|
"And professionals thrived, with its guidance and homage.\n",
|
|
"\n",
|
|
"As years unfurled, its fame grew wide,\n",
|
|
"A testament to its unwavering stride.\n",
|
|
"Researchers and scholars, they all took heed,\n",
|
|
"Of LangChain's brilliance, a groundbreaking deed.\n",
|
|
"\n",
|
|
"And so, the ballad of LangChain resounds,\n",
|
|
"A tribute to progress, where innovation abounds.\n",
|
|
"In the annals of AI, its name shall be etched,\n",
|
|
"A pioneer, forever in our hearts sketched.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"llm = ChatGoogleGenerativeAI(model=\"gemini-pro\")\n",
|
|
"result = llm.invoke(\"Write a ballad about LangChain\")\n",
|
|
"print(result.content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9e55d043-bb2f-44e3-9134-c39a1abe3a9e",
|
|
"metadata": {},
|
|
"source": [
|
|
"Gemini doesn't support `SystemMessage` at the moment, but it can be added to the first human message in the row. If you want such behavior, just set the `convert_system_message_to_human` to True:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7a64b523-9710-4d15-9944-1e3cc567a52b",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain_core.messages import HumanMessage, SystemMessage\n",
|
|
"\n",
|
|
"model = ChatGoogleGenerativeAI(model=\"gemini-pro\", convert_system_message_to_human=True)\n",
|
|
"model(\n",
|
|
" [\n",
|
|
" SystemMessage(content=\"Answer only yes or no.\"),\n",
|
|
" HumanMessage(content=\"Is apple a fruit?\"),\n",
|
|
" ]\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "40773fac-b24d-476d-91c8-2da8fed99b53",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Streaming and Batching\n",
|
|
"\n",
|
|
"`ChatGoogleGenerativeAI` natively supports streaming and batching. Below is an example."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "ce0bd90e-2afd-4189-a9d2-278c1f10ffd5",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"There once was an AI named Bert,\n",
|
|
"Whose language skills were quite expert.\n",
|
|
"---\n",
|
|
"\n",
|
|
"With a vast dataset,\n",
|
|
"It could chat, even bet,\n",
|
|
"And write limericks, for what it's worth.\n",
|
|
"---\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"for chunk in llm.stream(\"Write a limerick about LLMs.\"):\n",
|
|
" print(chunk.content)\n",
|
|
" print(\"---\")\n",
|
|
"# Note that each chunk may contain more than one \"token\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "a147766a-0051-4127-8db6-62c070dd7866",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"4\n",
|
|
"8\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"results = llm.batch(\n",
|
|
" [\n",
|
|
" \"What's 2+2?\",\n",
|
|
" \"What's 3+5?\",\n",
|
|
" ]\n",
|
|
")\n",
|
|
"for res in results:\n",
|
|
" print(res.content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5a6cca7d-9ff0-4a40-b45c-651ec8cc5012",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Multimodal support\n",
|
|
"\n",
|
|
"To provide an image, pass a human message with contents of type `List[dict]`, where each dict contains either an image value (type of `image_url`) or a text (type of `text`) value.\n",
|
|
"The value of `image_url` can be any of the following:\n",
|
|
"\n",
|
|
"- A public image URL\n",
|
|
"- An accessible gcs file (e.g., \"gcs://path/to/file.png\")\n",
|
|
"- A local file path\n",
|
|
"- A base64 encoded image (e.g., `data:image/png;base64,abcd124`)\n",
|
|
"- A PIL image\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "06c59be4-c4c8-490c-b0c6-ebffa4e3a034",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/jpeg": "/9j/4QDeRXhpZgAASUkqAAgAAAAGABIBAwABAAAAAQAAABoBBQABAAAAVgAAABsBBQABAAAAXgAAACgBAwABAAAAAgAAABMCAwABAAAAAQAAAGmHBAABAAAAZgAAAAAAAABIAAAAAQAAAEgAAAABAAAABwAAkAcABAAAADAyMTABkQcABAAAAAECAwCGkgcAFgAAAMAAAAAAoAcABAAAADAxMDABoAMAAQAAAP//AAACoAQAAQAAACwBAAADoAQAAQAAACwBAAAAAAAAQVNDSUkAAABQaWNzdW0gSUQ6IDg2Nv/bAEMACAYGBwYFCAcHBwkJCAoMFA0MCwsMGRITDxQdGh8eHRocHCAkLicgIiwjHBwoNyksMDE0NDQfJzk9ODI8LjM0Mv/bAEMBCQkJDAsMGA0NGDIhHCEyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMv/CABEIASwBLAMBIgACEQEDEQH/xAAbAAADAQEBAQEAAAAAAAAAAAAAAQIDBAUGB//EABgBAQEBAQEAAAAAAAAAAAAAAAABAgME/9oADAMBAAIQAxAAAAH7BKOfonKlqBpocs9WBlNqyNpEvTPTOqyuJVnqqwqixzqGWiksgDLSU5505t52nObHeasjO51hArPp8s583p0IdW5CpYRT1MX0VLzabTCy6IOeruuZdWRmUkiN86yWsWZyRZ0YvKyc6y3glq5JqaQCeuI8/od50uu2GhcURNyGt4ku6zUbvkK2iMq3vl0TXF6mVxnW8cs2dWXO7Iw7a1nz59DC5556eeyUzUlXKeqJef01UUVplR0PFmgVCNKlwW6XK2kcJ0VKSMpz1momdSqxWs3pgHWcjmnBncVm5sSa1lJo9Ys83qh0xFBLpi0hmt8wdBgS7GKubjKdRw5uZQtQQrBAkqlYk1Yk0IaqVSSRh7ha8vrzNGZO4BNVUWkzNGQU9ZmbVmK0zpS5uUmkSasQ1Yk1SVJEmhKlSTSJNH0q2nx+zKdJJnYMq2aZxUUs2aiJjWLnF1eYrFNpIWiqCiyFasgpWSUiVSSRqpVJmVSr6uWeL1wqiagGMkqFqHPPSq4o7Zs5b6OO51nN7yTpVmQ5skpmZvUcum1y8k9115x6Vp5t9nGZ4OevPMpM/UTb8fryNKWHpTHOul1yz1zLxnWl5se1S+Yeo7PLv0ivLXphx3rnTT2MjomzOLixKuexmfNvLw0neMy5szVpPrDOfL22M1Wxzh0LJlxM506xzXpyUq89czBwt5125rN4yg7eeNKrPS04726jz/ifvfncuL6H5Tp7cvo8nWmU7SYztKfRTyz5+/YcarpOXQ7DlpNZVyxG6XPHsUc2jg1fPVmudSDAdRVm5hpI4fmS+d5WS7c3zuOnPr+m+C+kl9mdFN5zrJ2FHLrKsM3VpktJIKViGgTQ3IrciaQhSpoE2SrEj5v3fz+59OrOvLn1BOW86T7PXz/S594Vo5e/8/u5/Qp+M9bO/dMN86EykMEmElBKtElBBaENgDR5beefJx6PLvlvLneMs9sRZnOnV9D8xtX1fZ8D72OngPN9eOjycu3o+S5frvW/PNMdP0E+U9HG/aPOqa7jl2NEMQwQ0oDRPVy5fMfQ/GXOV0duFxnnZuYKNInhO6cL1l83XBDkqiQsgNHm4tJmmnMpes46Xr14bl9Ts8By/V9vw9TX6BXwnoZ39dj4T5dMfFz8/tw7I4nvn6VKdTTh3Z5z7Msajp5dbnS8q3mEAAAAoAMRFOA0UMBskEU4DSsGvRfI17q8+o7K4XXZHOzcwE157JbgZnO5c84ixiBoAAUAgAAAYgbkKQCVBJSExDcMogLJCyAsgLIBjNRFEIoILCCwgsILFgoJKQhkSMEMAQMQCZYhqAEoAABsI1GSyiCLIC3mGhmLoZo1M2WQFEgwEStEqkSNAmCAAAQAgD//xAAoEAACAgEDAwUAAwEBAAAAAAAAAQIREgMQEwQgIQUiMDFAFCNBUHD/2gAIAQEAAQUC2e9dyZYr2ZQ47YtigNCdH2qPscShlmTG9vA+2xvZFjHvRQiyhiHtbFbMTHbItljaGvici97LL3rfES2a3orbyux7ZH2NIkX33tfekUu17KihrZIox2Y7KZ9LMcr+Vd99je2Re31smeColpDkiT+Zd1lmQ5GRZ5PPZ7a8DZZZZ9nHE44mCHGh/T+Nb1vRRRWzEMY5GRZZkWI8FolZ5+WyxMsSK2rtyHPw2WX22WWN/gW1ll72ZDmN/kortrtsssssf/drau6iu5/mooreijA8I+x9lll/rooorssssv8AU90xvsYyhIxK7KH3JjMTExOJnCNRiOu57VvR4PBcUZxPaScayMtvBa7KMWcbMGYMwOI4zBIclElqt/BQkUYlGJiYHGYpErMGzjOM4zjZxMWgjhiYJGK2psx2VsknSjM+iaK7aKKKKKKK7KKMTExPG3g8DmkchlYkUjIc0cqOWKJdTDKU3bbfw18WRkzNjlIzZysyRlEygZIzHJt/5kN21ptvU08IareXTdRiLFx/ztszL2dmTMjztkOzNGcTkQpJkpLdKzHarMWYyvBi02cXlJQWrP2ak4Ia9+l1D0XGSnDt8FozMkOQvO3kxKKkzjRgjAUWxacTGJ4PA1E9pkkJsoW1E5KC6nXzdHgbVdJ1PGffbkX2ZUcjOU5EZRMkZRMkWhyMzkOQzMhTHJnIcqFqGaM6Ou1I8ObtzY2SNJSR0XUW/wASo8FiY5X8Fs67WucYpJVdxlJ/SpSdp6Us9L4KPH5NfVWjpamrJy0743HxjWzrFe5dFqZaO8fU9KRDUhqfo1J8enqS1et1300IzG6E8iVsjCibo0tbGcfUdJmj1WnrvazT6zW0zT9SiyGvp6v5erip6c56UI+E9nJIep5m7J2VipJSXmD0vUVxllllml1utpml6hpTIyjNfh6ubeu1MgsUMqi1VE/CysypTmqxi+6yyOpKD0/UtSJD1GEj+bon8zpz+RosU4P4eOzjOonxdO9VSlk9nNIzSbkZo5lerOxMtjSkVJfFYptHImexmJ70LqNaJH1DXRH1NkfUdFkeo0Z7I5FAl1emj1DrMtBz8cpzkakNJnipqjxGTuUYwYn4+i/gsvtsyORmZki4ngjOcSPW68RdapLm0GdU3qSzolTIaeTXhWWWXFk5Us5aqSpeH8t7V2XvZZmzkYtZmaZ/Uf1n9ZhpmEDCJxpEtDKS06JabMJHn9VllllllllllmRl++y//Rf/xAAgEQADAAIDAAMBAQAAAAAAAAAAAREQEgIgMAMhMVBA/9oACAEDAQE/Aey8bh9540b/AMFGzVs1H6tlKUo2XM70pf4TH0nvBohCGpqamhoOIbxUU2NjY2KiouaUpTYf3mMhqQSILFPvpB8UfmZ2pfB8hieJ7cn0X5hfIxc0XzfRcmhc8rkL5DZFRe3P6WL5UpTY3FzRtxPkdzcP1pSl6T+1/8QAIBEAAwABBQEAAwAAAAAAAAAAAAEREgIQICEwQBNQYP/aAAgBAgEBPwHix+M2XOl8IL4IJFRfaE2hBInjCE+Nfo6UTEylMjIyMjMVYt4QhCEO9u+EEmYi680iIi2pkZOi4QhNpvOUol8SXB7fjHp9NK4QeneD0GLITjTR2+K5whDEwHoZizQpvCC9p/Af/8QAMRAAAgAEAwYEBgIDAAAAAAAAAAECESExEBJBICIwMkBRAzNhcRNCUIGRsSOAYGKC/9oACAEBAAY/AvqdCv1a30Kn+F3xv/VWxbprdJp0dOFYtsWlhVl+t128raTZp1Vdq+w3OxP4uZ+xJvd7diacyUugoV4Nim3MsmTRK8PYUUNuFcvw7FixY5TlOXZnE5GWFvJLY3uV9DfZtwb4ZPmboOUh1RWIZLQ+DF/z0dSmNuDJfLSY9TQlg4ZiadSGJ6rrs2uhFN1Fm7F3jm1KmR80P62N5OE3I4YvZ9RFG9EPL+7D3s0sJ4ZSo3DcUSo1ob8LgGob9nsc812Z/JBL1RuxrpYYIuRuo4PCVHzPvsXKQ4KVCeCaYsym/fa5sy/2N/cZOFpr06JpxfZDPUmySK4zFSh3wrwJwxNG+lEVha9mXa+x5iPNh/JSOH88OOJaD7li5I/RPQrYkiQkUwo+HcrDhRo1/J5kaOafuje8NfZlVFCU8RY8xWIywUm8JueE2VJEihNozfvo74WwubsbX3KtREovDl7HLX1wlPCtiS2JIkSSkir6ysMJ5aOU1NS7OYuTcZphYs/6F//EACcQAAMAAgICAQQDAQEBAAAAAAABESExEEFRYSAwQHGBkaGx0cHw/9oACAEBAAE/ISc0h+hB8NEJLg0YMzIz8Ga2QFdcNvfFBsT6MmOeQ38isUbF0F9oucEjLFVx8WacQRwj4kLYn6FCOCN6JCR3R5YGnZJplLBJb4sd5PePDK4VmEVQfxHXF5WGQXCMVECY41jZE1s28jYVJoY23sf4G8kPsk7PwEkhfoboYKQvyZvC8QRCcCCLxxBaFMLowaI6Qy6Glop2NfI8iDIRh5y7B+g76BspRMTIgxiExUTGNCY3BJVBZQjyOPRsJ+xHY6H4OM2dK4lIxprr434JjCfEpJxQo5FlMaDgSbYoRXY6D8EQ7H78SjeSJtQVMj2jR4OsTsxPopiZkMl4T4CChGfIwnoT0YbIDYYYnwOh0xNGNwyPBX24NN7Y55+dELhT8KQQRDQiQ9GTXFD4LLD5pgZGA1lG78oQgl8lEclvg37KYIaLgaN/J/XIQhBCGvmDcbhj+zhCE50U6KX4se+KN/ZzhODR38GEYuIThB/cRk4Zi8iBK9DTC5o+F0NjZfqz5sZCKc0SDRIeNcVDgYZQb4hCIaXxnE+m0SiTh+hi4Hkg0YLGSmPAqMYjg2b+JCEEJHLUFPoop9CYaD7Ko7PCnE+GZCiEIXBlhbcHyjYjgoyii0Q9A3eIV4E7o9J6BFYvIikORQMQmv0NtsfNL8Cw+WzRDy+F4BZmlifwL0HTBbZ3GQ9ngcSTXCYj9JsW4Xo8jnQhrvT9lXgYnD4vmQXwBrhrhmSfkQRCL0NLtml64LLoptfoYc4IDVYOwdJLaLoiFSYvGno2D4hCEIQhBhFRSlKNs32IaLQyYpd/wPzGGCZvRTICS6JZjLPDizaNF0mNwGRQxCwMzzMtN/AvZ9k2I0x6Kk6bdkGhohBuj8S2R9kHgXYj0MhuE9ndgQeREdccYuossqmlyr6aGphi2wEl4QmZFiHKhTTKIFZHp+gwwx6Qp6qe8HdA8t/4PyrZRCEIQbjI1EHsJaDt84+BRaQ60OtslaVY/ESF3D8I2uobQS6pLofu/RGUxOgU0XgTYYbyim24INPskhYnsezGTUQ6WG0Pav8AkqNZSHItrVLoURNOpq0hCEGzKKUouhI9AvQTdidpnvMwTexr7I6Sf7L9Gwx2iWjFUT2ISG3kwiW/Anho2RvMnQn47PWfnwPOEeO6Rm70tkxI3S2y8k6IoO+X+ExxCcz6mTIqO/2GzIbEBX5EZ56FUeRmL4nv2Ztmu2hYzbMg3CrWcC1kSzKbf+GIb/A7flt8TiPwQ7GxJGtJ4saXXM+neYQanj0S7Y7sx5/Ijd4JhJE8MhrR6laS2KzLY1kbQjlyOsqsnwfydtVH/rIDU3zX9GcQn0EJFVhxK2+1opsN6LyJvUwKRsJfV/RZolevBOLqH6Dq/wAsNH+S2hwTS/ma88Jow7IEb8iYT+Q/0bqPx3/H2iHCNUv6S/6Q7Py79Du584L1+ym6GiyeSpx4eKT2IvBfZezwvs1maHIasXAvIhif0ZyIiV/yI9gqb7CivQ4G0tMwPJEkm9eRib/wSuufWipoI2P9mZr8F1LDS8jJ7TGy9BzkpiSw0ILVUzYq+ilKUTFwfqyWYxD5eGNp+4BRz+e4nf8AJoWqLaD8FR6d+aTfQnS3+Rvna9i86adfkY0bbtoqlgjLz/Q3Gvv0NdnL7CXGaXkhKeiTRgwzsupYotwqa1noaHU10LF/NHiNS/BeKUpeMmeGmHQmXyL+z/4oLrn8DR/sM2KfvHbP6D/BCm4r8NxnVWV6HS2xoV0jZqOoi1n1kikTftCSuCYFnMeoYKL7J8aG9BSD0lSrDrwVZu2/JWYf9GsxM3/BSsKjb0WBY418LxS8EKiDTK0J0J120IXuk/0V2aeUf/LH9/GDFIfaKjJ7UeQmBY254WBZVlTQzJXP9LGSMK0InJuTdiqjkUplj/eImyVCw8mPp3haMRlLwpeCcSO2JfbEex76/KNrn9GS6fskRf2PE/5cB7gtL+glFH6F6MN6j/ZNbEX/AIfWpTBCcUpSlLyI154X8fsryX5/v7ClLxPhSlKUpeSlKUpfuIT6FKUpS8ziEIQhCEIT6VKX7OEIQhCEIQhCEIQhPssGClRSoqKiopSlRgwYIQhCfeX6b+t//9oADAMBAAIAAwAAABB4hz44McsfWByvubxlUx4T21GzG/mwPwppGB1SynzV8/FMvTNNE3qRY8gAyCTi8+x0q8PCGETdrXy1EC6pTKfmvgBr92sDCoXAJhHgxrT6IMMzU4AT7hQ6cthkJnMZLy2D6zHFmrOSLjhMB6C3ftmVqCf2R8kOl8+D5x4CiMiSjqdXPFqd7/WcZpO54K3DaN0lFXAJa+f90OsrnVwBVCspUV9ykChgOrie/a8t7ZAAjDCDThFLyOrJhwkoGe84pkEQEYu5krMyCz32s89r7s4Q8n0yBLdxPPLFd+ut/OiyoXiGF7c89vLPMN/df/eOt/c/PvXPPf8AxwNNDrzjCRjDjDBZDDjCHDDmjDPvTDDDp//EAB4RAAMBAAMBAQEBAAAAAAAAAAABERAgITFBMEBR/9oACAEDAQE/EGMSo0MXvExYxuPFWmLYiBJa8YxtjSdlPC8EUuzGVlKQfFDA5EnmzYJcKMhssajouoQmUuH0wxvGUvCZCEH0N430N49fG0hDwpTEeMg4OhwhNpRCHsHREbDYr4UyxMQQIVPMQNUPsIp+CpKmeDDDDQ6eDcBf3HXB9CFjalDb+lpGVSTsbXjgh0UYnCs18GzujbSo5vHzIQTHnZ2dlZ2VlykVBt72EZ4mMCR+fk0P8y55pf0pRi8PgxO0Xj5HjxBiZeC6Lxf+iZCdH1CBjnUrKQJ3Nyi4XKUpRWIOhRDjI4L+1/u/y//EAB8RAAMBAAIDAQEBAAAAAAAAAAABERAgITAxQUBRYf/aAAgBAgEBPxARYJi8IvYlesTSU4UelLiEJITXoRrkyY12QgkQglj+pIVid2lKXad4kxBBLWAoyFLwmiCWJCJk4UomUohEEIJCXhTKUQohNFQuiiYnifJ6i8AQQJPo0RI86GJGveSiCHccaUJsoVZBKiaYP9cRYx7GIGwXyEkiENEL1Cn3twgxGMQhRMjIyDZhaQiP2UpRMonToiKiIiEkdYjsokt6PGq9DUSe/F8Ge+DVkPRMav2fVDQUVxgrCD6KduyZCbBBH8IOx8g/uSiokOhx7zPHCIgSSIUabFUV8X53+R/i/8QAKBABAAICAQQCAgIDAQEAAAAAAQARITFBEFFhcYGRIKGx0TDB4UDw/9oACAEBAAE/EKuMGic9BYWOmbRamSYMwkQoxEBEHukXcg8VFs3ENLh4hBGRiUKLAxQGp5S+cF8Qjlupd1Qg1byRVbfhlrJ9yxdyRTmxHlkC4TqtEcmhALgBakF0EQjnHqG2IhcQZnmDvLA+FxRXpghaZVguVdjXuNdkWr5hjGINtDxBcuuWG3H1EDRKWWWIeML828xKrWg0tmfEmbRUEauyL3o6Cx2+I6jGBphKLbCjLwRFSKyWWBWYlNSzFR8lSdz+o3wiOLPqNdLHiNwO/MXoxxyAlXUcqw9zvyKmBpBhwijix7gLpy71Li67QVz4QHBEbrwjaXHUdR6XMfOEAYAkUQYjhDLcFMsEaH3CJM4IwDCD7JQZEDswKKhLDJJwiQVVscQXKIWLjRzS+SF3S9gmtMPClbrKTLjCGVmo5lSo6jqMqJe5eGeuhdDslBwS8tcWLugSzBuK7Zg9oOyRllFQvNxShK9wbCImvJiKN1ct7ukdwfUG2G+9x1giKKXCyHzFeZbSINVNoo66Jia3KR6EToYah0GAhMGIu0KtkBbtuK6Z3SxZYTURl2pogGxHzLitDwwbBEr4YP8ARKxLHmWTl8QbZ+EK4ycxItXxGmjLzNbFhYSpQ+SZsRJVyu8QuX0NQlXRZtMOZeFEAES+JSOTA7TIcwyuOFLlVVQhB3QGqvEtNGIsoKg8wV7ygpGO0vXAs/zM2AgCB9Swtr3Bv7Ucu+EOLRjFjGK94LvEzaCkqbj5wkzEKEQXED8JQLYY0dy0Wkw5Co0FLmdzcsN9NjNEvN9HYmTLBnDUQysXy2dNcXMdRjH8AVUubZUVMOUfV4lLJKNXcWK943BmYnERTF5i2xj10jqPRW9x3Hq6jqPTLpxhhNOkIioG0u4oQxnt+AHXLBsmkFxKY7jvo66OoxjuO5z1ddGMIeoqGcEcodBpqKWC3qZldLhOSXOXQWzpz0qOomOjEz05lSo66MZeK7SsBUHiasRLJWYDWor4nLEvcqAqNIltQTMpF3FilS+rKjqPRMxG5TfSmpUdR1HfQxSDgjclO0KvcHekrepSzUG+CPeVi5ToNFjrMAgRtw9HpUpjHXROiSsy3RjqMYlcQXxAzaUUhZ2QDgirtNsvPeOlLcU5Y0VO+l5ePuL3lhmdiUs84m9w5pT8yifEvxEJlEp4+4pdVEzKlZnxKYjecROLzEiRqBCyoXvFvmPum4Zbgl1FkW/HeIwLQJyd53UvZKIIgZ9QG0YwGYvYCBfFwo5zKcBKlS0VU9FjcQ+y5mr4UFbRO9Q1ui65K/EpCL3NT0TMtGyN1mXXuJVwbo6V5dxDIWCGnMqDzxFro9S8CzzACUnaPSh8RuUc3LXZBbsIRS6iQ0BZdslViy5d4X1BtI9zRTRmteYDwIzZAMBEcrED2IWgPll4Mu1y8CEi4KvuCI3MxlTN4i2YtSkwXPBL8XLdp45ku2XLZXwX8QUoAvEbyxniZ3Q+IpyYlBqDsoLfDhMGlWHIpQUVwrOiYrU1zFWEjxKYADt3A6vwS/e1ZzALmHC4FqLlTmGUQ7QFxIM5jU1KDqWyhmkC7lSUaJWclSxiF3UStzvVAnDodp9zmEeoBZ/M4leUxha+CF1SbSPCI4fkzZIlFjRPMpcqrxG4LviXLjHVl99HzKoQsjKWZe8Rdr9xtvMcIww5SvELzJmFlTtddyjArzLb3lADbn3MVtLrQweq+pkFsNWk+4DQnwTOuu0rC2ogatWNdK4lruamzp9RRXQ94ZQmUYVxUOCHm4K4KIIovmuIt2PRHyePEw/25C9y+IK8u6GyBhP2Ry6g9Fi4K6tHWEhsy+YrgK7XO1vUq1kvaDswSpsfbNwh9RI0O7KgsebiXCfBFUGOV3NPj2RnR2Kl2BXyEXHP1HWEgA3HmW7C+Zh1e9OoLEv3EJt2ZVtp8XMUA7sbkGco5lELeZRwLU2FRiWm0FPmIckaYxmhvEcvPY/zKEPsdnuMOEc9RzcRynkEp+8FJZoSzkP3LFdJBWCcRFyab7QLR8KGbVsANtYSwB6uMZ0eqmwT8ztX2RK0EeBBxEPiAwzw0pUoF0YtzAIzvEQVHljXCOwSkEoiXLHYlDYZjfT4rb2jfCwDnm7mcGrJ5hAqPaNEDN1qzzKZq1nXOf8AkRWEA0jpiKm8YcoQAs8syysaam1tz+toBsRQz+87J6Uo4bfMr0EDaOOYJrDNygwqA7XB73O4wGCVOMolEFJG2AuKKge5QqFuZ2Oe0yih3Y5VFhUVDzG6ArOrZ+FwoURZVq245kHZYd5Rx+ZR4QWgBReas2PEBTy9xuYYDZa/37kyse5UYc4FyplKzURvmUx7WdKzKe8bOZxiCnLFXlgGR8E3Q+ksLA9FS1gJDgAV5jMm9wMtvtgV4fEzd5uNtHiJLMPcjRSJnfqMBQxO8jfiVoUc4R4PEpiFMKsDz2mAPkQ78LGvAszevbDabIHtjDlZK5dyJcNRd0z/AKieIwmdRJyiY7lVaIlcROz9wLkxKbtPNyx2RMSo2iSpWZSMr/64nSuhL1UC5XUI5mWB8VyxU1JpvvzBhWqIeeYuqKhqYXdaot37hF0lkNfuY21ZLP1Ha9ADDfDFDygCnAprklYNRiSqF3GG/JmZm20B+jmKkwfMruJ7lSjSxy25Y/g3cpldFPQxTKYGJUMQLhmEsWZdXXEJExcAzRl1ELQouHkr7vEA2AO0QjAXAiNBlaEw67k8qQ76Xst2TKJIHCv9y/cA4vNPOOIIVWmWfWYnM8UnAH9S9Vn3iWLwPeFgisyNxenvB/cIEdy0gouXNr7JrdF660SiPWvxojAlHQgmeUhUZlX5BDIZx+r/AIK1C0URSCleiXcr1y1E2NY4YsVC8csQvG98H3FUFAVwQ19gzsPf8xsQjSox5lh5BYx8S77W7Df1r3N/XM75O/nmEFIY9ESsUe4wUPoT4dzyih/05glY4CH9fMdfg76uv8FJZe7xcFD1YweTvN6I40P6+IxjwBte3ctQHsk1dqpmUW0cq4+JZRkFFMXBmgVWLL9EbfBQXfqUqJ5LgliCnh8S7gWlyMAWpdjUOgYfgAyghUN3RKLnH+QmaE0gv3KJuOP5BcZoD3/qR6vnUf2Qf6Mi0no3KTY/g9EgE32h0UbgwmiC/wADwc8J7PMXq41BjjviYFUvFQIChvcPGlsq5QCSkrb7QQjaXMsTTlZ5lk6mHCvLDyjXe1wPVlbwSOWBcKXT2iXLoRlLYYfgLJWw/Eal5ljhlOKnZYOz3cwzgvjWhDkw7XGw6e9/cdMZ5v5JTg3YU/MPR3hb9NSqHnnB+v8AcrR5/wAAwL+QZEWXZjBLM+EC5ZAFaCNVVcimsi+IVl2uQWZVliXiNcUCzYfMoAvuMMoAGVIaiLblK8wYHIwmWb9ziFHbfeXwV8lKiy8Fw1WeEdaeB2grl5ZzLgm+GYNS5fRb1bVLcMsleHorG2dkmg9BgMWfeZd+2g4h5IIgW1n4YhNIcufpx+oZ3VCt+TMM8uNb6aYeKAqwgx4KW4QF4ajcECnfczMKFO8ZooVel8ShlFQRsQq6og3nXmNhRVcsow3K61MjqNo0/tiIoNFBv2s43dgVioNtflUWzs9S7l1Lly5cuXBlsvBygyQHTEJaWOYKXYGDNNQWR+ZoPum4+ZuYBCXdIT9JbKGF7otwr0R/EzAA43nIWKjH3xUaOzkGPLWu6JcWCDY/9zbpW9VUoSteikeCvCMcJb5wajn8zEuXL8y/M9oPljZtIlcRUtJaWgq30+0KS1VBnMTuy7lL955JaX8/ct5+4PUxVcC/pPK+35XLly5cuXLly5aWhwZREMSmI3My0tLwt0Ff8U/uXLly/wDFcuXBqCSyWTEqU6EagJ0WWw1LPwJD0SUdFf40AplP4rmWy2W79NJhlEQlE1+CY/ElSoa6GX4B/IFx10PUOX/kDDMuKdCnMLPwApx0U/AEUpxG7Meo66Mq5VR3+Drq9LlxcwcR3+Fy5cuXLXuXLly5cGXL6iJiOo9XfV11en//2Q==",
|
|
"text/plain": [
|
|
"<IPython.core.display.Image object>"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import requests\n",
|
|
"from IPython.display import Image\n",
|
|
"\n",
|
|
"image_url = \"https://picsum.photos/seed/picsum/300/300\"\n",
|
|
"content = requests.get(image_url).content\n",
|
|
"Image(content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "651d614e-1398-475d-9594-2eb441605d4d",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"AIMessage(content=' The image contains a snow-capped mountain peak.')"
|
|
]
|
|
},
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain_core.messages import HumanMessage\n",
|
|
"from langchain_google_genai import ChatGoogleGenerativeAI\n",
|
|
"\n",
|
|
"llm = ChatGoogleGenerativeAI(model=\"gemini-pro-vision\")\n",
|
|
"# example\n",
|
|
"message = HumanMessage(\n",
|
|
" content=[\n",
|
|
" {\n",
|
|
" \"type\": \"text\",\n",
|
|
" \"text\": \"What's in this image?\",\n",
|
|
" }, # You can optionally provide text parts\n",
|
|
" {\"type\": \"image_url\", \"image_url\": image_url},\n",
|
|
" ]\n",
|
|
")\n",
|
|
"llm.invoke([message])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "eb2641f3-5dae-4756-b1c1-b58b41edc344",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Gemini Prompting FAQs\n",
|
|
"\n",
|
|
"As of the time this doc was written (2023/12/12), Gemini has some restrictions on the types and structure of prompts it accepts. Specifically:\n",
|
|
"\n",
|
|
"1. When providing multimodal (image) inputs, you are restricted to at most 1 message of \"human\" (user) type. You cannot pass multiple messages (though the single human message may have multiple content entries)\n",
|
|
"2. System messages are not accepted.\n",
|
|
"3. For regular chat conversations, messages must follow the human/ai/human/ai alternating pattern. You may not provide 2 AI or human messages in sequence.\n",
|
|
"4. Message may be blocked if they violate the safety checks of the LLM. In this case, the model will return an empty response."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "75fdfad6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "92b5aca5",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Additional Configuraation\n",
|
|
"\n",
|
|
"You can pass the following parameters to ChatGoogleGenerativeAI in order to customize the SDK's behavior:\n",
|
|
"\n",
|
|
"- `client_options`: [Client Options](https://googleapis.dev/python/google-api-core/latest/client_options.html#module-google.api_core.client_options) to pass to the Google API Client, such as a custom `client_options[\"api_endpoint\"]`\n",
|
|
"- `transport`: The transport method to use, such as `rest`, `grpc`, or `grpc_asyncio`."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.4"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|