openai[patch]: release 0.2.6 (#27924)

Some additions in support of [predicted
outputs](https://platform.openai.com/docs/guides/latency-optimization#use-predicted-outputs)
feature:
- Bump openai sdk version
- Add integration test
- Add example to integration docs

The `prediction` kwarg is already plumbed through model invocation.
This commit is contained in:
ccurme
2024-11-05 18:02:24 -05:00
committed by GitHub
parent a8c473e114
commit 66966a6e72
4 changed files with 150 additions and 12 deletions

View File

@@ -509,6 +509,101 @@
"output_message.content"
]
},
{
"cell_type": "markdown",
"id": "5c35d0a4-a6b8-4d35-a02b-a37a8bda5692",
"metadata": {},
"source": [
"## Predicted output\n",
"\n",
":::info\n",
"Requires `langchain-openai>=0.2.6`\n",
":::\n",
"\n",
"Some OpenAI models (such as their `gpt-4o` and `gpt-4o-mini` series) support [Predicted Outputs](https://platform.openai.com/docs/guides/latency-optimization#use-predicted-outputs), which allow you to pass in a known portion of the LLM's expected output ahead of time to reduce latency. This is useful for cases such as editing text or code, where only a small part of the model's output will change.\n",
"\n",
"Here's an example:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "88fee1e9-58c1-42ad-ae23-24b882e175e7",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/// <summary>\n",
"/// Represents a user with a first name, last name, and email.\n",
"/// </summary>\n",
"public class User\n",
"{\n",
" /// <summary>\n",
" /// Gets or sets the user's first name.\n",
" /// </summary>\n",
" public string FirstName { get; set; }\n",
"\n",
" /// <summary>\n",
" /// Gets or sets the user's last name.\n",
" /// </summary>\n",
" public string LastName { get; set; }\n",
"\n",
" /// <summary>\n",
" /// Gets or sets the user's email.\n",
" /// </summary>\n",
" public string Email { get; set; }\n",
"}\n",
"{'token_usage': {'completion_tokens': 226, 'prompt_tokens': 166, 'total_tokens': 392, 'completion_tokens_details': {'accepted_prediction_tokens': 49, 'audio_tokens': None, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 107}, 'prompt_tokens_details': {'audio_tokens': None, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_45cf54deae', 'finish_reason': 'stop', 'logprobs': None}\n"
]
}
],
"source": [
"code = \"\"\"\n",
"/// <summary>\n",
"/// Represents a user with a first name, last name, and username.\n",
"/// </summary>\n",
"public class User\n",
"{\n",
" /// <summary>\n",
" /// Gets or sets the user's first name.\n",
" /// </summary>\n",
" public string FirstName { get; set; }\n",
"\n",
" /// <summary>\n",
" /// Gets or sets the user's last name.\n",
" /// </summary>\n",
" public string LastName { get; set; }\n",
"\n",
" /// <summary>\n",
" /// Gets or sets the user's username.\n",
" /// </summary>\n",
" public string Username { get; set; }\n",
"}\n",
"\"\"\"\n",
"\n",
"llm = ChatOpenAI(model=\"gpt-4o\")\n",
"query = (\n",
" \"Replace the Username property with an Email property. \"\n",
" \"Respond only with code, and with no markdown formatting.\"\n",
")\n",
"response = llm.invoke(\n",
" [{\"role\": \"user\", \"content\": query}, {\"role\": \"user\", \"content\": code}],\n",
" prediction={\"type\": \"content\", \"content\": code},\n",
")\n",
"print(response.content)\n",
"print(response.response_metadata)"
]
},
{
"cell_type": "markdown",
"id": "2ee1b26d-a388-4e7c-9f40-bfd1388ecc03",
"metadata": {},
"source": [
"Note that currently predictions are billed as additional tokens and may increase your usage and costs in exchange for this reduced latency."
]
},
{
"cell_type": "markdown",
"id": "feb4a499",
@@ -601,7 +696,7 @@
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
@@ -615,7 +710,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
"version": "3.10.4"
}
},
"nbformat": 4,