mirror of
https://github.com/hwchase17/langchain.git
synced 2025-09-20 01:54:14 +00:00
docs: Update a note about how to track azure openai's token usage when streaming (#30409)
- **Description:** Update a note about how to track azure openai's token usage when streaming - **Issue:** #30390 - **Dependencies:** None - **Twitter handle:** None --------- Co-authored-by: Chester Curme <chester.curme@gmail.com>
This commit is contained in:
@@ -29,6 +29,31 @@
|
|||||||
"%pip install -qU langchain-anthropic langchain-openai"
|
"%pip install -qU langchain-anthropic langchain-openai"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"id": "c71b676a",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
":::note A note on streaming with OpenAI\n",
|
||||||
|
"\n",
|
||||||
|
"OpenAI's Chat Completions API does not stream token usage statistics by default (see API reference\n",
|
||||||
|
"[here](https://platform.openai.com/docs/api-reference/completions/create#completions-create-stream_options)).\n",
|
||||||
|
"To recover token counts when streaming with `ChatOpenAI`, set `stream_usage=True` as\n",
|
||||||
|
"demonstrated in this guide.\n",
|
||||||
|
"\n",
|
||||||
|
"For `AzureChatOpenAI`, set `stream_options={\"include_usage\": True}` when calling\n",
|
||||||
|
"`.(a)stream`, or initialize with:\n",
|
||||||
|
"\n",
|
||||||
|
"```python\n",
|
||||||
|
"AzureChatOpenAI(\n",
|
||||||
|
" ...,\n",
|
||||||
|
" model_kwargs={\"stream_options\": {\"include_usage\": True}},\n",
|
||||||
|
")\n",
|
||||||
|
"```\n",
|
||||||
|
"\n",
|
||||||
|
":::"
|
||||||
|
]
|
||||||
|
},
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"id": "598ae1e2-a52d-4459-81fd-cdc68b06742a",
|
"id": "598ae1e2-a52d-4459-81fd-cdc68b06742a",
|
||||||
|
Reference in New Issue
Block a user