mirror of
https://github.com/hwchase17/langchain.git
synced 2025-09-19 00:58:32 +00:00
docs: Update a note about how to track azure openai's token usage when streaming (#30409)
- **Description:** Update a note about how to track azure openai's token usage when streaming - **Issue:** #30390 - **Dependencies:** None - **Twitter handle:** None --------- Co-authored-by: Chester Curme <chester.curme@gmail.com>
This commit is contained in:
@@ -29,6 +29,31 @@
|
||||
"%pip install -qU langchain-anthropic langchain-openai"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c71b676a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
":::note A note on streaming with OpenAI\n",
|
||||
"\n",
|
||||
"OpenAI's Chat Completions API does not stream token usage statistics by default (see API reference\n",
|
||||
"[here](https://platform.openai.com/docs/api-reference/completions/create#completions-create-stream_options)).\n",
|
||||
"To recover token counts when streaming with `ChatOpenAI`, set `stream_usage=True` as\n",
|
||||
"demonstrated in this guide.\n",
|
||||
"\n",
|
||||
"For `AzureChatOpenAI`, set `stream_options={\"include_usage\": True}` when calling\n",
|
||||
"`.(a)stream`, or initialize with:\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"AzureChatOpenAI(\n",
|
||||
" ...,\n",
|
||||
" model_kwargs={\"stream_options\": {\"include_usage\": True}},\n",
|
||||
")\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
":::"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "598ae1e2-a52d-4459-81fd-cdc68b06742a",
|
||||
|
Reference in New Issue
Block a user