From 428de88398b84977156de811413b28843e9511ce Mon Sep 17 00:00:00 2001 From: axiangcoding Date: Sat, 22 Mar 2025 02:18:50 +0800 Subject: [PATCH] docs: Update a note about how to track azure openai's token usage when streaming (#30409) - **Description:** Update a note about how to track azure openai's token usage when streaming - **Issue:** #30390 - **Dependencies:** None - **Twitter handle:** None --------- Co-authored-by: Chester Curme --- .../how_to/chat_token_usage_tracking.ipynb | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/docs/how_to/chat_token_usage_tracking.ipynb b/docs/docs/how_to/chat_token_usage_tracking.ipynb index a01ee9f01cb..95742c9371b 100644 --- a/docs/docs/how_to/chat_token_usage_tracking.ipynb +++ b/docs/docs/how_to/chat_token_usage_tracking.ipynb @@ -29,6 +29,31 @@ "%pip install -qU langchain-anthropic langchain-openai" ] }, + { + "cell_type": "markdown", + "id": "c71b676a", + "metadata": {}, + "source": [ + ":::note A note on streaming with OpenAI\n", + "\n", + "OpenAI's Chat Completions API does not stream token usage statistics by default (see API reference\n", + "[here](https://platform.openai.com/docs/api-reference/completions/create#completions-create-stream_options)).\n", + "To recover token counts when streaming with `ChatOpenAI`, set `stream_usage=True` as\n", + "demonstrated in this guide.\n", + "\n", + "For `AzureChatOpenAI`, set `stream_options={\"include_usage\": True}` when calling\n", + "`.(a)stream`, or initialize with:\n", + "\n", + "```python\n", + "AzureChatOpenAI(\n", + " ...,\n", + " model_kwargs={\"stream_options\": {\"include_usage\": True}},\n", + ")\n", + "```\n", + "\n", + ":::" + ] + }, { "cell_type": "markdown", "id": "598ae1e2-a52d-4459-81fd-cdc68b06742a",