Juanky Soriano dec3750875 Change method to calculate number of tokens for OpenAIChat (#1457) ...

Solves https://github.com/hwchase17/langchain/issues/1412

Currently `OpenAIChat` inherits the way it calculates the number of
tokens, `get_num_token`, from `BaseLLM`.
In the other hand `OpenAI` inherits from `BaseOpenAI`. 

`BaseOpenAI` and `BaseLLM` uses different methodologies for doing this.
The first relies on `tiktoken` while the second on `GPT2TokenizerFast`.

The motivation of this PR is:

1. Bring consistency about the way of calculating number of tokens
`get_num_token` to the `OpenAI` family, regardless of `Chat` vs `non
Chat` scenarios.
2. Give preference to the `tiktoken` method as it's serverless friendly.
It doesn't require downloading models which might make it incompatible
with `readonly` filesystems.

2023-03-06 07:20:25 -08:00

..

agents

Minor documentation update in initialize_agent (#1397)

2023-03-02 11:46:35 -08:00

callbacks

rfc: callback changes (#1165)

2023-02-20 22:54:15 -08:00

chains

fix always verbose on summarization checker (#1440)

2023-03-05 07:10:08 -08:00

docstore

Harrison/wiki update (#622)

2023-01-15 16:45:16 -08:00

document_loaders

Add PyMuPDF PDF loader (#1426)

2023-03-03 20:59:28 -08:00

embeddings

Return Cohere embeddings as lists of floats (#1394)

2023-03-02 09:02:10 -08:00

evaluation

Refactor some loops into list comprehensions (#1185)

2023-02-20 16:38:43 -08:00

graphs

catch networkx error (#1201)

2023-02-20 21:43:02 -08:00

indexes

bump version to 0.0.95 (#1324)

2023-02-27 07:45:54 -08:00

llms

Change method to calculate number of tokens for OpenAIChat (#1457)

2023-03-06 07:20:25 -08:00

prompts

Fix type hint of 'vectorstore_cls' arg in SemanticSimilarityExampleSelector (#1427)

2023-03-04 00:20:18 -08:00

tools

Bug: Fix Python version validation in PythonAstREPLTool (#1373)

2023-03-01 21:15:27 -08:00

utilities

searx: add install instructions, update doc and notebooks (#1420)

2023-03-03 20:57:50 -08:00

vectorstores

Harrison/similarity search chroma (#1434)

2023-03-04 08:10:15 -08:00

__init__.py

Add Writer, Banana, Modal, StochasticAI (#1270)

2023-02-24 06:58:58 -08:00

cache.py

Avoid IntegrityError for SQLiteCache updates (#1286)

2023-02-26 17:54:43 -08:00

docker-compose.yaml

add tracing support to langchain (#741)

2023-01-26 17:38:13 -08:00

example_generator.py

Harrison/improve cache (#368)

2022-12-18 16:22:42 -05:00

formatting.py

initial commit

2022-10-24 14:51:15 -07:00

input.py

Add asyncio support for LLM (OpenAI), Chain (LLMChain, LLMMathChain), and Agent (#841)

2023-02-07 21:21:57 -08:00

model_laboratory.py

Harrison/improve cache (#368)

2022-12-18 16:22:42 -05:00

py.typed

Add py.typed marker to package (#121)

2022-11-12 11:22:32 -08:00

python.py

Add a SQL agent for interacting with SQL Databases and JSON Agent for interacting with large JSON blobs (#1150)

2023-02-28 19:44:39 -08:00

requests.py

Add a SQL agent for interacting with SQL Databases and JSON Agent for interacting with large JSON blobs (#1150)

2023-02-28 19:44:39 -08:00

schema.py

add tracing support to langchain (#741)

2023-01-26 17:38:13 -08:00

serpapi.py

move serpapi wrapper (#1199)

2023-02-20 21:15:45 -08:00

server.py

add tracing support to langchain (#741)

2023-01-26 17:38:13 -08:00

sql_database.py

fix sql misinterpretation of % in query (#1408)

2023-03-02 16:03:16 -08:00

text_splitter.py

fix bug with length function (#1257)

2023-02-23 16:00:15 -08:00

utils.py

Harrison/bing wrapper (#656)

2023-01-19 14:48:30 -08:00