mirror of
https://github.com/hwchase17/langchain.git
synced 2025-05-09 01:00:01 +00:00
1efb9bae5f
- [Xorbits Inference(Xinference)](https://github.com/xorbitsai/inference) is a powerful and versatile library designed to serve language, speech recognition, and multimodal models. Xinference supports a variety of GGML-compatible models including chatglm, whisper, and vicuna, and utilizes heterogeneous hardware and a distributed architecture for seamless cross-device and cross-server model deployment. - This PR integrates Xinference models and Xinference embeddings into LangChain. - Dependencies: To install the depenedencies for this integration, run `pip install "xinference[all]"` - Example Usage: To start a local instance of Xinference, run `xinference`. To deploy Xinference in a distributed cluster, first start an Xinference supervisor using `xinference-supervisor`: `xinference-supervisor -H "${supervisor_host}"` Then, start the Xinference workers using `xinference-worker` on each server you want to run them on. `xinference-worker -e "http://${supervisor_host}:9997"` To use Xinference with LangChain, you also need to launch a model. You can use command line interface (CLI) to do so. Fo example: `xinference launch -n vicuna-v1.3 -f ggmlv3 -q q4_0`. This launches a model named vicuna-v1.3 with `model_format="ggmlv3"` and `quantization="q4_0"`. A model UID is returned for you to use. Now you can use Xinference with LangChain: ```python from langchain.llms import Xinference llm = Xinference( server_url="http://0.0.0.0:9997", # suppose the supervisor_host is "0.0.0.0" model_uid = {model_uid} # model UID returned from launching a model ) llm( prompt="Q: where can we visit in the capital of France? A:", generate_config={"max_tokens": 1024}, ) ``` You can also use RESTful client to launch a model: ```python from xinference.client import RESTfulClient client = RESTfulClient("http://0.0.0.0:9997") model_uid = client.launch_model(model_name="vicuna-v1.3", model_size_in_billions=7, quantization="q4_0") ``` The following code block demonstrates how to use Xinference embeddings with LangChain: ```python from langchain.embeddings import XinferenceEmbeddings xinference = XinferenceEmbeddings( server_url="http://0.0.0.0:9997", model_uid = model_uid ) ``` ```python query_result = xinference.embed_query("This is a test query") ``` ```python doc_result = xinference.embed_documents(["text A", "text B"]) ``` Xinference is still under rapid development. Feel free to [join our Slack community](https://xorbitsio.slack.com/join/shared_invite/zt-1z3zsm9ep-87yI9YZ_B79HLB2ccTq4WA) to get the latest updates! - Request for review: @hwchase17, @baskaryan - Twitter handle: https://twitter.com/Xorbitsio --------- Co-authored-by: Bagatur <baskaryan@gmail.com> |
||
|---|---|---|
| .. | ||
| ai21.ipynb | ||
| aleph_alpha.ipynb | ||
| amazon_api_gateway_example.ipynb | ||
| anyscale.ipynb | ||
| azure_openai_example.ipynb | ||
| azureml_endpoint_example.ipynb | ||
| banana.ipynb | ||
| baseten.ipynb | ||
| beam.ipynb | ||
| bedrock.ipynb | ||
| cerebriumai_example.ipynb | ||
| chatglm.ipynb | ||
| clarifai.ipynb | ||
| cohere.ipynb | ||
| ctransformers.ipynb | ||
| databricks.ipynb | ||
| deepinfra_example.ipynb | ||
| forefrontai_example.ipynb | ||
| google_vertex_ai_palm.ipynb | ||
| gooseai_example.ipynb | ||
| gpt4all.ipynb | ||
| huggingface_hub.ipynb | ||
| huggingface_pipelines.ipynb | ||
| huggingface_textgen_inference.ipynb | ||
| index.mdx | ||
| jsonformer_experimental.ipynb | ||
| koboldai.ipynb | ||
| llamacpp.ipynb | ||
| llm_caching.ipynb | ||
| manifest.ipynb | ||
| modal.ipynb | ||
| mosaicml.ipynb | ||
| nlpcloud.ipynb | ||
| octoai.ipynb | ||
| openai.ipynb | ||
| openllm.ipynb | ||
| openlm.ipynb | ||
| petals_example.ipynb | ||
| pipelineai_example.ipynb | ||
| predibase.ipynb | ||
| predictionguard.ipynb | ||
| promptlayer_openai.ipynb | ||
| rellm_experimental.ipynb | ||
| replicate.ipynb | ||
| runhouse.ipynb | ||
| sagemaker.ipynb | ||
| stochasticai.ipynb | ||
| textgen.ipynb | ||
| tongyi.ipynb | ||
| writer.ipynb | ||
| xinference.ipynb | ||