mirror of
https://github.com/imartinez/privateGPT.git
synced 2025-09-24 20:48:22 +00:00
Next version of PrivateGPT (#1077)
* Dockerize private-gpt * Use port 8001 for local development * Add setup script * Add CUDA Dockerfile * Create README.md * Make the API use OpenAI response format * Truncate prompt * refactor: add models and __pycache__ to .gitignore * Better naming * Update readme * Move models ignore to it's folder * Add scaffolding * Apply formatting * Fix tests * Working sagemaker custom llm * Fix linting * Fix linting * Enable streaming * Allow all 3.11 python versions * Use llama 2 prompt format and fix completion * Restructure (#3) Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local> * Fix Dockerfile * Use a specific build stage * Cleanup * Add FastAPI skeleton * Cleanup openai package * Fix DI and tests * Split tests and tests with coverage * Remove old scaffolding * Add settings logic (#4) * Add settings logic * Add settings for sagemaker --------- Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local> * Local LLM (#5) * Add settings logic * Add settings for sagemaker * Add settings-local-example.yaml * Delete terraform files * Refactor tests to use fixtures * Join deltas * Add local model support --------- Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local> * Update README.md * Fix tests * Version bump * Enable simple llamaindex observability (#6) * Enable simple llamaindex observability * Improve code through linting * Update README.md * Move to async (#7) * Migrate implementation to use asyncio * Formatting * Cleanup * Linting --------- Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local> * Query Docs and gradio UI * Remove unnecessary files * Git ignore chromadb folder * Async migration + DI Cleanup * Fix tests * Add integration test * Use fastapi responses * Retrieval service with partial implementation * Cleanup * Run formatter * Fix types * Fetch nodes asynchronously * Install local dependencies in tests * Install ui dependencies in tests * Install dependencies for llama-cpp * Fix sudo * Attempt to fix cuda issues * Attempt to fix cuda issues * Try to reclaim some space from ubuntu machine * Retrieval with context * Fix lint and imports * Fix mypy * Make retrieval API a POST * Make Completions body a dataclass * Fix LLM chat message order * Add Query Chunks to Gradio UI * Improve rag query prompt * Rollback CI Changes * Move to sync code * Using Llamaindex abstraction for query retrieval * Fix types * Default to CONDENSED chat mode for contextualized chat * Rename route function * Add Chat endpoint * Remove webhooks * Add IntelliJ run config to gitignore * .gitignore applied * Sync chat completion * Refactor total * Typo in context_files.py * Add embeddings component and service * Remove wrong dataclass from IngestService * Filter by context file id implementation * Fix typing * Implement context_filter and separate from the bool use_context in the API * Change chunks api to avoid conceptual class of the context concept * Deprecate completions and fix tests * Remove remaining dataclasses * Use embedding component in ingest service * Fix ingestion to have multipart and local upload * Fix ingestion API * Add chunk tests * Add configurable paths * Cleaning up * Add more docs * IngestResponse includes a list of IngestedDocs * Use IngestedDoc in the Chunk document reference * Rename ingest routes to ingest_router.py * Fix test working directory for intellij * Set testpaths for pytest * Remove unused as_chat_engine * Add .fleet ide to gitignore * Make LLM and Embedding model configurable * Fix imports and checks * Let local_data folder exist empty in the repository * Don't use certain metadata in LLM * Remove long lines * Fix windows installation * Typos * Update poetry.lock * Add TODO for linux * Script and first version of docs * No jekill build * Fix relative url to openapi json * Change default docs values * Move chromadb dependency to the general group * Fix tests to use separate local_data * Create CNAME * Update CNAME * Fix openapi.json relative path * PrivateGPT logo * WIP OpenAPI documentation metadata * Add ingest script (#11) * Add ingest script * Fix broken name refactor * Add ingest docs and Makefile script * Linting * Move transformers to main dependency * Move torch to main dependencies * Don't load HuggingFaceEmbedding in tests * Fix lint --------- Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local> * Rename file to camel_case * Commit settings-local.yaml * Move documentation to public docs * Fix docker image for linux * Installation and Running the Server documentation * Move back to docs folder, as it is the only supported by github pages * Delete CNAME * Create CNAME * Delete CNAME * Create CNAME * Improved API documentation * Fix lint * Completions documentation * Updated openapi scheme * Ingestion API doc * Minor doc changes * Updated openapi scheme * Chunks API documentation * Embeddings and Health API, and homogeneous responses * Revamp README with new skeleton of content * More docs * PrivateGPT logo * Improve UI * Update ingestion docu * Update README with new sections * Use context window in the retriever * Gradio Documentation * Add logo to UI * Include Contributing and Community sections to README * Update links to resources in the README * Small README.md updates * Wrap lines of README.md * Don't put health under /v1 * Add copy button to Chat * Architecture documentation * Updated openapi.json * Updated openapi.json * Updated openapi.json * Change UI label * Update documentation * Add releases link to README.md * Gradio avatar and stop debug * Readme update * Clean old files * Remove unused terraform checks * Update twitter link. * Disable minimum coverage * Clean install message in README.md --------- Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local> Co-authored-by: Iván Martínez <ivanmartit@gmail.com> Co-authored-by: RubenGuerrero <ruben.guerrero@boopos.com> Co-authored-by: Daniel Gallego Vico <daniel.gallego@bq.com>
This commit is contained in:
Pablo Orgaz
GitHub
0
private_gpt/server/chat/__init__.py
Normal file
0
private_gpt/server/chat/__init__.py
Normal file
82
private_gpt/server/chat/chat_router.py
Normal file
82
private_gpt/server/chat/chat_router.py
Normal file
@@ -0,0 +1,82 @@
|
||||
from fastapi import APIRouter
|
||||
from llama_index.llms import ChatMessage, MessageRole
|
||||
from pydantic import BaseModel
|
||||
from starlette.responses import StreamingResponse
|
||||
|
||||
from private_gpt.di import root_injector
|
||||
from private_gpt.open_ai.extensions.context_filter import ContextFilter
|
||||
from private_gpt.open_ai.openai_models import (
|
||||
OpenAICompletion,
|
||||
OpenAIMessage,
|
||||
to_openai_response,
|
||||
to_openai_sse_stream,
|
||||
)
|
||||
from private_gpt.server.chat.chat_service import ChatService
|
||||
|
||||
chat_router = APIRouter(prefix="/v1")
|
||||
|
||||
|
||||
class ChatBody(BaseModel):
|
||||
messages: list[OpenAIMessage]
|
||||
use_context: bool = False
|
||||
context_filter: ContextFilter | None = None
|
||||
stream: bool = False
|
||||
|
||||
model_config = {
|
||||
"json_schema_extra": {
|
||||
"examples": [
|
||||
{
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": "How do you fry an egg?",
|
||||
}
|
||||
],
|
||||
"stream": False,
|
||||
"use_context": True,
|
||||
"context_filter": {
|
||||
"docs_ids": ["c202d5e6-7b69-4869-81cc-dd574ee8ee11"]
|
||||
},
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@chat_router.post(
|
||||
"/chat/completions",
|
||||
response_model=None,
|
||||
responses={200: {"model": OpenAICompletion}},
|
||||
tags=["Contextual Completions"],
|
||||
)
|
||||
def chat_completion(body: ChatBody) -> OpenAICompletion | StreamingResponse:
|
||||
"""Given a list of messages comprising a conversation, return a response.
|
||||
|
||||
If `use_context` is set to `true`, the model will use context coming
|
||||
from the ingested documents to create the response. The documents being used can
|
||||
be filtered using the `context_filter` and passing the document IDs to be used.
|
||||
Ingested documents IDs can be found using `/ingest/list` endpoint. If you want
|
||||
all ingested documents to be used, remove `context_filter` altogether.
|
||||
|
||||
When using `'stream': true`, the API will return data chunks following [OpenAI's
|
||||
streaming model](https://platform.openai.com/docs/api-reference/chat/streaming):
|
||||
```
|
||||
{"id":"12345","object":"completion.chunk","created":1694268190,
|
||||
"model":"private-gpt","choices":[{"index":0,"delta":{"content":"Hello"},
|
||||
"finish_reason":null}]}
|
||||
```
|
||||
"""
|
||||
service = root_injector.get(ChatService)
|
||||
all_messages = [
|
||||
ChatMessage(content=m.content, role=MessageRole(m.role)) for m in body.messages
|
||||
]
|
||||
if body.stream:
|
||||
stream = service.stream_chat(
|
||||
all_messages, body.use_context, body.context_filter
|
||||
)
|
||||
return StreamingResponse(
|
||||
to_openai_sse_stream(stream), media_type="text/event-stream"
|
||||
)
|
||||
else:
|
||||
response = service.chat(all_messages, body.use_context, body.context_filter)
|
||||
return to_openai_response(response)
|
||||
116
private_gpt/server/chat/chat_service.py
Normal file
116
private_gpt/server/chat/chat_service.py
Normal file
@@ -0,0 +1,116 @@
|
||||
from collections.abc import Sequence
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from injector import inject, singleton
|
||||
from llama_index import ServiceContext, StorageContext, VectorStoreIndex
|
||||
from llama_index.chat_engine import ContextChatEngine
|
||||
from llama_index.indices.postprocessor import MetadataReplacementPostProcessor
|
||||
from llama_index.llm_predictor.utils import stream_chat_response_to_tokens
|
||||
from llama_index.llms import ChatMessage
|
||||
from llama_index.types import TokenGen
|
||||
|
||||
from private_gpt.components.embedding.embedding_component import EmbeddingComponent
|
||||
from private_gpt.components.llm.llm_component import LLMComponent
|
||||
from private_gpt.components.node_store.node_store_component import NodeStoreComponent
|
||||
from private_gpt.components.vector_store.vector_store_component import (
|
||||
VectorStoreComponent,
|
||||
)
|
||||
from private_gpt.open_ai.extensions.context_filter import ContextFilter
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from llama_index.chat_engine.types import (
|
||||
AgentChatResponse,
|
||||
StreamingAgentChatResponse,
|
||||
)
|
||||
|
||||
|
||||
@singleton
|
||||
class ChatService:
|
||||
@inject
|
||||
def __init__(
|
||||
self,
|
||||
llm_component: LLMComponent,
|
||||
vector_store_component: VectorStoreComponent,
|
||||
embedding_component: EmbeddingComponent,
|
||||
node_store_component: NodeStoreComponent,
|
||||
) -> None:
|
||||
self.llm_service = llm_component
|
||||
self.vector_store_component = vector_store_component
|
||||
self.storage_context = StorageContext.from_defaults(
|
||||
vector_store=vector_store_component.vector_store,
|
||||
docstore=node_store_component.doc_store,
|
||||
index_store=node_store_component.index_store,
|
||||
)
|
||||
self.service_context = ServiceContext.from_defaults(
|
||||
llm=llm_component.llm, embed_model=embedding_component.embedding_model
|
||||
)
|
||||
self.index = VectorStoreIndex.from_vector_store(
|
||||
vector_store_component.vector_store,
|
||||
storage_context=self.storage_context,
|
||||
service_context=self.service_context,
|
||||
show_progress=True,
|
||||
)
|
||||
|
||||
def _chat_with_contex(
|
||||
self,
|
||||
message: str,
|
||||
context_filter: ContextFilter | None = None,
|
||||
chat_history: Sequence[ChatMessage] | None = None,
|
||||
streaming: bool = False,
|
||||
) -> Any:
|
||||
vector_index_retriever = self.vector_store_component.get_retriever(
|
||||
index=self.index, context_filter=context_filter
|
||||
)
|
||||
chat_engine = ContextChatEngine.from_defaults(
|
||||
retriever=vector_index_retriever,
|
||||
service_context=self.service_context,
|
||||
node_postprocessors=[
|
||||
MetadataReplacementPostProcessor(target_metadata_key="window"),
|
||||
],
|
||||
)
|
||||
if streaming:
|
||||
result = chat_engine.stream_chat(message, chat_history)
|
||||
else:
|
||||
result = chat_engine.chat(message, chat_history)
|
||||
return result
|
||||
|
||||
def stream_chat(
|
||||
self,
|
||||
messages: list[ChatMessage],
|
||||
use_context: bool = False,
|
||||
context_filter: ContextFilter | None = None,
|
||||
) -> TokenGen:
|
||||
if use_context:
|
||||
last_message = messages[-1].content
|
||||
response: StreamingAgentChatResponse = self._chat_with_contex(
|
||||
message=last_message if last_message is not None else "",
|
||||
chat_history=messages[:-1],
|
||||
context_filter=context_filter,
|
||||
streaming=True,
|
||||
)
|
||||
response_gen = response.response_gen
|
||||
else:
|
||||
stream = self.llm_service.llm.stream_chat(messages)
|
||||
response_gen = stream_chat_response_to_tokens(stream)
|
||||
return response_gen
|
||||
|
||||
def chat(
|
||||
self,
|
||||
messages: list[ChatMessage],
|
||||
use_context: bool = False,
|
||||
context_filter: ContextFilter | None = None,
|
||||
) -> str:
|
||||
if use_context:
|
||||
last_message = messages[-1].content
|
||||
wrapped_response: AgentChatResponse = self._chat_with_contex(
|
||||
message=last_message if last_message is not None else "",
|
||||
chat_history=messages[:-1],
|
||||
context_filter=context_filter,
|
||||
streaming=False,
|
||||
)
|
||||
response = wrapped_response.response
|
||||
else:
|
||||
chat_response = self.llm_service.llm.chat(messages)
|
||||
response_content = chat_response.message.content
|
||||
response = response_content if response_content is not None else ""
|
||||
return response
|
||||
Reference in New Issue
Block a user