1
0
mirror of https://github.com/imartinez/privateGPT.git synced 2025-05-10 09:24:36 +00:00
Commit Graph

332 Commits

Author SHA1 Message Date
Pablo Orgaz
0d520026a3
fix: Windows 11 failing to auto-delete tmp file () 2023-11-17 18:23:57 +01:00
Lai Zn
4197ada626
feat: enable resume download for hf_hub_download () 2023-11-17 00:13:11 +01:00
Iván Martínez
09d9a91946
Create CNAME 2023-11-17 00:07:50 +01:00
Iván Martínez
f339f7608c
Move Docs to Fern () 2023-11-16 23:25:14 +01:00
Iván Martínez
ff7e2bc9dd
Delete CNAME 2023-11-16 22:53:00 +01:00
Iván Martínez
2a417d2f61
Fix/qdrant support ()
* Disable check same thread by default to enable disk-based Qdrant local client to work
2023-11-16 13:29:17 +01:00
Dominik Fuchs
23fa530c31
added wipe make command ()
* added `wipe` make command

* Apply suggestions from code review

Thanks for your suggestions, I like to apply them.

Co-authored-by: lopagela <lpglm@orange.fr>

* added `wipe` command to the documentation

* rebased to generate valid openapi.json

---------

Co-authored-by: lopagela <lpglm@orange.fr>
2023-11-16 11:44:02 +01:00
Anush
03d1ae6d70
feat: Qdrant support ()
* feat: Qdrant support

* Update private_gpt/components/vector_store/vector_store_component.py
2023-11-13 21:23:26 +01:00
Iván Martínez
86fc4781d8
Fix openai setting literal () 2023-11-12 22:29:26 +01:00
Pablo Orgaz
022bd718e3
fix: Remove global state ()
* Remove all global settings state

* chore: remove autogenerated class

* chore: cleanup

* chore: merge conflicts
2023-11-12 22:20:36 +01:00
Iván Martínez
f394ca61bb
Reuse existing stored index during ingestion () 2023-11-12 22:14:38 +01:00
lopagela
aa70d3d9f0
Add simple Basic auth ()
* Add simple Basic auth

To enable the basic authentication, one must set `server.auth.enabled`
to true.

The static string defined in `server.auth.secret` must be set in the
header `Authorization`.

The health check endpoint will always be accessible, no matter the API
auth configuration.

* Fix linting and type check

* Fighting with mypy being too restrictive

Had to disable mypy in the `auth` as we are not using the same signature
for the authenticated method.

mypy was complaining that the signatures of `authenticated` must be
identical, no matter in which logical branch we are.
Given that fastapi is accomodating itself of method signatures (it will
inject the dependencies in the method call), this warning of mypy is
actually preventing us to do something legit.

mypy doc: https://mypy.readthedocs.io/en/stable/common_issues.html

* Write tests to verify that the simple auth is working
2023-11-12 19:05:00 +01:00
Iván Martínez
b7647542f4
Curate sources to avoid the UI crashing ()
* Curate sources to avoid the UI crashing

* Remove sources from chat history to avoid confusing the LLM
2023-11-12 10:59:51 +01:00
lopagela
a579c9bdc5
Update poetry lock ()
* Update the version of llama_index used to fix transient openai errors

* Update poetry.lock file

* Make `local` mode the default mode by default
2023-11-11 22:44:19 +01:00
Iván Martínez
a22969ad1f
Add sources to completions APIs and UI () 2023-11-11 21:39:15 +01:00
César García
dbd99e7b4b
Update description.md ()
Added a section on how to customize low level args, proposing people to stick to suggested models.
2023-11-11 09:23:46 +01:00
lopagela
8487440a6f
Add basic CORS () 2023-11-10 14:29:43 +01:00
lopagela
a666fd5b73
Refactor UI state management ()
* Added logs at generation of the UI, and generate the UI in an object
* Make ingest script more verbose in case of an error at ingestion time
* Removed the explicit state in the UI containing ingested files
* Make script of ingestion a bit more verbose by displaying stack traces
* Change the browser tab title of privateGPT ui to `My Private GPT`
2023-11-10 10:42:43 +01:00
Iván Martínez
55e626eac7
Update README.md
Make installation docs and community more visibile
2023-11-10 10:33:17 +01:00
Iván Martínez
c81f4b2ebd
Search in Docs to UI ()
Move from Context Chunks JSON response to a more comprehensive Search in Docs functionality
2023-11-09 12:44:57 +01:00
Iván Martínez
1e96e3a29e
Stake issues/PRs not updated in 15 days ()
Moving temporarily to 15 days given all PRs and Issues were "updated" on Oct 19 to label them as "primordial"
2023-11-08 12:07:17 +01:00
Iván Martínez
f75f60b234
Create stale.yml ()
Workflow to automatically mark and close stale issues and PRs
2023-11-07 15:41:05 +01:00
lopagela
23cd3fea10
Parse JSON files using llama_index JSONReader ()
Patch the default list of llama_index to support JSON files.
This injection of JSON documents should improve the comprehension in
JSON files, as there is a parsing of JSON files.
2023-11-07 15:39:40 +01:00
lopagela
0c40cfb115
Endpoint to delete documents ingested ()
A file that is ingested will be transformed into several documents (that
are organized into nodes).
This endpoint is deleting documents (bits of a file). These bits can be
retrieved thanks to the endpoint to list all the documents.
2023-11-06 15:47:42 +01:00
lopagela
6583dc84c0
feat: Disable Gradio Analytics ()
* Disable Gradio Analytics

Gradio analytics can be disabled by either using the kwargs `enable_analytics` on `gr.Blocks`, or by setting the env variable `GRADIO_ANALYTICS_ENABLED` to something different from `True`.

Since that Gradio does not seem to respect their code contract (around `enable_analytics`), and that they are performing other operations only based on the value of `GRADIO_ANALYTICS_ENABLED` (c.f. `gradio.strings` https://github.com/gradio-app/gradio/blob/main/gradio/strings.py#L39), we are disabling gradio analytics by setting the required env variable to `False`.

Note: Setting an environment variables using `os.environ['foo'] = 'bar'` on system that are not based on unix might not work.

c.f. https://docs.python.org/3/library/os.html#os.environ for details on how `os.environ` works and all its caveats

* Update private_gpt/__init__.py
2023-11-06 14:31:26 +01:00
Pablo Orgaz
0d677e10b9
feat: move torch and transformers to local group () 2023-11-06 14:24:16 +01:00
Iván Martínez
ad512e3c42
Feature/sagemaker embedding ()
* Sagemaker deployed embedding model support

---------

Co-authored-by: Pablo Orgaz <pabloogc@gmail.com>
2023-11-05 16:16:49 +01:00
Pierre Marais
f29df84301
Disable chromaDB anonymous information collection ()
See https://docs.trychroma.com/telemetry
2023-11-02 12:45:48 +01:00
Pablo Orgaz
a517a588c4
fix: sagemaker config and chat methods () 2023-10-30 21:54:41 +01:00
NetroScript
b0e258265f
Improve logging and error handling when ingesting an entire folder () 2023-10-30 21:54:09 +01:00
Pablo Orgaz
5d1be6e94c
chore: only generate docker images on demand ()
* chore: only generate docker images on demand

* chore: consistent naming
2023-10-29 21:48:16 +01:00
lopagela
64c5ae214a
feat: Drop loguru and use builtin logging ()
* Configure simple builtin logging

Changed the 2 existing `print` in the `private_gpt` code base into actual python logging, stop using loguru (dependency will be dropped in a later commit).
Try to use the `key=value` logging convention in logs (to indicate what dynamic values represents, and what is dynamic vs not).
Using `%s` log style, so that the string formatting is pushed inside the logger, giving the ability to the logger to determine if the string need to be formatted or not (i.e. strings from debug logs might not be formatted if the log level is not debug)
The (basic) builtin log configuration have been placed in `private_gpt/__init__.py` in order to initialize the logging system even before we start to launch any python code in `private_gpt` package (ensuring we get any initialization log formatted as we want to)
Disabled `uvicorn` custom logging format, resulting in having uvicorn logs being outputted in our formatted.

Some more concise format could be used if we want to, especially:
```
COMPACT_LOG_FORMAT = '%(asctime)s.%(msecs)03d [%(levelname)s] %(name)s - %(message)s'
```

Python documentation and cookbook on logging for reference:
* https://docs.python.org/3/library/logging.html
* https://docs.python.org/3/howto/logging.html

* Removing loguru from the dependencies

Result of `poetry remove loguru`

* PR feedback: using `logger` variable name instead of `log`

---------

Co-authored-by: Louis Melchior <louis@jaris.io>
2023-10-29 19:11:02 +01:00
Pablo Orgaz
24cfddd60f
fix: fix pytorch version to avoid wheel bug ()
* fix: fix pytorch version

* fix: settings env var regex and split

* fix: add models folder for docker user
2023-10-27 20:27:40 +02:00
Pablo Orgaz
895588b82a
fix: Docker and sagemaker setup ()
* fix: docker copying extra files

* feat: allow configuring mode through env vars

* feat: Attempt to build and tag a docker image

* fix: run docker on release

* fix: typing in prompt transformation

* chore: remove tutorial comments
2023-10-27 13:29:29 +02:00
Shivam Singh
768e5ff505
chore: Update README.md ()
Removed Grammatical errors
2023-10-24 12:43:41 +02:00
Iván Martínez
78546524d0
Use OpenAI for embeddings when openai mode is selected () 2023-10-23 10:50:42 +02:00
Federico Grandi
769a047b54
chore: add GitHub metadata ()
* chore: add citation file

* chore: add LICENSE

* chore: update email

Co-authored-by: Daniel Gallego <danielgallegovico@gmail.com>

* chore: update email in citation file

Co-authored-by: Pablo Orgaz <pabloogc@gmail.com>

* chore: add ORCIDs to citation file

* docs: update README with citation info

---------

Co-authored-by: Daniel Gallego <danielgallegovico@gmail.com>
Co-authored-by: Pablo Orgaz <pabloogc@gmail.com>
2023-10-23 10:49:02 +02:00
Ikko Eltociear Ashimine
ba23443a70
fix: typo in README.md ()
componentes -> components
2023-10-23 08:54:12 +02:00
github-actions[bot]
b8383e00a6
chore(main): release 0.0.2 ()
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-10-20 18:29:21 +02:00
Iván Martínez
f5a9bf4e37
fix: chromadb max batch size () 2023-10-20 18:24:56 +02:00
Pablo Orgaz
b46c1087e2
chore: add linux instructions and C++ guide ()
* fix: add linux instructions

Co-authored-by: BW-Projects

* chore: Add C++ as a base requirement in the docs

* chore: Add clang for OSX

* chore: Update docs for OSX and gcc

* chore: make docs

---------

Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local>
2023-10-20 14:36:50 +02:00
github-actions[bot]
97d860a7c9
chore(main): release 0.0.1 ()
* chore(main): release 0.0.1

* Update CHANGELOG.md

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Pablo Orgaz <pabloogc@gmail.com>
2023-10-20 13:08:51 +02:00
Iván Martínez
490d93fdc1 chore: Initial version
Release-As: 0.0.1
2023-10-20 12:57:52 +02:00
Pablo Orgaz
aa4bb17a2e
fix: make docs more visible () 2023-10-19 22:12:30 +02:00
Iván Martínez
d249a17c33
feat(ui): add LLM mode to UI () 2023-10-19 19:21:29 +02:00
Pablo Orgaz
b7450911b2
feat: Release GitHub action ()
* feat: add release-please.yml

* feat: add release-please.yml

---------

Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local>
2023-10-19 17:34:41 +02:00
Iván Martínez
3ad1da019b
Update README.md
Primordial branch correct url
2023-10-19 16:17:35 +02:00
Pablo Orgaz
51cc638758
Next version of PrivateGPT ()
* Dockerize private-gpt

* Use port 8001 for local development

* Add setup script

* Add CUDA Dockerfile

* Create README.md

* Make the API use OpenAI response format

* Truncate prompt

* refactor: add models and __pycache__ to .gitignore

* Better naming

* Update readme

* Move models ignore to it's folder

* Add scaffolding

* Apply formatting

* Fix tests

* Working sagemaker custom llm

* Fix linting

* Fix linting

* Enable streaming

* Allow all 3.11 python versions

* Use llama 2 prompt format and fix completion

* Restructure ()

Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local>

* Fix Dockerfile

* Use a specific build stage

* Cleanup

* Add FastAPI skeleton

* Cleanup openai package

* Fix DI and tests

* Split tests and tests with coverage

* Remove old scaffolding

* Add settings logic ()

* Add settings logic

* Add settings for sagemaker

---------

Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local>

* Local LLM ()

* Add settings logic

* Add settings for sagemaker

* Add settings-local-example.yaml

* Delete terraform files

* Refactor tests to use fixtures

* Join deltas

* Add local model support

---------

Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local>

* Update README.md

* Fix tests

* Version bump

* Enable simple llamaindex observability ()

* Enable simple llamaindex observability

* Improve code through linting

* Update README.md

* Move to async ()

* Migrate implementation to use asyncio

* Formatting

* Cleanup

* Linting

---------

Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local>

* Query Docs and gradio UI

* Remove unnecessary files

* Git ignore chromadb folder

* Async migration + DI Cleanup

* Fix tests

* Add integration test

* Use fastapi responses

* Retrieval service with partial implementation

* Cleanup

* Run formatter

* Fix types

* Fetch nodes asynchronously

* Install local dependencies in tests

* Install ui dependencies in tests

* Install dependencies for llama-cpp

* Fix sudo

* Attempt to fix cuda issues

* Attempt to fix cuda issues

* Try to reclaim some space from ubuntu machine

* Retrieval with context

* Fix lint and imports

* Fix mypy

* Make retrieval API a POST

* Make Completions body a dataclass

* Fix LLM chat message order

* Add Query Chunks to Gradio UI

* Improve rag query prompt

* Rollback CI Changes

* Move to sync code

* Using Llamaindex abstraction for query retrieval

* Fix types

* Default to CONDENSED chat mode for contextualized chat

* Rename route function

* Add Chat endpoint

* Remove webhooks

* Add IntelliJ run config to gitignore

* .gitignore applied

* Sync chat completion

* Refactor total

* Typo in context_files.py

* Add embeddings component and service

* Remove wrong dataclass from IngestService

* Filter by context file id implementation

* Fix typing

* Implement context_filter and separate from the bool use_context in the API

* Change chunks api to avoid conceptual class of the context concept

* Deprecate completions and fix tests

* Remove remaining dataclasses

* Use embedding component in ingest service

* Fix ingestion to have multipart and local upload

* Fix ingestion API

* Add chunk tests

* Add configurable paths

* Cleaning up

* Add more docs

* IngestResponse includes a list of IngestedDocs

* Use IngestedDoc in the Chunk document reference

* Rename ingest routes to ingest_router.py

* Fix test working directory for intellij

* Set testpaths for pytest

* Remove unused as_chat_engine

* Add .fleet ide to gitignore

* Make LLM and Embedding model configurable

* Fix imports and checks

* Let local_data folder exist empty in the repository

* Don't use certain metadata in LLM

* Remove long lines

* Fix windows installation

* Typos

* Update poetry.lock

* Add TODO for linux

* Script and first version of docs

* No jekill build

* Fix relative url to openapi json

* Change default docs values

* Move chromadb dependency to the general group

* Fix tests to use separate local_data

* Create CNAME

* Update CNAME

* Fix openapi.json relative path

* PrivateGPT logo

* WIP OpenAPI documentation metadata

* Add ingest script ()

* Add ingest script

* Fix broken name refactor

* Add ingest docs and Makefile script

* Linting

* Move transformers to main dependency

* Move torch to main dependencies

* Don't load HuggingFaceEmbedding in tests

* Fix lint

---------

Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local>

* Rename file to camel_case

* Commit settings-local.yaml

* Move documentation to public docs

* Fix docker image for linux

* Installation and Running the Server documentation

* Move back to docs folder, as it is the only supported by github pages

* Delete CNAME

* Create CNAME

* Delete CNAME

* Create CNAME

* Improved API documentation

* Fix lint

* Completions documentation

* Updated openapi scheme

* Ingestion API doc

* Minor doc changes

* Updated openapi scheme

* Chunks API documentation

* Embeddings and Health API, and homogeneous responses

* Revamp README with new skeleton of content

* More docs

* PrivateGPT logo

* Improve UI

* Update ingestion docu

* Update README with new sections

* Use context window in the retriever

* Gradio Documentation

* Add logo to UI

* Include Contributing and Community sections to README

* Update links to resources in the README

* Small README.md updates

* Wrap lines of README.md

* Don't put health under /v1

* Add copy button to Chat

* Architecture documentation

* Updated openapi.json

* Updated openapi.json

* Updated openapi.json

* Change UI label

* Update documentation

* Add releases link to README.md

* Gradio avatar and stop debug

* Readme update

* Clean old files

* Remove unused terraform checks

* Update twitter link.

* Disable minimum coverage

* Clean install message in README.md

---------

Co-authored-by: Pablo Orgaz <pablo@Pablos-MacBook-Pro.local>
Co-authored-by: Iván Martínez <ivanmartit@gmail.com>
Co-authored-by: RubenGuerrero <ruben.guerrero@boopos.com>
Co-authored-by: Daniel Gallego Vico <daniel.gallego@bq.com>
2023-10-19 16:04:35 +02:00
Iván Martínez
78d1ef44ad
Update README.md 2023-09-25 16:03:09 +02:00
Iván Martínez
0b5a6687e3
Merge pull request from imartinez/990-cannot-submit-more-than-166-embeddings-at-once-while-ingesting
Batch embeddings to be processed by chromadb
2023-09-25 11:59:19 +02:00