Javier Martinez
091d5f7020
fix: random bugs ( #2301 )
...
* fix: celery callbacks
* fix: s3 + skill creator
* fix: resumable when there's params
* fix: add distributed cache
* fix: do durable context stack
* fix: mcp tools
* fix: present server tool
* fix: add cache to the skills
* fix: mcp
* fix: mypy
2026-07-16 09:14:43 +02:00
Javier Martinez
576589bdc5
feat: add websearch in a sandbox ( #2300 )
...
* feat: add websearch in a sandbox
* fix: celery callbacks
(cherry picked from commit fe7d0ce9d5
2026-07-16 08:39:52 +02:00
Javier Martinez
21d42fd97a
feat: code execution v4 ( #2295 )
...
* feat: add bundle to remove
* fix: spaces
* feat: add xml render as skill spec
* feat: add skill volume root to cache
* fix: deduplicate values
* fix: change the mount path to skill_id
* fix: use different paths
* feat: add principal
(cherry picked from commit 5db64fe721d5706440ce9f342b1388ffe742bc16)
# Conflicts:
# private_gpt/components/code_execution/base.py
# private_gpt/components/code_execution/code_execution_component.py
# private_gpt/components/code_execution/local.py
# private_gpt/components/environment/manager.py
# private_gpt/components/tools/builders/bash_tool_builder.py
# private_gpt/components/tools/builders/text_editor_tool_builder.py
# private_gpt/components/tools/processors/bash_processor.py
* fix: mypy
...
* fix: config
* fix: sandbox config
* feat: add forward cookies
* fix: loop
* feat: add present server
* feat: add feature flag for tools
* refactor: move principal to another better place
* feat: add api key principal
* docs: fix docs
* docs: add present server
* fix: principal
* fix: mypy
* fix: avoid to block the loop
* fix: blocks in expansion
* fix: remove maximum concurrent users
...
* fix: multiplexer
* fix: readers
* fix: more fixes
...
* fix: impl
* feat: tool scheduler
* feat: add adaptative
* feat: add chat worker
* fix: config
* fix: max
* feat: add chat/tools workers
* fix: mypy
* feat: add generic scheduler
* fix: get result
* feat: do serializable the tool executor
* fix: tools
* fix: config
* fix: config
* fix: args
* fix: config
* fix: serializer
* Revert "fix: blocks in expansion"
This reverts commit a2110f94a8175728472+Copilot@users.noreply.github.com >
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* Handle PGPT_WORKER_MODE=celery in health check worker status
* fix: cancel
* fix: arch
* fix: test ingestion
* fix: deserialization of chat messages
* fix: broken results
* fix: mypy
* fix: test
* fix: config
* fix: remove arq tool worker
* fix: output cls
* fix: preserve early resumable tool callbacks
* fix: preserve async tool result order
* refactor: address worker PR review comments
* fix: mypy
* test: colocate ARQ chat enqueue coverage
* fix: remove redis from tests
* fix: mypy
* fix: tests
* test: isolate chat mocks and cancellation timing
* fix: tests
* fix: tests
* fix: test
(cherry picked from commit f8ee460af21115ff2349175728472+Copilot@users.noreply.github.com >
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
2026-07-15 13:05:23 +02:00
Javier Martinez
cd8ca2214a
feat: resumable chat worker + tool worker + async tokenizer ( #2298 )
...
* fix: avoid to block the loop
* fix: blocks in expansion
* fix: remove maximum concurrent users
...
* fix: multiplexer
* fix: readers
* fix: more fixes
...
* fix: impl
* feat: tool scheduler
* feat: add adaptative
* feat: add chat worker
* fix: max
* feat: add chat/tools workers
* fix: mypy
* feat: add generic scheduler
* fix: get result
* feat: do serializable the tool executor
* fix: tools
* fix: config
* fix: config
* fix: args
* fix: config
* fix: serializer
* Revert "fix: blocks in expansion"
This reverts commit a2110f94a8175728472+Copilot@users.noreply.github.com >
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* Handle PGPT_WORKER_MODE=celery in health check worker status
* fix: cancel
* fix: arch
* fix: test ingestion
* fix: deserialization of chat messages
* fix: broken results
* fix: mypy
* fix: test
* fix: config
* fix: remove arq tool worker
* fix: output cls
* fix: preserve early resumable tool callbacks
* fix: preserve async tool result order
* refactor: address worker PR review comments
* fix: mypy
* test: colocate ARQ chat enqueue coverage
* fix: remove redis from tests
* test: isolate chat mocks and cancellation timing
* fix: tests
(cherry picked from commit 218b599c66fc5ec0f72a1371c275a1175728472+Copilot@users.noreply.github.com >
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
2026-07-15 12:12:45 +02:00
Javier Martinez
bfd2633d89
feat: add code execution ui ( #2289 )
2026-07-06 11:20:22 +02:00
Javier Martinez
f2cffd1ab9
feat: code execution v2 ( #2278 )
...
* fix: ensure that the publisher is ready to publish new messages
* fix: dockerfile
* fix: clones
(cherry picked from commit bc0a77e050175728472+Copilot@users.noreply.github.com >
* feat: M3 persistent sessions + M4 isolated bash + OpenSandbox provider
Introduces mount-aware sessions with canonical path abstraction:
- ContentBundle / SessionMount abstraction for skill/plugin mounting
- PathTranslator: rewrites commands and scrubs output (canonical ↔ real paths)
- LocalMount + ReadOnlyMount: local FS-backed mounts; read-only cache shared across sessions
- BashExecutor: asyncio subprocess with setsid + setrlimit isolation + killpg on timeout
- SkillLoader: downloads skill files from object storage as ContentBundles
- LocalCodeExecutionProvider: rewritten to use mounts, TTL reaper, BashExecutor
- OpenSandboxCodeExecutionProvider: new Docker/K8s backend via opensandbox SDK
- Async cascade: create_session / get_or_create_session / build_tool all async
- ObjectStorage.list_files() added to ABC + both implementations
- New settings: session_ttl_seconds, bash rlimit fields, OpenSandboxSettings
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
* refactor: generic ContentBundle with BundledFile, remove skill_filter from code_execution layer
- Replace ContentBundle/dataclass with pydantic BaseModel; add BundledFile with path, content, permissions
- Update ReadOnlyMount to use list[BundledFile] with per-file chmod
- Remove skill_filter and SkillLoader from create_session() in base, local, and code_execution_component
- Move skill-to-bundle resolution into BashToolBuilder (inject SkillLoader there)
- Update SkillLoader.load() to return list[BundledFile] objects
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
* fix: remove opensandbox
* feat: use sandbox inside of code executor
* fix: mypy
* feat: add workspace manager
* feat: improvements
* feat: add env abstraction
* feat: remove leftovers
* feat: improve skill content
* fix: macos
* fix: mypy
* feat: add code executor prompt
* feat: add content bundle in stack
* fix: mypy
* fix: layout
* feat: add skill prompt
* fix: order
* feat: allow to present final files
* feat: add container block
* fix: remove default config
* fix: session pers
* fix: do lazy env
* fix: mounter
* feat: refactor mounter
* fix: container
* feat: add requirements
* fix: add container registry
* fix: move to be lazy
* fix: mypy
* fix: current folder
* fix: stop sandbox in tabular
* fi: ensure to use absolute paths in bash & text editor
* feat: add files router
* feat: simplify
* feat: update present files
* fix: download files
* fix: mypy
* fix: tests
* fix: tests
* fix: bash executor in linux
* Revert "fix: bash executor in linux"
This reverts commit 483e208a9650d9288f5e175728472+Copilot@users.noreply.github.com >
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-06-29 18:29:29 +02:00
Javier Martinez
4cca7d0b97
feat: allow to attach files as Anthropic does ( #2265 )
...
* feat: add convert/parser service
(cherry picked from commit 2350ad0a3e60291fbb0ded1d1f1d1c784dcc6b3c)
* fix: mypy & test
(cherry picked from commit 4e5358ee156dc11400b6f1fdbcc71a58220a581c)
* feat: add initial content
(cherry picked from commit 6eaeb4f1a8e20ce09fb208aca8468185d3866e3c)
* fix: return content as tree
* docs: add convert examples
* feat: add parallel support
* feat: add concurrency in multimodal interceptor
* refactor: move config to a common class
* fix: apply preprocessors only over the last user message
* fix: final tweaks to allow to configure feature flag
* fix: support legacy blocks
* test: fix
* fix: mypy
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* fix: image/audio in the chat
* fix: copilot
* fix: name
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
2026-06-25 15:35:46 +02:00
Javier Martinez
42a6c047d0
feat: add validation skill endpoint + skill prompt ( #2272 )
...
* feat: add skill validator endpoint
* fix: skills
* feat: add skill prompt
(cherry picked from commit c36d27f008175728472+Copilot@users.noreply.github.com >
* fix: clarify skill visibility comment
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
* fix: rename has_loaded_non_eager variable to snake_case
* test: cover skill validation route
* test: fix invalid skill validation case
* test: cover skill tool visibility states
* feat: improve skill tools + prompt
* fix: mypy
* feat: allow to limit the skill maximum size
* feat: add validation error codes
* fix: tests
* fix: invalid frontmatter
* fix: edge cases
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
2026-06-25 15:35:26 +02:00
Alfonso Lozana
78a1e8b9f6
feat: control docs parser ( #2273 )
...
* fix: ensure that the publisher is ready to publish new messages
* fix: dockerfile
* fix: clones
(cherry picked from commit bc0a77e050175728472+Copilot@users.noreply.github.com >
* fix: add ExtractionUnsuccessfulError in docling reader
* feat: add vision pipeline
* feat: add fall back with vision
* feat: add settings of enable_vision_fallback
* doc: update doc
* chore: code refactor
* feat: remove pypdfium2 dependency
* fix: add settings and change only doc vision behaviour
* feat: change extraction_type_override to skip_strategy_inference (disable strategy on vision transform parser)
* chore: make check
---------
Co-authored-by: Javier Martinez <javiermartinezalvarez98@gmail.com >
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com >
2026-06-24 08:22:17 +02:00
Alfonso Lozana
5d60141f08
feat: improve web search ( #2274 )
...
* fix:[ZYL-3553] web scraper memory (#693 )
* fix: web scraper memory
* chore: make check
chore: fix make check errors
chore: fix make check
fix: async_playwright for test
chore: fix ruff check -
chore: try to fix mypy
chore: more fix...
chore: add playwright to evaluation
fix: mypy
Revert "chore: add playwright to evaluation"
This reverts commit d916b56dc144a405ae0ff0103d1b377764f1ecad.
# Conflicts:
# private_gpt/components/tools/builders/web_fetch_builder.py
# private_gpt/components/web/web_scraper_service.py
# private_gpt/components/web/web_search/processors/clean_content.py
# private_gpt/components/web/web_search/processors/select_best_links.py
# private_gpt/components/web/web_search/web_search_service.py
# private_gpt/server/tools/tool_service.py
* feat: improve web scraper pool (#695 )
* feat: [ZYL-3673] web search improve quality (#697 )
* feat: improve web scraper pool
* feat: improve select best links
* fix: add settings of context
* fix: brave config (#699 )
---------
Co-authored-by: Javier Martinez <javiermartinezalvarez98@gmail.com >
2026-06-24 08:20:24 +02:00
Javier Martinez
4302bd629a
fix: add missing docker database deps ( #2264 )
...
* fix: update deps names in dockerfile
* fix: missing prefix
2026-06-08 13:50:24 +02:00
Javier Martinez
183cd03857
feat!: PrivateGPT revamp v1 ( #2230 )
...
* feat!: PrivateGPT revamp v1
* chore(docs): update nodejs
2026-06-02 16:55:46 +02:00
J
fa3c30661d
fix: Add default mode option to settings ( #2078 )
...
* Add default mode option to settings
* Revise default_mode to Literal (enum) and add to settings.yaml
* Revise to pass make check/test
* Default mode: RAG
---------
Co-authored-by: Jason <jason@sowinsight.solutions >
2024-09-24 08:33:02 +02:00
Javier Martinez
54659588b5
fix: nomic embeddings ( #2030 )
...
* fix: allow to configure trust_remote_code
based on: https://github.com/zylon-ai/private-gpt/issues/1893#issuecomment-2118629391
* fix: nomic hf embeddings
2024-08-01 09:43:30 +02:00
Javier Martinez
8119842ae6
feat(recipe): add our first recipe Summarize ( #2028 )
...
* feat: add summary recipe
* test: add summary tests
* docs: move all recipes docs
* docs: add recipes and summarize doc
* docs: update openapi reference
* refactor: split method in two method (summary)
* feat: add initial summarize ui
* feat: add mode explanation
* fix: mypy
* feat: allow to configure async property in summarize
* refactor: move modes to enum and update mode explanations
* docs: fix url
* docs: remove list-llm pages
* docs: remove double header
* fix: summary description
2024-07-31 16:53:27 +02:00
Javier Martinez
40638a18a5
fix: unify embedding models ( #2027 )
...
* feat: unify embedding model to nomic
* docs: add embedding dimensions mismatch
* docs: fix fern
2024-07-31 14:35:46 +02:00
Javier Martinez
9027d695c1
feat: make llama3.1 as default ( #2022 )
...
* feat: change ollama default model to llama3.1
* chore: bump versions
* feat: Change default model in local mode to llama3.1
* chore: make sure last poetry version is used
* fix: mypy
* fix: do not add BOS (with last llamacpp-python version)
2024-07-31 14:35:36 +02:00
Javier Martinez
e54a8fe043
fix: prevent to ingest local files (by default) ( #2010 )
...
* feat: prevent to local ingestion (by default) and add white-list
* docs: add local ingestion warning
* docs: add missing comment
* fix: update exception error
* fix: black
2024-07-31 14:33:46 +02:00
Javier Martinez
20bad17c98
feat(llm): autopull ollama models ( #2019 )
...
* chore: update ollama (llm)
* feat: allow to autopull ollama models
* fix: mypy
* chore: install always ollama client
* refactor: check connection and pull ollama method to utils
* docs: update ollama config with autopulling info
2024-07-29 13:25:42 +02:00
Jackson
43cc31f740
feat(vectordb): Milvus vector db Integration ( #1996 )
...
* integrate Milvus into Private GPT
* adjust milvus settings
* update doc info and reformat
* adjust milvus initialization
* adjust import error
* mionr update
* adjust format
* adjust the db storing path
* update doc
2024-07-18 10:55:45 +02:00
Javier Martinez
01b7ccd064
fix(config): make tokenizer optional and include a troubleshooting doc ( #1998 )
...
* docs: add troubleshooting
* fix: pass HF token to setup script and prevent to download tokenizer when it is empty
* fix: improve log and disable specific tokenizer by default
* chore: change HF_TOKEN environment to be aligned with default config
* ifx: mypy
2024-07-17 10:06:27 +02:00
Proger666
2612928839
feat(vectorstore): Add clickhouse support as vectore store ( #1883 )
...
* Added ClickHouse vector sotre support
* port fix
* updated lock file
* fix: mypy
* fix: mypy
---------
Co-authored-by: Valery Denisov <valerydenisov@double.cloud >
Co-authored-by: Javier Martinez <javiermartinezalvarez98@gmail.com >
2024-07-08 16:18:22 +02:00
uw4
fc13368bc7
feat(llm): Support for Google Gemini LLMs and Embeddings ( #1965 )
...
* Support for Google Gemini LLMs and Embeddings
Initial support for Gemini, enables usage of Google LLMs and embedding models (see settings-gemini.yaml)
Install via
poetry install --extras "llms-gemini embeddings-gemini"
Notes:
* had to bump llama-index-core to later version that supports Gemini
* poetry --no-update did not work: Gemini/llama_index seem to require more (transient) updates to make it work...
* fix: crash when gemini is not selected
* docs: add gemini llm
---------
Co-authored-by: Javier Martinez <javiermartinezalvarez98@gmail.com >
2024-07-08 11:47:36 +02:00
Yevhenii Semendiak
3b3e96ad6c
Allow parameterizing OpenAI embeddings component (api_base, key, model) ( #1920 )
...
* Allow parameterizing OpenAI embeddings component (api_base, key, model)
* Update settings
* Update description
2024-05-17 09:52:50 +02:00
Fran García
966af4771d
fix(settings): enable cors by default so it will work when using ts sdk (spa) ( #1925 )
2024-05-10 14:13:46 +02:00
icsy7867
e21bf20c10
feat: prompt_style applied to all LLMs + extra LLM params. ( #1835 )
...
* Updated prompt_style to be moved to the main LLM setting since all LLMs from llama_index can utilize this. I also included temperature, context window size, max_tokens, max_new_tokens into the openailike to help ensure the settings are consistent from the other implementations.
* Removed prompt_style from llamacpp entirely
* Fixed settings-local.yaml to include prompt_style in the LLM settings instead of llamacpp.
2024-04-30 09:53:10 +02:00
imartinez
49ef729abc
Allow passing HF access token to download tokenizer. Fallback to default tokenizer.
2024-04-19 15:38:25 +02:00
imartinez
f469b4619d
Add required Ollama setting
2024-04-02 18:27:57 +02:00
Robin Boone
b3b0140e24
feat(llm): Ollama LLM-Embeddings decouple + longer keep_alive settings ( #1800 )
2024-04-02 16:23:10 +02:00
machatschek
83adc12a8e
feat(RAG): Introduce SentenceTransformer Reranker ( #1810 )
2024-04-02 10:29:51 +02:00
icsy7867
087cb0b7b7
feat(rag): expose similarity_top_k and similarity_score to settings ( #1771 )
...
* Added RAG settings to settings.py, vector_store and chat_service to add similarity_top_k and similarity_score
* Updated settings in vector and chat service per Ivans request
* Updated code for mypy
2024-03-20 22:25:26 +01:00
Iván Martínez
6f6c785dac
feat(llm): Ollama timeout setting ( #1773 )
...
* added request_timeout to ollama, default set to 30.0 in settings.yaml and settings-ollama.yaml
* Update settings-ollama.yaml
* Update settings.yaml
* updated settings.py and tidied up settings-ollama-yaml
* feat(UI): Faster startup and document listing (#1763 )
* fix(ingest): update script label (#1770 )
huggingface -> Hugging Face
* Fix lint errors
---------
Co-authored-by: Stephen Gresham <steve@gresham.id.au >
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com >
2024-03-20 21:33:46 +01:00
Otto L
1efac6a3fe
feat(llm - embed): Add support for Azure OpenAI ( #1698 )
...
* Add support for Azure OpenAI
* fix: wrong default api_version
Should be dashes instead of underscores.
see: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference
* fix: code styling
applied "make check" changes
* refactor: extend documentation
* mention azopenai as available option and extras
* add recommended section
* include settings-azopenai.yaml configuration file
* fix: documentation
2024-03-15 16:49:50 +01:00
Brett England
63de7e4930
feat: unify settings for vector and nodestore connections to PostgreSQL ( #1730 )
...
* Unify pgvector and postgres connection settings
* Remove local changes
* Update file pgvector->postgres
2024-03-15 09:55:17 +01:00
Brett England
68b3a34b03
feat(nodestore): add Postgres for the doc and index store ( #1706 )
...
* Adding Postgres for the doc and index store
* Adding documentation. Rename postgres database local->simple. Postgres storage dependencies
* Update documentation for postgres storage
* Renaming feature to nodestore
* update docstore -> nodestore in doc
* missed some docstore changes in doc
* Updated poetry.lock
* Formatting updates to pass ruff/black checks
* Correction to unreachable code!
* Format adjustment to pass black test
* Adjust extra inclusion name for vector pg
* extra dep change for pg vector
* storage-postgres -> storage-nodestore-postgres
* Hash change on poetry lock
2024-03-14 17:12:33 +01:00
Iván Martínez
d17c34e81a
fix(settings): set default tokenizer to avoid running make setup fail ( #1709 )
2024-03-13 09:53:40 +01:00
icsy7867
02dc83e8e9
feat(llm): adds serveral settings for llamacpp and ollama ( #1703 )
2024-03-11 22:51:05 +01:00
Iván Martínez
45f05711eb
feat: Upgrade to LlamaIndex to 0.10 ( #1663 )
...
* Extract optional dependencies
* Separate local mode into llms-llama-cpp and embeddings-huggingface for clarity
* Support Ollama embeddings
* Upgrade to llamaindex 0.10.14. Remove legacy use of ServiceContext in ContextChatEngine
* Fix vector retriever filters
2024-03-06 17:51:30 +01:00
TQ
cd40e3982b
feat(Vector): support pgvector ( #1624 )
2024-02-20 15:29:26 +01:00
Iván Martínez
aa13afde07
feat(UI): Select file to Query or Delete + Delete ALL ( #1612 )
...
---------
Co-authored-by: Robin Boone <rboone@sofics.com >
2024-02-16 17:36:09 +01:00
Ygal Blum
6bbec79583
feat(llm): Add support for Ollama LLM ( #1526 )
2024-02-09 15:50:50 +01:00
CognitiveTech
e326126d0d
feat: add mistral + chatml prompts ( #1426 )
2024-01-16 22:51:14 +01:00
imartinez
fee9f08ef3
Move back to 3900 for the context window to avoid melting local machines
2023-12-22 18:21:43 +01:00
Iván Martínez
4780540870
feat(settings): Configurable context_window and tokenizer ( #1437 )
2023-12-21 14:49:35 +01:00
Iván Martínez
8ec7cf49f4
feat(settings): Update default model to TheBloke/Mistral-7B-Instruct-v0.2-GGUF ( #1415 )
...
* Update LlamaCPP dependency
* Default to TheBloke/Mistral-7B-Instruct-v0.2-GGUF
* Fix API docs
2023-12-17 16:11:08 +01:00
cognitivetech
2564f8d2bb
fix(settings): correct yaml multiline string ( #1403 )
2023-12-16 19:02:46 +01:00
3ly-13
145f3ec9f4
feat(ui): Allows User to Set System Prompt via "Additional Options" in Chat Interface ( #1353 )
2023-12-10 19:45:14 +01:00
3ly-13
a072a40a7c
Allow setting OpenAI model in settings ( #1386 )
...
feat(settings): Allow setting openai model to be used. Default to GPT 3.5
2023-12-09 20:13:00 +01:00
lopagela
56af625d71
Fix the parallel ingestion mode, and make it available through conf ( #1336 )
...
* Fix the parallel ingestion mode, and make it available through conf
Also updated the documentation to show how to configure the ingest mode.
* PR feedback: redirect to documentation
2023-11-30 11:41:55 +01:00
Iván Martínez
64ed9cd872
Allow passing a system prompt ( #1318 )
2023-11-29 15:51:19 +01:00
