* feat: split vector_index_task into parse_task + extract_task
Split the single vector_index_task in extraction_tasks.py into two
atomic, independently-retryable tasks on the same queue:
- parse_task (private_gpt.ingestion.parse): validates and parses the
source file into tree nodes, serialises them into nodes_json on the
body, then dispatches extract_task on the same ingestion queue.
When parse_only=True it returns nodes_json directly (used by the
chat document preprocessing path) without dispatching extract_task.
- extract_task (private_gpt.ingestion.extract): deserialises the nodes
from body.nodes_json and runs load_index (embedding + persistence).
Both tasks share IngestAsyncBody (two new optional fields: nodes_json
and parse_only) so no extra models or files are needed.
CeleryIngestionScheduler.ingest_async dispatches parse_task instead of
vector_index_task; the existing vector_index_task is kept unchanged for
the synchronous blocking ingest() path.
IngestionTaskHelper.revoke_ingestion_task now also cancels pending or
running parse_task and extract_task instances.
BaseIngestionScheduler gains bytes_to_text() satisfying the
DocumentConverter protocol. LocalIngestionScheduler delegates to
ConvertService; CeleryIngestionScheduler dispatches a parse_only
parse_task and converts the returned nodes to plain text in-process.
DocumentFilePreprocessingInterceptor now injects IngestionSchedulerFactory
instead of ConvertService, and preprocess_document_history accepts the
DocumentConverter protocol instead of the concrete ConvertService type.
* refactor: clean split into parse_task + store_vectors_task in extraction_tasks.py
Rewrote the previous approach to be simpler and self-contained:
- Everything lives in extraction_tasks.py — no extra files or models.
- parse_task (private_gpt.ingestion.parse): validates + parses the file,
serialises nodes into body.nodes_json, then dispatches store_vectors_task
on the same ingestion queue. When body.parse_only=True the dispatch is
skipped and nodes_json is returned directly (chat bytes_to_text path).
- store_vectors_task (private_gpt.ingestion.store_vectors): deserialises
body.nodes_json and runs load_index. This is the terminal task that
triggers the final done/error AMQP callback.
- Both tasks set callback_task_name=VECTOR_INDEX_CALLBACK_TASK_NAME so
all progress, done and error events share the pgpt.vector_index_task.*
prefix — consumers see a unified event stream.
- vector_index_task is kept unchanged for the synchronous blocking path.
- IngestionTaskHelper.revoke_ingestion_task now covers all three task
names using the same IngestAsyncBody isinstance check.
* refactor: nodes as list[BaseNode] (pickle), drop parse_only and vector_index_task
- nodes field on IngestAsyncBody is now list[BaseNode] | None — Celery uses
pickle so the objects pass through unchanged, no dict/JSON round-trip needed.
- parse_only removed from IngestAsyncBody; it was an implementation detail of
the scheduler that had no business on the API model. bytes_to_text on both
schedulers now delegates directly to ConvertService.
- vector_index_task removed; _dispatch_sync_ingest dispatches parse_task,
waits for it (parse is fast), extracts the store_vectors task-id from the
result, and returns an AsyncResult for it — so ingest() and
ingest_for_request() poll store_vectors completion exactly as before.
- store_vectors_task reads body.nodes directly, no deserialization step.
* fix: reuse parse_task for chat document conversion via async scheduler
The chat attach-document path was still parsing in-process: the Celery
scheduler's bytes_to_text delegated to ConvertService, so the arq chat
worker ran ParseComponent directly instead of going through the ingestion
scheduler.
- parse_task gains a dispatch_store kwarg (default True). When False it
returns the parsed plain text via ConvertService and skips the
store_vectors dispatch, so the same task serves both the ingestion
pipeline and the chat convert path.
- CeleryIngestionScheduler.bytes_to_text now dispatches parse_task with
dispatch_store=False on the ingestion queue and returns the text, so
attached documents are converted on the Celery worker, not in the chat
worker.
- IngestAsyncBody keeps nodes typed as list[BaseNode] and carries
__getstate__/__setstate__ pickle hooks: Celery transports task args with
pickle, which chokes on the circular parent/children refs of TreeNode.
The hooks serialise nodes to their model_dump() dicts on the wire and
rebuild them on receipt, keeping the Celery limitation inside the model
abstraction instead of leaking list[dict] into the public type.
* fix: multimodality
* fix: update tests to use parse_task instead of removed vector_index_task
* fix: update tests and scheduler for parse_task + store_vectors_task split
- test_task_registry: update expected task names from vector_index to
parse + store_vectors.
- test_ingestion_scheduler: fix dispatch_task mock to return a proper
result (ready=True, failed=False, result='store-task-id') so the
_dispatch_sync_ingest poll loop exits cleanly.
- ingest_async: wait for parse_task and return the store_vectors task_id
so the status endpoint resolves to IngestResponse, not the intermediate
string. Same pattern as _dispatch_sync_ingest.
- parse_task: clear body.callback before returning so parse_task's
after_return hook does not fire an intermediate AMQP notification with
the store task-id as data; store_vectors_task owns the final callback.
test_ingest_uri_async and test_di are pre-existing failures on main
(unrelated to this branch).
* fix: restore test_ingest_uri_async — pin injector to preserve mock broker
StatelessBackgroundTask.create_application_injector() replaces the global
injector with a fresh one during eager task execution, making the mock
BrokerComponent unreachable from task_after_return. Patch
create_application_injector to return the test injector so the mock broker
is reachable and the AMQP callback assertion can fire.
The test also now explicitly documents that the event type must be
pgpt.vector_index_task.done (the legacy name) so existing consumers are
not broken by the parse_task / store_vectors_task split.
* fix: ingest_async is truly fire-and-forget, sync path waits on store_vectors
The three callers have distinct semantics:
- ingest_async: fire-and-forget. Dispatches parse_task, returns its
task_id immediately without waiting. parse_task dispatches
store_vectors_task internally; the AMQP done/error callback fires from
store_vectors_task under the legacy pgpt.vector_index_task.* name.
- ingest / ingest_for_request (sync path via _dispatch_sync_ingest):
must block until ingestion is complete. Dispatches parse_task, waits
for it (parsing only — fast), extracts the store_vectors task_id from
its result, then polls that AsyncResult for completion.
- bytes_to_text (convert path): dispatches parse_task(dispatch_store=False),
waits for it, returns the text. Unchanged.
test_ingest_uri_async updated: in eager mode parse_task's result is the
store_vectors task_id string; the test now follows the chain —
status(parse_task_id) → store_task_id → status(store_task_id) →
IngestResponse — which mirrors real client behaviour.
* refactor: normalize BaseIngestionScheduler to parse/store/ingest/delete × sync/async
Each operation now comes in two flavours — sync (runs to completion,
returns the result) and async (fire-and-forget, returns a task-id):
parse / parse_async
store / store_async
ingest / ingest_async (parse + store combined)
delete / delete_async
bytes_to_text (DocumentConverter protocol, always sync)
LocalIngestionScheduler implements all operations in-process.
CeleryIngestionScheduler dispatches each step to the configured queue:
- ingest_async: dispatches parse_task (which chains store_vectors_task)
and returns the parse task-id immediately — fire-and-forget.
- ingest / ingest_for_request: dispatches parse_task, waits for it,
then polls the store_vectors AsyncResult to completion.
- parse_async: dispatches parse_task with dispatch_store=False, returns
the parse task-id.
- store_async: dispatches store_vectors_task directly.
- bytes_to_text: dispatches parse_task(dispatch_store=False), waits,
returns plain text.
_dispatch_sync_ingest removed — its logic now lives in ingest().
Tests updated to use the public ingest() method and to stub both the
dispatch_task mock and the AsyncResult store poll.
* fix: rebuild httpx2 exceptions to an controllable exception
* fix: avoid to create loop before to control the env
* chore: enable verbose
* chore: run all tests
Add a provider guide showing how to configure PrivateGPT with DaoXE,
a multi-model multi-protocol AI API gateway at https://daoxe.com/v1.
Includes setup steps, advanced model profile examples, and a card in
the providers overview under a new "Cloud gateways" section.
Co-authored-by: hei <hei@heideMacBook-Pro.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Provide a clear private disclosure path so researchers are not forced
to file public issues when Private Vulnerability Reporting is not
enabled.
Fixes#2304
Signed-off-by: Solaris-star <820622658@qq.com>
* feat: add convert/parser service
(cherry picked from commit 2350ad0a3e60291fbb0ded1d1f1d1c784dcc6b3c)
* fix: mypy & test
(cherry picked from commit 4e5358ee156dc11400b6f1fdbcc71a58220a581c)
* feat: add initial content
(cherry picked from commit 6eaeb4f1a8e20ce09fb208aca8468185d3866e3c)
* fix: return content as tree
* docs: add convert examples
* feat: add parallel support
* feat: add concurrency in multimodal interceptor
* refactor: move config to a common class
* fix: apply preprocessors only over the last user message
* fix: final tweaks to allow to configure feature flag
* fix: support legacy blocks
* test: fix
* fix: mypy
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix: image/audio in the chat
* fix: copilot
* fix: name
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
create_chat_header_prompt rendered current_date with sub-second
datetime.now().isoformat() into the system prompt header, before the
guidelines and retrieved-context blocks. Since it changes every request,
the system-prompt prefix is never byte-stable, which defeats LLM
prompt-prefix caching (OpenAI automatic prefix caching, Anthropic
cache_control, local KV-cache reuse).
Use date-level granularity; the model only needs the calendar date for
relative-date reasoning and the prefix stays stable within a day.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix: ensure that the publisher is ready to publish new messages
* fix: dockerfile
* fix: clones
(cherry picked from commit bc0a77e05010cfb628842787954dc181323b8607)
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* feat: use default user folder
* docs: update references to local paths
* fix: windows
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix: align make test and wipe with PGPT_HOME paths
* fix: align wipe target with PGPT_HOME local_data
* fix: folders
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
