Upload API

What the Upload API is for

The Upload API turns knowledge ingestion into an automatable workflow. Use it when source material is produced by another system — a CI job, crawler, MCP tool, or back-office workflow — and should land directly in the Calypso RAG indexing pipeline.

This page is the integration view: how to wire a pipeline end to end. For the full per-endpoint reference, see Ingest.

Choosing a path

All three land the file in the path bucket, queue indexing, and converge on the same source object with the same ready bit.

The 25 MB per-file cap applies to all three — a session does not buy headroom, it buys resumability and keeps your API requests small by sending bytes straight to storage. A session create that declares a size_bytes over the cap is rejected at create time with 413 file_too_large, before you upload anything.

Core flow

1. Resolve or provision the destination bucket.
2. Create a one-call, single-file session, or batch upload with a project API key.
3. For sessions, upload each file directly to the returned upload_url with PUT.
4. Finalize the session or batch with JSON.
5. Treat the finalize response as durable acceptance, not retrieval readiness.
6. Poll source or batch status until indexing and bucket sync settle.

Authorization

Use a project API key as a bearer token:

Authorization: Bearer sk-...

Uploads require the knowledge:file:create capability; provisioning a bucket requires knowledge:bucket:create. See Capabilities.

Choosing the bucket

The destination bucket is in the URL on every route on this page — there is no bucket field to send. {bucket} accepts a bucket id or a slug.

To discover valid destinations, list the buckets the key can reach:

GET /v1/buckets
Authorization: Bearer sk-...

The response is scoped to the key's team and returns bucket ids, slugs, status, counts, and bucket-store readiness.

For pipelines that must run against a bucket that may not exist yet, provision it explicitly first — this is idempotent and safe to call on every run:

PUT /v1/buckets/{slug}
Authorization: Bearer sk-...

It returns created: true the first time and created: false on every later call, including when two pipeline runs race each other. See Manage → Buckets.

Single-file upload

The one-call form:

curl -X POST "https://api.calypso.so/v1/buckets/contracts/files" \
  -H "Authorization: Bearer sk-..." \
  -H "Idempotency-Key: crm-doc-123" \
  -F "file=@/absolute/path/to/contract.pdf" \
  -F "title=Customer Contract" \
  -F "tags=legal,customer" \
  -F 'metadata={"external_id":"crm_123","source":"local_cli"}'

To upload bytes directly to storage instead — resumable, and it keeps the request off your API call — create a session:

POST /v1/buckets/contracts/uploads
Authorization: Bearer sk-...
Content-Type: application/json
Idempotency-Key: your-stable-upload-key

{
  "filename": "contract.pdf",
  "content_type": "application/pdf",
  "size_bytes": 184233,
  "title": "Customer Contract",
  "tags": ["legal", "customer"],
  "metadata": { "external_id": "crm_123", "source": "local_cli" }
}

Typical create response:

{
  "session_id": "sess_123",
  "upload_strategy": "gcs_resumable",
  "upload_url": "https://storage.googleapis.com/...",
  "expires_at": "2026-06-08T21:00:00Z",
  "request_id": "req_current"
}

Upload bytes to the returned URL:

curl -X PUT "$UPLOAD_URL" \
  -H "Content-Type: application/pdf" \
  -H "Content-Range: bytes 0-184232/184233" \
  --data-binary @/absolute/path/to/contract.pdf

Finalize:

POST /v1/buckets/contracts/uploads/{session_id}/finalize
Authorization: Bearer sk-...
Content-Type: application/json

{}

Typical finalize response:

{
  "id": "file_123",
  "object": "knowledge_file",
  "status": "queued",
  "title": "Customer Contract",
  "filename": "contract.pdf",
  "content_type": "application/pdf",
  "size_bytes": 184233,
  "sha256": "abc123...",
  "task": {
    "id": "task_123",
    "type": "gemini_file_search_indexing",
    "status": "queued"
  },
  "bucket_sync_status": "queued",
  "request_id": "req_current"
}

Batch upload

Create a batch session in the destination bucket:

POST /v1/buckets/rag1/batches
Authorization: Bearer sk-...
Content-Type: application/json

{
  "manifest": {
    "version": 1,
    "batch_idempotency_key": "local-batch-001",
    "items": [
      { "client_file_id": "contract_pdf", "filename": "contract.pdf", "title": "Customer Contract" },
      { "client_file_id": "invoice_pdf", "filename": "invoice.pdf", "title": "Customer Invoice" }
    ]
  },
  "files": [
    { "client_file_id": "contract_pdf", "filename": "contract.pdf", "content_type": "application/pdf", "size_bytes": 184233 },
    { "client_file_id": "invoice_pdf", "filename": "invoice.pdf", "content_type": "application/pdf", "size_bytes": 91234 }
  ]
}

Upload every accepted item to its returned upload_url, then finalize:

POST /v1/buckets/rag1/batches/{batch_id}/finalize
Authorization: Bearer sk-...
Content-Type: application/json

{
  "mode": "finalize_uploaded"
}

Typical batch finalize response:

{
  "batch_id": "batch_123",
  "status": "partial",
  "finalized": [
    { "client_file_id": "contract_pdf", "session_id": "sess_1", "knowledge_id": "file_123", "task_id": "task_123" }
  ],
  "pending": [
    { "client_file_id": "invoice_pdf", "session_id": "sess_2", "reason": "blob_not_uploaded" }
  ],
  "failed": [],
  "replayed": [],
  "request_id": "req_current"
}

Call finalize again after uploading pending items. Already-finalized items are replayed, not duplicated, and each finalize accumulates onto the earlier results rather than replacing them.

Status endpoints

Poll source status — the one bit that covers both indexing and bucket sync:

GET /v1/sources/{id}

Poll task status:

GET /v1/knowledge/tasks/{task_id}

Poll batch status:

GET /v1/knowledge/batches/{batch_id}?include_items=true

While items are still outstanding, batch status deliberately withholds a terminal value — a batch will not report itself finished early.

Readiness

A finalize response means the successfully finalized items were stored and had indexing queued — nothing more. Read it per item, not as a verdict on the whole call:

• finalized / replayed — stored and queued. These are the only items this section applies to.
• pending — not stored. The blob is missing, or the call ran out of its time budget. Upload what is missing and call finalize again.
• failed — not stored, and retrying finalize will not change that. Inspect the item error and re-create the item.

Even for finalized items, queued does not mean retrievable.

Wait for ready: true on GET /v1/sources/{id} before testing retrieval. For deeper debugging, status, task and bucket_sync_status remain available, and GET /v1/sources/{id}?verify=gemini returns live provider truth.

Retries and idempotency

Send a stable Idempotency-Key on every create call. A replayed key resolves before the per-minute byte quota is consumed, so retrying a completed upload does not spend quota and does not risk quota_exceeded. Replays answer 200 with an Idempotent-Replayed: true header.

Idempotency keys must be non-empty, length-bounded, and visible ASCII.

Limits and best practices

• Max file size: 25 MB, on every path — one-call, session, and batch alike.
• Keep batch size at or below 100 files.
• Respect the 5 create requests per second per team rate limit.
• Metadata max: 8 KB. Tags max: 20.
• Generate deterministic client_file_id values when retrying the same batch.
• Treat upload_url as a short-lived bearer capability and do not log it.
• Provision buckets explicitly with PUT /v1/buckets/{slug} rather than relying on implicit creation during uploads.