Ingest overview

The one rule of ingestion

Every durable source lives in a bucket, and the bucket is in the URL. A bucket is a named collection of knowledge (support-handbook, pricing, legal) that agents search as a unit. Because the destination is part of the address, it is impossible to upload knowledge "nowhere":

POST /v1/buckets/{bucket}/files            Upload a file (one call, ≤25MB)
POST /v1/buckets/{bucket}/files/import     Import a file from a URL
POST /v1/buckets/{bucket}/pages            Ingest a web page
POST /v1/buckets/{bucket}/uploads          Upload session (large files)
POST /v1/buckets/{bucket}/batches          Batch upload (1–100 files)

{bucket} accepts a bucket id or slug. Authenticate every call with a project API key:

Authorization: Bearer sk-...

Choosing a door

If you post a URL to the wrong door, the API redirects you with a typed error rather than failing opaquely: file import returns 422 url_is_web_page for HTML pages, and page ingestion returns 422 url_is_file for document URLs — each carrying details.suggested_endpoint. See Ingest from a URL.

Readiness: one bit

Ingestion is asynchronous: accepting a source (201/202) means it is stored and queued, not yet searchable. Every source status response now carries a single readiness bit:

GET /v1/sources/{id}

{
  "id": "file_123",
  "status": "active",
  "ready": true,
  "bucket_sync_status": "active"
}

ready is the whole readiness contract: it flips to true only after both indexing and bucket synchronization complete, so a ready source is retrievable through its bucket right now with nothing further to check. For debugging, the detailed ladder is still there: status (queued → indexing → active/failed), task, bucket_sync_status, and GET /v1/sources/{id}?verify=gemini for live provider truth.

Adding files to a bucket that already has indexed content never interrupts answers — queries keep serving from what is indexed while new sources join as they become ready. Only a brand-new (or fully emptied) bucket has a short window with no grounded answers until its first source indexes.

Webhooks instead of polling

URL-based ingest requests (files/import, pages) accept a webhook, called when the source reaches a terminal state:

{
  "url": "https://example.com/whitepaper.pdf",
  "webhook": {
    "url": "https://hooks.example.com/calypso",
    "metadata": { "job_id": "sync-42" }
  }
}

Your endpoint receives source.ready or source.failed with your metadata echoed back, plus a stable event id (evt_…). Delivery is at-least-once (one retry, and rare duplicates are possible) — deduplicate on the event id or on source_id + event. Polling remains the source of truth. Sources whose indexing retries are exhausted receive source.failed with max_attempts_exhausted.

Bucket membership

GET    /v1/buckets/{bucket}/sources          List members (id, type, status, ready)
DELETE /v1/buckets/{bucket}/sources/{id}     Detach from this bucket only
DELETE /v1/sources/{id}                      Delete the source everywhere

Detaching removes the source from one bucket; the canonical source and its other bucket memberships are untouched. Global delete removes storage, index copies, and all memberships. Bucket creation and listing live under Manage → Buckets.

Source types in buckets

Idempotency

Every ingest create accepts an Idempotency-Key header, scoped to the calling API key. A retry with the same key and the same request replays the original result with Idempotent-Replayed: true. The two families differ on what happens when the body changes:

• Upload paths (one-call files, upload sessions, batches) validate the request against the original: reusing a key with a different body returns 409 idempotency_key_conflict.
• URL-based imports (files/import, pages) replay by key alone without comparing the body — reusing a key with different values replays the original import, so use a fresh key per logical operation.

A replay of a completed upload is resolved before the per-minute byte quota window is consumed, so retrying costs no quota and cannot fail with quota_exceeded for bytes it never re-sent. A rejected upload never claims its key either — if a create is refused by validation or a quota guard, the same key is immediately reusable.

Keys must be non-empty, length-bounded, and visible ASCII; anything else is 400 invalid_idempotency_key.

Next:

• Ingest from a URL
• Files
• Batches
• Manage buckets