Ingest

Ingest from a URL

Two doors, one lobby: import a document file by URL, or make a web page part of your knowledge — with typed cross-referral errors when you pick the wrong one.

Two doors, one decision

You have a URL. There are two different things you might be asking Calypso to do with it, and they are different promises:

	Import a file by URL	Add a web page
You're asking	"Copy this exact document"	"Make Calypso know this page"
Endpoint	`POST /v1/buckets/{bucket}/files/import`	`POST /v1/buckets/{bucket}/pages`
What runs	Fetch + store bytes + index	Crawl + LLM analysis (title, summary, tags generated)
Identity	Each import creates a new source	The normalized URL is the identity — one per team (`409 website_url_exists` on duplicates)
Capability	`knowledge:file:create`	`knowledge:website:create` — explicit-only (crawling runs third-party analysis per call)
Lifecycle	A snapshot, frozen at import time	A living page source

Whichever door you choose, the result converges: a source in your bucket, with the same status shape, the same ready bit, and the same citation attributes downstream.

Import a file by URL

POST /v1/buckets/{bucket}/files/import
Authorization: Bearer sk-...

{
  "url": "https://example.com/whitepaper.pdf",
  "title": "Product whitepaper",
  "tags": "product,launch"
}

Calypso fetches the URL server-side (public hosts only, http(s) only, 25 MB cap), stores the bytes durably, and queues indexing. The response returns the source id and task id; poll GET /v1/sources/{id} until ready, or attach a webhook.

Typed errors

Status	Code	Meaning
422	`url_is_web_page`	The URL serves HTML — use `/pages`, or pass `"force": true` to snapshot the HTML as a file. `details.suggested_endpoint` names the right door.
400	`url_fetch_blocked`	Rejected by fetch policy (not retryable).
404	`bucket_not_found`	The path bucket does not exist or is archived.
413	`file_too_large`	The remote file exceeds the 25 MB cap.
502	`url_fetch_failed`	The remote host failed to serve the file — retryable.

Add a web page

POST /v1/buckets/{bucket}/pages
Authorization: Bearer sk-...

{
  "url": "https://example.com/pricing",
  "tags_hint": "pricing, plans",
  "preferred_language": "en"
}

One shot: Calypso crawls the page, generates the title, summary, and tags, persists the page as knowledge, and indexes it into the bucket — pages are retrievable through bucket-scoped agents just like files.

/pages is the canonical name for what POST /v1/knowledge/websites does — the feature ingests one page. The /websites route remains a permanent alias; "website" is reserved for a future whole-site crawl.

Typed errors

Status	Code	Meaning
422	`url_is_file`	The URL names a document file (`.pdf`, `.docx`, …) — use `files/import`, or pass `"force": true` to crawl it as a page anyway.
400	`bucket_required`	No destination bucket (body-addressed alias route only — impossible on the bucket-addressed path).
404	`bucket_not_found`	The path bucket does not exist or is archived.
409	`website_url_exists`	This URL already exists for the team (details carry the source id).
502	`website_analysis_failed`	Crawl or analysis failed — retryable.

The wrong-door safety net

The API routes you at the moment of confusion instead of failing opaquely:

files/import fetches the URL, discovers text/html, and answers 422 url_is_web_page pointing at /pages.
/pages sees a URL path ending in a document extension and answers 422 url_is_file pointing at files/import.

Both errors include details.suggested_endpoint and details.force. Passing "force": true overrides the referral for the rare legitimate cross-cases (snapshotting an HTML file; crawling a URL that merely looks like a document).

Idempotency

Both doors accept an Idempotency-Key header. A retry with the same key replays the original result (200 + Idempotent-Replayed: true). On these two endpoints replay is keyed, not compared — reusing a key with a different body replays the original import rather than erroring (unlike the upload paths, which return 409 idempotency_key_conflict), so use a fresh key per logical operation. For pages the identity is also the normalized URL: the same URL with a different key (or none) returns 409 website_url_exists.

From MCP

The Calypso MCP server exposes both flows:

calypso-upload-file with sourceUrl — the backend fetches the URL; bytes never pass through the MCP process.
calypso-add-website — one-shot page ingestion with the same idempotency and conflict semantics.

Next steps

Overview

Files