Documents #

Documents are the knowledge base that powers the chat widget and semantic search. Every document is stored as markdown content with automatically generated vector embeddings for semantic retrieval.

Overview #

The document system handles the full lifecycle: ingestion (from raw content or URLs), storage, embedding generation, semantic search indexing, and deletion. All operations are available via API with API key authentication.

API Access #

All document endpoints support API key authentication:

• GET /docs — List documents with pagination
• GET /docs/:id — Get a single document
• POST /docs/ingest — Ingest single or batch (raw content or URLs, auto-scrape, embeddings auto-generated)
• PUT /docs/:id — Update a document
• DELETE /docs/:id — Delete a document and its embeddings
• POST /docs/batch-delete — Delete multiple documents at once

See the API Reference for full details.

Ingestion #

Raw Content #

Send markdown content directly. Embeddings are generated automatically on creation and update.

curl -X POST https://api.canny.bot/api/docs/ingest \
  -H "x-api-key: <your_api_key>" \
  -H "Content-Type: application/json" \
  -d '{"title": "Product: Widget Pro", "content": "The Widget Pro is our flagship product..."}'

URL Auto-Scrape #

Provide a URL without content and the system scrapes it using Firecrawl v2 with:

• onlyMainContent — removes headers, navs, footers
• onlyCleanContent — LLM-based pass removing residual boilerplate (cookie banners, ads, social widgets)
• removeBase64Images — strips bloated base64 images
• Image references (![alt](url)) are stripped from the final markdown

curl -X POST https://api.canny.bot/api/docs/ingest \
  -H "x-api-key: <your_api_key>" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products/widget-pro"}'

Batch Ingestion #

Send an array to ingest multiple documents in a single request. All items are processed in parallel.

curl -X POST https://api.canny.bot/api/docs/ingest \
  -H "x-api-key: <your_api_key>" \
  -H "Content-Type: application/json" \
  -d '[
    {"title": "Widget Pro", "content": "..."},
    {"url": "https://example.com/products/widget-lite"},
    {"title": "Gadget Max", "content": "...", "url": "https://...", "is_dynamic": true, "renew_interval_hours": 48}
  ]'

Dynamic Documents #

Documents can be marked as dynamic to enable automatic content refresh from the source URL:

• is_dynamic: true — enables auto-refresh
• renew_interval_hours — hours between refreshes (default: 24, range: 1-720)

Dynamic documents are re-scraped and re-indexed at the configured interval. If a refresh fails, the last successful content is retained.

Embeddings #

Embeddings are generated automatically using the semantic search plugin's indexer service:

• On create: embeddings are generated for the content field, using title as metadata
• On update: embeddings are re-generated when content changes
• On delete: embeddings are removed before the document is deleted

Embedding generation is non-blocking — if it fails, the document is still created/updated and an error is logged.

Agentic Use Cases #

Import a Product Catalog from an External API #

1. Fetch products from your product database API
2. Map each product to { title, content, url } format
3. Send the array to POST /docs/ingest in a single request
4. Verify with GET /docs and test with GET /semantic-search/search

Sync Documents from a CMS #

1. List existing documents via GET /docs
2. Compare with your CMS content
3. Create new documents via POST /docs/ingest
4. Update changed documents via PUT /docs/:id
5. Delete removed documents via POST /docs/batch-delete

Best Practices #

• Use batch ingestion for bulk imports — it's faster than individual create calls
• Provide URLs when possible — the system handles scraping and markdown cleaning automatically
• Check account usage before large imports via GET /account/status
• Test search after ingestion via GET /semantic-search/search to verify embeddings work
• Use batch delete for cleanup — it removes embeddings in parallel