Content extraction APIGuide

URL to Markdown API for AI pipelines

Turn a public webpage or raw HTML into readable Markdown with one API call. Remove navigation, keep useful document structure and measure the token difference before passing content to your agent or RAG pipeline.

Primary use
Convert a public webpage or raw HTML to Markdown with token metrics. Try the free API and integrate with Python, JavaScript or n8n.
Recommended flow
Fetch, clean, measure tokens, then hand consistent Markdown to agents or retrieval systems.
Next step
Use the Playground to compare raw HTML against optimized output before integrating the API.

Convert a public URL to Markdown

The public endpoint is available for integration testing without an API key. Send a JSON request to POST /api/v1/convert and read the markdown field in the response. Try the live playground to inspect the result before integrating.

Working URL request
curl -X POST https://aingestor.com/api/v1/convert \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/",
    "options": {
      "remove_navigation": true,
      "remove_ads": true,
      "preserve_code_blocks": true
    }
  }'

Use raw HTML when you already have the document

Send html instead of url when your application already has the source. This avoids an extra fetch and makes controlled examples reproducible. Provide exactly one source field per request; do not send URL and HTML together.

Raw HTML request
curl -X POST https://aingestor.com/api/v1/convert \
  -H 'Content-Type: application/json' \
  -d '{"html":"<main><h1>Hello</h1><p>Useful content for an agent.</p></main>"}'

Keep content and token metrics together

The response includes Markdown, optional title and description, tokens_before, tokens_after, reduction_percent, processing_ms and a request_id. Inspect tables, lists and code blocks on your own document types before relying on unattended processing.

  • Read and store Markdown in your own pipeline.
  • Compare token counts using the same cl100k_base tokenizer.
  • Keep the request ID to investigate a conversion.
  • Treat estimated dollar savings as an illustration, not your model's actual bill.

Integrate with Python, JavaScript or n8n

Use a standard HTTP client; there is no proprietary SDK to install. The API documentation includes copyable Python and JavaScript requests plus an n8n HTTP Request workflow. Route errors separately from successful Markdown and use bounded retries for transient failures.

For RAG, convert first and then pass the result to your own chunking, embedding and retrieval steps. For documentation, start with one useful page and check heading, table and code fidelity before expanding.

Know which sources are supported

URL mode fetches public HTTP/HTTPS HTML; it does not execute browser JavaScript or bypass login, CAPTCHA or paywalls. Pages that block automated access may fail. Use raw HTML you are permitted to process, or another supported source.

This API handles one document per request. It does not crawl an entire website, create embeddings or implement semantic chunking. See limits and error handling before using a source at scale.

Evaluate the result before a production rollout

Use the sandbox samples to compare navigation removal, documentation code and tables. Then evaluate representative pages from your actual workflow. Token reduction varies with page structure and is not a fixed percentage.

For production traffic, tell us your expected volume and source types. We can assess fit and integration requirements before you rely on the service.

FAQ

Can I use it without an API key?

The current public endpoint is available for integration testing without an API key, subject to rate and size limits.

Does it render JavaScript or crawl a whole site?

No. It fetches one public HTML document or accepts pasted HTML. JavaScript rendering, multi-page crawling and authenticated browsing are not part of this endpoint.

Does the response include token savings?

Yes. It includes before/after cl100k_base token counts and a percentage. Actual results depend on your input; the estimated dollar figure is illustrative.

What if the source site cannot be fetched?

Check the structured error code. A blocked, unavailable or non-HTML source may fail. Retry only transient errors with backoff, try another permitted source or send raw HTML.