Oncrawl Data Reader

The Oncrawl Data Reader node queries pages, links, clusters or structured data of a completed Oncrawl crawl with an OQL filter validated against the crawl schema.

What does the Oncrawl Data Reader node do?#

The Oncrawl Data Reader queries the crawl data Oncrawl collected, through the official Data API. Four data types are available — pages, links, duplicate clusters and structured data — and the filter is written in OQL, Oncrawl’s own query language.

Where the Oncrawl KPI Reader answers “how many pages were fetched”, this node answers “which pages return a 404 below depth 3, and what are their titles”. It is the node to reach for when the question is specific.

Three design points matter in practice:

Field names are checked before the query runs. Oncrawl’s schema is per-crawl and case-sensitive: the fields available depend on how the crawl was configured, and Status_Code is not status_code. The node reads the crawl’s own schema first, so a wrong field is reported by name — with close matches suggested — instead of coming back as a generic API error.

The 10 000-item ceiling belongs to one transport, not to your dataset. Oncrawl caps a search query at 10 000 items regardless of the limit you ask for. With automatic pagination on — the default — the node pages itself until it has the rows you asked for, and switches to the export transport on its own when the search ceiling gets in the way, up to a 100 000-row safety cap. Knowing which transport applies is not the user’s job.

A partial result always says it is partial. Whatever stopped the read — the requested limit, the ceiling in the cases where it still applies, a failed transport switch — provenance.complete and an explicit warning say so. This matters because a truncated result looks exactly like a complete one, and a report built on the first 10 000 of four million rows is wrong without appearing to be.

Common use cases:

• Listing every page that returns an error, with its depth and title, for a technical SEO audit.
• Counting pages by status code or depth without downloading a single row.
• Exporting a full crawl dataset to a spreadsheet for further processing.
• Reviewing internal links or structured-data markup found during the crawl.
• Giving an LLM a precise, filtered page set instead of an entire crawl.

Quick setup#

The three modes#

Choosing the mode matters more than any other setting, because the three are different transports and are not subject to the same limits.

Mode	Returns	Limit	Use it when
Rows	Individual records, paginated automatically	100 000 rows with automatic pagination on; 10 000 without it, or when an offset is set	You need the actual rows
Aggregations	Counts computed by Oncrawl	None — nothing is downloaded row by row	You need totals, distributions or breakdowns
Full export	The matching dataset as streamed CSV or JSONL	100 000 lines or 10 MB, whichever comes first; past that the file is cut and truncated is true	You need the matching rows as a file, and sorting does not matter

Automatic pagination in Rows mode#

Oncrawl’s Data API offers two transports for rows, and they have different rules. A search query returns paginated JSON but stops answering past 10 000 items, whatever limit and offset ask for. An export query has no such ceiling, but it answers with a stream and rejects sort and offset.

With Automatic pagination on, which is the default, the node handles that split itself. It walks the search pages until it has the rows you asked for; if the 10 000-item ceiling gets in the way, it re-reads the same query through the export transport and parses the stream back into rows, up to a 100 000-row cap. The output records the switch with a warning: “More than 10,000 rows matched, so the read switched to the export transport automatically and returned 25,000 rows.”

Two things are worth knowing about that switch. A sort you configured is dropped, because the export transport rejects it — a warning says so, and you can order the rows downstream. And if the switch fails even after its retries, or the export brings back no more rows than the search already had — it stops at 10 MB, which wide rows can reach first — the rows the search had already returned are kept, with a warning saying the result is the capped search instead. Losing data that was already read would be worse than handing it over labelled as partial.

The 100 000-row cap is a safety limit of the node, not of Oncrawl. A read that stopped only when the dataset did would pull millions of rows into a single workflow payload; the cap turns that into a reported truncation instead.

Writing an OQL filter#

OQL is a JSON object, not a text expression. A single condition is a field leaf, and leaves compose with and and or.

{ "field": ["status_code", "equals", 404] }

{
  "and": [
    { "field": ["status_code", "gte", 400] },
    { "field": ["depth", "between", [1, 4]] }
  ]
}

Operators#

Group	Operators	Notes
Presence	has_value, has_no_value	Pass an empty string as the value
String	equals, contains, startswith, endswith	Each also accepts a not_ prefix, such as not_contains
Numeric	gt, gte, lt, lte, between	between takes a two-element array; the lower bound is inclusive, the upper exclusive

String operators accept a fourth element of options: {"ci": true} for a case-insensitive match, {"regex": true} when the value is a regular expression.

{ "field": ["urlpath", "startswith", "/blog/([0-9]+)/", { "regex": true }] }

Configuration parameters#

Required fields#

Optional fields#

Output#

In Rows mode:

{
  "provenance": {
    "crawl_id": "abc123",
    "data_type": "pages",
    "mode": "query",
    "offset": 0,
    "returned": 842,
    "total": 842,
    "complete": true,
    "truncated": false,
    "pagination": {
      "strategy": "search",
      "requests": 1,
      "auto_paginate": true
    },
    "warnings": [],
    "retrieved_at": "2026-10-20T14:00:00+00:00",
    "source": "Oncrawl API v2"
  },
  "rows": [{ "url": "/a", "status_code": 404 }]
}

Always check complete. It is the authoritative answer to “did this read see every matching row”, and when it is false a warning explains what stopped the read. truncated is simply its opposite, kept for workflows that already read it.

Note that honouring your limit is not the same as completeness. Asking for 10 rows of a filter that matches 128 828 returns complete: false with the warning “The read stopped at the requested limit of 10 rows and more rows match it. Raise the limit, or narrow the OQL filter.” That is deliberate: the node reports what it saw of the dataset, not whether it did as it was told.

The pagination block explains how the rows were obtained: strategy is search or export, requests counts the calls spent by the transport that produced them — useful when a read is slow, an export stream counting as one — and auto_paginate echoes the setting. After a switch to the export transport, total keeps the count the search reported, since the export stream carries none.

Aggregations mode returns an aggs array with Oncrawl’s cols / rows structure. Full export mode returns content, the streamed file as text, plus a line_count.

Troubleshooting#

Field 'status_kode' does not exist in this crawl's schema

Cause: The field name is misspelled, or absent from this crawl’s configuration.

Solution: The message lists close matches — use one of them. The Fields selector shows the exact spelling for this crawl.

Did you mean 'status_code'? Field names are case-sensitive

Cause: The field exists, but with different capitalisation. Oncrawl’s schema is case-sensitive.

Solution: Use the exact spelling from the message.

Field 'body_text' exists but cannot be filtered on in this crawl

Cause: Not every field in the schema is filterable; Oncrawl marks which ones are.

Solution: The message lists filterable fields. Pick one of those, or return the field without filtering on it.

Operator 'contains' is not available on field 'fetched'

Cause: The operator does not apply to that field’s type — contains is a string operator, fetched is a boolean.

Solution: The message lists the operators the crawl allows on that field.

Oncrawl caps a search query at 10,000 items, and that ceiling was reached

Cause: More than 10 000 rows matched, and the read could not switch to the export transport that lifts the ceiling — either because Automatic pagination is off, or because an Offset is set, which the export transport rejects. The result is partial.

Solution: Turn Automatic pagination on and remove the offset, which lets the read go up to 100 000 rows on its own. Otherwise narrow the OQL filter, switch to Aggregations if you only need counts, or use Full export to get every row as a file.

The read stopped at the requested limit of N rows and more rows match it

Cause: The read did exactly what you asked, but your filter matches more rows than the limit — so the result is a slice of the dataset, not the dataset. complete is false for that reason.

Solution: Raise the Limit (up to 100 000 with automatic pagination on), narrow the OQL filter, or treat the slice as a sample on purpose. provenance.total tells you how many rows actually match.

Oncrawl rejects an offset on the export transport, so the read could not go past the 10,000-item ceiling

Cause: An offset was set on a read that needed more than 10 000 rows. The export transport is the only way past the ceiling and it refuses an offset, so the node kept the offset and the ceiling.

Solution: Remove the offset. It exists to page manually, which automatic pagination makes unnecessary; with it gone the read reaches up to 100 000 rows in one go.

Oncrawl rejects a sort on the export transport, so the sort order was dropped

Cause: The read passed the 10 000-item ceiling and switched to the export transport, which does not accept a sort.

Solution: Sort the rows downstream, or lower the limit below 10 000 if the order matters more than the volume.

The read tried to go past the 10,000-item ceiling through the export transport and it failed

Cause: The switch to the export transport failed even after the node’s retries, or the export returned no more rows than the search already had. Rather than failing the run, the node returned the rows the search had already read, which is why truncated is true.

Solution: Re-run later. If the warning says the export returned no more rows than the search, select fewer Fields so each row is smaller. Otherwise narrow the OQL filter so the read fits under 10 000 rows and never needs the switch.

'links' is a high-volume data type and no OQL filter was given

Cause: links and structured_data routinely hold millions of rows, and the query has no filter.

Solution: Add an OQL filter, or switch to Aggregations. Paging through millions of links row by row is not viable.

Missing segmentation name in query

Cause: The query references a segmentation — what Oncrawl also calls page groups. A segmentation is defined on the project, not on the crawl, and it has to be named for its groups to be resolved. This node queries one crawl’s own data, so it has no slot to name one.

Most often the culprit is a selected field, not the filter: Oncrawl’s schema lists segmentation fields alongside ordinary ones, and selecting one is enough to fail the whole query. The node now hides them from the Fields picker and refuses them by name before the call, so this error should only reach you from an OQL filter written by hand, or from a node saved before that check existed.

Solution: Remove the segmentation field from both the filter and the selected fields, and use plain crawl fields such as urlpath, depth or status_code. To reproduce a page group, express its rule directly in OQL: a group covering a directory becomes {"field": ["urlpath", "startswith", "/blog/"]}. The error message echoes the query that was sent, which is how you find the offending field.

Oncrawl crawl is not completed

Cause: The crawl is still running, or it failed. Only a completed crawl has a queryable dataset.

Solution: Wait for it to finish, or use an Oncrawl Crawl Resolver, which only ever returns completed crawls.

Oncrawl refused the request (403)

Cause: Oncrawl recognised the token but refused the action. The message names the reason Oncrawl gave:

• quota_error — the account’s quota does not allow the request.
• no_active_subscription — the account has no active subscription.
• feature_not_available — the plan does not include this feature, for example the API option.
• otherwise — the token lacks the projects:read scope, or access to this project.

Solution: Free up or raise the quota, renew the subscription or the plan, or regenerate the token with the projects:read scope on an account that can see the project.

Oncrawl rate limit exceeded (429)

Cause: Oncrawl answered 429 to three attempts in a row. The node retries a 429 or a 5xx error twice, a few seconds apart, before giving up. Oncrawl does not publish its rate limit.

Solution: Re-run later, and avoid starting several workflows that read Oncrawl at the same moment.

Best practices and pitfalls#

How does it fit into a workflow?#

A typical technical-audit chain, with the resolver keeping the data fresh:

graph LR
    Resolver[Oncrawl Crawl Resolver] --> Reader[Oncrawl Data Reader<br/>pages, status_code >= 400]
    Reader --> Extractor[JSON Path Extractor<br/>rows]
    Extractor --> LLM[LLM<br/>prioritise the fixes]
    LLM --> Email[Email Sender]

Related nodes#