Data Hub Writer

The Data Hub Writer node inserts, updates, merges or deletes rows in your organization's Data Hub tables, without any integration to configure.

What does the Data Hub Writer node do?

The Data Hub Writer node runs an INSERT, UPDATE, MERGE or DELETE query on the tables of your organization’s Data Hub. It works like the Google BigQuery Writer, with the same SQL editor, the same array handling and the same output, but needs no BigQuery integration: the query runs with your organization’s own Data Hub identity, which can only write to your organization’s datasets.

Info

The node is available only to organizations with Data Hub enabled. Contact your DNG administrator to activate it.

Common use cases:

  • Saving content generated per product (titles, descriptions) into a Data Hub table
  • Storing classifications or scores computed by a workflow run from a Data Hub pipeline
  • Updating or merging rows of a Data Hub table from a workflow

Quick setup

Add the node to the canvas

Find Data Hub Writer in the Integrations section of the node library and drop it on the canvas.

Choose the query type

Pick INSERT, UPDATE, MERGE or DELETE.

Write the SQL query

Open the SQL editor and click Tables to browse your Data Hub tables, switch location (EU, US), filter by name or label, and click a table to insert its full name (for example `dng_org_123456789_eu.product_content`) at the cursor. Use {{variables}} for values coming from upstream nodes. Validate and Run test check the query with a dry run: nothing is written from the editor.

Connect the output

The output port emits the execution result (affected rows, job IDs) as JSON.

Configuration parameters

Required fields

query_type string required

Query Type — insert, update, merge or delete. Must match the first keyword of the query.

sql_query string required

SQL Query — The DML query to run on your Data Hub tables, referenced as dataset.table. Template variables like {{sku_id}} are resolved at runtime. A variable that resolves to an array of JSON objects is rendered as a multi-row VALUES clause and split into several queries when it is too large.

INSERT INTO dng_org_123456789_eu.product_content (sku_id, new_title, updated_at)
VALUES ('{{sku_id}}', r"""{{new_title}}""", CURRENT_TIMESTAMP())

What does the node output?

execution_result string

A JSON string with the query type, the number of affected rows, the BigQuery job IDs and the number of chunks sent.

{
  "success": true,
  "query_type": "insert",
  "job_id": "job_abc123",
  "job_ids": ["job_abc123"],
  "num_dml_affected_rows": 1,
  "total_bytes_processed": 0,
  "chunks": 1
}

Usage examples

Example 1: Save a generated title per product

Run the workflow from a Data Hub pipeline batch stage (one run per product), generate the title with an LLM, then:

  • Query Type: insert
  • SQL Query: 
    INSERT INTO dng_org_123456789_eu.product_content (sku_id, new_title, updated_at)
VALUES ('{{sku_id}}', r"""{{new_title}}""", CURRENT_TIMESTAMP())

Example 2: Upsert a list of results

A Code Block returns the results as a JSON array in items:

  • Query Type: merge
  • SQL Query: 
    MERGE dng_org_123456789_eu.gpc_results T
USING (
  SELECT JSON_VALUE(item, '$.sku_id') AS sku_id,
         JSON_VALUE(item, '$.gpc_path') AS gpc_path
  FROM UNNEST(JSON_QUERY_ARRAY(r'''{{items}}''')) AS item
) S
ON T.sku_id = S.sku_id
WHEN MATCHED THEN UPDATE SET gpc_path = S.gpc_path
WHEN NOT MATCHED THEN INSERT (sku_id, gpc_path) VALUES (S.sku_id, S.gpc_path)

Common issues

Data Hub is not enabled for this organization

Cause: Data Hub is not activated for your organization, or it was deactivated.

Solution: Contact your DNG administrator to activate Data Hub.

Access Denied on a table

Cause: The query targets a dataset outside your organization’s Data Hub.

Solution: Only your organization’s Data Hub tables can be written. Insert the table from the Tables panel of the SQL editor.

Query must start with INSERT, UPDATE, MERGE, or DELETE

Cause: The query is a SELECT or another statement.

Solution: Use the Data Hub Reader to read data; this node only writes.

Best practices

Tip

Prefer one array over many small writes. Bind a {{rows}} variable to an array of JSON objects: the node writes them in a few queries instead of one per row.

Warning

Writes are not atomic across chunks. When a large array is split into several queries, a failure leaves the earlier chunks written.

Data Hub Reader

Query your organization’s Data Hub tables.
Google BigQuery Writer

Write to BigQuery projects outside Data Hub through a BigQuery integration.
Loop

Write one row per item of a list.