Data Comparison

🔍

Data Comparison

Reconcile two datasets on composite keys

The Data Comparison node compares a source dataset against a target dataset and reports how they differ. Records are matched on one or more composite key fields, then compared field by field. The result classifies every record as matched, mismatched, missing, extra, or duplicated — making the node well suited to reconciliation, migration validation, and data-quality checks.

Common uses include:

• Reconciling records between two systems after a migration.
• Validating that an export matches its source of truth.
• Detecting rows that exist in one dataset but not the other.
• Flagging fields whose values differ between source and target.

Before You Start

What you may need	Where to set it up	Why it matters
The datasets to compare	Files	Reference each dataset by path or file id when comparing files rather than inline data.
Common key and field names	Global Variables	Keep uniqueKeys and field lists consistent across comparison workflows.

Actions

Action	Description
dataComparison:compareData	Compare a source dataset against a target dataset on composite keys

---

dataComparison:compareData

Compares the source and target datasets record by record. Each record is identified by the combination of uniqueKeys; matched records are then compared on fieldsToCompare (or all shared fields). Optional normalization makes the comparison tolerant of case or whitespace differences. All configuration lives under the config.dataCompareDetails root.

Datasets can be supplied inline, or read from files. When reading from files, the supported types are .json, .csv, .xlsx, and .xls.

Parameters

Parameter	Type	Required	Default	Description
inputMode	string	No	raw	How datasets are supplied: raw, filePath, or fileId
normalization	string	No	none	Value normalization before comparison: none, case, whitespace, or both
uniqueKeys	array or CSV	Yes	--	Composite key fields used to match records. Empty produces a VALIDATION_ERROR
fieldsToCompare	array or CSV	No	all shared fields	Fields compared on matched records
fieldMappings	object	No	{}	Map of source field name to target field name
sourceData	JSON array	If inputMode is raw	--	Inline source records
targetData	JSON array	If inputMode is raw	--	Inline target records
sourceFile	path	If inputMode is filePath	--	Source dataset path (read via the artifact client)
targetFile	path	If inputMode is filePath	--	Target dataset path (read via the artifact client)
sourceFileId	string	If inputMode is fileId	--	Source file identifier (read via the web client helper)
targetFileId	string	If inputMode is fileId	--	Target file identifier (read via the web client helper)

Example: Reconcile customer records after a migration

Field	Value
inputMode	raw
uniqueKeys	customerId
fieldsToCompare	name, status
normalization	both
sourceData	[{"customerId": "1", "name": "Acme", "status": "active"}]
targetData	[{"customerId": "1", "name": "acme ", "status": "active"}]

What happens:

1. The node keys both datasets by customerId.
2. It normalizes values for case and whitespace before comparing.
3. Record 1 matches on the key and compares equal on name and status.
4. The record is reported as matched; the brief and detailed results summarize the outcome.

---

Output

The action returns a JSON result with a summary (briefResult) and a per-record breakdown (detailedResult).

Field	Type	Description
briefResult.isSuccess	boolean	true when the comparison completed
briefResult.result	string	Overall outcome of the comparison
briefResult.sourceRecords	integer	Number of records in the source dataset
briefResult.targetRecords	integer	Number of records in the target dataset
briefResult.matched	integer	Records present in both with equal compared fields
briefResult.mismatches	integer	Records present in both but with differing fields
briefResult.missing	integer	Records in the source but not the target
briefResult.extra	integer	Records in the target but not the source
briefResult.duplicates	integer	Records with duplicate composite keys
detailedResult	array	Per-record comparison detail

Example output:

{
  "briefResult": {
    "isSuccess": true,
    "result": "COMPLETED",
    "sourceRecords": 100,
    "targetRecords": 98,
    "matched": 95,
    "mismatches": 3,
    "missing": 2,
    "extra": 0,
    "duplicates": 0
  },
  "detailedResult": [
    {"key": "1001", "status": "matched"},
    {"key": "1002", "status": "mismatch", "fields": {"status": {"source": "active", "target": "inactive"}}}
  ]
}

---

Common Error Scenarios

Error	Likely Cause	What to Check
VALIDATION_ERROR	uniqueKeys is empty, or a required dataset input is missing	Provide at least one unique key and the source/target inputs for the chosen inputMode
VALIDATION_ERROR	inputMode set but matching inputs not supplied	Supply sourceData/targetData for raw, sourceFile/targetFile for filePath, or sourceFileId/targetFileId for fileId
COMPARISON_ERROR	A file could not be read or parsed	Confirm the file exists and is a supported type (.json, .csv, .xlsx, .xls). Retry if the failure is transient

---

Best Practices

• Choose keys that are truly unique. Composite keys with duplicates inflate the duplicates count and skew matching. Combine fields until each record is uniquely identified.
• Normalize when sources disagree on formatting. Use case or both when the same value appears with different casing or trailing spaces across systems.
• Limit compared fields. Set fieldsToCompare to the fields that matter so incidental differences (timestamps, ids) do not register as mismatches.
• Map renamed columns. Use fieldMappings when source and target label the same data differently, so records still align.
• Read the brief first. briefResult gives the reconciliation at a glance; drill into detailedResult only for the records that differ.