Deduplicate
Remove duplicate items from an array by comparing specific fields, all fields, or all fields except excluded ones.
The Deduplicate node removes duplicate entries from an array. You can compare items by a specific field, by all fields (exact match), or by all fields except a set you exclude. When duplicates are found, you control whether the first or last occurrence is kept. Use it after merging datasets, importing CSVs, or collecting webhook events where duplicates are expected.
Configuration
Source Array
| Field | Type | Required | Description |
|---|---|---|---|
| Source Array | template string | Yes | Variable path to the array to deduplicate, e.g. nodes.merge_1.merged |
The source must resolve to an array. If it resolves to a non-array value, the node fails with a NOT_ARRAY error.
Compare Mode
| Value | Behavior |
|---|---|
| By Specific Field | Compare items using a single named property |
| All Fields (exact match) | Compare the full JSON serialization of each item |
| All Fields Except... | Compare all fields except the ones you exclude |
Defaults to By Specific Field.
Deduplicate Key
Only shown when Compare Mode is By Specific Field.
| Field | Type | Required | Description |
|---|---|---|---|
| Deduplicate Key | text | No | Object property to check for duplicates (e.g. email, id). Leave empty for primitive value comparison. |
Except Fields
Only shown when Compare Mode is All Fields Except....
| Field | Type | Required | Description |
|---|---|---|---|
| Except Fields | text | No | Comma-separated field names to exclude from comparison (e.g. id, updatedAt). |
Keep
| Value | Behavior |
|---|---|
| Keep first occurrence | Retains the first item encountered and discards later duplicates |
| Keep last occurrence | Retains the last item encountered and discards earlier duplicates |
Defaults to Keep first occurrence.
Output
| Field | Type | Description |
|---|---|---|
| unique | array | The deduplicated array |
| duplicateCount | number | Number of duplicate items that were removed |
Example
Two API sources return user lists that may overlap. After merging them, you want to ensure each email address appears only once:
- Add a Merge node to combine both API results into a single array.
- Add a Deduplicate node after the merge.
- Set Source Array to
nodes.merge_1.merged. - Set Compare Mode to By Specific Field.
- Set Deduplicate Key to
email. - Set Keep to Keep first occurrence.
The output at nodes.removeDuplicates_1.unique contains one entry per unique email address, and nodes.removeDuplicates_1.duplicateCount tells you how many duplicates were removed.
See the Logic Nodes overview for a full list of available nodes.