How to: Remove Duplicate Merge Indexes Using the Omit Operator

The Omit operator removes specific data points from a data set. A good example for removing unwanted data is when you're working with a Merge operator. When merging in a Data Workflow, you sometimes end up with duplicate indexes that you can remove using an Omit operator.

Configuration

In this how-to guide, you'll set up a data table to store some sample client information. Then, you'll use a Data Workflow to take two columns from that data table and merge them into an array. Then, use an Omit operator to remove the duplicate index from that array. You'll include two Console operators in your Data Workflow so you can view your data before and after omitting the duplicate index.

These instructions assume you have a new module open, saved, and with a title.

Configure the Data Table Component

First, let's set up some sample data in a Data Table component.

1. In the Module Builder, drag and drop a Data Table component onto your canvas.

2. In the Property ID field, enter dtClientInfo.

3. In the data table, enter the following:

   #	A	B	C	D
   1	firstName	lastName	city	email
   2	Sam	Smith	New York	sam@gmail.com
   3	Matt	Walensky	Brooklyn	matt@gmail.com
   4	Dafna	Klein	Chicago	dafna@gmail.com
   5	Olga	Apply	Boston	olga@gmail.com
   6	Shannon	Kempel	LA	shannon@gmail.com

   

4. Click Save Component.

Configure the Data Workflow Component

Now, let's set up your Data Workflow to separate the city and email data using two Pick operators. Then, you'll merge that data into its own data set using a Merge operator. To merge this data, you need to create an index for each item using two Create Index operators. Lastly, you’ll use an Omit operator to remove your duplicate merge indexes.

1. Drag and drop a Data Workflow component onto your canvas, placing it below your Data Table component.

2. In the Property ID field, enter dwfMergeAndOmit.

3. In the Canvas Label Text, enter dwfMergeAndOmit.

   

Configure the Input Operator

1. Drag and drop an Input operator onto your Data Workflow canvas.

2. Configure the operator's Info window as follows:

   Setting	Value
   Category	Input
   Component	dtClientInfo
   Required	Yes
   Source	Default

Configure the First Pick Operator

1. Drag and drop a Pick operator onto your Data Workflow canvas.

2. Configure the operator's Info window as follows:

   Setting	Value
   Category	Pick
   Label	Pick Email
   Pick Keys	email

3. Connect the output port (right) of the dtClientInfo Input operator to the input port (left) of the Pick Email Pick operator.

Configure the First Create Index Operator

1. Drag and drop a Create Index operator onto your Data Workflow canvas.

2. Configure the operator's Info window as follows:

   Setting	Value
   Category	Create Index
   Label	Email Index
   Index Name	rowId
   Starting Index	1
   Keys

3. Connect the output port (right) of the Pick Email Pick operator to the input port (left) of the Email Index Create Index operator.

Configure the Second Pick Operator

1. Drag and drop another Pick operator onto your Data Workflow canvas.

2. Configure the operator's Info window as follows:

   Setting	Value
   Category	Pick
   Label	Pick City
   Pick Keys	city

3. Connect the output port (right) of the dtClientInfo Input operator to the input port (left) of the Pick City Pick operator.

Configure the Second Create Index Operator

1. Drag and drop another Create Index operator onto your Data Workflow canvas.

2. Configure the operator's Info window as follows:

   Setting	Value
   Category	Create Index
   Label	City Index
   Index Name	rowId
   Starting Index	1
   Keys

3. Connect the output port (right) of the Pick City Pick operator to the input port (left) of the City Index Create Index operator.

Configure the Merge Operator

1. Drag and drop a Merge operator onto your Data Workflow canvas.

2. Configure the operator's Info window as follows:

   Setting	Value
   Category	Merge
   Label	Merge Email and City
   Merge Type	Inner Left
   Keep Flag	False
   Renaming	Replace
   Rows to Keep	First
   Merge Key	rowId

3. Connect the output port (right) of the Email Index Create Index operator to the upper input port (left) of the Merge Email and City Merge operator.

4. Connect the output port (right) of the City Index Create Index operator to the lower input port (left) of the Merge Email and City Merge operator.

Configure the First Console Operator

1. Drag and drop a Console operator onto your Data Workflow canvas.

2. Configure the operator's Info window as follows:

   Setting	Value
   Category	Console
   Label	Before Omit

3. Connect the output port (right) of the Merge Email and City Merge operator to the input port (left) of the Before Omit Console operator.

Configure the Omit Operator

1. Drag and drop an Omit operator onto your Data Workflow canvas.

2. Configure the operator's Info window as follows:

   Setting	Value
   Category	Omit
   Label	Omit Duplicate Index
   Omit Keys	rowId

3. Connect the output port (right) of the Merge Email and City Merge operator to the input port (left) of the Omit Duplicate Index Omit operator.

Configure the Second Console Operator

1. Drag and drop another Console operator onto your Data Workflow canvas.

2. Configure the operator's Info window as follows:

   Setting	Value
   Category	Console
   Label	After Omit

3. Connect the output port (right) of the Omit Duplicate Index Omit operator to the input port (left) of the After Omit Console operator.

4. Click Save Component.

Configure the Button Component

Lastly, let's add a Button component to trigger the entire operation.

1. Drag and drop a Button component onto your canvas, placing it below your Data Workflow.

2. In the Property ID field, enter btnMerge.

3. In the Label Text field, enter Merge.

4. Set the Action Type as Event.

5. From the On Click drop-down, select dwfMergeAndOmit.

   

6. Click Save Component.

7. Save your module.

Here's how your module looks in the Module Builder:

Preview your module in Express View and open the DevTools Console. Click the Merge button, and you'll see your data from your Data Workflow both before and after it passes through the Omit operator. In the Before Omit Console, you'll see the rowId key for each item in the array. In the After Omit Console, you'll see your data without the key.