Data Workflow Operators

Estimated Reading Time:  26 minutes

Overview

Your Unqork applications rely on data to function. The Data Workflow component is the space where you input data, map it, and manipulate it into what you want. Operators are the individual components of your workflow. They let you manipulate large and complex data structures from multiple sources. Much like components in a module, you'll string together operators to create workflows.

Common uses of operators include:

  • Inputting data.

  • Unwinding complex data structures.

  • Filtering data to obtain specific items.

  • Viewing data at different points in the workflow.

  • Appending data items to create new structures.

  • Outputting data.

Here's an extract of a Data Workflow canvas showing an orchestration of operators. We've pulled this extract from the Address Parsing snippet.

As you can see, the Input operator brings in the data from an Address Search component. The Unwind operator separates the individual data points. And the remaining operators take each data point and output them to different fields.

What You'll Learn

In this article, you’ll learn:

Understanding What Operator Ports Do

Let's take a closer look at an operator. Do you see the small extensions around the edges? These are your operator's ports. Lines drawn between these ports connect your operators.

The three types of ports are:

  • Input

  • Argument

  • Output

An operator can have more than one input or output port. Below are the ports you'll see associated with a Filter operator.

Input

The input port is the entry point into your operator. The previous operator in the workflow connects to the next operator's input port.

Some data operations have two inputs. For example, the Merge operator, which combines two inputted data sets. In this instance, a data set enters Input 1, and another data set enters Input 2.

Argument

Arguments add a dynamic aspect to data processing. They can create conditions, rules, and more functionality for an operator. When you bring an input into an operator's top port, you're bringing the input in as an argument.

It's important to make a distinction between an argument and an expression. Most operators have an Expression or Key field that controls what the operator does. For example, say you have a data set of savings accounts and checking accounts. You want to filter the two account types into separate outputs using a Filter operator. Without an argument, your expression might be account="savings". From there, the savings accounts filter out from an Output port. Anything that isn't a savings account filters out from the other. The operator performs a single operation. If you want an operator to always have the same function, there's no need for an argument.

Where an argument comes into play is when you want different functions based on dynamic data. Let's use the example from earlier, where you have a data set of savings and checking accounts. With an argument, you can make the filter more complex. You could add more filtering options, sorting accounts by their size. You could also add logic to your filter, calculating multiple account sizes. Or, you could set up decisions based on the accounts your end-user selects.

NOTE  Unqork references an argument as _arg in an expression.

Output

Outputs are where you send the final result of the data operation. You'll connect one output to the next via the right-side ports.

Some data operations have two outputs. For example, Filter, which separates data and outputs into multiple locations. Output 1 would be all the expression results that are true, or what you filtered for. Output 2 would be the expression results that are all false, or what you filtered out.

Connecting Operators

Remember, you connect operators with lines drawn from an output port to an input or Argument port. This is commonly referred to as "wiring" your operators. In the following example, the output port (right) of the Input operator connects to the input port (left) of the Col2Array operator.

Grouping Operators

It's easy to build complex systems using the Data Workflow component. But, you might find yourself with a canvas crowded with operators and connecting lines. Grouping operators together can help you organize your Data Workflow. This creates a clearer process flow and helps keep track of your operators' functions.

Let's look at the Address Parsing snippet again as an example. By default, its Data Workflow canvas looks like this:

After grouping operators you get this much easier to follow layout:

To group Data Workflow operators together:

1. Hold down the Shift key while clicking on the Data Workflow operators you want to group.
2. Press the Ctrl + G keys (PC) or Cmd + G keys (Mac). The selected operators group together, then collapse into an expandable frame.

TIP  You can expand or collapse the frame by clicking the plus (+) or minus (-) sign in the frame's top bar.

3. In the Info window that appears, enter a Label for your group of operations.
4. Click Save.

List of Operators For The Data Workflow Component

Here is the full list of the operators you can use in the Data Workflow component, organized by category.

Frequently Used

We put together a list of operators used most often. We chose these from various Data Workflow canvas categories.

Operator

Description

Aggregate

You have a few options for combining elements:

  • Sum By: Finds the sum of all numbers in a given range.

  • Count By: Counts the total number of values in a given range.

  • Mean By: Calculates the average of values in a given range.

  • Max By/Min By: Finds the highest or lowest value in a given range.

In Value Key, enter the column header (key) to indicate the range you want to perform your selected operation on.

Clean Keys

Fills in missing keys or empty values in each row of a table.

  • Missing: Creates a key when it's missing from a row. In the Keys setting, add the key name you'd like to add. Under Fill With, enter the value to add to the key.

  • Empty: Fills in values for keys that exist but have empty values. Under Keys, add the key name you'd like to fill when empty. Under Fill With, enter the value to add.

  • Missing or Empty: Fills values for rows that are missing a key, or that have a key but are missing values.

Console

Watches the progress of data passing through your Data Workflow operators. Useful for monitoring and troubleshooting individual steps. Accessible through the DevTools Console.

NOTE  It's a best practice to remove Console operators when you finish configuration and testing.

Create Field

Creates or updates key/value pairs (fields) in a table. Use the structure key=value, or key="value" for strings. Also supports conditional expressions and referencing other values. Use the structure key=IF(condition,valueIfTrue,valueIfFalse) to create conditional expressions.

TIP  Create Field operations happen in order, from Field 1 to Field 5. You can reference values created in previous fields.

Filter

Filters table-based data using the criteria specified in the Expression setting. The operator separates rows that meet the criteria from rows that do not. Matching rows pass through the upper output port. Non-matching rows pass through the lower output port. For example, to filter for rows that contain the key/value pair "member":"yes", use the Expression member="yes".

TIP  The Expression setting supports AND and OR statements, as well as _arg. Using _arg references data coming into the top argument port.

Formula

Performs calculations on your data based on the Formula defined in the Formula/Expression setting. You can use A as an alias for the data coming into the input port. You can use _arg to reference values coming into the top argument port. For example, the expression =A*12 multiplies your input by 12. The expression =A*_arg multiplies your input by the value entering the argument port.

Get

Pulls out data from your data set, based on the value entered in the Path field. Can be used to pull out individual values or entire rows of an array. For example, a Path of [0] retrieves the first row of your data set. A Path of [0].keyName pulls the specified key/value pair from the first row of your data set. Also useful for referencing Global Variables.

TIP  You can reference _arg in the Path setting to reference input coming into the top argument port.

Size

Returns the magnitude of the structure or value passed to it. If the input is a character string, the operator outputs the number of characters. If the input is an array, the operator outputs the number of elements in the array.

I/O

Operator

Description

Input

Lets you pull data from a single component into your data workflow. Use the Component setting to select an existing component in your module. Set Required to Yes if a value must exist in the component for the Data Workflow to run. To set up a Binded Table, connect a Data Table and select Binded Table from the Source setting.

Group Input

Creates a single object that contains all fields referenced in the Component and Component 1-5 fields. You must select one field in the Component setting. Use the Component 1-5 settings to add additional fields to your object. You can use this instead of an Input operator if you need to draw data from more than one source.

Output

An Output operator marks the end of a Data Workflow path. Select an action from the Action setting. The action applies to the component selected in the Component setting. Commonly used for outputting data (set the Action to Value) or triggering other events.

TIP  Some actions require creating a value to trigger the event. For example, a value of Yes for an Action of Reset. You can use a Create Value operator and connect it as an input. Or, use a Set Outputs operator instead of an Output operator. Set Outputs lets you create the trigger value in the Info window, using the setValue setting.

Set Output

Applies the same output action to multiple components at once. Select up to 5 existing components in the Component 1-5 fields. The end action varies based on your selection from the Action setting. Enter the value to output to all components in the setValue setting. Common uses include outputting data (set the Action to Value) or triggering other events.

NOTE  The same action and value applies to every component set as an output.

Console

Watches the progress of data passing through your Data Workflow operators. Useful for monitoring and troubleshooting individual steps. Accessible through the DevTools Console.

NOTE  It's a best practice to remove Console operators when you finish configuration and testing.

Create Table

Creates an empty table (array of objects). Use Number of Rows to define how many empty rows (objects) to add. You can set Create Index to Yes to add a key named index to each row. The value is an integer value of the index position in the array.

Create Value

Lets you create a value and pass it into your Data Workflow. This operator can create a number, string, or Boolean value. You can either set the value directly in the Value/Expression setting, or use _arg to reference input coming into the top argument port. Commonly used with an Output operator to set values for Post Triggers.

Gateways

Operator

Description

Branch Merge

Merges two pathways created by the Branch Split operator. This operator has two inputs, but only receives the input pathway passed by the Branch Split operator. The pathway not taken by Branch Split is blocked.

Branch Split

Splits a single input into two possible pathways based on the condition entered in the Condition setting. When the Condition is True, the upper output/pathway proceeds. When False, the lower output/pathway proceeds. The pathway not taken is blocked. Pairs with Branch Merge.

Decision

Chooses one of two possible paths for your data to take, based on criteria set in the Condition field. If your data meets the criteria, it follows the upper output port. If your data doesn't meet the criteria, it follows the lower output port.

TIP  The Condition statement must use <, >, =, >=, or <=. You can use A as an alias for the data coming into the input port. You can use _arg to reference values coming into the top argument port. For example, A > _arg. You can also use OR statements. For example, OR(A="a",A="b",A="c").

NOTE  The Input List setting is optional. If using it, the selected component must be the component connected to the input port.

Gate

When the condition is True, your data proceeds through the output. When the condition is False, the Gate blocks data, and the Data Workflow path stops.

TIP  You can reference _arg in the Condition setting to block or allow data based on input coming into the top argument port.

Input Switch

Switches between two inputs depending on set criteria. Connect two different inputs to the operator. Then, enter a condition in the operator's Condition setting. If the result of your condition is True, data from the upper input port passes. If the result is False, data from the lower input port passes.

TIP  You can use _arg in the condition to reference an input coming into the top argument port. For example, _arg='false' or _arg=0.

Table

Operator

Description

Aggregate

You have a few options for combining elements:

  • Sum By: Finds the sum of all numbers in a given range.

  • Count By: Counts the total number of values in a given range.

  • Mean By: Calculates the average of values in a given range.

  • Max By/Min By: Finds the highest or lowest value in a given range.

In Value Key, enter the column header (key) to indicate the range you want to perform your selected operation on.

Append

Appends two values or arrays together. When appending strings, acts as a concatenation. E.g., appending "Jane" and "Doe" returns "JaneDoe". When appending arrays, the operator joins both arrays into one. The data connected to the lower input port appends to the data connected to the upper input port.

NOTE  This operator won't append objects. To append objects, use the Extend operator instead.

Cartesian

Takes two arrays, and outputs an array containing each possible combination of the source arrays' values. For example, with source arrays of ["A", "B"] and [1,2], the output becomes [["A",1], ["A",2], ["B",1], ["B",2]]. Similar to performing a Cross Join (aka Cartesian Product) in SQL. When Type is set to Array, input must be a single array containing two objects. Each object contains one of the arrays you want to cross join.

Clean Keys

Fills in missing keys or empty values in each row of a table.

  • Missing: Creates a key when it's missing from a row. In the Keys setting, add the key name you'd like to add. Under Fill With, enter the value to add to the key.

  • Empty: Fills in values for keys that exist but have empty values. Under Keys, add the key name you'd like to fill when empty. Under Fill With, enter the value to add.

  • Missing or Empty: Fills values for rows that are missing a key, or that have a key but are missing values.

Col2Array

Converts a single table column into an array of values. In the Path setting, enter the column header (key) of the column you want to convert to an array. For example, with an input of [{"keyName":"hello"},{"keyName":"hi"}], output is ["hello","hi"] when "keyName" is entered in the Path setting.

Convert Field

Converts entire values in entire columns in a data set from one data type to another. You'll input a data set to the operator, indicating which key(s) you want to convert. Then, you'll use the Cast To setting to select either Number, Integer, or String as the data type. Your output is in the form of your selected data type.

NOTE  Passing an object results in an output of [object,object].

Create Field

Creates or updates key/value pairs (fields) in a table. Use the structure key=value, or key="value" for strings. Also supports conditional expressions and referencing other values. Use the structure key=IF(condition,valueIfTrue,valueIfFalse) to create conditional expressions.

TIP  Create Field operations happen in order, from Field 1 to Field 5. You can reference values created in previous fields.

Create Index

Lets you add index numbers to each row of a Data Table. Creates a key/value pair, where the value is the index number. In Index Name, enter a key name. In Starting Index, enter a number to start the indexing at.

TIP  You can also use _arg in the Starting Index field, to reference input coming into the top argument port. For example, _arg+1.

NOTE  The Create Index operator has other uses, including adding new rows of data to a table, starting from either the top or the bottom of the table.

Create Table

Creates an empty table (array of objects). Use Number of Rows to define how many empty rows (objects) to add. You can set Create Index to Yes to add a key named index to each row. The value is an integer value of the index position in the array.

Filter

Filters table-based data using the criteria specified in the Expression setting. The operator separates rows that meet the criteria from rows that do not. Matching rows pass through the upper output port. Non-matching rows pass through the lower output port. For example, to filter for rows that contain the key/value pair "member":"yes", use the Expression member="yes".

TIP  The Expression setting supports AND and OR statements, as well as _arg. Using _arg references data coming into the top argument port.

HTML Table

Converts a data table to an HTML table. Uses include displaying the resulting HTML table in a Content component or downloading it in an Excel file.

Map Keys

Lets you update the names of multiple keys in a table at once. Connect your original table to the upper input port. Then, create and connect a Mapping Table to the lower input port. The Mapping Table should contain a column with the original (source) key names and a column with the new (target) key names. Reference these column headers in the operator's Source Column and Target Column fields.

TIP  Use a Data Table to create your Mapping Table.

Map Values

Lets you update multiple values in a table at once. Connect your original table to the upper input port. Then, create and connect a Mapping Table to the lower input port. The Mapping Table should contain a column with the original (source) values and a column with the new (target) values. Enter these column headers in the operator's Source Column and Target Column fields.

TIP  Use a Data Table to create your Mapping Table.

Merge

Combines data from two Data Tables. You can merge information from one table into the other or combine both tables into an array. In the Merge Key setting, enter the reference key of the data you want to merge. This key could be the heading of a column in your Data Tables. To merge columns that have the same values but different keys, you can enter columnOneName=columnTwoName as the Merge key.

Multi-row

Duplicates a table column, while shifting the duplicated rows up or down several rows. In Key Name, enter the column to duplicate. In New Key Name, enter a name for the new column. In the Steps field, enter a positive or negative integer to determine row shifting. A positive Steps value shifts rows up. For example, Steps = 1 shifts values up one row. A negative Steps value shifts rows down. For example, Steps = -2 shifts values down two rows. By default, with no Steps value entered, values shift down one row.

NOTE  Rows left empty after adjacent rows have shifted have an undefined value.

Nest By

Lets you create a nested table (array of objects) in the connected parent table. In Parent Keys, enter the key that remains un-nested in each row. In Child Name, enter a key for the nested array to exist under. All other fields in the row nest under the Child Name key. Useful for grouping rows by a common Parent Key value. For example, You can nest by a Parent Key that has the same value across multiple rows. All rows with the same Parent Key value group together into one row. Their remaining fields nest as individual rows in the same Child Name array.

NOTE  Keys that are not consistent across all arrays are dropped, which can result in data loss.

Nested Array

Lets you add or update fields in a nested table (array of objects). Input must be a table that contains a nested table. In Table Path, enter the key that contains the nested table. In Field 1 to Field 4, define fields to add or update. Use the structure key=value, or key="value" for strings.

Omit

Lets you remove specific columns from a table. In Omit Keys, enter the key(s) (column headers) of the column(s) you want to remove from the table. To remove multiple keys, use a comma-separated list.

TIP  You can use _arg in Omit Keys to reference input coming into the top argument port. This is useful for dynamically passing a list of keys to omit.

Pick

Lets you retrieve specific columns from a table. In Pick Keys, enter the key(s) (column headers) of the column(s) you want to pull from the table. To retrieve multiple keys, use a comma-separated list.

TIP  You can use _arg in Pick Keys to reference input coming into the top argument port. This is useful for dynamically passing a list of columns to pick from the table.

Rename Field

Renames keys (fields) in key/value pairs. Enter the current key name in Current Name, and the new key name in Target Name. The Target Name field supports strings, formulas, and _arg. Using _arg lets you reference input coming into the top argument port.

Size

Returns the magnitude of the structure or value passed to it. If the input is a character string, the operator outputs the number of characters. If the input is an array, the operator outputs the number of elements in the array.

Sort

Sorts table-based data in ascending or descending order. You can reference _arg in the Sort Keys or Asc/Desc field to sort based on input coming into the top argument port. Or, hard-code the Sort Keys or Asc/Desc value. To hard-code the Asc/Desc value, enter asc or desc, all lower-case (default sort order is ascending).

Split Field

Lets you split values in key/value pair objects, in an array of objects. The split values are added as a nested array of values in each object. In Path/Field, enter the key that contains the value you want to split. In Separator, enter the separator to split the value on. The separator is not included in the final arrays. In New Name, enter a key for the field that contains the array of split values.

TIP  Split Field can also be used like the Split String operator, to split a string input into an array of values.

Table2HTML

Takes two inputs (an empty table and a value) and outputs a string. The string contains the value, repeated as many times as there are rows in the table. Each value in the string is separated by the HTML break specified in the operator's HTML Break setting. You can use a Create Table operator to create an empty table with as many rows as you need. Connect the table to the upper input port, and the value to the lower input port.

Table2Object

Converts table-based data to an object. Try setting up a Data Table with the column headers Key and Value. In the operator's Key Column Name field, enter Key. In Value Column Name, enter Value. Each row in the Data Table becomes a key/value pair in the final object.

NOTE  Key Column Name and Value Column Name must exactly match the table's column headers.

Transpose

Transposes (rotates) data from columns to rows in a table. Multiple columns can be transposed at once. In Transpose Keys, enter the keys (column headers) of the columns you want to transpose into rows.

Unique

Lets you filter out duplicate key/value pairs in an array of objects. Outputs only the objects with unique values. In Unique Keys, enter the key to look at when filtering for unique values.

NOTE  The operator keeps the first instance of a duplicate.

Unwind

Brings values in a nested array up a level in the data structure. Use the Unwind Key setting to define the key that contains the nested array.

Array

Operator

Description

Array2Col

Converts an array of values to an array of key/value pair objects (a column in a table). Each value in the source array becomes one key/value pair object. In Key Name, enter a key to use in the key/value pairs. The value comes from the source array of values.

Get

Pulls out data from your data set, based on the value entered in the Path field. Can be used to pull out individual values or entire rows of an array. For example, a Path of [0] retrieves the first row of your data set. A Path of [0].keyName pulls the specified key/value pair from the first row of your data set. Also useful for referencing Global Variables.

TIP  You can reference _arg in the Path setting to reference input coming into the top argument port.

Includes

Determines if a specified value exists in the input. Input should be a value or an array of values. In the Value setting, enter the value you want to check for. If your data includes that value, the output is Boolean True. If not, the output becomes Boolean False.

Join

Joins together an array of values and outputs a string containing those values. In the Separator setting, enter a character to act as a separator between values in the final string. When joining string values, the final string does not include quotation marks around each value.

TIP  You can use Col2Array to quickly create an array of values from a single column in a table.

Set

Lets you add values to a data structure. Connect the data structure where you want to add values to the upper input port. In Path, enter the key of the key/value pair you want to create. You can use dot notation to specify index positions in an array. For example, [0].key. In Value, enter the value for the key/value pair you want to create. Or, leave Value blank and connect the value or structure to add to the lower input port.

TIP  You can use _arg to add values based on input coming into the top argument port.

Size

Returns the magnitude of the structure or value passed to it. If the input is a character string, the operator outputs the number of characters. If the input is an array, the operator outputs the number of elements in the array.

Object

Operator

Description

Clean Object

Removes or converts unwanted data values, including null values, in a data set. The input must be an object. The operator's Clean Action setting has 3 options:

  • Convert All to String: Converts every value in the data set to a string.

  • Omit Null: Drops any fields with a value of "null".

  • Set Null to Empty String: Replaces every value of "null" with an empty string (""). Clean Actions apply to all nested fields, as well.

Diff

Performs a diff analysis between two inputs and highlights structural differences. Outputs change records.

NOTE  This operator uses the deep-diff npm package. To interpret change records, you can review the package documentation here: https://www.npmjs.com/package/deep-diff#differences.

Extend

Takes two inputs (which must be objects) and merges them into one object. Often used by connecting two Group Input operators, as Group Input has a limit of 6 fields. Extend lets you create a single object with more than 6 fields.

TIP  To merge arrays, use the Append operator instead.

Get

Pulls out data from your data set, based on the value entered in the Path field. Can be used to pull out individual values or entire rows of an array. For example, a Path of [0] retrieves the first row of your data set. A Path of [0].keyName pulls the specified key/value pair from the first row of your data set. Also useful for referencing Global Variables.

TIP  You can reference _arg in the Path setting to reference input coming into the top argument port.

Has

Determines if a path or key exists in an object. Use the Path setting to define what key to look for. If the input has the key specified, the output is Boolean True. If not, the output becomes Boolean False. The value in the Path setting must match the key exactly, or the output becomes false.

NOTE  Input cannot be an array or table, only an object.

Object2Table

Creates a table out of an object. Each key/value pair in the object becomes a row in the final table. Input must be a single object.

Set

Lets you add values to a data structure. Connect the data structure where you want to add values to the upper input port. In Path, enter the key of the key/value pair you want to create. You can use dot notation to specify index positions in an array. For example, [0].key. In Value, enter the value for the key/value pair you want to create. Or, leave Value blank and connect the value or structure to add to the lower input port.

TIP  You can use _arg to add values based on input coming into the top argument port.

Value/String

Operator

Description

Convert Value

Converts a value from one data type to another, based on the value selected in the Cast To setting: Number, Integer, or String. Input must be a single key/value pair. To convert to a Number or Integer, the input must be a numeric value. When converting to Integer, any values after the decimal are removed, not rounded.

Create Value

Lets you create a value and pass it into your Data Workflow. This operator can create a number, string, or Boolean value. You can either set the value directly in the Value/Expression setting, or use _arg to reference input coming into the top argument port. Commonly used with an Output operator to set values for Post Triggers.

Excel to JSON

Converts the base64 encoding of an Excel file to JSON. Data in the sheet is converted into two attributes: a headings array and a JSON array. The headings array contains the header fields of the file. The JSON array contains the file contents. We recommend using an Excel file that contains headers. This ensures the JSON array has key/value pairs, where the key corresponds to the header.

Formula

Performs calculations on your data based on the Formula defined in the Formula/Expression setting. You can use A as an alias for the data coming into the input port. You can use _arg to reference values coming into the top argument port. For example, the expression =A*12 multiplies your input by 12. The expression =A*_arg multiplies your input by the value entering the argument port.

Includes

Determines if a specified value exists in the input. Input should be a value or an array of values. In the Value setting, enter the value you want to check for. If your data includes that value, the output is Boolean True. If not, the output becomes Boolean False.

JSON Parse

Parses a string containing properly-formatted JSON and outputs the defined data structure as a JSON object.

TIP  You can use a Hidden component as the Input, where the Hidden component's Default Value field contains the written JSON.

NLP

Compares text entering the upper input port to a Keyword vs Mood table. Outputs an array listing any matching keywords and their corresponding moods. Create a Data Table with the column names Keyword and Mood; connect it to the NLP operator's lower input port. This is your Keyword vs Mood table. Under Keyword, create a list of words. Under Mood, assign emotions (moods) to each word. In the NLP operator's Source Column setting, enter Keyword. When the operator finds a match between the text in your input and the keywords in your table, it adds the Keyword/Mood pair to the output.

Split String

Splits a string into an array of values. In the Separator setting, enter the character(s) acting as a separator in your string. The separator is not included in the final array.

NOTE  The default separator is a comma. If the string is comma-separated you can leave the Separator setting blank.

 

Best Practices

  • Data Workflows timeout after five minutes in all environments. Build Data Workflows to complete operations in five minutes to prevent timeouts.

  • If you don't plan to use disabled components in your application, remove them to ensure optimal performance. Remember to check all active components that connect to disabled components. Ensure active components still function properly after you remove the disabled ones.

  • Add labels to all Data Workflow operators to describe their function. These labels make it easier to know the purpose of an operator without having to open the Info window.

  • Select the Do Not Sanitize setting in all your operators to improve application performance.

  • Organize Data Workflow components based on their function in your application.

  • Use the component's Notes tab to comment on complex data processes. Add notes to explain what components are being triggered, trigger types, and the importance of each component.

Resources