Data Workflow Operators

Overview

Your Unqork applications rely on data to function. The Data Workflow component is the space where you input data, map it, and manipulate it. Operators are the individual pieces of your Data Workflow. They let you manipulate large and complex data structures from multiple sources. Much like components in a module, you'll connect operators to create operations.

Common uses of operators include:

  • Inputting data.

  • Unwinding complex data structures.

  • Filtering data to obtain specific items.

  • Viewing data at different points in the Data Workflow.

  • Appending data to create new structures.

  • Outputting data.

Here's an extract of a Data Workflow canvas showing an orchestration of operators. This image represents the Data Workflow of the Address Parsing snippet.

As you can see, the Input operator brings in the data from an Address Search component. The Unwind operator separates the individual data points, and the remaining operators take each data point and output them to different fields.

What You'll Learn

In this article, you'll learn the basics of operators, the different types of operator ports, and how to group them .

Understanding What Operator Ports Do

Let's take a closer look at an operator. Do you see the small extensions around the edges? These are your operator's ports. Lines drawn between these ports connect your operators.

The three port types include:

Input

Argument

Output

An operator can also have more than one input or output port. Below are the ports you'll see associated with a Filter operator:

Input

The input port is the entry point into your operator. The previous operator in the Data Workflow connects to the next operator's input port.

Some data operations have two inputs—for example, the Merge operator, which combines two inputted data sets. In this case, a data set enters Input 1, and another data set enters Input 2.

Argument

Arguments add a dynamic aspect to data processing. They can create conditions, rules, and more functionality for an operator. When you bring an input into an operator's top port, you're bringing the input in as an argument.

It's important to make a distinction between an argument and an expression. Most operators have an Expression or Key field that controls what the operator does. For example, say you have a data set of savings accounts and checking accounts. You want to filter the two account types into separate outputs using a Filter operator. Without an argument, your expression might be account="savings". From there, the savings accounts filter output to another operator. Anything that isn't a savings account filters out from the rest. The operator performs a single operation. If you want an operator to always have the same function, there's no need for an argument.

Where an argument comes into play is when you want different functions based on dynamic data. Let's use the example of a data set of savings and checking accounts. With an argument, you can make the filter more complex. You could add more filtering operations, like sorting accounts by their size. You could also add logic to your filter, calculating multiple account sizes. Or, you could set up decision based on the accounts your end-user End-users, also known as Express Users, are the individuals accessing an application through Express View. In most cases, end-users are the customers using the product. selects.

Unqork references an argument as _arg in an expression.

Output

Outputs are where you send the final result of the operation. You'll connect one output to the next using the right port.

Some data operations have two outputs—for example, the Filter operator, which separates data and outputs into multiple locations. Output 1 would be all the expression results that are true, or what you filtered for. Output 2 would be the expression results that are false, or what you do not what to filter for.

Connecting Operators

Remember, you connect operators using arrow connectors from an output port to an input or argument port. In the following image, the output port (right) of the Input operator connects to the input port (left) of the Col2Array operator.

Grouping Operators

It's easy to build complex systems using the Data Workflow component. But, you might find yourself with a crowded canvas of operators and connecting lines. Grouping operators can help you organize your Data Workflow. Doing so creates a clear process flow and helps you track your operators' functions.

Let's look at the Address Parsing snippet again. By default, its Data Workflow canvas looks like the following:

After grouping operators you can see a cleaner layout:

To group Data Workflow operators:

1. Hold down the Shift key while clicking on the Data Workflow operators you want to group.
2. Press the Ctrl + G keys (PC) or Cmd + G keys (Mac). The selected operators group together, then collapse into an expandable frame.

You can expand or collapse the frame by clicking the plus (+) or minus (-) sign in the frame's top bar.

3. In the Info window that displays, enter a Label for your group of operations.
4. Click Save.

Data Workflow Operators

The Data Workflow component's operators are organized into six categories, based on the actions they perform. Click the following links to learn more about the various Data Workflow categories:

I/O Operators

Gateways Operators

Table Operators

Array Operators

Object Operators

Value/String Operators

Frequently Used

There is also a frequently-used Data Workflow operator category. Below is a list of these Data Workflow operators with a description of how they work.

Operator

Description

Input

Lets you retrieve data from a another component and bring it into your Data Workflow. Use the Component drop-down to select an existing component in your module. Set Required to Yes if a value must exist in the component for the Data Workflow to run. To set up a Binded Table, connect a Data Table component to the Input operator and select Binded Table from the Source setting.

Output

An Output operator marks the end of a Data Workflow path. Use the Component setting to select a component in your module to output the data. The Action setting applies to the component selected in the Component field. This operator is commonly used for outputting data by setting the Action field to Value or triggering events.

Some actions require creating a value to trigger the event. For example, a Value of Yes for an Action of Reset. You can use a Create Value operator and connect it as an input. Or, use a Set Outputs operator instead of an Output operator. The Set Outputs operator lets you create the trigger value using the setValue setting.

Console

Watches the progress of data passing through your Data Workflow operators. This operator is useful for monitoring and troubleshooting individual steps in the DevTools Console The DevTools Console helps you securely store, build, test, and deploy your software..

It's a best practice to remove Console operators when you finish configuration and testing.

Create Field

Creates or updates key/value pairs (fields) in a table. Use the structure key=value or key="value" for strings. The operator also supports conditional expressions and referencing other values. Use the structure key=IF(condition,valueIfTrue,valueIfFalse) to create conditional expressions.

The Create Field operations occur in order, from Field 1 to Field 5. You can reference values created in previous fields.

Create Value

Lets you create a value and pass it into the Data Workflow. This operator can create a number, string, or Boolean value. You can either set the value directly in the Value/Expression field or use _arg to reference the input connected to the top argument port. This operator is commonly used with an Output operator to set values for Post Triggers.

Filter

Filters table-based data using the criteria specified in the Expression field. The operator separates rows that meet the criteria from rows that do not. Matching rows pass through the upper output port while non-matching rows pass through the lower output port. For example, to filter for rows containing the key/value pair "member":"yes", use the Expressionmember="yes".

The Expression field also supports AND and OR statements and _arg. Using _arg references data coming into the top argument port.

Formula

Performs calculations on your data based on the formula defined in the Formula/Expression field. You can use an alias of A to represent the data coming into the input port. You can also use _arg to reference values coming into the top argument port. For example, the expression =A*12 multiplies your input by 12. The expression =A*_arg multiplies your input by the value entered in the argument port.

Get

Retrieves data from your data set based on the value entered in the Path field. You can use the operator to retrieve individual values or entire rows of an array. For example, a Path of [0] retrieves the first row of your data set. A Path of [0].keyName retrieves the specified key/value pair from the first row of your data set. It's also useful for referencing Global Variables.

You can use _arg in the Path field to reference the input coming into the top argument port.

Best Practices

  • For operators that require fields, like the Formula operator, it is best practice to avoid using symbols and numerical values in the fields. For example, instead of using Test1Key="value", use TestOneKey="value".

  • Data Workflows timeout after five minutes in all environments. Build Data Workflows to complete operations in five minutes to prevent timeouts.

  • If you don't plan to use disabled components in your application, remove them to ensure optimal performance. Remember to check all active components that connect to disabled components. Ensure active components still function properly after you remove the disabled ones.

  • Add labels to all Data Workflow operators to describe their function. These labels make it easier to know the purpose of an operator without having to open the Info window.

  • Select the Do Not Sanitize setting in all your operators to improve application performance.

  • Organize Data Workflow components based on their function in your application.

  • Use the component's Notes tab to comment on complex data processes. Add notes to explain what components are being triggered, trigger types, and the importance of each component.

Resources