Best Practices: Secure File Transfer Protocol (SFTP) Integration

Estimated Reading Time:  11 minutes

Overview

FTP (File Transfer Protocol) is a method of transferring large folders and files on the web. SFTP (Secure File Transfer Protocol) offers the same functionality, but with an added layer of security.

An SFTP setup lets you connect to a server using a password-protected folder. This folder is also called a directory. In some cases, you might need extra security keys to access this folder. SFTP is crucial when working with sensitive information, so it's best practice to implement SFTP instead of FTP whenever possible.

What You'll Learn

In this article, you'll learn how to:

Connect an Unqork Environment to an SFTP Server

Connecting to an SFTP server happens in two places. First, add your connection to the Services Administration of your environment. Then, configure a File Storage component in a module.

In Services Administration, four authentication types use some version of FTP:

  • SFTP

  • FTP

  • Encryption (GPG)

  • Decryption (GPG)

Selecting one option lets you reference this connection in a File Storage component. You can configure a File Storage component once you add an FTP-type service to your environment. In the component's configuration window, select either SFTP or FTP from the File Storage Type drop-down. Then, your FTP-type services display in the Service drop-down.

NOTE  The FTP authentication option is in the process of being removed. As best pratice for optimal security, we don't recommend using this option.

The next step in configuring your File Storage component is to select a request method. You can find a request method in the Method drop-down in the configuration window. Here's a brief description of each:

  • GET: Gets a specific file.

  • LIST: Lists all files in a directory (or folder).

  • PUT: Writes a file to a directory.

  • DELETE: Deletes a file from a directory.

NOTE  GET, PUT, and DELETE request methods each allow only single actions. To perform these actions on multiple files, you need to create a list of individual files and loop through them. Only the LIST request method lets you list multiple files simultaneously.

A File Storage component can only run server-side. So, you can't monitor its progress in Express View. You must ensure the component's module is part of a server-side execution. Then, you can test your configuration using the Server Side Execution option at the top of the Module Builder.

SFTP Request Methods

Each SFTP request method has its own requirements.

GET

A File Storage component configured with a GET request reads a specific file from a file directory. Selecting GET from the Method drop-down populates the File Storage component's Inbound table. This table has a single row called Path, which is the path leading to the file you want to read or get:

Say you want to get a file located in the following location on the SFTP server: /upload/training/icon-presentation.png.

You can do this in two ways:

  • You can hard code this file path in the Property ID column of the Inbound table:

Get From Property ID Watch Required

Path

/upload/training/icon-presentation.png

 

Yes (checked)

  • You can store the file path as the Default Value in a Hidden component. Then, enter the Property ID of your Hidden component (filePath) in the Property ID column:

Get From Property ID Watch Required

Path

filePath

 

Yes (checked)

With this complete, your File Storage component retrieves the file icon-presentation.png. You can see the response in the file object data.resolved.file.

Here, you see a few key/value pairs:

  • storage: Displays a value of fs, which stands for File Storage.

  • name: The name of the file returned.

  • path: A string referring to the file's location in the Amazon S3 bucket.

NOTE  S3 is the AWS (Amazon Web Services) file system Unqork uses. Whenever you upload a file to Unqork, you're uploading to an S3 bucket. Using a GET request retrieves a file from another server and uploads it to your Amazon S3 bucket. Only those authenticated to your Unqork environment can retrieve files. That way, you don't need to worry about unwanted files being uploaded.

Here's an example of how this looks on the Server Side Execution Testing page:

LIST

A File Storage component configured using a LIST request lists all files found in a file directory. Selecting LIST from the Method drop-down populates the File Storage component's Inbound table. This table has a single row called Path. This is the path leading to the directory of files you want to list:

LIST works much like a GET request. The only difference is that you exclude the file name in the path. Instead, the path ends with the folder that holds the files you want to list.

Say you want to list files located in the following folder on the SFTP server: /upload/training/.

Like with a GET request, you can do this in two ways:

  • You can hard code this path in the Property ID column of the Inbound table:

Get From Property ID Watch Required

Path

/upload/training/

 

Yes (checked)

  • You can store the path as the Default Value in a Hidden component. Then, enter the Property ID of your Hidden component (fileDirectory) in the Property ID column:

Get From Property ID Watch Required

Path

fileDirectory

 

Yes (checked)

With this complete, your File Storage component lists the files found in /upload/training/. You can see the response in the file object data.resolved.files. Here, each file in the requested folder has its own nested object:

PUT

A File Storage component configured with a PUT request stores a file in a specified location. Selecting PUT from the Method drop-down populates the File Storage component's Inbound table. This table has 2 rows:

  • Path: The location you want your file stored.

  • File Object: The file you want to store.

The path plays the same role here as in LIST requests. The path you enter ends with the folder you want to store your file.

For the File Object, you need to have the same object format as what was returned in the GET and LIST request methods. To help demonstrate, here's the object from a sample file upload as seen in the DevTools Console:

This object has the following key/value pairs:

  • data: The base64 representation of the file.

  • name: The name of the file.

  • size: The size of the file.

  • storage: The format of the stored file.

  • type: The type of file uploaded.

  • url: The data representation of the file.

The uploaded file is called Values and Satisfaction Exercise_Unqork.pdf. It also has the following url:

data:application/pdf;base64,JVBERi0xLjYNJeLjz9MNC....

The url starts with data, meaning it's the data representation of the uploaded file. The file type application/pdf follows the data representation. This format is the same as what you see under the type key. Next is base64, which is how the file is encoded. Everything after is a base64-encoded representation of the uploaded file.

To pass your PUT request, you need the name and url keys. But, the process doesn't begin with the File Storage component. Instead, you (or your end-user) must upload a file using a module configured with a File component. Let's look at the configuration.

Calculating the File Path

First, you need your Path value to reflect the uploaded file's name. Use a Calculator component to add the file name to the end of a file directory. Then, you can store this full file path in a Hidden component and reference it as you did with the other request types.

Creating an Attachment

To upload files, Unqork requires that you create an attachment to move your file to the SFTP server. Configure a Plug-In component using the following Data Source URL: /fbu/uapi/modules/{{data.currentModuleId}}/submissions/attachments.

NOTE  Replace currentModuleID in the Data Source URL field with your own module ID. You must also set the Make a Multipart API Call toggle to ON.

This Plug-In compnent makes an API (application programming interface) call that takes the base64 URL of your uploaded file and uploads it to the Amazon S3 bucket in your environment. The response is a URL pointing to that file in the S3 bucket.

Retrieving the File Object

This API call serves as a conversion, turning the uploaded file into a format the SFTP transfer recognizes. The next step is to retrieve that converted file. Configure a second Plug-In component that performs a GET API call on the URL returned by the first Plug-In component. You must ensure the Expect Binary Response toggle is set to ON.

Submitting the Request

With the above complete, you can configure your File Storage component. You must reference your calculated file path and your final API call response in the Inbound table as follows:

Get From Property ID Watch Required

Path

filePath

 

Yes (checked)

File Object

pluginGetFileObject

 

Yes (checked)

DELETE

A File Storage component configured with a DELETE request deletes a file from a directory. Selecting DELETE from the Method drop-down populates the File Storage component's Inbound table with a row called Path. This is the path leading to the file you want to delete:

Say you want to delete this file on the SFTP server: /upload/training/icon-presentation.png.

You can populate the path using either of the methods you have learned about so far:

  • You can hard code this file path in the Property ID column of the Inbound table:

Get From Property ID Watch Required

Path

/upload/training/icon-presentation.png

 

Yes (checked)

  • You can store the file path as the Default Value in a Hidden component. Then, enter the Property ID of your Hidden component (filePath) in the Property ID column:

Get From Property ID Watch Required

Path

filePath

 

Yes (checked)

With this complete, your File Storage component deletes the file icon-presentation.png. A DELETE request only returns a 200 response.

With these basic operations, you can create more complex functions. Say you want to rename or move files. You can accomplish that using a combination of these operations.

Test an SFTP Connection

These basic operations let you create more complex functions, like renaming or moving files. You can accomplish these actions using a combination of these operations you've learned in this article.

You can test your SFTP connections and operations using the Server Side Execution option in the Module Builder. You can also test SFTP outside of Unqork using external applications like Cyberduck. Cyberduck is an FTP testing application similar to Postman, except Postman is for APIs and Cyberduck is for FTP. You can download Cyberduck and add it to your SFTP information for testing.

Lab

WARNING  You only have viewing access to the below modules. Do not attempt to alter it in any way. Altering an example module might diminish the experience of other Unqork users.

Resources