Bulk Data Operations: Export

This feature is currently in production early-access. Functionality is subject to change as this feature is prepared for general access.

Overview

The Bulk Data Operations (BDO) Export API APIs (application programming interfaces) are a set of protocols and definitions developers use to build and integrate application software. APIs act as the connective tissue between products and services. Endpoint lets you export your submission data Also known as Record Data. Submission data consists of information saved by Unqork components. View submission data by using the angular command, or in Record Collections. out of the Unqork platform and into a file or object storage service Object storage, also known as object-based storage, is a computer data storage architecture designed to handle large amounts of unstructured data.Popular object storage services include Amazon S3, Azure Blob Storage, and Google Cloud Storage.. You can make an export request using curl, Postman, or any other service that supports a POST The HTTP POST method is used to send data to a server to create or update a resource, typically including the data in the body of the request. request.

To discover how to set up API calls using a third-party program, view our Testing API Endpoints Using Third-Party Applications article.

You can test the BDO Export endpoint using the Unqork Services page. To access the Unqork Services page, enter the following address into your browser: https://{environment}.unqork.io/fbu/uapi/docs/documentation/#/Bulk%20Operations/import. Replace {environment} with the name of your Unqork environment.

Exporting Files Using Bulk Data Operations

BDO Export uses the HTTP HTTP (Hypertext Transfer Protocol) is an application-layer protocol used to transmit hypermedia documents like HTML. POST The HTTP POST method is used to send data to a server to create or update a resource, typically including the data in the body of the request. method and the following endpoint: https://{environment}.unqork.io/fbu/uapi/bulk-operations/import. Replace {environment} with the name of your Unqork environment.

Copy
{
  "cloudStorageBucketName": "string",
  "cloudStorageServiceName": "string",
  "cloudStorageTargetDirectory": "string",
  "dataModelOrModuleId": "string",
  "dateFieldEndToFilterOn": "string",
  "dateFieldStartToFilterOn": "string",
  "dateFieldToFilterOn": "created",
  "exportFullRecord": "true",
  "fileFormat": "CSV",
  "name": "string",
  "numberOfFilesToGenerate": 1,
  "targetRecordType": "ALL"
}

The following parameters are available for the Bulk Data Export schema Schema is a declarative language that defines the structure, content, and constraints of the data.:

Key

Possible Values

Description

Required

cloudStorageBucketName

string (no spaces)

Enter the name of the cloud service bucket A bucket is a container in cloud computing that stores data. Popular cloud services that use buckets include Amazon S3, Azure Blob Storage, and Google Cloud Storage where you want to export the data. For example, your Amazon S3 storage might have a bucket named user-data-bucket.

When using cloud storage, cloudStorageServiceName, cloudStorageBucketName, and cloudStorageTargetDirectory must be included in your request.

Bucket names do not have empty spaces in them as they are used as a URL A URL (uniform resource locator) is a unique identifier used to locate a resource on the internet. Also known as a web address. or URI URI (Uniform Resource Identifier) identifies a resource by name, location, or both. address. For example, http://s3.amazonaws.com/bucket-name/.

Optional

cloudStorageServiceName

string (no spaces)

Enter the Service Name of the service created in Services Administration. Unqork supports connecting the following object storage services Object storage, also known as object-based storage, is a computer data storage architecture designed to handle large amounts of unstructured data.Popular object storage services include Amazon S3, Azure Blob Storage, and Google Cloud Storage.:

Add Amazon S3 or Azure Blob Storage services using our Services Administration - How to: Add a Service article.

Optional

cloudStorageTargetDirectory

string (no spaces)

If your bucket A bucket is a container in cloud computing that stores data. Popular cloud services that use buckets include Amazon S3, Azure Blob Storage, and Google Cloud Storage has directories, specify the path where you want to export the data. For example: userData/moduleName/submissions.

Optional

dataModelOrModuleId

string

Specify the Data Model ID, or Module ID, containing the data you want to export.

To find a Data Model's ID value, view our How to: Query Data Model Records Data article.

Yes

dateFieldEndToFilterOn

string (YYYY-MM-DDTHH:mm:ss:sssZ)

Export data record up to this date. Enter the date and time in ISO 8601 Format A standard format for representing dates and times. ISO 8601 format is represented in the following order: year, month, day, hour, minutes, seconds, and milliseconds. (UTC Coordinated Universal Time (UTC) is a standard, globally-accepted timezone used to regulate clocks and time around the world.). For example, setting this key's value to 2024-10-17T18:02:55.101Z would export data recorded before October 10th, 2024 at 6:02:55.101PM Zulu time.

To export data from a specific time frame, use dateFieldStartToFilterOn as the start date, and dateFieldEndToFilterOn as the end date.

Optional

dateFieldStartToFilterOn

string (YYYY-MM-DDTHH:mm:ss:sssZ)

Export data recorded after this date. Enter the date and time in ISO 8601 Format A standard format for representing dates and times. ISO 8601 format is represented in the following order: year, month, day, hour, minutes, seconds, and milliseconds. (UTC Coordinated Universal Time (UTC) is a standard, globally-accepted timezone used to regulate clocks and time around the world.). For example, setting this key's value to 2024-10-17T18:02:55.101Z would export data recorded after October 10th, 2024 at 6:02:55.101PM Zulu time.

To export data from a specific time frame, use dateFieldStartToFilterOn as the start date, and dateFieldEndToFilterOn as the end date.

Optional

dateFieldToFilterOn

created

modified

Filter exported data by record created, or modified dates.

When using this field, you must include dateFieldEndToFilterOn or dateFieldStartToFilterOn, or both in your request.

 

exportFullRecord

Boolean

When set to true, it includes all record details, like _id, owner, metadata, and so on.

When set to false, it includes only record data nested in the data object.

Unqork recommends setting this key to false.

Yes

fileFormat

CSV

ZIPWITHCSV

JSON

ZIPWITHJSON

ZIPWITHMULTIPLEJSON

Specify the file format to extract to. File formats include:

Value must be uppercase.

Yes

name

string

Enter a value as the name of the extracted file. Names can include spaces. This value also displays under the Job Tracker's File Name column.

Yes

numberOfFilesToGenerate

number

Set the preferred amount of files to export. This number cannot be larger than the total files exported.

Unqork recommends setting this value to 1. If the resulting file would be larger than 100 MB, the export operation automatically creates additional files.

Yes

targetRecordType

ALL

ACTIVE

ARCHIVED

Specify the type of record to export. The following options are available:

  • ALL: Export Active and Archived records.

  • ACTIVE: Export records that have not been archived.

  • ARCHIVED: Export records that have been archived, but not permanently deleted.

The value must be uppercase.

Yes

Example Response

In the BDO Export response below, the location field contains the address of the exported file for download:

Larger export operations might require additional time to process. Use the /fbu/uapi/bulk-operations/job/{jobId} endpoint to check the status of the job.

Copy
{
    "id": "67129acfc2f4b88db7a26b59",
    "created": 1729272527398,
    "modified": 1729272527398,
    "createdBy": "creatorAccount@creatorCompany.com",
    "modifiedBy": "creatorAccount@creatorCompany.com",
    "data": {
        "type": "EXPORT",
        "collection": "658c3c6270a73gb027a0f3a2",
        "dataLocation": {
            "database": "unqork",
            "collection": "submissions",
            "entityType": "MODULE"
        },
        "name": "Example Export",
        "steps": [
            {
                "stepType": "EXPORT",
                "status": "created",
                "options": {
                    "export": {
                        "chunks": 1,
                        "filter": {
                            "showActive": true,
                            "showArchive": true,
                            "form": "658c3c6270a73gb027a0f3a2",
                            "dateFilter": {
                                "start": "2024-09-17T18:02:55.101Z",
                                "end": "2024-10-17T20:02:55.101Z",
                                "dateField": "created"
                            }
                        },
                        "projection": {
                            "data": true
                        }
                    }
                },
                "files": [
                    {
                        "destination": {
                            "name": "0",
                            !!"location": "unqorkResource/export/9adf217f-5ccb-449c-9920-f1e0000597d7/0.csv",!!
                            "format": "CSV"
                        }
                    }
                ],
                "collection": "658c3c6270a73gb027a0f3a2",
                "dataLocation": {
                    "database": "unqork",
                    "collection": "submissions",
                    "entityType": "MODULE"
                }
            }
        ]
    }
}

In the response above, the location key's value contains the file's address. Use your preferred file download method to extract the file.

To learn how to use the location key to create a download URL, view our Bulk Operations: Download-Link article.

Viewing a Bulk Data Export Operation Job's Details

After starting a Bulk Data Export operation, you can view the job details using the /fbu/uapi/bulk-operations/job/{jobId} endpoint with the GET The HTTP GET method is used to request data from a server, typically retrieving information without modifying any resources. method. Copy the export response's id value and replace the job detail's {jobId} placeholder. For example, https://{environment}.unqork.io/fbu/uapi/bulk-operations/job/67169684ec1bf4776fa760a5.

Example Response

Below is an example of a Get Job Details endpoint response:

Copy
{
    "id": "67114d26340b0b7c0510c680",
    "created": 1729187110966,
    "modified": 1729187221199,
    "createdBy": "creatorEmail@creatorCompany.com",
    "modifiedBy": null,
    "data": {
        "type": "EXPORT",
        "collection": "658c3c6b70573db027a0f3a2",
        "dataLocation": {
            "database": "unqork",
            "collection": "submissions",
            "entityType": "MODULE"
        },
        "name": "Example Export",
        "steps": [
            {
                "stepType": "EXPORT",
                "id": "67114d50b056c32e103b11e3",
                "status": "completed",
                "files": [
                    {
                        "destination": {
                            "name": "0",
                            !!"location": "unqorkResource/export/78b9abdc-df4a-42e0-8b02-1ce0377a1a61/0.csv",!!
                            "format": "CSV"
                        }
                    }
                ],
                "options": {
                    "export": {
                        "chunks": 1,
                        "filter": {
                            "showActive": true,
                            "showArchive": false,
                            "form": "658c3c6b70573db027a0f3a2"
                        },
                        "projection": {
                            "data": false
                        }
                    }
                },
                "dataLocation": {
                    "database": "unqork",
                    "collection": "submissions",
                    "entityType": "MODULE"
                },
                "collection": "658c3c6b70573db027a0f3a2",
                "details": {
                    "dataLocation": {
                        "database": "unqork",
                        "collection": "submissions",
                        "entityType": "MODULE"
                    },
                    "chunks": [
                        {
                            "chunkQueueId": "0",
                            "status": "completed",
                            "docs": {
                                "total": 4,
                                "processed": 4,
                                "processedSize": 1723
                            },
                            "startTime": "2024-10-17T17:46:52.716Z",
                            "endTime": "2024-10-17T17:46:52.755Z"
                        }
                    ],
                    !!"docs": {
                        "total": 4,
                        "processed": 4
                    }!!
                }
            }
        ],
        !!"status": "completed"!!
    }
}

Using the above example response, the following properties might be important for Creator Also known as Unqork Users, or Designer Users; is anyone who is inside the Unqork platform. processes or pipeline development:

Setting Description Example

status

The current status of the operation. The overall operation status is displayed at the bottom of the data object. For troubleshooting, review the status property contained in each stepType in the steps array. This value is also reflected in the Job Tracker's Job Status column.

Status values include:

  • inProgress: The export operation is still processing the file(s).
  • completed: The export is complete.

  • canceled: The export has been manually stopped by the user.

  • failed: The export failed. Review each stepType's value to determine which process failed.

Copy
{
    "data": {
        "type": "EXPORT",
        "collection": "658c3c6b70573db027a0f3a2",
        "dataLocation": {},
        "name": "Example Export 2",
        "steps": [],
        !!"status": "completed"!!
    }
}

docs

Located in the details object, the docs objects displays the following properties:

  • processed: The amount of records successfully processed by the operation.

  • total: The total records processed.

Copy
"details": {
  "dataLocation": {},
  "chunks": [],
  "docs": {
    !!"total": 4,!!
    !!"processed": 4!!
    }
  }
}

location

Located in the files array, the location property contains the file key used to create the download file.

Copy
"files": [
    {
      "destination": {
      "name": "0",
      !!"location": "unqorkResource/export/78bdc-df4a-41a61/0.csv",!!
      "format": "CSV"
        }
    }
 ]

Resources