Bulk Data Operations: Export

Overview

The Bulk Data Operations (BDO) Export API endpoint lets you export your submission data out of the Unqork platform and into a file or object storage service. You can make an export request using curl, Postman, or any other service that supports a POST request.

To discover how to set up API calls using a third-party program, view our Testing API Endpoints Using Third-Party Applications article.

You can test the BDO Export endpoint using the Unqork Services page. To access the Unqork Services page, enter the following address into your browser: https://{environment}.unqork.io/fbu/uapi/docs/documentation/#/Bulk%20Operations/export. Replace {environment} with the name of your Unqork environment.

Exporting Files Using Bulk Data Operations

BDO Export uses the HTTP POST method and the following endpoint: https://{environment}.unqork.io/fbu/uapi/bulk-operations/export. Replace {environment} with the name of your Unqork environment.

Export jobs are limited to 100 MB.

Copy 
{
  "cloudStorageBucketName": "string",
  "cloudStorageServiceName": "string",
  "cloudStorageTargetDirectory": "string",
  "dataModelOrModuleId": "string",
  "dateFieldEndToFilterOn": "string",
  "dateFieldStartToFilterOn": "string",
  "dateFieldToFilterOn": "created",
  "exportFullRecord": "true",
  "fileFormat": "CSV",
  "name": "string",
  "numberOfFilesToGenerate": 1,
  "targetRecordType": "ALL"
}

The following parameters are available for the Bulk Data Export schema:

Key

Possible Values

Description

 Required

cloudStorageBucketName

string (no spaces)

Enter the name of the cloud service bucket where you want to export the data. For example, your Amazon S3 storage might have a bucket named user-data-bucket.

When using cloud storage, cloudStorageServiceName, cloudStorageBucketName, and cloudStorageTargetDirectory must be included in your request.

Bucket names do not have empty spaces in them as they are used as a URL or URI address. For example, http://s3.amazonaws.com/bucket-name/.

Optional

cloudStorageServiceName

string (no spaces)

Enter the Service Name of the service created in Services Administration. Unqork supports connecting the following object storage services:

Add Amazon S3 or Azure Blob Storage services using our Services Administration - How to: Add a Service article.

Optional

cloudStorageTargetDirectory

string (no spaces)

If your bucket has directories, specify the path where you want to export the data. For example: userData/moduleName/submissions.

Optional

dataModelOrModuleId

string

Specify the Data Model ID, or Module ID, containing the data you want to export.

To find a Data Model's ID value, view our How to: Query Data Model Records Data article.

Yes

dateFieldEndToFilterOn

string (YYYY-MM-DDTHH:mm:ss:sssZ)

Export data record up to this date. Enter the date and time in ISO 8601 Format (UTC). For example, setting this key's value to 2024-10-17T18:02:55.101Z would export data recorded before October 10th, 2024 at 6:02:55.101PM Zulu time.

To export data from a specific time frame, use dateFieldStartToFilterOn as the start date, and dateFieldEndToFilterOn as the end date.

Optional

dateFieldStartToFilterOn

string (YYYY-MM-DDTHH:mm:ss:sssZ)

Export data recorded after this date. Enter the date and time in ISO 8601 Format (UTC). For example, setting this key's value to 2024-10-17T18:02:55.101Z would export data recorded after October 10th, 2024 at 6:02:55.101PM Zulu time.

To export data from a specific time frame, use dateFieldStartToFilterOn as the start date, and dateFieldEndToFilterOn as the end date.

Optional

dateFieldToFilterOn

created

modified

Filter exported data by record created, or modified dates.

When using this field, you must include dateFieldEndToFilterOn or dateFieldStartToFilterOn, or both in your request.

Optional

exportFullRecord

Boolean

When set to true, it includes all record details, like _id, owner, metadata, and so on.

When set to false, it includes only record data nested in the data object.

Unqork recommends setting this key to false.

Yes

fileFormat

CSV

ZIPWITHCSV

JSON

ZIPWITHJSON

ZIPWITHMULTIPLEJSON

Specify the file format to extract to. File formats include:

Value must be uppercase.

Yes

name

string

Enter a value as the name of the extracted file. Names can include spaces. This value also displays under the Job Tracker's File Name column.

Yes

numberOfFilesToGenerate

number

Set the preferred amount of files to export. This number cannot be larger than the total files exported.

Unqork recommends setting this value to 1. If the resulting file would be larger than 100 MB, the export operation automatically creates additional files.

Yes

targetRecordType

ALL

ACTIVE

ARCHIVED

Specify the type of record to export. The following options are available:

  • ALL: Export Active and Archived records.

  • ACTIVE: Export records that have not been archived.

  • ARCHIVED: Export records that have been archived, but not permanently deleted.

The value must be uppercase.

Yes

Example Response

In the BDO Export response below, the location field contains the address of the exported file for download:

Larger export operations might require additional time to process. Use the /fbu/uapi/bulk-operations/job/{jobId} endpoint to check the status of the job.

Copy 
{
    "id": "67129acfc2f4b88db7a26b59",
    "created": 1729272527398,
    "modified": 1729272527398,
    "createdBy": "creatorAccount@creatorCompany.com",
    "modifiedBy": "creatorAccount@creatorCompany.com",
    "data": {
        "type": "EXPORT",
        "collection": "658c3c6270a73gb027a0f3a2",
        "dataLocation": {
            "database": "unqork",
            "collection": "submissions",
            "entityType": "MODULE"
        },
        "name": "Example Export",
        "steps": [
            {
                "stepType": "EXPORT",
                "status": "created",
                "options": {
                    "export": {
                        "chunks": 1,
                        "filter": {
                            "showActive": true,
                            "showArchive": true,
                            "form": "658c3c6270a73gb027a0f3a2",
                            "dateFilter": {
                                "start": "2024-09-17T18:02:55.101Z",
                                "end": "2024-10-17T20:02:55.101Z",
                                "dateField": "created"
                            }
                        },
                        "projection": {
                            "data": true
                        }
                    }
                },
                "files": [
                    {
                        "destination": {
                            "name": "0",
                            "location": "unqorkResource/export/9adf217f-5ccb-449c-9920-f1e0000597d7/0.csv",
                            "format": "CSV"
                        }
                    }
                ],
                "collection": "658c3c6270a73gb027a0f3a2",
                "dataLocation": {
                    "database": "unqork",
                    "collection": "submissions",
                    "entityType": "MODULE"
                }
            }
        ]
    }
}

In the response above, the location key's value contains the file's address. Use your preferred file download method to extract the file.

To learn how to use the location key to create a download URL, view our Bulk Operations: Download-Link article.

Viewing a Bulk Data Export Operation Job's Details

After starting a Bulk Data Export operation, you can view the job details using the /fbu/uapi/bulk-operations/job/{jobId} endpoint with the GET method. Copy the export response's id value and replace the job detail's {jobId} placeholder. For example, https://{environment}.unqork.io/fbu/uapi/bulk-operations/job/67169684ec1bf4776fa760a5.

Example Response

Below is an example of a Get Job Details endpoint response:

Copy 
{
    "id": "67114d26340b0b7c0510c680",
    "created": 1729187110966,
    "modified": 1729187221199,
    "createdBy": "creatorEmail@creatorCompany.com",
    "modifiedBy": null,
    "data": {
        "type": "EXPORT",
        "collection": "658c3c6b70573db027a0f3a2",
        "dataLocation": {
            "database": "unqork",
            "collection": "submissions",
            "entityType": "MODULE"
        },
        "name": "Example Export",
        "steps": [
            {
                "stepType": "EXPORT",
                "id": "67114d50b056c32e103b11e3",
                "status": "completed",
                "files": [
                    {
                        "destination": {
                            "name": "0",
                            "location": "unqorkResource/export/78b9abdc-df4a-42e0-8b02-1ce0377a1a61/0.csv",
                            "format": "CSV"
                        }
                    }
                ],
                "options": {
                    "export": {
                        "chunks": 1,
                        "filter": {
                            "showActive": true,
                            "showArchive": false,
                            "form": "658c3c6b70573db027a0f3a2"
                        },
                        "projection": {
                            "data": false
                        }
                    }
                },
                "dataLocation": {
                    "database": "unqork",
                    "collection": "submissions",
                    "entityType": "MODULE"
                },
                "collection": "658c3c6b70573db027a0f3a2",
                "details": {
                    "dataLocation": {
                        "database": "unqork",
                        "collection": "submissions",
                        "entityType": "MODULE"
                    },
                    "chunks": [
                        {
                            "chunkQueueId": "0",
                            "status": "completed",
                            "docs": {
                                "total": 4,
                                "processed": 4,
                                "processedSize": 1723
                            },
                            "startTime": "2024-10-17T17:46:52.716Z",
                            "endTime": "2024-10-17T17:46:52.755Z"
                        }
                    ],
                    "docs": {
                        "total": 4,
                        "processed": 4
                    }
                }
            }
        ],
        "status": "completed"
    }
}

Using the above example response, the following properties might be important for Creator processes or pipeline development:

Setting Description Example

status

The current status of the operation. The overall operation status is displayed at the bottom of the data object. For troubleshooting, review the status property contained in each stepType in the steps array. This value is also reflected in the Job Tracker's Job Status column.

Status values include:

  • inProgress: The export operation is still processing the file(s).

  • completed: The export is complete.

  • canceled: The export has been manually stopped by the user.

  • failed: The export failed. Review each stepType's value to determine which process failed.
Copy 
{
    "data": {
        "type": "EXPORT",
        "collection": "658c3c6b70573db027a0f3a2",
        "dataLocation": {},
        "name": "Example Export 2",
        "steps": [],
        "status": "completed"
    }
}

docs

Located in the details object, the docs objects displays the following properties:

  • processed: The amount of records successfully processed by the operation.

  • total: The total records processed.
Copy 
"details": {
  "dataLocation": {},
  "chunks": [],
  "docs": {
    "total": 4,
    "processed": 4
    }
  }
}

location

Located in the files array, the location property contains the file key used to create the download file.
Copy 
"files": [
    {
      "destination": {
      "name": "0",
      "location": "unqorkResource/export/78bdc-df4a-41a61/0.csv",
      "format": "CSV"
        }
    }
 ]

Resources