Bulk Data Operations: Export

Overview

The Bulk Data Operations (BDO) Export API APIs (application programming interfaces) are a set of protocols and definitions developers use to build and integrate application software. APIs act as the connective tissue between products and services. endpoint lets you export your submission data Also known as Record Data. Submission data consists of information saved by Unqork components. View submission data by using the angular command, or in Record Collections. out of the Unqork platform and into a file or object storage service Object storage, also known as object-based storage, is a computer data storage architecture designed to handle large amounts of unstructured data.Popular object storage services include Amazon S3, Azure Blob Storage, and Google Cloud Storage.. You can make an export request using curl, Postman, or any other service that supports a POST The HTTP POST method is used to send data to a server to create or update a resource, typically including the data in the body of the request. request.

To discover how to set up API calls using a third-party program, view our Testing API Endpoints Using Third-Party Applications article.

You can test the BDO Export endpoint using the Unqork Services page. To access the Unqork Services page, enter the following address into your browser: https://{environment}.unqork.io/fbu/uapi/docs/documentation/#/Bulk%20Operations/export. Replace {environment} with the name of your Unqork environment.

Exporting Files Using Bulk Data Operations

BDO Export uses the HTTP HTTP (Hypertext Transfer Protocol) is an application-layer protocol used to transmit hypermedia documents like HTML. POST The HTTP POST method is used to send data to a server to create or update a resource, typically including the data in the body of the request. method and the following endpoint: https://{environment}.unqork.io/fbu/uapi/bulk-operations/export. Replace {environment} with the name of your Unqork environment.

Export jobs are limited to 100 MB.

Copy

{
  "cloudStorageBucketName": "string",
  "cloudStorageServiceName": "string",
  "cloudStorageTargetDirectory": "string",
  "dataModelOrModuleId": "string",
  "dateFieldEndToFilterOn": "string",
  "dateFieldStartToFilterOn": "string",
  "dateFieldToFilterOn": "created",
  "exportFullRecord": "true",
  "fileFormat": "CSV",
  "name": "string",
  "numberOfFilesToGenerate": 1,
  "targetRecordType": "ALL"
}

The following parameters are available for the Bulk Data Export schema Schema is a declarative language that defines the structure, content, and constraints of the data.:

Key	Possible Values	Description	Required
cloudStorageBucketName	string (no spaces)	Enter the name of the cloud service bucket A bucket is a container in cloud computing that stores data. Popular cloud services that use buckets include Amazon S3, Azure Blob Storage, and Google Cloud Storage where you want to export the data. For example, your Amazon S3 storage might have a bucket named user-data-bucket. When using cloud storage, cloudStorageServiceName, cloudStorageBucketName, and cloudStorageTargetDirectory must be included in your request. Bucket names do not have empty spaces in them as they are used as a URL A URL (uniform resource locator) is a unique identifier used to locate a resource on the internet. Also known as a web address. or URI URI (Uniform Resource Identifier) identifies a resource by name, location, or both. address. For example, http://s3.amazonaws.com/bucket-name/.	Optional
cloudStorageServiceName	string (no spaces)	Enter the Service Name of the service created in Services Administration. Unqork supports connecting the following object storage services Object storage, also known as object-based storage, is a computer data storage architecture designed to handle large amounts of unstructured data.Popular object storage services include Amazon S3, Azure Blob Storage, and Google Cloud Storage.: Amazon S3: https://aws.amazon.com/s3/ Azure Blob Storage: https://azure.microsoft.com/en-us/products/storage/blobs Add Amazon S3 or Azure Blob Storage services using our Services Administration - How to: Add a Service article.	Optional
cloudStorageTargetDirectory	string (no spaces)	If your bucket A bucket is a container in cloud computing that stores data. Popular cloud services that use buckets include Amazon S3, Azure Blob Storage, and Google Cloud Storage has directories, specify the path where you want to export the data. For example: userData/moduleName/submissions.	Optional
dataModelOrModuleId	string	Specify the Data Model ID, or Module ID, containing the data you want to export. To find a Data Model's ID value, view our How to: Query Data Model Records Data article.	Yes
dateFieldEndToFilterOn	string (YYYY-MM-DDTHH:mm:ss:sssZ)	Export data record up to this date. Enter the date and time in ISO 8601 Format A standard format for representing dates and times. ISO 8601 format is represented in the following order: year, month, day, hour, minutes, seconds, and milliseconds. (UTC Coordinated Universal Time (UTC) is a standard, globally-accepted timezone used to regulate clocks and time around the world.). For example, setting this key's value to 2024-10-17T18:02:55.101Z would export data recorded before October 10th, 2024 at 6:02:55.101PM Zulu time. To export data from a specific time frame, use dateFieldStartToFilterOn as the start date, and dateFieldEndToFilterOn as the end date.	Optional
dateFieldStartToFilterOn	string (YYYY-MM-DDTHH:mm:ss:sssZ)	Export data recorded after this date. Enter the date and time in ISO 8601 Format A standard format for representing dates and times. ISO 8601 format is represented in the following order: year, month, day, hour, minutes, seconds, and milliseconds. (UTC Coordinated Universal Time (UTC) is a standard, globally-accepted timezone used to regulate clocks and time around the world.). For example, setting this key's value to 2024-10-17T18:02:55.101Z would export data recorded after October 10th, 2024 at 6:02:55.101PM Zulu time. To export data from a specific time frame, use dateFieldStartToFilterOn as the start date, and dateFieldEndToFilterOn as the end date.	Optional
dateFieldToFilterOn	created modified	Filter exported data by record created, or modified dates. When using this field, you must include dateFieldEndToFilterOn or dateFieldStartToFilterOn, or both in your request.	Optional
exportFullRecord	Boolean	When set to true, it includes all record details, like _id, owner, metadata, and so on. When set to false, it includes only record data nested in the data object. Unqork recommends setting this key to false.	Yes
fileFormat	CSV ZIPWITHCSV JSON ZIPWITHJSON ZIPWITHMULTIPLEJSON	Specify the file format to extract to. File formats include: CSV: Exports records in a single CSV Comma-separated values (CSV) is a plain text file format that stores tabular data, such as spreadsheets or databases. CSV files are a common way to store and send data because they are easy to import and export between different programs. file. ZIPWITHCSV: Exports records in a single CSV Comma-separated values (CSV) is a plain text file format that stores tabular data, such as spreadsheets or databases. CSV files are a common way to store and send data because they are easy to import and export between different programs. file, then compresses it into a ZIP A zip file is a file format that can contain multiple files combined and compressed into one file. Since it's a type of compressed file, a zip file can be smaller in size than the files it contains. file. JSON: Exports records in a single JSON Lines JSON Lines is a format for storing structured data that may be processed one record at a time. It works with unix-style text processing tools and shell pipelines. file. ZIPWITHJSON: Exports records in a single JSON Lines JSON Lines is a format for storing structured data that may be processed one record at a time. It works with unix-style text processing tools and shell pipelines. file, then compresses it into a ZIP A zip file is a file format that can contain multiple files combined and compressed into one file. Since it's a type of compressed file, a zip file can be smaller in size than the files it contains. file. ZIPWITHMULTIPLEJSON: Exports each record into individual JSON JSON (JavaScript Object Notation) is an open standard file and data interchange format. Unqork uses JSON for submission (record) data. files where each file represents a single record, then compresses them into a ZIP A zip file is a file format that can contain multiple files combined and compressed into one file. Since it's a type of compressed file, a zip file can be smaller in size than the files it contains. file. Value must be uppercase.	Yes
name	string	Enter a value as the name of the extracted file. Names can include spaces. This value also displays under the Job Tracker's File Name column.	Yes
numberOfFilesToGenerate	number	Set the preferred amount of files to export. This number cannot be larger than the total files exported. Unqork recommends setting this value to 1. If the resulting file would be larger than 100 MB, the export operation automatically creates additional files.	Yes
targetRecordType	ALL ACTIVE ARCHIVED	Specify the type of record to export. The following options are available: ALL: Export Active and Archived records. ACTIVE: Export records that have not been archived. ARCHIVED: Export records that have been archived, but not permanently deleted. The value must be uppercase.	Yes

Example Response

In the BDO Export response below, the location field contains the address of the exported file for download:

Larger export operations might require additional time to process. Use the /fbu/uapi/bulk-operations/job/{jobId} endpoint to check the status of the job.

Copy

{
    "id": "67129acfc2f4b88db7a26b59",
    "created": 1729272527398,
    "modified": 1729272527398,
    "createdBy": "creatorAccount@creatorCompany.com",
    "modifiedBy": "creatorAccount@creatorCompany.com",
    "data": {
        "type": "EXPORT",
        "collection": "658c3c6270a73gb027a0f3a2",
        "dataLocation": {
            "database": "unqork",
            "collection": "submissions",
            "entityType": "MODULE"
        },
        "name": "Example Export",
        "steps": [
            {
                "stepType": "EXPORT",
                "status": "created",
                "options": {
                    "export": {
                        "chunks": 1,
                        "filter": {
                            "showActive": true,
                            "showArchive": true,
                            "form": "658c3c6270a73gb027a0f3a2",
                            "dateFilter": {
                                "start": "2024-09-17T18:02:55.101Z",
                                "end": "2024-10-17T20:02:55.101Z",
                                "dateField": "created"
                            }
                        },
                        "projection": {
                            "data": true
                        }
                    }
                },
                "files": [
                    {
                        "destination": {
                            "name": "0",
                            !!"location": "unqorkResource/export/9adf217f-5ccb-449c-9920-f1e0000597d7/0.csv",!!
                            "format": "CSV"
                        }
                    }
                ],
                "collection": "658c3c6270a73gb027a0f3a2",
                "dataLocation": {
                    "database": "unqork",
                    "collection": "submissions",
                    "entityType": "MODULE"
                }
            }
        ]
    }
}

In the response above, the location key's value contains the file's address. Use your preferred file download method to extract the file.

To learn how to use the location key to create a download URL, view our Bulk Operations: Download-Link article.

Viewing a Bulk Data Export Operation Job's Details

After starting a Bulk Data Export operation, you can view the job details using the /fbu/uapi/bulk-operations/job/{jobId} endpoint with the GET The HTTP GET method is used to request data from a server, typically retrieving information without modifying any resources. method. Copy the export response's id value and replace the job detail's {jobId} placeholder. For example, https://{environment}.unqork.io/fbu/uapi/bulk-operations/job/67169684ec1bf4776fa760a5.

Example Response

Below is an example of a Get Job Details endpoint response:

Copy

{
    "id": "67114d26340b0b7c0510c680",
    "created": 1729187110966,
    "modified": 1729187221199,
    "createdBy": "creatorEmail@creatorCompany.com",
    "modifiedBy": null,
    "data": {
        "type": "EXPORT",
        "collection": "658c3c6b70573db027a0f3a2",
        "dataLocation": {
            "database": "unqork",
            "collection": "submissions",
            "entityType": "MODULE"
        },
        "name": "Example Export",
        "steps": [
            {
                "stepType": "EXPORT",
                "id": "67114d50b056c32e103b11e3",
                "status": "completed",
                "files": [
                    {
                        "destination": {
                            "name": "0",
                            !!"location": "unqorkResource/export/78b9abdc-df4a-42e0-8b02-1ce0377a1a61/0.csv",!!
                            "format": "CSV"
                        }
                    }
                ],
                "options": {
                    "export": {
                        "chunks": 1,
                        "filter": {
                            "showActive": true,
                            "showArchive": false,
                            "form": "658c3c6b70573db027a0f3a2"
                        },
                        "projection": {
                            "data": false
                        }
                    }
                },
                "dataLocation": {
                    "database": "unqork",
                    "collection": "submissions",
                    "entityType": "MODULE"
                },
                "collection": "658c3c6b70573db027a0f3a2",
                "details": {
                    "dataLocation": {
                        "database": "unqork",
                        "collection": "submissions",
                        "entityType": "MODULE"
                    },
                    "chunks": [
                        {
                            "chunkQueueId": "0",
                            "status": "completed",
                            "docs": {
                                "total": 4,
                                "processed": 4,
                                "processedSize": 1723
                            },
                            "startTime": "2024-10-17T17:46:52.716Z",
                            "endTime": "2024-10-17T17:46:52.755Z"
                        }
                    ],
                    !!"docs": {
                        "total": 4,
                        "processed": 4
                    }!!
                }
            }
        ],
        !!"status": "completed"!!
    }
}

Using the above example response, the following properties might be important for Creator Also known as Unqork Users, or Designer Users; is anyone who is inside the Unqork platform. processes or pipeline development:

Setting Description Example

Setting	Description	Example
status	The current status of the operation. The overall operation status is displayed at the bottom of the data object. For troubleshooting, review the status property contained in each stepType in the steps array. This value is also reflected in the Job Tracker's Job Status column. Status values include: inProgress: The export operation is still processing the file(s). completed: The export is complete. canceled: The export has been manually stopped by the user. failed: The export failed. Review each stepType's value to determine which process failed.	Copy `{ "data": { "type": "EXPORT", "collection": "658c3c6b70573db027a0f3a2", "dataLocation": {}, "name": "Example Export 2", "steps": [], !!"status": "completed"!! } }`
docs	Located in the details object, the docs objects displays the following properties: processed: The amount of records successfully processed by the operation. total: The total records processed.	Copy `"details": { "dataLocation": {}, "chunks": [], "docs": { !!"total": 4,!! !!"processed": 4!! } } }`
location	Located in the files array, the location property contains the file key used to create the download file.	Copy `"files": [ { "destination": { "name": "0", !!"location": "unqorkResource/export/78bdc-df4a-41a61/0.csv",!! "format": "CSV" } } ]`

status

The current status of the operation. The overall operation status is displayed at the bottom of the data object. For troubleshooting, review the status property contained in each stepType in the steps array. This value is also reflected in the Job Tracker's Job Status column.

Status values include:

inProgress: The export operation is still processing the file(s).
completed: The export is complete.
canceled: The export has been manually stopped by the user.
failed: The export failed. Review each stepType's value to determine which process failed.

Copy

{
    "data": {
        "type": "EXPORT",
        "collection": "658c3c6b70573db027a0f3a2",
        "dataLocation": {},
        "name": "Example Export 2",
        "steps": [],
        !!"status": "completed"!!
    }
}

docs

Located in the details object, the docs objects displays the following properties:

processed: The amount of records successfully processed by the operation.
total: The total records processed.

Copy

"details": {
  "dataLocation": {},
  "chunks": [],
  "docs": {
    !!"total": 4,!!
    !!"processed": 4!!
    }
  }
}

location

Located in the files array, the location property contains the file key used to create the download file.

Copy

"files": [
    {
      "destination": {
      "name": "0",
      !!"location": "unqorkResource/export/78bdc-df4a-41a61/0.csv",!!
      "format": "CSV"
        }
    }
 ]

Bulk Data Operations: Export

Overview

Exporting Files Using Bulk Data Operations

Example Response

Viewing a Bulk Data Export Operation Job's Details

Example Response

Resources

Feedback

Sorry about that

Great!

Looking for more help?