Bulk Data Operations: Export
This feature is currently in production early-access. Functionality is subject to change as this feature is prepared for general access.
Overview
The Bulk Data Operations (BDO) Export API APIs (application programming interfaces) are a set of protocols and definitions developers use to build and integrate application software. APIs act as the connective tissue between products and services. Endpoint lets you export your submission data Also known as Record Data. Submission data consists of information saved by Unqork components. View submission data by using the angular command, or in Record Collections. out of the Unqork platform and into a file or object storage service Object storage, also known as object-based storage, is a computer data storage architecture designed to handle large amounts of unstructured data.Popular object storage services include Amazon S3, Azure Blob Storage, and Google Cloud Storage.. You can make an export request using curl, Postman, or any other service that supports a POST The HTTP POST method is used to send data to a server to create or update a resource, typically including the data in the body of the request. request.
To discover how to set up API calls using a third-party program, view our Testing API Endpoints Using Third-Party Applications article.
You can test the BDO Export endpoint using the Unqork Services page. To access the Unqork Services page, enter the following address into your browser: https://{environment}.unqork.io/fbu/uapi/docs/documentation/#/Bulk%20Operations/import. Replace {environment} with the name of your Unqork environment.
Exporting Files Using Bulk Data Operations
BDO Export uses the HTTP HTTP (Hypertext Transfer Protocol) is an application-layer protocol used to transmit hypermedia documents like HTML. POST The HTTP POST method is used to send data to a server to create or update a resource, typically including the data in the body of the request. method and the following endpoint: https://{environment}.unqork.io/fbu/uapi/bulk-operations/import. Replace {environment} with the name of your Unqork environment.
{
"cloudStorageBucketName": "string",
"cloudStorageServiceName": "string",
"cloudStorageTargetDirectory": "string",
"dataModelOrModuleId": "string",
"dateFieldEndToFilterOn": "string",
"dateFieldStartToFilterOn": "string",
"dateFieldToFilterOn": "created",
"exportFullRecord": "true",
"fileFormat": "CSV",
"name": "string",
"numberOfFilesToGenerate": 1,
"targetRecordType": "ALL"
}
The following parameters are available for the Bulk Data Export schema Schema is a declarative language that defines the structure, content, and constraints of the data.:
Key |
Possible Values |
Description |
Required |
---|---|---|---|
cloudStorageBucketName |
string (no spaces) |
Enter the name of the cloud service bucket A bucket is a container in cloud computing that stores data. Popular cloud services that use buckets include Amazon S3, Azure Blob Storage, and Google Cloud Storage where you want to export the data. For example, your Amazon S3 storage might have a bucket named user-data-bucket. When using cloud storage, cloudStorageServiceName, cloudStorageBucketName, and cloudStorageTargetDirectory must be included in your request. Bucket names do not have empty spaces in them as they are used as a URL A URL (uniform resource locator) is a unique identifier used to locate a resource on the internet. Also known as a web address. or URI URI (Uniform Resource Identifier) identifies a resource by name, location, or both. address. For example, http://s3.amazonaws.com/bucket-name/. |
Optional |
cloudStorageServiceName |
string (no spaces) |
Enter the Service Name of the service created in Services Administration. Unqork supports connecting the following object storage services Object storage, also known as object-based storage, is a computer data storage architecture designed to handle large amounts of unstructured data.Popular object storage services include Amazon S3, Azure Blob Storage, and Google Cloud Storage.:
Add Amazon S3 or Azure Blob Storage services using our Services Administration - How to: Add a Service article. |
Optional |
cloudStorageTargetDirectory |
string (no spaces) |
If your bucket A bucket is a container in cloud computing that stores data. Popular cloud services that use buckets include Amazon S3, Azure Blob Storage, and Google Cloud Storage has directories, specify the path where you want to export the data. For example: userData/moduleName/submissions. |
Optional |
dataModelOrModuleId |
string |
Specify the Data Model ID, or Module ID, containing the data you want to export. To find a Data Model's ID value, view our How to: Query Data Model Records Data article. |
Yes |
dateFieldEndToFilterOn |
string (YYYY-MM-DDTHH:mm:ss:sssZ) |
Export data record up to this date. Enter the date and time in ISO 8601 Format A standard format for representing dates and times. ISO 8601 format is represented in the following order: year, month, day, hour, minutes, seconds, and milliseconds. (UTC Coordinated Universal Time (UTC) is a standard, globally-accepted timezone used to regulate clocks and time around the world.). For example, setting this key's value to 2024-10-17T18:02:55.101Z would export data recorded before October 10th, 2024 at 6:02:55.101PM Zulu time. To export data from a specific time frame, use dateFieldStartToFilterOn as the start date, and dateFieldEndToFilterOn as the end date. |
Optional |
dateFieldStartToFilterOn |
string (YYYY-MM-DDTHH:mm:ss:sssZ) |
Export data recorded after this date. Enter the date and time in ISO 8601 Format A standard format for representing dates and times. ISO 8601 format is represented in the following order: year, month, day, hour, minutes, seconds, and milliseconds. (UTC Coordinated Universal Time (UTC) is a standard, globally-accepted timezone used to regulate clocks and time around the world.). For example, setting this key's value to 2024-10-17T18:02:55.101Z would export data recorded after October 10th, 2024 at 6:02:55.101PM Zulu time. To export data from a specific time frame, use dateFieldStartToFilterOn as the start date, and dateFieldEndToFilterOn as the end date. |
Optional |
dateFieldToFilterOn |
created modified |
Filter exported data by record created, or modified dates. When using this field, you must include dateFieldEndToFilterOn or dateFieldStartToFilterOn, or both in your request. |
|
exportFullRecord |
Boolean |
When set to true, it includes all record details, like _id, owner, metadata, and so on. When set to false, it includes only record data nested in the data object. Unqork recommends setting this key to false. |
Yes |
fileFormat |
CSV ZIPWITHCSV JSON ZIPWITHJSON ZIPWITHMULTIPLEJSON |
Specify the file format to extract to. File formats include:
Value must be uppercase. |
Yes |
name |
string |
Enter a value as the name of the extracted file. Names can include spaces. This value also displays under the Job Tracker's File Name column. |
Yes |
numberOfFilesToGenerate |
number |
Set the preferred amount of files to export. This number cannot be larger than the total files exported. Unqork recommends setting this value to 1. If the resulting file would be larger than 100 MB, the export operation automatically creates additional files. |
Yes |
targetRecordType |
ALL ACTIVE ARCHIVED |
Specify the type of record to export. The following options are available:
The value must be uppercase. |
Yes |
Example Response
In the BDO Export response below, the location field contains the address of the exported file for download:
Larger export operations might require additional time to process. Use the /fbu/uapi/bulk-operations/job/{jobId} endpoint to check the status of the job.
{
"id": "67129acfc2f4b88db7a26b59",
"created": 1729272527398,
"modified": 1729272527398,
"createdBy": "creatorAccount@creatorCompany.com",
"modifiedBy": "creatorAccount@creatorCompany.com",
"data": {
"type": "EXPORT",
"collection": "658c3c6270a73gb027a0f3a2",
"dataLocation": {
"database": "unqork",
"collection": "submissions",
"entityType": "MODULE"
},
"name": "Example Export",
"steps": [
{
"stepType": "EXPORT",
"status": "created",
"options": {
"export": {
"chunks": 1,
"filter": {
"showActive": true,
"showArchive": true,
"form": "658c3c6270a73gb027a0f3a2",
"dateFilter": {
"start": "2024-09-17T18:02:55.101Z",
"end": "2024-10-17T20:02:55.101Z",
"dateField": "created"
}
},
"projection": {
"data": true
}
}
},
"files": [
{
"destination": {
"name": "0",
!!"location": "unqorkResource/export/9adf217f-5ccb-449c-9920-f1e0000597d7/0.csv",!!
"format": "CSV"
}
}
],
"collection": "658c3c6270a73gb027a0f3a2",
"dataLocation": {
"database": "unqork",
"collection": "submissions",
"entityType": "MODULE"
}
}
]
}
}
In the response above, the location key's value contains the file's address. Use your preferred file download method to extract the file.
To learn how to use the location key to create a download URL, view our Bulk Operations: Download-Link article.
Viewing a Bulk Data Export Operation Job's Details
After starting a Bulk Data Export operation, you can view the job details using the /fbu/uapi/bulk-operations/job/{jobId} endpoint with the GET The HTTP GET method is used to request data from a server, typically retrieving information without modifying any resources. method. Copy the export response's id value and replace the job detail's {jobId} placeholder. For example, https://{environment}.unqork.io/fbu/uapi/bulk-operations/job/67169684ec1bf4776fa760a5.
Example Response
Below is an example of a Get Job Details endpoint response:
{
"id": "67114d26340b0b7c0510c680",
"created": 1729187110966,
"modified": 1729187221199,
"createdBy": "creatorEmail@creatorCompany.com",
"modifiedBy": null,
"data": {
"type": "EXPORT",
"collection": "658c3c6b70573db027a0f3a2",
"dataLocation": {
"database": "unqork",
"collection": "submissions",
"entityType": "MODULE"
},
"name": "Example Export",
"steps": [
{
"stepType": "EXPORT",
"id": "67114d50b056c32e103b11e3",
"status": "completed",
"files": [
{
"destination": {
"name": "0",
!!"location": "unqorkResource/export/78b9abdc-df4a-42e0-8b02-1ce0377a1a61/0.csv",!!
"format": "CSV"
}
}
],
"options": {
"export": {
"chunks": 1,
"filter": {
"showActive": true,
"showArchive": false,
"form": "658c3c6b70573db027a0f3a2"
},
"projection": {
"data": false
}
}
},
"dataLocation": {
"database": "unqork",
"collection": "submissions",
"entityType": "MODULE"
},
"collection": "658c3c6b70573db027a0f3a2",
"details": {
"dataLocation": {
"database": "unqork",
"collection": "submissions",
"entityType": "MODULE"
},
"chunks": [
{
"chunkQueueId": "0",
"status": "completed",
"docs": {
"total": 4,
"processed": 4,
"processedSize": 1723
},
"startTime": "2024-10-17T17:46:52.716Z",
"endTime": "2024-10-17T17:46:52.755Z"
}
],
!!"docs": {
"total": 4,
"processed": 4
}!!
}
}
],
!!"status": "completed"!!
}
}
Using the above example response, the following properties might be important for Creator Also known as Unqork Users, or Designer Users; is anyone who is inside the Unqork platform. processes or pipeline development:
Setting | Description | Example |
---|---|---|
status |
The current status of the operation. The overall operation status is displayed at the bottom of the data object. For troubleshooting, review the status property contained in each stepType in the steps array. This value is also reflected in the Job Tracker's Job Status column. Status values include:
|
Copy
|
docs |
Located in the details object, the docs objects displays the following properties:
|
Copy
|
Located in the files array, the location property contains the file key used to create the download file. |
Copy
|
Resources