Bulk Operations

CSV Data Load

Ingests CSV data, provided directly in the operation as an insert, update or upsert into the specified database table.

• operation (required) - must always be csv_data_load

• action (optional) - type of action you want to perform - insert, update or upsert. The default is insert

• database (optional) - name of the database where you are loading your data. The default is data

• table (required) - name of the table where you are loading your data

• data (required) - csv data to import into Harper

Body

{
  "operation": "csv_data_load",
  "database": "dev",
  "action": "insert",
  "table": "breed",
  "data": "id,name,section,country,image\n1,ENGLISH POINTER,British and Irish Pointers and Setters,GREAT BRITAIN,http://www.fci.be/Nomenclature/Illustrations/001g07.jpg\n2,ENGLISH SETTER,British and Irish Pointers and Setters,GREAT BRITAIN,http://www.fci.be/Nomenclature/Illustrations/002g07.jpg\n3,KERRY BLUE TERRIER,Large and medium sized Terriers,IRELAND,\n"
}

Response: 200

  {
      "message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69",
      "job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69"
  }

---

CSV File Load

Ingests CSV data, provided via a path on the local filesystem, as an insert, update or upsert into the specified database table.

Note: The CSV file must reside on the same machine on which Harper is running. For example, the path to a CSV on your computer will produce an error if your Harper instance is a cloud instance.

• operation (required) - must always be csv_file_load

• action (optional) - type of action you want to perform - insert, update or upsert. The default is insert

• database (optional) - name of the database where you are loading your data. The default is data

• table (required) - name of the table where you are loading your data

• file_path (required) - path to the csv file on the host running Harper

Body

{
  "operation": "csv_file_load",
  "action": "insert",
  "database": "dev",
  "table": "breed",
  "file_path": "/home/user/imports/breeds.csv"
}

Response: 200

{
  "message": "Starting job with id 3994d8e2-ec6a-43c4-8563-11c1df81870e",
  "job_id": "3994d8e2-ec6a-43c4-8563-11c1df81870e"
}

---

CSV URL Load

Ingests CSV data, provided via URL, as an insert, update or upsert into the specified database table.

• operation (required) - must always be csv_url_load

• action (optional) - type of action you want to perform - insert, update or upsert. The default is insert

• database (optional) - name of the database where you are loading your data. The default is data

• table (required) - name of the table where you are loading your data

• csv_url (required) - URL to the csv

Body

{
  "operation": "csv_url_load",
  "action": "insert",
  "database": "dev",
  "table": "breed",
  "csv_url": "https://s3.amazonaws.com/complimentarydata/breeds.csv"
}

Response: 200

{
  "message": "Starting job with id 332aa0a2-6833-46cd-88a6-ae375920436a",
  "job_id": "332aa0a2-6833-46cd-88a6-ae375920436a"
}

---

Import from S3

This operation allows users to import CSV or JSON files from an AWS S3 bucket as an insert, update or upsert.

• operation (required) - must always be import_from_s3

• action (optional) - type of action you want to perform - insert, update or upsert. The default is insert

• database (optional) - name of the database where you are loading your data. The default is data

• table (required) - name of the table where you are loading your data

• s3 (required) - object containing required AWS S3 bucket info for operation:

  • aws_access_key_id - AWS access key for authenticating into your S3 bucket

  • aws_secret_access_key - AWS secret for authenticating into your S3 bucket

  • bucket - AWS S3 bucket to import from

  • key - the name of the file to import - the file must include a valid file extension ('.csv' or '.json')

  • region - the region of the bucket

Body

{
  "operation": "import_from_s3",
  "action": "insert",
  "database": "dev",
  "table": "dog",
  "s3": {
    "aws_access_key_id": "YOUR_KEY",
    "aws_secret_access_key": "YOUR_SECRET_KEY",
    "bucket": "BUCKET_NAME",
    "key": "OBJECT_NAME",
    "region": "BUCKET_REGION"
  }
}

Response: 200

{
  "message": "Starting job with id 062a1892-6a0a-4282-9791-0f4c93b12e16",
  "job_id": "062a1892-6a0a-4282-9791-0f4c93b12e16"
}