BigQuery API

This page documents BigQuery-specific payload fields for the datastore API. For generic operations (get, list, update, delete, multi-schema bulk), see Datastore API.

BigQuery authenticates with a single credential: the service account key. You can create a datastore with a new inline connection or reuse an existing connection, and BigQuery supports both source and enrichment datastores. All modes use the same endpoint:

Endpoint: POST /api/datastores

Permission: Manager

UI ↔ API field mapping

The API names a few fields differently from the UI. The mapping below avoids the most common copy-paste mistake (sending the dataset name where the project belongs).

Connection Fields

password and parameters belong inside connection when creating an inline connection. When reusing connection_id, its saved billing project and temp dataset settings are reused too. The service account key's project_id does not select billing.

Create a Source Datastore

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "BigQuery Production",
    "connection": {
      "name": "bigquery_production",
      "type": "bigquery",
      "password": "<contents of the service account key JSON, escaped as a string>",
      "parameters": {
        "billing_project_id": "acme-billing",
        "temp_dataset": "qualytics_temp"
      }
    },
    "database": "acme-analytics",
    "schema": "sales_curated",
    "teams": ["Data Platform"],
    "trigger_sync": true
  }'

{
  "id": 42,
  "name": "BigQuery Production",
  "type": "bigquery",
  "store_type": "jdbc",
  "database": "acme-analytics",
  "schema": "sales_curated",
  "connected": true
}

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "BigQuery Finance",
    "connection_id": 7,
    "database": "acme-analytics",
    "schema": "finance_curated",
    "teams": ["Finance Team"],
    "trigger_sync": true
  }'

{
  "id": 43,
  "name": "BigQuery Finance",
  "type": "bigquery",
  "store_type": "jdbc",
  "database": "acme-analytics",
  "schema": "finance_curated",
  "connected": true
}

Create an Enrichment Datastore

BigQuery also supports enrichment datastores. The payload is the same as for a source datastore, plus "enrichment_only": true. Point schema at the dataset Qualytics should write enrichment outputs to; the service account needs the read-write roles listed on the Permissions page.

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "BigQuery Enrichment",
    "connection_id": 7,
    "database": "acme-analytics",
    "schema": "qualytics_enrichment",
    "teams": ["Data Platform"],
    "enrichment_only": true
  }'

{
  "id": 44,
  "name": "BigQuery Enrichment",
  "type": "bigquery",
  "store_type": "jdbc",
  "database": "acme-analytics",
  "schema": "qualytics_enrichment",
  "connected": true
}

To link an enrichment destination to a source datastore, see Datastore API › Link Enrichment Destination.

Other Operations

For get, list, update, delete, test connection, multi-schema bulk creation, and end-to-end automation examples, see the Datastore API reference.

Change or Clear the Billing Project

Endpoint: PUT /api/connections/{id}

Permission: Manager, with permission to edit the connection. See Manage Connections for access requirements.

Changing a connection affects every datastore that uses it. Keep their database and schema values unchanged. Prepare the query permissions and any temp dataset in the new billing project before updating the connection.

Read the connection with GET /api/connections/{id} first. The parameters object replaces the saved object, so include every parameter you want to retain. The following example assumes the connection is named bigquery_production and has only the two parameters shown. Omit password to retain the saved service account key.

curl -X PUT "https://your-instance.qualytics.io/api/connections/7" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "bigquery_production",
    "type": "bigquery",
    "parameters": {
      "billing_project_id": "acme-billing",
      "temp_dataset": "qualytics_temp"
    }
  }'

To test the proposed settings without saving them, send the same body to POST /api/connections/7/test. Success returns 204 No Content.

curl -X PUT "https://your-instance.qualytics.io/api/connections/7" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "bigquery_production",
    "type": "bigquery",
    "parameters": {
      "billing_project_id": "",
      "temp_dataset": "qualytics_temp"
    }
  }'