BigQuery API¶

This page documents BigQuery-specific payload fields for the datastore API. For generic operations (get, list, update, delete, multi-schema bulk), see Datastore API.

BigQuery authenticates with a single credential: the service account key. You can create a datastore with a new inline connection or reuse an existing connection, and BigQuery supports both source and enrichment datastores. All modes use the same endpoint:

Endpoint: POST /api/datastores

Permission: Manager, plus the Editor team permission on at least one of the assigned teams (the Public team when teams is left out). Admins are exempt.

UI ↔ API field mapping¶

The API names a few fields differently from the UI. The mapping below avoids the most common copy-paste mistake (sending the dataset name where the project belongs).

Connection Fields¶

password and parameters belong inside connection when creating an inline connection. When reusing connection_id, its saved billing project and temp dataset settings are reused too. The service account key's project_id does not select billing.

Create a Source Datastore¶

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "BigQuery Production",
    "connection": {
      "name": "bigquery_production",
      "type": "bigquery",
      "password": "<contents of the service account key JSON, escaped as a string>",
      "parameters": {
        "billing_project_id": "acme-billing",
        "temp_dataset": "qualytics_temp"
      }
    },
    "database": "acme-analytics",
    "schema": "sales_curated",
    "teams": ["Data Platform"],
    "trigger_sync": true
  }'

{
  "id": 42,
  "name": "BigQuery Production",
  "type": "bigquery",
  "store_type": "jdbc",
  "database": "acme-analytics",
  "schema": "sales_curated",
  "connected": true
}

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "BigQuery Finance",
    "connection_id": 7,
    "database": "acme-analytics",
    "schema": "finance_curated",
    "teams": ["Finance Team"],
    "trigger_sync": true
  }'

{
  "id": 43,
  "name": "BigQuery Finance",
  "type": "bigquery",
  "store_type": "jdbc",
  "database": "acme-analytics",
  "schema": "finance_curated",
  "connected": true
}

Create an Enrichment Datastore¶

BigQuery also supports enrichment datastores. The payload is the same as for a source datastore, plus "enrichment_only": true. Point schema at the dataset Qualytics should write enrichment outputs to; the service account needs the read-write roles listed on the Permissions page.

curl -X POST "https://your-instance.qualytics.io/api/datastores" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "BigQuery Enrichment",
    "connection_id": 7,
    "database": "acme-analytics",
    "schema": "qualytics_enrichment",
    "teams": ["Data Platform"],
    "enrichment_only": true
  }'

{
  "id": 44,
  "name": "BigQuery Enrichment",
  "type": "bigquery",
  "store_type": "jdbc",
  "database": "acme-analytics",
  "schema": "qualytics_enrichment",
  "connected": true
}

To link an enrichment destination to a source datastore, see Datastore API › Link Enrichment Destination.

Other Operations¶

For get, list, update, delete, test connection, multi-schema bulk creation, and end-to-end automation examples, see the Datastore API reference.

Change or Clear the Billing Project¶

Endpoint: PUT /api/connections/{id}

Permission: Manager, with permission to edit the connection. See Manage Connections for access requirements.

Changing a connection affects every datastore that uses it. Keep their database and schema values unchanged. Prepare the query permissions in the new billing project before updating the connection. The temp dataset stays in each datastore's data project.

Read the connection with GET /api/connections/{id} first. The parameters object replaces the saved object, so include every parameter you want to retain. The following example assumes the connection is named bigquery_production and has only the two parameters shown. Omit password to retain the saved service account key.

curl -X PUT "https://your-instance.qualytics.io/api/connections/7" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "bigquery_production",
    "type": "bigquery",
    "parameters": {
      "billing_project_id": "acme-billing",
      "temp_dataset": "qualytics_temp"
    }
  }'

To test the proposed settings without saving them, send the same body to POST /api/connections/7/test. Success returns 204 No Content.

curl -X PUT "https://your-instance.qualytics.io/api/connections/7" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "bigquery_production",
    "type": "bigquery",
    "parameters": {
      "billing_project_id": "",
      "temp_dataset": "qualytics_temp"
    }
  }'