Add a BigQuery Source Datastore
A source datastore is a storage location Qualytics connects to so it can profile, scan, and monitor data. Adding BigQuery as a source lets Qualytics query the datasets in your Google Cloud project through the BigQuery JDBC driver and run quality operations on the tables it discovers.
Before you start, review the BigQuery Permissions and the Authentication page for the service account setup.
Field reference
The Add Datastore page shows the sections below when BigQuery is selected. When reusing an existing connection, the Connection Properties and Secrets Management sections come already filled in and read-only: Qualytics has already validated those credentials, so you fill in only the Datastores Extraction and the Datastore Properties. To change a saved connection's credentials, edit the connection through the Manage Connections page; edits there apply to every datastore that reuses the connection.
Connection Properties
These fields define the identity and working storage of the connection. They belong to the connection: when reusing an existing connection, they come already filled in and read-only.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Connection Name | Text | A label for the saved connection (e.g., acme_bigquery_warehouse), so other datastores can reuse it later. |
|
| Temp Dataset ID | Text | A dataset where the driver stores large query results and temporary tables. Recommended for performance and cost control; see Create a BigQuery Temp Dataset for the setup and the extra permission it needs. | |
| Service Account Key | File | The JSON key file of the service account Qualytics uses. It already contains the account's email and private key; see Authentication for how to create one. |
Secrets Management
This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Login URL | Text | The Vault endpoint Qualytics uses to authenticate (e.g., https://vault.example.com/v1/auth/approle/login). |
|
| Credentials Payload | Text | A JSON body containing the credentials Vault expects (e.g., {"role_id":"...","secret_id":"..."}). |
|
| Token JSONPath | Text | The JSONPath that extracts the client token from Vault's response. Defaults to $.auth.client_token. |
|
| Secret URL | Text | The Vault path where the secret is stored (e.g., https://vault.example.com/v1/secret/data/bigquery). |
|
| Token Header Name | Text | The HTTP header name used to send the token. Defaults to X-Vault-Token. |
|
| Data JSONPath | Text | The JSONPath that extracts the secret payload from Vault's response. Defaults to $.data. |
Note
Once the secrets manager is configured, reference any secret value in the connection fields using ${key}. Qualytics resolves the secret at the moment the connection is opened, so changed keys take effect on the next connection.
Datastores Extraction
Pick the project and dataset(s) Qualytics should read from. You fill these in on both flows.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Project ID | Option | The Google Cloud project that holds your datasets. Click the refresh icon to fetch the projects visible to the credentials. | |
| Dataset ID | Option | One or more datasets inside the selected project. Each dataset you pick becomes its own Qualytics datastore. Click the refresh icon to load the available datasets. |
Multiple datasets
Selecting more than one dataset creates one source datastore per dataset, named from the Name Template. See Multi-Schema Source Datastore Creation for details.
Datastore Properties
Common fields for every source datastore, shown below the Datastores Extraction section. You fill these in on both flows.
| FIELD | REQUIRED | TYPE | DESCRIPTION |
|---|---|---|---|
| Name Template | Text | Defines the naming pattern for each source datastore being created. Use {{schema}} as a placeholder that gets replaced with the actual dataset name (e.g., bq_{{schema}} becomes bq_sales_curated). Left empty, the datastore is named from the connection name and the dataset. |
|
| Group | Option | Organizes your datastores under a shared group in the navigation tree. Select an existing group or create a new one with the Add New Group toggle. | |
| Teams | Option | Select one or more teams to associate with this source datastore. | |
| Initiate Sync | Checkbox | Automatically sync the datastore to detect containers and fields after creation. |
Steps
There are two ways to set up the connection: reuse a connection you already saved (Existing Connection) or create a new one from scratch (New Connection). The tabs below walk through each option; pick the one you want to follow. Each field is described in the Field reference above.
Step 1: Navigate to the Source Datastores page.
Step 2: Click the Add Source Datastore button at the top-right corner.
Step 3: The Add Datastore page opens.
Step 4: Select New Connection next to the Search field.
Step 5: Select BigQuery from the connector grid. Use the search field to filter connectors by name.
Step 6: Fill in the Connection Properties: the Connection Name, optionally a Temp Dataset ID, and the Service Account Key JSON file, attached through Upload File .
Step 7: Optionally, expand Secrets Management to retrieve credentials from a secrets manager.
Step 8: Fill in the Datastores Extraction fields (Project ID and Dataset ID) and the Datastore Properties.
Step 9: Click Test connection. A success message confirms that the connection has been verified.
Info
The Finish and Next buttons stay disabled until the connection test passes on the current values. If you change a connection field after a successful test, test again. If the test fails, see Troubleshooting.
Step 10: Click Finish to create the datastore.
Tip
To link an enrichment datastore so Qualytics can store anomalies and metadata from the first operation, click Next instead of Finish. See Link Enrichment on Datastore Creation.
Step 11: A success dialog confirms that your datastore has been added. Click Go to your datastore to open its page.
Step 1: Navigate to the Source Datastores page.
Step 2: Click the Add Source Datastore button at the top-right corner.
Step 3: The Add Datastore page opens.
Step 4: Select Existing Connection next to the Search field.
Step 5: Select the saved BigQuery connection from the grid. Use the search field to filter connections by name. The Connection Properties and Secrets Management sections come already filled in and read-only.
Start a new connection from this one
To use the selected connection as a starting point for a brand-new connection instead, click the Duplicate as a new connection button on the selected connection. The form switches to New Connection mode with the connection's settings already filled in for you to adjust.
Step 6: Fill in the Datastores Extraction fields (Project ID and Dataset ID) and the Datastore Properties. These are the only fields left to fill in, and among the properties only Teams is required.
Step 7: Click Test connection. A success message confirms that the connection has been verified.
Info
The Finish and Next buttons stay disabled until the connection test passes on the current values. If the test fails, see Troubleshooting.
Step 8: Click Finish to create the datastore.
Tip
To link an enrichment datastore so Qualytics can store anomalies and metadata from the first operation, click Next instead of Finish. See Link Enrichment on Datastore Creation.
Step 9: A success dialog confirms that your datastore has been added. Click Go to your datastore to open its page.