Google Cloud Storage

Adding and configuring a Google Cloud Storage connection within Qualytics empowers the platform to build a symbolic link with your file system to perform operations like data discovery, visualization, reporting, syncing, profiling, scanning, anomaly surveillance, and more.

This documentation provides a step-by-step guide on how to add Google Cloud Storage as a source datastore in Qualytics. It covers the entire process, from initial connection setup to testing and finalizing the configuration. A Google Cloud Storage datastore can also be the place Qualytics writes to; see Use as an Enrichment Datastore.

By following these instructions, enterprises can ensure their Google Cloud Storage environment is properly connected with Qualytics, unlocking the platform's potential to help you proactively manage your full data quality lifecycle.

Let’s get started 🚀

Google Cloud Storage Setup Guide

This guide walks you through what you need from Google Cloud before adding the datastore to Qualytics: the bucket URI and a service account key.

Retrieve the Google Cloud Storage URI

To retrieve the Cloud Storage URI, follow the given steps:

1. Go to the Cloud Storage Console.
2. Navigate to the location of the object (file) that holds the source data.
3. At the top of the Cloud Storage console, locate and note down the path to the object.
4. Create the URI using the following format:

gs://<bucket_name>

The URI stops at the bucket. You choose the folder inside it with Root Path when you add the datastore, so do not append a path here.

Create a Service Account Key

The connector authenticates with a service account key, which you upload as a JSON file when you add the datastore. To create one:

Step 1: In the Google Cloud console, open IAM & Admin, then Service Accounts, and select the project that owns the bucket.

Step 2: Click Create Service Account, give it a name, and create it. You can skip the optional project-level role step, because the role that matters is granted on the bucket, as described below.

Step 3: Open the service account, go to its Keys tab, and click Add Key, then Create new key.

Step 4: Choose JSON and create the key. The file downloads once and cannot be downloaded again, so store it somewhere safe.

Step 5: Grant the service account the role it needs on the target bucket, per the privileges below.

Datastore Google Cloud Storage Privileges

The permissions required depend on whether the Google Cloud Storage datastore is used as a source or enrichment datastore. Qualytics accesses the bucket with the service account whose key you upload.

Minimum Permissions (Source Datastore)

The service account must have the following permissions:

Additional Permissions for an Enrichment Datastore

When Google Cloud Storage is used as an enrichment datastore, the following additional permissions are required:

Example IAM Policy

Replace <SERVICE_ACCOUNT_EMAIL> and <BUCKET_NAME> with your actual values.

Source Datastore (Read-Only)

{
  "bindings": [
    {
      "role": "roles/storage.objectViewer",
      "members": [
        "serviceAccount:<SERVICE_ACCOUNT_EMAIL>"
      ]
    }
  ]
}

Enrichment Datastore (Read-Write)

{
  "bindings": [
    {
      "role": "roles/storage.objectAdmin",
      "members": [
        "serviceAccount:<SERVICE_ACCOUNT_EMAIL>"
      ]
    }
  ]
}

Assigning via gcloud CLI

# Source Datastore (Read-Only)
gsutil iam ch \
  serviceAccount:<SERVICE_ACCOUNT_EMAIL>:roles/storage.objectViewer \
  gs://<BUCKET_NAME>

# Enrichment Datastore (Read-Write)
gsutil iam ch \
  serviceAccount:<SERVICE_ACCOUNT_EMAIL>:roles/storage.objectAdmin \
  gs://<BUCKET_NAME>

GCS Roles Summary

Troubleshooting Common Errors

Detailed Troubleshooting Notes

Authentication Errors

The error Invalid credentials indicates that the service account key is incorrect or malformed.

Common causes:

• Malformed service account key: the JSON key file is corrupted, truncated, or belongs to a different project.
• Service account disabled: the service account has been disabled in the Google Cloud Console.

Permission Errors

The error 403 Forbidden or The caller does not have storage.objects.list access means the credentials are valid but lack the required IAM permissions.

Common causes:

• Missing IAM role: the service account does not have Storage Object Viewer (source) or Storage Object Admin (enrichment) assigned on the target bucket.
• Role assigned at wrong level: the role is assigned at the project level but a bucket-level policy overrides it.
• Uniform bucket-level access: if the bucket uses uniform bucket-level access (recommended), ensure IAM policies are set at the bucket level, not through ACLs.
• Source vs. enrichment mismatch: the service account has Storage Object Viewer but the operation requires write access (enrichment).

Connection Errors

The error 404 Not Found: Bucket not found indicates a configuration issue with the bucket name or URI.

Common causes:

• Bucket does not exist: the bucket name was misspelled or the bucket has been deleted.
• Wrong project: the service account belongs to a different Google Cloud project than the bucket.
• Invalid URI format: the URI must follow gs://<bucket_name>. Extra path segments or incorrect formatting will cause failures.

Add a Source Datastore

A source datastore is a storage location Qualytics connects to so it can profile, scan, and monitor data. Adding Google Cloud Storage as a source lets Qualytics read files directly from your bucket and run quality operations on the data they contain.

Before you start, create the key described in Create a Service Account Key and review the required privileges.

Field reference

The Add Datastore page shows the sections below when Google Cloud Storage is selected. When reusing an existing connection, the Connection Properties and Secrets Management sections come already filled in and read-only: Qualytics has already validated those credentials, so you fill in only the Location and the General fields. To change a saved connection's credentials, edit the connection through the Manage Connections page; edits there apply to every datastore that reuses the connection.

Connection Properties

These fields define where the bucket lives and how Qualytics authenticates to it. They belong to the connection: when reusing an existing connection, they come already filled in and read-only.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.

Location

Pick the folder inside the bucket Qualytics should read from. You fill this in on both flows.

General

Common fields for every source datastore, shown below the Location section. You fill these in on both flows.

Below the form, an information banner lists the Public addresses and Private addresses your datastore connections originate from. Allow the ones that match your network setup through your security groups or firewall rules; see How Connections Work for where the addresses come from.

Steps

There are two ways to set up the connection: reuse a connection you already saved (Existing Connection) or create a new one from scratch (New Connection). The tabs below walk through each option; pick the one you want to follow. Each field is described in the Field reference above.

Use as an Enrichment Datastore

An enrichment datastore is where Qualytics writes its findings: scan results, source record examples, remediation snapshots, materialized copies, and exports. Keep it on a dedicated schema or storage path, separate from the source data you govern. When a source datastore is linked to it, that source's writes land there. The destination can share the source's connection when those credentials have the required write access.

To use a Google Cloud Storage datastore as a destination, the service account behind its connection needs the write permissions listed under Additional Permissions for an Enrichment Datastore. Then link it from the datastore that will write to it: open its Settings menu, click Enrichment , and either pick the Google Cloud Storage datastore under Use existing or create it there under Add new. The same choice is offered in the second step of the Add Datastore page. The flow is the same for every connector and is documented once, in Link an Enrichment Destination and Link Enrichment on Datastore Creation.

The rest of this section covers only what is specific to Google Cloud Storage: the fields you fill in when creating the destination on Google Cloud Storage from inside that dialog. For what Qualytics writes and how the outputs are named, see Enrichment Outputs.

Field reference

The Enrichment Destination step shows the sections below when Google Cloud Storage is selected. When reusing an existing connection, the Connection Properties and Secrets Management sections come already filled in and read-only.

Connection Properties

These fields define where the bucket lives and how Qualytics authenticates to it. They are the same fields as on the source datastore flow, repeated here so this section stands on its own.

Secrets Management

This group is optional: use it only if you want Qualytics to pull credentials from a secrets manager instead of typing them into the form. Turn on HashiCorp Vault to show the fields below. Despite the label, any secrets manager that exposes a compatible REST API works, not only HashiCorp Vault; see Secrets Management. It also belongs to the connection: read-only when reusing an existing connection.

Location

Where Qualytics writes the enrichment files.

General

Write Behavior

Whether a Scan also writes a snapshot of the anomalous records, and whether Qualytics syncs and profiles the destination after each write.

Steps

Fill in these fields under Add new, either in the Enrichment Destination dialog of the datastore that will write here or in the second step of the Add Datastore page, then click Test connection and save. The steps are the same for every connector: see Link an Enrichment Destination or Link Enrichment on Datastore Creation.

API Payload Examples

This section provides detailed examples of API payloads to guide you through the process of creating and managing datastores using Qualytics API. Each example includes endpoint details, sample payloads, and instructions on how to replace placeholder values with actual data relevant to your setup.

Creating a Source Datastore

This section provides sample payloads for creating the Google Cloud Storage datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint: /api/datastores (post)

        {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "trigger_sync": true,
        "root_path": "/gcs_root_path",
        "enrichment_only": false,
        "connection": {
            "name": "your_connection_name",
            "type": "gcs",
            "uri": "gs://<bucket_name>",
            "secret_key": "gcs_service_account_key"
        }
    }

   {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "trigger_sync": true,
        "root_path": "/gcs_root_path",
        "enrichment_only": false,
        "connection_id": 123
    }

# Step 1: Create a Connection
qualytics connections create \
    --type gcs \
    --name "your_connection_name" \
    --uri "gs://<bucket_name>" \
    --secret-key ${GCS_SERVICE_ACCOUNT_KEY}

# Step 2: Create a Source Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --database . \
    --schema /

Create an Enrichment Datastore

This section provides sample payloads for creating an enrichment datastore. Replace the placeholder values with actual data relevant to your setup.

Endpoint: /api/datastores (post)

    {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "trigger_sync": true,
        "root_path": "/gcs_root_path",
        "enrichment_only": true,
        "connection": {
            "name": "your_connection_name",
            "type": "gcs",
            "uri": "gs://<bucket_name>",
            "secret_key": "gcs_service_account_key"
        }
    }

    {
        "name": "your_datastore_name",
        "teams": ["Public"],
        "trigger_sync": true,
        "root_path": "/gcs_root_path",
        "enrichment_only": true,
        "connection_id": 123
    }

# Step 1: Create a Connection
qualytics connections create \
    --type gcs \
    --name "your_connection_name" \
    --uri "gs://<bucket_name>" \
    --secret-key ${GCS_SERVICE_ACCOUNT_KEY}

# Step 2: Create an Enrichment Datastore
qualytics datastores create \
    --name "your_datastore_name" \
    --connection-name "your_connection_name" \
    --database . \
    --schema /your_enrichment_path \
    --enrichment-only

Link an Enrichment Destination

Use the provided endpoint to link a datastore to its enrichment destination:

Endpoint Details: /api/datastores/{datastore-id}/enrichment/{enrichment-id} (patch)