Skip to content

Custom JDBC Drivers Troubleshooting

  • Self-hosted

This page lists the errors you can hit when registering and running a custom JDBC driver. Each section names the message you see, the underlying cause, and the resolution. Registration problems surface in the Dataplane startup log; use your log search to find the exact text of the message you saw.

YAML validation errors at startup

Refusing to publish BuiltInDriverRegistry: <N> custom driver YAML parse/validation error(s):
  - <file>: <key path>: <reason>

Cause: One or more definitions failed the strict schema: an unknown key, a missing required key, a duplicate key, a file name that does not match config.prefix, or a URL placeholder that no connection field satisfies. Registration is all-or-nothing, so the Dataplane refuses to start instead of dropping the broken driver.

Resolution: Fix every error listed under the message against the Requirements contract, rebuild the image, and redeploy. The Dataplane does not fall back to a partial driver list, so a clean start means every definition registered.

Prefix conflicts at startup

The exact wording varies with the conflict:

Refusing to publish BuiltInDriverRegistry: <N> custom driver(s) attempt to override built-in prefixes:
  - custom JAR ships META-INF/jdbc-drivers/<name>.yaml; built-in '<name>' cannot be overridden

Refusing to publish BuiltInDriverRegistry: <N> custom driver(s) collide with built-in prefixes:
  - custom driver '<prefix>' collides with a built-in prefix and cannot override it

Refusing to publish BuiltInDriverRegistry: <N> duplicate custom driver prefix(es):
  - custom prefix '<prefix>' is defined by more than one custom YAML resource

Cause: A custom definition uses a prefix (or ships a YAML file named after one) that belongs to a built-in connector, or two custom JARs define the same prefix. Built-in connectors cannot be overridden, and every custom prefix must be unique.

Resolution: Rename the custom driver's prefix, remembering that the YAML file name must match it, or remove the duplicate JAR from the image. Rebuild and redeploy.

Dialect class fails to register

Custom JDBC dialect for driver prefix '<prefix>' (class '<className>') could not be registered: <reason>

Cause: The definition declares a dialectClass that the Dataplane could not load and instantiate. The reason at the end of the message names the specific failure, such as class not found on classpath, a class that does not extend the required parent, or expected a Scala object (MODULE$ field) or a public no-arg constructor; found neither.

Resolution: Check the class against the dialectClass contract: the class must ship in a JAR on the same classpath, extend org.apache.spark.sql.jdbc.JdbcDialect, carry a trailing $ in the declared class name when it is a Scala object, and expose a public no-argument constructor when it is a Java class. Fix, rebuild, and redeploy.

The connector is missing from the picker

No error appears anywhere: the Dataplane starts cleanly, the startup log does not mention the driver, and the add-datastore picker simply does not list it.

Cause: The definition was never discovered. The usual reasons:

  • The JAR does not record the META-INF/jdbc-drivers/ directory entry. Some zip tools and build plugins write only the file entries, and such a JAR is silently invisible: the YAML is on the classpath, yet the driver never registers.
  • The YAML does not sit directly under META-INF/jdbc-drivers/.
  • The JAR is not on the Dataplane classpath.
  • The Dataplane was not restarted after the JAR was added. Discovery runs once, at startup.

Resolution: Run jar tf my-driver.jar and confirm a META-INF/jdbc-drivers/ line ending in / and the <prefix>.yaml entry under it are both present. Confirm the JAR landed in /opt/spark/jars/ (or your classpath location) in the running image, then restart the Dataplane.

Datastores of a custom type stopped connecting after an upgrade

No JDBC driver definition found for prefix '<prefix>'. Registered drivers: <list>

Cause: The upgrade rebuilt the Dataplane image without the driver JAR. The connector disappears from the picker, and operations against existing datastores of that type fail with the message above, because the registry no longer knows the prefix.

Resolution: Add the driver JAR back into the rebuilt image and restart the Dataplane. The datastores recover as soon as the driver registers again; nothing about them needs to be recreated.

The connection test fails on a working database

The message varies: it wraps whatever error the database returned for the probe statement.

Cause: The connection test runs the definition's config.connectionTest statement, or SELECT 1 when none is set. Some databases cannot run a bare SELECT 1 (for example, ones that require a FROM clause), so the probe fails even though the credentials and URL are right.

Resolution: Set config.connectionTest to a statement the database accepts (for example, SELECT 1 FROM DUAL), rebuild, and restart.