Connect to Databricks

​

Prerequisites

• An active Databricks workspace.
• Permissions to create and manage Service Principals within your Databricks account or workspace.
• A Databricks SQL Warehouse set up and running.

​

1. Obtain Databricks Connection Details

1. Host: The URL of your Databricks workspace, which typically looks like https://<region>.cloud.databricks.com/. You can find this in your SQL Warehouse connection details.
2. HTTP Path: The specific path to your SQL Warehouse, usually starting with /sql/1.0/warehouses/. You can find this in the “Connection Details” tab of your SQL Warehouse.
3. Client ID: The Application ID of your Databricks Service Principal.
4. Client Secret: The secret key generated for your Databricks Service Principal.

​

How to Find Your Connection Details

​

Host and HTTP Path

1. Navigate to your Databricks Workspace.
2. Go to **SQL Warehouses **(or SQL Endpoints).
3. Select the warehouse you want to connect to.
4. Click on the Connection Details tab. Here you will find the Server Hostname (your Host) and the HTTP Path.

​

Client ID and Client Secret (using a Databricks-managed Service Principal)

1. Log in to your Databricks environment.
   • A) account-level permissions: Log in to your Databricks account console.
   • B) administrator-level permissions: Log in to your Databricks workspace as an administrator.
2. Navigate to the Service Principals section.
   • A) From Account Console: Navigate to User management > Service principals.
   • B) From Workspace: Navigate to Admin Settings (click your username in the top right, then Admin Settings) > Identity and access > Service principals.
3. Click Add service principal.
4. Select Databricks managed.
5. Provide a descriptive Name for the service principal (e.g., WisdomAI-ServicePrincipal).
6. Click Add service principal.
7. Once created, you will see the Application ID for this service principal. This is your Client ID. Copy this value.

1. From the service principal’s page, go to the Secrets tab.
2. Under OAuth secrets, click Generate secret.
3. Set a lifetime for the secret (up to 730 days) and click Generate.
   Choose an expiration that balances security and operational convenience.
4. Immediately copy the displayed secret. This is your Client Secret.

​

2. Assign Permissions in Databricks

• SQL Warehouse/Cluster Permissions: Grant CAN USE permission to the Service Principal on the SQL Warehouse you are connecting to. You can do this from the Permissions tab of the SQL Warehouse.
• Data Privileges: The service principal requires the following privileges in Unity Catalog (or Hive Metastore):
  • USE CATALOG on the target catalog.
  • USE SCHEMA on the target schema.
  • SELECT on the tables and views you want WisdomAI to access.

-- Grant permission to use a specific catalog
GRANT USAGE ON CATALOG <your_catalog_name> TO `<service_principal_id>`;

-- Grant permission to use a specific schema
GRANT USAGE ON SCHEMA <your_catalog_name>.<your_schema_name> TO `<service_principal_id>`;

-- Grant select permission on table
GRANT SELECT ON TABLE <your_catalog_name>.<your_schema_name>.<your_table_name> TO `<service_principal_id>`;

-- Grant select permission on all tables in a schema
GRANT SELECT ON ALL TABLES IN SCHEMA <your_catalog_name>.<your_schema_name> TO `<service_principal_id>`;

​

3. Connect WisdomAl to Databricks

1. In WisdomAl, navigate to the Connections section and click Add Connection.
2. Select Databricks as the data source type.
3. Fill in the connection details in the “Databricks connection details” section.
   • Connection Name: Choose a descriptive name for your connection (e.g., “Databricks Prod”).
   • Host: Paste the Server Hostname you obtained from Databricks.
   • HTTP Path: Paste the HTTP Path for your SQL Warehouse.
   • Client ID: Enter your Client ID (Application ID). This field is mandatory.
   • Client Secret: Enter your Client Secret (the value copied immediately after creation). This field is mandatory.
   • Catalog Filters (Optional): Specify any Catalog Filters if you want to further restrict the catalogs/schemas WisdomAl crawls.
4. Click Save and Sync metadata. WisdomAl will use the provided credentials to connect to Databricks and scan the metadata of the specified catalogs and tables.

​

Security Considerations and Best Practices

• Least Privilege: Always follow the principle of least privilege. Only grant the necessary permissions to the Service Principal.
• Secure Key Management: Treat your Client ID and Client Secret like passwords. Avoid sharing them via insecure channels, such as unencrypted email. Use secure methods, such as LastPass or other secure file-sharing services.
• Credential Rotation: Regularly rotate your Client Secrets to enhance security, especially before their expiration.

​

Troubleshooting Common Issues

​

Authentication Failed or Invalid Credentials

• Ensure the Host, HTTP Path, Client ID, and Client Secret are complete and correct. No extra characters, spaces, or missing lines.
• Verify that the Client Secret has not expired or been revoked in Databricks.
• Confirm that the Service Principal exists and is enabled in Databricks.

​

Permission Denied Errors

• Double-check that the Service Principal has the necessary CAN USE permission on the SQL Warehouse.
• Verify that the Service Principal has the correct USE CATALOG, USE SCHEMA, and SELECT privileges on the Databricks catalogs, schemas, and tables you are trying to access.
• Confirm that the SQL Warehouse is running and accessible.

​

Catalog Not Found or Table Not Found

• Verify the spelling of the catalog or table names.
• If using catalog filters in WisdomAl, ensure the catalog is included in the filter.
• Confirm that the Service Principal has permissions on the specific catalog you are trying to access.

​

Connection Timeout or Network Errors

• Verify that WisdomAl’s CIDR block (35.238.115.103/32 or 34.82.248.105/32) is allowlisted in your Snowflake network policies or any corporate firewalls.