Skip to main content
Data Engineering·How-To

How to Connect ADF to Databricks

Securely orchestrate Databricks notebooks from Azure Data Factory using Key Vault and PAT tokens.

Uma Kiran Ramadugu
Uma Kiran Ramadugu·Sep 3, 2026·5 min read

In a modern Azure Data Platform, Azure Data Factory (ADF) handles the orchestration (the schedule and dependencies), while Databricks handles the heavy lifting (the Spark compute).

To make them work together, ADF needs permission to spin up clusters and trigger notebooks inside your Databricks workspace. Here is the secure, production-ready way to connect them.

Step 1: Generate a Databricks Personal Access Token (PAT)

ADF needs a token to authenticate with Databricks.

  1. Log into your Azure Databricks workspace.
  2. Click your profile icon in the top right corner and select User Settings.
  3. Under the Developer tab, click Manage next to Access Tokens.
  4. Click Generate New Token. Name it “ADF-Orchestration” and give it a long lifetime (or leave it blank for no expiration, though this is less secure).
  5. Copy the token immediately. You will not be able to see it again.

Step 2: Store the Token in Azure Key Vault

Never paste access tokens directly into ADF linked services. Always use Azure Key Vault.

  1. Navigate to your Azure Key Vault in the Azure Portal.
  2. Go to Secrets and click Generate/Import.
  3. Name the secret something like databricks-pat-token and paste the token from Step 1 into the Value field.
  4. Ensure your ADF has an Access Policy (or RBAC assignment) allowing it to “Get” secrets from this Key Vault.

Step 3: Create the Linked Service in ADF

Now we tell ADF how to talk to Databricks using the secret.

  1. Open Azure Data Factory Studio and go to the Manage hub (the toolbox icon).
  2. Under Connections, click Linked services → New.
  3. Search for “Azure Databricks” and select it.
  4. Select your Azure subscription and the Databricks workspace.
  5. Under Authentication type, select Access Token.
  6. Instead of pasting the token, click the Azure Key Vault radio button. Select your Key Vault linked service and type the secret name (databricks-pat-token).
  7. For the cluster, select New job cluster. (Do not use an existing interactive cluster for production pipelines; job clusters are cheaper and spin up cleanly for the job).
  8. Select the Spark version and node types. Click Test Connection to verify it works, then click Create.

Step 4: Trigger a Notebook

The connection is established. You can now use it in a pipeline.

  1. Go to the Author hub in ADF and create a new pipeline.
  2. Drag a Notebook activity from the Databricks section onto the canvas.
  3. In the activity settings, go to the Azure Databricks tab and select the Linked Service you just created.
  4. Go to the Settings tab and browse your Databricks workspace to select the notebook you want to run.
  5. Publish your pipeline and click Trigger Now!
Chat with us