Skip to main content
Data Engineering·How-To

CI/CD for Databricks

How to deploy notebooks safely using Azure DevOps and Databricks Repos.

Uma Kiran Ramadugu
Uma Kiran Ramadugu·Sep 3, 2026·7 min read

Setting up CI/CD for Databricks using Azure DevOps ensures that your data pipelines are version controlled and can be rolled back if something goes wrong. Here is the standard flow.

Step 1: Databricks Repos

Databricks has a feature called Databricks Repos (or Git Folders). This allows you to link your Databricks Workspace directly to your Azure DevOps Git repository.

  1. Create a Git repository in Azure DevOps (e.g., data-platform-notebooks).
  2. In your Databricks Dev Workspace, go to the Workspace menu, select Repos, and click Add Repo.
  3. Paste the URL of your Azure DevOps repo and provide a PAT (Personal Access Token) generated from Azure DevOps.
  4. You can now edit notebooks and commit them to branches directly from the Databricks UI.

Step 2: The Build Pipeline (CI)

When you open a Pull Request (PR) to merge your branch into main, Azure DevOps should trigger a Build Pipeline.

In the context of Databricks notebooks, the Build pipeline is usually quite simple. It simply zips the notebooks or copies them into an “Artifact” so that the release pipeline can pick them up. If you are writing Python modules instead of notebooks, this is where you would run pytest to run unit tests on your code before allowing the PR to merge.

Step 3: The Release Pipeline (CD)

Once the code is merged into main, the Release pipeline is triggered. This pipeline pushes the code to your Prod Workspace.

The most robust way to do this today is using the Databricks CLI or the Databricks Asset Bundles (DABs).

  1. In Azure DevOps, create a Release Pipeline triggered by the Build artifact.
  2. Add an Azure CLI or Bash task.
  3. Install the Databricks CLI on the DevOps build agent.
  4. Authenticate to the Prod Workspace using a Service Principal (do not use a personal user token for prod deployments).
  5. Run a command like databricks workspace import_dir to sync the notebooks from the artifact into a locked production folder in the workspace.
Chat with us