Azure Databricks Cluster Config Builder

MonitoringAzure

Build Databricks cluster configs with autoscaling, Photon runtime, Spark tuning, instance pools, and cluster policies.

Azure Databricks Configuration

Build Databricks cluster configs with autoscaling, Photon runtime, Spark tuning, instance pools, init scripts, libraries, and cluster policies.

Required Fields

workspaceNameresourceGroupclusterConfig.clusterNameclusterConfig.sparkVersionclusterConfig.nodeTypeId

{
  "workspaceName": "dbw-contoso-analytics",
  "resourceGroup": "rg-databricks",
  "location": "eastus2",
  "pricingTier": "Premium",
  "clusterConfig": {
    "clusterName": "analytics-interactive",
    "sparkVersion": "14.3.x-scala2.12",
    "nodeTypeId": "Standard_DS4_v2",
    "driverNodeTypeId": "Standard_DS5_v2",
    "numWorkers": 0,
    "autoscale": {
      "minWorkers": 2,
      "maxWorkers": 16
    },
    "autoTerminationMinutes": 30,
    "enableElasticDisk": true,
    "sparkConf": {
      "spark.databricks.delta.optimizeWrite.enabled": "true",
      "spark.databricks.delta.autoCompact.enabled": "true",
      "spark.databricks.delta.merge.enableLowShuffle": "true",
      "spark.sql.adaptive.enabled": "true",
      "spark.sql.shuffle.partitions": "auto",
      "spark.databricks.io.cache.enabled": "true"
    },
    "sparkEnvVars": {
      "PYSPARK_PYTHON": "/databricks/python3/bin/python3",
      "ENVIRONMENT": "production"
    },
    "customTags": {
      "team": "data-engineering",
      "costCenter": "DE-001",
      "project": "analytics-platform"
    },
    "clusterLogConf": {
      "dbfs": {
        "destination": "dbfs:/cluster-logs/analytics-interactive"
      }
    },
    "initScripts": [
      {
        "workspace": {
          "destination": "/Shared/init-scripts/install-libraries.sh"
        }
      }
    ],
    "singleUserName": null,
    "dataSecurityMode": "USER_ISOLATION",
    "runtimeEngine": "PHOTON"
  },
  "libraries": [
    {
      "pypi": {
        "package": "great-expectations==0.18.0"
      }
    },
    {
      "pypi": {
        "package": "delta-spark==3.1.0"
      }
    },
    {
      "maven": {
        "coordinates": "com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.22"
      }
    }
  ],
  "instancePool": {
    "instancePoolName": "pool-analytics",
    "nodeTypeId": "Standard_DS4_v2",
    "minIdleInstances": 2,
    "maxCapacity": 20,
    "idleInstanceAutoTerminationMinutes": 60,
    "preloadedSparkVersions": ["14.3.x-scala2.12"],
    "azureAttributes": {
      "availability": "SPOT_WITH_FALLBACK_AZURE",
      "spotBidMaxPrice": -1
    }
  },
  "clusterPolicies": {
    "policyName": "analytics-team-policy",
    "maxWorkersLimit": 20,
    "autoterminationMinutesRange": { "min": 10, "max": 120, "default": 30 },
    "allowedNodeTypes": ["Standard_DS4_v2", "Standard_DS5_v2", "Standard_E8s_v5"]
  }
}

Generated Output

Output will appear here...

Overview

Build Databricks cluster configs with autoscaling, Photon runtime, Spark tuning, instance pools, and cluster policies. This tool helps Azure engineers generate valid configurations quickly without consulting documentation, reducing errors and accelerating infrastructure deployment. All processing runs in your browser with no data sent to external servers.

How Engineers Use This

•Producing a Databricks Cluster starting point for a new service that follows the Azure reference architecture rather than copy-paste from another team's repo.
•Sketching a Databricks Cluster migration from one Azure subscription to another with subscription and resource-group differences called out.
•Pre-flighting a Databricks Cluster change against a change-management ticket so the implementation matches what was approved.
•Comparing two candidate Databricks Cluster layouts side by side to pick the one with the cleaner permission boundary.

Questions & Answers

Can the output of this Databricks Cluster config builder be checked into source control safely?

Yes. The output is plain text containing only configuration — no Azure credentials, subscription IDs (unless you typed them), or secrets are included. Treat it like any other infrastructure-as-code artifact: review in pull requests, gate on validation, apply via your change process.

Will the output of this Databricks Cluster config builder pass `az deployment validate` or `terraform validate`?

It produces structurally valid output for the Azure schemas it supports. We still recommend running provider validation locally before applying — schemas evolve and a recently-released property may not yet be reflected. When validation does fail, the error message points at the exact attribute the schema rejected.

Related Learning Guides

Cost Management Guide24 min read
Azure Monitor & Application Insights26 min read

Was this tool helpful?

Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.