What's the practical difference between autoScale and dynamicExecutorAllocation?

autoScale changes the actual number of underlying VM nodes provisioned for the pool (a slower, infrastructure-level operation, adding/removing nodes takes real time), while dynamicExecutorAllocation adjusts how many Spark executor processes run within whatever nodes are currently provisioned (a faster, application-level operation that doesn't require new infrastructure). A job can scale executors up quickly via dynamicExecutorAllocation using existing node capacity, while autoScale only kicks in when the workload's aggregate demand genuinely requires more or fewer underlying nodes over a longer window.

Does autoPause fully deallocate the pool, or just idle it?

It fully deallocates the pool's compute after the configured delayInMinutes of inactivity, meaning compute cost stops entirely during the paused period, not just a reduced idle-state billing rate. The tradeoff is a cold-start delay (provisioning fresh nodes) the next time a job or notebook session actually needs the pool, similar to the tradeoff seen in other autoscale-to-zero services.

Can a customLibraries whl package be updated without recreating the Spark pool?

Generally yes, updating the referenced package at its data lake path (or bumping the version and updating the path) and having subsequent sessions pick up the new version doesn't require recreating the entire Spark pool resource, though depending on how sessions are configured, already-running sessions may need to be restarted to pick up a library change rather than picking it up live mid-session.

Azure Synapse Spark Pool Builder

MonitoringAzure

Build Synapse Spark pool configs with autoscale, dynamic executor allocation, Spark properties, and library requirements.

Azure Synapse Analytics Configuration

Build Synapse Spark pool configs with autoscale, dynamic executor allocation, Spark properties, library requirements, and session-level packages.

Required Fields

workspaceNamesparkPoolNamenodeSizesparkVersionautoScale

{
  "workspaceName": "syn-contoso-analytics",
  "resourceGroup": "rg-synapse",
  "location": "eastus2",
  "sparkPoolName": "sparkpool01",
  "nodeSize": "Medium",
  "nodeSizeFamily": "MemoryOptimized",
  "autoScale": {
    "enabled": true,
    "minNodeCount": 3,
    "maxNodeCount": 20
  },
  "autoPause": {
    "enabled": true,
    "delayInMinutes": 15
  },
  "sparkVersion": "3.4",
  "nodeCount": 0,
  "dynamicExecutorAllocation": {
    "enabled": true,
    "minExecutors": 1,
    "maxExecutors": 10
  },
  "sparkConfigProperties": {
    "spark.sql.adaptive.enabled": "true",
    "spark.sql.adaptive.coalescePartitions.enabled": "true",
    "spark.sql.parquet.int96RebaseModeInWrite": "CORRECTED",
    "spark.databricks.delta.optimizeWrite.enabled": "true",
    "spark.synapse.logAnalytics.enabled": "true",
    "spark.synapse.logAnalytics.workspaceId": "law-workspace-id"
  },
  "libraryRequirements": {
    "pipRequirements": "great-expectations==0.18.0\npandas==2.2.0\nscikit-learn==1.4.0\nazure-identity==1.15.0",
    "condaEnvironment": null
  },
  "customLibraries": [
    {
      "name": "contoso-ml-utils",
      "type": "whl",
      "path": "abfss://libraries@stdatalakecontoso.dfs.core.windows.net/python/contoso_ml_utils-1.2.0-py3-none-any.whl"
    }
  ],
  "sessionLevelPackagesEnabled": true,
  "cacheSize": 50,
  "isComputeIsolationEnabled": false,
  "sparkEventsFolder": "abfss://synapse@stdatalakecontoso.dfs.core.windows.net/spark-events",
  "tags": {
    "environment": "production",
    "team": "data-engineering",
    "costCenter": "analytics-001"
  }
}

Generated Output

Output will appear here...

About This Tool

Build an Azure Synapse Spark pool with autoScale bounds, autoPause (deallocating the pool entirely after delayInMinutes of inactivity, the primary cost control for an intermittently-used pool), and dynamicExecutorAllocation as a separate, Spark-level scaling mechanism operating within whatever nodes autoScale has currently provisioned. autoScale and dynamicExecutorAllocation solve genuinely different scaling problems and are easy to conflate: autoScale changes the actual number of underlying compute nodes in the pool (a slower, infrastructure-level scaling operation), while dynamicExecutorAllocation adjusts how many Spark executors run within the currently-provisioned nodes (a faster, Spark-application-level scaling operation), a pool can have dynamicExecutorAllocation scaling executors up and down rapidly within a completely stable node count that autoScale hasn't changed at all.

When to Use This Tool

•Setting autoPause with a bounded delayInMinutes (15) so a Spark pool used only for periodic notebook sessions doesn't accrue node-hour cost during the (often much longer) idle periods between actual usage.
•Understanding dynamicExecutorAllocation as a Spark-level scaling mechanism distinct from autoScale, so a job's executor count can adjust quickly within currently-provisioned nodes without needing to wait for the slower node-level autoScale process to add or remove infrastructure.
•Setting nodeSizeFamily to MemoryOptimized specifically for a workload with a genuinely memory-intensive Spark workload (large shuffle operations, big broadcast joins), rather than defaulting to a general-purpose or compute-optimized family that would bottleneck on memory instead.
•Referencing a customLibraries whl package from a data lake path, so a team's shared internal utility code is available to every notebook/job on the pool without needing each individual notebook to install it separately at runtime.

Pro Tips

TIP

Don't conflate autoScale and dynamicExecutorAllocation as the same scaling knob, they operate at genuinely different layers (infrastructure nodes versus Spark executors within those nodes) and tuning one without understanding the other can leave a workload bottlenecked at whichever layer wasn't actually adjusted.

TIP

Set autoPause's delayInMinutes based on actual observed usage gaps, too short a delay causes frequent cold-start penalties for a pool used somewhat regularly throughout the day, too long wastes cost on idle compute between genuinely infrequent usage sessions.

TIP

Match nodeSizeFamily to the workload's actual bottleneck (memory versus compute), a memory-bottlenecked Spark job on a compute-optimized node family hits performance ceilings that adding more nodes doesn't fix, since the constraint is per-node memory, not aggregate compute.

How It Works Under the Hood

The builder validates that workspaceName, sparkPoolName, nodeSize, sparkVersion, and autoScale all resolve before accepting the JSON as a valid Spark pool definition, the fields Synapse needs to provision the pool with its sizing and scaling behavior; it can't verify the specified node size is available in the target region or that library requirements (pip packages, custom whl files) are mutually compatible, those are only confirmed when a session actually starts against the pool.

Frequently Asked Questions

What's the practical difference between autoScale and dynamicExecutorAllocation?: autoScale changes the actual number of underlying VM nodes provisioned for the pool (a slower, infrastructure-level operation, adding/removing nodes takes real time), while dynamicExecutorAllocation adjusts how many Spark executor processes run within whatever nodes are currently provisioned (a faster, application-level operation that doesn't require new infrastructure). A job can scale executors up quickly via dynamicExecutorAllocation using existing node capacity, while autoScale only kicks in when the workload's aggregate demand genuinely requires more or fewer underlying nodes over a longer window.
Does autoPause fully deallocate the pool, or just idle it?: It fully deallocates the pool's compute after the configured delayInMinutes of inactivity, meaning compute cost stops entirely during the paused period, not just a reduced idle-state billing rate. The tradeoff is a cold-start delay (provisioning fresh nodes) the next time a job or notebook session actually needs the pool, similar to the tradeoff seen in other autoscale-to-zero services.
Can a customLibraries whl package be updated without recreating the Spark pool?: Generally yes, updating the referenced package at its data lake path (or bumping the version and updating the path) and having subsequent sessions pick up the new version doesn't require recreating the entire Spark pool resource, though depending on how sessions are configured, already-running sessions may need to be restarted to pick up a library change rather than picking it up live mid-session.

Related Learning Guides

Cost Management Guide24 min read
Azure Monitor & Application Insights26 min read

Was this tool helpful?

Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.