Build Cloud Composer environment configurations with Airflow settings, workload sizing, and private networking.
Build Cloud Composer environment configurations with Airflow settings, workload sizing, private networking, and CMEK encryption.
Required Fields
nameconfig.softwareConfig.imageVersionconfig.nodeConfig.networkconfig.environmentSizeOutput will appear here...Build a Cloud Composer environment spec covering the Airflow image version, workload sizing (scheduler/webserver/worker/triggerer CPU, memory, and worker autoscaling bounds), private networking, and CMEK encryption. Composer 3's workloadsConfig replaces Composer 2's node-pool-based sizing with per-component resource requests, so worker.minCount/maxCount control horizontal autoscaling directly rather than through a separate GKE node pool config. The private environment's cloudSqlIpv4CidrBlock and webServerIpv4CidrBlock must not overlap the environment's own VPC ranges or the GKE cluster's pod/service secondary ranges underneath it, a collision here fails environment creation with an opaque network error rather than a clear CIDR conflict message.
Workload sizing (CPU/memory/worker count bounds) can be updated in place without recreating the environment. environmentSize itself (SMALL/MEDIUM/LARGE) can also be updated, but it changes the underlying Cloud SQL tier backing the Airflow metadata database, so expect a maintenance-style operation with brief control-plane unavailability, not an instant resize.
Airflow version bumps between Composer image versions can deprecate or remove operators, change default argument behavior, or update provider package versions bundled in pypiPackages. Pin imageVersion explicitly rather than relying on environment auto-upgrade, and test DAGs against a staging environment on the target image before rolling the upgrade to production.
Environment creation fails, but the error surfaces as a general network provisioning failure rather than pointing at the specific overlapping range. Reserve cloudSqlIpv4CidrBlock and webServerIpv4CidrBlock from a part of your address space you know is disjoint from the environment's primary VPC range and the ipAllocationPolicy secondary ranges before submitting, rather than debugging the collision after a failed create.
A data engineering team's morning DAG backlog balloons every day around 6am when the overnight batch jobs all fire at once, with tasks sitting in queued state for 8-10 minutes before workers pick them up, even though the environment eventually catches up by 7am. Reviewing the config shows worker.minCount is still the Composer default of 1, meaning autoscaling has to react from a cold start every single morning instead of already having headroom. They use the builder to raise minCount to 3 based on the observed steady-state concurrent task count, and the next morning's backlog clears within two minutes instead of ten.
The builder requires name, config.softwareConfig.imageVersion, config.nodeConfig.network, and config.environmentSize to resolve before accepting the JSON, the fields the Composer API needs to place the environment's GKE cluster into a network and size its Airflow metadata database, everything else (worker bounds, private config, CMEK) is validated structurally but left to your judgment.
worker.minCount below 2 means a single worker restart (during a patch or a crash) can leave zero workers picking up tasks for a window, most production environments set minCount to at least 2 for that reason alone, independent of throughput needs.
pypiPackages version pins using >= without an upper bound can silently pull in a new major version during environment recreation or an image rebuild, pin exact or constrained ranges for anything DAG-critical.
kmsKeyName for CMEK must be in the same region as the environment; a cross-region key reference fails at creation time, and once an environment is created without CMEK, encryption can't be added retroactively, it's a create-time-only setting.
Was this tool helpful?
Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.