Why do workers fail to launch when using WORKER_IP_PRIVATE?

Workers without a public IP still need to reach Google APIs (for Dataflow control plane communication, GCS access, etc.), and that path requires Private Google Access enabled on the subnetwork they're launched in. Without it, a private-IP-only worker has no route to those services and job startup fails, usually with a fairly generic worker-startup error rather than one that names Private Google Access directly.

What does enableStreamingEngine actually change about how the job runs?

It moves windowing, state, and timer management off worker VMs and into a Google-managed backend service. Without it, workers hold pipeline state locally, which both caps how aggressively the autoscaler can add/remove workers (since state would need to migrate) and increases per-worker memory pressure for stateful operations. With it enabled, workers become closer to stateless compute, letting maxWorkers actually get exercised under load.

Can I update a running Flex Template job's parameters without stopping it?

Only a limited set of runtime parameters and worker counts support in-place updates via the update API for compatible streaming jobs; most containerSpec or environment-level changes (like the Docker image itself, or the network) require draining or cancelling the existing job and launching a new one, not a live parameter patch.

GCP Dataflow Template Builder

MonitoringGCP

Build Dataflow flex template configurations with container specs, worker settings, and streaming engine options.

Dataflow Flex Templates Configuration

Build Dataflow flex template configurations with container specs, worker settings, networking, and streaming engine options.

Required Fields

jobNameprojectIdregioncontainerSpec.imageenvironment.tempLocation

{
  "jobName": "streaming-events-to-bigquery",
  "projectId": "my-project",
  "region": "us-central1",
  "templateType": "FLEX",
  "containerSpec": {
    "image": "us-central1-docker.pkg.dev/my-project/dataflow/event-pipeline:v2.0",
    "sdkInfo": {
      "language": "JAVA"
    },
    "metadata": {
      "name": "Streaming Events to BigQuery",
      "description": "Reads events from Pub/Sub and writes to BigQuery with windowing"
    }
  },
  "parameters": {
    "inputSubscription": "projects/my-project/subscriptions/events-sub",
    "outputTable": "my-project:analytics.raw_events",
    "windowDuration": "60s",
    "numShards": "5"
  },
  "environment": {
    "numWorkers": 3,
    "maxWorkers": 20,
    "workerMachineType": "n2-standard-4",
    "tempLocation": "gs://my-project-dataflow-temp/staging",
    "stagingLocation": "gs://my-project-dataflow-temp/staging",
    "network": "prod-vpc",
    "subnetwork": "regions/us-central1/subnetworks/dataflow-subnet",
    "serviceAccountEmail": "dataflow-sa@my-project.iam.gserviceaccount.com",
    "enableStreamingEngine": true,
    "additionalExperiments": [
      "enable_streaming_engine",
      "use_runner_v2"
    ],
    "kmsKeyName": "projects/my-project/locations/us-central1/keyRings/dataflow-ring/cryptoKeys/dataflow-key",
    "ipConfiguration": "WORKER_IP_PRIVATE"
  },
  "labels": {
    "team": "data-engineering",
    "pipeline": "event-ingestion"
  }
}

Generated Output

Output will appear here...

About This Tool

Build a Dataflow Flex Template job configuration: a Docker containerSpec image (built and pushed to Artifact Registry ahead of time) plus runtime parameters and environment settings, as opposed to a classic template's precompiled, parameter-substitution-only model. enableStreamingEngine offloads windowing and state management to the Dataflow service rather than worker VMs, which is what makes maxWorkers autoscaling actually effective for stateful streaming pipelines, without it, worker-local state pins scaling far more conservatively. ipConfiguration: WORKER_IP_PRIVATE requires the target subnetwork to have Private Google Access enabled, or workers can't reach the Dataflow control plane and Google APIs at all despite having no public IP to do so through.

Real-World Scenario

A data engineering team migrates a Dataflow pipeline's networking to a private-IP-only subnet as part of a security hardening pass, and the very next deploy fails with every worker stuck in a starting state that never progresses, timing out after the standard startup window. Nothing in the pipeline code changed, only the ipConfiguration setting. They use the builder to review the environment block, realize WORKER_IP_PRIVATE was flipped without checking whether the target subnetwork had Private Google Access, enable it on the subnet, and the exact same job definition launches cleanly on the next attempt.

When to Use This Tool

•Debugging workers that fail to start with connectivity errors after switching ipConfiguration to WORKER_IP_PRIVATE, tracing it to the target subnetwork not having Private Google Access enabled.
•Enabling enableStreamingEngine on a stateful streaming pipeline that was scaling poorly under load, since worker-local state was pinning the autoscaler well below maxWorkers regardless of backlog.
•Setting a tighter numWorkers starting count for a job with a predictable steady-state load, instead of relying entirely on autoscaling to ramp up from a cold-start minimum during the first minutes of a job.
•Attaching kmsKeyName for CMEK on a pipeline processing regulated data, and confirming the key resides in the same region as the Dataflow job, a cross-region key reference fails job submission outright.

Pro Tips

TIP

A stateful streaming pipeline without enableStreamingEngine can look 'stuck' under increasing backlog even with a generous maxWorkers, because state-migration cost caps how fast the autoscaler will actually add workers, don't assume a scaling problem is a quota or maxWorkers issue before checking this flag.

TIP

kmsKeyName location must match the job's region exactly, a common mistake is reusing a multi-region key reference or a key from a different region than the workers, which fails at submission, not at runtime, so at least the failure is fast.

TIP

additionalExperiments flags like use_runner_v2 aren't purely cosmetic, they change execution engine behavior in ways that can affect which Beam features and connectors are available, verify template compatibility with any experiment flags before assuming they're safe to add or remove.

How It Works Under the Hood

The builder requires jobName, projectId, region, containerSpec.image, and environment.tempLocation to resolve before accepting the config, the minimum fields the Dataflow API needs to launch a Flex Template job — an image to run, a project/region to run it in, and scratch storage to stage state — while networking, streaming engine, and encryption settings are validated as well-formed JSON but their runtime implications are yours to reason through.

Frequently Asked Questions

Why do workers fail to launch when using WORKER_IP_PRIVATE?: Workers without a public IP still need to reach Google APIs (for Dataflow control plane communication, GCS access, etc.), and that path requires Private Google Access enabled on the subnetwork they're launched in. Without it, a private-IP-only worker has no route to those services and job startup fails, usually with a fairly generic worker-startup error rather than one that names Private Google Access directly.
What does enableStreamingEngine actually change about how the job runs?: It moves windowing, state, and timer management off worker VMs and into a Google-managed backend service. Without it, workers hold pipeline state locally, which both caps how aggressively the autoscaler can add/remove workers (since state would need to migrate) and increases per-worker memory pressure for stateful operations. With it enabled, workers become closer to stateless compute, letting maxWorkers actually get exercised under load.
Can I update a running Flex Template job's parameters without stopping it?: Only a limited set of runtime parameters and worker counts support in-place updates via the update API for compatible streaming jobs; most containerSpec or environment-level changes (like the Docker image itself, or the network) require draining or cancelling the existing job and launching a new one, not a live parameter patch.

Related Learning Guides

Firestore Data Modeling26 min read
Cost Optimization Guide24 min read

Was this tool helpful?

Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.