GCP Dataproc Serverless Batch Builder

MonitoringGCP

Build Dataproc Serverless Spark batch configurations with runtime settings and dynamic allocation.

Dataproc Serverless Configuration

Build Dataproc Serverless Spark batch configurations with runtime settings, dynamic allocation, networking, and metastore integration.

Required Fields

nameruntimeConfig.versionenvironmentConfig.executionConfig.serviceAccount

{
  "name": "projects/my-project/locations/us-central1/batches/etl-daily-20260314",
  "sparkBatch": {
    "mainJarFileUri": "gs://my-project-jars/etl-pipeline-2.0.jar",
    "mainClass": "com.example.etl.DailyPipeline",
    "args": [
      "--input=gs://my-project-data/raw/2026-03-14/",
      "--output=gs://my-project-data/processed/2026-03-14/",
      "--date=2026-03-14"
    ],
    "jarFileUris": [
      "gs://my-project-jars/connectors/spark-bigquery-latest.jar"
    ],
    "fileUris": [
      "gs://my-project-config/etl-config.properties"
    ],
    "archiveUris": []
  },
  "pysparkBatch": null,
  "runtimeConfig": {
    "version": "2.2",
    "containerImage": "us-central1-docker.pkg.dev/my-project/spark/custom-spark:v2.0",
    "properties": {
      "spark.executor.memory": "8g",
      "spark.executor.cores": "4",
      "spark.driver.memory": "4g",
      "spark.sql.adaptive.enabled": "true",
      "spark.dynamicAllocation.enabled": "true",
      "spark.dynamicAllocation.minExecutors": "2",
      "spark.dynamicAllocation.maxExecutors": "20"
    },
    "cohort": ""
  },
  "environmentConfig": {
    "executionConfig": {
      "serviceAccount": "dataproc-sa@my-project.iam.gserviceaccount.com",
      "networkUri": "projects/my-project/global/networks/prod-vpc",
      "subnetworkUri": "projects/my-project/regions/us-central1/subnetworks/dataproc-subnet",
      "kmsKey": "projects/my-project/locations/us-central1/keyRings/dataproc-ring/cryptoKeys/dataproc-key",
      "ttl": "14400s",
      "stagingBucket": "my-project-dataproc-staging"
    },
    "peripheralsConfig": {
      "metastoreService": "projects/my-project/locations/us-central1/services/hive-metastore",
      "sparkHistoryServerConfig": {
        "dataprocCluster": "projects/my-project/regions/us-central1/clusters/history-cluster"
      }
    }
  },
  "labels": {
    "pipeline": "daily-etl",
    "team": "data-engineering"
  }
}

Generated Output

Output will appear here...

How It Helps

Build Dataproc Serverless Spark batch configurations with runtime settings and dynamic allocation. This tool helps GCP engineers generate valid configurations quickly without consulting documentation, reducing errors and accelerating infrastructure deployment. All processing runs in your browser with no data sent to external servers.

Things Engineers Ask

Can the output of this Dataproc Serverless Batch builder be checked into source control safely?

Yes. The output is plain text containing only configuration — no Google Cloud credentials, project IDs (unless you typed them), or secrets are included. Treat it like any other infrastructure-as-code artifact: review in pull requests, gate on validation, apply via your change process.

Can I use this Dataproc Serverless Batch builder for Google Cloud sovereign or assured workloads?

Most Dataproc Serverless Batch primitives behave the same in standard, Assured Workloads, and sovereign Google Cloud deployments, but available services, regions, and access controls differ. The output is portable in shape; you must verify service availability and any Assured Workloads constraints before applying in a controlled environment.

Practical Applications

1Producing a Dataproc Serverless Batch starting point for a new service that follows the Google Cloud reference architecture rather than copy-paste from another team's repo.
2Translating a Google Cloud architect's whiteboard sketch into a concrete Dataproc Serverless Batch spec engineers can apply.
3Pre-flighting a Dataproc Serverless Batch change against a change-management ticket so the implementation matches what was approved.
4Working out a Dataproc Serverless Batch configuration during incident response, when the on-call needs a corrected version applied within minutes.

Related Learning Guides

Firestore Data Modeling26 min read
Cost Optimization Guide24 min read

Was this tool helpful?

Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.