AWS EMR Serverless Job Builder

What This Tool Does

Build an EMR Serverless Spark job run referencing a pre-created Application, with SparkSubmitParameters tuning executor/driver sizing and dynamic allocation bounds directly in the spark-submit argument string rather than through separate structured fields, plus ConfigurationOverrides for logging and Glue Data Catalog integration via the AWSGlueDataCatalogHiveClientFactory Hive metastore client. ExecutionTimeoutMinutes is a hard kill switch, a job still legitimately running past this timeout gets forcibly terminated regardless of progress, so it needs to be sized with real margin above the job's expected worst-case runtime, not just its typical runtime, or an occasional larger-than-usual data volume causes a false job failure rather than a slower-but-successful completion.

Technical Details

Common Use Cases

1. 1Setting spark.dynamicAllocation.minExecutors/maxExecutors bounds within SparkSubmitParameters so the job can scale executor count with actual workload within a controlled range, rather than running with a fixed executor count that's either wasteful for a light run or insufficient for a heavy one.
2. 2Setting spark.hadoop.hive.metastore.client.factory.class to the AWS Glue Data Catalog Hive client factory, so the Spark job reads/writes tables registered in the Glue Data Catalog directly, using it as the Hive metastore rather than requiring a separately-managed Hive metastore.
3. 3Enabling ManagedPersistenceMonitoringConfiguration so Spark event logs and job history persist beyond the job's own ephemeral runtime, letting you review the Spark UI for a completed job's performance characteristics after the fact.
4. 4Sizing ExecutionTimeoutMinutes with real margin above the job's typical runtime (not just matching it), so a larger-than-usual data volume on a given day doesn't get killed mid-run as a false failure when it would have otherwise completed successfully just a bit slower.

Common Questions

What happens if a job is still running when ExecutionTimeoutMinutes is reached?

It's forcibly terminated, regardless of how close it was to completing, this is a hard kill, not a warning or a graceful wind-down. Set this value with meaningful margin above the job's expected worst-case (not typical) runtime, since an occasional larger data volume or a slower Spark stage shouldn't turn into a hard failure for a job that would have completed successfully given a bit more time.

Does the job driver's resource request in SparkSubmitParameters need to fit within the application's own configured capacity?

Yes, the EMR Serverless Application itself has its own configured maximum capacity (separate from the individual job run's SparkSubmitParameters), a job requesting more executor/driver resources than the application's overall capacity allows will fail to acquire sufficient resources. Check the target Application's configured capacity limits, not just the job's own resource request, when a job unexpectedly fails to launch fully-sized executors.

Is spark.dynamicAllocation.enabled required for EMR Serverless jobs?

It's not strictly required, but it's the mechanism that lets EMR Serverless's own elastic compute scaling actually take effect at the Spark executor level, without it, the job runs with a fixed executor count for its entire duration regardless of actual workload, missing much of the cost/performance benefit that makes EMR Serverless attractive over a fixed-size provisioned cluster in the first place.

Expert Tips

Related Learning Guides

• DynamoDB Data Modeling28 min read
• Cost Optimization Strategies26 min read