Is AWS Data Pipeline still a recommended service for new ETL pipelines?

No, AWS has directed customers toward alternatives like Step Functions (for general workflow orchestration), Glue workflows/triggers (for Glue-centric ETL), or Amazon MWAA (managed Apache Airflow) for new pipeline development, Data Pipeline is still operational for existing customers but isn't the service AWS points new pipeline work toward. If you're building something new, evaluate one of these alternatives rather than Data Pipeline.

What's the difference between StringValue and RefValue in a PipelineObjects field?

StringValue is a literal, static value (a table name, a schedule period). RefValue is a reference to another object's Id elsewhere in the PipelineObjects array, used to build the pipeline's dependency graph, an activity's input/output/schedule/runsOn fields typically use RefValue to point at the actual data node, schedule, or resource objects they depend on, rather than duplicating that object's definition inline.

What does failureAndRerunMode CASCADE actually change compared to the alternative?

CASCADE propagates a failed object's failure status to its dependents automatically, so a chain of dependent activities all show as failed (or get canceled) rather than sitting indefinitely in a waiting state for an upstream object that will never successfully complete. The alternative (a non-cascading mode) can leave downstream activities stuck waiting without clearly surfacing that the whole pipeline run has effectively failed.

AWS Data Pipeline Config Builder

MonitoringAWS

Build Data Pipeline definition configurations with schedules, data nodes, activities, and resource specifications.

AWS Data Pipeline Configuration

Build Data Pipeline definition configurations with schedules, data nodes, activities, and resource specifications.

Required Fields

NamePipelineObjects

{
  "PipelineId": "df-0abc123def456789",
  "Name": "daily-s3-to-redshift-pipeline",
  "Description": "Daily ETL pipeline from S3 to Redshift with data validation",
  "PipelineObjects": [
    {
      "Id": "Default",
      "Name": "Default",
      "Fields": [
        {"Key": "type", "StringValue": "Default"},
        {"Key": "scheduleType", "StringValue": "cron"},
        {"Key": "failureAndRerunMode", "StringValue": "CASCADE"},
        {"Key": "role", "StringValue": "DataPipelineDefaultRole"},
        {"Key": "resourceRole", "StringValue": "DataPipelineDefaultResourceRole"},
        {"Key": "pipelineLogUri", "StringValue": "s3://pipeline-logs/daily-etl/"}
      ]
    },
    {
      "Id": "Schedule",
      "Name": "DailySchedule",
      "Fields": [
        {"Key": "type", "StringValue": "Schedule"},
        {"Key": "period", "StringValue": "1 day"},
        {"Key": "startAt", "StringValue": "FIRST_ACTIVATION_DATE_TIME"},
        {"Key": "startDateTime", "StringValue": "2024-01-01T06:00:00"}
      ]
    },
    {
      "Id": "S3Input",
      "Name": "S3InputDataNode",
      "Fields": [
        {"Key": "type", "StringValue": "S3DataNode"},
        {"Key": "directoryPath", "StringValue": "s3://source-data/daily/#{format(@scheduledStartTime,'YYYY-MM-dd')}/"},
        {"Key": "dataFormat", "RefValue": "CsvFormat"}
      ]
    },
    {
      "Id": "RedshiftOutput",
      "Name": "RedshiftDataNode",
      "Fields": [
        {"Key": "type", "StringValue": "RedshiftDataNode"},
        {"Key": "database", "RefValue": "RedshiftDatabase"},
        {"Key": "tableName", "StringValue": "daily_events"},
        {"Key": "primaryKeys", "StringValue": "event_id"}
      ]
    },
    {
      "Id": "CopyActivity",
      "Name": "S3ToRedshiftCopy",
      "Fields": [
        {"Key": "type", "StringValue": "RedshiftCopyActivity"},
        {"Key": "input", "RefValue": "S3Input"},
        {"Key": "output", "RefValue": "RedshiftOutput"},
        {"Key": "insertMode", "StringValue": "OVERWRITE_EXISTING"},
        {"Key": "schedule", "RefValue": "Schedule"},
        {"Key": "runsOn", "RefValue": "Ec2Resource"}
      ]
    }
  ],
  "Tags": [
    {
      "Key": "Pipeline",
      "Value": "daily-etl"
    }
  ]
}

Generated Output

Output will appear here...

About This Tool

Build an AWS Data Pipeline definition, a graph of PipelineObjects (Schedule, data nodes, activities) linked via RefValue references, for AWS's original, now largely legacy ETL orchestration service, which AWS has stopped onboarding new customers to in favor of Step Functions, Glue workflows, or MWAA (managed Airflow) for new pipeline work. failureAndRerunMode: CASCADE means a failed activity's failure propagates to dependent downstream activities (marking them as failed too, rather than leaving them stuck waiting indefinitely), which matters for understanding why a single upstream failure shows as multiple failed activities in the pipeline's status rather than just one.

When to Use This Tool

•Understanding a legacy Data Pipeline definition inherited from an existing production system well enough to safely maintain it, since AWS Data Pipeline is not the recommended service for new pipeline development at this point, this tool is primarily useful for working with existing pipelines, not designing new ones.
•Using #{format(@scheduledStartTime,'YYYY-MM-dd')} expression syntax within a directoryPath field to dynamically construct a date-partitioned S3 path per scheduled run, a Data-Pipeline-specific templating mechanism distinct from more modern tools' parameterization approaches.
•Setting failureAndRerunMode to CASCADE so a failure in an upstream activity (like the S3-to-Redshift copy) correctly propagates failure status to anything depending on it, rather than leaving downstream activities in an ambiguous stuck-waiting state.
•Referencing a prior object by Id via RefValue (like the CopyActivity referencing S3Input and RedshiftOutput as its input/output) to build the pipeline's dependency graph, since RefValue-based linking, not implicit ordering, is what defines the actual execution dependencies.

Pro Tips

TIP

If you're starting new ETL work, don't reach for Data Pipeline, AWS's own guidance points toward Step Functions, Glue workflows, or MWAA for anything new, treat this service as legacy-maintenance territory.

TIP

Trace RefValue references carefully when modifying an existing pipeline definition, the actual execution dependency graph is defined by these references, not by the order objects appear in the PipelineObjects array, a change that looks isolated can actually break a dependency chain elsewhere in the array.

TIP

The Data-Pipeline-specific expression syntax (#{format(...)}, #{myVariable}) doesn't carry over to other AWS orchestration tools, don't assume familiarity with this syntax transfers if you eventually migrate a pipeline to Step Functions or Airflow, those use entirely different templating approaches.

How It Works Under the Hood

The builder validates that Name and PipelineObjects resolve before accepting the JSON as a valid Data Pipeline definition, the minimum fields the (legacy) PutPipelineDefinition API needs; it can't verify that every RefValue in the object graph actually references a valid Id present elsewhere in PipelineObjects, a dangling reference is only caught when the pipeline definition is actually validated against the live service.

Frequently Asked Questions

Is AWS Data Pipeline still a recommended service for new ETL pipelines?: No, AWS has directed customers toward alternatives like Step Functions (for general workflow orchestration), Glue workflows/triggers (for Glue-centric ETL), or Amazon MWAA (managed Apache Airflow) for new pipeline development, Data Pipeline is still operational for existing customers but isn't the service AWS points new pipeline work toward. If you're building something new, evaluate one of these alternatives rather than Data Pipeline.
What's the difference between StringValue and RefValue in a PipelineObjects field?: StringValue is a literal, static value (a table name, a schedule period). RefValue is a reference to another object's Id elsewhere in the PipelineObjects array, used to build the pipeline's dependency graph, an activity's input/output/schedule/runsOn fields typically use RefValue to point at the actual data node, schedule, or resource objects they depend on, rather than duplicating that object's definition inline.
What does failureAndRerunMode CASCADE actually change compared to the alternative?: CASCADE propagates a failed object's failure status to its dependents automatically, so a chain of dependent activities all show as failed (or get canceled) rather than sitting indefinitely in a waiting state for an upstream object that will never successfully complete. The alternative (a non-cascading mode) can leave downstream activities stuck waiting without clearly surfacing that the whole pipeline run has effectively failed.

Related Learning Guides

DynamoDB Data Modeling28 min read
Cost Optimization Strategies26 min read

Was this tool helpful?

Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.

AWS Data Pipeline Config Builder

MonitoringAWS

Build Data Pipeline definition configurations with schedules, data nodes, activities, and resource specifications.

AWS Data Pipeline Configuration

Build Data Pipeline definition configurations with schedules, data nodes, activities, and resource specifications.

Required Fields

NamePipelineObjects

Generated Output

Output will appear here...

About This Tool

When to Use This Tool

•Understanding a legacy Data Pipeline definition inherited from an existing production system well enough to safely maintain it, since AWS Data Pipeline is not the recommended service for new pipeline development at this point, this tool is primarily useful for working with existing pipelines, not designing new ones.
•Using #{format(@scheduledStartTime,'YYYY-MM-dd')} expression syntax within a directoryPath field to dynamically construct a date-partitioned S3 path per scheduled run, a Data-Pipeline-specific templating mechanism distinct from more modern tools' parameterization approaches.
•Setting failureAndRerunMode to CASCADE so a failure in an upstream activity (like the S3-to-Redshift copy) correctly propagates failure status to anything depending on it, rather than leaving downstream activities in an ambiguous stuck-waiting state.
•Referencing a prior object by Id via RefValue (like the CopyActivity referencing S3Input and RedshiftOutput as its input/output) to build the pipeline's dependency graph, since RefValue-based linking, not implicit ordering, is what defines the actual execution dependencies.

Pro Tips

TIP

How It Works Under the Hood

Frequently Asked Questions

Is AWS Data Pipeline still a recommended service for new ETL pipelines?: No, AWS has directed customers toward alternatives like Step Functions (for general workflow orchestration), Glue workflows/triggers (for Glue-centric ETL), or Amazon MWAA (managed Apache Airflow) for new pipeline development, Data Pipeline is still operational for existing customers but isn't the service AWS points new pipeline work toward. If you're building something new, evaluate one of these alternatives rather than Data Pipeline.
What's the difference between StringValue and RefValue in a PipelineObjects field?: StringValue is a literal, static value (a table name, a schedule period). RefValue is a reference to another object's Id elsewhere in the PipelineObjects array, used to build the pipeline's dependency graph, an activity's input/output/schedule/runsOn fields typically use RefValue to point at the actual data node, schedule, or resource objects they depend on, rather than duplicating that object's definition inline.
What does failureAndRerunMode CASCADE actually change compared to the alternative?: CASCADE propagates a failed object's failure status to its dependents automatically, so a chain of dependent activities all show as failed (or get canceled) rather than sitting indefinitely in a waiting state for an upstream object that will never successfully complete. The alternative (a non-cascading mode) can leave downstream activities stuck waiting without clearly surfacing that the whole pipeline run has effectively failed.

Related Learning Guides

DynamoDB Data Modeling28 min read
Cost Optimization Strategies26 min read

Was this tool helpful?