Does Firehose wait for BufferingHints.SizeInMBs to be reached before ever delivering data?

No, it delivers on whichever threshold, SizeInMBs or IntervalInSeconds, is reached first, not requiring both. On a lower-throughput stream, the time-based threshold typically triggers first, resulting in many smaller files well under the configured size hint, which is a common and often surprising cause of small-file proliferation in the destination even with a seemingly generous SizeInMBs setting.

What happens to a record that fails DataFormatConversionConfiguration's schema-based conversion?

It's typically routed to the ErrorOutputPrefix location in S3 rather than being dropped silently or blocking the rest of the stream's delivery, letting you inspect and reprocess malformed or schema-mismatched records separately. Monitor this error prefix, a source producing records that don't match the expected Glue schema (a new field, a type change) accumulates there without necessarily surfacing as an obvious alarm otherwise.

Does a Lambda processor in ProcessingConfiguration add meaningful latency to delivery?

It adds the Lambda invocation's own processing time plus its own BufferSizeInMBs/BufferIntervalInSeconds buffering (configured separately within the processor's Parameters), layered on top of the destination's own BufferingHints. For a stream with tight end-to-end latency requirements, both buffering layers (processor and destination) need to be tuned together, not just the destination's BufferingHints alone.

AWS Kinesis Firehose Delivery Builder

MonitoringAWS

Build Kinesis Firehose delivery stream configurations with S3/Redshift destinations, Parquet conversion, and dynamic partitioning.

Amazon Data Firehose Configuration

Build Kinesis Firehose delivery stream configurations with S3/Redshift destinations, Parquet conversion, and dynamic partitioning.

Required Fields

DeliveryStreamNameDeliveryStreamTypeExtendedS3DestinationConfiguration.BucketARN

{
  "DeliveryStreamName": "clickstream-to-s3",
  "DeliveryStreamType": "KinesisStreamAsSource",
  "KinesisStreamSourceConfiguration": {
    "KinesisStreamARN": "arn:aws:kinesis:us-east-1:123456789012:stream/clickstream",
    "RoleARN": "arn:aws:iam::123456789012:role/firehose-kinesis-role"
  },
  "ExtendedS3DestinationConfiguration": {
    "BucketARN": "arn:aws:s3:::clickstream-data-lake",
    "Prefix": "data/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/",
    "ErrorOutputPrefix": "errors/year=!{timestamp:yyyy}/month=!{timestamp:MM}/!{firehose:error-output-type}/",
    "RoleARN": "arn:aws:iam::123456789012:role/firehose-s3-role",
    "BufferingHints": {
      "SizeInMBs": 128,
      "IntervalInSeconds": 300
    },
    "CompressionFormat": "UNCOMPRESSED",
    "EncryptionConfiguration": {
      "KMSEncryptionConfig": {
        "AWSKMSKeyARN": "arn:aws:kms:us-east-1:123456789012:key/abcd1234"
      }
    },
    "DataFormatConversionConfiguration": {
      "Enabled": true,
      "SchemaConfiguration": {
        "RoleARN": "arn:aws:iam::123456789012:role/firehose-glue-role",
        "DatabaseName": "clickstream_db",
        "TableName": "events",
        "Region": "us-east-1"
      },
      "InputFormatConfiguration": {
        "Deserializer": {
          "OpenXJsonSerDe": {}
        }
      },
      "OutputFormatConfiguration": {
        "Serializer": {
          "ParquetSerDe": {
            "Compression": "SNAPPY"
          }
        }
      }
    },
    "DynamicPartitioningConfiguration": {
      "Enabled": true,
      "RetryOptions": {
        "DurationInSeconds": 300
      }
    },
    "ProcessingConfiguration": {
      "Enabled": true,
      "Processors": [
        {
          "Type": "Lambda",
          "Parameters": [
            {
              "ParameterName": "LambdaArn",
              "ParameterValue": "arn:aws:lambda:us-east-1:123456789012:function:clickstream-transform"
            },
            {
              "ParameterName": "BufferSizeInMBs",
              "ParameterValue": "3"
            },
            {
              "ParameterName": "BufferIntervalInSeconds",
              "ParameterValue": "60"
            }
          ]
        }
      ]
    },
    "CloudWatchLoggingOptions": {
      "Enabled": true,
      "LogGroupName": "/aws/firehose/clickstream",
      "LogStreamName": "S3Delivery"
    }
  },
  "Tags": [
    {
      "Key": "Pipeline",
      "Value": "clickstream"
    }
  ]
}

Generated Output

Output will appear here...

What This Tool Does

Build a Kinesis Firehose (Amazon Data Firehose) delivery stream with a Kinesis Data Stream source, DataFormatConversionConfiguration transforming incoming JSON into Parquet via a Glue Data Catalog schema reference, DynamicPartitioningConfiguration for content-based S3 prefixing, and an optional Lambda ProcessingConfiguration for custom record transformation before delivery. BufferingHints controls delivery latency versus file-size efficiency as a real tradeoff, Firehose delivers whichever threshold (SizeInMBs or IntervalInSeconds) is hit first, so a 128MB/300s config on a lower-throughput stream might deliver every 300 seconds on the time threshold long before accumulating 128MB, producing many small files, which hurts downstream query efficiency in exactly the way DataFormatConversionConfiguration's Parquet output was meant to avoid.

Technical Details

The builder validates that DeliveryStreamName, DeliveryStreamType, and ExtendedS3DestinationConfiguration.BucketARN all resolve before accepting the JSON as a valid CreateDeliveryStream request, the fields Firehose needs to know the stream's identity, source type, and destination; it can't verify the referenced Glue schema, KMS key, or Lambda processor ARNs are correctly configured and mutually compatible, those are only confirmed once real records start flowing through the stream.

Common Use Cases

1Enabling DataFormatConversionConfiguration with a Glue Data Catalog schema reference so incoming JSON records are converted to Parquet before landing in S3, directly improving downstream Athena/Redshift Spectrum query cost and speed versus storing raw JSON.
2Enabling DynamicPartitioningConfiguration so S3 output prefixes can be derived from record content (not just delivery timestamp), letting the S3 layout reflect a meaningful business dimension (like customer ID or event type) rather than only time-based partitioning.
3Adding a Lambda ProcessingConfiguration to enrich or filter records before delivery (adding a derived field, dropping malformed records) when the transformation logic is too complex for Firehose's built-in format conversion alone.
4Tuning BufferingHints.SizeInMBs upward for a high-throughput stream specifically to produce fewer, larger Parquet files, since many small files degrade downstream query engine performance more than the added buffering latency typically costs.

Common Questions

Does Firehose wait for BufferingHints.SizeInMBs to be reached before ever delivering data?: No, it delivers on whichever threshold, SizeInMBs or IntervalInSeconds, is reached first, not requiring both. On a lower-throughput stream, the time-based threshold typically triggers first, resulting in many smaller files well under the configured size hint, which is a common and often surprising cause of small-file proliferation in the destination even with a seemingly generous SizeInMBs setting.
What happens to a record that fails DataFormatConversionConfiguration's schema-based conversion?: It's typically routed to the ErrorOutputPrefix location in S3 rather than being dropped silently or blocking the rest of the stream's delivery, letting you inspect and reprocess malformed or schema-mismatched records separately. Monitor this error prefix, a source producing records that don't match the expected Glue schema (a new field, a type change) accumulates there without necessarily surfacing as an obvious alarm otherwise.
Does a Lambda processor in ProcessingConfiguration add meaningful latency to delivery?: It adds the Lambda invocation's own processing time plus its own BufferSizeInMBs/BufferIntervalInSeconds buffering (configured separately within the processor's Parameters), layered on top of the destination's own BufferingHints. For a stream with tight end-to-end latency requirements, both buffering layers (processor and destination) need to be tuned together, not just the destination's BufferingHints alone.

Expert Tips

TIP

Small-file proliferation from a time-threshold-dominated buffering pattern is a common, quiet cost and performance problem for downstream Athena/Redshift Spectrum queries, monitor actual delivered file sizes, not just the configured BufferingHints values, to catch this.

TIP

Always monitor the ErrorOutputPrefix location for a stream using DataFormatConversionConfiguration, schema-mismatched records land there without necessarily triggering an obvious alert, and can silently accumulate if the source starts producing malformed data.

TIP

When chaining a Lambda processor with a format-converting destination, tune both the processor's own buffering parameters and the destination's BufferingHints together, treating them as one combined latency/file-size budget rather than optimizing each in isolation.

Related Learning Guides

DynamoDB Data Modeling28 min read
Cost Optimization Strategies26 min read

Was this tool helpful?

Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.

AWS Kinesis Firehose Delivery Builder

MonitoringAWS

Build Kinesis Firehose delivery stream configurations with S3/Redshift destinations, Parquet conversion, and dynamic partitioning.

Amazon Data Firehose Configuration

Build Kinesis Firehose delivery stream configurations with S3/Redshift destinations, Parquet conversion, and dynamic partitioning.

Required Fields

DeliveryStreamNameDeliveryStreamTypeExtendedS3DestinationConfiguration.BucketARN

Generated Output

Output will appear here...

What This Tool Does

Technical Details

Common Use Cases

1Enabling DataFormatConversionConfiguration with a Glue Data Catalog schema reference so incoming JSON records are converted to Parquet before landing in S3, directly improving downstream Athena/Redshift Spectrum query cost and speed versus storing raw JSON.
2Enabling DynamicPartitioningConfiguration so S3 output prefixes can be derived from record content (not just delivery timestamp), letting the S3 layout reflect a meaningful business dimension (like customer ID or event type) rather than only time-based partitioning.
3Adding a Lambda ProcessingConfiguration to enrich or filter records before delivery (adding a derived field, dropping malformed records) when the transformation logic is too complex for Firehose's built-in format conversion alone.
4Tuning BufferingHints.SizeInMBs upward for a high-throughput stream specifically to produce fewer, larger Parquet files, since many small files degrade downstream query engine performance more than the added buffering latency typically costs.

Common Questions

Does Firehose wait for BufferingHints.SizeInMBs to be reached before ever delivering data?: No, it delivers on whichever threshold, SizeInMBs or IntervalInSeconds, is reached first, not requiring both. On a lower-throughput stream, the time-based threshold typically triggers first, resulting in many smaller files well under the configured size hint, which is a common and often surprising cause of small-file proliferation in the destination even with a seemingly generous SizeInMBs setting.
What happens to a record that fails DataFormatConversionConfiguration's schema-based conversion?: It's typically routed to the ErrorOutputPrefix location in S3 rather than being dropped silently or blocking the rest of the stream's delivery, letting you inspect and reprocess malformed or schema-mismatched records separately. Monitor this error prefix, a source producing records that don't match the expected Glue schema (a new field, a type change) accumulates there without necessarily surfacing as an obvious alarm otherwise.
Does a Lambda processor in ProcessingConfiguration add meaningful latency to delivery?: It adds the Lambda invocation's own processing time plus its own BufferSizeInMBs/BufferIntervalInSeconds buffering (configured separately within the processor's Parameters), layered on top of the destination's own BufferingHints. For a stream with tight end-to-end latency requirements, both buffering layers (processor and destination) need to be tuned together, not just the destination's BufferingHints alone.

Expert Tips

TIP

Related Learning Guides

DynamoDB Data Modeling28 min read
Cost Optimization Strategies26 min read

Was this tool helpful?