Why does my sink show as successfully created but nothing ever lands in the destination?

The sink's auto-generated writerIdentity service account needs an explicit IAM grant on the destination, this doesn't happen automatically. For BigQuery, grant roles/bigquery.dataEditor on the dataset; for Cloud Storage, roles/storage.objectCreator on the bucket; for Pub/Sub, roles/pubsub.publisher on the topic. Sink creation itself doesn't validate or require this grant exists.

Do exclusion filters reduce Cloud Logging's own ingestion cost, or only the sink's export cost?

Only the sink's export volume and downstream storage cost. Log entries are still ingested and billed at the Cloud Logging layer regardless of sink exclusions, those filters only control what gets copied out to BigQuery/Storage/Pub/Sub through this specific sink. To reduce ingestion-level cost, you need an exclusion at the log router / _Default sink level instead.

What's the difference between includeChildren and a project-level sink?

includeChildren only applies to sinks created at the organization or folder level, and controls whether the sink also captures matching logs from child projects/folders, not just the resource it's directly attached to. A project-level sink has no children to include, it only ever sees that project's own logs.

GCP Log Sink Builder

MonitoringGCP

Build Cloud Logging sink configurations for exporting logs to BigQuery, Cloud Storage, or Pub/Sub.

Cloud Logging Sinks Configuration

Build Cloud Logging sink configurations for exporting logs to BigQuery, Cloud Storage, or Pub/Sub with inclusion and exclusion filters.

Required Fields

namedestinationfilter

{
  "name": "security-audit-sink",
  "description": "Export security audit logs to BigQuery for compliance analysis",
  "destination": "bigquery.googleapis.com/projects/my-project/datasets/security_audit_logs",
  "filter": "logName=\"projects/my-project/logs/cloudaudit.googleapis.com%2Factivity\" OR logName=\"projects/my-project/logs/cloudaudit.googleapis.com%2Fdata_access\" severity>=WARNING",
  "disabled": false,
  "exclusions": [
    {
      "name": "exclude-health-checks",
      "description": "Exclude health check log entries",
      "filter": "protoPayload.methodName=\"compute.healthChecks.get\"",
      "disabled": false
    },
    {
      "name": "exclude-read-only",
      "description": "Exclude read-only data access logs",
      "filter": "protoPayload.methodName=~\"*.get\" OR protoPayload.methodName=~\"*.list\"",
      "disabled": true
    }
  ],
  "bigqueryOptions": {
    "usePartitionedTables": true,
    "usesTimestampColumnPartitioning": true
  },
  "includeChildren": true,
  "writerIdentity": "serviceAccount:p123456789-abcdef@gcp-sa-logging.iam.gserviceaccount.com"
}

Generated Output

Output will appear here...

See It in Action

A compliance team sets up a new BigQuery sink for security audit logs ahead of an upcoming SOC 2 audit, confirms the sink shows ACTIVE in the console, and moves on. Three weeks later, pulling a report for the auditor, the destination dataset has zero rows for the entire period, because the sink's writerIdentity was never granted BigQuery Data Editor on the dataset, an easy step to miss since sink creation doesn't fail or warn about it. They use the builder to double check the sink's filter and destination are correctly formed, grant the missing IAM role to the writerIdentity service account, and new log entries start landing in BigQuery within minutes, though the three-week gap for historical data has to be backfilled separately from raw log exports.

What This Tool Does

Build a Cloud Logging sink that routes matching log entries to BigQuery, Cloud Storage, or Pub/Sub, with inclusion and exclusion filters. The writerIdentity service account Cloud Logging generates for the sink must be separately granted write access on the destination (BigQuery Data Editor, Storage Object Creator, or Pub/Sub Publisher depending on destination type), sink creation succeeds regardless of whether that grant exists, so a sink can sit silently producing zero exported rows if the IAM step is skipped. Exclusion filters are evaluated before the sink's own destination cost accrues, so a well-tuned exclusion (dropping health-check or read-only access logs) is a direct lever on both noise and BigQuery/Storage spend.

Technical Details

The builder requires name, destination, and filter to resolve before accepting the JSON as a valid LogSink, the minimum fields the Logging API needs to know what to route, where, and under what match condition; it can't validate that the destination's IAM policy actually grants the sink's writerIdentity write access, that check only happens against the live resource.

Common Use Cases

1Debugging a sink that shows as ACTIVE with zero rows landing in the destination BigQuery dataset, tracing it to the sink's writerIdentity never having been granted BigQuery Data Editor on that dataset.
2Adding an exclusion filter for compute.healthChecks.get calls so routine load-balancer health-check noise doesn't inflate the exported audit log volume and BigQuery storage cost.
3Setting includeChildren true on an organization-level aggregated sink so security audit logs from every project in the org land in one centralized BigQuery dataset for compliance review.
4Scoping a sink's filter to just severity>=WARNING for cost control, after discovering a default all-severity sink was exporting orders of magnitude more INFO-level noise than anyone was actually querying.

Common Questions

Why does my sink show as successfully created but nothing ever lands in the destination?: The sink's auto-generated writerIdentity service account needs an explicit IAM grant on the destination, this doesn't happen automatically. For BigQuery, grant roles/bigquery.dataEditor on the dataset; for Cloud Storage, roles/storage.objectCreator on the bucket; for Pub/Sub, roles/pubsub.publisher on the topic. Sink creation itself doesn't validate or require this grant exists.
Do exclusion filters reduce Cloud Logging's own ingestion cost, or only the sink's export cost?: Only the sink's export volume and downstream storage cost. Log entries are still ingested and billed at the Cloud Logging layer regardless of sink exclusions, those filters only control what gets copied out to BigQuery/Storage/Pub/Sub through this specific sink. To reduce ingestion-level cost, you need an exclusion at the log router / _Default sink level instead.
What's the difference between includeChildren and a project-level sink?: includeChildren only applies to sinks created at the organization or folder level, and controls whether the sink also captures matching logs from child projects/folders, not just the resource it's directly attached to. A project-level sink has no children to include, it only ever sees that project's own logs.

Expert Tips

TIP

The IAM grant on the destination is a separate step from sink creation and is the single most common reason a sink 'doesn't work' despite showing healthy status, always verify writerIdentity's permissions immediately after creating a sink, not after the first missing-data report.

TIP

bigqueryOptions.usePartitionedTables should almost always be true for anything beyond a short-lived sink, an unpartitioned destination table for a high-volume log sink becomes expensive to query and slow to prune well before anyone notices the storage bill creeping up.

TIP

Disabled exclusions (disabled: true) left in a filter list are easy to forget about during a review, since they read like active filters at a glance, verify the disabled flag specifically when auditing what a sink is actually excluding versus what someone intended to exclude.

Related Learning Guides

Firestore Data Modeling26 min read
Cost Optimization Guide24 min read

Was this tool helpful?

Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.