Does filtering with transformKql reduce actual data ingestion cost, or just what's visible in queries?

It reduces actual ingestion, transformKql runs at the DCR level before data is written to the destination (Log Analytics), so a filtered-out event is never ingested and never billed as ingested data. This is meaningfully different from collecting everything and filtering only in downstream KQL queries, which still pays full ingestion cost for the discarded data on every single query run against it.

What happens if a dataFlow references a stream name that doesn't match any configured dataSource?

The DCR is generally rejected at validation/creation, since a dataFlow's streams entries need to correspond to streams actually declared by a dataSources entry (like Microsoft-Perf being declared by the performanceCounters block), a mismatched or typo'd stream name in dataFlows is a common configuration error that surfaces as a validation failure rather than silently collecting nothing.

Can a single DCR route different streams to different destinations?

Yes, as the example shows, dataFlows is an array where each entry independently specifies which streams go to which destinations array, this is exactly how the config routes Microsoft-SecurityEvent only to the Log Analytics destination while Microsoft-Perf goes to both Log Analytics and Azure Monitor Metrics, different data types can have entirely different destination routing within one DCR.

Azure Monitor Data Collection Rule Builder

MonitoringAzure

Build DCR configs for Azure Monitor Agent with performance counters, Windows event logs, KQL transforms, and Log Analytics destinations.

Azure Monitor Configuration

Build DCR configs for Azure Monitor Agent with performance counters, Windows event logs, syslog, KQL transforms, and Log Analytics destinations.

Required Fields

ruleNameresourceGroupdataSourcesdestinationsdataFlows

{
  "ruleName": "dcr-windows-perf-events",
  "resourceGroup": "rg-monitoring",
  "location": "eastus2",
  "kind": "Windows",
  "description": "Collect Windows performance counters and security events from Azure Monitor Agent.",
  "dataSources": {
    "performanceCounters": [
      {
        "name": "ds-perf-cpu-memory",
        "streams": ["Microsoft-Perf"],
        "samplingFrequencyInSeconds": 60,
        "counterSpecifiers": [
          "\\Processor(_Total)\\% Processor Time",
          "\\Memory\\Available MBytes",
          "\\Memory\\% Committed Bytes In Use",
          "\\LogicalDisk(_Total)\\% Free Space",
          "\\LogicalDisk(_Total)\\Disk Reads/sec",
          "\\LogicalDisk(_Total)\\Disk Writes/sec",
          "\\Network Interface(*)\\Bytes Total/sec"
        ]
      }
    ],
    "windowsEventLogs": [
      {
        "name": "ds-security-events",
        "streams": ["Microsoft-SecurityEvent"],
        "xPathQueries": [
          "Security!*[System[(EventID=4624 or EventID=4625 or EventID=4648 or EventID=4672 or EventID=4720 or EventID=4726)]]",
          "Security!*[System[(EventID=4732 or EventID=4756 or EventID=4740 or EventID=4767)]]"
        ]
      },
      {
        "name": "ds-system-events",
        "streams": ["Microsoft-Event"],
        "xPathQueries": [
          "System!*[System[(Level=1 or Level=2 or Level=3)]]",
          "Application!*[System[(Level=1 or Level=2)]]"
        ]
      }
    ],
    "syslog": [],
    "extensions": [
      {
        "name": "ds-change-tracking",
        "extensionName": "ChangeTracking-Windows",
        "streams": ["Microsoft-ConfigurationChange"],
        "extensionSettings": {
          "enableFiles": true,
          "enableSoftware": true,
          "enableRegistry": true,
          "enableServices": true,
          "fileSettings": {
            "fileCollectionFrequency": 3600
          }
        }
      }
    ]
  },
  "destinations": {
    "logAnalytics": [
      {
        "name": "dest-law-security",
        "workspaceResourceId": "/subscriptions/a1b2c3d4/resourceGroups/rg-monitoring/providers/Microsoft.OperationalInsights/workspaces/law-security-prod",
        "workspaceId": "workspace-guid-here"
      }
    ],
    "azureMonitorMetrics": {
      "name": "dest-metrics"
    }
  },
  "dataFlows": [
    {
      "streams": ["Microsoft-Perf"],
      "destinations": ["dest-law-security", "dest-metrics"],
      "transformKql": "source | where ObjectName == 'Processor' or ObjectName == 'Memory'"
    },
    {
      "streams": ["Microsoft-SecurityEvent"],
      "destinations": ["dest-law-security"],
      "transformKql": "source"
    },
    {
      "streams": ["Microsoft-Event"],
      "destinations": ["dest-law-security"],
      "transformKql": "source | where EventLevelName == 'Error' or EventLevelName == 'Critical'"
    }
  ],
  "tags": {
    "environment": "production",
    "managedBy": "monitoring-team"
  }
}

Generated Output

Output will appear here...

What This Tool Does

Build an Azure Monitor Agent Data Collection Rule (DCR) wiring dataSources (performance counters, Windows event logs via xPathQueries, extensions like ChangeTracking) through dataFlows that route each named stream to one or more destinations, optionally applying a transformKql filter or projection before ingestion. dataFlows' transformKql runs at ingestion time, before data lands in Log Analytics, which is a genuinely different cost/filtering point than filtering in a query afterward, a transformKql filtering out noisy, low-value events (like the example's Error/Critical-only filter on Microsoft-Event) reduces actual Log Analytics ingestion volume and cost, whereas filtering the same data only in downstream KQL queries still pays full ingestion cost for data that's discarded on every query anyway.

Technical Details

The builder validates that ruleName, resourceGroup, dataSources, destinations, and dataFlows all resolve before accepting the JSON as a valid Data Collection Rule definition, the fields Azure Monitor needs to know what to collect, where it goes, and how it's transformed en route; it can't verify the transformKql syntax is valid or that referenced workspace resource IDs exist, those are only confirmed against the live Azure Monitor resource provider.

Common Use Cases

1Using xPathQueries to scope Windows Security event collection to specific, security-relevant EventIDs (4624, 4625, 4648, etc.) rather than ingesting the entire noisy Security log, directly controlling both signal quality and ingestion cost.
2Applying a transformKql filter on the Microsoft-Event stream to keep only Error/Critical level events, reducing ingested volume for a stream that would otherwise include large amounts of routine Information-level noise.
3Routing the same performance counter stream (Microsoft-Perf) to both a Log Analytics workspace and Azure Monitor Metrics simultaneously via dataFlows' multi-destination array, so the same collected data serves both long-term log-based analysis and near-real-time metric-based alerting.
4Enabling the ChangeTracking-Windows extension with specific enableFiles/enableSoftware/enableRegistry flags, scoping file-integrity and configuration-change monitoring to exactly the categories relevant to a security or compliance requirement, rather than tracking everything by default.

Common Questions

Does filtering with transformKql reduce actual data ingestion cost, or just what's visible in queries?: It reduces actual ingestion, transformKql runs at the DCR level before data is written to the destination (Log Analytics), so a filtered-out event is never ingested and never billed as ingested data. This is meaningfully different from collecting everything and filtering only in downstream KQL queries, which still pays full ingestion cost for the discarded data on every single query run against it.
What happens if a dataFlow references a stream name that doesn't match any configured dataSource?: The DCR is generally rejected at validation/creation, since a dataFlow's streams entries need to correspond to streams actually declared by a dataSources entry (like Microsoft-Perf being declared by the performanceCounters block), a mismatched or typo'd stream name in dataFlows is a common configuration error that surfaces as a validation failure rather than silently collecting nothing.
Can a single DCR route different streams to different destinations?: Yes, as the example shows, dataFlows is an array where each entry independently specifies which streams go to which destinations array, this is exactly how the config routes Microsoft-SecurityEvent only to the Log Analytics destination while Microsoft-Perf goes to both Log Analytics and Azure Monitor Metrics, different data types can have entirely different destination routing within one DCR.

Expert Tips

TIP

Use transformKql filtering deliberately for high-volume, low-signal streams (routine Information-level events, verbose performance counters you don't actually query), the ingestion cost savings from filtering at the DCR level rather than downstream can be substantial for a high-volume Windows fleet.

TIP

Scope Windows Security event collection via xPathQueries to specific, meaningful EventIDs rather than the entire Security log, both for cost control and to keep the resulting Log Analytics data genuinely useful for security investigation rather than diluted with routine noise.

TIP

Verify every stream referenced in dataFlows actually corresponds to a declared dataSources entry, a mismatch is a common and easy-to-introduce error when a DCR is edited incrementally over time.

Related Learning Guides

Cost Management Guide24 min read
Azure Monitor & Application Insights26 min read

Was this tool helpful?

Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.