How does OCI Streaming compare to self-managed Apache Kafka?

OCI Streaming is Kafka-compatible at the API level, so existing Kafka clients (producers and consumers) work with minimal configuration changes. The key differences are: Streaming is fully managed (no brokers to patch or scale), automatically handles replication and failover, charges per PUT request and data retention rather than per-broker-hour, and integrates natively with OCI services via Service Connector Hub. Self-managed Kafka offers more tuning control, custom plugins, and Kafka Connect ecosystem but requires significant operational overhead.

How do I choose the right number of partitions?

Partitions determine parallelism — each partition can be consumed by one consumer in a consumer group. Choose partitions based on your target throughput and consumer count. A single partition supports approximately 1 MB/s write and 2 MB/s read. If you need 10 MB/s write throughput, provision at least 10 partitions. Also consider the number of consumers — you need at least as many partitions as consumers in a group. Partitions can be increased but never decreased, so start conservatively.

OCI Streaming Config Builder

ServerlessOCI

Build OCI Streaming (Kafka-compatible) stream pool and stream configurations.

OCI Streaming Configuration

Build OCI Streaming (Kafka-compatible) stream pool, stream, and connector configurations.

Required Fields

compartmentIdstreamPoolstreams

{
  "compartmentId": "ocid1.compartment.oc1..aaaaaaaexample",
  "streamPool": {
    "displayName": "prod-stream-pool",
    "compartmentId": "ocid1.compartment.oc1..aaaaaaaexample",
    "kafkaSettings": {
      "bootstrapServers": "cell-1.streaming.us-ashburn-1.oci.oraclecloud.com:9092",
      "autoCreateTopicsEnabled": false,
      "numPartitions": 1,
      "logRetentionHours": 24
    },
    "customEncryptionKey": {
      "kmsKeyId": "ocid1.key.oc1.iad.aaaaaaaexample"
    },
    "privateEndpointSettings": {
      "subnetId": "ocid1.subnet.oc1.iad.aaaaaaaexample",
      "privateEndpointIp": "10.0.1.50",
      "nsgIds": ["ocid1.networksecuritygroup.oc1.iad.aaaaaaaexample"]
    }
  },
  "streams": [
    {
      "displayName": "order-events",
      "partitions": 8,
      "retentionInHours": 72,
      "streamPoolId": "ocid1.streampool.oc1.iad.aaaaaaaexample"
    },
    {
      "displayName": "user-activity",
      "partitions": 4,
      "retentionInHours": 24,
      "streamPoolId": "ocid1.streampool.oc1.iad.aaaaaaaexample"
    },
    {
      "displayName": "audit-log-stream",
      "partitions": 2,
      "retentionInHours": 168,
      "streamPoolId": "ocid1.streampool.oc1.iad.aaaaaaaexample"
    }
  ],
  "connectHarness": {
    "displayName": "s3-compatible-connector",
    "sourceType": "STREAMING",
    "targetType": "OBJECT_STORAGE",
    "targetBucket": "stream-archive",
    "batchSize": 100,
    "batchTimeInMs": 60000
  },
  "freeformTags": {
    "platform": "event-driven",
    "team": "data-engineering"
  }
}

Generated Output

Output will appear here...

About This Tool

OCI Streaming is a managed Apache Kafka-compatible event streaming platform for ingesting and processing high-volume, real-time data streams. Streams organize messages into partitions for parallel consumption, with configurable retention periods and consumer group semantics. This builder helps you configure streams with partition counts, retention settings, consumer group definitions, and integration patterns for connecting to OCI Functions, Service Connector Hub, and custom consumers.

When to Use This Tool

•Configuring a stream with optimal partition count and retention period for an IoT telemetry pipeline ingesting 100,000 events per second
•Building a stream configuration with multiple consumer groups for fan-out processing — one for real-time analytics and another for long-term archival
•Setting up stream integration with Service Connector Hub to automatically forward events to Object Storage, Autonomous Database, or OCI Functions
•Creating a Kafka-compatible stream configuration for migrating existing Kafka producers and consumers to OCI with minimal code changes

Frequently Asked Questions

How does OCI Streaming compare to self-managed Apache Kafka?: OCI Streaming is Kafka-compatible at the API level, so existing Kafka clients (producers and consumers) work with minimal configuration changes. The key differences are: Streaming is fully managed (no brokers to patch or scale), automatically handles replication and failover, charges per PUT request and data retention rather than per-broker-hour, and integrates natively with OCI services via Service Connector Hub. Self-managed Kafka offers more tuning control, custom plugins, and Kafka Connect ecosystem but requires significant operational overhead.
How do I choose the right number of partitions?: Partitions determine parallelism — each partition can be consumed by one consumer in a consumer group. Choose partitions based on your target throughput and consumer count. A single partition supports approximately 1 MB/s write and 2 MB/s read. If you need 10 MB/s write throughput, provision at least 10 partitions. Also consider the number of consumers — you need at least as many partitions as consumers in a group. Partitions can be increased but never decreased, so start conservatively.

Related Learning Guides

OCI Streaming & Events22 min read

Disclaimer: This tool runs entirely in your browser. No data is sent to our servers. Always verify outputs before using them in production. AWS, Azure, and GCP are trademarks of their respective owners.