IBM Cloud Object Storage Guide

IBM Cloud Object Storage Overview

IBM Cloud Object Storage (COS) is a highly scalable, durable storage service designed for unstructured data — documents, images, videos, backups, logs, and data lake assets. COS provides S3-compatible APIs, making it easy to migrate workloads from AWS S3 or use existing S3-compatible tools and libraries. With 99.999999999% (11 nines) durability and flexible data residency options, COS is the foundation for data storage on IBM Cloud.

What distinguishes IBM Cloud Object Storage from competitors is its unique resiliency options. COS stores data using Information Dispersal Algorithm (IDA) technology, which slices data into fragments and distributes them across storage nodes. This approach provides durability without the overhead of traditional replication, resulting in lower storage costs. You can choose from three resiliency options: Cross Region (data distributed across 3+ regions), Regional (data distributed across 3 zones in one region), and Single Data Center (data in one facility).

This guide covers COS instance creation, bucket configuration, storage classes, lifecycle policies, encryption, access control, data transfer tools, and integration with other IBM Cloud services.

COS Architecture

Service Instances

A COS service instance is the top-level container for your buckets. You can create multiple service instances to separate billing and access control for different projects or teams. Each instance has its own set of credentials, IAM policies, and usage metrics.

# Create a COS service instance
ibmcloud resource service-instance-create my-cos \
  cloud-object-storage standard global \
  -g prod-resources

# List COS instances
ibmcloud resource service-instances --service-name cloud-object-storage

# Get the CRN of an instance
ibmcloud resource service-instance my-cos --id

Buckets

Buckets are containers for objects. Each bucket has a globally unique name, a storage class, and a location type (cross-region, regional, or single data center). You choose these settings at bucket creation time and they cannot be changed later.

# Create a regional bucket with Smart Tier storage
ibmcloud cos bucket-create \
  --bucket prod-data-2024 \
  --class smart \
  --ibm-service-instance-id <cos-crn> \
  --region us-south

# Create a cross-region bucket for disaster recovery
ibmcloud cos bucket-create \
  --bucket dr-backup-2024 \
  --class cold \
  --ibm-service-instance-id <cos-crn> \
  --region us

# List all buckets
ibmcloud cos buckets --ibm-service-instance-id <cos-crn>

Storage Classes

COS offers four storage classes with different pricing and access characteristics. Choose the storage class based on how frequently you access your data:

• Smart Tier: Automatically moves objects between hot, cool, and cold tiers based on access patterns. No retrieval fees. Best for workloads with unpredictable access patterns. Recommended for most use cases.
• Standard: Lowest latency access with no retrieval fees. Highest storage cost. Best for frequently accessed data like application assets, databases backups, and active datasets.
• Vault: Lower storage cost with a small per-GB retrieval fee. Data must be stored for at least 30 days. Best for compliance archives, infrequently accessed backups, and regulatory data retention.
• Cold Vault: Lowest storage cost with higher retrieval fees. Data must be stored for at least 90 days. Best for long-term archives, disaster recovery copies, and regulatory compliance data that is rarely accessed.

Lifecycle Policies

Lifecycle policies automate data management by transitioning objects between storage classes or expiring (deleting) objects after a specified period. Common use cases include archiving old logs, expiring temporary files, and transitioning infrequently accessed data to cheaper storage classes.

{
  "Rules": [
    {
      "ID": "archive-old-logs",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "logs/"
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "GLACIER"
        }
      ]
    },
    {
      "ID": "expire-temp-files",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "tmp/"
      },
      "Expiration": {
        "Days": 7
      }
    },
    {
      "ID": "cleanup-multipart",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 3
      }
    }
  ]
}

Encryption

All data in COS is encrypted at rest by default using IBM-managed keys (SSE-COS). For regulatory requirements that mandate customer-managed keys, integrate with IBM Key Protect or IBM Hyper Protect Crypto Services (HPCS) for SSE-KP encryption. With customer-managed keys, you control the root key lifecycle including creation, rotation, and revocation.

# Create a bucket with Key Protect encryption
ibmcloud cos bucket-create \
  --bucket encrypted-data \
  --class standard \
  --ibm-service-instance-id <cos-crn> \
  --region us-south \
  --kp-root-key-crn <key-protect-root-key-crn>

Object Lock and Immutability

Object Lock provides write-once-read-many (WORM) protection that prevents objects from being deleted or modified for a specified retention period. This is essential for regulatory compliance in industries like finance and healthcare where data must be preserved unchanged for audit purposes.

Access Control

COS supports multiple access control mechanisms that can be used together:

• IAM Policies: Grant access to service instances and individual buckets using IBM Cloud IAM roles.
• Bucket Policies: S3-compatible bucket policies for fine-grained, resource-level access control.
• HMAC Credentials: S3-compatible access key ID and secret access key for tools that require S3 authentication.
• Pre-signed URLs: Temporary URLs that grant time-limited access to specific objects without requiring IBM Cloud credentials.
• Public Access: Enable public read access for buckets hosting static websites or public datasets.

# Create HMAC credentials for S3-compatible access
ibmcloud resource service-key-create cos-hmac-key Writer \
  --instance-name my-cos \
  --parameters '{"HMAC": true}'

# View the credentials
ibmcloud resource service-key cos-hmac-key

Data Transfer

For large data transfers, COS supports several options beyond the standard S3 API:

• Aspera High-Speed Transfer: Built-in high-speed data transfer technology that can achieve transfer speeds up to 10 Gbps, regardless of network conditions. Aspera uses a patented FASP protocol that overcomes TCP throughput limitations.
• Multipart Upload: Upload large objects (up to 10 TB) by splitting them into parts that can be uploaded in parallel.
• rclone: Open-source tool that syncs files between local storage and COS, supporting incremental transfers.
• IBM Cloud Mass Data Migration: Physical device shipped to your location for petabyte-scale data migration.

Integration with IBM Cloud Services

COS integrates natively with many IBM Cloud services:

• SQL Query: Run SQL queries directly on COS data using serverless SQL Query (now IBM Cloud Data Engine).
• Functions: Trigger Cloud Functions when objects are created, updated, or deleted.
• Event Streams: Stream COS events to Kafka topics for real-time processing.
• Activity Tracker: Audit all COS management and data events.
• Key Protect: Customer-managed encryption keys for data at rest.
• Log Analysis: Archive logs from Log Analysis to COS for long-term retention.
• IKS/ROKS: Use COS as persistent storage for Kubernetes pods via the s3fs CSI driver.

Best Practices

• Use Smart Tier storage class for workloads with variable or unknown access patterns.
• Enable versioning on buckets that store critical data.
• Configure lifecycle policies to transition old data to cheaper storage classes and expire temporary data.
• Use customer-managed encryption with Key Protect or HPCS for regulated workloads.
• Access COS through private endpoints (Virtual Private Endpoints or direct endpoints) to avoid egress charges.
• Enable Activity Tracker data events for security-sensitive buckets.
• Use Object Lock for compliance data that must be immutable.
• Implement bucket policies for fine-grained access control beyond IAM roles.
• Configure abort rules for incomplete multipart uploads to avoid storage waste.
• Use Aspera for large data transfers to maximize throughput.