Introduction

Container Object Storage Interface (COSI)

Container Object Storage Interface (COSI) is a set of abstractions for provisioning and management of object storage. It aims to be a common layer of abstraction across multiple object storage vendors, such that workloads can request and automatically be provisioned with object storage buckets.

The goals of this project are:

• Automate object storage provisioning, access and management
• Provide a common layer of abstraction for consuming object storage
• Facilitate lift and shift of workloads across object storage providers (i.e. prevent vendor lock-in)

Why another standard?

Kubernetes abstracts file/block storage via the CSI standard. The primitives for file/block storage do not extend well to object storage. Here is the extremely concise and incomplete list of reasons why:

• Unit of provisioned storage - Bucket instead of filesystem mount or blockdevice.
• Access is over the network instead of local POSIX calls.
• No common protocol for consumption across various implementations of object storage.
• Management policies and primitives - for instance, mounting and unmounting do not apply to object storage.

The existing primitives in CSI do not apply to objectstorage. Thus the need for a new standard to automate the management of objectstorage.

Community, discussion, contribution, and support

You can reach the maintainers and interact with community through the following channels:

• #sig-storage-cosi slack channel
• container-object-storage-interface mailing list

Community Meeting

• Thursday at 10:30AM - 11:00AM PT
• SIG Storage Zoom Meeting Room (Password is 77777)
• Meeting Notes

Quick Start

COSI components are out-of-tree and need manual installation.

To install COSI Custom Resource Definitions and COSI Controller, issue the following command:

kubectl create -k 'https://github.com/kubernetes-sigs/container-object-storage-interface//?ref=v0.2.0'

COSI Management Tasks

This section provides details for some of the operations that need to be performed when managing COSI components.

Administrative Tasks

Installing Custom Resources and Controller

Refer to Quickstart Guide for installation instructions.

Installing Driver

Refer to Installing Driver for detailed steps.

Creating BucketClasses and BucketAccessClasses

These resources define storage classes and access policies for object storage.

---
apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketAccessClass
metadata:
  name: example-accessclass
driverName: cosi.example.com
authenticationType: Key
parameters:
  foo: bar
---
apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClass
metadata:
  name: example-class
driverName: cosi.example.com
deletionPolicy: Delete
parameters:
  foo: bar

User Tasks

Creating BucketClaims

A BucketClaim requests a new bucket provisioned by the COSI driver.

apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketClaim
metadata:
  name: example-claim
spec:
  bucketClassName: example-class
  protocols: [ 'S3' ]

Creating BucketAccesses

A BucketAccess grants access to a previously created bucket claim.

apiVersion: objectstorage.k8s.io/v1alpha1
kind: BucketAccess
metadata:
  name: example-access
spec:
  bucketClaimName: example-claim
  protocol: S3
  bucketAccessClassName: example-accessclass
  credentialsSecretName: example-secret

Using the COSI-Provisioned Object Storage Credentials

Applications can access COSI-provisioned object storage credentials using Kubernetes Secrets.

spec:
  template:
    spec:
      containers:
        - volumeMounts:
            - mountPath: /conf
              name: example-secret-vol
      volumes:
        - name: example-secret-vol
          secret:
            secretName: example-secret
            items:
              - key: BucketInfo
                path: BucketInfo

Installing COSI Driver

There is no single and definitive guide for installing a COSI driver. The installation process may vary depending on the specific driver you are using and the platform it is being deployed on.

General Installation Steps

1. Check Compatibility: Ensure the driver is compatible with your Kubernetes version and COSI API version.
2. Refer to Driver Documentation: Each driver may have unique requirements and steps. Refer to the documentation of each driver for specific installation details. Many drivers provide Helm charts or Kubernetes manifests for easier deployment.

Troubleshooting

This document provides troubleshooting steps for common issues encountered when using COSI components including Custom Resource Definitions (CRDs), Custom Resources (CRs), the COSI Controller, and Drivers with Sidecars.

CRD Issues

Symptoms

• CRDs fail to apply to the cluster.
• Resources are not recognized by kubectl.

Possible Causes & Resolution

1. Configuration Misalignment
   • Check: Validate whether the installation followed official documentation steps.
   • Resolve: Reinstall CRDs by strictly following the quick-start guide, step-by-step.

Custom Resource (CR) Issues

Symptoms

• BucketClaim/BucketAccess CRs are not processed.
• Status bucketReady or accessGranted conditions remain false.

Possible Causes & Resolution

1. Misconfigured CR Spec

   • Check: Validate required fields (e.g., bucketName, driverName).
   • Fix: Refer to the CR examples in the driver documentation.

2. Controller Not Responding

   • Check: Verify the controller pods is running (kubectl get pods -n container-object-storage-system).
   • Fix: Inspect controller logs for errors.

Controller Issues

Symptoms

• Controller pod crashes or enters CrashLoopBackOff.
• No events generated for CRs.

Possible Causes & Resolution

1. Missing Permissions

   • Check: Review RBAC roles for the controller service account.
   • Fix: Ensure the controller has permissions to manage CRDs and watch resources.

2. Reconciliation Failures

   • Check: Look for Reconcile errors in controller logs.
   • Fix: Validate driver connectivity or CR configurations (e.g., invalid bucket class parameters).

Driver with Sidecar Issues

Symptoms

• Sidecar fails to communicate with the driver.
• Bucket provisioning times out.

Possible Causes & Resolution

1. Sidecar-Driver Communication Failure

   • Check: Ensure the sidecar and driver share the communication socket (e.g. shared volumeMounts).
   • Fix: Adjust driver manifests and COSI_ENDPOINT for both driver and sidecar.

2. Resource Conflicts

   • Check: Multiple drivers using the same driver name.
   • Fix: Ensure unique driverName values per driver instance.

3. Sidecar Liveness Probe Failures

   • Check: Inspect sidecar logs for health check errors.
   • Fix: Adjust liveness/readiness probe thresholds in the sidecar deployment.

FAQs

• Q: Why is my CRD not recognized after applying?
  A: Ensure the CRD is compatible with your Kubernetes cluster version and the COSI controller is running.

• Q: The driver isn't responding to provisioning requests. What should I check?
  A: Verify driver-sidecar communication.

• Q: Why is my bucket stuck in ready: false state?
  A: Check storage quotas, driver availability, and controller logs for reconciliation errors.

Monitoring

// TODO: add once metrics are in place

Developer guide

The Developer Guide provides an in-depth overview of developing for the Container Object Storage Interface (COSI). This chapter is designed for developers who want to contribute to COSI, build drivers, or integrate object storage solutions with Kubernetes using the COSI specification.

Developing "core" COSI

With “core” COSI we refer to the common set of API and controllers that are required to run any COSI driver.

Before your first contribution, you should follow the Kubernetes Contributor Guide.

To further understand the COSI architecture, please refer to KEP-1979: Object Storage Support.

Developing a COSI Driver

Overview

The starting point for developing a COSI driver is the COSI Driver Sample. This repository provides a foundational implementation that you can build upon.

Implementing the Servers

A COSI driver requires implementing two main servers:

Identity Server

The IdentityServer provides driver metadata and implements the following interface:

type IdentityServer interface {
	DriverGetInfo(context.Context, *cosi.DriverGetInfoRequest) (*cosi.DriverGetInfoResponse, error)
}

Provisioner Server

The ProvisionerServer handles bucket provisioning and access management:

type ProvisionerServer interface {
	DriverCreateBucket(context.Context, *cosi.DriverCreateBucketRequest) (*cosi.DriverCreateBucketResponse, error)
	DriverDeleteBucket(context.Context, *cosi.DriverDeleteBucketRequest) (*cosi.DriverDeleteBucketResponse, error)
	DriverGrantBucketAccess(context.Context, *cosi.DriverGrantBucketAccessRequest) (*cosi.DriverGrantBucketAccessResponse, error)
	DriverRevokeBucketAccess(context.Context, *cosi.DriverRevokeBucketAccessRequest) (*cosi.DriverRevokeBucketAccessResponse, error)
}

Entrypoint

The driver entrypoint initializes logging, parses flags, and starts the gRPC server:

package main

func main() {
	klog.InitFlags(nil)
	flag.Parse()
	if err := run(context.Background()); err != nil {
		klog.ErrorS(err, "Exiting on error")
		os.Exit(1)
	}
}

func run(ctx context.Context) error {
	ctx, stop := signal.NotifyContext(ctx,
		os.Interrupt,
		syscall.SIGINT,
		syscall.SIGTERM,
	)
	defer stop()

	cfg := config.Load() // placeholder

	identityServer := &driver.IdentityServer{Name: "cosi.example.com"}
	provisionerServer := &driver.ProvisionerServer{
		Config: cfg,
	}

	server, err := grpcServer(identityServer, provisionerServer)
	if err != nil {
		return fmt.Errorf("gRPC server creation failed: %w", err)
	}

	cosiEndpoint, ok := os.LookupEnv("COSI_ENDPOINT")
	if !ok {
		cosiEndpoint = "unix:///var/lib/cosi/cosi.sock"
	}

	lis, cleanup, err := listener(ctx, cosiEndpoint)
	if err != nil {
		return fmt.Errorf("failed to create listener for %s: %w", cosiEndpoint, err)
	}
	defer cleanup()

	var wg sync.WaitGroup
	wg.Add(1)
	go shutdown(ctx, &wg, server)

	if err = server.Serve(lis); err != nil {
		return fmt.Errorf("gRPC server failed: %w", err)
	}

	wg.Wait()
	return nil
}

Creating Listener

The listener sets up a gRPC connection for handling requests:

func listener(
	ctx context.Context,
	cosiEndpoint string,
) (net.Listener, func(), error) {
	endpointURL, err := url.Parse(cosiEndpoint)
	if err != nil {
		return nil, nil, fmt.Errorf("unable to parse COSI endpoint: %w", err)
	}

	listenConfig := net.ListenConfig{}

	if endpointURL.Scheme == "unix" {
		_ = os.Remove(endpointURL.Path) // Cleanup stale socket
	}

	listener, err := listenConfig.Listen(ctx, endpointURL.Scheme, endpointURL.Path)
	if err != nil {
		return nil, nil, fmt.Errorf("unable to create listener: %w", err)
	}

	cleanup := func() {
		if endpointURL.Scheme == "unix" {
			if err := os.Remove(endpointURL.Path); err != nil {
				klog.ErrorS(err, "Failed to remove old socket")
			}
		}
	}

	return listener, cleanup, nil
}

Creating gRPC Server

The gRPC server registers both the IdentityServer and ProvisionerServer:

func grpcServer(
	identity cosi.IdentityServer,
	provisioner cosi.ProvisionerServer,
) (*grpc.Server, error) {
	server := grpc.NewServer()

	if identity == nil || provisioner == nil {
		return nil, errors.New("provisioner and identity servers cannot be nil")
	}

	cosi.RegisterIdentityServer(server, identity)
	cosi.RegisterProvisionerServer(server, provisioner)

	return server, nil
}

Graceful Shutdown

To ensure clean shutdown, implement a graceful termination mechanism:

const (
	gracePeriod = 5 * time.Second
)

func shutdown(
	ctx context.Context,
	wg *sync.WaitGroup,
	g *grpc.Server,
) {
	<-ctx.Done()
	defer wg.Done()
	defer klog.Info("Stopped")

	klog.Info("Shutting down")

	dctx, stop := context.WithTimeout(context.Background(), gracePeriod)
	defer stop()

	c := make(chan struct{})

	if g != nil {
		go func() {
			g.GracefulStop()
			c <- struct{}{}
		}()

		for {
			select {
			case <-dctx.Done():
				klog.Info("Forcing shutdown")
				g.Stop()
				return
			case <-c:
				return
			}
		}
	}
}

Developing client applications

No official client libraries

We do not provide official client libraries for interacting with COSI secrets. Instead, we encourage users to build their own clients using standard tools and APIs.

• Different users have different needs, and maintaining an official client library might limit their ability to customize or optimize for specific use cases.
• Providing and maintaining client libraries across multiple languages is a significant effort, requiring continuous updates and support.
• By relying on standard APIs, users can integrate directly with COSI without additional abstraction layers that may introduce unnecessary complexity.

Stability and breaking changes

We follow a strict versioning policy to ensure stability while allowing for necessary improvements.

• Patch Releases (v1alphaX): No breaking changes are introduced between patch versions.
• Version Upgrades (v1alpha1 to v1alpha2): Breaking changes, including format modifications, may occur between these versions.

For more details, refer to the Kubernetes Versioning and Deprecation Policy.

Existing Guides

For guidance on developing clients, refer to our language-specific documentation:

• Go Client Guide

If additional client guides are needed, we welcome contributions from the community.

Developing Client Apps in Go

Configuration Structure

The Config struct is the primary configuration object for the storage package. It encapsulates all necessary settings for interacting with different storage providers. This design ensures that all configuration details are centralized and easily maintainable, allowing your application to switch storage backends with minimal code changes.

The nested Spec struct defines both generic and provider-specific parameters:

• BucketName: Specifies the target storage container or bucket. This value directs where the data will be stored or retrieved.
• AuthenticationType: Indicates the method of authentication (for example, "key"). This ensures that the correct credentials are used when accessing a storage provider.
• Protocols: An array of strings that informs the system which storage protocols (e.g., "s3" or "azure") are supported. The factory uses this to determine the appropriate client to initialize.
• SecretS3 / SecretAzure: These fields hold pointers to the respective secret structures needed for authenticating with S3 or Azure services. Their presence is conditional on the protocols configured.

// import "example.com/pkg/storage"
package storage

type Config struct {
	Spec Spec `json:"spec"`
}

type Spec struct {
	BucketName         string             `json:"bucketName"`
	AuthenticationType string             `json:"authenticationType"`
	Protocols          []string           `json:"protocols"`
	SecretS3           *s3.SecretS3       `json:"secretS3,omitempty"`
	SecretAzure        *azure.SecretAzure `json:"secretAzure,omitempty"`
}

Azure Secret Structure

The SecretAzure struct holds authentication credentials for accessing Azure-based storage services. It is essential when interacting with Azure Blob storage, as it contains a shared access token along with an expiration timestamp. The inclusion of the ExpiryTimestamp allows your application to check token validity.

While current COSI implementation doesn't auto-renew tokens, the ExpiryTimestamp provides hooks for future refresh logic.

// import "example.com/pkg/storage/azure"
package azure

type SecretAzure struct {
	AccessToken     string    `json:"accessToken"`
	ExpiryTimestamp time.Time `json:"expiryTimeStamp"`
}

S3 Secret Structure

The SecretS3 struct holds authentication credentials for accessing S3-compatible storage services. This struct includes the endpoint, region, and access credentials required to securely interact with the S3 service. By isolating these values into a dedicated structure, the design helps maintain clear separation between configuration types, thus enhancing code clarity.

// import "example.com/pkg/storage/s3"
package s3

type SecretS3 struct {
	Endpoint        string `json:"endpoint"`
	Region          string `json:"region"`
	AccessKeyID     string `json:"accessKeyID"`
	AccessSecretKey string `json:"accessSecretKey"`
}

Factory

The factory pattern¹ is used to instantiate the appropriate storage backend based on the provided configuration. We will hide the implementation behind the interface.

The factory function examines the configuration’s Protocols array and validates the AuthenticationType along with the corresponding secret. It then returns a concrete implementation of the Storage interface. This method of instantiation promotes extensibility, making it easier to support additional storage protocols in the future, as the COSI specification evolves.

Here is a minimal interface that supports only basic Delete/Get/Put operations:

type Storage interface {
	Delete(ctx context.Context, key string) error
	Get(ctx context.Context, key string, wr io.Writer) error
	Put(ctx context.Context, key string, data io.Reader, size int64) error
}

Our implementation of factory method can be defined as following:

// import "example.com/pkg/storage"
package storage

import (
	"fmt"
	"slices"
	"strings"

	"example.com/pkg/storage/azure"
	"example.com/pkg/storage/s3"
)

func New(config Config, ssl bool) (Storage, error) {
	if slices.ContainsFunc(config.Spec.Protocols, func(s string) bool { return strings.EqualFold(s, "s3") }) {
		if !strings.EqualFold(config.Spec.AuthenticationType, "key") {
			return nil, fmt.Errorf("invalid authentication type for s3")
		}

		s3secret := config.Spec.SecretS3
		if s3secret == nil {
			return nil, fmt.Errorf("s3 secret missing")
		}

		return s3.New(config.Spec.BucketName, *s3secret, ssl)
	}

	if slices.ContainsFunc(config.Spec.Protocols, func(s string) bool { return strings.EqualFold(s, "azure") }) {
		if !strings.EqualFold(config.Spec.AuthenticationType, "key") {
			return nil, fmt.Errorf("invalid authentication type for azure")
		}

		azureSecret := config.Spec.SecretAzure
		if azureSecret == nil {
			return nil, fmt.Errorf("azure secret missing")
		}

		return azure.New(config.Spec.BucketName, *azureSecret)
	}

	return nil, fmt.Errorf("invalid protocol (%v)", config.Spec.Protocols)
}

Clients

As we alredy defined the factory and uppermost configuration, let's get into the details of the clients, that will implement the Storage interface.

S3

In the implementation of S3 client, we will use MinIO client library, as it's more lightweight than AWS SDK.

// import "example.com/pkg/storage/s3"
package s3

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"time"

	"github.com/minio/minio-go/v7"
	"github.com/minio/minio-go/v7/pkg/credentials"
)

type Client struct {
	s3cli      *minio.Client
	bucketName string
}

func New(bucketName string, s3secret SecretS3, ssl bool) (*Client, error) {
	s3cli, err := minio.New(s3secret.Endpoint, &minio.Options{
		Creds:  credentials.NewStaticV4(s3secret.AccessKeyID, s3secret.AccessSecretKey, ""),
		Region: s3secret.Region,
		Secure: ssl,
	})
	if err != nil {
		return nil, fmt.Errorf("unable to create client: %w", err)
	}

	return &Client{
		s3cli:      s3cli,
		bucketName: bucketName,
	}, nil
}

func (c *Client) Delete(ctx context.Context, key string) error {
	return c.s3cli.RemoveObject(ctx, c.bucketName, key, minio.RemoveObjectOptions{})
}

func (c *Client) Get(ctx context.Context, key string, wr io.Writer) error {
	obj, err := c.s3cli.GetObject(ctx, c.bucketName, key, minio.GetObjectOptions{})
	if err != nil {
		return err
	}
	_, err = io.Copy(wr, obj)
	return err
}

func (c *Client) Put(ctx context.Context, key string, data io.Reader, size int64) error {
	_, err := c.s3cli.PutObject(ctx, c.bucketName, key, data, size, minio.PutObjectOptions{})
	return err
}

Azure Blob

In the implementation of Azure client, we will use Azure SDK client library. Note, that the configuration is done with NoCredentials client, as the Azure secret contains shared access signatures (SAS)².

// import "example.com/pkg/storage/azure"
package azure

import (
	"context"
	"errors"
	"fmt"
	"io"
	"time"

	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
)

type Client struct {
	azCli         *azblob.Client
	containerName string
}

func New(containerName string, azureSecret SecretAzure) (*Client, error) {
	azCli, err := azblob.NewClientWithNoCredential(azureSecret.AccessToken, nil)
	if err != nil {
		return nil, fmt.Errorf("unable to create client: %w", err)
	}

	return &Client{
		azCli:         azCli,
		containerName: containerName,
	}, nil
}

func (c *Client) Delete(ctx context.Context, blobName string) error {
	_, err := c.azCli.DeleteBlob(ctx, c.containerName, blobName, nil)
	return err
}

func (c *Client) Get(ctx context.Context, blobName string, wr io.Writer) error {
	stream, err := c.azCli.DownloadStream(ctx, c.containerName, blobName, nil)
	if err != nil {
		return fmt.Errorf("unable to get download stream: %w", err)
	}
	_, err = io.Copy(wr, stream.Body)
	return err
}

func (c *Client) Put(ctx context.Context, blobName string, data io.Reader, size int64) error {
	_, err := c.azCli.UploadStream(ctx, c.containerName, blobName, data, nil)
	return err
}

Summing up

Once all components are in place, using the storage package in your application becomes straightforward. The process starts with reading a JSON configuration file, which is then decoded into the Config struct. The factory method selects and initializes the appropriate storage client based on the configuration, enabling seamless integration with either S3 or Azure storage.

import (
	"encoding/json"
	"os"

	"example.com/pkg/storage"
)

func example() {
	f, err := os.Open("/opt/cosi/BucketInfo")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	var cfg storage.Config
	if err := json.NewDecoder(f).Decode(&cfg); err != nil {
		panic(err)
	}

	client, err := storage.New(cfg, true)
	if err != nil {
		panic(err)
	}

	// use client Put/Get/Delete
	// ...
}

Drivers

• Platform will take you to platform documentation;
• COSI Driver Name will take you to driver repository.

Deprecated drivers

Deprecated drivers are no longer maintained or recommended for use with COSI; users should migrate to supported alternatives to ensure compatibility and security.

API Reference

Packages

• objectstorage.k8s.io/v1alpha1

objectstorage.k8s.io/v1alpha1

AuthenticationType

Underlying type: string

Appears in:

• BucketAccessClass

Bucket

Appears in:

• BucketList

BucketAccess

Appears in:

• BucketAccessList

BucketAccessClass

Appears in:

• BucketAccessClassList

BucketAccessSpec

Appears in:

• BucketAccess

BucketAccessStatus

Appears in:

• BucketAccess

BucketClaim

Appears in:

• BucketClaimList

BucketClaimSpec

Appears in:

• BucketClaim

BucketClaimStatus

Appears in:

• BucketClaim

BucketClass

Appears in:

• BucketClassList

BucketSpec

Appears in:

• Bucket

BucketStatus

Appears in:

• Bucket

DeletionPolicy

Underlying type: string

Appears in:

• BucketClass
• BucketSpec

Protocol

Underlying type: string

Appears in:

• BucketAccessSpec
• BucketClaimSpec
• BucketSpec