Skip to content

Architecture

Jump to bottom
Gosia edited this page Mar 7, 2024 · 2 revisions

Registry operator

1. Introduction

1.1 Purpose

This document provides a comprehensive architectural overview of the registry operator, using a number of different architectural views to depict different aspects of the system. It is intended to capture and convey the significant architectural decisions which have been made during registry operator designing. This document will address the background for this project, and the architecturally significant functional requirements.

1.2 Scope

The document describes the whole registry operator architecture. In examples there are shown protocols such as filesystem and s3 as they are planed for the Minimal Viable Product (MVP) or features to be implemented in one of the first releases (for more information see adrs).

1.3 Definitions

Definition of each abbreviation or term that might be not clear to the reader:

  • CNCF - Cloud Native Computing Foundation,
  • Registry - a stateless, highly scalable server side application that stores and distributes container images and other content.
  • Kubernetes operator - a method of packaging, deploying, and managing applications on Kubernetes clusters. They extend the capabilities of Kubernetes by capturing the domain-specific knowledge required to run a particular application or service.
  • CRDs - Custom Resource Definitions - a method of extending the Kubernetes API and defining custom objects specific to an application or service.
  • CSI - Container Storage Interface, handles block and file storage, defines CSI Custom Resources:
    • StorageClass - Represents a class of PersistentVolume with similar characteristics (cluster scoped resource),
    • PersistentVolumeClaim - Represents a request to provision a PersistentVolume (namespace scoped resource),
    • PersistentVolume - Represents a storage volume.
  • COSI - Container Storage Interface, handles objects storage, defines COSI Custom Resources:
    • BucketClaim- Represents a request to provision a Bucket (namespace scoped resource),
    • BucketClass - Represents a class of Buckets with similar characteristics (cluster scoped resource),
    • Bucket - Represents a Bucket or its equivalent in the storage backend (namespace scoped resource),
    • BucketAccessClass - Represents a class of accessors with similar access requirements (cluster scoped resource),
    • BucketAccess - Represents a request to access a Bucket and then a access token or service account in the storage backend (namespace scoped resource).

1.4 References

2. Architectural overview

2.1 Motivation

The Registry configuration is based on a YAML file. Configuration file contains data about the type of storage used in backend. Information about the storage system, on which user would like to install Registry are placed in the COSI/CSI CRs and secrets. From the user perspective the configuration of the new Registry will be possible if CSI/COSI driver is installed and CRs are created. The registry operator takes the configuration with reference to CSI/COSI CRs and creates Registy YAML file. This solution reduces the manual effort required to configure registry, making it easier to operate and maintain complex systems.

2.2 Big Picture

Kubernetes operator handles the configuration of applications, allowing users to define and manage configurations using Kubernetes manifests. This enables a declarative approach to managing application state and configurations. Operators use a reconciliation loop to continuously compare the observed state of the application (based on the Kubernetes cluster's current state) with the desired state specified in the CRs.

2.3 Technical Details

2.3.1 Desired Registry configuration file

CNCF describes Registry configuration file. Automation with registry operator assumes changing below parameters:

  • storage - defines which storage backend is in use. Only one backend can be configure, otherwise the registry returns an error. Types of backend storage drivers:

    • filesystem- uses the local disk to store registry files (for small-scale production applications).
    • azure- uses Microsoft Azure Blob Storage.
    • gcs- uses Google Cloud Storage.
    • s3- uses Amazon Simple Storage Service (S3) and is compatible Storage Services.

    ⚠ NOTE
    The MVP will provide filesystem and s3 implementation.

  • inmemory - uses inmemory storage driver for purely tests purposes. This driver is an implementation of the storagedriver.StorageDriver interface which uses local memory for object storage.

    ⚠ WARNING
    This storage driver does not persist data across runs. This is why it is only suitable for testing. Never use this driver in production.

  • delete - enables the deletion of image blobs and manifests by digest and it's default valuse is set to false.

  • maintanance - enables maintanance functions like uploadpurging and readonly.

  • auth - configures authentication provider, only one is allowed.

config.yaml file with sections to be changed by registry operator:

storage:
  filesystem:                                # MVP (via CSI)                                      
    rootdirectory: /var/lib/registry
    maxthreads: 100
  azure:
  ...
  gcs:
  ...
  s3:
    accesskey: awsaccesskey                  # MVP (via COSI)                         
    secretkey: awssecretkey                  # MVP (via COSI)                            
    region: us-west-1                        # MVP (via COSI)                                  
    regionendpoint: http://myobjects.local   # MVP (via COSI)                     
    forcepathstyle: true
    accelerate: false
    bucket: bucketname                       # MVP (via COSI)                                
    encrypt: true
    keyid: mykeyid
    secure: true
    v4auth: true
    chunksize: 5242880
    multipartcopychunksize: 33554432
    multipartcopymaxconcurrency: 100
    multipartcopythresholdsize: 33554432
    rootdirectory: /s3/object/name/prefix
    usedualstack: false
    loglevel: debug
  inmemory:                                  # MVP
  delete:                                    # MVP                      
    enabled: false
  redirect:
    disable: false
  cache:
    blobdescriptor: redis
    blobdescriptorsize: 10000
  maintenance:                               # MVP                    
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
    readonly:
      enabled: false
auth:
  silly:
    realm: silly-realm
    service: silly-service
  token:
    autoredirect: true
    realm: token-realm
    service: token-service
    issuer: registry-token-issuer
    rootcertbundle: /root/certs/bundle # MVP
  htpasswd:
    realm: basic-realm
    path: /path/to/htpasswd

2.3.2 Operator pattern

The operator design pattern defines how to manage application and infrastructure resources using domain-specific knowledge and declarative state. The goal of the pattern is to reduce the amount of manual imperative work by capturing that domain specific knowledge in code and exposing it using a declarative API.

operator

The Operator pattern consists of three components:

  • The application or infrastructure that we want to manage - Registry YAML configuration file.
  • A domain specific language that enables the user to specify the desired state of the application in a declarative way - CRDs containing data about the storage.
  • A controller that runs continuously:
    • Reads and is aware of the state.
    • Runs actions when operations state changes in an automated way.
    • Report the state of the application in a declarative way.

A single service (also called a controller) has a knowledge how to adjust and maintain the resource. It can read the desired spec and can create and manage the resources that were described.

Operators use Custom Resource Definitions to define custom objects that represent the application or service being managed. These CRDs extend the Kubernetes API, allowing users to interact with and manage their applications using native Kubernetes tools.

controller

Controllers watch for changes to the custom resources (CRs) they manage. When a CR is created, modified, or deleted, the Operator controller reacts by taking appropriate actions to ensure the desired state of the application via reconciliation loop.

2.3.3 Custorm Resource Definition

Definitions of storage Kubernetes resources are loaded to the spec.yaml. Parameters from the spec.yaml are loaded to the Custom Resource.

⚠ NOTE
bucketClaim, bucketAccess and persistentVolumeClaim are optional. If they are not incluede in the spec.yaml file, they will be automatically created respectively based on the bucketClass, bucketAccessClass and storageClass.

spec:
  image: "docker.io/library/registry:2"
  config:
    log:
      level: debug
    storage:
      filesystem:
        persistentVolumeClaim: "<PVC name>"
        storageClass: "<SC name>" 
      objectStorage:
        bucketClaim: "<BucketClaim name>"
        bucketClass: "<BucketClass name>" 
        bucketAccess: "<BucketAccess name>"
        bucketAccessClass: "<BucketAccessClass name>" 
        chunkSize: 5242880
      inmemory: {} # This driver takes no parameters
      delete:
        enabled: true
      redirect:
        disabled: false
      cache:
        blobdescriptor: redis # inmemory
        blobdescriptorsize: 10000
      maintenance:
        uploadpurging:
          enabled: true
          age: 168h
          interval: 24h
          dryrun: false
        readonly:
          enabled: false
      auth:
        htpasswd:
          secret:
            name: "<Secret name>"
            field: "htpasswd"
  http:
    addr: ":5000"
    host: ""
    secret:
      name: "<Secret name>"
      field: "secretForLocalDev"
    relativeURLs: false
    drainTimeout: 60s
    http2:
      disabled: false
    h2c:
      enabled: false
  redis:
    secret:
      name: "<Secret name>"
      field: "redisConnection"
    db: 0
    dialtimeout: 10ms
    readtimeout: 10ms
    writetimeout: 10ms
    pool:
      maxidle: 16
      maxactive: 64
      idletimeout: 300s
    tls:
      enabled: false
  health:
    storagedriver:
      enabled: true
      interval: 10s
      threshold: 3
    tcp:
      - addr: redis-server.domain.com:6379
        timeout: 3s
        interval: 10s
        threshold: 3
  validation:
    manifests:
      urls:
        allow:
          - ^https?://([^/]+\.)*example\.com/
        deny:
          - ^https?://www\.example\.com/

2.3.4 Workflow

Definitions of storage Kubernetes resources are loaded to the spec.yaml, parameters from the spec.yaml are loaded to the custom resource.

register_operator_overview

cr

2.3.5 Use cases

3. Testing plan

[Outline a testing plan for the proposed enhancement, including any specific test cases that should be considered.]

Clone this wiki locally