A collection of examples for Strimzi/Kafka, Spark, and Volcano

Goals

Kafka

• Install Strimzi

• Deploy cluster

• Produce / consume messages

• Adjust number of brokers

• Benchmark perfomace

• Deploy Kafka UI

• Topic compression

• Backup / restore

• OIDC

• Rack awareness

• Monitoring

• Cruise Control

• Quotas

• Retention policies

• Scale up/down

• Mirror maker

• autoscaling brokers

• Karapace deployment

• Karapace security

Minio

• single node single disk deploy to k8s
• bucket notifications into kafka

VAP emulator

• Object input
• Object output
• Auto scaling
• ffmpeg process
• Get args from kafka

S3 <--> POSIX

Smart guy

Smart guys video

S3 == eventual consistency

S3Proxy

S3Proxy - access a file system using the S3 API

Native Object Access

S3FS

s3fs

Goofys

Goofys

Faster but less POSIX-y

rclone

rclone

like rsync for cloud, but offers a fuse mount option

Weka S3

based on MinIO - allows s3 access to SMB/NFS/WekaFS data

CunoFS

Paid solution

Block S3 access

s3ql

treats object store like block device. Lacks native object access.

JuiceFS

treats like block storage.