icecheng/freeleaps-ops
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
291388fa0e
freeleaps-ops/cluster/manifests/freeleaps-data-platform/kafka
History
Nicolas e7ec6a4258 Installed Strimzi Kafka Operator version 0.45.0 
Three Kafka nodes + Three ZooKeeper nodes
Can tolerate 1 node failure
3 replicas distributed across different nodes
Use Azure Disk SSD SCRAM-SHA-512 + ACLs
2025-08-20 17:44:55 +08:00
..
README.md Installed Strimzi Kafka Operator version 0.45.0 2025-08-20 17:44:55 +08:00
storage-class.yaml Installed Strimzi Kafka Operator version 0.45.0 2025-08-20 17:44:55 +08:00
strimzi-kafka-cluster.yaml Installed Strimzi Kafka Operator version 0.45.0 2025-08-20 17:44:55 +08:00
values.yaml feat(kafka, pinot, star-rocks): update configurations and resource limits across multiple components 2025-06-26 23:04:03 +08:00
vpa.yaml feat(kafka, pinot, star-rocks): update configurations and resource limits across multiple components 2025-06-26 23:04:03 +08:00

README.md

Kafka Configuration for Freeleaps Data Platform

This directory contains the Kafka configuration for the Freeleaps Data Platform.

Files

  • values.yaml - Original Bitnami Kafka Helm values (legacy)
  • vpa.yaml - Vertical Pod Autoscaler configuration
  • strimzi-kafka-cluster.yaml - Strimzi Kafka Operator cluster configuration
  • README.md - This file

Migration from Bitnami to Strimzi

Current State

  • Existing: Bitnami Kafka deployment (single node)
  • Target: Strimzi Kafka Operator with HA cluster (3 brokers + 3 ZK nodes)

Migration Steps

  1. Install Strimzi Operator (already done)

    helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator \
  --namespace freeleaps-data-platform

  2. Delete existing Kafka deployment

    kubectl delete statefulset kafka-controller -n freeleaps-data-platform
kubectl delete service kafka kafka-controller-headless -n freeleaps-data-platform
kubectl delete configmap kafka-controller-configuration -n freeleaps-data-platform
kubectl delete pvc data-kafka-controller-0 -n freeleaps-data-platform

  3. Deploy new Strimzi Kafka cluster

    kubectl apply -f strimzi-kafka-cluster.yaml

New Cluster Features

  • High Availability: 3 Kafka brokers + 3 Zookeeper nodes
  • Authentication: SCRAM-SHA-512
  • Storage: 20Gi per broker, 10Gi per ZK node
  • Replication: 3x replication factor
  • Security: TLS support for internal communication

Monitoring

Check cluster status:

kubectl get kafka -n freeleaps-data-platform
kubectl get kafkatopic -n freeleaps-data-platform
kubectl get kafkauser -n freeleaps-data-platform

Client Configuration

For applications connecting to the new Kafka cluster, use:

  • Bootstrap servers: kafka-cluster-kafka-bootstrap.freeleaps-data-platform.svc.freeleaps.cluster:9092
  • Authentication: SCRAM-SHA-512
  • Username: freeleaps-user
  • Password: Generated by Strimzi (check the secret)

Storage Requirements

  • Kafka brokers: 3 × 20Gi = 60Gi
  • Zookeeper: 3 × 10Gi = 30Gi
  • Total: 90Gi

Network Policy

The cluster includes network policies that allow:

  • Internal cluster communication
  • External client access (with proper authentication)
  • Metrics collection (if enabled)