feat: Update Pinot configuration and RBAC rules

- Enhanced the Pinot Helm chart values.yaml with comprehensive configurations for controller, broker, server, minion, and zookeeper components.
- Added support for pod disruption budgets and custom resource definitions in RBAC rules.
- Introduced a new script for managing Kubernetes service port forwarding, allowing users to easily forward, stop, and list active services.
- Updated helm repository list to ensure proper access to necessary charts.

Signed-off-by: zhenyus <zhenyus@mathmast.com>

cluster/bin/README.md

@@ -1,14 +1,32 @@
-# Freeleaps Cluster Authenticator
+# Freeleaps Cluster Toolchains
+
+A collection of tools to help you work with Freeleaps Kubernetes Cluster.
+
+- [Freeleaps Cluster Toolchains](#freeleaps-cluster-toolchains)
+  - [Freeleaps Cluster Authenticator](#freeleaps-cluster-authenticator)
+    - [Prerequisites](#prerequisites)
+    - [How to use?](#how-to-use)
+    - [Commands](#commands)
+  - [Freeleaps Cluster Proxifier](#freeleaps-cluster-proxifier)
+    - [Prerequisites](#prerequisites-1)
+    - [How to use?](#how-to-use-1)
+    - [Commands](#commands-1)
+  - [Infrastructure Authentication Retriever](#infrastructure-authentication-retriever)
+    - [Prerequisites](#prerequisites-2)
+    - [How to use?](#how-to-use-2)
+    - [Commands](#commands-2)
+
+## Freeleaps Cluster Authenticator
 
 > Helps to `kubectl` authenticated to `Freeleaps Kubernetes Cluster`.
 
-## Prerequisites
+### Prerequisites
 
 - `mathmast.com` Azure Directory account
 - `Linux` with `Shell / Bash` support
 - `Internet` access
 
-## How to use ?
+### How to use?
 
 First, you need grant this script to executable with commands:
 
@@ -104,7 +122,7 @@ Now you are already finished authentication for `Freeleaps Kubernetes Cluster`.
 
 You can using `kubectl` to work with `Freeleaps Kubernetes CLuster` right now.
 
-## Commands
+### Commands
 
 `Freeleaps Cluster Authenticator` provides lots of useful command to help you complete authentication flow.
 
@@ -116,3 +134,103 @@ You can using `kubectl` to work with `Freeleaps Kubernetes CLuster` right now.
 | `doctor`     | `-d`, `--doctor`     | Check if all the required tools are installed                                                          |
 | `dashboard`  | `-db`, `--dashboard` | Open dashboard for Freeleaps Kubernetes Cluster and forward the port to localhost and listen on `8443` |
 | `get-token`  | `-gt`, `--get-token` | Get the token for the current user, usually to using for Kubernetes Dashboard authentication           |
+
+## Freeleaps Cluster Proxifier
+
+> Helps to forward Kubernetes service ports to your local machine.
+
+### Prerequisites
+
+- Authenticated kubectl access to Freeleaps Kubernetes Cluster
+- Linux with Shell/Bash support
+
+### How to use?
+
+First, grant executable permission to the script:
+
+```bash
+chmod ug+x freeleaps-cluster-proxifier
+```
+
+View available commands:
+
+```bash
+freeleaps-cluster-proxifier -h
+```
+
+List all available services that can be forwarded:
+
+```bash
+freeleaps-cluster-proxifier list-available
+```
+
+Forward a service port:
+
+```bash
+freeleaps-cluster-proxifier forward namespace/service -p local-port:service-port
+```
+
+List all active port forwards:
+
+```bash
+freeleaps-cluster-proxifier list
+```
+
+Stop forwarding a service:
+
+```bash
+freeleaps-cluster-proxifier stop namespace/service
+```
+
+### Commands
+
+| Command          | Alias                    | Usage                                   |
+| ---------------- | ------------------------ | --------------------------------------- |
+| `forward`        | `-f`,`--forward`         | Forward a service port to local machine |
+| `stop`           | `-s`,`--stop`            | Stop forwarding a service               |
+| `list`           | `-l`,`--list`            | List all forwarded services             |
+| `list-available` | `-la`,`--list-available` | List all available services             |
+| `help`           | `-h`,`--help`            | Show help message                       |
+
+## Infrastructure Authentication Retriever
+
+> Helps to retrieve authentication tokens for various infrastructure services.
+
+### Prerequisites
+
+- Authenticated kubectl access to Freeleaps Kubernetes Cluster
+- Linux with Shell/Bash support
+
+### How to use?
+
+First, grant executable permission:
+
+```bash
+chmod ug+x infra-auth-retriver
+```
+
+View available commands:
+
+```bash
+infra-auth-retriver -h
+```
+
+Retrieve authentication token for a specific service:
+
+```bash
+infra-auth-retriver get-token <service-name>
+```
+
+List all available services:
+
+```bash
+infra-auth-retriver list
+```
+
+### Commands
+
+| Command     | Alias               | Usage                                               |
+| ----------- | ------------------- | --------------------------------------------------- |
+| `get-token` | `-gt`,`--get-token` | Retrieve authentication token for specified service |
+| `list`      | `-l`,`--list`       | List all available services                         |
+| `help`      | `-h`,`--help`       | Show help message                                   |

cluster/bin/freeleaps-cluster-proxifier

@@ -0,0 +1,212 @@
+#!/bin/sh
+
+set -eu
+
+VERSION="0.0.1-20250509"
+PROXIFIER_DIR="${HOME}/.freeleaps/proxifier"
+
+help() {
+    echo "Freeleaps Cluster Proxifier (Version: ${VERSION})"
+    echo ""
+    echo "This script helps you to forward Kubernetes service ports to your local machine."
+    echo "It maintains the forwarding state and provides commands to manage port forwarding."
+    echo ""
+    echo "Usage: freeleaps-cluster-proxifier <sub-command>"
+    echo ""
+    echo "Sub Commands:"
+    echo "    forward,-f,--forward <namespace>/<service> -p <local-port>:<service-port>    Forward a service port to local"
+    echo "    stop,-s,--stop <namespace>/<service>                                         Stop forwarding a service"
+    echo "    list,-l,--list                                                               List all forwarded services"
+    echo "    list-available,-la,--list-available                                          List all available services"
+    echo "    help,-h,--help                                                               Show this help message"
+}
+
+ensure_proxifier_dir() {
+    if [ ! -d "${PROXIFIER_DIR}" ]; then
+        mkdir -p "${PROXIFIER_DIR}"
+    fi
+}
+
+get_process_file() {
+    namespace="$1"
+    service="$2"
+    echo "${PROXIFIER_DIR}/${namespace}-${service}.pid"
+}
+
+forward_port() {
+    if [ $# -lt 1 ]; then
+        echo "[ERROR] Invalid number of arguments for forward command"
+        echo "[TIP] Usage: freeleaps-cluster-proxifier forward <namespace>/<service> -p <local-port>:<service-port>"
+        exit 1
+    fi
+
+    # Parse namespace/service
+    IFS='/' read -r namespace service <<EOF
+$1
+EOF
+    
+    if [ -z "${namespace}" ] || [ -z "${service}" ]; then
+        echo "[ERROR] Invalid format. Use namespace/service"
+        exit 1
+    fi
+
+    # Parse port mapping
+    if [ "$2" != "-p" ] || [ -z "$3" ]; then
+        echo "[ERROR] Invalid port format. Use -p <local-port>:<service-port>"
+        exit 1
+    fi
+    ports="$3"
+
+    # Validate service exists and user has permissions
+    if ! kubectl get svc "${service}" -n "${namespace}" >/dev/null 2>&1; then
+        if kubectl get namespace "${namespace}" >/dev/null 2>&1; then
+            echo "[ERROR] Either the service '${service}' doesn't exist in namespace '${namespace}' or you don't have permission to access it"
+            echo "[TIP] Please contact your cluster administrator to request access to this service"
+        else
+            echo "[ERROR] Namespace '${namespace}' doesn't exist or you don't have permission to access it"
+            echo "[TIP] Please contact your cluster administrator to request access to this namespace"
+        fi
+        exit 1
+    fi
+
+    process_file=$(get_process_file "${namespace}" "${service}")
+    
+    if [ -f "${process_file}" ]; then
+        echo "[ERROR] Service ${service} in namespace ${namespace} is already being forwarded"
+        echo "[TIP] Use 'freeleaps-cluster-proxifier list' to see active forwards"
+        exit 1
+    fi
+
+    ensure_proxifier_dir
+
+    echo "[FORWARD] Starting port forward for ${service} in namespace ${namespace}..."
+    kubectl port-forward -n "${namespace}" "svc/${service}" "${ports}" > /dev/null 2>&1 & 
+    pid=$!
+
+    # Store PID and port mapping
+    echo "${pid}:${ports}" > "${process_file}"
+    
+    echo "[FORWARD] Port forward started successfully"
+    echo "[INFO] Service ${service}.${namespace} is now mapping with ${ports}"
+}
+
+stop_forward() {
+    if [ $# -ne 1 ]; then
+        echo "[ERROR] Invalid number of arguments for stop command"
+        echo "[TIP] Usage: freeleaps-cluster-proxifier stop <namespace>/<service>"
+        exit 1
+    fi
+
+    # Parse namespace/service
+    IFS='/' read -r namespace service <<EOF
+$1
+EOF
+    
+    if [ -z "${namespace}" ] || [ -z "${service}" ]; then
+        echo "[ERROR] Invalid format. Use namespace/service"
+        exit 1
+    fi
+
+    process_file=$(get_process_file "${namespace}" "${service}")
+
+    if [ ! -f "${process_file}" ]; then
+        echo "[ERROR] No active forward found for service ${service} in namespace ${namespace}"
+        exit 1
+    fi
+
+    pid=$(cat "${process_file}" | cut -d: -f1)
+    if kill "${pid}" >/dev/null 2>&1; then
+        rm "${process_file}"
+        echo "[STOP] Stopped forwarding service ${service} in namespace ${namespace}"
+    else
+        echo "[WARNING] Process not found, cleaning up state file"
+        rm "${process_file}"
+    fi
+}
+
+list_forwards() {
+    ensure_proxifier_dir
+
+    echo "Belows are all active port forwards:"
+    printf "%-30s %-60s %-15s %-10s\n" "Namespace" "Service" "Port Mapping" "PID"
+    
+    for file in "${PROXIFIER_DIR}"/*.pid; do
+        if [ -f "${file}" ]; then
+            name=$(basename "${file}" .pid)
+            namespace=$(echo "${name}" | cut -d'-' -f1)
+            service=$(echo "${name}" | cut -d'-' -f2-)
+            data=$(cat "${file}")
+            pid=$(echo "${data}" | cut -d: -f1)
+            ports=$(echo "${data}" | cut -d: -f2-)
+            
+            # Check if process is still running
+            if kill -0 "${pid}" >/dev/null 2>&1; then
+                printf "%-30s %-60s %-15s %-10s\n" "${namespace}" "${service}" "${ports}" "${pid}"
+            else
+                echo "[WARNING] Cleaning up stale forward for ${service} in namespace ${namespace}"
+                rm "${file}"
+            fi
+        fi
+    done
+}
+
+list_available_services() {
+    echo "Belows are all available services that you can forward:"
+    printf "%-30s %-60s %-10s\n" "Namespace" "Service" "Ports"
+    
+    # Get all namespaces user has access to
+    kubectl get namespaces -o name | cut -d'/' -f2 | while read -r ns; do
+        # Get services in each namespace
+        if kubectl auth can-i get services -n "${ns}" >/dev/null 2>&1; then
+            kubectl get services -n "${ns}" \
+                --no-headers \
+                -o custom-columns="Namespace:.metadata.namespace,Service:.metadata.name,Ports:.spec.ports[*].port" | \
+            while read -r line; do
+                # Only show if user has permission to port-forward
+                svc_name=$(echo "${line}" | awk '{print $2}')
+                if kubectl auth can-i get services/"${svc_name}" -n "${ns}" >/dev/null 2>&1; then
+                    namespace=$(echo "${line}" | awk '{print $1}')
+                    service=$(echo "${line}" | awk '{print $2}')
+                    ports=$(echo "${line}" | awk '{print $3}')
+                    printf "%-30s %-60s %-10s\n" "${namespace}" "${service}" "${ports}"
+                fi
+            done
+        fi
+    done
+}
+
+main() {
+    if [ $# -lt 1 ]; then
+        echo "[ERROR] No sub-command provided"
+        echo "[TIP] Run 'freeleaps-cluster-proxifier -h' to see available sub-commands"
+        exit 1
+    fi
+
+    subcommand="$1"
+    shift
+
+    case "${subcommand}" in
+        forward|-f|--forward)
+            forward_port "$@"
+        ;;
+        stop|-s|--stop)
+            stop_forward "$@"
+        ;;
+        list|-l|--list)
+            list_forwards
+        ;;
+        list-available|-la|--list-available)
+            list_available_services
+        ;;
+        help|-h|--help)
+            help
+        ;;
+        *)
+            echo "[ERROR] Invalid sub-command: ${subcommand}"
+            help
+            exit 1
+        ;;
+    esac
+}
+
+main "$@"

cluster/bin/infra-auth-retriver

@@ -15,6 +15,7 @@ help() {
   echo "  help,-h,--help        Show help"
   echo "  grafana               Retrieve Grafana Auth"
   echo "  argocd                Retrieve ArgoCD Auth"
+  echo "  kafka                 Retrieve Kafka password"
 }
 
 main() {
@@ -38,6 +39,10 @@ main() {
       echo "ArgoCD User: admin"
       echo "ArgoCD Auth: $(kubectl get secret argocd-initial-admin-secret -n freeleaps-devops-system -o jsonpath='{.data.password}' | base64 -d)"
       ;;
+    kafka)
+      echo "Kafka Username: freeleaps"
+      echo "Kafka Password: $(kubectl get secret kafka-user-passwords --namespace freeleaps-data-platform -o jsonpath='{.data.client-passwords}' | base64 -d | cut -d , -f 1)"
+      ;;
     *)
       help
       ;;

cluster/manifests/freeleaps-data-platform/doris/deploy.sh

@@ -1,16 +0,0 @@
-#!/bin/bash
-
-# 创建命名空间
-kubectl create namespace freeleaps-data-platform
-
-# 安装Doris Operator CRD
-kubectl create -f https://raw.githubusercontent.com/apache/doris-operator/master/config/crd/bases/doris.apache.com_dorisclusters.yaml
-
-# 部署 Doris Operator
-kubectl apply -f https://raw.githubusercontent.com/apache/doris-operator/master/config/operator/operator.yaml
-
-# 创建 Doris 集群配置
-kubectl apply -f doris-cluster.yaml -n freeleaps-data-platform
-
-# 检查部署状态
-kubectl get pods -n freeleaps-data-platform -l app=doris

cluster/manifests/freeleaps-data-platform/doris/doris-cluster.yaml

@@ -1,57 +0,0 @@
-apiVersion: doris.apache.com/v1
-kind: DorisCluster
-metadata:
-  name: doris-cluster
-  namespace: freeleaps-data-platform
-spec:
-  clusterDomain: "freeleaps.cluster"
-
-  feSpec:
-    replicas: 1
-    image: apache/doris:2.0.2
-    resources:
-      requests:
-        cpu: "1"
-        memory: "2Gi"
-      limits:
-        cpu: "2"
-        memory: "4Gi"
-    service:
-      type: ClusterIP
-    configMap:
-      fe.conf: |
-        JAVA_OPTS="-Xmx2048m -XX:+UseG1GC"
-
-  beSpec:
-    replicas: 1
-    image: apache/doris:2.0.2
-    storage:
-      storageSize: "50Gi"
-      storageClassName: "azure-disk-std-ssd-lrs"
-    resources:
-      requests:
-        cpu: "2"
-        memory: "4Gi"
-      limits:
-        cpu: "4"
-        memory: "8Gi"
-    storage:
-      storageSize: "50Gi"
-      storageClassName: "standard"
-    configMap:
-      be.conf: |
-        JAVA_OPTS="-Xmx8192m -XX:+UseG1GC"
-        BE_ADDR=${POD_IP}:9060
-        BE_HTTP_PORT=8040
-        BE_PORT=9060
-        HEARTBEAT_SERVICE_PORT=9050
-        BRPC_PORT=8060
-
-  feAddress: doris-cluster-fe-service
-
-  monitoring:
-    enabled: true
-    prometheus:
-      serviceMonitor:
-        enabled: true
-        namespace: freeleaps-monitoring-system

cluster/manifests/freeleaps-data-platform/kafka/kafka.sh

@@ -1,6 +0,0 @@
-#!/usr/bin/env bash
-# 查看kafka的pod
-kubectl get pods -n freeleaps-data-platform
-# 查看kafka的服务
-kubectl get svc -n freeleaps-data-platform
-

cluster/manifests/freeleaps-data-platform/pinot/values.yaml

@@ -1,42 +1,804 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Default values for Pinot.
+
+namespaceOverride:
+namespaceAnnotations: {}
+
+image:
+  repository: apachepinot/pinot
+  # Pinot docker images are available at https://hub.docker.com/r/apachepinot/pinot/tags
+  #   - `latest` tag is always available and points to the nightly pinot master branch build
+  #   - `release-x.y.z` or `x.y.z` tags are available for each release, e.g. release-1.0.0, release-0.12.1, 1.0.0, 0.12.1, etc.
+  #
+  # Default JDK comes with Amazon Corretto 11, here are also images with different JDKs:
+  #   - Amazon Corretto 11, e.g. `latest-11`, `1.0.0-11`, `latest-11-amazoncorretto`, `1.0.0-11-amazoncorretto`
+  #   - Amazon Corretto 17, e.g. `latest-17-amazoncorretto`, `1.0.0-17-amazoncorretto`
+  #   - MS OpenJDK 11, e.g. `latest-11-ms-openjdk`, `1.0.0-11-ms-openjdk`
+  #   - MS OpenJDK 17, e.g. `latest-17-ms-openjdk`, `1.0.0-17-ms-openjdk`
+  #   - OpenJDK 21, e.g. `latest-21-openjdk`, `1.0.0-21-openjdk`
+  tag: latest # 1.0.0, 0.12.1, latest
+  pullPolicy: Always # Use IfNotPresent when you pinged a version of image tag
+
 cluster:
-  name: pinot-cluster
+  name: freeleaps
 
-namespaceOverride: "freeleaps-data-platform"
+imagePullSecrets: []
 
-controller:
+terminationGracePeriodSeconds: 30
-  replicaCount: 1
-  persistence:
-    enabled: true
-    size: 20Gi
-  ingress:
-    enabled: true
-    annotations:
-      kubernetes.io/ingress.class: nginx
-      nginx.ingress.kubernetes.io/rewrite-target: /$2
-      cluster-issuer: mathmast-dot-com
-    hosts:
-      - host: pinot.freeleaps.com
-        paths:
-          - path: /pinot(/|$)(.*)
-            pathType: Prefix
-            port: 9000
-    tls:
-      - secretName: pinot-dot-mathmast-dot-com-tls
-        hosts:
-          - pinot.freeleaps.com
 
-broker:
+securityContext: {}
-  replicaCount: 1
+  # capabilities:
+  #   drop:
+  #   - ALL
+  # readOnlyRootFilesystem: true
+  # runAsNonRoot: true
+  # runAsUser: 1000
 
-server:
+# default values of the probes i.e. liveness and readiness.
-  replicaCount: 1
+# customization of values is present at the component level.
-  persistence:
+probes:
-    enabled: true
+  initialDelaySeconds: 60
-    size: 50Gi
+  periodSeconds: 10
+  failureThreshold: 10
+  # should be 1 for liveness and startup probe, as per K8s doc.
+  successThreshold: 1
+  timeoutSeconds: 10
 
-zookeeper:
+serviceAccount:
+  # Specifies whether a service account should be created
+  create: true
+  # Annotations to add to the service account
+  annotations: {}
+  # The name of the service account to use.
+  # If not set and create is true, a name is generated using the fullname template
+  name: ""
+
+additionalMatchLabels: {}
+
+
+pinotAuth:
   enabled: false
+  controllerFactoryClass: org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory
+  brokerFactoryClass: org.apache.pinot.broker.broker.BasicAuthAccessControlFactory
+  configs:
+  #  - access.control.principals=admin,user
+  #  - access.control.principals.admin.password=verysecret
+  #  - access.control.principals.user.password=secret
+  #  - access.control.principals.user.tables=baseballStats,otherstuff
+  #  - access.control.principals.user.permissions=READ
+
+# ------------------------------------------------------------------------------
+# Pinot Controller:
+# ------------------------------------------------------------------------------
+controller:
+  name: controller
+  # Controls whether controller.port is included in the configuration.
+  # Set to false to exclude controller.port when using TLS-only mode or when
+  # you want to specify the port in controller.access.protocols.https.port instead.
+  configureControllerPort: true
+  replicaCount: 1
+  podManagementPolicy: Parallel
+  podSecurityContext: {}
+    # fsGroup: 2000
+  securityContext: {}
+  startCommand: "StartController"
+
+  probes:
+    endpoint: "/health"
+    livenessEnabled: false
+    readinessEnabled: false
+    startupEnabled: false
+    liveness:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    readiness:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    startup:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+  persistence:
+    enabled: true
+    accessMode: ReadWriteOnce
+    size: 1G
+    mountPath: /var/pinot/controller/data
+    storageClass: ""
+    extraVolumes: []
+    extraVolumeMounts: []
+
+  data:
+    dir: /var/pinot/controller/data
+
+  vip:
+    enabled: false
+    host: pinot-controller
+    port: 9000
+
+  jvmOpts: "-XX:ActiveProcessorCount=2 -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-controller.log -Djute.maxbuffer=4000000"
+
+  log4j2ConfFile: /opt/pinot/etc/conf/pinot-controller-log4j2.xml
+  pluginsDir: /opt/pinot/plugins
+
+  pdb:
+    enabled: false 
+    minAvailable: ""
+    maxUnavailable: 50%
+
+  service:
+    annotations: {}
+    clusterIP: "None"
+    externalIPs: []
+    loadBalancerIP: ""
+    loadBalancerSourceRanges: []
+    type: ClusterIP
+    port: 9000
+    nodePort: ""
+    protocol: TCP
+    name: controller
+    extraPorts: []
+      # - containerPort: 1234
+      #   protocol: PROTOCOL
+      #   name: extra-port
+
   external:
     enabled: true
-    host: "kafka-zookeeper"
+    type: LoadBalancer
-    port: 2181
+    port: 9000
+    annotations: {}
+
+  ingress:
+    v1beta1:
+      enabled: false
+      annotations: { }
+      tls: { }
+      path: /
+      hosts: [ ]
+      # port: 9433
+    v1:
+      enabled: false
+      ingressClassName: ""
+      annotations: {}
+      tls: []
+      path: /
+      hosts: []
+      # port: 9433
+
+  resources:
+    requests:
+      memory: "1.25Gi"
+
+  nodeSelector: {}
+
+  tolerations: []
+
+  initContainers: []
+
+  affinity: {}
+
+  podAnnotations: {}
+
+  # set enabled as true, to automatically roll controller stateful set for configmap change
+  automaticReload:
+    enabled: false
+
+  updateStrategy:
+    type: RollingUpdate
+
+  # Use envFrom to define all of the ConfigMap or Secret data as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables
+  envFrom: []
+  #  - configMapRef:
+  #      name: special-config
+  #  - secretRef:
+  #      name: test-secret
+
+  # Use extraEnv to add individual key value pairs as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
+  extraEnv:
+    - name: LOG4J_CONSOLE_LEVEL
+      value: info
+  #  - name: PINOT_CUSTOM_ENV
+  #    value: custom-value
+
+  # Extra configs will be appended to pinot-controller.conf file
+  extra:
+    configs: |-
+      pinot.set.instance.id.to.hostname=true
+      controller.task.scheduler.enabled=true
+
+# ------------------------------------------------------------------------------
+# Pinot Broker:
+# ------------------------------------------------------------------------------
+broker:
+  name: broker
+  # Controls whether pinot.broker.client.queryPort is included in the configuration.
+  # Set to false to exclude pinot.broker.client.queryPort when using TLS-only mode or when
+  # you want to specify the port in pinot.broker.access.protocols.https.port instead.
+  configureBrokerPort: true
+  replicaCount: 1
+  podManagementPolicy: Parallel
+  podSecurityContext: {}
+    # fsGroup: 2000
+  securityContext: {}
+  startCommand: "StartBroker"
+
+  jvmOpts: "-XX:ActiveProcessorCount=2 -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-broker.log -Djute.maxbuffer=4000000"
+
+  log4j2ConfFile: /opt/pinot/etc/conf/pinot-broker-log4j2.xml
+  pluginsDir: /opt/pinot/plugins
+
+  routingTable:
+    builderClass: random
+
+  probes:
+    endpoint: "/health"
+    livenessEnabled: true
+    readinessEnabled: true
+    startupEnabled: false
+    liveness:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    readiness:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    startup:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+  persistence:
+    extraVolumes: []
+    extraVolumeMounts: []
+
+  pdb:
+    enabled: false
+    minAvailable: ""
+    maxUnavailable: 50%
+
+  service:
+    annotations: {}
+    clusterIP: "None"
+    externalIPs: []
+    loadBalancerIP: ""
+    loadBalancerSourceRanges: []
+    type: ClusterIP
+    protocol: TCP
+    port: 8099
+    name: broker
+    nodePort: ""
+    extraPorts: []
+      # - containerPort: 1234
+      #   protocol: PROTOCOL
+      #   name: extra-port
+
+  external:
+    enabled: true
+    type: LoadBalancer
+    port: 8099
+    # For example, in private GKE cluster, you might add cloud.google.com/load-balancer-type: Internal
+    annotations: {}
+
+  ingress:
+    v1beta1:
+      enabled: false
+      annotations: {}
+      tls: {}
+      path: /
+      hosts: []
+      # port: 8443
+    v1:
+      enabled: false
+      ingressClassName: ""
+      annotations: {}
+      tls: []
+      path: /
+      hosts: []
+      # port: 8443
+
+  resources:
+    requests:
+      memory: "1.25Gi"
+
+  nodeSelector: {}
+
+  affinity: {}
+
+  tolerations: []
+
+  initContainers: []
+
+  podAnnotations: {}
+
+  # set enabled as true, to automatically roll broker stateful set for configmap change
+  automaticReload:
+    enabled: false
+
+  updateStrategy:
+    type: RollingUpdate
+
+  # Use envFrom to define all of the ConfigMap or Secret data as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables
+  envFrom: []
+  #  - configMapRef:
+  #      name: special-config
+  #  - secretRef:
+  #      name: test-secret
+
+  # Use extraEnv to add individual key value pairs as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
+  extraEnv:
+    - name: LOG4J_CONSOLE_LEVEL
+      value: info
+  #  - name: PINOT_CUSTOM_ENV
+  #    value: custom-value
+
+  # Extra configs will be appended to pinot-broker.conf file
+  extra:
+    configs: |-
+      pinot.set.instance.id.to.hostname=true
+      pinot.query.server.port=7321
+      pinot.query.runner.port=7732
+
+# ------------------------------------------------------------------------------
+# Pinot Server:
+# ------------------------------------------------------------------------------
+server:
+  name: server
+  # Controls whether pinot.server.netty.port is included in the configuration.
+  # Set to false to exclude pinot.server.netty.port when using TLS-only mode or when
+  # you want to specify the port in pinot.server.nettytls.port instead.
+  configureServerPort: true
+  replicaCount: 1
+  podManagementPolicy: Parallel
+  podSecurityContext: {}
+    # fsGroup: 2000
+  securityContext: {}
+  startCommand: "StartServer"
+
+  probes:
+    endpoint: "/health"
+    livenessEnabled: false
+    readinessEnabled: false
+    startupEnabled: false
+    liveness:
+      endpoint: "/health/liveness"
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    readiness:
+      endpoint: "/health/readiness"
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    startup:
+      endpoint: "/health/liveness"
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+  dataDir: /var/pinot/server/data/index
+  segmentTarDir: /var/pinot/server/data/segment
+
+  persistence:
+    enabled: true
+    accessMode: ReadWriteOnce
+    size: 4G
+    mountPath: /var/pinot/server/data
+    storageClass: ""
+    #storageClass: "ssd"
+    extraVolumes: []
+    extraVolumeMounts: []
+
+  jvmOpts: "-Xms512M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log -Djute.maxbuffer=4000000"
+
+  log4j2ConfFile: /opt/pinot/etc/conf/pinot-server-log4j2.xml
+  pluginsDir: /opt/pinot/plugins
+
+  pdb:
+    enabled: false 
+    minAvailable: ""
+    maxUnavailable: 1
+
+  service:
+    annotations: {}
+    clusterIP: ""
+    externalIPs: []
+    loadBalancerIP: ""
+    loadBalancerSourceRanges: []
+    type: ClusterIP
+    nettyPort: 8098
+    nettyPortName: netty
+    adminPort: 8097
+    adminExposePort: 80
+    adminPortName: admin
+    nodePort: ""
+    protocol: TCP
+    extraPorts: []
+      # - containerPort: 1234
+      #   protocol: PROTOCOL
+      #   name: extra-port
+
+  resources:
+    requests:
+      memory: "1.25Gi"
+
+  nodeSelector: {}
+
+  affinity: {}
+
+  tolerations: []
+
+  initContainers: []
+
+  podAnnotations: {}
+
+  # set enabled as true, to automatically roll server stateful set for configmap change
+  automaticReload:
+    enabled: false
+
+  updateStrategy:
+    type: RollingUpdate
+
+  # Use envFrom to define all of the ConfigMap or Secret data as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables
+  envFrom: []
+  #  - configMapRef:
+  #      name: special-config
+  #  - secretRef:
+  #      name: test-secret
+
+  # Use extraEnv to add individual key value pairs as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
+  extraEnv:
+    - name: LOG4J_CONSOLE_LEVEL
+      value: info
+  #  - name: PINOT_CUSTOM_ENV
+  #    value: custom-value
+
+  # Extra configs will be appended to pinot-server.conf file
+  extra:
+    configs: |-
+      pinot.set.instance.id.to.hostname=true
+      pinot.server.instance.realtime.alloc.offheap=true
+      pinot.query.server.port=7321
+      pinot.query.runner.port=7732
+
+# ------------------------------------------------------------------------------
+# Pinot Minion:
+# ------------------------------------------------------------------------------
+minion:
+  enabled: false
+  name: minion
+  # Controls whether pinot.minion.port is included in the configuration.
+  # Set to false to exclude pinot.minion.port when using TLS-only mode
+  # or when you're configuring ports through another mechanism.
+  configureMinionPort: true
+  replicaCount: 0
+  podManagementPolicy: Parallel
+  podSecurityContext: {}
+    # fsGroup: 2000
+  securityContext: {}
+  startCommand: "StartMinion"
+
+  probes:
+    endpoint: "/health"
+    livenessEnabled: true
+    readinessEnabled: true
+    startupEnabled: false
+    liveness:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    readiness:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    startup:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+  dataDir: /var/pinot/minion/data
+  jvmOpts: "-XX:ActiveProcessorCount=2 -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-minion.log -Djute.maxbuffer=4000000"
+
+  log4j2ConfFile: /opt/pinot/etc/conf/pinot-minion-log4j2.xml
+  pluginsDir: /opt/pinot/plugins
+
+  persistence:
+    enabled: true
+    accessMode: ReadWriteOnce
+    size: 4G
+    mountPath: /var/pinot/minion/data
+    storageClass: ""
+    #storageClass: "ssd"
+    extraVolumes: []
+    extraVolumeMounts: []
+
+  service:
+    annotations: {}
+    clusterIP: ""
+    externalIPs: []
+    loadBalancerIP: ""
+    loadBalancerSourceRanges: []
+    type: ClusterIP
+    port: 9514
+    nodePort: ""
+    protocol: TCP
+    name: minion
+    extraPorts: []
+      # - containerPort: 1234
+      #   protocol: PROTOCOL
+      #   name: extra-port
+
+  resources:
+    requests:
+      memory: "1.25Gi"
+
+  nodeSelector: {}
+
+  affinity: {}
+
+  tolerations: []
+
+  initContainers: []
+
+  podAnnotations: {}
+
+  automaticReload:
+    enabled: false
+
+  updateStrategy:
+    type: RollingUpdate
+
+  # Use envFrom to define all of the ConfigMap or Secret data as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables
+  envFrom: []
+  #  - configMapRef:
+  #      name: special-config
+  #  - secretRef:
+  #      name: test-secret
+
+  # Use extraEnv to add individual key value pairs as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
+  extraEnv:
+    - name: LOG4J_CONSOLE_LEVEL
+      value: info
+  #  - name: PINOT_CUSTOM_ENV
+  #    value: custom-value
+
+  # Extra configs will be appended to pinot-minion.conf file
+  extra:
+    configs: |-
+      pinot.set.instance.id.to.hostname=true
+
+
+# ------------------------------------------------------------------------------
+# Pinot Minion Stateless:
+# ------------------------------------------------------------------------------
+minionStateless:
+  enabled: true
+  name: minion-stateless
+  # Controls whether pinot.minion.port is included in the configuration.
+  # Set to false to exclude pinot.minion.port when using TLS-only mode
+  # or when you're configuring ports through another mechanism.
+  configureMinionStatelessPort: true
+  replicaCount: 1
+  podSecurityContext: {}
+    # fsGroup: 2000
+  securityContext: {}
+  startCommand: "StartMinion"
+
+  probes:
+    endpoint: "/health"
+    livenessEnabled: true
+    readinessEnabled: true
+    startupEnabled: true
+    liveness:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    readiness:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+    startup:
+      initialDelaySeconds: 60
+      failureThreshold: 10
+      timeoutSeconds: 10
+      successThreshold: 1
+      periodSeconds: 10
+
+  dataDir: /var/pinot/minion/data
+  jvmOpts: "-XX:ActiveProcessorCount=2 -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-minion.log -Djute.maxbuffer=4000000"
+
+  log4j2ConfFile: /opt/pinot/etc/conf/pinot-minion-log4j2.xml
+  pluginsDir: /opt/pinot/plugins
+
+  persistence:
+    enabled: false
+    pvcName: minion-data-vol
+    accessMode: ReadWriteOnce
+    size: 4G
+    mountPath: /var/pinot/minion/data
+    storageClass: ""
+    #storageClass: "ssd"
+    extraVolumes: []
+    extraVolumeMounts: []
+
+  service:
+    port: 9514
+    protocol: TCP
+    name: minion
+    extraPorts: []
+      # - containerPort: 1234
+      #   protocol: PROTOCOL
+      #   name: extra-port
+      
+  resources:
+    requests:
+      memory: "1.25Gi"
+
+  nodeSelector: {}
+
+  affinity: {}
+
+  tolerations: []
+
+  initContainers: []
+
+  podAnnotations: {}
+
+  # Use envFrom to define all of the ConfigMap or Secret data as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#configure-all-key-value-pairs-in-a-configmap-as-container-environment-variables
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#configure-all-key-value-pairs-in-a-secret-as-container-environment-variables
+  envFrom: []
+  #  - configMapRef:
+  #      name: special-config
+  #  - secretRef:
+  #      name: test-secret
+
+  # Use extraEnv to add individual key value pairs as container environment variables.
+  # ref: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
+  extraEnv:
+    - name: LOG4J_CONSOLE_LEVEL
+      value: info
+  #  - name: PINOT_CUSTOM_ENV
+  #    value: custom-value
+
+  # Extra configs will be appended to pinot-minion.conf file
+  extra:
+    configs: |-
+      pinot.set.instance.id.to.hostname=true
+
+# ------------------------------------------------------------------------------
+# Zookeeper:
+# NOTE: IN PRODUCTION USE CASES, IT's BEST TO USE ZOOKEEPER K8S OPERATOR
+# ref: https://github.com/pravega/zookeeper-operator#install-the-operator
+# ------------------------------------------------------------------------------
+
+zookeeper:
+  ## If true, install the Zookeeper chart alongside Pinot
+  ## ref: https://github.com/bitnami/charts/tree/master/bitnami/zookeeper
+  enabled: true
+
+  ## If the Zookeeper Chart is disabled a URL override is required to connect
+  urlOverride: "my-zookeeper:2181/my-pinot"
+
+  ## Zookeeper port
+  port: 2181
+
+  ## Configure Zookeeper resource requests and limits
+  ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
+  resources:
+    requests:
+      memory: "1.25Gi"
+
+  ## Replicas
+  replicaCount: 1
+
+  ## Ongoing data directory cleanup configuration
+  autopurge:
+
+    ## The time interval (in hours) for which the purge task has to be triggered
+    ## Set to a positive integer to enable the auto purging
+    purgeInterval: 1
+
+    ## The most recent snapshots amount (and corresponding transaction logs) to retain
+    snapRetainCount: 5
+
+  ## Size (in MB) for the Java Heap options (Xmx and Xms)
+  ## This env var is ignored if Xmx an Xms are configured via `zookeeper.jvmFlags`
+  heapSize: "1024"
+
+  ## Extra JVM Flags for Zookeeper
+  jvmFlags: "-Djute.maxbuffer=4000000"
+
+  persistence:
+    enabled: true
+    storageClass: ""
+    #storageClass: "ssd"
+
+    ## The amount of PV storage allocated to each Zookeeper pod in the statefulset
+    size: "8Gi"
+
+  ## Specify a Zookeeper imagePullPolicy
+  ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
+  image:
+    PullPolicy: "IfNotPresent"
+
+  ## Pod scheduling preferences (by default keep pods within a release on separate nodes).
+  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
+  ## By default we don't set affinity:
+  affinity: {}  # Criteria by which pod label-values influence scheduling for zookeeper pods.
+  # podAntiAffinity:
+  #   requiredDuringSchedulingIgnoredDuringExecution:
+  #     - topologyKey: "kubernetes.io/hostname"
+  #       labelSelector:
+  #         matchLabels:
+  #           release: zookeeper

cluster/manifests/freeleaps-data-platform/rbac/rbac.yaml

@@ -5,7 +5,7 @@ metadata:
   namespace: freeleaps-data-platform
 rules:
   - apiGroups: [""]
-    resources: ["secrets", "pods", "configmaps", "endpoints", "services", "persistentvolumeclaims"]
+    resources: ["secrets", "pods", "configmaps", "endpoints", "services", "serviceaccounts", "persistentvolumeclaims"]
     verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   - apiGroups: ["apps"]
     resources: ["deployments", "replicasets"]
@@ -25,6 +25,12 @@ rules:
   - apiGroups: [""]
     resources: ["nodes"]
     verbs: ["get", "list", "watch"]
+  - apiGroups: ["policy"]
+    resources: ["poddisruptionbudgets"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+  - apiGroups: ["apiextensions.k8s.io"]
+    resources: ["customresourcedefinitions"]
+    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding

cluster/manifests/helm-repos/REPO.list

@@ -13,4 +13,4 @@ kubernetes-dashboard,https://kubernetes.github.io/dashboard/,force-update
 grafana,https://grafana.github.io/helm-charts,force-update
 fluent,https://fluent.github.io/helm-charts,force-update
 pinot,https://raw.githubusercontent.com/apache/pinot/master/helm,force-update
-starrocks,https://starrocks.github.io/starrocks-kubernetes-operator,force-update
+starrocks,https://starrocks.github.io/starrocks-kubernetes-operator,force-update

first-class-pipeline/src/com/freeleaps/devops/ArgoApplicationVersionUpdater.groovy

@@ -30,22 +30,6 @@ class ArgoApplicationVersionUpdater {
       steps.writeYaml(file: valuesFile, data: data, overwrite: true)
 
       steps.withCredentials([steps.usernamePassword(credentialsId: 'freeleaps-ops-credentials', passwordVariable: 'OPS_GIT_PASSWORD', usernameVariable: 'OPS_GIT_USERNAME')]) {
-        // steps.sh """
-        //   echo "Install required tools for git..."
-        //   apt-get -y update && apt-get install -y --no-install-recommends git apt-transport-https ca-certificates gnupg
-        //   echo "Set ${workspace} as a safe directory..."
-        //   git config --global --add safe.directory ${workspace}
-        //   echo "Configure git user..."
-        //   git config user.name "freeleaps-gitops-bot"
-        //   git config user.email "gitops@mathmast.com"
-        //   echo "Add and commit changes..."
-        //   git remote add ci_origin https://${steps.env.OPS_GIT_USERNAME}:${steps.env.OPS_GIT_PASSWORD}@dev.azure.com/freeleaps/freeleaps-ops/_git/freeleaps-ops
-        //   git add ${valuesFile}
-        //   git commit -m "ci(bump): bump ${component.name} image version for ${environmentSlug} to ${steps.env.BUILD_IMAGE_VERSION}"
-        //   echo "Push changes to freeleaps-ops repository..."
-        //   git push ci_origin HEAD:master
-        //   echo "Done."
-        // """
         steps.sh """
           #!/bin/bash