freeleaps-ops/docs/prometheus-metrics-intergration-guideline.md

# 1. Service Provides Metrics Interface

Create a `metrics.py` file in the `webapi/providers` directory to provide the Metrics interface.

Content as follows:
```
# Prometheus metric prefix is : freeleaps_metrics
def register(app):
  instrumentator = (
      Instrumentator().instrument(
        app,
        metric_namespace="freeleaps",
        metric_subsystem="metrics",)
  )

  @app.on_event("startup")
  async def startup():
    instrumentator.expose(app, endpoint="/api/_/metrics", should_gzip=True)
    logging.info("Metrics endpoint exposed at /api/_/metrics")
```

Reference the integration approach at https://gitea.freeleaps.mathmast.com/freeleaps/freeleaps-service-hub/src/branch/master/apps/metrics


# 2. Configure Metrics Collection

## 2.1. Find the helm pkg directory
For freeleaps internal services, you need to find the helm-package configuration for the service deployment in the freeleaps-ops repository
> For example, the metrics service configuration is at `https://gitea.freeleaps.mathmast.com/freeleaps/freeleaps-ops/src/branch/master/freeleaps/helm-pkg/metrics`

For freeleaps tenant services (deployed via one-click), you need to find `.freeleaps/devops/helm-pkg` in the corresponding service repository
> For example, the magicleaps service configuration is at `https://gitea.freeleaps.mathmast.com/products/magicleaps/src/branch/main/.freeleaps/devops/helm-pkg`

## 2.2. Add servicemonitor configuration
Add a `servicemonitor.yaml` file in the templates folder with the following content, referencing `https://gitea.freeleaps.mathmast.com/freeleaps/freeleaps-ops/src/branch/master/freeleaps/helm-pkg/notification/templates/notification/servicemonitor.yaml`

Update necessary content based on the configuration in value.yaml, for example, change line 6 from notification to metrics

## 2.3. Modify value.{alpha|prod}.yaml
Add serviceMonitor configuration, referencing `https://gitea.freeleaps.mathmast.com/freeleaps/freeleaps-ops/src/branch/master/freeleaps/helm-pkg/notification/values.prod.yaml`
```
  - name: notification-service
    type: ClusterIP
    port: 8003
    targetPort: 8003
    serviceMonitor:
      enabled: true # Set to false for alpha environment
      labels:
        release: kube-prometheus-stack
      namespace: freeleaps-monitoring-system
      internal: 30s
      scrapeTimeout: ''
```

## 2.4. Verify Metrics Collection Results

Forward the service [kube-prometheus-stack-prometheus] to local port through LENS, access `http://localhost:9090/targets` to check if the corresponding Endpoint can be found
![alt text](asserts/image.png)
Access `http://localhost:9090/query` to check if the corresponding metrics can be found
![alt text](asserts/image2.png)

# 3. Configure Grafana Dashboard

# 3. Configure Grafana Dashboard

## 3.1. Add `dashboard.yaml` under helm-pkg

Content reference `https://gitea.freeleaps.mathmast.com/freeleaps/freeleaps-ops/src/branch/master/freeleaps/helm-pkg/notification/templates/notification/dashboard.yaml`


## 3.2. Modify value.{alpha|prod}.yaml

```yaml
dashboard:
  enabled: true # Set to false for alpha environment
  name: freeleaps-prod-metrics-dashboard
  title: Freeleaps Metrics Dashboard (PROD/ALPHA)
  metricsPrefix: freeleaps_metrics
```

## 3.3. Verify Dashboard Addition

Login to `https://grafana.mathmast.com` to check if the dashboard was added successfully

![alt text](asserts/image1.png)

![alt text](asserts/image3.png)