← Back to Recording

Summary

Overview

This course session focuses on configuring and troubleshooting observability in a Kubernetes environment using Grafana, Prometheus, and sidecar patterns. The instructor guides students through importing pre-built dashboards for Node Explorer, CoreDNS, Kubernetes views, and MongoDB, while addressing common issues with Prometheus data sources. The session concludes with a detailed explanation of sidecar containers for metric collection, service monitors, and methods to persist dashboards via Helm values.yaml configuration. Emphasis is placed on understanding the data flow from application → sidecar → service → service monitor → Prometheus → Grafana.

Topic (Timeline)

1. Dashboard Import: Node Explorer and CoreDNS [00:00:02.830 - 00:03:31.880]

  • Instructor guides students to import Grafana dashboards via ID: 1860 (Node Explorer) and 14981 (CoreDNS).
  • Steps: Navigate to Home → Dashboards → New → Import → enter ID → Load → select Prometheus data source → Import.
  • CoreDNS dashboard is renamed to “CoreDNS 2” to avoid conflict with an existing dashboard.
  • Issue identified: Some students lack Prometheus data source. Suspected root cause: misconfigured Helm values.yaml, particularly in ServiceMonitor CRD fields.
  • Instructor notes that a single character error in the ServiceMonitor YAML can prevent metric collection.

2. Troubleshooting Prometheus Data Source and Helm Installation [00:03:33.000 - 00:07:06.710]

  • Students without Prometheus data are advised to check ServiceMonitor configuration in Helm values.
  • Instructor suggests re-installing Helm charts after verifying YAML files, as incorrect initial installation may leave orphaned or malformed ServiceMonitors.
  • Emphasis on namespace consistency (monitoring namespace) and potential need to delete and recreate Minikube cluster for clean state.
  • Confirmed: Node Explorer and Kubernetes Views dashboards are successfully imported and display global CPU/RAM usage.
  • Instructor acknowledges time constraints prevent full troubleshooting of ServiceMonitor issues.

3. MongoDB Dashboard and Sidecar Pattern Introduction [00:07:06.710 - 00:11:26.340]

  • MongoDB dashboard (ID 20867) imported but shows no data due to missing metrics collection.
  • Explanation of sidecar pattern: secondary container running alongside primary container (e.g., MongoDB) within the same pod.
  • Sidecar purpose: intercept metrics from MongoDB and expose them via an HTTP endpoint for Prometheus scraping.
  • Sidecar increases pod readiness count (e.g., from 3/3 to 4/4) and appears in pod description.
  • Production best practice: Use dedicated metrics user with limited roles instead of reusing admin credentials (e.g., Mongo Express admin).
  • Sidecar is not standalone; it only functions when attached to a primary container.

4. Configuring MongoDB Sidecar and Service Monitor [00:11:26.340 - 00:16:07.920]

  • Sidecar configuration shown in PSMDB Helm values.yaml: uncommented section with image: percona/mongodb-exporter:v0.14.0, name: metrics.
  • Environment variables: MongoDB URI, password (insecure for demo; production should use secrets), and scrape arguments.
  • Service created: mongodb-psmdb-dbmetrics-service to expose sidecar metrics endpoint.
  • ServiceMonitor defined to allow Prometheus to discover and scrape the service (referenced from Slide 81).
  • Data flow: MongoDB → sidecar (exposes metrics) → service (exposes endpoint) → service monitor (tells Prometheus to scrape) → Prometheus → Grafana dashboard.
  • Without this chain, MongoDB dashboard remains empty.

5. Dashboard Persistence and Course Wrap-up [00:16:07.920 - 00:17:30.960]

  • Dashboards imported via UI are lost on Helm uninstall/reinstall.
  • Solution: Define dashboard providers in values.yaml under Grafana section to persist dashboards via ConfigMap or JSON files.
  • Instructor confirms all files (YAML, PDFs, service monitor templates) are available in Lesson 9 for student reference.
  • Videos will be available for at least two weeks; students encouraged to rewatch and re-deploy on Minikube.
  • Final reminder: Understanding the full observability stack (sidecar + service + service monitor + Prometheus + Grafana) is critical for real-world deployments.

Appendix

Key Principles

  • Sidecar Pattern: A secondary container in the same pod that extends functionality (e.g., metrics collection) without modifying the primary app.
  • ServiceMonitor: Custom Resource Definition (CRD) that tells Prometheus how to discover and scrape metrics from a Kubernetes service.
  • Dashboard Persistence: Use Grafana’s dashboardProviders in Helm values.yaml to avoid losing dashboards on Helm reinstallation.

Tools Used

  • Grafana: Dashboard visualization
  • Prometheus: Metrics collection and storage
  • Kube-Prometheus-Stack: Helm chart bundling Prometheus, Alertmanager, Grafana, and ServiceMonitors
  • MongoDB Exporter (Percona): Sidecar container for exposing MongoDB metrics
  • Helm: Package manager for deploying Kubernetes applications
  • Minikube: Local Kubernetes cluster for testing

Common Pitfalls

  • Incorrect ServiceMonitor CRD fields (e.g., typo in selector, namespace, or port) → no metrics collected.
  • Reusing admin credentials for metrics scraping → security risk.
  • Not persisting dashboards via ConfigMap → dashboards lost on Helm upgrade/uninstall.
  • Not waiting long enough after Helm uninstall → orphaned ServiceMonitors remain.

Practice Suggestions

  • Rebuild the MongoDB observability stack from scratch using Lesson 9 files.
  • Modify the sidecar image or args and observe changes in pod readiness and metrics.
  • Create a custom dashboard for a different service using the same pattern (sidecar + service + service monitor).
  • Use kubectl describe pod and kubectl logs -c <sidecar-name> to debug sidecar behavior.