Kubernetes - Basic to Advanced — Session 7

Overview

This course session (Lesson 7) provides a comprehensive hands-on introduction to Helm, Kubernetes Operators, and Container Storage Interface (CSI) for managing persistent storage in Kubernetes clusters. The session covers creating and deploying Helm charts from scratch, integrating with upstream Helm repositories, installing and configuring operators (specifically Percona’s MongoDB operator), and dynamically provisioning, snapshotting, and restoring persistent volumes using CSI drivers. The instructor emphasizes the distinction between imperative Helm-based deployments and declarative GitOps workflows (e.g., Argo CD), and highlights best practices for production-grade infrastructure as code, including namespace isolation, version pinning, and avoiding in-place upgrades. The lab environment uses Minikube with CSI HostPath for local storage simulation.

Topic (Timeline)

1. Course Introduction and Helm Overview [00:00:01 - 00:04:00]

• Introduction to Lesson 7: Helm, Operators, and Storage.
• Clarification that Helm is used for testing and templating, but in Infrastructure as Code (IaC), Helm templates are consumed declaratively (e.g., by Argo CD), not for state management.
• Outline of upcoming topics: Helm chart creation, Helm repository usage, CSI, Persistent Volumes (PVs), Persistent Volume Claims (PVCs), dynamic provisioning via StorageClass, and operator-based deployments.
• Mention of future lessons: Day of training on IaC (Terraform/Ansible) and declarative GitOps (Argo CD).
• Clarification that Proxmox API integration with Ansible is pending; current setup uses Terraform for VM creation and Ansible for Kubernetes provisioning, with plans to consolidate into Ansible-only IaC.

2. Container Storage Interface (CSI) and Persistent Storage [00:04:00 - 00:07:05]

• Definition of CSI as the Kubernetes standard for block storage access, analogous to CNI for networking.
• Overview of CSI drivers: Longhorn (bare-metal friendly, supports snapshots, encryption, metrics), Minikube’s CSI HostPath (for local development).
• Persistent Volumes (PVs): Cluster-scoped storage resources with lifecycle independent of pods; analogous to Docker external volumes.
• Persistent Volume Claims (PVCs): Namespaced requests for storage; support access modes: ReadWriteOnce, ReadOnlyMany, ReadWriteMany.
• Dynamic Provisioning via StorageClass: Enables on-demand PV creation; allows classification by performance (NVMe, SSD, HDD) and sensitivity (encrypted vs. unencrypted).
• Helm provisioning of storage resources: Helm charts can automate creation of StorageClass, PV, and PVC resources.

3. Helm Chart Structure and Templating [00:07:05 - 00:09:45]

• Standard Helm chart layout: Chart.yaml (chart and app version), values.yaml (user-configurable variables), templates/ (Kubernetes manifest templates).
• Templates use Go templating to interpolate values from values.yaml.
• charts/ directory for subcharts; umbrella charts for demonstrations (not recommended for production with GitOps due to sequencing issues).
• Helm Captain: Person responsible for maintaining Helm charts; upstream maintainers often lack dedicated Helm expertise.
• Common upstream Helm issues: Misconfigured ingress, headless services, incorrect ports or image tags.
• Solutions: Fork and maintain custom chart repos (e.g., GitHub, GitLab); manual patching and version tracking required, leading to operational overhead.

4. Kubernetes Operators and Custom Resource Definitions (CRDs) [00:09:45 - 00:12:30]

• Operators: Automate application lifecycle management by mimicking human operators; interact with Kubernetes API via CRDs.
• Operators are newer and may lack full feature parity with non-operator deployments (e.g., MongoDB without operator may have more features).
• MongoDB StatefulSet: Uses indexed pods (e.g., mongodb-0, mongodb-1); enables high availability with primary/secondary replication and arbiter nodes (no data storage).
• PVs decouple storage lifecycle from app lifecycle: Data persists even if MongoDB pods are deleted; new pods can reattach to existing PVs.
• Minikube requires local-path-storage for StatefulSet PVs.

5. CSI and Storage Lab: PVC, Snapshot, and Restore [00:12:30 - 00:21:56]

• Lab setup: Fresh Minikube single-node cluster.
• Enable volume snapshots: minikube addons enable csi-hostpath-driver.
• Verify CSI components: csi-hostpath-attacher, resizer, plugin pods in kube-system.
• Create PVC using pvc.yaml → dynamically provisioned PV.
• Verify PV creation: kubectl get pv (cluster-scoped) vs. kubectl get pvc (namespaced).
• Create VolumeSnapshotClass and snapshot via csisnapshot.yaml.
• Restore from snapshot: csirestore.yaml must specify correct namespace (PVC namespace); failure occurs if namespace mismatch.
• Fix: Delete failed restore, correct namespace in YAML, reapply.
• Emphasis on namespace best practices: Always define namespace in YAML for production; default namespace acceptable for testing.

6. Helm Chart Creation and Deployment [00:21:56 - 00:27:43]

• Create Helm chart: helm create test-app → generates standard structure.
• Examine Chart.yaml (chart/app version), values.yaml (configurable values), templates/ (manifests).
• Deploy Helm chart: helm install nginx-app test-app -n test-app.
• Verify deployment: helm list -A, kubectl get all -n test-app.
• Uninstall: helm uninstall nginx-app -n test-app → confirms clean removal.
• Note: Helm manages state; Argo CD will replace this in GitOps workflows.

7. Integrating with Upstream Helm Repositories [00:27:43 - 00:34:55]

• Add Cilium Helm repo: helm repo add cilium https://helm.cilium.io/.
• Verify: helm repo list, helm search repo cilium -l to list versions.
• Best practice: Avoid 0.x or 1.x versions for production; wait for 2.x+ (e.g., Longhorn requires 3.x+ due to instability in early releases).
• Update repo: helm repo update.
• Export values: helm show values cilium/cilium > cilium-helm-values.yaml.
• Deploy Cilium: helm install cilium cilium/cilium -n kube-system -f cilium-helm-values.yaml.
• Debug: Cilium operator fails to schedule second replica on single-node cluster due to port conflict (9963) and HA configuration.
• Uninstall: helm uninstall cilium -n kube-system.

8. MongoDB Operator and Database Deployment [00:34:55 - 00:55:36]

• Add Percona MongoDB operator repo: helm repo add percona https://percona.github.io/percona-helm-charts/.
• Create new Minikube profile: 3-node cluster with --cpus=2 --memory=2G.
• Install storage provisioner: minikube addons enable storage-provisioner-rancher to resolve local PV issues.
• Create namespace: kubectl create ns mongodb.
• Install MongoDB operator: helm install mongo-operator percona/psmdb-operator -n mongodb -f psmdb-operator-helm-values.yaml (with telemetry.enabled: false).
• Install MongoDB database: helm install mongo-db percona/psmdb-db -n mongodb -f psmdb-db-helm-values.yaml (with sharding and backup disabled).
• Verify: kubectl get psmdb -n mongodb, kubectl get all -n mongodb.
• SSH into Minikube: minikube ssh → navigate to /opt/local-path-provisioning/ → observe MongoDB data files (WiredTiger, collection, journal, etc.).
• Check encryption: kubectl logs mongodb-psmdb-db-rs-0-0 -n mongodb -c mongod → encryption status not confirmed due to version mismatch (1.21.1); instructor notes need to pin version and investigate flag changes.

9. Summary, Q&A, and Next Steps [00:55:36 - 00:56:56]

• Operators enable scalable, multi-database management via single controller (e.g., one operator managing mongo-db, mongo-db1, etc. in separate namespaces).
• Emphasis on IaC best practices: Use minimal values.yaml files, avoid full upstream values, and use Jinja templates for IaC (not compatible with full Helm values).
• PDFs for Lessons 8 and 9 to be sent via email; typos expected and will be corrected in follow-up.
• Next session: Lesson 8 and 9 (Kubernetes IaC and GitOps with Argo CD) at 11:00 a.m.

Appendix

Key Principles

• Helm in IaC: Use Helm as a templating engine, not a state manager. Declarative GitOps (Argo CD) should control deployment state.
• Version Pinning: Avoid latest in production. Pin to stable versions (e.g., Longhorn ≥ 1.3.x, Cilium ≥ 1.18.2).
• Namespaces: Always define namespace in YAML for production; default namespace is for testing only.
• CSI Drivers: Use csi-hostpath for Minikube; Longhorn for production bare-metal clusters.
• Operators: Use CRDs to automate complex stateful applications (e.g., MongoDB, Redis, Kafka); separate operator and database namespaces in production.

Tools Used

• Helm: Chart creation, templating, deployment, and repository management.
• Minikube: Local Kubernetes cluster for development and testing.
• CSI HostPath: Local storage driver for volume snapshots and dynamic provisioning in Minikube.
• Percona PSMDB Operator: Operator for deploying and managing MongoDB clusters.
• Terraform & Ansible: Used for VM provisioning and Kubernetes setup (future consolidation planned).
• kubectl: Cluster inspection, debugging, and resource management.

Common Pitfalls

• Helm Chart Issues: Upstream charts often have misconfigured ingress, service, or image values; require manual patching and repo maintenance.
• Single-Node HA Conflicts: Deploying HA operators (e.g., Cilium) on single-node clusters causes port conflicts and pending pods.
• Namespace Mismatch: Restore PVCs fail if namespace in csirestore.yaml does not match original PVC’s namespace.
• Storage Provisioning: Minikube lacks default local storage; requires storage-provisioner-rancher addon.
• Encryption Configuration: New MongoDB versions may disable encryption by default; require explicit flags or version pinning.

Practice Suggestions

• Recreate the entire CSI snapshot/restore workflow on a fresh Minikube cluster.
• Build a custom Helm chart for a simple app (e.g., Nginx) and deploy it with overridden values.
• Install a second MongoDB database using the same operator in a new namespace (mongo-db2).
• Compare helm show values output for Cilium and Percona charts to understand structure differences.
• Attempt to deploy Cilium on a 2-node Minikube cluster to resolve the HA scheduling issue.