Docker and Kubernetes Training Course — Session 9

Overview

This course provides a comprehensive hands-on deep dive into advanced Kubernetes concepts, focusing on DaemonSets, StatefulSets, security (authentication, authorization, RBAC), network policies, ingress controllers, and monitoring. The session guides learners through practical demonstrations of pod lifecycle management with DaemonSets, node-specific pod scheduling using node selectors and taints/tolerations, stateful application deployment with persistent storage and ordered scaling, fine-grained access control via roles and role bindings, network traffic restriction using network policies, and exposing services via ingress controllers. The course concludes with an introduction to Prometheus and Grafana for cluster monitoring and touches on Helm for package management.

Topic (Timeline)

1. DaemonSets: Auto-Recovery and Node-Specific Scheduling [00:00:20 - 00:06:35]

• Demonstrated deletion of a pod and observed automatic recreation by a DaemonSet, contrasting with ReplicaSets in Deployments.
• Explained that DaemonSets ensure one pod runs on every node (or selected nodes), critical for system agents (e.g., logging, monitoring).
• Introduced node selectors to restrict DaemonSet pods to nodes with specific labels (e.g., storage=ssd).
• Demonstrated adding labels to worker nodes via kubectl label node, verifying with get nodes --show-labels.
• Created and applied a DaemonSet YAML with a node selector, confirming pod placement only on labeled nodes.
• Showed that DaemonSets support rolling updates and revision history similar to Deployments.

2. DaemonSets: Taints, Tolerations, and Master Node Scheduling [00:06:35 - 00:09:29]

• Explained taints on master nodes (e.g., node-role.kubernetes.io/control-plane:NoSchedule) preventing regular pods from scheduling.
• Introduced tolerations as a mechanism to override taints: pods with matching tolerations can schedule on tainted nodes.
• Demonstrated creating a DaemonSet with a toleration for node-role.kubernetes.io/control-plane:NoSchedule, enabling it to run on the master.
• Emphasized that this is a controlled exception — not a recommendation to remove taints entirely, as master nodes should remain reserved for control plane components.
• Confirmed the pod deployed successfully in the kube-system namespace on both master and worker nodes.

3. DaemonSets: Updates, Rollbacks, and Revision History [00:09:29 - 00:16:19]

• Demonstrated two update strategies for DaemonSets: OnDelete (manual pod deletion) and RollingUpdate (default, zero-downtime).
• Showed updating a DaemonSet image version using kubectl set image daemonset/<name> <container>=<new-image> (e.g., from 2.5.2 to 2.6.0).
• Verified the update by checking pod image versions with get pods -o wide and describe pod.
• Introduced kubectl rollout history daemonset/<name> to view revision history.
• Demonstrated rolling back to a previous revision using kubectl rollout undo daemonset/<name>.
• Emphasized that DaemonSet rollbacks preserve pod state and storage, similar to Deployments.

4. StatefulSets: Concepts, Ordering, and Persistent Storage [00:17:28 - 00:24:27]

• Differentiated stateful (persistent data, stable network identity) vs. stateless (ephemeral) applications.
• Explained that StatefulSets preserve: (1) stable, unique network identifiers (hostname: <statefulset-name>-<index>), (2) persistent storage via PVCs, and (3) ordered creation/deletion (0 → 1 → 2, then 2 → 1 → 0).
• Used MySQL/DB cluster as a use case: primary must start before replicas; replicas must terminate in reverse order.
• Introduced headless services (clusterIP: None) to provide stable DNS entries for each pod (e.g., mysql-0.mysql.default.svc.cluster.local).
• Explained that StatefulSets use PVCs bound to each pod, which persist even if the pod is deleted and recreated.
• Noted that StatefulSets support rolling updates and can be scaled up/down with ordered operations.

5. StatefulSets: Limitations and Lab Constraints [00:24:27 - 00:29:31]

• Clarified that a full StatefulSet lab requires dynamic storage provisioners (e.g., via kops or cloud provider), which were unavailable in the kube-adm environment.
• Demonstrated attempted StatefulSet creation, observing PVCs stuck in Pending state due to missing StorageClass.
• Emphasized conceptual understanding over execution: StatefulSets require headless services, PVCs, ordered scaling, and stable hostnames.
• Showed a complex StatefulSet YAML including: service (headless and clusterIP), StatefulSet spec, volume claims, readiness probes, and resource limits.
• Concluded that StatefulSets are essential for databases, message queues, and distributed systems requiring persistent state and ordered lifecycle management.

6. Kubernetes Security: Authentication and Authorization [00:29:50 - 00:39:58]

• Defined authentication (how users/apps log in) and authorization (what they can do).
• Covered authentication methods: client certificates, tokens, passwords, SSO (via OpenID Connect/OAuth2).
• Explained SSO federation: user logs into external identity provider (e.g., Google, LDAP), receives JWT token, which Kubernetes validates.
• Defined authorization models: ABAC (deprecated), Node, and RBAC (recommended).
• Introduced Roles (namespace-scoped) and ClusterRoles (cluster-scoped) as sets of permissions (get, list, create, delete, etc.).
• Explained RoleBindings and ClusterRoleBindings to bind users/groups/service accounts to roles.
• Demonstrated creating a Role for get pods in a namespace and binding it to a user via RoleBinding.
• Showed use of kubectl auth can-i to test permissions before granting access (e.g., auth can-i get pods --as=user1).

7. Kubernetes Security: Certificate Management and RBAC Lab [00:39:58 - 00:44:48]

• Demonstrated generating a private key and Certificate Signing Request (CSR) using OpenSSL for a new user.
• Showed approving the CSR via kubectl certificate approve <csr-name>.
• Created a kubeconfig file for the user with the signed certificate and private key.
• Created a Role and RoleBinding to grant read access to pods in a namespace.
• Used kubectl auth can-i to verify the user could get pods but not delete pods.
• Showed creation of a ClusterRole for cluster-wide access (e.g., to secrets) and binding it to a user via ClusterRoleBinding.
• Emphasized principle of least privilege and separation of concerns: roles define permissions, bindings assign them.

8. Network Policies: Ingress and Egress Control [00:49:19 - 01:00:09]

• Introduced NetworkPolicies as Kubernetes-native firewall rules for L3/L4 traffic control between pods.
• Explained default behavior: all traffic allowed within a namespace; policies are additive and must be explicitly defined.
• Demonstrated creating a NetworkPolicy to deny all ingress traffic (policyTypes: [Ingress] with empty from).
• Created a backend pod (e.g., database) and a frontend pod; confirmed default connectivity.
• Applied a NetworkPolicy to allow ingress only from pods labeled app=frontend on port 80.
• Verified connectivity: frontend pod could reach backend; other pods could not.
• Demonstrated egress policy: created a policy to block all outbound traffic from a pod.
• Used curl from within a test pod to confirm inability to reach external websites (e.g., curl web:80 failed).

9. Ingress and Ingress Controllers: External Access and Routing [01:21:58 - 01:38:58]

• Differentiated Ingress (rules) from Ingress Controller (implementation, e.g., NGINX, HAProxy).
• Compared to cloud load balancers: Ingress rules route traffic by host/path (e.g., example.com/hello → service A; example.com/api → service B).
• Deployed NGINX Ingress Controller in ingress-nginx namespace.
• Created a Deployment and ClusterIP Service for a sample app (hello-server).
• Created an Ingress resource with a path-based rule (/hello → hello-server:80).
• Modified the Ingress Controller’s Service type from LoadBalancer to NodePort to expose it via worker node IPs.
• Accessed the app via http://<worker-node-IP>:<nodeport>/hello and confirmed successful routing.
• Emphasized that Ingress enables multiple services to share a single public IP via host/path routing.

10. Monitoring: Prometheus and Grafana [01:41:23 - 01:46:06]

• Introduced Prometheus as the standard open-source monitoring tool for Kubernetes.
• Explained Prometheus architecture: exporters (on nodes/pods) scrape metrics → store in time-series database → exposed via HTTP API.
• Described Grafana as a visualization layer: connects to Prometheus as a data source to create dashboards.
• Mentioned that Prometheus requires configuration via prometheus.yml to define scrape targets and intervals.
• Noted that Grafana plugins allow integration with Prometheus and other data sources.
• Briefly mentioned Helm as a package manager to simplify deployment of Prometheus, Grafana, and other tools.
• Concluded that monitoring requires both data collection (Prometheus) and visualization (Grafana).

Appendix

Key Principles

• DaemonSets: Ensure pod presence on every (or selected) node; auto-recover on failure; support rolling updates.
• StatefulSets: Required for stateful apps (databases); provide stable network IDs, ordered scaling, and persistent storage via PVCs.
• RBAC: Use Roles/ClusterRoles to define permissions; use RoleBindings/ClusterRoleBindings to assign them to users/groups/service accounts.
• NetworkPolicies: Default-deny; explicitly allow traffic using labels, namespaces, ports, and protocols.
• Ingress: Decouples routing rules from infrastructure; requires an Ingress Controller (e.g., NGINX) to implement rules.
• Monitoring: Use Prometheus for scraping and storage; use Grafana for dashboards; avoid built-in metrics server for production.

Tools Used

• kubectl (get, describe, apply, edit, set image, rollout, auth can-i)
• OpenSSL (certificate generation)
• NGINX Ingress Controller
• Prometheus (metrics collection)
• Grafana (visualization)
• Helm (package management, mentioned)

Common Pitfalls

• Forgetting to apply Role/ClusterRole YAML before binding.
• Using private IPs instead of public IPs when accessing NodePort services externally.
• Misconfiguring node selectors or taints/tolerations, leading to pods not scheduling.
• Assuming StatefulSets work without a dynamic StorageClass (common in kube-adm).
• Not labeling pods correctly in NetworkPolicies, causing traffic to be blocked.
• Confusing Ingress (rules) with Ingress Controller (implementation).

Practice Suggestions

• Recreate all DaemonSet, StatefulSet, and NetworkPolicy labs in a local cluster (Minikube, Kind).
• Generate custom certificates and bind them to users to simulate RBAC in a test environment.
• Deploy NGINX Ingress Controller and create multiple Ingress rules for different paths/hosts.
• Install Prometheus and Grafana using Helm charts and build custom dashboards.
• Experiment with taints and tolerations to simulate node specialization (e.g., GPU, SSD nodes).