← Back to Recording

Summary

Overview

This course session provides a comprehensive deep dive into Kubernetes security and automation, focusing on RBAC (Role-Based Access Control), service accounts, JWT token management, resource quotas, network policies, data encryption, and GitOps-driven deployment workflows. The session begins with a correction of prior MongoDB versioning issues (Migrating from v7 to v8) and transitions into hands-on lab exercises for securing Kubernetes clusters using declarative configurations. Participants learn to create and bind roles to service accounts, generate and validate API tokens via JWT.io, configure kubeconfig contexts, and enforce least-privilege access. The session concludes with a team-based lab requiring deployment of MongoDB via PSMDB operator and validation of RBAC for database access, reinforcing secure cluster operations in production-like environments.

Topic (Timeline)

1. MongoDB Version Update and Environment Context [00:00:01 - 00:02:37]

  • Clarified prior session’s error: Helm chart was using MongoDB 7 due to unpinned version; updated to MongoDB 8.0 for consistency.
  • Noted that some libraries (e.g., Next.js, FastAPI) had incomplete MongoDB 8.0 support as of November; confirmed current setup now works.
  • Instructed students to update all previous materials (slide decks, repos) to reflect MongoDB 8.0.
  • Announced focus for today: dashboards, metrics, logs, admin interfaces, JWTs, Kubernetes security, and automation.

2. Kubernetes Security Fundamentals: Access Control, Secrets, and Hardening [00:02:37 - 00:06:23]

  • Introduced multi-layered Kubernetes security model:
    • Access Control: Enforced via RBAC (Role-Based Access Control) with no deny rules; permissions are additive.
    • Roles: Namespace-scoped; must specify namespace. Used for team/namespace isolation (e.g., dev team restricted to dev namespace).
    • ClusterRoles: Cluster-wide permissions (e.g., get nodes, list pods across all namespaces).
    • RoleBindings & ClusterRoleBindings: Bind roles to subjects (users, groups, service accounts).
  • Secrets Management: Recommended use of Kubernetes Secrets or external tools (Vault, InfraPhysical — noted as increasingly adopted).
  • Data Security:
    • Encrypt data in transit using modern TLS.
    • Encrypt data at rest at volume level (block storage) and optionally at database level (may require enterprise license).
    • Application-level encryption as fallback.
  • Node Hardening: Key-only access, firewalls, disable unused ports.
  • Pod Security: Enforce Pod Security Standards (PSS) or Pod Security Policies (PSP) to prevent privileged containers.
  • API Server Security: Central access point must be authenticated and authorized via RBAC.

3. Service Accounts, Token Lifecycle, and Security Contexts [00:06:23 - 00:09:15]

  • Service accounts (SA) are used by pods to authenticate with the Kube API server.
  • Every namespace has at least one default SA; if unspecified, pod inherits default SA.
  • Service account tokens:
    • Auto-mounted as projected volume in pod.
    • Contain CA cert data via ConfigMap.
    • Default lifetime: 1 hour; auto-refreshed by kubelet.
    • Invalidated on pod deletion.
  • Security Context: Defines pod/container-level privilege controls:
    • User/Group IDs, Linux capabilities, privilege escalation, read-only root FS, AppArmor, SELinux.
  • Resource Quotas:
    • Enforced per namespace; limit CPU/memory usage per team/application.
    • Must specify requests/limits for CPU/memory if quota is enabled.
    • Quotas do not affect existing resources; require restart/reinstall to apply.
    • Can be gated via Validating Admission Policy.
    • Suggested phased rollout (e.g., next sprint) for large clusters.

4. Network Policies, Data Encryption, and Infrastructure as Code [00:09:15 - 00:14:10]

  • Network Policies:
    • Implemented by CNI plugins (e.g., Cilium); default policy allows all traffic.
    • Must be updated when container ports change — upstream teams often overlook this.
    • Emphasized documentation of port usage by upstream teams.
  • Data Encryption:
    • In-transit: Use Gateway API or legacy Ingress API with TLS.
    • At-rest:
    • Volume encryption via modern block storage.
    • Database-level encryption (key-value or value-only).
    • etcd secret encryption required (with encryption provider enabled).
  • Automation & IaC:
    • Modern clusters use Infrastructure as Code (IaC), not manual kubeadm.
    • Terraform: provisions VMs, networks.
    • Ansible: configures Kubernetes distro, CNI, CSI, CI/CD components.
  • CI/CD & GitOps:
    • Custom containers and Helm charts stored in private registries.
    • Continuous Integration (CI): automates build and push.
    • Continuous Delivery (CD) + GitOps: declarative state management via Git (e.g., Argo CD).
    • Rollback: Change version in Git → Argo auto-syncs to previous state on failure.
    • UI visibility (e.g., Argo CD dashboard) shows resource status (red “broken heart” = failure), logs, and events.

5. Hands-On Lab: RBAC Implementation with Service Account, Role, and Token [00:14:10 - 00:29:33]

  • Lab setup: Clone repo (gitlab.com/automatedk8s/digress/k8s4dbasic2advanced), pull Lesson 8 files.
  • Step 1: Start fresh Minikube single-node cluster.
  • Step 2: Apply serviceaccount.yaml → create dev-user SA.
  • Step 3: Apply role.yaml → define pod-reader role (read-only pods in namespace).
  • Step 4: Apply rolebinding.yaml → bind pod-reader role to dev-user SA.
  • Step 5: Generate JWT token via kubectl create token dev-user -n <namespace>.
  • Step 6: Validate token at jwt.io → confirm issuer, subject, namespace.
  • Step 7: Set kubeconfig context:
    • kubectl config set-credentials dev-user --token=<JWT>
    • kubectl config set-context dev-user --user=dev-user --namespace=default
    • kubectl config use-context dev-user
  • Step 8: Test RBAC:
    • kubectl get pods → ✅ works (within namespace).
    • kubectl get pods -A → ❌ forbidden (no cluster-wide access).
    • kubectl create pod nginx → ❌ forbidden (no create permission).
  • Step 9: Switch back to minikube context → verify full access (kubectl get pods -A ✅).
  • Step 10: Clean up:
    • Delete nginx pod: kubectl delete pod nginx
    • Delete context: kubectl config delete-context dev-user
    • Delete user: kubectl config delete-user dev-user
    • Verify only minikube context remains.

6. Team Exercise: PSMDB Operator, MongoDB 8, and RBAC Validation [00:29:33 - 00:33:09]

  • Objective: Deploy PSMDB operator and MongoDB 8 on a 3-node Minikube cluster.
  • Tasks:
    • Use existing values.yaml (MongoDB 8 pre-configured).
    • Install PSMDB operator and database.
    • Create service account, role, rolebinding for a custom user (e.g., mdb-reader).
    • Generate token, set context, test RBAC (read-only access to MongoDB resources).
    • Delete all resources upon completion.
  • Note on Error: kubeadm on Minikube v1.34.0 has a known integer overflow bug (32/64-bit); use v1.34.1+ in production.
  • Instructor advised: minikube stop && minikube delete to reset environment; kubeconfig will be wiped.

7. Lesson Review and Break [00:33:09 - 00:34:12]

  • Summary of key learnings:
    • Kubernetes security layers: RBAC, secrets, network policies, encryption.
    • Service account token lifecycle and JWT validation.
    • Role/RoleBinding scoping and context switching in kubeconfig.
    • GitOps (Argo CD) for declarative, auditable, rollback-capable deployments.
    • Importance of data-at-rest encryption (etcd, volumes, apps).
  • Announced next session: Lesson 9 (immediate break).

Appendix

Key Principles

  • Least Privilege: Always restrict permissions to the minimum required (e.g., pod-reader role).
  • No Deny Rules in RBAC: Permissions are additive; absence = denial.
  • Token Security: Service account tokens are short-lived, auto-refreshed, and invalidated on pod deletion.
  • GitOps = Declarative Truth: State defined in Git; system self-corrects via Argo CD or similar.
  • IaC First: Never manually provision clusters; use Terraform + Ansible for reproducibility.

Tools Used

  • Kubernetes CLI (kubectl): Core interaction tool.
  • jwt.io: JWT token decoder and validator.
  • Minikube: Local single/multi-node cluster for testing.
  • Terraform: Infrastructure provisioning (VMs, networks).
  • Ansible: Configuration management (K8s distro, CNI, CSI).
  • Argo CD: GitOps operator for declarative app deployment and rollback.
  • PSMDB Operator: Percona MongoDB operator for Kubernetes.

Common Pitfalls

  • Unpinned Helm chart versions → unexpected MongoDB version changes.
  • Upstream teams changing container ports without documentation → network policy breaks.
  • Applying resource quotas to existing pods → no effect; requires restart.
  • Using kubeadm v1.34.0 → known integer conversion bug; use v1.34.1+.
  • Forgetting to specify namespace in roles → role fails to bind.

Practice Suggestions

  • Recreate the RBAC lab with different roles (e.g., pod-writer, secret-reader).
  • Test network policies with Cilium: block all traffic except specific ports.
  • Use kubectl auth can-i to test permissions before creating resources.
  • Set up a private Helm chart repo and simulate a GitOps rollback using Argo CD.
  • Enable etcd encryption in Minikube (via --extra-config=apiserver.etcdEncryption=true).