389 Directory Server for Administrators (AU) - andres-8e2b-20251030-072838

Overview

This course provides an in-depth technical training on Red Hat 389 Directory Server, covering core architecture, replication topologies, plugin configurations, security, backup strategies, migration from OpenLDAP, and automation using Ansible. The session is structured as a hands-on lab-based lecture, guiding participants through the design, deployment, and management of LDAP directory services in enterprise environments. Key topics include schema management, multi-supplier and cascading replication, SSSD integration, TLS configuration, and operational best practices for high availability and disaster recovery.

Topic (Timeline)

1. Introduction & Voice Recording Setup [00:03:26.830 - 00:27:50.670]

• Instructor provides guidance on optimal voice recording practices for course materials: speak clearly, avoid background noise, maintain natural cadence.
• Mentions completion of voice recording and cessation of recording session.
• Brief, non-technical segment unrelated to core curriculum.

2. Travel Context & Casual Discussion [00:27:50.670 - 00:30:31.970]

• Informal conversation about travel to Australia, specifically Cairns and the Great Barrier Reef.
• Mentions use of WeWork spaces in Sydney and reference to a New Zealand-based system (A&C).
• Transition to technical content via casual remarks.

3. Review of Yesterday’s LDAP Architecture Fundamentals [00:30:31.970 - 00:37:41.800]

• Recap of core LDAP concepts from prior session: schema, LDIF, and directory structure.
• Explanation of 389 Directory Server architecture:
  • Core components: ns-slapd (LDAP server daemon).
  • Plugin system: handles operations (bind, search, modify), enforces policies, manages relationships.
  • Backend databases: legacy BDB vs. modern LMDB (technical preview, recommended for performance).
• Discussion of replication fundamentals: how data flows between servers via replication agreements.

4. Directory Server Plugins Deep Dive [00:37:41.800 - 00:45:26.340]

• Detailed review of enabled plugins from prior lab:
  • Unique attributes, numeric assignment (DNA), memberOf, referential integrity, retroactive plugin.
• Plugin classification by execution timing:
  • Pre-operation (e.g., password policy, ACL validation).
  • Post-operation (e.g., logging).
  • In-transaction (e.g., DNA, memberOf, uniqueness).
• Plugin types: syntax validation, account policy, ACLs, aliases, attribute uniqueness, auto membership, DNA, referral integrity, linked attributes, pass-through (IPA, PAM), POSIX (deprecated post v12.6).
• Emphasis on plugin roles in enforcing data integrity and access control.

5. SSSD Architecture and Authentication Framework [00:45:30.280 - 00:50:45.810]

• Introduction to SSSD (System Security Services Daemon):
  • Centralized authentication/authorization framework for Linux clients.
  • Integrates with LDAP, Active Directory, Kerberos, and cloud providers (e.g., Azure via OAuth).
• Components:
  • Responder modules (PAM, NSS, sudo, autofs).
  • Controller process handling authentication requests.
  • Local cache for offline access (critical for resilience during backend outages).
• Discussion of cache implications: potential stale data, need for manual cache flush.
• Role of PAM (Pluggable Authentication Modules) as the low-level authentication interface.
• NSS (Name Service Switch) as the configuration map for user/group data sources (files, SSSD, Winbind).

6. Directory Replication: Principles and Topologies [00:50:45.810 - 01:44:03.970]

• Purpose of replication: high availability, load balancing, reduced latency, local data control.
• Key concepts:
  • Supplier (read-write master) vs. Consumer (read-only replica).
  • Replication unit = suffix (database), not partial subtree.
• Topologies covered:
  • Single Supplier-Consumer: Simple one-way replication.
  • Cascading (Hub-and-Spoke): Hub server acts as intermediary to reduce load on primary supplier; improves performance and reduces WAN traffic.
  • Multi-Supplier Replication (MSR): Multiple read-write masters; enables concurrent updates.
  • Full Mesh: All servers replicate with all others; high fault tolerance but complex (20 servers → ~380 agreements).
  • Partial Mesh: Reduced connections for manageability (e.g., ring or star patterns).
• Replication agreement configuration: unique server IDs, replication traffic, conflict resolution (last-write-wins).
• Best practices: Red Hat recommends ≤4 suppliers; avoid full mesh unless critical redundancy required.

7. Replication Tuning, Performance, and Resource Impact [01:44:03.970 - 02:34:58.390]

• Latency optimization in WAN environments:
  • releaseTimeout (default 60s): prevents supplier monopolization of consumer.
  • replicationLogin + busyWaitTime + sessionPauseTime: mitigate replication lockouts.
  • connectionTimeout: hard limit on replication session duration.
• Resource consumption:
  • CPU: Each replication agreement spawns threads; high agreement count → context switching overhead.
  • I/O: Change log writes + database updates → high disk I/O.
  • File descriptors: Each connection consumes FDs; must be tuned in OS limits.
• Change log management:
  • changelogMaxEntries: must exceed total directory entries.
  • changelogMaxAge: auto-purge old entries (in hours).
  • replicaTombstoneAge: delay deletion cleanup to ensure propagation to all replicas.

8. SSSD and Certificate Integration in IPA/389DS [02:34:58.390 - 02:46:08.410]

• SSSD integration with 389DS: uses certificates for secure replication (TLS).
• IPA topology example: domain suffix replication + certificate replication.
• Geographic topology: 4 data centers, 4 replicas per DC, 2 inter-DC replication links for redundancy.
• Emphasis on failover design: ensure connectivity survives single-point failures.

9. Replication Lab Setup and Troubleshooting [02:46:08.410 - 06:04:53.870]

• Lab instructions: configure 4 instances (server1–server4) as suppliers/consumers.
• Key steps:
  • Create identical suffixes on supplier and consumer.
  • Enable replication on consumer first (create replication account).
  • Create replication agreement on supplier pointing to consumer.
• Common issues:
  • DNS resolution failure → use /etc/hosts entries.
  • Firewall blocking ports 389/636.
  • Incorrect server ID or suffix mismatch.
• Use dsconf and dsctl for configuration and diagnostics.
• Force replication via disable/enable cycle.
• Use nmap to verify port accessibility.

10. Migration from OpenLDAP to 389DS [06:04:55.230 - 06:42:45.900]

• Migration strategy: offline, not live.
• Tool: openldap2ds (best-effort, not guaranteed).
• Key challenges:
  • Schema incompatibility (object classes, must/may attributes).
  • Password hash differences (e.g., SSHA vs. PBKDF2).
  • Missing overlays (e.g., memberOf as plugin vs. overlay).
• Best practices:
  • Clean data: remove unused service accounts, orphaned entries.
  • Export each suffix separately.
  • Dry-run first to generate migration checklist.
  • Rebuild indexes post-migration for performance.
  • Test applications for behavior consistency.
• Real-world case: Only used for Unix authentication (PAM, sudo, POSIX groups); applications use Active Directory directly → simplified migration.

11. Admin Tasks: ACIs, Password Policies, SSSD, TLS [06:42:45.900 - 06:51:19.340]

• Admin tasks covered:
  • ACI (Access Control Instructions): define fine-grained permissions.
  • Global password policy: enforce complexity, expiration, history.
  • Disable anonymous binds.
  • SSSD configuration: authenticate Linux clients against 389DS.
  • TLS certificate management:
  • Use internal CA (not Let’s Encrypt).
  • Import CA, server cert, private key.
  • Enable LDAPS (636) and start TLS.
• Automation: Ansible playbooks for certificate deployment.

12. Ansible Automation for 389DS [06:51:19.340 - 07:06:56.500]

• Two Ansible automation options:
  1. Red Hat community collection (outdated, basic): deploy instances, backups, basic config.
  2. Advanced role-based repo (beta, more functional): supports plugins, TLS, replication, change log tuning.
• Playbook capabilities:
  • Deploy instances, configure logins, enable plugins, manage TLS.
  • Replication roles: configure suppliers, consumers, agreements, certificates.
• No built-in health-check playbooks.
• Backup automation: use dsctl backup (online) or db2bak (offline); combine for redundancy.
• Recommend storing backups on NFS or external storage; automate via cron or Ansible.

13. Backup Strategies and Operational Best Practices [07:06:56.500 - 07:20:44.130]

• Backup methods:
  • Online backup: dsctl backup — fast, but resets changelog → requires re-initializing replicas.
  • Offline backup: db2bak — requires stopping instance; preserves changelog.
• Enterprise practice: daily online + weekly offline backups.
• Avoid VM snapshots for BDB databases (risk of corruption); safe for LMDB.
• Current customer practice: nightly slapcat dump during maintenance window (2–3 AM).
• Two masters → no downtime during backup.

14. Migration, Upgrades, and Patch Management [07:20:44.130 - 07:30:56.340]

• Upgrade path:
  • 389DS v12 → v13: major OS-level changes expected; no immediate urgency.
  • Patching: minor dot releases (e.g., 12.5 → 12.6) are safe in-place.
  • Avoid upgrading if using Windows Sync (deprecated post v12.6).
• Patch management:
  • Use Red Hat Satellite or custom repos to control package versions.
  • Freeze 389DS packages until tested in Dev/QA.
  • Avoid automatic updates in production.
• LMDB recommendation: use despite being “technical preview” — superior performance and self-healing vs. BDB.
• Migration tool exists (bdb2lmdb) for future conversion if needed.

15. Real-World Deployments and Q&A [07:30:56.340 - 07:35:20.870]

• Customer examples:
  • 7 locations, 5k users → 3 masters (geographically distributed), 5 consumers.
  • University with 400k historical student records → replication performance challenges.
• Final Q&A:
  • No health-check Ansible playbooks available.
  • Cockpit provides detailed SNMP-based monitoring stats.
  • Certificate automation via Vault or internal CA.
  • Migration success hinges on data cleansing and schema alignment.
• Closing remarks: appreciation for participation, course materials and certificates to follow.

Appendix

Key Principles

• Replication Unit: Suffix (database), not subtree.
• Supplier vs. Consumer: Only suppliers can write; consumers refer writes.
• Conflict Resolution: Last-write-wins based on timestamp.
• SSSD: Centralizes authentication across Linux services (PAM, NSS, sudo).
• LMDB: Preferred backend over BDB — faster, self-healing, scalable.
• Change Log: Critical for replication; must be sized to hold all directory entries.

Tools Used

• dsctl: Command-line tool for instance management.
• dsconf: Configuration tool for replication, plugins, ACIs.
• slapcat: Export LDIF from directory.
• db2bak: Offline backup tool.
• openldap2ds: Migration tool (best-effort).
• bdb2lmdb: Backend migration tool.
• Ansible roles for deployment and replication automation.
• Cockpit: Built-in monitoring dashboard.

Common Pitfalls

• DNS resolution failure between servers → use /etc/hosts.
• Firewall blocking LDAP ports (389/636).
• Mismatched suffixes between supplier and consumer.
• Using BDB without proper repair tools → prefer LMDB.
• Online backup resets changelog → requires re-initializing replicas.
• Full mesh topology → excessive replication agreements → avoid beyond 4 suppliers.
• Password hash incompatibility during migration → force password reset or enable legacy hash support.

Practice Suggestions

• Clone VMs before configuring replication to enable safe experimentation.
• Test replication topologies in lab before production.
• Always validate schema compatibility before migration.
• Use Ansible for repeatable, auditable deployments.
• Monitor changelog size and set purge policies.
• Conduct dry-run migrations with openldap2ds before production cutover.
• Combine online and offline backups for resilience.