OpenLDAP Everywhere Reloaded, Part II

This multipart series covers how to engineer an OpenLDAP Directory Service to create a unified login for heterogeneous environments. With current software and a modern approach to server design, the aim is to reduce the number of single points of failure for the directory. In this installment, I discuss the differences between single and multi-master replication. I also describe how to configure OpenLDAP for single master replication between two servers. [See the April 2012 issue for Part I of this series or visit www.linuxjournal.com/content/openldap-everywhere-reloaded-part-i.]

Figure 1. Example redundant server pair—in Part I of the series, NTP, DNS and DHCP were configured.

On both servers, use your preferred package manager to install the slapd and ldap-utils packages if they haven't been installed already.

OpenLDAP 2.3 offered the start of a dynamic configuration back end to replace the traditional slapd.conf and schema files. This dynamic configuration engine (also known as cn=config) is now the default method in OpenLDAP 2.4 to store the slapd(8) configuration.

The benefits for using cn=config over traditional slapd.conf(5) are namely:

OpenLDAP 2.4 still can be configured through slapd.conf(5) for now; however, this functionality may be removed from a future release of OpenLDAP. If you have an existing OpenLDAP server configured via slapd.conf, now is the time to get acquainted with cn=config.

OpenLDAP 2.4 changes the terminology in regard to replication. Replication nodes no longer are referred to as either “master” or “slave”. They are instead referred to as either a “provider” (a node that provides directory updates) or a “consumer” (a node that consumes directory updates from the provider or sometimes another consumer). The change is subtle but important to note.

In addition to LDAP Sync Replication (aka Syncrepl), which uses a Single Master Replication (SMR) model, OpenLDAP 2.4 introduces new replication types, such as N-Way Multi-Master Replication.

N-Way Multi-Master Replication, as the name suggests, uses a Multi-Master Replication (MMR) model. It is akin in operation to 389 Directory Server's replication of similar name. Multiple providers can write changes to the Directory Information Tree (DIT) concurrently.

For more information on the changes in OpenLDAP 2.4, consult the OpenLDAP 2.4 Software Administrator's Guide (see Resources).

Neither replication model is better than the other per se. They both have their own benefits and drawbacks. It's really just a matter of which benefits and drawbacks are better aligned to your individual needs.

The benefit of SMR (via Syncrepl) is that it guarantees data consistency. Data will not corrupt or conflict because only one provider is allowed to make changes to the DIT. All other consumers, in effect, just make a read-only shadow copy of the DIT. Should the single provider go off-line, clients still can read from the shadow copy on the consumer.

This benefit also can be its drawback. SMR removes the single point of failure for Directory reads, but it still has the disadvantage of a single point of failure for Directory writes. If a client tries to write to the Directory when the provider is off-line, it will be unable to do so and will receive an error.

Generally speaking, this might not be a problem if the data within LDAP is very static or the outage is corrected in a relatively short amount of time. After all, a Directory by its very nature is intended to be read from far more than it ever will be written to.

But, if the provider's outage lasts for a significant amount of time, this can cause some sticky problems with account management. While the provider is unavailable, users are unable to change their expired or forgotten passwords, which might cause problems with logins. If an employee is terminated, you cannot disable that person's account in LDAP until the provider is returned to service. Additionally, employees will be unable to change address-book data (although most users would not consider this an urgent problem).

The benefit of MMR is that it removes the single point of failure for Directory writes. If one provider goes off-line, the other provider(s) still can make changes to the DIT. Those changes will be replicated back to the failed provider when it comes back on-line. However, as is the case with all high-availability clusters, this can introduce what is referred to as the “split-brain” problem.

Figure 2. An over-simplified view of the split-brain problem: replication fails between the two servers despite the local network still being available.

The split-brain problem is where neither provider has failed, but network communication between the two has been disrupted. The “right side” of the split can modify the DIT blindly without consideration of what the “left side” already had changed (and vice versa). This can cause damage or corruption to the shared data store that is supposed to be consistent between both providers.

As time goes on, the two independent copies of the DIT start to diverge further and further from each other, and they become inconsistent. When the split is repaired, there is no automagic way for either provider to know which server has the truly correct copy of the DIT. At this point, a system administrator must intervene manually to repair any divergence between the two servers.

As Directories are read from more than they are written to, you may perceive the risk of divergence during split-brain to be very low. In this case, N-Way Multi-Master Replication is a good way to remove the single point of failure for Directory writes.

On the other hand, the single point of failure for Directory writes may be only a minor nuisance if you can avoid the hassles of data inconsistency. In this case, Syncrepl is the better option.

It's all a matter of which risk you perceive to have a bigger impact on your organization. You'll need to make an assessment as to which of the two replication methods is more appropriate, then implement one or the other—but not both!

After Debian installs the slapd package, it asks you for the “Administrator” password. It preconfigures the Directory Information Tree (DIT) with a top-level namespace of dc=nodomain if getdomainname(2) was not configured locally. The RootDN becomes cn=admin,dc=nodomain, which is a Debian-ism and a departure from OpenLDAP's default of cn=Manager,$BASEDN.

dc=nodomain is not actually useful though. The Debian OpenLDAP maintainers essentially leave it up to the user to re-create a more appropriate namespace.

You can delete the dc=nodomain DIT and start again with the dpkg-reconfigure slapd command. Run this on both linux01.example.com and linux02.example.com. The reconfigure scripts for the slapd package will ask you some questions. I've provided the answers I used as an example. Of course, select more appropriate values where you see fit:

"Omit OpenLDAP server configuration" = No
"DNS domain name" = example.com
"Organisation name" = Example Corporation
"Administrator password" = linuxjournal
"Confirm Administrator password" = linuxjournal
"Database backend to use" = HDB
"Do you want the database to be removed when slapd is purged?" = No
"Move old database?" = Yes
"Allow LDAPv2 protocol?" = No

The question about “DNS domain name” has nothing to do with DNS; it is a Debian-ism. The answer supplied as a domain name will be converted to create the top-level namespace ($BASEDN) of the DIT. For example, if you intend to use dc=pixie,dc=dust as your top-level namespace, enter pixie.dust for the answer.

The questions about “Administrator password” refer to the OpenLDAP RootDN password, aka RootPW, aka olcRootPW. Here you will set the password for the cn=admin,$BASEDN account, which in this example is cn=admin,dc=example,dc=com.

If you run the slapcat(8) command, it now shows a very modest DIT, with only dc=example,dc=com and cn=admin,dc=example,dc=com populated.

OpenLDAP by default (for performance reasons) does not log a large amount information to syslog(3). You might want to increase OpenLDAP's log levels to assist the diagnosis of any replication problems that occur:

# set_olcLogLevel.ldif
#
# Run on linux01 and linux02
#
dn: cn=config
changetype: modify
replace: olcLogLevel
olcLogLevel: acl stats sync

Modify cn=config on both servers with the ldapmodify -Q -Y EXTERNAL -H ldapi:/// -f set_olcloglevel.ldif command to make this change effective.

If you have chosen to use LDAP Sync Replication (Syncrepl), the instructions below demonstrate a way to replicate dc=example,dc=com between both servers using one provider (linux01.example.com) and one consumer (linux02.example.com).

As Syncrepl is a consumer-side replication engine, it requires the consumer to bind to the provider with a security object (an account) to complete its replication operations.

To create a new security object on linux01.example.com, create a new text file called smr_create_security_object.ldif, and populate it as follows:

# smr_create_security_object.ldif
#
# Run on linux01
#
# 1. Create an OU for all replication accounts
dn: ou=Replicators,dc=example,dc=com
description: Security objects (accounts) used by
 Consumers that will replicate the DIT.
objectclass: organizationalUnit
objectclass: top
ou: Replicators

# 2. Create security object for linux02.example.com
dn: cn=linux02.example.com,ou=Replicators,dc=example,dc=com
cn: linux02.example.com
description: Security object used by linux02.example.com
 for replicating dc=example,dc=com.
objectClass: simpleSecurityObject
objectClass: organizationalRole
userPassword: {SSHA}qzhCiuIJb3NVJcKoy8uwHD8eZ+IeU5iy
# userPassword is 'linuxjournal' in encrypted form.

The encrypted password was obtained with the slappasswd -s <password> command. Use ldapadd(1) to add the security object to dc=example,dc=com:

root@linux01:~# ldapadd -x -W -H ldapi:/// \
> -D cn=admin,dc=example,dc=com \
> -f smr_create_security_object.ldif
Enter LDAP Password:
adding new entry "ou=Replicators,dc=example,dc=com"

adding new entry "cn=linux02.example.com,ou=
↪Replicators,dc=example,dc=com"

root@linux01:~#

If you encounter an error, there may be a typographical error in the LDIF file. Be careful to note lines that are broken with a single preceding space on the second line. If in doubt, see the Resources section for a copy of smr_create_security_object.ldif.

Run slapcat(8) to show the security object and the OU it's contained by.

On linux01.example.com, create a new text file called smr_set_dcexample_provider.ldif, and populate it as follows:

# smr_set_dcexample_provider.ldif
#
# Run on linux01
#
# 1. Load the Sync Provider (syncprov) Module
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov

# 2. Enable the syncprov overlay on
#    dc=example,dc=com
dn: olcOverlay=syncprov,olcDatabase={1}hdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov
olcSpCheckpoint: 100 10
olcSpSessionlog: 100
# olcSpCheckpoint (syncprov-checkpoint) every 100
#   operations or every 10 minutes, whichever is
#   first
# olcSpSessionlog (syncprov-sessionlog) maximum
#   100 session log entries

# 3.1.1. Delete the existing ACL for
#        userPassword/shadowLastChange
dn: olcDatabase={1}hdb,cn=config
changetype: modify
delete: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange
  by self write
  by anonymous auth
  by dn="cn=admin,dc=example,dc=com" write
  by * none
-
# 3.1.2. Add a new ACL to allow the replication
#        security object read access to
#        userPassword/shadowLastChange
add: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange
  by self write
  by anonymous auth
  by dn="cn=admin,dc=example,dc=com" write
  by dn="cn=linux02.example.com,ou=Replicators,dc=ex
 ↪ample,dc=com" read
  by * none
-
# 3.2. Indices can speed searches up. Though, every
#      index used, adds to slapd's memory
#      requirements
add: olcDbIndex
#
# Required indices
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
#
# Not quite required, not quite optional. The logs
# fill up without this index present
olcDbIndex: uid pres,sub,eq
#
# Optional indices
olcDbIndex: cn pres,sub,eq
olcDbIndex: displayName pres,sub,eq
olcDbIndex: givenName pres,sub,eq
olcDbIndex: mail pres,eq
olcDbIndex: sn pres,sub,eq
#
# Debian already includes an index for
# objectClass eq, which is also a requirement
-
# 3.3. Allow Replicator account limitless searches
add: olcLimits
olcLimits: dn.exact="cn=linux02.example.com,ou=Repli
 cators,dc=example,dc=com"
  time.soft=unlimited
  time.hard=unlimited
  size.soft=unlimited
  size.hard=unlimited

When this LDIF file is applied, it will tell slapd(8) to load the syncprov (Sync Provider) module and will enable the syncprov overlay on the database that contains dc=example,dc=com. It will modify Debian's default password ACL to allow the newly created security object read access (so it can replicate passwords to linux02.example.com). It also adds some required and optional indices, and removes any time and size limits for the security object (so as not to restrict it when it queries linux01.example.com).

Apply this LDIF file on linux01.example.com with ldapmodify(1) as follows:

root@linux01:~# ldapmodify -Q -Y EXTERNAL \
> -H ldapi:/// \
> -f smr_set_dcexample_provider.ldif
modifying entry "cn=module{0},cn=config"

adding new entry "olcOverlay=syncprov,olcDatabase={1}hdb,cn=config"

modifying entry "olcDatabase={1}hdb,cn=config"

root@linux01:~#

Again, if there are errors, they could be typographical errors. Be sure to note which lines in the file are broken with a preceding single space or a preceding double space. Also, be sure to note which sections are separated with a blank line and which are separated with a single dash (-) character. If in doubt, see the Resources section for a copy of smr_set_dcexample_provider.ldif.

Now, on linux02.example.com, create a text file called smr_set_dcexample_consumer.ldif, and populate it with the following:

# smr_set_dcexample_consumer.ldif
#
# Run on linux02
#
# 1.1.
dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcSyncRepl
olcSyncRepl: rid=001
  provider=ldap://linux01.example.com/
  type=refreshAndPersist
  retry="5 6 60 5 300 +"
  searchbase="dc=example,dc=com"
  schemachecking=off
  bindmethod=simple
  binddn="cn=linux02.example.com,ou=Replicators,dc=example,dc=com"
  credentials=linuxjournal
# retry every 5 seconds for 6 times (30 seconds),
#  then every 60 seconds for 5 times (5 minutes)
#  then every 300 seconds (5 minutes) thereafter
# schemachecking=off as checking gets done on
#  linux01. we do not want records received from
#  linux01 ignored because they fail the ill-
#  defined (or missing) schemas on linux02.
-
# 1.2.1. Delete the existing ACL for
#        userPassword/shadowLastChange
delete: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange
  by self write
  by anonymous auth
  by dn="cn=admin,dc=example,dc=com" write
  by * none
-
# 1.2.2. Add a new ACL which removes all write
#        access
add: olcAccess
olcAccess: {0}to attrs=userPassword,shadowLastChange
  by anonymous auth
  by * none
-
# 1.3.1. Delete the existing ACL for *
delete: olcAccess
olcAccess: {2}to *
  by self write
  by dn="cn=admin,dc=example,dc=com" write
  by * read
-
# 1.3.2. Add a new ACL for * removing all write
#        access
add: olcAccess
olcAccess: {2}to *
  by * read
-
# 1.4. Indices can speed searches up. Though, every
#      index used, adds to slapd's memory
#      requirements
add: olcDbIndex
#
# Required indices
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
#
# Not quite required, not quite optional. The logs
# fill up without this index present
olcDbIndex: uid pres,sub,eq
#
# Optional indices
olcDbIndex: cn pres,sub,eq
olcDbIndex: displayName pres,sub,eq
olcDbIndex: givenName pres,sub,eq
olcDbIndex: mail pres,eq
olcDbIndex: sn pres,sub,eq
#
# Debian already includes an index for
# objectClass eq, which is also a requirement
-
# 1.5. If a LDAP client attempts to write changes
#      on linux02, linux02 will return with a
#      referral error telling the client to direct
#      the change at linux01 instead.
add: olcUpdateRef
olcUpdateRef: ldap://linux01.example.com/
-
# 1.6.1. Rename cn=admin to cn=manager.
#        Modifications are only made by linux01
replace: olcRootDN
olcRootDN: cn=manager
-
# 1.6.2. Remove the local olcRootPW. Modifications
#        are only made on linux01
delete: olcRootPW

When this LDIF file is applied, it configures slapd(8) to use LDAP Sync Replication (olcSyncRepl) to replicate from linux01.example.com. It authenticates with the newly created security object. As this is a read-only copy of dc=example,dc=com, it replaces two existing ACLs with ones that remove all write access. It adds some required and optional indices, adds a referral URL for linux01.example.com and (in effect) cripples the RootDN on linux02.example.com (because no modifications to the DIT will occur here).

Apply smr_set_dcexample_consumer.ldif on linux02.example.com with ldapmodify(1) as follows:

root@linux02:~# ldapmodify -Q -Y EXTERNAL \
> -H ldapi:/// \
> -f smr_set_dcexample_consumer.ldif
modifying entry "olcDatabase={1}hdb,cn=config"

root@linux02:~#

Finally, on linux02.example.com, stop slapd(8), delete the database files created by the dpkg-reconfigure slapd command run earlier, and restart slapd(8). This will allow slapd(8) to regenerate the database files in light of the new configuration:

root@linux02:~# /etc/init.d/slapd stop
Stopping OpenLDAP: slapd.
root@linux02:~# rm /var/lib/ldap/*
root@linux02:~# /etc/init.d/slapd start
Starting OpenLDAP: slapd.
root@linux02:~#

To show that the replication works, you can add something to the DIT on linux01.example.com and use slapcat(8) on linux02.example.com to see if it arrives there.

Create a text file on linux01.example.com called set_dcexample_test.ldif, and populate it with some dummy records:

# set_dcexample_test.ldif
#
# Run on linux01
#
dn: ou=People,dc=example,dc=com
description: Testing dc=example,dc=com replication
objectclass: organizationalUnit
objectclass: top
ou: People

dn: ou=Soylent.Green.is,ou=People,dc=example,dc=com
description: Chuck Heston would be proud
objectclass: organizationalUnit
ou: Soylent.Green.is

Use ldapadd(1) to add the entries to the DIT:

root@linux01:~# ldapadd -x -W -H ldapi:/// \
> -D cn=admin,dc=example,dc=com \
> -f set_dcexample_test.ldif
Enter LDAP Password:
adding new entry "ou=People,dc=example,dc=com"

adding new entry "ou=Soylent.Green.is,ou=People,dc=e
xample,dc=com"

root@linux01:~#

On linux02.example.com, use slapcat(8) to see that the records are present:

root@linux02:~# slapcat | grep -i soylent
dn: ou=Soylent.Green.is,ou=People,dc=example,dc=com
ou: Soylent.Green.is
root@linux02:~#

On linux01.example.com, create a new text file called unset_dcexample_test.txt, and populate it as follows:

ou=Soylent.Green.is,ou=People,dc=example,dc=com
ou=People,dc=example,dc=com

Use the command ldapdelete -x -W -H ldapi:/// -D cn=admin,dc=example,dc=com -f unset_dcexample_test.txt to delete the test entries.

Once replication is working properly between the two servers, you should remove the change to the logging level (olcLogLevel) performed earlier, so that queries to LDAP do not affect server performance.

On both linux01.example.com and linux02.example.com create a text file called unset_olcLogLevel.ldif, and populate it as follows:

# unset_olcLogLevel.ldif
#
# Run on linux01 and linux02
#
dn: cn=config
changetype: modify
delete: olcLogLevel

Then, use it to remove olcLogLevel with the ldapmodify -Q -Y EXTERNAL -H ldapi:/// -f unset_olcLogLevel.ldif command.

Also, configure the LDAP clients to point at the LDAP servers. Modify /etc/ldap/ldap.conf on both servers, and add the following two lines:

BASE   dc=example,dc=com
URI    ldap://linux01.example.com/ ldap://linux02.example.com/

If you opted for MMR, use the above two lines for /etc/ldap/ldap.conf on linux01.example.com only. On linux02.example.com, use the following two lines instead:

BASE   dc=example,dc=com
URI    ldap://linux02.example.com/ ldap://linux01.example.com/

I'll continue this in Part III of this series, where I describe how to configure the two OpenLDAP servers to replicate using N-Way Multi-Master Replication instead.