So it looks like I’m running out of memory on our ldap machine, not sure why but the kernel’s oom-killer then takes out slapd, the ldap server daemon. This can leave, and in this morning’s case, the backend database in a dirty state and thus unable to load.
Checking configuration files for slapd: bdb_db_open: unclean shutdown detected; attempting recovery.
bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered.
Luckily I wasn’t the only one who had this problem and found a solution in this blog post.
[root@ldap]# /usr/sbin/slapd_db_recover -v -h /var/lib/ldap
Finding last valid log LSN: file: 1 offset 5315883
Recovery starting from 
Recovery complete at Sun Jul 30 11:31:56 2006
Maximum transaction ID 8000040d Recovery checkpoint