Knowledge Base

Preserving for the future: Shell scripts, AoC, and more

Random troubleshooting story

While programming my new (and still as-of-yet unpublished) python program that uses ldap authentication, I ran into a weird issue. I logged in to the web app, and got an ldap error! The ldap server was refusing connections. I logged into my freeipa servers, and investigated. sudo ipactl status showed all clear:

Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

Both servers showed the same results. So I tried my ldap login again and it failed again. So this time I ran an ldapsearch command on my client, and discovered that indeed, I couldn't get an ldap connection on dns2. Host dns1 was still working. The cool interactive flask troubleshooter session in the web page told me the app had picked dns2 (from my kerberos TXT nslookups) of the available pool. So, that's why the app was failing.

So, back to dns2. I then checked the directory service directly:

sudo journalctl -n200 -u dirsrv@IPA-EXAMPLE-COM.service

The output indicated the system was out of disk space! And sure enough, my root partition / was completely filled. Apparently my SpiderOakONE config directory takes up 50GB of space. So I moved it to /home partition which had way more space, and restarted ipa.

Comments