Random troubleshooting story
While programming my new (and still as-of-yet unpublished) python program that uses ldap authentication, I ran into a weird issue. I logged in to the web app, and got an ldap error! The ldap server was refusing connections. I logged into my freeipa servers, and investigated. sudo ipactl status
showed all clear:
Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa: INFO: The ipactl command was successful
Both servers showed the same results. So I tried my ldap login again and it failed again. So this time I ran an ldapsearch command on my client, and discovered that indeed, I couldn't get an ldap connection on dns2. Host dns1 was still working. The cool interactive flask troubleshooter session in the web page told me the app had picked dns2 (from my kerberos TXT nslookups) of the available pool. So, that's why the app was failing.
So, back to dns2. I then checked the directory service directly:
sudo journalctl -n200 -u dirsrv@IPA-EXAMPLE-COM.service
The output indicated the system was out of disk space! And sure enough, my root partition / was completely filled. Apparently my SpiderOakONE config directory takes up 50GB of space. So I moved it to /home partition which had way more space, and restarted ipa.
Comments