I had an issue with one of my servers going down repeatedly. This is very bad news for someone like me who makes a living as a web developer. Hosting websites and keeping my pages up and my clients happy is something that is very important to me, and when a server goes down, my clients are not happy and I am not happy.

Server and the Issue

I took a look at the server and the issue was that all of the available hard drive space had been used up. I deleted a bunch of files and cleared up several gigabytes of space. A couple of days later the server went down again. Same issue.

I also got an email regarding spam emails being sent from one of my IP addresses. I went looking for the largest files and directories on my server and discovered that my mail logs had been going crazy. I discovered then that my server had been compromised to send spam email.

Even worse.

I set about looking to discover what I could. I searched google because I know that sometimes hackers like to post about their accomplishments, sometimes hackers work in teams with other hackers and leave their chat logs open and searchable to the internet. I took a look through google and discovered a conversation showing how they had gotten into my server. I discovered that I had stupidly left an old account on that I had created for a customer that had the same username and password based on a dictionary word. I realized that I was an idiot.

I searched around and did what I could. Deleted the mail queues. Deleted the logs. Deleted extra users. Deleted the insecure user. Yet still my server continued sending email. I looked through the crontabs for all users and couldn’t find any processes or scripts. I then asked for help.

I found a very skilled programmer on twitter named @wh1zzz0 and approached him for help. He helped me go through and secure my server and then also showed me how to search for rootkits. A rootkit is something that a hacker can leave on a server for him to gain access later even if you’ve changed your passwords.

He told me about rkhunter (short for rootkit hunter), which is a piece of software that allows you to search your server for rootkits.

I downloaded and installed rootkit hunter from sourceforge: http://rkhunter.sourceforge.net/

Installing and running this searched for hundreds of commonly used rootkits and helped me discover the source of my problems.

I hope that my mistakes may help someone else learn and protect their server in the future.