I originally posted this on my blog for my 100 Hour Board identity, but I'll copy it here since it's kind of interesting. It is rather long though.

So, all the readers of the 100 Hour Board should know by now (assuming you check your email account) that the Board was attacked and compromised recently.

I was waiting for all the disclosure emails to go out before posting. So, now we can discuss what happened.

On Thursday November 13, 2008 I was asked by an editor to make a code change on part of the site. Well, wasn't I surprised that when I tried to login to the server the account password had been changed. So this was when the fun began. I used an alternative access method to get onto the machine with root privileges and regain control of the normal user account. Then, of course, it was time to start digging to find out what "they" had been up to, where they came from, and how they got in.

I quickly discovered from the log files that the attackers were clever enough to gain access to the machine, however apparently incredibly stupid at the same time. So, there's a good chance it was mostly a script-kiddie attack. They remembered to clear the bash_history for the normal user, but (having accessed the root account using "sudo su"; I've revoked sudo privileges now, since I'm the only one that should be executing commands as root, and I have the root password to use that account when needed) forgot to clear root's bash_history from their last session. So I got to see some of their stupid in action, line by line. For example they list out a directory and then try to change directories into a regular file. Or run "cat" on a compressed file (so you get a bunch of garbage sent to the screen). They were, however, properly paranoid, checking for other logins several times during what appeared to be a short session.

After locking down the user accounts to prevent further shell access I went digging through more logs to find out who they were. So, while they were smart enough to erase their bash_history (most of it) they were also dumb enough to leave behind all of their auth_logs (the ones that say when people log in and from where) as well as all the of the apache logs from their attacks. So it was trivial to discover that the attackers were hitting the machine from locations in Ukraine and Russia (of course these were just their last bounces before coming to our machine and they could be coming through any number of points before that).

Based on the Apache logs they found the Board by doing a Google search looking for any ".edu" addresses that contain "id" in the URL anywhere and are running PHP. So, they appeared to be trolling for sites that might be vulnerable to SQL Injection attacks. My first thought was that they exploited a vulnerability in the Board software and executed an SQL Injection; but further analysis of the events suggest that was not the case. The particular part of the Board they attempted to attack was actually immune. Which, of course, is to suggest that other areas were less immune. I say "were" because I've been spending many waking hours of my life the last couple of weeks doing a line-by-line code review patching up security holes. It's been a tedious and tiring process, but overall the Board is much safer now.

So, what were they doing with the machine they had gained access to? Hosting spam. They weren't sending spam from our machine, which is nice, fewer headaches for us to deal with (being blacklisted, etc). But they had been hosting a Cialis/Viagra spam website in a directory and they embedded a hidden link on our site to promote search engine traffic for that page. They also installed several web interface back-doors to the system, at least 5 if I recall correctly. They were not very smart about hiding those, however, since a PHP file in an "Images" directory sticks out pretty obviously. So I cleaned out all of that garbage as well. Initial reviews of the database suggest that no malicious content has been injected there either.

So, all in all we weren't hit too badly. And had they not been arrogant enough to change the account password there's a good chance I wouldn't know about it still.



The fun, from my end, begins again when they find out that they no longer have access to the machine. Keeping a close eye on the machine they discovered the next day that their shell access was no longer valid. Then in the Apache logs they quickly discovered that all of their web-interface back-doors were missing as well. (This actually helps me because all the ones they checked were ones I had removed, which suggests I got them all.) Then they found that their spam site was no longer running, and that their hidden link on our site was gone. Now the crucial step was about to happen, they would check their next point of access to see if we had completely locked down the machine.

Much to my surprise they quickly and easily accessed the database via our phpMyAdmin interface. This was to my surprise because I had also changed the MySQL account passwords to prevent this very thing. I was afraid they had used some exploit in the phpMyAdmin code, but doing some very fast digging and checking I was sad/happy to discover that I was at fault. In my haste to get our new machine running I had apparently forgotten to remove one of the default "nobody" accounts in MySQL. I now have found that Ubuntu MySQL installations come with a script "mysql_secure_installation" which steps you through a script to make MySQL secure by removing anonymous accounts, setting the root password, deleting default databases, etc. I highly recommend it, it's apparently not well known since none of the Ubuntu MySQL Installation Tutorials I've read through have ever mentioned it.

So, now MySQL is locked down tight, and to prevent further issues of any kind access to phpMyAdmin has been restricted to on-campus only. It appears that they either found out about this change on November 20, or not at all yet. That depends on if they have a machine available to launch attacks from inside an insurance company in Canada which tried to access phpMyAdmin.

It's now been a couple days since I was writing this post, so I've lost my train of thought. So, I think I'll just end it here. We've cleaned out the machine and everything seems to be pretty good. There doesn't appear to by any XSS attacks running or malicious content hiding in the database, but if you see anything suspicious please send us an email so we can check into it.

[This Entry] [0 Comments]

I haven't posted anything interesting this month. Mostly because there hasn't been anything interesting that I want to write about that I didn't write about on my other blog and I don't particularly want to just copy posts.

Last night I stayed up later than I should have to finish reading The Man Who Mistook his Wife for a Hat. It's a collection of stories about patients that Oliver Sacks encountered during his years as a practicing neurologist (I think he was a neurologist anyways). It's rather interesting. There are a lot of weird ways that the human brain can malfunction.

The first story, where the title comes from, is about a man who loses all brain function pertaining to faces. He doesn't see faces as faces anymore, nor does he have any memory of what faces are, or that he can't see them. The title comes from his visit to Dr. Sacks when he was preparing to leave and began looking around for his hat. His wife was standing near him and he reached over and tried to pick up her head as his hat to put on so he could leave.

The book contains several other random stories. People with Korsakov's syndrome (like the guy in Memento if you've seen it), people who have phantom pains and how phantom limbs make using prosthetics easier, a pair of twin savants who can't perform simple arithmetic but "see" numbers instantly when a collection is presented to them and who communicate with each other in a type of game where they some come up with prime numbers and tell them to each other and each really enjoys the process (Prime numbers up to 25 digits long!). The same twins can instantly tell you what day of the week any date within the next 10,000 or previous 10,000 years will/did fall. Same thing for calculating the date of Easter for that time period. Fascinating.

So, it's an interesting little read.

I will have a rather interesting post to read about soon, but it will be lengthy and will require more work to compose.

[This Entry] [0 Comments]

In case anyone is wondering, there is a really easy way to secure a MySQL installation (at least on Ubuntu). After installing mysql just run the command "mysql_secure_installation" which will automatically allow you to setup the passwords, delete anonymous accounts, remove default DBs and Tables, etc. I wish I had known about it earlier.

[This Entry] [0 Comments]

I wanted to do this again the other day, and had to dig through my old code to figure out how I had done it, so I thought I'd put it here so I can find it more easily in the future, and so anyone can find it too.

So, say you're running through a loop and you want to output some basic information about that loop without scrolling through a million lines on the console output. You're using Python, of course, and the standard "print" function automatically adds a newline to your input (unless you end it with a comma, but you're still stuck at the end of the line).

You can write to the stdout stream instead though and be rather tricky:
sys.stdout.write("\rLoop Iteration:" + str(i) + "/" + str(max_iterations))
sys.stdout.flush()

On Linux the "\r" will return the cursor to the beginning of the line, so you can overwrite the entire line with the new information without moving to the next line on the console. The flush() command is needed so that the information you just put into the buffer gets written out (which normally happens when a newline is encountered).

Enjoy.

[This Entry] [0 Comments]