This morning the Apache website was compromised. They gave a quick update on the situation in the following post:
On August 27th, starting at about 18:00 UTC an account used for automated backups for the ApacheCon website hosted on a 3rd party hosting provider was used to upload files to minotaur.apache.org. The account was accessed using SSH key authentication from this host.
<snip>
The attackers created several files in the directory containing files for www.apache.org, including several CGI scripts. These files were then rsynced to our production webservers by automated processes. At about 07:00 on August 28 2009 the attackers accessed these CGI scripts over HTTP, which spawned processes on our production web services.
At about 07:45 UTC we noticed these rogue processes on eos.apache.org, the Solaris 10 machine that normally serves our websites.
Within the next 10 minutes we decided to shutdown all machines involved as a precaution.
This type of attack is getting more press recently. Website for several members of the US House were compromised recently as reported here in the Washington Post. Additionally, many security professionals were made keenly aware of the risks of shared hosting.
These attacks are interesting for several reasons. The most obvious is that they are very high profile targets. Hacking important sites gets attention. Second, it’s a common assumption that 3rd party hosting is safe (or at least mitigates risk by moving it to a 3rd party). Most security professionals accept that all websites will be eventually compromised so move them elsewhere. The problem is that most organizations cannot maintain the necessary discipline to keep everything important off their 3rd party hosted sites. When you consider that constituents and clients require access to various documents and web apps, it’s a considerable problem. Additionally as web2.0 becomes more popular and the even more invasive “cloud computing” becomes a standard business process, organizations will face further security problems. Thus while some risk is shifted by 3rd party hosting, some isn’t and security complexities increase.
The reason that this subject is particularly interesting to me is that 3rd party hosts are almost always excluded from penetration tests. Most organizations have to exclude 3rd party hosts as they’re generally shared and acquiring permission is virtually impossible. That of course leads to numerous problems.
I’d make a few suggestions:
- Push assessment of 3rd party hosting as far as possible. The line between ethical and out of bounds is blurry, but it’s a line pen testers are familiar with. If you’re working with a reputable firm that you trust, let them know they can investigate your corporate resources hosted on 3rd parties (careful tho that you can only authorize assessing your resources)
- In every network (external/internal) pen test, include web apps as in scope. This shouldn’t replace regular, more comprehensive web app assessments, but including web apps in the RoE can enable an attacker to leverage realistic attack vectors — including third party hosted web apps.
- Have the pen testers go ahead and buy an account on the hosting provider. Generally web hosting is cheap. For less than $50, they can get an account with the same provider and determine what risks might be of interest.
I’m obviously not a lawyer, so please don’t take my word for what is legal or contractually acceptable. But it’s the job of pen testers to realistically portray the threat of real-world attackers. To do this we need to adapt techniques we see in use.