cyberwart

This morning the Apache website was compromised. They gave a quick update on the situation in the following post:

On August 27th, starting at about 18:00 UTC an account used for automated backups for the ApacheCon website hosted on a 3rd party hosting provider was used to upload files to minotaur.apache.org.  The account was accessed using SSH key authentication from this host.

<snip>

The attackers created several files in the directory containing files for www.apache.org, including several CGI scripts.  These files were then rsynced to our production webservers by automated processes.  At about 07:00 on August 28 2009 the attackers accessed these CGI scripts over HTTP, which spawned processes on our production web services.

At about 07:45 UTC we noticed these rogue processes on eos.apache.org, the Solaris 10 machine that normally serves our websites.

Within the next 10 minutes we decided to shutdown all machines involved as a precaution.

This type of attack is getting more press recently. Website for several members of the US House were compromised recently as reported here in the Washington Post. Additionally, many security professionals were made keenly aware of the risks of shared hosting.

These attacks are interesting for several reasons. The most obvious is that they are very high profile targets. Hacking important sites gets attention. Second, it’s a common assumption that 3rd party hosting is safe (or at least mitigates risk by moving it to a 3rd party). Most security professionals accept that all websites will be eventually compromised so move them elsewhere. The problem is that most organizations cannot maintain the necessary discipline to keep everything important off their 3rd party hosted sites. When you consider that constituents and clients require access to various documents and web apps, it’s a considerable problem. Additionally as web2.0 becomes more popular and the even more invasive “cloud computing” becomes a standard business process, organizations will face further security problems. Thus while some risk is shifted by 3rd party hosting, some isn’t and security complexities increase.

The reason that this subject is particularly interesting to me is that 3rd party hosts are almost always excluded from penetration tests. Most organizations have to exclude 3rd party hosts as they’re generally shared and acquiring permission is virtually impossible. That of course leads to numerous problems.

I’d make a few suggestions:

1. Push assessment of 3rd party hosting as far as possible. The line between ethical and out of bounds is blurry, but it’s a line pen testers are familiar with. If you’re working with a reputable firm that you trust, let them know they can investigate your corporate resources hosted on 3rd parties (careful tho that you can only authorize assessing your resources)
2. In every network (external/internal) pen test, include web apps as in scope. This shouldn’t replace regular, more comprehensive web app assessments, but including web apps in the RoE can enable an attacker to leverage realistic attack vectors — including third party hosted web apps.
3. Have the pen testers go ahead and buy an account on the hosting provider. Generally web hosting is cheap. For less than $50, they can get an account with the same provider and determine what risks might be of interest.

I’m obviously not a lawyer, so please don’t take my word for what is legal or contractually acceptable. But it’s the job of pen testers to realistically portray the threat of real-world attackers. To do this we need to adapt techniques we see in use.

Recently, I’ve been thinking about the role of incident response. This isn’t typically an area that I directly deal with, but is often related to most of my work. I’ve spent untold hours writing network sensor software, and even more trying to avoid detection during pen tests. I’ve even been the handler on a few incidents, but it’s not a direct activity in my daily work.

That being said, I’m surprised with the documentation that’s available on the topic. Most books are at least 5 years old. Various websites have quick briefs (much like this one) and there are a few government funded guides there are at least 100 pages. I’m writing this post as general guidance and to link some of the better information I found.

First, as every guide states, have a plan! This is tremendously important. An incident response without a plan is likely to result in massive problems. Why is the plan so important? A smooth response is the easy answer. The more accurate is that when an incident occurs decision makers are going to be afraid. They’ll make poor decisions or want to meet with several executives. This is simply not practical during a time sensitive event.

Many guides will detail the various hardware, skill sets, and other random things you might need, etc. I’m not going get into that. There are a few basics that need to be established at the policy level and then the team should implement a detailed plan. The high level requirements are:

1. Guidance on declaring an incident
2. How to appoint an Incident Manager (IM)
3. Authorities and budget of the IM
4. The business objectives  the IM

Officially declaring an incident seems like an overly formal event. But business processes are going to change, hours are going to be burned, and someone needs to get in control. Therefore there needs to be an official process for this happening. This process can simply be an email decision by the CIO/CISO/CEO/whoever but it needs to happen. It also needs to happen in time. Responding to an incident after sys admins have started making furious changes is only going to complicate the situation.

Appointing the Incident Manager is key. Various guides will call this person the Incident Handler. I believe IM is a better term as this person’s key responsibility is coordinating technical staff, executives, and communication. This person needs to be solid under pressure, senior enough to lead a team and to communicate with executives. Technical abilities should be significant enough to understand the technology and attacks, but this person doesn’t need to reverse a themidia packed custom binary. The IM should immediately select a technical lead/deputy that can handle the technical details. While the IM is meeting with the CIO someone needs to make sure the technical work is getting done!

As I said, authorities change during an incident. The most basic change is that all IT decisions now go through the IM. An incident cannot be handled if forensic staff are trying to image a drive that sys admins are trying to repair, or if web developers are fixing the SQL injection on the production host. I’ve seen both. Fighting the technical details such as these or arguing with an IT/operations manager just wastes time. The other common problem, is the fight with change management. High availability organizations likely have a week or longer review and testing process for change management. If the executives want faster change, the IM needs authority to do so and the business needs to understand the associated risks.

Everything needs to be tied back to business goals. Contrary to what geeks think, myself included, the organization does not exist for the network. There are business needs that need to be foremost in mind. Some businesses will want to fully investigate an incident, press charges, sue, etc. Others will have complex legal reporting requirements. Most will simply want the incident fixed as quickly as possible. Knowing what the organization wants and when is key. The job of the IM is take the current situation and move it best available business objective, which implies the business objectives are known.

More detailed planning should define the standard operating procedures for how different types of incidents should be handled. These should begin with high level business objectives and classes of attacks. In my opinion these are important but less so. The particulars of an incident tend to make sticking to a detailed plan difficult. Keeping these guides flexible is important. I find it best to notate the tasks that need to be accomplished and important checkpoints required to ensure every one is on the same page and that stake holders are informed. The guides should also keep important contact information centralized.

The exception to flexible planning is the execution of common tasks. These common tasks occur as part of large incidents or as the whole response to low level incidents, meaning a less formal more common incident requiring a response but not full escalation. Every team should be able to rapidly:

1. Block an external IP address at the gateway
2. Monitor/log all traffic for a particular IP addresses anywhere in the network for analysis
3. Redirect an internal or external host to an untrusted subnet/sandbox
4. Block an internal hosts traffic by IP, MAC Address, and wall jack
5. Deploy a new IDS/IPS signature
6. Deploy patches/software to a server or workstation

The six above items are core tasks any incident response team should be able to execute in minutes. Obviously one wouldn’t want to decide to push software out in a few minutes, but once the decision is made software should begin moving out as quickly as possible.

Any organization building the basic policy and operating procedure guides is well on its way to effectively responding to incidents. The next step is to develop an efficient process to quickly execute the common six activities above. The two elephants in the room are: detecting the incident and forensics/reverse-engineering. These items require extreme specialization. Detection is about monitoring your network and hosts, establishing a baseline, and observing malicious traffic. In my experience breaking into systems this is an incredibility hard task. As bad as these products can be, the only real answer for an enterprise is to use them, love them, and tweak them. Reverse-engineering is easier — just go find a guy that owns a license to IDA Pro and launches into a rant when comparing Windbg with Olly/Imdbg.

If you look at recent posts, I’m sure you can figure out where I stand on the last item.

Useful Links:

• http://csrc.nist.gov/publications/nistpubs/800-61/sp800-61.pdf
• http://www.cert.org/csirts/Creating-A-CSIRT.html
• http://en.wikipedia.org/wiki/Computer_security_incident_management
• http://www.sei.cmu.edu/pub/documents/03.reports/pdf/03hb002.pdf

Here’s a link over to Adam’s site where he links a quick report we did on some malware. The malware itself isn’t especially interesting. Conficker definitely blows it out of the water. What is interesting is that it appears to be a custom implant used to target a particular financial organization. This isn’t a bot that happened to get into a network, this is someone that wrote software and is carefully maintaining access to the network.

#!/usr/bin/env python
'''
Matthew Wollenweber
mjw@cyberwart.com

watchfunc just run somethign temporarily

'''
DESC = "just a temporary script to do some task"
USAGE = "!watchfunc 0xDEADBEEF"

import immlib
from immlib import LogBpHook
#import time
import struct
import unicodedata
import getopt

def main(args):
    imm = immlib.Debugger()
    load_hook = MyLoadHook()

    addr = long(args[0], 16)
    load_hook.add("I'm watching you", addr, 0, 0, 0)
    imm.Log("watching hook set")

    return "Watching the function"

def usage():
    print USAGE

if __name__=="__main__":
    print "This module is for use within Immunity Debugger only"

class MyLoadHook(LogBpHook):
    def __init__(self):
        LogBpHook.__init__(self)
        self.imm = immlib.Debugger()

    def run(self,regs):
        imm = self.imm
        r = imm.getRegs()
        eax = r['EAX']
        ecx = r['ECX']
        esi = r['ESI']
        edi = r['EDI']
        esp = r['ESP']
        calledfrom = imm.callStack()[0].calledfrom

        args =[]
        arg_s = ""

        try:
            for x in range(1, 16):
                tmp = imm.callStack()[x].procedure
                if tmp.find(" = ") >= 0:
                    args.append(tmp.strip())
                else:
                    break
            raise Exception('secret', '42')

        except:
            for x in args:
                arg_s += x + " "

        imm.Log("[WATCHFUNC]: Called from: 0x%08x %s eax=0x%08x ecx=0x%08x esi=0x%08x edi=0x%08x esp=0x%08x" % (calledfrom, arg_s, eax, ecx, esi, edi, esp))

        return 

#!/usr/bin/env python
'''
Matthew Wollenweber
mjw@cyberwart.com
Wait for x iterations of function y

'''
DESC = "Skip through x iterations of function y"
USAGE = "!wait function_addr count"

import immlib
from immlib import LogBpHook
#import time
import struct
import unicodedata
import getopt

def main(args):
    imm = immlib.Debugger()
    load_hook = MyLoadHook()

    addr = long(args[0], 16)
    count = int(args[1])

    imm.addKnowledge("wait_count", count)
    imm.addKnowledge("curr_count", 0)

    load_hook.add("waiting hook", addr, 0, 0, 0)
    imm.Log("waiting hook set")

    return "waiting hook set"

def usage():
    print USAGE

if __name__=="__main__":
    print "This module is for use within Immunity Debugger only"

class MyLoadHook(LogBpHook):
    def __init__(self):
        LogBpHook.__init__(self)
        self.imm = immlib.Debugger()

    def run(self,regs):
        imm = self.imm
        curr_count = imm.getKnowledge("curr_count")
        wait_count = imm.getKnowledge("wait_count")
        #imm.Log("curr count %i"  % (curr_count))
        if curr_count == wait_count:
            imm.setBreakpoint(long(imm.getKnowledge("wait_bp")))
            imm.Pause()
        else:
            curr_count += 1
            imm.forgetKnowledge("curr_count")
            imm.addKnowledge("curr_count", curr_count)

        return 

Note: updated 8/21 to account for more generic function arguments in !watchfunc

Here’s a bit of code for Immunity Debugger. It hooks LoadLibraryA and LoadLibraryW
to stalk a module until it’s loaded. Once it is, it adds breakpoints the user provides
inside that module

#!/usr/bin/env python
'''
Matthew Wollenweber
August 6 2009
mjw@cyberwart.com

I want to debug modules that are dynamically loaded at runtime and then unloaded.
It's a pain to set andreset breakpoints. This script watches for the module to be loaded.
When the module is loaded breakpoints are set. 

-r implies the use of a relative base for the target module
'''
DESC = "Set breakpoints when a specific library is loaded"
USAGE = "!stalkmod <-r> modname  ... "

import immlib
from immlib import LogBpHook
#import time
import struct
import unicodedata
import getopt

def main(args):
    imm = immlib.Debugger()
    imm.Log("starting modstalker")
    breakpoints = []
    module_name = ""

    try:
        opts, extra = getopt.getopt(args, "r")

        for flag, val in opts:
            if flag == "-r":
                imm.addKnowledge("getbase", 1)

        module_name = extra[0]
        for x in range(1, len(extra)):
            breakpoints.append(long(extra[x], 16))
    except:
        usage()
        imm.Log("ERROR in modstalker")
        return "ERROR. Exiting module"

    allmodules=imm.getAllModules()
    for x in allmodules:
        if x.getName().find(module_name) >= 0:
            for b in breakpoints:
                imm.setBreakpoint(b)
            imm.Log("Module Already loaded. Breakpoints set")
            return "Module already loaded. Breakpoints set"

    imm.addKnowledge("breakpoints", breakpoints)
    imm.addKnowledge("modname", module_name)

    load_hook = MyLoadHook()

    load_addr = imm.getAddress("kernel32.LoadLibraryW")
    load_hook.add("bp on loadlibraryW", load_addr, 0, 0, 0)
    load_addr = imm.getAddress("kernel32.LoadLibraryA")
    load_hook.add("bp on loadlibraryW", load_addr, 0, 0, 0)
    imm.Log("Placed hook on loadlibraryA and LoadLibraryW")

    return "Hooks loaded"

class MyLoadHook(LogBpHook):
    def __init__(self):
        LogBpHook.__init__(self)
        self.imm = immlib.Debugger()
        self.found = 0

    def run(self,regs):
        imm = self.imm
        ucode = 0
        getbase = 0
        baseoffset = long(0)

        modname = imm.getKnowledge("modname")
        getbase = imm.getKnowledge("getbase")

        regs = imm.getRegs()
        ptr = imm.readMemory(regs['ESP']+0x4, 0x4)
        ptr = int(struct.unpack("L", ptr)[0])
        filename = imm.readString(ptr)

        #fuck unicode
        if len(filename) < 3:
            ucode = 1
            tmp = imm.readWString(ptr)
            filename = ""
            x = 0
            while x <  len(tmp):
                filename += str(tmp[x])
                x+=2

        filename = filename.lower()

        #imm.Log("I'm in: " + filename)
        #imm.Log("I'm stalking: " + modname)

        if filename.find(modname) >= 0:
            imm.Log("MATCHED. Setting breakpoints")
            if ucode == 0:
                imm.setBreakpoint(0x7c801DAB) #End of LoadLibraryA
            else:
                imm.setBreakpoint(0x7c80AEEB) #End of LoadLibraryW

            imm.Run() #let it run until the end of loadlibrary so that our memory is mapped
            #time.sleep(2)
            if getbase == 1:
                allmodules=imm.getAllModules()
                for x in allmodules:
                    if x.getName().find(modname) >= 0:
                        baseoffset = x.getBase()
                        break

            bps = imm.getKnowledge("breakpoints")
            for x in bps:
                x = x + baseoffset
                imm.Log("set BP: %08x" % (x))
                imm.setBreakpoint(x)

            imm.Log("breakpoints set")
            self.found = 1
            self.UnHook()

        return 

def usage():
    print USAGE

if __name__=="__main__":
    print "This module is for use within Immunity Debugger only"

Thanks to Justin at Immunity for providing guidance. FYI it would appear simpler to use LoadDLL hooks — good luck with that.

At Blackhat I took Offensive Security’s Advanced Windows Exploitation (AWE) class. The class isn’t exactly what I expected but it was definitely good. Below is a brief summary of the pros and cons:

Pros:

1. All exploits were entirely real-world. There were no sample applications. Everything we exploited was a real application and it was fairly recent.
2. The class was basically exploiting one piece of software after another.
3. I learned a lot of shellcode — which has been a weakness

Cons:

1. The class largely focused on bypassing MS Windows protections. We spent a fair amount of time bypassing DEP, but we didn’t go over SafeSEH/GS. Personally, all three mitigations are about the same difficulty to bypass IMO so I didn’t understand why covering some and not others.
2. The workbook wasn’t a real-work book. It didn’t match the slides and it wasn’t exactly meant to supplement them either. It was more a solutions manual. I found this frustrating as I tend to learn by reading so I was disappointed that there wasn’t a way to read through the course.
3. The class largely used Windbg. I guess this is just a pet-peeve but I can’t stand windbg. Oddly they’d switch back and forth to Immunity or Ollydbg for various tasks.
4. They focused far too much on shellcode. I generally consider shellcode a solved problem. Sure there are  cases where you need something special but overall Metasploit does well enough for me.
5. I really wanted to do more heap fun. I know it can take a long time to do modern heap exploits and maybe it isn’t feasible for a class but a guy can dream.
6. I’d have liked to consider more generic cases — everyone says generic exploitation is dead, but sometimes it’s not. It would have been interesting to discuss what is and isn’t generically exploitable.

I’m a critic so please don’t compare the pro/con counts. I enjoyed the class and the instructor was top notch. The assistant was a little overly dramatic about a the difficulty of a simple task, but it was 4 days and lots of exploits. In my case, I didn’t learn a lot but the practice was very good. I also improved my shellcode, which seemed to be a big thing for Ryu. If exploiting real software isn’t something you do very often, I’d definitely recommend the class.

One of the most exciting talks at Blackhat was Immunity’s CLOUDBURST talk. Basically a few months back a bunch of VMware bugs popped. Immunity people being insane decided to attempt to exploit them and was successful. This is incredibly cool because many organizations are using VMware as a core network component. Additionally researchers use VMware to test exploits and malware. This type of bug is therefore very serious.

For me, this bug is particularly meaningful. At my last company I had an arguement with a VP about deploying a client portal, black testing (exploits/malware), and development all on one physical box separated by “virtual networks”. I thought this was insane — especially for a security consulting company. I argued, unsuccessfully, that a “virtual firewall” is just more software that could be broken and offers no defense-in-depth.

Within a few weeks the CLOUDBURST info was hinted at. At Blackhat details were published and yeah, they can pop out of a client onto the host running ESX. Awesome.