Automating PDF Analysis

This post assumes a knowledge of basic LINUX commands. For help, consult the looping section of the BASH manual (http://linux.die.net/man/1/bash).

Suggested readings:

http://isc.sans.org/diary.html?storyid=7903
http://isc.sans.org/diary.html?storyid=7906
http://isc.sans.org/diary.html?storyid=7867
http://isc.sans.org/diary.html?storyid=7984

In response to these and other posts, I think it's time to get serious about 1) shortening the time from starting analysis to the determination of 'malicious' and 2) start tackling the massive numbers of these files swarming the enterprise. Both of these techniques require essentially the same techniques described above to me implemented in repeatable ways to script and automate them.

The malicious PDFs I've analyzed have a few things in common that will make this process easier. First, they almost always contain the dropper payload they want to execute. They usually come from free (gmail, yahoo, hotmail) or weakly secured (AOL, MSN) webmail accounts. And, best of all, the encoding scheme used to protect the droppers is always the same, a 255-byte decrementing XOR key.

So, to build a body of files for analysis, you want to start isolating or collecting all of the PDFs delivered from webmail accounts. Once you have these hundred or thousands of files, you need to start ripping through them and identifying the evil ones.

Before we start, get the latest version of Didier Stevens pdf-parser.py (http://blog.didierstevens.com/programs/pdf-tools/). Now, these pdf files sometime contain duplicate object numbers, lots of unlinked objects, and blobs in the unmapped spaces of the PDF (like after the %EOF tag). So, to begin, let's assume one hundred objects and start ripping all of the encoded/flated objects from the pdf. There will be a lot of blank objects since some don't exist in the PDF. Get rid of those with a remove statement on 0-length files.

$ mkdir pdf.analysis
$ cd pdf.analysis
$ cp ../1.pdf .
$ for (( i=0; i<100;> $i.out; done
$ rm `ls -l | egrep " 0 2010-" | awk '{ print $8}'`

Now we have a collection of extracted objects. As mentioned in Bojan's ISC Diary (http://isc.sans.org/diary.html?storyid=7867), we can search for failed FlateDecodes. This may indicate an intersting PDF for follow-up and can be an easy malicious PDF indicator.

$ grep failed *
31.out: FlateDecode decompress failed
31.out: FlateDecode decompress failed
Binary file 35.out matches
52.out: FlateDecode decompress failed
52.out: FlateDecode decompress failed

The malicious PDFs contain a dropper that is encoded. We've seen simple XOR encoding before, but the nefarious folk of the world appear to have moved into rotating XOR encoding techniques. The key is either incremented or decremented by some amount for every byte processed. When the keyset rotates to the end of the 0x00-FF scale, it turns the corner and picks up at the other end. So, to deal with this, I updated a previously written multi-byte XOR script to handle 256-byte rotating XOR keys with a given offset. Pair it with a for loop to cycle through all 256 possible start keys, and the encoded blob will be decoded and discovered with a simple GREP for a known string. Here's how it works. In this example, I had already located and carved the unknown blob from the PDF capsule. However, for automation you would can pass the entire PDF file and just not worry about the other bytes that will get mulched. We're only looking to identify the EXE, not carve it at this point.

$ for ((i=0; i<256; i++)); do echo $i; perl multi-xor-v2.pl -f 1.pdf -o $i.ex_ -k "$i" -R -1; done $ grep -i KERNEL *
Binary file 0.ex_ matches

Apparently the ROTXOR key starts at 0x00 and rotates at a decrement of -1 for every byte processed. The rotation is typical for PDFs of the day, though I have also seen different start points. Now that we have our decoded blob, the rest can be disposed of.

$ mv 0.ex_ Carved_decoded_ROTXOR255_key0_step-1.exe
$ rm *.ex_
$ ls
1.pdf Carved_decoded_ROTXOR255_key0_step-1.exe multi-xor-v2.pl


The .EXE can be run through standard analysis routines to discover the call-outs and second stage drops. This PDF is definitely malicious.

So, to take it to step two, addressing the large numbers of these PDFs, just take the above steps, codify into a script, and run in another loop.

$ mkdir pdf.analysis
$ cp *.pdf pdf.analysis
$ cd pdf.analysis
$ find . -type f -name *.pdf | while read i; do echo "processing $i"; ../analyzepdf.sh "$i"; done && find . -type -f -name *.exe | while read i; do echo "MATCH: $i"; done | tee matches.txt

The above loop creates a directory for analysis, creates an array of the PDF files available to be analyzed, and initiates the analysis script for each of them. The anlaysis script will create analysis subdirectories for each PDF, perform the above analysis steps and decodings, identify the interesting tidbits, and leave behind the interesting artifacts. When the loop finishes, the FIND command is used to locate the executables left behind, and create a notification for those PDFs found to have drops, recording this information to the matches.txt file.

Now you can revisit the PDFs identified in the matches .txt file and carve the droppers out of them.

$ dd if=1.pdf of=c1.bin bs=1 skip=27598 count=834887 && xxd c1.bin | less
834887+0 records in
834887+0 records out
834887 bytes (835 kB) copied, 1.7896 s, 467 kB/s

Apply the ROTXOR decoder scripts to the blob to reveal the executable.

$ for ((i=0; i<256; i++)); do echo $i; perl multi-xor-v2.pl -f c1.bin -o $i.ex_ -k "$i" -R -1; done $ grep KERNEL *
Binary file 0.ex_ matches

$ mv 0.ex_ Carved_decoded_ROTXOR255_key0_step-1.exe
$ rm *.ex_
$ ls
c1.bin Carved_decoded_ROTXOR255_key0_step-1.exe multi-xor-v2.pl

Quick shell script to extract the contents of an image

- assuming TSK is installed, the image "image.dd" is in the local directory, and a directory "files" exists for the extracts. Change the offset and disk type to suit. This particular image was a 1GB FAT16 USB drive image.

# for i in `fls -Dr -m / -f fat -o 63 image.dd | grep -v ".Trash" | grep -v "(deleted)" | cut -f 2 -d"|"`; do mkdir files/$i; done

# for i in `fls -Fr -m / -f fat -o 63 image.dd | grep -v ".Trash" | grep -v "(deleted)" | cut -d "|" -f 2,4`; do echo $i; icat -o 63 -f fat image.dd `echo $i | cut -d "|" -f 2` > files/`echo $i | cut -d "|" -f 1`; done

Idea for enterprise scanning

Pseudocode for an internal scanner. Attempts to combat environmental manipulation through self-integrity checking, but a better mechanism may be needed.

Assumed this operates in a client/server model with the server offering messages to clients in a one-to-many or several-to-many relationship. Ulitmately, the server should be able to post a request (hash list, updated files) and the clients should pull the list, self check, perform the tests, and report back. The central system should them be able to generate reports based on the results.

INteresting reports might include which scanned successfully, which didn't report, any anomalies discovered. All hashes are passed back to central, so the tool could be used for forensic anomalies, known discovery of artifacts, discovery of similar artifacts within a defined threshhold, or compliance applications (similarity or direct matching). The insider threat model could be integrated by allowing the tracking of defined critical documents within reporting systems.

Client structure follows:

internal scanner

pull updates and signatures. check sig, decode to mem, load hashes

provide non-DOM driver to access disk filesystem - driver client bindings
provide access to memory - Volatility

check self integrity of all components, static files
Walk VAD and dump all processes/dll injects to disk
identify self in proc dump and validate hash vice known

dump registry hives in memory
extract registry values for known hostiles - regripper

for each proc/file
perform static hash scanning >> hashfile
perform context piecewise hashing >> cphfile

compile xml/soap response
encrypt, sign, report back to central


Needed tools:

http://code.google.com/p/pyssdeep/
http://www.py2exe.org/
https://www.volatilesystems.com/default/volatility
http://www.regripper.net/
http://ssdeep.sourceforge.net/
http://www.indigostar.com/perl2exe.htm

Opinion on SMTP Honeypots

honeyd is an infrastructure honeypot that refers to other services. it's a little heavy. If you are trying to emulate the Interweb on an open access point for research, it's great. For this, you want something more focused, either a honeytrapd type of service (dangerous on your border) or a full-time script running in its own process(s) to capture and handle load. Look at truman's (http://www.secureworks.com/research/tools/truman.html) smtp script and consider reversing it's interally-focused intent to external. Add some support scripts for housekeeping and you should be good to go. Obviously, run in a dmz, with limited perms, on a box that is easily rebuilt and doesn't have other dependent, critical apps/processes. VM should be fine.

On Tue, Apr 28, 2009 at 12:28 PM, private investigation <xxx> wrote:
I tried to use honeyd but seems that honeyd cannot handle much of smtp request

So You Thought You Were in Control of Your Friend List

Check this tool out! Facebook Controller:

http://my.opera.com/quakerdoomer/blog/2009/04/30/fbcontroller-facebook-controller-the-ultimate-facebook-controller-without-the-pa

Nice, using social media against you to subvert authentication controls, do recon, and manipulate data. Great POC!

SSH Command Monitoring

This was an interesting post from the secureshell list. Thanks Richard!


Hi "J",

you can do that with your unix/linux onboard tools. just attach strace
to the sshd process of the user you want to monitor:

strace -s 4096 -e trace=read -p PROCESS_ID

than have a look for the shell prompt (e.g.):

read(10, "\33]0;USERNAME@HOSTNAME:~\7".

.., 16384) = 22

now you know that the FD (file handle) is 10 for the users ssh session terminal.

then you can do something like that:

strace -s 4096 -e trace=read -p 10417 2>&1 | grep -E '^read\(10,' |
grep -oE '".+"'

and you should get an output like:

"uname -a"
"\r\n"
"Linux HOSTNAME 2.6.29.1 #1 SMP Sat Apr 18 11:22:05 CEST 2009 i686
Intel(R) Core(TM)2 Duo CPU L7500 @ 1.60GHz GenuineIntel GNU/Linux\r\n"
"\33]0;USERNAME@HOSTNAME:~\7"


well, this will only work if you have root permission on the server
running sshd.


have fun,
richard

Open Source Security Information Manager - OSSIM

I've been playing with OSSIM (http://www.ossim.net) for the last week. The stand-alone installation from AlienVault was trivially easy, thanks guys! I was able to install a main 'trusted net' stand-alone and integrate a DMZ sub-sensor in an hour. The dashboard is pretty, with many reporting features and does a decent job of aggregating the infeed of data from the wide collection of tools it provides.

But...

Having put this Unified Threat Management (http://en.wikipedia.org/wiki/Unified_Threat_Management) device on the network, I find it to be the least secure thing out there. While AlienVault did an excellent job of bringing all of these wonderful security monitoring tools together, having the production interface on the main network acting as both collector, sensor, and admin access is a bad idea. It also uses so many products and services that it is terribly insecure itself. Having the main sensor in the trusted network isn't too bad for this, but having one of these in the more exposed DMZ makes me wary. OSSIM needs a lot of custom configuration to implement restricted access, split the collection interface to a promiscuous-only and have a separate admin interface. This can be done with taps to ensure only one-way traffic occurs, but that still leaves the service open to injection if one were to expect the box to be there. In all, the UTM sensor-with-everything idea needs to be rethought.

It's been fun to play with, but ultimately I'm going to explore running with some functionalized implementations that might prove more secure.