Installing Yara on Ubuntu 10.04

Installation for YARA on Ubuntu 10.04. First you will need the PCRE development and runtime libraries.

$ sudo apt-get install libpcre3 libpcre3-dev

Now acquire the YARA source code.

$ wget http://yara-project.googlecode.com/files/yara-1.4.tar.gz
$ wget http://yara-project.googlecode.com/files/yara-python-1.4.tar.gz

Untar and configure YARA.

$ tar xvfz yara-1.4.tar.gz
$ cd yara-1.4.tar.gz
$ ./configure

If there are no errors, make the executables.

$ make
$ make check
$ sudo make install

Now add python support.

$ cd ..
$ tar xvfz yara-python-1.4.tar.gz
$ cd yara-python-1.4.tar.gz
$ python setup.py build
$ sudo python setup.py install

You should now be able to call YARA from a shell prompt.

$ yara
usage: yara [OPTION]... [RULEFILE]... FILE
options:
-t print rules tagged as and ignore the rest. Can be used more than once.
-i print rules named and ignore the rest. Can be used more than once.
-n print only not satisfied rules (negate).
-g print tags.
-m print metadata.
-s print matching strings.
-d = define external variable.
-r recursively search directories.
-f fast matching mode.
-v show version information.

Report bugs to:

Installing log2timeline on Ubuntu

Here is a script to ease installation of Kristinn Gudjonsson's Log2Timeline tool on Ubuntu hosts. Tested on Ubuntu Lucid 10.04.

############################################
# log2timeline_ubuntu_deps.sh
############################################
#!/bin/sh
sudo apt-get install libdigest-crc-perl libnetpacket-perl libparse-win32registry-perl libarchive-zip-perl libtimedate-perl libcarp-assert-perl libclass-dbi-perl libdatetime-perl libhtml-scrubber-perl libnet-pcap-perl libparams-validate-perl libimage-exiftool-perl libdbd-sqlite3-perl libdate-manip-perl libdatetime-format-strptime-perl
sudo perl -MCPAN -e 'install File::Mork'
sudo perl -MCPAN -e 'install Data::Hexify'

Verizon FIOS Faux Pas

Exemplifying the problem many folks have been having with Verizon FIOS, our router is version C of the Actiontec line. For the first two years of use, the bulk of our bandwidth was light web surfing and providng data to the VOD/Guide service for the DVR. Not very taxing on a 12Mb/s line. However, two things happened this year that started to tax the FIOS service, as provided by Verizon. First, we got an internet-ready DVD player. For this, we also signed up for NetFlix, an online movie service whose monthly subscription for online access to streaming content is less than a couple of video store rentals. We also discovered that our Wii had onboard wireless and had recently had software written for it to make it NetFlix compatible. This increased our bandwidth usage when streaming a movie by about ten fold. Unfortunately, the little NAT table in the ActionTec router just couldn't handle it and ended up crashing about every other day, requiring a manual power cycle. This just wouldn't fly, so, having spare hardware laying around and a desire to amp up the firewall capabilities of the home connection anyway, I decided to attempt to replace the Actiontec.


I followed the advice provided at DSLReports and set up a double-bridge bypass to my own firewall. This worked well for bypassing the NAT table issue... for about 20 days. After that, the Internet continued to work flawlessly, providing plenty of connections for the demands of streaming media, but the VOD/Guide services consistently failed due to the DVR's inability to pull an IP address. After a couple hours of poking at my firewall and the Actiontec, I found that the COAX (Ethernet) connection was not pulling an IP for the Actiontec, nor was it passing the bridge from the MOCA adapter for the STB. When I realized this, I looked at the configuration of the Actiontec router and saw that the COAX (Ethernet) interface was completely disabled. Now how could this be? It worked before. After re-enabling it and some power cycles, I determined that the Actiontec will only enable the device for the time it is powered on. For some reason, every so often it will self-reboot, which resets the device state to DISABLED, killing the VOD/Guide. ARGH!! So, to keep the STB working as it should, I have to attach to the Ethernet switch, manually set my IP address to the 192.168.1.0/24 net, enter the 192.168.1.1 default address, login, and reenable the COAX (Ethernet) device. What a pain, Verizon!


This also brings to light a security vulnerability in this method of double-bridging. The router automagically assigns itself this 192.168.1.1 address available from its Ethernet ports. As the Ethernet switch is bridged to the COAX (Broadband) device, this means that the router may be remotely accessible to an attack should an aggressor push spoofed RFC 1918 packets to your public IP address. They will drop off of the WAN firewall, but will the local ethernet bind to 192.168.1.1 answer? Since the Ethernet ports are set to make up the outer bridge to the WAN, this seems plausible, making for a potential hole in this setup. So much for adding security with a better firewall!

Automating PDF Analysis

This post assumes a knowledge of basic LINUX commands. For help, consult the looping section of the BASH manual (http://linux.die.net/man/1/bash).

Suggested readings:

http://isc.sans.org/diary.html?storyid=7903
http://isc.sans.org/diary.html?storyid=7906
http://isc.sans.org/diary.html?storyid=7867
http://isc.sans.org/diary.html?storyid=7984

In response to these and other posts, I think it's time to get serious about 1) shortening the time from starting analysis to the determination of 'malicious' and 2) start tackling the massive numbers of these files swarming the enterprise. Both of these techniques require essentially the same techniques described above to me implemented in repeatable ways to script and automate them.

The malicious PDFs I've analyzed have a few things in common that will make this process easier. First, they almost always contain the dropper payload they want to execute. They usually come from free (gmail, yahoo, hotmail) or weakly secured (AOL, MSN) webmail accounts. And, best of all, the encoding scheme used to protect the droppers is always the same, a 255-byte decrementing XOR key.

So, to build a body of files for analysis, you want to start isolating or collecting all of the PDFs delivered from webmail accounts. Once you have these hundred or thousands of files, you need to start ripping through them and identifying the evil ones.

Before we start, get the latest version of Didier Stevens pdf-parser.py (http://blog.didierstevens.com/programs/pdf-tools/). Now, these pdf files sometime contain duplicate object numbers, lots of unlinked objects, and blobs in the unmapped spaces of the PDF (like after the %EOF tag). So, to begin, let's assume one hundred objects and start ripping all of the encoded/flated objects from the pdf. There will be a lot of blank objects since some don't exist in the PDF. Get rid of those with a remove statement on 0-length files.

$ mkdir pdf.analysis
$ cd pdf.analysis
$ cp ../1.pdf .
$ for (( i=0; i<100;> $i.out; done
$ rm `ls -l | egrep " 0 2010-" | awk '{ print $8}'`

Now we have a collection of extracted objects. As mentioned in Bojan's ISC Diary (http://isc.sans.org/diary.html?storyid=7867), we can search for failed FlateDecodes. This may indicate an intersting PDF for follow-up and can be an easy malicious PDF indicator.

$ grep failed *
31.out: FlateDecode decompress failed
31.out: FlateDecode decompress failed
Binary file 35.out matches
52.out: FlateDecode decompress failed
52.out: FlateDecode decompress failed

The malicious PDFs contain a dropper that is encoded. We've seen simple XOR encoding before, but the nefarious folk of the world appear to have moved into rotating XOR encoding techniques. The key is either incremented or decremented by some amount for every byte processed. When the keyset rotates to the end of the 0x00-FF scale, it turns the corner and picks up at the other end. So, to deal with this, I updated a previously written multi-byte XOR script to handle 256-byte rotating XOR keys with a given offset. Pair it with a for loop to cycle through all 256 possible start keys, and the encoded blob will be decoded and discovered with a simple GREP for a known string. Here's how it works. In this example, I had already located and carved the unknown blob from the PDF capsule. However, for automation you would can pass the entire PDF file and just not worry about the other bytes that will get mulched. We're only looking to identify the EXE, not carve it at this point.

$ for ((i=0; i<256; i++)); do echo $i; perl multi-xor-v2.pl -f 1.pdf -o $i.ex_ -k "$i" -R -1; done $ grep -i KERNEL *
Binary file 0.ex_ matches

Apparently the ROTXOR key starts at 0x00 and rotates at a decrement of -1 for every byte processed. The rotation is typical for PDFs of the day, though I have also seen different start points. Now that we have our decoded blob, the rest can be disposed of.

$ mv 0.ex_ Carved_decoded_ROTXOR255_key0_step-1.exe
$ rm *.ex_
$ ls
1.pdf Carved_decoded_ROTXOR255_key0_step-1.exe multi-xor-v2.pl


The .EXE can be run through standard analysis routines to discover the call-outs and second stage drops. This PDF is definitely malicious.

So, to take it to step two, addressing the large numbers of these PDFs, just take the above steps, codify into a script, and run in another loop.

$ mkdir pdf.analysis
$ cp *.pdf pdf.analysis
$ cd pdf.analysis
$ find . -type f -name *.pdf | while read i; do echo "processing $i"; ../analyzepdf.sh "$i"; done && find . -type -f -name *.exe | while read i; do echo "MATCH: $i"; done | tee matches.txt

The above loop creates a directory for analysis, creates an array of the PDF files available to be analyzed, and initiates the analysis script for each of them. The anlaysis script will create analysis subdirectories for each PDF, perform the above analysis steps and decodings, identify the interesting tidbits, and leave behind the interesting artifacts. When the loop finishes, the FIND command is used to locate the executables left behind, and create a notification for those PDFs found to have drops, recording this information to the matches.txt file.

Now you can revisit the PDFs identified in the matches .txt file and carve the droppers out of them.

$ dd if=1.pdf of=c1.bin bs=1 skip=27598 count=834887 && xxd c1.bin | less
834887+0 records in
834887+0 records out
834887 bytes (835 kB) copied, 1.7896 s, 467 kB/s

Apply the ROTXOR decoder scripts to the blob to reveal the executable.

$ for ((i=0; i<256; i++)); do echo $i; perl multi-xor-v2.pl -f c1.bin -o $i.ex_ -k "$i" -R -1; done $ grep KERNEL *
Binary file 0.ex_ matches

$ mv 0.ex_ Carved_decoded_ROTXOR255_key0_step-1.exe
$ rm *.ex_
$ ls
c1.bin Carved_decoded_ROTXOR255_key0_step-1.exe multi-xor-v2.pl

Quick shell script to extract the contents of an image

- assuming TSK is installed, the image "image.dd" is in the local directory, and a directory "files" exists for the extracts. Change the offset and disk type to suit. This particular image was a 1GB FAT16 USB drive image.

# for i in `fls -Dr -m / -f fat -o 63 image.dd | grep -v ".Trash" | grep -v "(deleted)" | cut -f 2 -d"|"`; do mkdir files/$i; done

# for i in `fls -Fr -m / -f fat -o 63 image.dd | grep -v ".Trash" | grep -v "(deleted)" | cut -d "|" -f 2,4`; do echo $i; icat -o 63 -f fat image.dd `echo $i | cut -d "|" -f 2` > files/`echo $i | cut -d "|" -f 1`; done

Idea for enterprise scanning

Pseudocode for an internal scanner. Attempts to combat environmental manipulation through self-integrity checking, but a better mechanism may be needed.

Assumed this operates in a client/server model with the server offering messages to clients in a one-to-many or several-to-many relationship. Ulitmately, the server should be able to post a request (hash list, updated files) and the clients should pull the list, self check, perform the tests, and report back. The central system should them be able to generate reports based on the results.

INteresting reports might include which scanned successfully, which didn't report, any anomalies discovered. All hashes are passed back to central, so the tool could be used for forensic anomalies, known discovery of artifacts, discovery of similar artifacts within a defined threshhold, or compliance applications (similarity or direct matching). The insider threat model could be integrated by allowing the tracking of defined critical documents within reporting systems.

Client structure follows:

internal scanner

pull updates and signatures. check sig, decode to mem, load hashes

provide non-DOM driver to access disk filesystem - driver client bindings
provide access to memory - Volatility

check self integrity of all components, static files
Walk VAD and dump all processes/dll injects to disk
identify self in proc dump and validate hash vice known

dump registry hives in memory
extract registry values for known hostiles - regripper

for each proc/file
perform static hash scanning >> hashfile
perform context piecewise hashing >> cphfile

compile xml/soap response
encrypt, sign, report back to central


Needed tools:

http://code.google.com/p/pyssdeep/
http://www.py2exe.org/
https://www.volatilesystems.com/default/volatility
http://www.regripper.net/
http://ssdeep.sourceforge.net/
http://www.indigostar.com/perl2exe.htm

Opinion on SMTP Honeypots

honeyd is an infrastructure honeypot that refers to other services. it's a little heavy. If you are trying to emulate the Interweb on an open access point for research, it's great. For this, you want something more focused, either a honeytrapd type of service (dangerous on your border) or a full-time script running in its own process(s) to capture and handle load. Look at truman's (http://www.secureworks.com/research/tools/truman.html) smtp script and consider reversing it's interally-focused intent to external. Add some support scripts for housekeeping and you should be good to go. Obviously, run in a dmz, with limited perms, on a box that is easily rebuilt and doesn't have other dependent, critical apps/processes. VM should be fine.

On Tue, Apr 28, 2009 at 12:28 PM, private investigation <xxx> wrote:
I tried to use honeyd but seems that honeyd cannot handle much of smtp request