About a year ago, I started playing around with the PacketProtector (www.packetprotector.org/) load on a Linksys WRTSL54GS. I had intended it to be a replacement for my existing firewall, but it never got past acceptance testing as it crashed out after a while. After a lull and a VLAN misconfiguration that locked me out from access to the LAN ports, I decided to look into the error.
To recover the unit, I needed to see what at what point it was failing and try to regain control of it. Since PacketProtector is derived from OpenWRT, I perused their forums for answers. I purchased a USB TTL controller (http://www.amazon.com/gp/product/B006JKNWLE/ref=oh_details_o01_s00_i00), as recommended by the OpenWRT recovery article (http://wiki.openwrt.org/doc/hardware/port.serial) and the hardware specific article (http://wiki.openwrt.org/toh/linksys/wrtsl54gs?s[]=wrtsl54gs). When allowed to run for a while, this error shows up on the console, ad nauseum:
__alloc_pages: 0-order allocation failed (gfp=0x20/0)
It seemed to be indicative of either a USB OHCI error, or a memory error from Google searches. The following article had the most relevant data (https://packetprotector.org/forum/viewtopic.php?id=4362). Other Openwrt forum posts indicate that usb ohci failures are possible. However, most of the posts pointed to a memory leak that consumes everything. To test this, I set the unit to run for a while with periodic process and 'free' assessments to watch how the memory flows out. If it was leaking, I wanted to determine which service it is. The analysis script follows:
# cat /packetprotector/memanal.sh
#!/bin/sh
while [[ 1 == 1 ]]; do
echo;
echo "####################";
ps | sort | uniq;
date;
free;
sleep 60;
done
When run from the TTL console at /dev/ttyUSB0 under minicom with console capture engaged, it provided a usable log from which to determine how long expected runtime was, as well as what the process is hogging up the memory. Here was the last log before failure:
Sat Jan 1 00:54:36 UTC 2000
total used free shared buffers
Mem: 30512 28284 2228 0 316
Swap: 65524 38804 26720
Total: 96036 67088 28948
####################
1 root 136 S init
2 root SW [keventd]
3 root RWN [ksoftirqd_CPU0]
4 root DW [kswapd]
5 root SW [bdflush]
6 root SW [kupdated]
9 root SW [mtdblockd]
54 root SWN [jffs2_gcd_mtd4]
67 root 136 S logger -s -p 6 -t
69 root 152 S /bin/ash --login
78 root 124 S /sbin/syslogd -C16 -m 0
81 root 112 S /sbin/klogd
118 root 92 S /sbin/hotplug2 --override --persistent --max-children
222 root 168 S udhcpc -t 0 -i eth1 -b -p /var/run/eth1.pid -R
255 root SW [khubd]
376 root SW [usb-storage-0]
377 root SW [scsi_eh_0]
641 root 244 S crond -c /etc/crontabs
646 root 96 S /usr/sbin/dropbear -p 22
663 nobody 108 S /usr/sbin/dnsmasq --dhcp-range=lan,192.168.1.100,192.
713 root 100 S mini_httpd -C /etc/mini_httpd/mini_httpd.conf
731 root 160 S /packetprotector/usr/sbin/snort-inline -Q -D -c /pack
786 root 208 S /packetprotector/usr/sbin/openvpn /packetprotector/et
799 nobody 124 S /packetprotector/usr/sbin/tinyproxy -c /packetprotect
800 nobody 100 S /packetprotector/usr/sbin/tinyproxy -c /packetprotect
801 nobody 100 S /packetprotector/usr/sbin/tinyproxy -c /packetprotect
802 nobody 100 S /packetprotector/usr/sbin/tinyproxy -c /packetprotect
803 nobody 100 S /packetprotector/usr/sbin/tinyproxy -c /packetprotect
804 nobody 100 S /packetprotector/usr/sbin/tinyproxy -c /packetprotect
805 nobody 100 S /packetprotector/usr/sbin/tinyproxy -c /packetprotect
806 nobody 100 S /packetprotector/usr/sbin/tinyproxy -c /packetprotect
807 nobody 100 S /packetprotector/usr/sbin/tinyproxy -c /packetprotect
808 root 120 S /packetprotector/usr/sbin/dansguardian -c /packetprot
809 root 116 S /packetprotector/usr/sbin/dansguardian -c /packetprot
810 root 116 S /packetprotector/usr/sbin/dansguardian -c /packetprot
811 root 116 S /packetprotector/usr/sbin/dansguardian -c /packetprot
812 root 120 S /packetprotector/usr/sbin/dansguardian -c /packetprot
813 root 120 S /packetprotector/usr/sbin/dansguardian -c /packetprot
814 root 120 S /packetprotector/usr/sbin/dansguardian -c /packetprot
816 root 120 S /packetprotector/usr/sbin/dansguardian -c /packetprot
817 root 120 S /packetprotector/usr/sbin/dansguardian -c /packetprot
818 root 120 S /packetprotector/usr/sbin/dansguardian -c /packetprot
819 root 120 S /packetprotector/usr/sbin/dansguardian -c /packetprot
836 root 120 S /bin/sh /packetprotector/usr/sbin/snort_check
839 root 120 S /bin/sh /packetprotector/usr/sbin/snort_check
840 root 248 R top -n1
841 root 112 S grep snort
842 root 112 S grep 731
843 root 136 S /bin/sh /packetprotector/usr/sbin/snort_check
844 root 124 S /bin/sh /packetprotector/usr/sbin/snort_check
848 root 120 S /bin/sh /packetprotector/usr/sbin/snort_check
851 root 120 S /bin/sh /packetprotector/usr/sbin/snort_check
852 root 260 R top -n1
853 root 120 S grep snort
854 root 112 S grep 731
855 root 136 S /bin/sh /packetprotector/usr/sbin/snort_check
856 root 124 S /bin/sh /packetprotector/usr/sbin/snort_check
860 root 120 S /bin/sh /packetprotector/usr/sbin/snort_check
863 root 120 S /bin/sh /packetprotector/usr/sbin/snort_check
[...] x 1798 times
1852 root 196 D top -n1
1853 root 92 S grep snort
1854 root 80 S grep 731
1855 root 132 S /bin/sh /packetprotector/usr/sbin/snort_check
1856 root 120 S /bin/sh /packetprotector/usr/sbin/snort_check
1861 root 116 S /bin/sh /packetprotector/usr/sbin/snort_check
1863 root 116 S /bin/sh /packetprotector/usr/sbin/snort_check
1864 root 164 D top -n1
1865 root 104 S grep snort
1866 root 80 S grep 731
1867 root 132 S /bin/sh /packetprotector/usr/sbin/snort_check
1868 root 120 S /bin/sh /packetprotector/usr/sbin/snort_check
1870 root 104 S /bin/sh -c /packetprotector/usr/sbin/proxylogrotate.s
1873 root 116 S /bin/sh /packetprotector/usr/sbin/snort_check
1875 root 124 D /bin/sh /packetprotector/usr/sbin/proxylogrotate.sh
1877 root 104 S /bin/sh -c /packetprotector/usr/sbin/watchdog > /dev/
1878 root 112 D /bin/sh /packetprotector/usr/sbin/watchdog
1879 root /bin/sh /packetprotector/usr/sbin/watchdog
1880 root 116 D /bin/sh /packetprotector/usr/sbin/snort_check
The failure happened at 55 minutes after boot. This was interesting, as the swap was being consumed by prior runs at about one megabyte per minute. Something caused it to die before it hit the end of swap, possibly due to overallocation of processing resources. However, the memory leak could clearly be traced back to the many 'snort_check' scripts that are not dying as expected. A copy of that script follows:
$ cat /packetprotector/usr/sbin/snort_check
#!/bin/sh
#
# Changed snort_check 12/22/09 - CMH714
#
#
# Set High that snort shouldnt go over
# Default should be something very high like 140
# The memory snort eats is relative to the rules enabled.
HI_PCT="130"
echo "hi pct - " $HI_PCT
pid=`pidof snort-inline`
echo "pid - " "$pid"
if [ -n "$pid" ] ; then
SNORTPCT=`top -n1 | grep snort | grep $pid | cut -d% -f1 | awk '{print $NF}'`
fi
echo "snortpct - " "$SNORTPCT"
SNORT_INLINE=`grep "snort-inline=" /etc/packetprotector.conf | cut -d "=" -f 2`
if [ -n "$SNORTPCT" ] && [ -n "$pid" ] ; then
echo "Snorts running - Checking its Memory Usage"
if [ "$SNORTPCT" -gt "$HI_PCT" ] && [ -n "$pid" ] ; then
EPOCHTIME=`date +%s`
LOGDIR="/packetprotector/logs"
mv $LOGDIR/snort-inline/alert $LOGDIR/snort-inline/alert.$EPOCHTIME && touch $LOGDIR/snort-inline/alert
# Create a log file, assumes directory exists already
echo "snort needs to restart - killing snort" > /packetprotector/logs/snort-restart.log
echo `date` >> /packetprotector/logs/snort-restart.log
# killall snort-inline 2> /dev/null
killall -1 snort-inline 2> /dev/null
# restart snort - dont need to do this as watchdog will get it next time around
# /packetprotector/etc/init.d/S11snort-inline
fi
elif ([ -n "$SNORTPCT" ] && [ -z "$pid" ]) || ([ -z "$SNORTPCT" ] && [ -n "$pid" ]) ; then
echo "Snort may be going up or down. Will check again next minute..."
elif [ -z "$SNORTPCT" ] && [ -z "$pid" ] && [ "X$SNORT_INLINE" = "X1" ] ; then
echo "restarting snort"
echo "restarting snort" >> /packetprotector/logs/snort-restart.log
echo `date` >> /packetprotector/logs/snort-restart.log
/packetprotector/etc/init.d/S11snort-inline
else
return
fi
After review, it became clear that the script wouldn't reach conclusion and close on its own. Consider the following line:
SNORTPCT=`top -n1 | grep snort | grep $pid | cut -d% -f1 | awk '{print $NF}'`
The TOP command was called without the '-b' option to put it into batch mode. This kept the TOP process in memory, constantly updating and consuming memory as a running process, keeping the script and its following pipes alive until memory is exhausted.
Patching the line to add the batch flag seemed to have slowed the stacking up of the scripts, but did not eliminate the problem. The modification merely dragged the failure out for a few hours more:
# diff snort_check snort_check.mod
16c16
< SNORTPCT=`top -n1 | grep snort | grep $pid | cut -d% -f1 | awk '{print $NF}'`
---
> SNORTPCT=`top -b -n1 | grep snort | grep $pid | cut -d% -f1 | awk '{print $NF}'`
Further investigation running the command manually revealed that TOP did not exit as it should from the count mode (-n 1), holding and updating the process as it ran forward. The following memory status at failure mirrored the one before.
Sat Jan 1 01:03:37 UTC 2000
total used free shared buffers
Mem: 30512 29868 644 0 288
Swap: 65524 33356 32168
Total: 96036 63224 32812
top proc: 49
snort procs: 252
TOP not exiting caused over-consumption and, ultimately, a process contention problem. TOP was called within the /packetprotector/usr/sbin/snort_check script, which itself was executed every minute from the /packetprotector/usr/sbin/watchdog script as scheduled by CRON. To alleviate the problem immediately, the watchdog was modified to remove the snort_check script, as below.
# cat /packetprotector/usr/sbin/watchdog
#!/bin/sh
/packetprotector/usr/sbin/nas_check &
#/packetprotector/usr/sbin/snort_check &
After seven hours of running with no traffic, the memory utilization stabilized and held constant at 32MB. This indicated a need to modify the snort_check script to improve the memory monitoring process. Since TOP was not performing as documented per the options, another method was needed to get the memory information. /proc/[pid]/status has the VmRSS information that is reported to TOP, so this was used to draw the information directly for the script.
/proc/[pid]/status - look for VmRSS for the in memory value
/proc/[pid]/cmdline - look for the command line executed for a grep
The following modifications were made:
# diff snort_check snort_check.mod
10c10,12
< HI_PCT="130"
---
> # 20120805 - Changing PCT to an absolute value for a limit. Adjust as needed. Removes the need to use TOP.
> #HI_PCT="130"
> HI_PCT="20000"
16c18
< SNORTPCT=`top -n1 | grep snort | grep $pid | cut -d% -f1 | awk '{print $NF}'`
---
> SNORTPCT=`cat /proc/$pid/status | grep VmRSS | awk '{print $2}'`
/packetprotector/usr/sbin/watchdog was modified to call the new script.
# cat usr/sbin/watchdog
#!/bin/sh
/packetprotector/usr/sbin/nas_check &
#/packetprotector/usr/sbin/snort_check &
/packetprotector/usr/sbin/snort_check.mod &
The new script was run under monitoring for a while to ensure the new watchdog process was working. Consumed memory was now unaffected and the script is confirmed to be working properly. The change was made permanent to my unit and has eliminated crashing for more than a week.
Saturday, August 25, 2012
Saturday, January 7, 2012
Analysis of a Freshly Installed Windows 7's Teredo Tunneling Traffic
While reviewing packet drops, I noticed a system that looked like a low, slow series of repeated failures.
Digging into it, I identified the system as our newly installed Windows 7 host. This seemed a bit odd. Most of the dropped packets were to a single destination, 65.55.158.118.
Searching through the DNS logs, I found that the IP was only involved in a request for teredo.ipv6.microsoft.com. The picture below shows the result of the query, however the googleusercontent.com return is an error in the way DNSMASQ mixes the logs for simultaneous return records. So, there is only one domain to IP relationship between teredo.ipv6.microsoft.com and 65.55.158.118.
To see the activity of just the traffic to this host, I decided to graph the activity to IP address 65.55.158.118 across all of the datasets. This would show the interrelationship of DNS, firewall, and proxy resources over the last seven days. I used this query in my Splunk search:
The columns only indicated DNS and firewall would be factors since no proxy logs matched. I clicked 'Show Report' at the upper-right and selected a line graph with a logrithmic aspect. Here are my results:
So, my new Windows 7 host commits nearly constant DNS resolutions to teredo.ipv6.microsoft.com, resolves to 65.55.158.118, then sends roughly nine times as much direct traffic to that IP to die at the firewall as it is apparently not proxy aware.
My understanding has been that Microsoft disabled Teredo Tunneling by default after Windows XP Service Pack 1 due to industry concerns about data control. This is confirmed by a TechNet article released in 2007. A separate post found at the Microsoft Answers site indicates for Windows 7, "Teredo is installed by default and enabled so that it remains in a standby mode and comes into action only when required." Apparently not since it makes queries ALL THE TIME.
Teredo-by-default remains a potential security and data-loss issue for companies, many of whom lack tools to analyze the 6to4 wrapped protocol for network intrusion detection, enforcing proxy use for regulatory compliance, or other purposes. Some analysis of the protocol has been undertaken, as indicated by this Internet Storm Center article about a year ago, and followed by an execellent analysis by Johannes Ullrich in May. However, this level of depth in monitoring this protocol is not packaged or attainable to most businesses today.
To address the issue, administrators not utilizing Teredo for their domain should disable it via Global Policy. For home users of Windows 7 and Vista, this article at Windows Seven Forums provides instructions to disable the service.
In the Windows menu search bar, type 'cmd.exe', but don't open it. Right-click cmd.exe and select 'Run as Administrator'. Click Yes if prompted to confirm running the program with elevated privileges. Now follow the commands indicated in the output below:
Reboot the system to have the setting take effect.
Digging into it, I identified the system as our newly installed Windows 7 host. This seemed a bit odd. Most of the dropped packets were to a single destination, 65.55.158.118.
Searching through the DNS logs, I found that the IP was only involved in a request for teredo.ipv6.microsoft.com. The picture below shows the result of the query, however the googleusercontent.com return is an error in the way DNSMASQ mixes the logs for simultaneous return records. So, there is only one domain to IP relationship between teredo.ipv6.microsoft.com and 65.55.158.118.
To see the activity of just the traffic to this host, I decided to graph the activity to IP address 65.55.158.118 across all of the datasets. This would show the interrelationship of DNS, firewall, and proxy resources over the last seven days. I used this query in my Splunk search:
65.55.158.118 earliest=-7d | fields sourcetype | timechart limit=0 span=1h count(sourcetype) by sourcetype
The columns only indicated DNS and firewall would be factors since no proxy logs matched. I clicked 'Show Report' at the upper-right and selected a line graph with a logrithmic aspect. Here are my results:
So, my new Windows 7 host commits nearly constant DNS resolutions to teredo.ipv6.microsoft.com, resolves to 65.55.158.118, then sends roughly nine times as much direct traffic to that IP to die at the firewall as it is apparently not proxy aware.
My understanding has been that Microsoft disabled Teredo Tunneling by default after Windows XP Service Pack 1 due to industry concerns about data control. This is confirmed by a TechNet article released in 2007. A separate post found at the Microsoft Answers site indicates for Windows 7, "Teredo is installed by default and enabled so that it remains in a standby mode and comes into action only when required." Apparently not since it makes queries ALL THE TIME.
Teredo-by-default remains a potential security and data-loss issue for companies, many of whom lack tools to analyze the 6to4 wrapped protocol for network intrusion detection, enforcing proxy use for regulatory compliance, or other purposes. Some analysis of the protocol has been undertaken, as indicated by this Internet Storm Center article about a year ago, and followed by an execellent analysis by Johannes Ullrich in May. However, this level of depth in monitoring this protocol is not packaged or attainable to most businesses today.
To address the issue, administrators not utilizing Teredo for their domain should disable it via Global Policy. For home users of Windows 7 and Vista, this article at Windows Seven Forums provides instructions to disable the service.
In the Windows menu search bar, type 'cmd.exe', but don't open it. Right-click cmd.exe and select 'Run as Administrator'. Click Yes if prompted to confirm running the program with elevated privileges. Now follow the commands indicated in the output below:
C:\Windows\system32>netsh
netsh>interface
netsh interface>teredo
netsh interface teredo>set state disabled
Ok.
netsh interface teredo>exit
C:\Windows\system32>
Reboot the system to have the setting take effect.
Saturday, December 10, 2011
Building a Debian/Ubuntu Package for DNSCrypt-proxy
Referring to X4's issue for Linux compiling, the following are instructions for builiding a debian package.
Create a working directory. Then acquire the source by cloning the repository. Create a source tarball in the Debian package naming format.
$ cd ~
$ mkdir dnscryptworkdir
$ cd dnscryptworkdir
$ git clone https://github.com/opendns/dnscrypt-proxy
$ mv dnscrypt-proxy dnscrypt-proxy-0.1
$ tar cvfz dnscrypt-proxy_0.1.orig.tar.gz dnscrypt-proxy-0.1
$ cd dnscrypt-proxy-0.1
Now it's time to lay the groundwork for the package. Use DebHelper to generate the package framework. If you haven't built a package before, install the necessary packages.
$ sudo apt-get install build-essential devscripts ubuntu-dev-tools debhelper dh-make diff patch cdbs quilt gnupg \
fakeroot lintian pbuilder piuparts
Use DebHelper to lay the package framework. This will generate the debian/ directory and associated files. Go ahead and remove the defaults. The README.Debian file is also unnecessary
$ dh_make -f ../dnscrypt-proxy_0.1.orig.tar.gz -s -b
$ cd debian
$ rm *.ex *.EX
$ rm README.Debian
Edit the control file with your favorite editor. It should look like this
Source: dnscrypt-proxy
Section: misc
Priority: optional
Maintainer: pinowudi
Build-Depends: cdbs, debhelper (>= 7), automake
Standards-Version: 3.8.3
Homepage: https://github.com/opendns/dnscrypt-proxy
Package: dnscrypt-proxy
Architecture: all
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: A tool for securing communications between a client and a DNS resolver.
DNSCrypt is a slight variation on DNSCurve.
.
DNSCurve improves the confidentiality and integrity of DNS requests using high-speed high-security elliptic-curve cryptography. Best of all, DNSCurve has very low overhead and adds virtually no latency to queries.
.
DNSCurve aims at securing the entire chain down to authoritative servers. However, it only works with authoritative servers that explicitly support the protocol. And unfortunately, DNSCurve hasn't received much adoption yet.
.
The DNSCrypt protocol is very similar to DNSCurve, but focuses on securing communications between a client and its first-level resolver. While not providing end-to-end security, it protects the local network (which is often the weakest link in the chain) against man-in-the-middle attacks. It also provides some confidentiality to DNS queries.
.
The DNSCrypt daemon acts as a DNS proxy between a regular client, like a DNS cache or an operating system stub resolver, and a DNSCrypt-aware resolver, like OpenDNS.
Now edit the rules file. Using the CDBS packager makes implementing the autoconf features easy, with one exception. Since the autoconf has not been run already, it must be done before the package build. This will require the extra 'autoreconf' as a preconfiguration action. The rules file should look akin to the following.
#!/usr/bin/make -f
include /usr/share/cdbs/1/class/autotools.mk
include /usr/share/cdbs/1/rules/debhelper.mk
include /usr/share/cdbs/1/class/makefile.mk
# Add here any variable or target overrides you need.
makebuilddir/dnscrypt-proxy-0.1::
autoreconf --install
Update the changelog file using the dce tool. Mark the issue as change number zero.
$ dch -e
dnscrypt-proxy (0.1-1) unstable; urgency=low
* Initial release (Closes: #0)
-- pinowudi Sat, 10 Dec 2011 14:11:02 -0500
Edit the copyright file with the copyright information from the package.
This work was packaged for Debian by:
pinowudi on Sat, 10 Dec 2011 14:01:36 -0500
It was downloaded from
Upstream Author(s):
Frank Denis
Copyright:
Copyright (c) 2011 OpenDNS, Inc.
License:
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
====
This license applies to all parts of dnscrypt-proxy that are not externally
maintained libraries.
The externally maintained libraries used by dnscrypt-proxy are:
- NaCl (http://nacl.cr.yp.to/). Public domain.
- libuv (https://github.com/joyent/libuv). MIT license.
+ libuv dependencies, see src/libuv/LICENSE.
- alt_arc4random.c reuses code from OpenBSD. BSD license,
see the alt_arc4random.c header.
The Debian packaging is:
Copyright (C) 2011 pinowudi
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
The software package does not have a .config file yet, so making one modification is necessary to get the CDBS manager to handle the package. Run autoreconfig to generate the autoconf files. Then execute the package build. The package should be produced in the directory below the current build dir.
$ autoreconf --install
$ debuild -us -uc
$ ls ..
dnscrypt-proxy-0.1 dnscrypt-proxy_0.1-1_all.deb dnscrypt-proxy_0.1-1.diff.gz dnscrypt-proxy_0.1-1.dsc dnscrypt-proxy_0.1-1_i386.build dnscrypt-proxy_0.1-1_i386.changes dnscrypt-proxy_0.1.orig.tar.gz
Now test the package installation.
$ cd ..
$ sudo dpkg -i dnscrypt-proxy_0.1-1_all.deb
[sudo] password :
Selecting previously deselected package dnscrypt-proxy.
(Reading database ... 122394 files and directories currently installed.)
Unpacking dnscrypt-proxy (from dnscrypt-proxy_0.1-1_all.deb) ...
Setting up dnscrypt-proxy (0.1-1) ...
Processing triggers for man-db ...
$ whereis dnscrypt-proxy
dnscrypt-proxy: /usr/sbin/dnscrypt-proxy /usr/share/man/man8/dnscrypt-proxy.8.gz
$ dnscrypt-proxy -h
dnscrypt-proxy 0.8
Copyright (C) 2011 OpenDNS, Inc.
Options:
-a --local-address=...
-d --daemonize
-e --edns-payload-size=...
-h --help
-k --provider-key=...
-l --logfile=...
-n --max-active-requests=...
-p --pidfile=...
-r --resolver-address=...
-t --tcp-port=...
-u --user=...
-N --provider-name=...
-P --local-port=...
-V --version
Please consult the dnscrypt-proxy(8) man page for details.
Good to go!
Create a working directory. Then acquire the source by cloning the repository. Create a source tarball in the Debian package naming format.
$ cd ~
$ mkdir dnscryptworkdir
$ cd dnscryptworkdir
$ git clone https://github.com/opendns/dnscrypt-proxy
$ mv dnscrypt-proxy dnscrypt-proxy-0.1
$ tar cvfz dnscrypt-proxy_0.1.orig.tar.gz dnscrypt-proxy-0.1
$ cd dnscrypt-proxy-0.1
Now it's time to lay the groundwork for the package. Use DebHelper to generate the package framework. If you haven't built a package before, install the necessary packages.
$ sudo apt-get install build-essential devscripts ubuntu-dev-tools debhelper dh-make diff patch cdbs quilt gnupg \
fakeroot lintian pbuilder piuparts
Use DebHelper to lay the package framework. This will generate the debian/ directory and associated files. Go ahead and remove the defaults. The README.Debian file is also unnecessary
$ dh_make -f ../dnscrypt-proxy_0.1.orig.tar.gz -s -b
$ cd debian
$ rm *.ex *.EX
$ rm README.Debian
Edit the control file with your favorite editor. It should look like this
Source: dnscrypt-proxy
Section: misc
Priority: optional
Maintainer: pinowudi
Build-Depends: cdbs, debhelper (>= 7), automake
Standards-Version: 3.8.3
Homepage: https://github.com/opendns/dnscrypt-proxy
Package: dnscrypt-proxy
Architecture: all
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: A tool for securing communications between a client and a DNS resolver.
DNSCrypt is a slight variation on DNSCurve.
.
DNSCurve improves the confidentiality and integrity of DNS requests using high-speed high-security elliptic-curve cryptography. Best of all, DNSCurve has very low overhead and adds virtually no latency to queries.
.
DNSCurve aims at securing the entire chain down to authoritative servers. However, it only works with authoritative servers that explicitly support the protocol. And unfortunately, DNSCurve hasn't received much adoption yet.
.
The DNSCrypt protocol is very similar to DNSCurve, but focuses on securing communications between a client and its first-level resolver. While not providing end-to-end security, it protects the local network (which is often the weakest link in the chain) against man-in-the-middle attacks. It also provides some confidentiality to DNS queries.
.
The DNSCrypt daemon acts as a DNS proxy between a regular client, like a DNS cache or an operating system stub resolver, and a DNSCrypt-aware resolver, like OpenDNS.
Now edit the rules file. Using the CDBS packager makes implementing the autoconf features easy, with one exception. Since the autoconf has not been run already, it must be done before the package build. This will require the extra 'autoreconf' as a preconfiguration action. The rules file should look akin to the following.
#!/usr/bin/make -f
include /usr/share/cdbs/1/class/autotools.mk
include /usr/share/cdbs/1/rules/debhelper.mk
include /usr/share/cdbs/1/class/makefile.mk
# Add here any variable or target overrides you need.
makebuilddir/dnscrypt-proxy-0.1::
autoreconf --install
Update the changelog file using the dce tool. Mark the issue as change number zero.
$ dch -e
dnscrypt-proxy (0.1-1) unstable; urgency=low
* Initial release (Closes: #0)
-- pinowudi
Edit the copyright file with the copyright information from the package.
This work was packaged for Debian by:
pinowudi
It was downloaded from
Upstream Author(s):
Frank Denis
Copyright:
Copyright (c) 2011 OpenDNS, Inc.
License:
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
====
This license applies to all parts of dnscrypt-proxy that are not externally
maintained libraries.
The externally maintained libraries used by dnscrypt-proxy are:
- NaCl (http://nacl.cr.yp.to/). Public domain.
- libuv (https://github.com/joyent/libuv). MIT license.
+ libuv dependencies, see src/libuv/LICENSE.
- alt_arc4random.c reuses code from OpenBSD. BSD license,
see the alt_arc4random.c header.
The Debian packaging is:
Copyright (C) 2011 pinowudi
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
The software package does not have a .config file yet, so making one modification is necessary to get the CDBS manager to handle the package. Run autoreconfig to generate the autoconf files. Then execute the package build. The package should be produced in the directory below the current build dir.
$ autoreconf --install
$ debuild -us -uc
$ ls ..
dnscrypt-proxy-0.1 dnscrypt-proxy_0.1-1_all.deb dnscrypt-proxy_0.1-1.diff.gz dnscrypt-proxy_0.1-1.dsc dnscrypt-proxy_0.1-1_i386.build dnscrypt-proxy_0.1-1_i386.changes dnscrypt-proxy_0.1.orig.tar.gz
Now test the package installation.
$ cd ..
$ sudo dpkg -i dnscrypt-proxy_0.1-1_all.deb
[sudo] password :
Selecting previously deselected package dnscrypt-proxy.
(Reading database ... 122394 files and directories currently installed.)
Unpacking dnscrypt-proxy (from dnscrypt-proxy_0.1-1_all.deb) ...
Setting up dnscrypt-proxy (0.1-1) ...
Processing triggers for man-db ...
$ whereis dnscrypt-proxy
dnscrypt-proxy: /usr/sbin/dnscrypt-proxy /usr/share/man/man8/dnscrypt-proxy.8.gz
$ dnscrypt-proxy -h
dnscrypt-proxy 0.8
Copyright (C) 2011 OpenDNS, Inc.
Options:
-a --local-address=...
-d --daemonize
-e --edns-payload-size=...
-h --help
-k --provider-key=...
-l --logfile=...
-n --max-active-requests=...
-p --pidfile=...
-r --resolver-address=...
-t --tcp-port=...
-u --user=...
-N --provider-name=...
-P --local-port=...
-V --version
Please consult the dnscrypt-proxy(8) man page for details.
Good to go!
Thursday, September 8, 2011
Bigger Sets to Visualize
After yesterdays post, I revisited the data and came up with a better static query. It now just looks for 40 byte packets fom port 12200. The supporting query looks like this:
The flash clearly indicates a ramp-up of scanning activity last month, but just how much? Open the Report Builder and re-render the data as a stacked area graph. Again, treat nullvalues as zero for this purpose.
Thats a pretty dramatic take-off from a wide selection of IP addresses. None of them have reverse DNS records, and all have with infrastructure-style WHOIS records. A brief review indicates several clusters of IP addresses at large virtual hostsers with a few smaller hosters and individual servers. Perhaps a compromised management console (remember CPanel?) or a clutch of vulnerable LAMP stacks?
Visualizing and Investigating a Distributed Scan
I was checking my logs and trying out a new app I created for Splunk. Normally my graph looks like the one below, with the exception of one host that I recently added that was wildly dropping packets on the left. At the bottom-right, there are typically some bursts, but for the most part the averages are low. The scatterplots are fairly tight and low volume. For this packet length average vs. summation plot, most things cluster at the bottom left (low volume), occasionally blip into one of the upper-left (lots of low, slow) or lower-right (a few large packets) corners, and rarely pop-up in the upper-right (lots of big packets, blasting) quadrant. Here is an example:
The Bird is the Word
Today I check the dashboard and the left side (ingress) looked remarkably different. The averages were pretty normal, but there was a steady baseline at the bottom. Also, the summation of dropped packets was much higher than the 'normal' sampling. The scatterplot also diverged more in the ~40 byte range, with a sample of hosts clearly engaged in a lot of small transactions. This peaqued my interest.
And Now It's Time for Breakdown...
Investigating the increase in ingress summation (the chart at the bottom-right), I clicked the 'View Results' link on that chart. I then modified it slightly to reduce the number of extracted fields. This improved the search speed remarkably. I also eliminated the aggregate summing the 'timechart' command performs by default after the top 10, which returned all of the hosts. The 'limit=0' parameter achieved this. Now I had a chart that I could graph on a line chart and analyze more closely.
I adjusted the default settings slightly to treat null values as zero and added some axis labels. TIP: If you hover the mouse over one of the IP addresses in legend, Splunk highlights the line for that selection. I noticed the 58.218.199.0/24 IP addresses all have that matching bumpity-bump along the bottom, low and slow. This is pretty good evidence of an undiversified distributed scan. I decided to generate some better metrics.
Which of these is Not Like the Others?
Looking at the rejected packet meta, I wanted to determine just how alike the packets were that were dropping from these hosts. I limited the search to just that subnet with a straight text search. Again, I limited the selected fields to improve the extraction speed. Then I created a timechart to bin by day the average packet length of each server involved. Once the search got underway, I clicked 'Show Report'.
In the Report Builder, I selected an area chart, treating nulls as zero, and stacked the results. As you see in this screenshot, the report showed the distribution of the packet sizes between the servers was even. The scanning cluster also ramped up just after a preliminary poke from a single server, 58.218.199.49.
Peek-a-boo, I See You
Having identified the scanning cluster, I wanted to see what ports the aggressor was looking to find. During my investigation, I came across this comment at the Internet Storm Center website, followed by many other confirmatory posts at other sites, regarding the nature of the attacker. It appeared they have been active for a number of years.
After identifying the affected ports, I prettied it up a bit with a lookup table to provide the common uses of those ports. It became plain that the scanner was looking for open proxies, whether legitimate, ill-configured, left by malware, or just plain hacked. There were many supporting statements for all of these in the resources culled to identify these ports.
Playing Favorites?
Now that I had an inkling what they were up to, I was interested in seeing if they favor certain ports. The data showed that the scanners were coordinated to test the target port set just about evenly.
Summing It Up
It seems the aggressors utilizing these Chinese servers really like to scan for open proxies. Intent for such a collection, who can say? It could be folks looking for a way to circumvent Chinese proxy filters. However, judging from the technical coordination and longevity of this scanning behavior, it would seem more coordinated than that. In the least, better funded, and possibly sanctioned. Regardless of speculation, it is without doubt that the aggressors seek open proxy servers without permission, even trolling for botnet proxy remnants, and at the expense of those they locate when they turn on the tubes.
 |
| Normal Dashboard |
The Bird is the Word
Today I check the dashboard and the left side (ingress) looked remarkably different. The averages were pretty normal, but there was a steady baseline at the bottom. Also, the summation of dropped packets was much higher than the 'normal' sampling. The scatterplot also diverged more in the ~40 byte range, with a sample of hosts clearly engaged in a lot of small transactions. This peaqued my interest.
 |
| Abby Normal |
And Now It's Time for Breakdown...
Investigating the increase in ingress summation (the chart at the bottom-right), I clicked the 'View Results' link on that chart. I then modified it slightly to reduce the number of extracted fields. This improved the search speed remarkably. I also eliminated the aggregate summing the 'timechart' command performs by default after the top 10, which returned all of the hosts. The 'limit=0' parameter achieved this. Now I had a chart that I could graph on a line chart and analyze more closely.
 |
| Prep for launch... |
I adjusted the default settings slightly to treat null values as zero and added some axis labels. TIP: If you hover the mouse over one of the IP addresses in legend, Splunk highlights the line for that selection. I noticed the 58.218.199.0/24 IP addresses all have that matching bumpity-bump along the bottom, low and slow. This is pretty good evidence of an undiversified distributed scan. I decided to generate some better metrics.
 |
| Launch! Graph of FW drops over a timespan. |
Which of these is Not Like the Others?
Looking at the rejected packet meta, I wanted to determine just how alike the packets were that were dropping from these hosts. I limited the search to just that subnet with a straight text search. Again, I limited the selected fields to improve the extraction speed. Then I created a timechart to bin by day the average packet length of each server involved. Once the search got underway, I clicked 'Show Report'.
In the Report Builder, I selected an area chart, treating nulls as zero, and stacked the results. As you see in this screenshot, the report showed the distribution of the packet sizes between the servers was even. The scanning cluster also ramped up just after a preliminary poke from a single server, 58.218.199.49.
 |
| Poke then plunge. This stacked area graph shows even traffic from the scanning cluster. |
Peek-a-boo, I See You
Having identified the scanning cluster, I wanted to see what ports the aggressor was looking to find. During my investigation, I came across this comment at the Internet Storm Center website, followed by many other confirmatory posts at other sites, regarding the nature of the attacker. It appeared they have been active for a number of years.
After identifying the affected ports, I prettied it up a bit with a lookup table to provide the common uses of those ports. It became plain that the scanner was looking for open proxies, whether legitimate, ill-configured, left by malware, or just plain hacked. There were many supporting statements for all of these in the resources culled to identify these ports.
 |
| Proxy ports, proxy ports... Oh, more proxy ports. No secrets here. |
Playing Favorites?
Now that I had an inkling what they were up to, I was interested in seeing if they favor certain ports. The data showed that the scanners were coordinated to test the target port set just about evenly.
 |
| Even distribution of scanned ports. |
Summing It Up
It seems the aggressors utilizing these Chinese servers really like to scan for open proxies. Intent for such a collection, who can say? It could be folks looking for a way to circumvent Chinese proxy filters. However, judging from the technical coordination and longevity of this scanning behavior, it would seem more coordinated than that. In the least, better funded, and possibly sanctioned. Regardless of speculation, it is without doubt that the aggressors seek open proxy servers without permission, even trolling for botnet proxy remnants, and at the expense of those they locate when they turn on the tubes.
Monday, August 29, 2011
This article is in response to a query about securing a FIOS network in a reasonable way for family and home use.
This design will create three separate zones. The first zone is the WAN, which simply picks up the IP address, routing, and DNS information from the FIOS servers. This implements the simple SPI firewall filters.
The second zone is the Trusted LAN, which will be locked down in what it can pass to the other zones. Why treat the Trusted LAN with less trust? Well, if we want to trust assets on that network, we need them to perform in compliant ways. So, instead of inherently trusting assets on that network, we enforce that that segment will act in an assured manner. This means implementing egress controls, limiting weird protocols, and forcing the use of proxies. Think of it less as 'a zone of trusted hosts' and more like 'a zone of hosts I need to trust'. These hosts are subject to more scrutiny as they or their data are more valuable.
The third zone is the DMZ. The hardened proxy/dns server goes in here, but the rest of that segment is for hosts that can't comply with the mandatory restrictions of the Trusted LAN. These are usually consumer appliance devices like iPods, iPads, tablets, video game consoles, Internet-enabled DVD players and the like. My rubric: consumer appliance devices that need Internet access to be useful but are too dumb to print.
I also recommend placing wireless connections into yet another zone to provide granular access controls. Wireless is typically the weakest infrastructure conduit in a home network, excluding PEBKAC. However, attaching wireless access points into the DMZ and Trusted zones may be suitable depending on your ability to secure them.
Pick list:
Bringing in the WAN
My FIOS installation has the unfortunate distinction of not using the Ethernet for the WAN connection, but bringing in the WAN backhaul over the COAX. This requires the use of a MOCA adapter to bridge the WAN to Ethernet. Since I don't have any spare MOCA units around and the COAX LAN to the video set-top boxes run over that same COAX line, that means using the ActionTec in a double-bridge confirguration, as documented in my previous post "Verizon FIOS Faux Paus".
Firewall Installation
Download and install m0n0wall on the firewall host. Once the steps are completed to create the double-bridge, or if you have Ethernet WAN, run the WAN line to eth1 of the firewall host. The default setup via the console of the FW host should allow for easy implementation of the WAN (eth1) and LAN (eth0) ports in simple no-ingress, all-egress configuration. Apply a common RFC 1918 private class-C subnet to the trusted LAN. e.g. 192.168.1.0/24.
Now set up the third interface as a DMZ (eth2). Set it up in standard SPI configuration: block ingress initiates, allow all outbound traffic. This will allow just about anything to work in this network. It also provides a nice guest network for visitors. Apply a different class-C subnet for the DMZ. e.g. 192.168.2.0/24
Connect the switches to the respective ports. You should definitely be using separate physical switches for each segment. Otherwise, employ VLAN tagging for the segments on a single switch that can support it. Wireless routers with integrated switches can serve this purpose, just use the Ethernet LAN ports, not the WAN port to avoid double NAT'ing. If using a switch with node presence, like most consumer-grade [wireless] routers, be sure to set the router IP to a private subnet not routable by that served by the LAN it is on. For example, on a 192.168.n.n LAN with a gateway of 192.168.n.1, the router management IP should be 10.n.n.n. Management of the switch should be performed by a direct hookup to a host with a static IP set to the same class-C subnet. It is not a perfect solution, but it will help prevent drive-by XSS exploits on weak router configuration pages and casual scans. Neither is it invisible, as described later. Routers are noisy things.
Establish Essential Services
With basic services going, it is time to establish control over essential services. This will be done on a host operating in the DMZ. This will provide it the access it requires to the Internet for services, make it available to both TRUSTED and DMZ hosts that might make use of it, and protect the TRUSTED network from probes should the box become compromised. However, if the host becomes compromised, it will give the attacker control over traffic direction (DNS) and content (proxy) for the TRUSTED net. I chose Ubuntu for the easy package management, though any *nix operating system should be appropriate. This box should be hardened and tightly monitored.
Most of the Internet operates via HTTP(S) after performing a DNS resolution. Therefore, controlling these protocols is essential to establishing control in the network. With the intent of completely locking down egress protocols on the TRUSTED LAN later on, a fully-featured proxy server is necessary. Caching is also a nice feature to reduce the load of things like updates, which most of the hosts on the network will be doing from Internet sources. SQUID provides caching support, access enforcement, and robust logging features.
Optionally, one can layer DansGuardian as a free regex-based enforcement layer on top of SQUID. It provides access controls suited for many common web ailments. Be advised, it will require a bit of tuning at setup to keep from blocking things like system updates. It also does not respond well for some foreign-language matches, which can become a burden. Otherwise, if you are technically inclined and have time to maintain your own custom lists, this is a great tool.
On the proxy host, also install DNSMASQ. This is the same utility that provides forwarding DNS and DHCP services on many wireless routers. We will only be using it for its DNS capabilities. Edit the dnsmasq configuration file to manually set OpenDNS as the DNS forwarders. OpenDNS provides filtered DNS resolution for free, protecting against many of the most obvious violations by botnet herders by default, even without an registered account. If you choosed to register and add your home network, you can gain further content control based on their domain categorizations.
Set the logging of both squid and dnsmasq to the syslog daemon. Modern syslog services, such as rsyslog and syslog-ng, are capable of filtering different program's logs into rotated files. Establish log directories and rotated log files for squid and dnsmasq. Also fork these feeds to the central syslog server (audit server) to be established later.
Locking Down
Now that DNS and Proxy services are established, adjust the configuration of the firewall. Change the default DHCP behavior from issuing the firewall's IP as the DNS passthrough to issuing the IP address of your proxy/dns server in the DMZ. This will encourage hosts to use the DMZ DNS server, which provides logging and domain access control. In the TRUSTED LAN, adjust your hosts' settings to use the proxy server in the DMZ. This is typically done by setting the proxy values to "http://proxyip:3128/", and selecting 'apply to all protocols'. For Windows boxes, this must be done in Internet Explorer to make system updates work, as well as any other installed browsers that do not integrate with IE's proxy settings. For Gnome Linux hosts, this is typically under System -> Preferences -> Network Proxy as the administrative user. When prompted, confirm 'apply systemwide'.
With hosts in the TRUSTED LAN using the proxy, the firewall can now be prepared to lock out all outbound protocols for that subnet. Establish a rule allowing http/s and dns traffic to the proxy server. Create a rule allowing syslog from the dmz proxy server into your trusted audit server. Establish another rule allowing SSH from the trusted server to the dmz proxy server. This will allow you to manage the proxy server to manage it after first authenticating to the trusted server. Prepare for a day where you will be able to monitor the blocked traffic from the firewall on your syslog server. There will be a few things like anti-virus software updates and email server access that may not respond well to the demand for a proxy server. These will have to be manually updated in the firewall as exceptions. There may be a rule explicitly allowing traffic from the LAN to the DMZ. It should be disabled. Now, as the last rule, reject everything sourced from the TRUSTED LAN. Apply and start watching the logs.
Wireless
If you have chosen to employ wireless, ensure that the wireless access points are configured to different ESSIDs and encryption keys. They do not service the same networks.
Review
In this post, we've reviewed locking down the network of hosts we need to trust, while still allowing non-proxy-capable devices an allowance to the Internet. The DMZ allows cart-blanche access to all devices, providing a network suitable for Internet-ready appliances and guests. The proxy services enforce proper accesses on the Trusted hosts and the DNS service logs all of the hosts requests.
Next time: Auditing the Logs
This design will create three separate zones. The first zone is the WAN, which simply picks up the IP address, routing, and DNS information from the FIOS servers. This implements the simple SPI firewall filters.
The second zone is the Trusted LAN, which will be locked down in what it can pass to the other zones. Why treat the Trusted LAN with less trust? Well, if we want to trust assets on that network, we need them to perform in compliant ways. So, instead of inherently trusting assets on that network, we enforce that that segment will act in an assured manner. This means implementing egress controls, limiting weird protocols, and forcing the use of proxies. Think of it less as 'a zone of trusted hosts' and more like 'a zone of hosts I need to trust'. These hosts are subject to more scrutiny as they or their data are more valuable.
The third zone is the DMZ. The hardened proxy/dns server goes in here, but the rest of that segment is for hosts that can't comply with the mandatory restrictions of the Trusted LAN. These are usually consumer appliance devices like iPods, iPads, tablets, video game consoles, Internet-enabled DVD players and the like. My rubric: consumer appliance devices that need Internet access to be useful but are too dumb to print.
I also recommend placing wireless connections into yet another zone to provide granular access controls. Wireless is typically the weakest infrastructure conduit in a home network, excluding PEBKAC. However, attaching wireless access points into the DMZ and Trusted zones may be suitable depending on your ability to secure them.
Pick list:
- Pentium-3 class computer with 256 MB RAM. As this machine is likely 10 years old, purchasing an IDE-to-Compact Flash adapter with an 8GB CF card will give it some pick-up. Assuming there is already one Ethernet onboard the motherboard, this unit should have at least a PCI expansion port capable of supporting a 2- or 4-port Ethernet card. Keep in mind, three ports are required to make this configuration work. With a minimalist firewall OS, like m0n0wall, this will become a capable firewall.
- Matching PCI 2- or 4-port Ethernet card. Server-class coprocessors by Intel are desirable.
- Pentium-4 class computer with 512 MB RAM + 10GB drive. This machine is probably 5-7 years old and needs a little more spunk to deliver layer-7 services. Consider the above-mentioned PCI-to-CF converter for the base OS with a secondary drive for bulk storage. This will become the DMZ proxy and DNS services.
- Pentium class computer, configuration negotiable, lots of bulk disk (fast I/O preferred). This machine will become the internal security auditor, receiving logs from the DMZ proxy and the firewall.
Bringing in the WAN
My FIOS installation has the unfortunate distinction of not using the Ethernet for the WAN connection, but bringing in the WAN backhaul over the COAX. This requires the use of a MOCA adapter to bridge the WAN to Ethernet. Since I don't have any spare MOCA units around and the COAX LAN to the video set-top boxes run over that same COAX line, that means using the ActionTec in a double-bridge confirguration, as documented in my previous post "Verizon FIOS Faux Paus".
Firewall Installation
Download and install m0n0wall on the firewall host. Once the steps are completed to create the double-bridge, or if you have Ethernet WAN, run the WAN line to eth1 of the firewall host. The default setup via the console of the FW host should allow for easy implementation of the WAN (eth1) and LAN (eth0) ports in simple no-ingress, all-egress configuration. Apply a common RFC 1918 private class-C subnet to the trusted LAN. e.g. 192.168.1.0/24.
Now set up the third interface as a DMZ (eth2). Set it up in standard SPI configuration: block ingress initiates, allow all outbound traffic. This will allow just about anything to work in this network. It also provides a nice guest network for visitors. Apply a different class-C subnet for the DMZ. e.g. 192.168.2.0/24
Connect the switches to the respective ports. You should definitely be using separate physical switches for each segment. Otherwise, employ VLAN tagging for the segments on a single switch that can support it. Wireless routers with integrated switches can serve this purpose, just use the Ethernet LAN ports, not the WAN port to avoid double NAT'ing. If using a switch with node presence, like most consumer-grade [wireless] routers, be sure to set the router IP to a private subnet not routable by that served by the LAN it is on. For example, on a 192.168.n.n LAN with a gateway of 192.168.n.1, the router management IP should be 10.n.n.n. Management of the switch should be performed by a direct hookup to a host with a static IP set to the same class-C subnet. It is not a perfect solution, but it will help prevent drive-by XSS exploits on weak router configuration pages and casual scans. Neither is it invisible, as described later. Routers are noisy things.
Establish Essential Services
With basic services going, it is time to establish control over essential services. This will be done on a host operating in the DMZ. This will provide it the access it requires to the Internet for services, make it available to both TRUSTED and DMZ hosts that might make use of it, and protect the TRUSTED network from probes should the box become compromised. However, if the host becomes compromised, it will give the attacker control over traffic direction (DNS) and content (proxy) for the TRUSTED net. I chose Ubuntu for the easy package management, though any *nix operating system should be appropriate. This box should be hardened and tightly monitored.
Most of the Internet operates via HTTP(S) after performing a DNS resolution. Therefore, controlling these protocols is essential to establishing control in the network. With the intent of completely locking down egress protocols on the TRUSTED LAN later on, a fully-featured proxy server is necessary. Caching is also a nice feature to reduce the load of things like updates, which most of the hosts on the network will be doing from Internet sources. SQUID provides caching support, access enforcement, and robust logging features.
Optionally, one can layer DansGuardian as a free regex-based enforcement layer on top of SQUID. It provides access controls suited for many common web ailments. Be advised, it will require a bit of tuning at setup to keep from blocking things like system updates. It also does not respond well for some foreign-language matches, which can become a burden. Otherwise, if you are technically inclined and have time to maintain your own custom lists, this is a great tool.
On the proxy host, also install DNSMASQ. This is the same utility that provides forwarding DNS and DHCP services on many wireless routers. We will only be using it for its DNS capabilities. Edit the dnsmasq configuration file to manually set OpenDNS as the DNS forwarders. OpenDNS provides filtered DNS resolution for free, protecting against many of the most obvious violations by botnet herders by default, even without an registered account. If you choosed to register and add your home network, you can gain further content control based on their domain categorizations.
Set the logging of both squid and dnsmasq to the syslog daemon. Modern syslog services, such as rsyslog and syslog-ng, are capable of filtering different program's logs into rotated files. Establish log directories and rotated log files for squid and dnsmasq. Also fork these feeds to the central syslog server (audit server) to be established later.
Locking Down
Now that DNS and Proxy services are established, adjust the configuration of the firewall. Change the default DHCP behavior from issuing the firewall's IP as the DNS passthrough to issuing the IP address of your proxy/dns server in the DMZ. This will encourage hosts to use the DMZ DNS server, which provides logging and domain access control. In the TRUSTED LAN, adjust your hosts' settings to use the proxy server in the DMZ. This is typically done by setting the proxy values to "http://proxyip:3128/", and selecting 'apply to all protocols'. For Windows boxes, this must be done in Internet Explorer to make system updates work, as well as any other installed browsers that do not integrate with IE's proxy settings. For Gnome Linux hosts, this is typically under System -> Preferences -> Network Proxy as the administrative user. When prompted, confirm 'apply systemwide'.
With hosts in the TRUSTED LAN using the proxy, the firewall can now be prepared to lock out all outbound protocols for that subnet. Establish a rule allowing http/s and dns traffic to the proxy server. Create a rule allowing syslog from the dmz proxy server into your trusted audit server. Establish another rule allowing SSH from the trusted server to the dmz proxy server. This will allow you to manage the proxy server to manage it after first authenticating to the trusted server. Prepare for a day where you will be able to monitor the blocked traffic from the firewall on your syslog server. There will be a few things like anti-virus software updates and email server access that may not respond well to the demand for a proxy server. These will have to be manually updated in the firewall as exceptions. There may be a rule explicitly allowing traffic from the LAN to the DMZ. It should be disabled. Now, as the last rule, reject everything sourced from the TRUSTED LAN. Apply and start watching the logs.
Wireless
If you have chosen to employ wireless, ensure that the wireless access points are configured to different ESSIDs and encryption keys. They do not service the same networks.
Review
In this post, we've reviewed locking down the network of hosts we need to trust, while still allowing non-proxy-capable devices an allowance to the Internet. The DMZ allows cart-blanche access to all devices, providing a network suitable for Internet-ready appliances and guests. The proxy services enforce proper accesses on the Trusted hosts and the DNS service logs all of the hosts requests.
Next time: Auditing the Logs
Wednesday, August 17, 2011
Review of a Clean Android Installation
After acquiring a new Android device recently, I decided to profile the traffic. Using Splunk to analyze the traffic, I came up with this nice little chart. No surprises here.
Thursday, November 11, 2010
Ecryptfs and profile errors
I recently had an Ubuntu fail on ecryptfs. I would log into the system and it would present a default empty home dir with the Access-ecrypt-fs file. The problem was similar to these discussions on lost ecrypt profiles.
https://answers.launchpad.net/ecryptfs/+question/46307
http://ubuntuforums.org/showthread.php?t=1459250
My profile looked like this:
$ ls
Access-Your-Private-Data.desktop README.txt
$ ecryptfs-mount-private
ERROR: Encrypted private directory is not setup properly
After running an strace, I finally discovered that my movement of the /home directory contents had upset the delicate balance of ecryptfs. The program was not finding the .ecryptfs files it was looking for. ecryptfs profiles are not stored in your home directory, but rather are linked to another profile store.
$ ls -la
lrwxrwxrwx 1 user user 31 2010-09-07 21:51 .ecryptfs -> /home/.ecryptfs/user/.ecryptfs
The system had lost its way to /home/.ecryptfs/user/.ecryptfs. Fixing the links in /home recovered my encrypted profile.
$ ls -la /data1/home
total 16
drwxr-xr-x 4 root root 4096 2010-09-07 21:51 .
drwxrwxrwx 11 root root 4096 2010-09-26 10:42 ..
drwx------ 4 user user 4096 2010-09-25 22:00 user
drwxr-xr-x 3 root root 4096 2010-09-07 21:51 .ecryptfs
$ cd /
$ sudo rm -rf home
$ sudo ln -s /data1/home home
https://answers.launchpad.net/ecryptfs/+question/46307
http://ubuntuforums.org/showthread.php?t=1459250
My profile looked like this:
$ ls
Access-Your-Private-Data.desktop README.txt
$ ecryptfs-mount-private
ERROR: Encrypted private directory is not setup properly
After running an strace, I finally discovered that my movement of the /home directory contents had upset the delicate balance of ecryptfs. The program was not finding the .ecryptfs files it was looking for. ecryptfs profiles are not stored in your home directory, but rather are linked to another profile store.
$ ls -la
lrwxrwxrwx 1 user user 31 2010-09-07 21:51 .ecryptfs -> /home/.ecryptfs/user/.ecryptfs
The system had lost its way to /home/.ecryptfs/user/.ecryptfs. Fixing the links in /home recovered my encrypted profile.
$ ls -la /data1/home
total 16
drwxr-xr-x 4 root root 4096 2010-09-07 21:51 .
drwxrwxrwx 11 root root 4096 2010-09-26 10:42 ..
drwx------ 4 user user 4096 2010-09-25 22:00 user
drwxr-xr-x 3 root root 4096 2010-09-07 21:51 .ecryptfs
$ cd /
$ sudo rm -rf home
$ sudo ln -s /data1/home home
Tuesday, October 12, 2010
Scanning Memory Objects with Yara
Below are some Python inserts to enable yara scanning of in-memory objects while parsing something, like a PDF. This particular example enables Yara signature scanning of parsed, filtered PDF objects via Didier Steven's PDF-Parser.
[...imports...]
import yara
import mmap
rules = yara.compile('path/rulefile')
[...parsing code...]
############################ yara insert around line 558
############################ just before
######### print ' %s' % FormatOutput(filtered, options.raw)
memmap=mmap.mmap(-1,len(filtered))
memmap.write(filtered)
memmap.seek(0)
matches = rules.match(data=memmap.read(len(filtered)))
memmap.close()
for m in matches:
__ print ' yara: %s' % (m)
##################################
[...resume Didier's code...]
print ' %s' % FormatOutput(filtered, options.raw)
[...imports...]
import yara
import mmap
rules = yara.compile('path/rulefile')
[...parsing code...]
############################ yara insert around line 558
############################ just before
######### print ' %s' % FormatOutput(filtered, options.raw)
memmap=mmap.mmap(-1,len(filtered))
memmap.write(filtered)
memmap.seek(0)
matches = rules.match(data=memmap.read(len(filtered)))
memmap.close()
for m in matches:
__ print ' yara: %s' % (m)
##################################
[...resume Didier's code...]
print ' %s' % FormatOutput(filtered, options.raw)
Thursday, September 30, 2010
Installing Yara on Ubuntu 10.04
Installation for YARA on Ubuntu 10.04. First you will need the PCRE development and runtime libraries.
$ sudo apt-get install libpcre3 libpcre3-dev
Now acquire the YARA source code.
$ wget http://yara-project.googlecode.com/files/yara-1.4.tar.gz
$ wget http://yara-project.googlecode.com/files/yara-python-1.4.tar.gz
Untar and configure YARA.
$ tar xvfz yara-1.4.tar.gz
$ cd yara-1.4.tar.gz
$ ./configure
If there are no errors, make the executables.
$ make
$ make check
$ sudo make install
Now add python support.
$ cd ..
$ tar xvfz yara-python-1.4.tar.gz
$ cd yara-python-1.4.tar.gz
$ python setup.py build
$ sudo python setup.py install
You should now be able to call YARA from a shell prompt.
$ yara
usage: yara [OPTION]... [RULEFILE]... FILE
options:
-t print rules tagged as and ignore the rest. Can be used more than once.
-i print rules named and ignore the rest. Can be used more than once.
-n print only not satisfied rules (negate).
-g print tags.
-m print metadata.
-s print matching strings.
-d= define external variable.
-r recursively search directories.
-f fast matching mode.
-v show version information.
Report bugs to:
$ sudo apt-get install libpcre3 libpcre3-dev
Now acquire the YARA source code.
$ wget http://yara-project.googlecode.com/files/yara-1.4.tar.gz
$ wget http://yara-project.googlecode.com/files/yara-python-1.4.tar.gz
Untar and configure YARA.
$ tar xvfz yara-1.4.tar.gz
$ cd yara-1.4.tar.gz
$ ./configure
If there are no errors, make the executables.
$ make
$ make check
$ sudo make install
Now add python support.
$ cd ..
$ tar xvfz yara-python-1.4.tar.gz
$ cd yara-python-1.4.tar.gz
$ python setup.py build
$ sudo python setup.py install
You should now be able to call YARA from a shell prompt.
$ yara
usage: yara [OPTION]... [RULEFILE]... FILE
options:
-t
-i
-n print only not satisfied rules (negate).
-g print tags.
-m print metadata.
-s print matching strings.
-d
-r recursively search directories.
-f fast matching mode.
-v show version information.
Report bugs to:
Thursday, September 23, 2010
Installing log2timeline on Ubuntu
Here is a script to ease installation of Kristinn Gudjonsson's Log2Timeline tool on Ubuntu hosts. Tested on Ubuntu Lucid 10.04.
############################################
# log2timeline_ubuntu_deps.sh
############################################
#!/bin/sh
sudo apt-get install libdigest-crc-perl libnetpacket-perl libparse-win32registry-perl libarchive-zip-perl libtimedate-perl libcarp-assert-perl libclass-dbi-perl libdatetime-perl libhtml-scrubber-perl libnet-pcap-perl libparams-validate-perl libimage-exiftool-perl libdbd-sqlite3-perl libdate-manip-perl libdatetime-format-strptime-perl
sudo perl -MCPAN -e 'install File::Mork'
sudo perl -MCPAN -e 'install Data::Hexify'
############################################
# log2timeline_ubuntu_deps.sh
############################################
#!/bin/sh
sudo apt-get install libdigest-crc-perl libnetpacket-perl libparse-win32registry-perl libarchive-zip-perl libtimedate-perl libcarp-assert-perl libclass-dbi-perl libdatetime-perl libhtml-scrubber-perl libnet-pcap-perl libparams-validate-perl libimage-exiftool-perl libdbd-sqlite3-perl libdate-manip-perl libdatetime-format-strptime-perl
sudo perl -MCPAN -e 'install File::Mork'
sudo perl -MCPAN -e 'install Data::Hexify'
Tuesday, August 10, 2010
Verizon FIOS Faux Pas
Exemplifying the problem many folks have been having with Verizon FIOS, our router is version C of the Actiontec line. For the first two years of use, the bulk of our bandwidth was light web surfing and providng data to the VOD/Guide service for the DVR. Not very taxing on a 12Mb/s line. However, two things happened this year that started to tax the FIOS service, as provided by Verizon. First, we got an internet-ready DVD player. For this, we also signed up for NetFlix, an online movie service whose monthly subscription for online access to streaming content is less than a couple of video store rentals. We also discovered that our Wii had onboard wireless and had recently had software written for it to make it NetFlix compatible. This increased our bandwidth usage when streaming a movie by about ten fold. Unfortunately, the little NAT table in the ActionTec router just couldn't handle it and ended up crashing about every other day, requiring a manual power cycle. This just wouldn't fly, so, having spare hardware laying around and a desire to amp up the firewall capabilities of the home connection anyway, I decided to attempt to replace the Actiontec.
I followed the advice provided at DSLReports and set up a double-bridge bypass to my own firewall. This worked well for bypassing the NAT table issue... for about 20 days. After that, the Internet continued to work flawlessly, providing plenty of connections for the demands of streaming media, but the VOD/Guide services consistently failed due to the DVR's inability to pull an IP address. After a couple hours of poking at my firewall and the Actiontec, I found that the COAX (Ethernet) connection was not pulling an IP for the Actiontec, nor was it passing the bridge from the MOCA adapter for the STB. When I realized this, I looked at the configuration of the Actiontec router and saw that the COAX (Ethernet) interface was completely disabled. Now how could this be? It worked before. After re-enabling it and some power cycles, I determined that the Actiontec will only enable the device for the time it is powered on. For some reason, every so often it will self-reboot, which resets the device state to DISABLED, killing the VOD/Guide. ARGH!! So, to keep the STB working as it should, I have to attach to the Ethernet switch, manually set my IP address to the 192.168.1.0/24 net, enter the 192.168.1.1 default address, login, and reenable the COAX (Ethernet) device. What a pain, Verizon!
This also brings to light a security vulnerability in this method of double-bridging. The router automagically assigns itself this 192.168.1.1 address available from its Ethernet ports. As the Ethernet switch is bridged to the COAX (Broadband) device, this means that the router may be remotely accessible to an attack should an aggressor push spoofed RFC 1918 packets to your public IP address. They will drop off of the WAN firewall, but will the local ethernet bind to 192.168.1.1 answer? Since the Ethernet ports are set to make up the outer bridge to the WAN, this seems plausible, making for a potential hole in this setup. So much for adding security with a better firewall!
I followed the advice provided at DSLReports and set up a double-bridge bypass to my own firewall. This worked well for bypassing the NAT table issue... for about 20 days. After that, the Internet continued to work flawlessly, providing plenty of connections for the demands of streaming media, but the VOD/Guide services consistently failed due to the DVR's inability to pull an IP address. After a couple hours of poking at my firewall and the Actiontec, I found that the COAX (Ethernet) connection was not pulling an IP for the Actiontec, nor was it passing the bridge from the MOCA adapter for the STB. When I realized this, I looked at the configuration of the Actiontec router and saw that the COAX (Ethernet) interface was completely disabled. Now how could this be? It worked before. After re-enabling it and some power cycles, I determined that the Actiontec will only enable the device for the time it is powered on. For some reason, every so often it will self-reboot, which resets the device state to DISABLED, killing the VOD/Guide. ARGH!! So, to keep the STB working as it should, I have to attach to the Ethernet switch, manually set my IP address to the 192.168.1.0/24 net, enter the 192.168.1.1 default address, login, and reenable the COAX (Ethernet) device. What a pain, Verizon!
This also brings to light a security vulnerability in this method of double-bridging. The router automagically assigns itself this 192.168.1.1 address available from its Ethernet ports. As the Ethernet switch is bridged to the COAX (Broadband) device, this means that the router may be remotely accessible to an attack should an aggressor push spoofed RFC 1918 packets to your public IP address. They will drop off of the WAN firewall, but will the local ethernet bind to 192.168.1.1 answer? Since the Ethernet ports are set to make up the outer bridge to the WAN, this seems plausible, making for a potential hole in this setup. So much for adding security with a better firewall!
Tuesday, January 12, 2010
Automating PDF Analysis
This post assumes a knowledge of basic LINUX commands. For help, consult the looping section of the BASH manual (http://linux.die.net/man/1/bash).
Suggested readings:
http://isc.sans.org/diary.html?storyid=7903
http://isc.sans.org/diary.html?storyid=7906
http://isc.sans.org/diary.html?storyid=7867
http://isc.sans.org/diary.html?storyid=7984
In response to these and other posts, I think it's time to get serious about 1) shortening the time from starting analysis to the determination of 'malicious' and 2) start tackling the massive numbers of these files swarming the enterprise. Both of these techniques require essentially the same techniques described above to me implemented in repeatable ways to script and automate them.
The malicious PDFs I've analyzed have a few things in common that will make this process easier. First, they almost always contain the dropper payload they want to execute. They usually come from free (gmail, yahoo, hotmail) or weakly secured (AOL, MSN) webmail accounts. And, best of all, the encoding scheme used to protect the droppers is always the same, a 255-byte decrementing XOR key.
So, to build a body of files for analysis, you want to start isolating or collecting all of the PDFs delivered from webmail accounts. Once you have these hundred or thousands of files, you need to start ripping through them and identifying the evil ones.
Before we start, get the latest version of Didier Stevens pdf-parser.py (http://blog.didierstevens.com/programs/pdf-tools/). Now, these pdf files sometime contain duplicate object numbers, lots of unlinked objects, and blobs in the unmapped spaces of the PDF (like after the %EOF tag). So, to begin, let's assume one hundred objects and start ripping all of the encoded/flated objects from the pdf. There will be a lot of blank objects since some don't exist in the PDF. Get rid of those with a remove statement on 0-length files.
$ mkdir pdf.analysis
$ cd pdf.analysis
$ cp ../1.pdf .
$ for (( i=0; i<100;> $i.out; done
$ rm `ls -l | egrep " 0 2010-" | awk '{ print $8}'`
Now we have a collection of extracted objects. As mentioned in Bojan's ISC Diary (http://isc.sans.org/diary.html?storyid=7867), we can search for failed FlateDecodes. This may indicate an intersting PDF for follow-up and can be an easy malicious PDF indicator.
$ grep failed *
31.out: FlateDecode decompress failed
31.out: FlateDecode decompress failed
Binary file 35.out matches
52.out: FlateDecode decompress failed
52.out: FlateDecode decompress failed
The malicious PDFs contain a dropper that is encoded. We've seen simple XOR encoding before, but the nefarious folk of the world appear to have moved into rotating XOR encoding techniques. The key is either incremented or decremented by some amount for every byte processed. When the keyset rotates to the end of the 0x00-FF scale, it turns the corner and picks up at the other end. So, to deal with this, I updated a previously written multi-byte XOR script to handle 256-byte rotating XOR keys with a given offset. Pair it with a for loop to cycle through all 256 possible start keys, and the encoded blob will be decoded and discovered with a simple GREP for a known string. Here's how it works. In this example, I had already located and carved the unknown blob from the PDF capsule. However, for automation you would can pass the entire PDF file and just not worry about the other bytes that will get mulched. We're only looking to identify the EXE, not carve it at this point.
$ for ((i=0; i<256; i++)); do echo $i; perl multi-xor-v2.pl -f 1.pdf -o $i.ex_ -k "$i" -R -1; done $ grep -i KERNEL *
Binary file 0.ex_ matches
Apparently the ROTXOR key starts at 0x00 and rotates at a decrement of -1 for every byte processed. The rotation is typical for PDFs of the day, though I have also seen different start points. Now that we have our decoded blob, the rest can be disposed of.
$ mv 0.ex_ Carved_decoded_ROTXOR255_key0_step-1.exe
$ rm *.ex_
$ ls
1.pdf Carved_decoded_ROTXOR255_key0_step-1.exe multi-xor-v2.pl
The .EXE can be run through standard analysis routines to discover the call-outs and second stage drops. This PDF is definitely malicious.
So, to take it to step two, addressing the large numbers of these PDFs, just take the above steps, codify into a script, and run in another loop.
$ mkdir pdf.analysis
$ cp *.pdf pdf.analysis
$ cd pdf.analysis
$ find . -type f -name *.pdf | while read i; do echo "processing $i"; ../analyzepdf.sh "$i"; done && find . -type -f -name *.exe | while read i; do echo "MATCH: $i"; done | tee matches.txt
The above loop creates a directory for analysis, creates an array of the PDF files available to be analyzed, and initiates the analysis script for each of them. The anlaysis script will create analysis subdirectories for each PDF, perform the above analysis steps and decodings, identify the interesting tidbits, and leave behind the interesting artifacts. When the loop finishes, the FIND command is used to locate the executables left behind, and create a notification for those PDFs found to have drops, recording this information to the matches.txt file.
Now you can revisit the PDFs identified in the matches .txt file and carve the droppers out of them.
$ dd if=1.pdf of=c1.bin bs=1 skip=27598 count=834887 && xxd c1.bin | less
834887+0 records in
834887+0 records out
834887 bytes (835 kB) copied, 1.7896 s, 467 kB/s
Apply the ROTXOR decoder scripts to the blob to reveal the executable.
$ for ((i=0; i<256; i++)); do echo $i; perl multi-xor-v2.pl -f c1.bin -o $i.ex_ -k "$i" -R -1; done $ grep KERNEL *
Binary file 0.ex_ matches
$ mv 0.ex_ Carved_decoded_ROTXOR255_key0_step-1.exe
$ rm *.ex_
$ ls
c1.bin Carved_decoded_ROTXOR255_key0_step-1.exe multi-xor-v2.pl
Suggested readings:
http://isc.sans.org/diary.html?storyid=7903
http://isc.sans.org/diary.html?storyid=7906
http://isc.sans.org/diary.html?storyid=7867
http://isc.sans.org/diary.html?storyid=7984
In response to these and other posts, I think it's time to get serious about 1) shortening the time from starting analysis to the determination of 'malicious' and 2) start tackling the massive numbers of these files swarming the enterprise. Both of these techniques require essentially the same techniques described above to me implemented in repeatable ways to script and automate them.
The malicious PDFs I've analyzed have a few things in common that will make this process easier. First, they almost always contain the dropper payload they want to execute. They usually come from free (gmail, yahoo, hotmail) or weakly secured (AOL, MSN) webmail accounts. And, best of all, the encoding scheme used to protect the droppers is always the same, a 255-byte decrementing XOR key.
So, to build a body of files for analysis, you want to start isolating or collecting all of the PDFs delivered from webmail accounts. Once you have these hundred or thousands of files, you need to start ripping through them and identifying the evil ones.
Before we start, get the latest version of Didier Stevens pdf-parser.py (http://blog.didierstevens.com/programs/pdf-tools/). Now, these pdf files sometime contain duplicate object numbers, lots of unlinked objects, and blobs in the unmapped spaces of the PDF (like after the %EOF tag). So, to begin, let's assume one hundred objects and start ripping all of the encoded/flated objects from the pdf. There will be a lot of blank objects since some don't exist in the PDF. Get rid of those with a remove statement on 0-length files.
$ mkdir pdf.analysis
$ cd pdf.analysis
$ cp ../1.pdf .
$ for (( i=0; i<100;> $i.out; done
$ rm `ls -l | egrep " 0 2010-" | awk '{ print $8}'`
Now we have a collection of extracted objects. As mentioned in Bojan's ISC Diary (http://isc.sans.org/diary.html?storyid=7867), we can search for failed FlateDecodes. This may indicate an intersting PDF for follow-up and can be an easy malicious PDF indicator.
$ grep failed *
31.out: FlateDecode decompress failed
31.out: FlateDecode decompress failed
Binary file 35.out matches
52.out: FlateDecode decompress failed
52.out: FlateDecode decompress failed
The malicious PDFs contain a dropper that is encoded. We've seen simple XOR encoding before, but the nefarious folk of the world appear to have moved into rotating XOR encoding techniques. The key is either incremented or decremented by some amount for every byte processed. When the keyset rotates to the end of the 0x00-FF scale, it turns the corner and picks up at the other end. So, to deal with this, I updated a previously written multi-byte XOR script to handle 256-byte rotating XOR keys with a given offset. Pair it with a for loop to cycle through all 256 possible start keys, and the encoded blob will be decoded and discovered with a simple GREP for a known string. Here's how it works. In this example, I had already located and carved the unknown blob from the PDF capsule. However, for automation you would can pass the entire PDF file and just not worry about the other bytes that will get mulched. We're only looking to identify the EXE, not carve it at this point.
$ for ((i=0; i<256; i++)); do echo $i; perl multi-xor-v2.pl -f 1.pdf -o $i.ex_ -k "$i" -R -1; done $ grep -i KERNEL *
Binary file 0.ex_ matches
Apparently the ROTXOR key starts at 0x00 and rotates at a decrement of -1 for every byte processed. The rotation is typical for PDFs of the day, though I have also seen different start points. Now that we have our decoded blob, the rest can be disposed of.
$ mv 0.ex_ Carved_decoded_ROTXOR255_key0_step-1.exe
$ rm *.ex_
$ ls
1.pdf Carved_decoded_ROTXOR255_key0_step-1.exe multi-xor-v2.pl
The .EXE can be run through standard analysis routines to discover the call-outs and second stage drops. This PDF is definitely malicious.
So, to take it to step two, addressing the large numbers of these PDFs, just take the above steps, codify into a script, and run in another loop.
$ mkdir pdf.analysis
$ cp *.pdf pdf.analysis
$ cd pdf.analysis
$ find . -type f -name *.pdf | while read i; do echo "processing $i"; ../analyzepdf.sh "$i"; done && find . -type -f -name *.exe | while read i; do echo "MATCH: $i"; done | tee matches.txt
The above loop creates a directory for analysis, creates an array of the PDF files available to be analyzed, and initiates the analysis script for each of them. The anlaysis script will create analysis subdirectories for each PDF, perform the above analysis steps and decodings, identify the interesting tidbits, and leave behind the interesting artifacts. When the loop finishes, the FIND command is used to locate the executables left behind, and create a notification for those PDFs found to have drops, recording this information to the matches.txt file.
Now you can revisit the PDFs identified in the matches .txt file and carve the droppers out of them.
$ dd if=1.pdf of=c1.bin bs=1 skip=27598 count=834887 && xxd c1.bin | less
834887+0 records in
834887+0 records out
834887 bytes (835 kB) copied, 1.7896 s, 467 kB/s
Apply the ROTXOR decoder scripts to the blob to reveal the executable.
$ for ((i=0; i<256; i++)); do echo $i; perl multi-xor-v2.pl -f c1.bin -o $i.ex_ -k "$i" -R -1; done $ grep KERNEL *
Binary file 0.ex_ matches
$ mv 0.ex_ Carved_decoded_ROTXOR255_key0_step-1.exe
$ rm *.ex_
$ ls
c1.bin Carved_decoded_ROTXOR255_key0_step-1.exe multi-xor-v2.pl
Sunday, October 18, 2009
Quick shell script to extract the contents of an image
- assuming TSK is installed, the image "image.dd" is in the local directory, and a directory "files" exists for the extracts. Change the offset and disk type to suit. This particular image was a 1GB FAT16 USB drive image.
# for i in `fls -Dr -m / -f fat -o 63 image.dd | grep -v ".Trash" | grep -v "(deleted)" | cut -f 2 -d"|"`; do mkdir files/$i; done
# for i in `fls -Fr -m / -f fat -o 63 image.dd | grep -v ".Trash" | grep -v "(deleted)" | cut -d "|" -f 2,4`; do echo $i; icat -o 63 -f fat image.dd `echo $i | cut -d "|" -f 2` > files/`echo $i | cut -d "|" -f 1`; done
# for i in `fls -Dr -m / -f fat -o 63 image.dd | grep -v ".Trash" | grep -v "(deleted)" | cut -f 2 -d"|"`; do mkdir files/$i; done
# for i in `fls -Fr -m / -f fat -o 63 image.dd | grep -v ".Trash" | grep -v "(deleted)" | cut -d "|" -f 2,4`; do echo $i; icat -o 63 -f fat image.dd `echo $i | cut -d "|" -f 2` > files/`echo $i | cut -d "|" -f 1`; done
Monday, May 4, 2009
Idea for enterprise scanning
Pseudocode for an internal scanner. Attempts to combat environmental manipulation through self-integrity checking, but a better mechanism may be needed.
Assumed this operates in a client/server model with the server offering messages to clients in a one-to-many or several-to-many relationship. Ulitmately, the server should be able to post a request (hash list, updated files) and the clients should pull the list, self check, perform the tests, and report back. The central system should them be able to generate reports based on the results.
INteresting reports might include which scanned successfully, which didn't report, any anomalies discovered. All hashes are passed back to central, so the tool could be used for forensic anomalies, known discovery of artifacts, discovery of similar artifacts within a defined threshhold, or compliance applications (similarity or direct matching). The insider threat model could be integrated by allowing the tracking of defined critical documents within reporting systems.
Client structure follows:
internal scanner
pull updates and signatures. check sig, decode to mem, load hashes
provide non-DOM driver to access disk filesystem - driver client bindings
provide access to memory - Volatility
check self integrity of all components, static files
Walk VAD and dump all processes/dll injects to disk
identify self in proc dump and validate hash vice known
dump registry hives in memory
extract registry values for known hostiles - regripper
for each proc/file
perform static hash scanning >> hashfile
perform context piecewise hashing >> cphfile
compile xml/soap response
encrypt, sign, report back to central
Needed tools:
http://code.google.com/p/pyssdeep/
http://www.py2exe.org/
https://www.volatilesystems.com/default/volatility
http://www.regripper.net/
http://ssdeep.sourceforge.net/
http://www.indigostar.com/perl2exe.htm
Assumed this operates in a client/server model with the server offering messages to clients in a one-to-many or several-to-many relationship. Ulitmately, the server should be able to post a request (hash list, updated files) and the clients should pull the list, self check, perform the tests, and report back. The central system should them be able to generate reports based on the results.
INteresting reports might include which scanned successfully, which didn't report, any anomalies discovered. All hashes are passed back to central, so the tool could be used for forensic anomalies, known discovery of artifacts, discovery of similar artifacts within a defined threshhold, or compliance applications (similarity or direct matching). The insider threat model could be integrated by allowing the tracking of defined critical documents within reporting systems.
Client structure follows:
internal scanner
pull updates and signatures. check sig, decode to mem, load hashes
provide non-DOM driver to access disk filesystem - driver client bindings
provide access to memory - Volatility
check self integrity of all components, static files
Walk VAD and dump all processes/dll injects to disk
identify self in proc dump and validate hash vice known
dump registry hives in memory
extract registry values for known hostiles - regripper
for each proc/file
perform static hash scanning >> hashfile
perform context piecewise hashing >> cphfile
compile xml/soap response
encrypt, sign, report back to central
Needed tools:
http://code.google.com/p/pyssdeep/
http://www.py2exe.org/
https://www.volatilesystems.com/default/volatility
http://www.regripper.net/
http://ssdeep.sourceforge.net/
http://www.indigostar.com/perl2exe.htm
Friday, May 1, 2009
Opinion on SMTP Honeypots
honeyd is an infrastructure honeypot that refers to other services. it's a little heavy. If you are trying to emulate the Interweb on an open access point for research, it's great. For this, you want something more focused, either a honeytrapd type of service (dangerous on your border) or a full-time script running in its own process(s) to capture and handle load. Look at truman's (http://www.secureworks.com/research/tools/truman.html ) smtp script and consider reversing it's interally-focused intent to external. Add some support scripts for housekeeping and you should be good to go. Obviously, run in a dmz, with limited perms, on a box that is easily rebuilt and doesn't have other dependent, critical apps/processes. VM should be fine.
On Tue, Apr 28, 2009 at 12:28 PM, private investigation <xxx> wrote:
I tried to use honeyd but seems that honeyd cannot handle much of smtp request
On Tue, Apr 28, 2009 at 12:28 PM, private investigation <xxx> wrote:
I tried to use honeyd but seems that honeyd cannot handle much of smtp request
So You Thought You Were in Control of Your Friend List
Check this tool out! Facebook Controller:
http://my.opera.com/quakerdoomer/blog/2009/04/30/fbcontroller-facebook-controller-the-ultimate-facebook-controller-without-the-pa
Nice, using social media against you to subvert authentication controls, do recon, and manipulate data. Great POC!
http://my.opera.com/quakerdoomer/blog/2009/04/30/fbcontroller-facebook-controller-the-ultimate-facebook-controller-without-the-pa
Nice, using social media against you to subvert authentication controls, do recon, and manipulate data. Great POC!
SSH Command Monitoring
This was an interesting post from the secureshell list. Thanks Richard!
Hi "J",
you can do that with your unix/linux onboard tools. just attach strace
to the sshd process of the user you want to monitor:
strace -s 4096 -e trace=read -p PROCESS_ID
than have a look for the shell prompt (e.g.):
read(10, "\33]0;USERNAME@HOSTNAME:~\7"..., 16384) = 22
now you know that the FD (file handle) is 10 for the users ssh session terminal.
then you can do something like that:
strace -s 4096 -e trace=read -p 10417 2>&1 | grep -E '^read\(10,' |
grep -oE '".+"'
and you should get an output like:
"uname -a"
"\r\n"
"Linux HOSTNAME 2.6.29.1 #1 SMP Sat Apr 18 11:22:05 CEST 2009 i686
Intel(R) Core(TM)2 Duo CPU L7500 @ 1.60GHz GenuineIntel GNU/Linux\r\n"
"\33]0;USERNAME@HOSTNAME:~\7"
well, this will only work if you have root permission on the server
running sshd.
have fun,
richard
Hi "J",
you can do that with your unix/linux onboard tools. just attach strace
to the sshd process of the user you want to monitor:
strace -s 4096 -e trace=read -p PROCESS_ID
than have a look for the shell prompt (e.g.):
read(10, "\33]0;USERNAME@HOSTNAME:~\7".
now you know that the FD (file handle) is 10 for the users ssh session terminal.
then you can do something like that:
strace -s 4096 -e trace=read -p 10417 2>&1 | grep -E '^read\(10,' |
grep -oE '".+"'
and you should get an output like:
"uname -a"
"\r\n"
"Linux HOSTNAME 2.6.29.1 #1 SMP Sat Apr 18 11:22:05 CEST 2009 i686
Intel(R) Core(TM)2 Duo CPU L7500 @ 1.60GHz GenuineIntel GNU/Linux\r\n"
"\33]0;USERNAME@HOSTNAME:~\7"
well, this will only work if you have root permission on the server
running sshd.
have fun,
richard
Saturday, April 18, 2009
Open Source Security Information Manager - OSSIM
I've been playing with OSSIM (http://www.ossim.net) for the last week. The stand-alone installation from AlienVault was trivially easy, thanks guys! I was able to install a main 'trusted net' stand-alone and integrate a DMZ sub-sensor in an hour. The dashboard is pretty, with many reporting features and does a decent job of aggregating the infeed of data from the wide collection of tools it provides.
But...
Having put this Unified Threat Management (http://en.wikipedia.org/wiki/Unified_Threat_Management) device on the network, I find it to be the least secure thing out there. While AlienVault did an excellent job of bringing all of these wonderful security monitoring tools together, having the production interface on the main network acting as both collector, sensor, and admin access is a bad idea. It also uses so many products and services that it is terribly insecure itself. Having the main sensor in the trusted network isn't too bad for this, but having one of these in the more exposed DMZ makes me wary. OSSIM needs a lot of custom configuration to implement restricted access, split the collection interface to a promiscuous-only and have a separate admin interface. This can be done with taps to ensure only one-way traffic occurs, but that still leaves the service open to injection if one were to expect the box to be there. In all, the UTM sensor-with-everything idea needs to be rethought.
It's been fun to play with, but ultimately I'm going to explore running with some functionalized implementations that might prove more secure.
But...
Having put this Unified Threat Management (http://en.wikipedia.org/wiki/Unified_Threat_Management) device on the network, I find it to be the least secure thing out there. While AlienVault did an excellent job of bringing all of these wonderful security monitoring tools together, having the production interface on the main network acting as both collector, sensor, and admin access is a bad idea. It also uses so many products and services that it is terribly insecure itself. Having the main sensor in the trusted network isn't too bad for this, but having one of these in the more exposed DMZ makes me wary. OSSIM needs a lot of custom configuration to implement restricted access, split the collection interface to a promiscuous-only and have a separate admin interface. This can be done with taps to ensure only one-way traffic occurs, but that still leaves the service open to injection if one were to expect the box to be there. In all, the UTM sensor-with-everything idea needs to be rethought.
It's been fun to play with, but ultimately I'm going to explore running with some functionalized implementations that might prove more secure.
Subscribe to:
Posts (Atom)