15-Minute Linux Live Analysis

Dr. Phil (Polstra)Bloomsburg University of Pennsylvania

What is this talk about?

● Determining with some certainty if you have been hacked● In a matter of minutes● With minimal disturbance to the subject system● Automating the process with shell scripting

Why should you care?

● Someone calls you about a suspected breach● You need to need to figure out if they were hacked

● Quickly so as to avoid further harm to your client● Without destroying evidence● Without taking down a critical machine

Who is the handsome man before you?

● Digital forensics professor at Bloomsburg University● Programming since age 8 (in Assembly since age 10)● Known to hack electronics from age 12● Linux user from its start● Author USB

Forensics

Insert coolCover art

Roadmap

● Opening a case● Talking to the users● Mounting known-good binaries● Minimizing disturbance to the system● Collecting Data● Automation with scripting● Next steps if there is a breach

Opening a Case

● Decide on case name (date?)● Create a case folder on your laptop● Start making entries in your notebook

● Bound notebook with numbered pages● Easy to carry● Hard to insert/remove pages● No batteries required

First Talk to the Users

● They know the situation better than you● Might be able to tell a false alarm before digging in● Why did you call me?

● What they suspect● No internal experts or policy to use outsider?

● Why do you think there was an incident?● Everything they know about the subject system

USB Response Drive

● Contains known-good binaries● Minimum /bin, /sbin, /lib for same architecture● Might also grab /usr/sbin, /usr/bin, /usr/lib● Must be on an ext2, ext3, or ext4 partition

● Could contain a bootable Linux on another partition● This partition will probably be FAT ● Should be first partition● See http://linuxforensicsbook.com/hdvideos.html Chapter 1

Mounting Known-Good Binaries

● Insert response drive● Exec your bash binary● Set path to your binaries (and only your binaries)● Set LD_LIBRARY_PATH● Run all shell scripts as bash <script>

● Don't use she-bang (#!) in scripts!

Demo: Mounting Binaries

Minimize Disturbance to System

● You will always change the system a little● Goal is to

● Minimize memory footprint● Never write to subject media

● Two basic options● Save everything to your USB response drive● Send it over the network

Sending data over the network

● Better than USB drive due to caching● Use netcat

● Create a listener for “log” information on forensics workstation● Send “log” information from client● Also create a listener for interesting files on forensics

workstation– Spawn a new listener when files are sent

Setting Up Log Listener

● netcat -k -l 9999 >> case-log.txt● (-k) keep alive (-l) listen (>>) append● From subject

● {command} | netcat {forensic ws IP} 9999

● Let's use shell scripting to automate this● Shell not Python because we want to minimize memory

footprint

Automating the Log Listener

usage () {

echo "usage: $0 <case number>"

echo "Simple script to create case folder and start listeners"

exit 1

}

if [ $# -lt 1 ] ; then

usage

else

echo "Starting case $1"

fi

#if the directory doesn't exist create it

if [ ! -d $1 ] ; then

mkdir $1

fi

# create the log listener

`nc -k -l 4444 >> $1/log.txt` &

echo "Started log listener for case $1 on $(date)" | nc localhost 4444

# start the file listener

`./start-file-listener.sh $1` &

Automating the Log Client-Part 1

usage () {

echo "usage: $0 <IP> [log port] [fn port] [ft port]"

exit 1

}

# did you specify a file?

if [ $# -lt 1 ] ; then

usage

fi

export RHOST=$1

if [ $# -gt 1 ] ; then

export RPORT=$2

else

export RPORT=4444

fi

if [ $# -gt 2 ] ; then

export RFPORT=$3

else

export RFPORT=5555

fi

if [ $# -gt 3 ] ; then

export RFTPORT=$4

else

export RFTPORT=5556

fi

Automating the Log Client – Part 2

# defaults primarily for testing

[ -z "$RHOST" ] && { export RHOST=localhost; }

[ -z "$RPORT" ] && { export RPORT=4444; }

usage () {

echo "usage: $0 <command or script>"

echo "Simple script to send a log entry to listener"

exit 1

}

# did you specify a command?

if [ $# -lt 1 ] ; then

usage

else

echo -e "++++Sending log for $@ at $(date) ++++\n $($@) \n----end----\n" | nc $RHOST $RPORT

fi

Automating Sending Files

● Listener on forensics workstation listens for file name● When a new file name is received

● Create a new listener to receive file● Redirect file to one with correct name● Also log in the main case log (optional)

● On the client side● File name is sent● After brief pause send file to data listener port

Automating the File Listener

usage () {

echo "usage: $0 <case name>"

echo "Simple script to start a file listener"

exit 1

}

# did you specify a file?

if [ $# -lt 1 ] ; then

usage

fi

while true

do

filename=$(nc -l 5555)

nc -l 5556 > $1/$(basename $filename)

done

Automating the File Client

# defaults primarily for testing

[ -z "$RHOST" ] && { export RHOST=localhost; }

[ -z "$RPORT" ] && { export RPORT=4444; }

[ -z "$RFPORT" ] && { export RFPORT=5555; }

[ -z "$RFTPORT" ] && { export RFTPORT=5556; }

usage () {

echo "usage: $0 <filename>"

echo "Simple script to send a file to listener"

exit 1

}

# did you specify a file?

if [ $# -lt 1 ] ; then

usage

fi

#log it

echo "Attempting to send file $1 at $(date)" | nc $RHOST $RPORT

#send name

echo $(basename $1) | nc $RHOST $RFPORT

#give it time

sleep 5

nc $RHOST $RFTPORT < $1

Cleaning Up

# close the case and clean up the listeners

echo "Shutting down listeners at $(date) at user request" | nc localhost 4444

killall start-case.sh

killall start-file-listener.sh

killall nc

Collecting Data

● Date (date)● Clock skew on subject● Time zone on subject

● Kernel version (uname -a)● Needed for memory analysis● Might be useful for researching vulnerabilities

Collecting Data (continued)

● Network interfaces (ifconfig -a)● Any new interfaces?● Strange addresses assigned?

● Network connections (netstat -anp)● Connects to suspicious Internet addresses?● Strange localhost connections?● Suspicious ports?● Programs on wrong ports (i.e malware on port 80)

Collecting Data (continued)

● Open files (lsof -V)● What programs are using certain files/ports● Might fail if malware installed

● Running processes (ps -ef and/or ps -aux)● Things run as root that shouldn't be● No login accounts logged in and running things● Might fail if malware installed

Collecting Data (continued)

● Routing info (netstat -rn and route)● Routed through new interface● New gateways● Conflicting results = malware● Failure to run = malware

Collecting Data (continued)

● Mounted filesystems (mount and df)● Things mounted that shouldn't be (especially tempfs)● Mount options that have changed● Filesystem filling up● Disagreement = malware

● Kernel modules (lsmod)● New device drivers● Modules that have changed

Collecting Data (continued)

● Who is logged in now (w)● System accounts that shouldn't be allowed to login

● Who has been logging in (last)● System accounts that shouldn't be allowed to login● Accounts that don't normally use this machine

● Failed logins (lastb)● Repeated failures then success = password cracked

Collecting Data (continued)

● User login info (send /etc/passwd and /etc/shadow)● Newly created login● System accounts with shells and home directories● Accounts with ID 0● Accounts with passwords that shouldn't be there

Putting It Together with a Script

usage () {

echo "usage: $0 [listening host]"

echo "Simple script to send a log entry to listener"

exit 1

}

# did you specify a listener IP?

if [ $# -gt 1 ] || [ "$1" == "--help" ] ; then

usage

fi

# did you specify a listener IP?

if [ "$1" != "" ] ; then

source setup-client.sh $1

fi

# now collect some info!

send-log.sh date

send-log.sh uname -a

send-log.sh ifconfig -a

send-log.sh netstat -anp

send-log.sh lsof -V

send-log.sh ps -ef

send-log.sh netstat -rn

send-log.sh route

send-log.sh lsmod

send-log.sh df

send-log.sh mount

send-log.sh w

send-log.sh last

send-log.sh lastb

send-log.sh cat /etc/passwd

send-log.sh cat /etc/shadow

Running the Initial Scan

Have I been hacked?

Who is Johnn?

/etc/passwd

Why do these accounts have passwords?

/etc/shadow

Who's been logging in?Results from last

Who failed to login?

Results from lastb

Looks Like They Were HackedNow What?

Live Analysis

● Use techniques described to● Grab file metadata for key directories (/sbin, /bin, etc.)● Grab users' command history● Get system log files● Get hashes of suspicious files

● Dump RAM● Must compile LiME (off subject machine!)● Risky – can cause a crash

Dead Analysis

● Unless the machine absolutely cannot be taken offline it is strongly recommended that you shut it down and get a filesystem image

● If you cannot shutdown the machine● You can still get a filesystem image with dcfldd● You probably cannot use this evidence in court

More on Dead Analysis

● Filesystem analysis is much more mature and powerful than memory analysis

● The Linux support in Volatility is somewhat lacking● Relatively new addition to a new tool● Seems to fall down a lot with late 3.x and 4.x kernels

● None of the investigators I've talked to could come up with a case where evidence existed only in memory

Finding Out More

● Heard there was a new book out (1 kg+ of knowledge)● Resources on http://linuxforensicsbook.com● Feel free to talk to me later

Questions?