server forensics: linux - ulisboa · server forensics: linux ... using some text-based...

Server Forensics: Linux

Part II.A. Techniques and Tools: Computer Forensics

CSF: Forensics Cyber-Security Fall 2015

Nuno Santos

Summary

2015/16 CSF - Nuno Santos 2

}  Linux forensics

}  Overview of Linux (Unix) systems

}  System subversion with rootkits

}  Rootkit detection techniques

Operating systems we will be focusing on


}  Desktop platforms: Windows

}  Server platforms: Linux

}  Mobile platforms: Android

Today!

Previous class

Our main focus when analyzing a server’s OS


}  On desktops, we were mostly concerned about evidence left by local users’ activities

}  On servers, we will look for evidence left by external agents that resulted in security violations to the OS

}  Such external agents typically consist of stealthy malware called rootkits

attack

rootkit

Overview of Linux systems


Linux belongs to the big family of Unix systems


Unix systems’ bootstrap sequence


}  At a high-level, Unix systems have same boot sequence

Unix systems’ architecture


}  Unix systems also have same arch, at a high-level }  The kernel can be extended with loadable modules

System subversion with rootkits


What’s wrong with this picture?


[root@dobro bin]# ls -‐a . dd igawk nisdomainname tar .. df ipcalc pgawk tcsh .. dmesg kbd_mode ping touch alsaunmute dnsdomainname kill ping6 tracepath arch doexec ksh ps tracepath6 ash domainname link pwd traceroute ash.static dumpkeys ln red traceroute6 aumix-‐minimal echo loadkeys rm true awk ed login rmdir umount

Let’s take a closer look


How can there be two “..” directories?...

[root@dobro]# ls -‐a . dd igawk nisdomainname tar .. df ipcalc pgawk tcsh .. dmesg kbd_mode ping touch

How did this happen?


}  This is actually:

mkdir <dot><dot><backslash><space><enter>

}  It creates a directory named “dot-dot-space”

[root@dobro bin]# mkdir ..\

What’s in this “mystery” directory?


Here’s a simple trick to hide malicious files in plain sight

[root@dobro bin]# cd ..\ [root@dobro .. ]# ls -‐l total 24 -‐rw-‐r-‐-‐r-‐-‐ 1 root root 0 Dec 15 12:19 rootkit_file_01 -‐rw-‐r-‐-‐r-‐-‐ 1 root root 0 Dec 15 12:19 rootkit_file_02 -‐rw-‐r-‐-‐r-‐-‐ 1 root root 0 Dec 15 12:19 rootkit_file_03 -‐rw-‐r-‐-‐r-‐-‐ 1 root root 0 Dec 15 12:19 rootkit_file_04 -‐rw-‐r-‐-‐r-‐-‐ 1 root root 0 Dec 15 12:19 rootkit_file_05 -‐rw-‐r-‐-‐r-‐-‐ 1 root root 0 Dec 15 12:19 rootkit_file_06

Rootkits


}  The behavior of the operating system can be affected by the presence of rootkits

}  Rootkits are a category of malware which has the ability to hide itself and cover up traces of activities

Sample scenario


}  A rootkit runs a password logger on a compromised system

Compromised system

Password Logger

Email with sniffed passwords encoded

using some text-based steganography

techniques

Another example


}  A rootkit installs a backdoor on the compromised system

Compromised system

Backdoor communication with backdoor (covert channel

in HTTP or DNS)

Shell

Rootkit goals


1.  Remove evidence of original attack and activity that led to rootkit installation

2.  Hide future attacker activity (files, net connections, processes) and prevent it from being logged

3.  Enable future access to system by attacker

4.  Install tools to widen scope of penetration

5.  Secure system so other attackers can’t take control of system from original attacker

Rootkits include several attacker tools


}  Attack tools installed after an intruder gained access: }  Backdoor programs }  Network sniffers }  Log cleaners }  File / process / user hiding tools }  etc.

Rootkit tools: Backdoor programs


}  Backdoor is an unauthorized way of gaining access to a program, online service or an entire computer system }  Let attackers log in to the hacked system w/o using an exploit again

Examples Description

Login Backdoor Modify login.c to look backdoor password before stored password

Telnetd Backdoor Trojaned the “in.telnetd” to allow attacker gain access with backdoor password

Services Backdoor Replacing and manipulate services like “ftp”, “rlogin”, even “inetd” as backdoor to gain access

Cronjob backdoor Backdoor could also be added in “crontjob” to run on specific time for example at 12 midnight to 1 am

Library backdoors Shared libraries can be backdoor to do malicious activity including giving a root or administrator access

Kernel backdoors This backdoor is basically exploiting the kernel

Rootkit tools: Sniffers and wipers


}  Packet sniffers }  Packet Sniffer is a program and/or device that monitor data

traveling over a network, TCP/IP or other network protocol }  Used to listen or to steal valuable information off a network; many

services such as “ftp” and “telnet” transfer their password in plain text and it is easily capture by sniffer

}  Log-wiping utilities }  Log file are the lists actions that have occurred, e.g., in UNIX, wtmp

logs time and date user log in into the system }  Log file enable admins to monitor, review system performance and

detect any suspicious activities }  Deleting intrusion records helps prevent detection of the intrusion

Rootkit tools: Miscellaneous attacker tools


}  DDOS program }  To turn the compromised server into a DDOS client such as, trinoo

}  IRC program }  Connects to some remote server waiting for the attacker to issue a

command (e.g., to trigger a distributed denial of service attack)

}  System patch }  Attacker may patch the system after successful attack; this will

prevent other attacker to gain access into the system again

}  Password cracker }  Vulnerability scanners }  Hiding utilities

}  Utilities to conceal the rootkit files on compromised system

Rootkit types


}  Command-level rootkits }  Replace user-level programs

}  Library-level rootkits }  Replace system libraries

}  Kernel-level rootkits }  Modify the kernel itself

Command-level subversion


}  Command-level rootkits hide the presence of malware by making changes to system commands }  Based on principle: To suppress bad news, silence the messenger

}  Table shows typical command-level rootkit modifications

Evolution of command-level rootkits


}  First rootkits came with network sniffers to collect user names and passwords, e.g., the esniff program

}  Later versions came with remotely controlled agents for distributed denial-of-service attacks, e.g., T0rn

Command-level rootkit evasion


}  To evade detection, early rootkits not only replaced system utility sw but also erased records in system logs }  Some rootkits even gave modified system utilities the same file

time stamps and cyclic redundancy check (CRC) values as the original files

}  Later command-level rootkits simply install modified programs that hide the presence of malware

Installation concealment


}  Use a subdirectory of a busy system directory like /dev, /etc, /lib, or /usr/lib

}  Use dot files, which aren’t in ls output

}  Use spaces to make filenames look like expected dot files: “. “ and “.. “

}  Use filenames that system might use }  /dev/hdd (if no 4th IDE disk exists) }  /usr/lib/libX.a (libX11 is real Sun X-Windows)

}  Delete rootkit install directory once installation is complete

Rootkit types


}  Command-level rootkits

}  Library-level rootkits

}  Kernel-level rootkits

Library-level subversion


}  Instead of replacing system utilities, rootkits can hide their existence by making changes at the next level down in the system architecture: the system run-time library

}  A good example is redirecting the open() and stat() calls }  The purpose of these modifications is to fool file-integrity-checking

software that examines executable file contents and attributes }  By redirecting the open() and stat() calls to the original file, the

rootkit makes it appear as if the file is still intact }  However, execve() executes the subverted file

Example of library-level subversion


}  Redirect specific open() system calls

real_syscall3() is a macro (not entirely shown) that modifies the standard _syscall13() macro real_syscall3 is defines our real_open() function that invokes Sys_open system call

Rootkit types





Kernel-level subversion


}  Just like user-level rootkits, kernel-level rootkits are installed after a system’s security has been breached

}  Kernel-level rootkits compromise the kernel }  Kernel runs in supervisor processor mode }  Thus, the rootkit gains complete control over the machine

}  Advantage: stealth, e.g., }  Runtime integrity checkers cannot see rootkit changes }  All programs in the system can be affected by the rootkit }  Open backdoors/sniff network without running processes

Kernel rootkit operation


}  The purpose of many kernel rootkits is to hide malware processes, files, and network ports (& the rootkit itself)

}  There are to sides to information hiding

}  Output: the kernel must censor the output from system calls that produce a list of processes, files, network ports, etc.

}  Input: any attempts to manipulate a hidden process, file, network port, and so on must fail as if the object did not exist }  May also redirect system calls to subvert operation of software that

checks integrity of executable files

Methods to inject rootkit code into a kernel


1.  Loading kernel module into a running kernel }  Uses official LKM interface, hence it’s easier to use }  Hide module names from external (/proc/ksyms) & internal tables }  Might also intercept syscalls that report on status of kernel modules

2.  Injecting code into the memory of a running kernel that has no support for module loading

}  Involves writing new code to unused kernel memory via /dev/kmem and activating the new code by redirecting, e.g., a system call

3.  Injecting code into the kernel file or a kernel module file }  These changes are persistent across boot, but require that the

system is rebooted to activate the subverted code

Early kernel rootkit architecture


}  Based on system-call interposition: Early kernel rootkits subvert syscalls close to the process-kernel boundary

Rootkit interposition code


}  To prevent access to a hidden file, process, and so on, rootkits redirect specific system calls to wrapper code

}  To prevent rootkit disclosure, syscalls that produce lists of files, etc., are intercepted to suppress info to be hidden

System-call interposition: good and bad


}  Advantage: The code is easy to understand }  The change is made at a point that is close to the user of

those system calls

}  Disadvantage: Many system calls need to be intercepted }  E.g., to hide the existence of a file, need to intercept all

system calls that have file name arguments, about 50 syscalls

}  Alternative: subvert the kernel at a level that is closer to the information being hidden

Later kernel rootkit architectures


}  Based on file system interposition: Rootkits based on interposition at object interfaces subvert several FS methods

Hiding file names


}  Of all the operations provided by the VFS interface the lookup() operation is of particular interest }  lookup() translates the pathname into underlying FS object }  Redirect can hide a file from all syscalls that access a file by name

}  Doesn’t work for syscalls that create a new directory entry }  E.g., link(), mkdir(), socket(), etc. }  Unless the malware does not handle this, operation will fail

Rootkit detection techniques


Detection techniques for each rootkit type





Approaches to detecting command-level rootkits


1.  If a rootkit listens for connections, the network port will be visible to an external network port scanner

2.  Some tools can reveal the names of all directory entries, including hidden or deleted files

3.  Corrupted versions of ps and similar hide malware processes, but these can still be found using, e.g., the /proc file system

4.  Deleted login/logout records in the wtmp file leave behind holes that can be detected using an appropriate tool

5.  Ifconfig might report that a network interface is not in sniffer mode, but we can query the kernel for the interface status

6.  CRC checksums reported by compromised cksum, can be detected using MD5 or SHA1

7.  When examining a low-level copy of the file system on a trusted machine, all hidden files and modifications will be visible

Example of exposing a command-level rootkit1/7


}  We examine two utilities of the T0rn rootkit for Linux }  This rootkit was in widespread use in 2001

}  First, search the /bin/ls executable file using strings and grep

}  The file name /usr/src/.puta/.1file looks very suspicious



}  If we try to list the /usr/src/.puta directory, the ls command hides the name, as we would expect:

}  However, the directory name still shows up when we use the echo command, together with the .* wildcard expansion feature that is built into the command shell



}  File .puta/.1file is a rootkit configuration file }  Has list of file and directory names that must remain hidden }  Contain the malware program files, config files, and data files



}  To detect modified process status check against /proc }  Table shows that ps command is hiding with ID 153



}  The system utilities that were replaced by the rootkit do a good job of hiding process 153

}  Not only it is censored by process status tools like ps, but also it does not show up with network status tools such as netstat

}  However, it does not replace the lsof command, which can help reveal the purpose of process 153



}  File name /usr/sbin/nscd suggests that it is a system program, but comparison with uncompromised systems shows this program exists only in later Linux versions }  nscd: name service cache daemon



}  Connecting with telnet to TCP port 47017 on the local machine confirms it is a backdoor process }  In this case we are welcomed by the opening banner of what

appears to be an SSH server

}  Find more about the T0rn rootkit at: https://www.sans.org/security-resources/malwarefaq/t0rn_rootkit.php

Automated approaches for detection


}  Each rootkit differs slightly in its approach to hiding the presence of malware and therefore requires us to take a slightly different approach to detect it

}  An example of software that automates the search for known rootkits is the Chkrootkit toolkit }  Runs on a dozen different UNIX platforms }  Recognizes more that fifty different rootkits }  Looks for deleted login / logout records, signatures of replaced system

utilities, rootkit configuration files and directories, missing processes, and signs of kernel-level subversion

Detection of library-level rootkits


}  Strawman: check library hashes. But, would an MD5 or SHA-1 hash reveal library modifications?

}  Not necessarily: while the runtime linker uses the low-level open() system call when it accesses the modified library file, md5sum use the fopen() library routine }  Therefore, can be redirected to the unmodified library file

}  Workaround: rootkit detection tools need to carry their own trusted copy of the system library routines

Detection of kernel-level rootkits


}  Kernel rootkits may be exposed because they introduce little inconsistencies into a system

}  Some may show up externally, in the results from system calls that manipulate processes, files, kernel modules, etc.

}  Others show up only internally, in the contents of kernel data structures }  E.g., hidden objects occupy some storage even though the

storage does not appear in kernel symbol tables

Inconsistencies that may reveal kernel rootkits


}  Output of tools that bypass the file system can reveal information that is hidden by compromised FS code }  E.g., TSK

}  Oversight }  E.g., the modification time of an important system directory is

changed, not to the contents of that directory

}  Inconsistencies in the results from process-manipulating system calls and from the /proc file system }  E.g., in reporting a process as “not found”)



}  Unexpected behavior of some system calls }  E.g., when the Adore rootkit is installed, setuid() – change process

privileges – will report success for some parameter value even though the user does not have sufficient privileges

}  E.g., when the Knark rootkit is installed, settimeofday() – set the system clock – will report suffess for some parameter values even though is should always fail when invoked by an unprivileged user

}  Directory hard link count inconsistencies }  The hard link count of a directory, as reported by stat(), should

equal the number of subdirectories, as reported by getdents() }  If a directory is being hidden, it may show up as a missing hard link



}  Modifications to kernel tables, such as system call table or the virtual FS table }  May be detected after the fact by reading kernel memory

via /dev/kmem }  Or by examining kernel memory from inside with a forensic

kernel module such as Carbonite

}  Modifications to kernel tables or kernel code may be detected using a kernel module that samples critical data structures periodically



}  Checking against raw data sources }  Modifications that hide files can show up as inconsistencies

between information from the raw disc device and information returned by the kernel file system code

}  Modifications that hide network ports, processes, or kernel modules may be exposed by reading kernel memory and comparing the contents of kernel data structures with results from system calls

Example of kernel toolkit detector tool


}  Findrootkit: examines kernel mem through /dev/kmem and checks consistency from multiple sources: }  The /dev/ksyms symbol table, with kernel function and data

addresses and sizes }  The in-kernel module list, with executable code and data

segment addresses and sizes of loaded kernel modules }  The in-kernel “text arena” table, with executable code

segment addresses and sizes }  The function addresses in the system call jump tables and in

file system operation jump tables }  The executable code and data segment sizes, as specified in

the symbol tables of kernel module files

Example of kernel toolkit detector tool


}  Findrootkit can produce modification reports }  Report example for a Solaris kernel

}  Changes to (a) the file system operations table, and (b) the system-call jump table

Conclusions


}  Many attacks to operating systems are performed through rootkit software

}  There are three different types of rootkits depending on the OS layer targeted by the rootkit: command-, library-, and kernel-level rootkits

}  Depending on the rootkit, the forensic analyst needs to employ different rootkit-detection techniques

References

2015/16 CSF - Nuno Santos

}  Primary bibliography }  Dan Farmer, Wietse Venema, Forensic Discovery, Chapter 5

61

Next class

CSF - Nuno Santos

}  Mobile forensics: Android

2015/16 62

server forensics: linux - ulisboa · server forensics: linux ... using some text-based...

Documents