assessing the performance impact of the linux kernel audit trail
DESCRIPTION
Security breach into information systems is almost always caused by unpatched or unknown vulnerabilities. To counter this threat, system administrators often rely on effective auditing to discover such security violations. The benefits of auditing include accountability, reconstruction of higher-level events, intrusion detection, and problem analysis. Furthermore, the mere presence of auditing is a powerful deterrence to potential attackers. Despite these well-understood benefits, system administrators are reluctant to turn on auditing on their systems because of the fear of performance overhead. In this thesis, we design, implement and benchmark a low overhead auditing subsystem for the Linux 2.4 kernel.TRANSCRIPT
Honours Year Project Report
Assessing the Performance Impact of theLinux Kernel Audit Trail
By
Qiu Jiang
Department of Computer Science
School of Computing
National University of Singapore
2004/05
Honours Year Project Report
Assessing the Performance Impact of theLinux Kernel Audit Trail
By
Qiu Jiang
Department of Computer Science
School of Computing
National University of Singapore
2004/05
Project No: H089020Advisor: Prof. Sandeep KumarDeliverables:
Report: 1 Volume
Abstract
Security breach into information systems is almost always caused by unpatched or unknownvulnerabilities. To counter this threat, system administrators often rely on effective auditing todiscover such security violations. The benefits of auditing include accountability, reconstructionof higher-level events, intrusion detection, and problem analysis. Furthermore, the mere pres-ence of auditing is a powerful deterrence to potential attackers. Despite these well-understoodbenefits, system administrators are reluctant to turn on auditing on their systems because ofthe fear of performance overhead. In this thesis, we design, implement and benchmark a lowoverhead auditing subsystem for the Linux 2.4 kernel.
Subject Descriptors:C5 Computer System ImplementationD.4.6 Security and Protection
Keywords:audit, kernel, operating system, linux
Implementation Software and Hardware:Linux Kernel 2.4
Acknowledgement
I would like to express my utmost gratitude to my project supervisor, Assistant Professor
Sandeep Kumar, who provided me with inspiration and guidance that set me on the right course
in this study.
Also, my hat goes off to Jeremy Banford and Nenad Ocelic for their attempt to contribute
a high quality kernel-level auditing tool to the open source community.
List of Figures
3.1 Audit records are generated at system calls . . . . . . . . . . . . . . . . . . . . . 123.2 Security sensitive system calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143.3 Auditor device with double buffering . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.1 Integer, float and double opreations . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2 Memory and file system results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.3 Process related results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244.4 Communication results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4
List of Tables
3.1 System call group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Threat level classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.1 Linux kernel compilation macrobenchmarks, time in seconds . . . . . . . . . . . . 27
5
Table of Contents
Title 1
Abstract 2
Acknowledgement 3
List of Figures 4
List of Tables 5
1 Introduction 11.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Report Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Related Work 42.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.2 Audit Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 Built-in Logging Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2.2 Custom Audit Trails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Auditing Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 System Design and Implementation 93.1 Naive Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Improved Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2.1 Kernel Patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2.2 Auditor Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.2.3 Audit Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4 Evaluation 194.1 Test Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.2 Microbenmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.2.1 Arithmetic Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.2.2 Memory and File System Operations . . . . . . . . . . . . . . . . . . . . . 224.2.3 Process Related Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . 244.2.4 Communication Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3 Macrobenchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.4 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.4.1 Ideal Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284.4.2 Worse Case Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6
4.4.3 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5 Conclusions and Future Work 315.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.1.1 Tamper Proof Audit Trail . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.1.2 LSM Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
References 33
7
Chapter 1
Introduction
Information security is becoming an ever critial aspect of computing today. Widely publicized
security breaches, for example (Leyden, 2004) and (CERT, 2003), highlight the need for a
forensic tool that will greatly assist system administrators in understanding the modus operandi
of an attack and assessing the extent of damage to their systems. This information can also help
fix the vulnerabilities that lead to successful exploitation. In this thesis, we describe the design
and implementation of a kernel-level subsystem that can record process activity at a very fine
granualarity. We also argue that, contrary to popular belief, there is little performance impact
resulting from a well-implemented kernel-level auditing system.
1.1 Background
An audit trail, as defined in (NIST, 1995), is a sequential record of computer events about an
operating system, an application, or user activity. The process of constructing an audit trail
and its subsequent examination is called auditing. The benefits of auditing include individual
accountability, reconstruction of higher-level events, intrusion detection, and problem analy-
sis. Additionally, the knowledge of the existence of an auditing mechanism can be a powerful
deterrence to potential attackers.
The ultimate goal of auditing is the reconstruction of high-level and human-understandable
events that attackers commit to breach security. Examples of these high-level events are user-
1
logon, buffer-overflow, privilege-escalation and password-stealing. However, currently there is
no mechanism that can be used to describe such high-level events. Instead, what is often used
is a record of system events that are at a very fine granularity. It is hoped that with such
a record, system administrators can understand the chain of high-level events that lead to
successful attacks and, more importantly, patch the very security vulnerability which made the
attacks possible in the first place. To this end, operating system vendors have implemented a
kernel-level auditing solution that records the arguments and return values of every system call
executed on the system.
With such a fine granularity of record generation and storage comes the fear of performance
overhead as a result of auditing. Many users and system administrators are loathe to install a
full featured auditing system, or to turn it on during normal daily activity because of the fear
of performance degradation on the host platform, in spite of the aforementioned benefits,
To better understand the cause of this performance degradation, and to investigate solutions
to mitigate the overhead, we proposed and implemented a system that significantly extends the
original LinuxBSM (Banford, 2001) (Ocelic, 2001) project. The improved LinuxBSM consists
of three components: (1) a set of kernel patches that augment the audit record generation of
the security-relevant system calls, (2) a kernel-space auditing device that provides a means for
user-space applications to interact with the kernel audit subsystem and, (3) a user-space audit
daemon that stores audit records into the local filesystem. These are described in detail in
Chapter 3.
1.2 Contributions
The contribution of our work is two fold. We have designed and implemented a kernel-level
security auditing system for the Linux operating system that significantly improves over the
most recent system (Banford, 2001) (Ocelic, 2001). Secondly, we have performed in-depth
benchmarking to quantify the actual performance overhead in practice. Based on these results,
we can say that a well designed and implemented auditing system incurs little overhead, so
system administrators can reap the full benefits of auditing without the fear of its performance
2
impact.
1.3 Report Organization
The report is organized as follows. We review the state-of-the-art in kernel-level auditing in
Chapter 2. Chapter 3 describes the design and implementation of our improved LinuxBSM.
Chapter 4 examines the performance impact of the auditing system, and we conclude and discuss
future work in Chapter 5.
3
Chapter 2
Related Work
2.1 Introduction
Given the benefits of auditing, many application-level software such as relational database
management systems and web servers provide comprehensive audit capabilities with extensive
information that can even be used in performance tuning. Even operating systems, such as the
recent Microsoft Windows Server 2003 (Microsoft, 2003) and Sun’s Solaris 10 (Sun, 2005) allow
comprehensive and highly configurable auditing policies.
Application-level auditing is commonly known as logging. They provide many benefits such
as system-health monitoring, application debugging and performance tuning. However, with
the specific aim of security auditing, we restrict our investigation to kernel-level auditing in the
following sections.
In this chapter, we survey the auditing tools available either as part of the operating system,
in section 2.2.1, or as a custom built solution in section 2.2.2. Then, in section 2.3, we look at
some audit benchmarks that were performed in the past and understand how it has become a
major obstacle to audit adoption.
4
2.2 Audit Tools
2.2.1 Built-in Logging Tool
At a basic level, the various Unix-like operating systems provide a logging utility called klogd
(Schwarz, 2000), the kernel logging daemon that intercepts and logs Linux kernel messages in
much the same way as the system logging daemon. The kernel can add an entry to the kernel log
by calling printk() that behaves almost identically to the C library printf() function. Invoking
the printk() function stores the new entry in a circular buffer and subsequently the user-space
klog daemon retrieves these messages from the buffer and feeds them into the log file via the
syslogd daemon.
Klogd is remarkable in that it aims to provide a unified logging framework to the kernel.
This enables critical kernel messages to be retrieved and reviewed easily by developers as well
as administrators and acted upon. However, klog entries are entirely customary and there
is no common standard on how, when and what to audit. Furthermore, the requirement for
security auditing mandates a custom auditing system that allows the effecient recording of
data-intensive audit trails. In recognizing this need, and in enabling rich and flexible auditing,
various operating system vendors, as well as the open source community, have implemented
their auditing solution.
2.2.2 Custom Audit Trails
The SunShield Basic Security Module (Sun, 2000) provides an auditing system for both SunOS
(Sunshield) and Solaris (Sunshield BSM). The BSM permits the actions of specific users to be
recorded and written to an audit file. Sun’s BSM is the originial inspiration to develop a high
quality audit tool for the open source Linux operating system, which is known as LinuxBSM.
LinuxBSM (Banford, 2001) is an initiative at University of California, Davis to provide a
comprehesive auditing package for Linux that is fully compliant with the U.S. Government’s C2
standards for security, defined in the standards document ’Trusted Computer System Evaluation
Criteria’ (DoD, 1985) or more commonly called the Orange Book. Specifically, the LinuxBSM
5
is an auditing tool that aims to bring the capabilities of Sun’s Solaris Basic Security Module
to Linux. LinuxBSM monitors and records system events on the kernel level, equipping system
adminstrators a mechanism to trace the activities in the system. The project’s development
stopped in November 2000 and its capabilities at that time included auditing a total of six
system calls for the Linux 2.2 kernel.
LinuxBSM2 (Ocelic, 2001) is the next evolution from LinuxBSM and it consists of several
architectural improvements over the original LinuxBSM, including a flexible system call audit
record generator. This tool allows an administrator to specify desired audit record fields for
new system calls in a plain text file. The file is used in inserting the relevant source codes to
enable auditing of these system calls automatically. In this way, kernel developers can avoid a
certain amount of manual duplication of identical code section in audit record construction. The
project is also targeted at the same Linux 2.2 kernel and the numbers of system calls audited
remains at six. Meanwhile, major kernel changes took place in 2001 and the first release of new
version 2.4 kernel was released at the same time. LinuxBSM2 cannot be used in the new kernel
due to these breaking changes. Similarly, development stopped towards the end of 2001.
Linux Auditing Subsystem (SuSE, 2003), LAuS, is a very similar auditing system imple-
mented by SuSE Inc. The system is composed of kernel-resident and user-space components
that facilitate highly-configurable and robust logging of system call invocations. It is currently
distributed as part of Red Hat Enterprise Linux 3 and SuSE Linux Enterprise Server 8 that are
based on the Linux 2.4 kernel. The design principle of LAuS is significantly different from the
improved LinuxBSM and we will describe its key features as follows.
• LAuS adds a new field to the task structure to store audit parameters to enable selective
auditing of processes. If a processes is marked to be audited, then relevant audit fields
will be recorded at the system call dispatch point. This has the disadvantage of the need
to modify individual system call handlers for each supported hardware architecture.
• LAuS stores the audit records in a memory buffer in the kernel-space. This buffer is like a
queue in that the audit records are read, one at a time, by invoking the read() system call
on the auditor device. Such a design necessitates frequent disk write operations and hence
6
places a I/O load on host systems. Also, the rate at which audit records are generated
may outpace that of dequeuing from the memory buffer and this results in audit data loss.
The improved LinuxBSM solves these problems by avoiding an architecture-dependent de-
sign and employing a double-buffer strategy, as detailed in Chapter 3.
The abovementioned audit solutions, including the improved LinuxBSM, share a common
need for kernel patches. This is because the Linux kernel, by default, does not have any provision
for system call interception or monitoring. Hence, a kernel patch is the precursor for any system
call level auditing systems. However, the recent development in the Linux Security Modules
(Wright, Cowan, Smalley, Morris, & Kroah-Hartman, 2002) project introduces a lightweight,
general purpose access control framework as loadable kernel modules. The LSM framework,
itself a set of kernel patches, is on the verge of being integrated into the mainstream kernel. It
provides access to various kernel objects such as processes, IPCs and file system. Any access
control policy can be implemented as a LSM security kernel module. Currently there are some
problems with LSM, such as the lack of support for persistent security attribute binding without
resorting to external metadata storage. However, once LSM is stablized in the mainstream
kernel, the improved LinuxBSM can leverage the access to auditable kernel objects without
having to place system call hooks manually.
2.3 Auditing Benchmarks
Despite the well understood benefits of auditing, many system administrators are reluctant to
install an auditing system primarily due to two reasons.
1. Auditing tools have not been standardized and often the format of audit records are
customary to individual applications. As a result, audit records from different auditing
systems exist as information islands and there is no coherent view across these potentially
useful data. This is a classical data warehousing problem and solutions to it include
standardizing audit format for the various auditing sources, as proposed by (Bishop, 1995).
Alternatively, we can perform extraction, transform and load (ETL) by custom-built data
7
adapters and store the result into a central repository. This method is favored by hybrid
intrusion detection systems such as Prelude (Mathieu Blanc, 2003).
2. The second major hurdle to widespread adoption of auditing is due to the fear of its
performance impact on the host system. To this end, we find surprisingly little literature
on documented performance benchmarks on auditing. (Nitzberg, 1994) performed an
early analysis of the performance impact of the audit component available in the HP-UX
version 8.0, a hybrid between AT&T System V and BSD Unix, and found that the worse
case impact is as high as 85% degradation. This might explain the fear of implementing an
auditing solution. More recently, (Veronique Abily, 2000), used the SunShield BSM (Sun,
2000) to measure the performance impact of an intrusion detection system and discovered
that the auditing component contributed to about 4% performance hit. This result is
much more encouraging than the earlier study. However, the result is buried as part of
benchmarking process of an intrusion detection system.
In chapter 4, we describe the benchmark results of our auditing system that provide con-
clusive evidence about the fact that auditing does not impact system performance in most
systems.
8
Chapter 3
System Design and Implementation
This chapter describes the design and implementation issues in the improved LinuxBSM. It
starts with a discussion on the obvious (and inefficient) approach to kernel auditing in section
3.1. Section 3.2 explains the approach employed by the improved LinuxBSM to what and how
to audit in the improved LinuxBSM.
3.1 Naive Approach
As argued in (Sekar, Bowen, & Segal, 1999), regardless of how an attack is delivered, any dam-
age to a host is effected via the system calls made by a process running on the host computer.
Therefore, auditing system calls is an efficient way of analyzing (potential) security breaches.
Note that this is different from auditing policies in the Microsoft Windows Server family (Mi-
crosoft, 2003) where a level of abstraction above system calls is provided.
A naive approach to system call auditing would involve ’capturing’ system calls in the ar-
chitecture dependent system call handler1. As user-space programs invoke system calls, threads
of execution trap into the kernel from where the system call is dispatched. At this point, an
audit record containing the system call number and associated arguments can be generated and
written to the local file system. However, there are at least two problems with this approach.
1. Invoking system calls can have different post-conditions and return values. For example,1For example, in the Linux kernel, the system call handler resides in entry.s
9
the read() function call can succeed or fail and when it succeeds, the file offset can have
different values based on how many bytes are read. As a result, if only a set of generic
parameters are recorded in the audit trail, then it will be remarkably more difficult to
distinguish one system call from another.
2. More seriously, if every system call results in a disk write operation to the file system, it
is clear that auditing will become a bottleneck to system performance.
3.2 Improved Approach
To overcome the abovementioned problems, the improved LinuxBSM consists of three compo-
nents, a set of kernel patches to the security sensitive system calls, a kernel device driver that
records audit events in memory buffers (shared with the user-space daemon), and a user-space
daemon that periodically writes to the local file system. This design enhances the availability
of the host on which it runs while providing the benefits of auditing. We describe the three
components in the following sections.
3.2.1 Kernel Patches
The main design policy of the system is to maximize its utility and minimize its load in a
production environment. To maximize its utility, we must recognize the ingenuity of attackers
and their pattern of intrusions. In an ideal situation, we will audit all system calls. A system
administrator will have a complete record of events to reconstruct the steps of an attack.
However, such an approach will not only overwhelm the system administrator with mostly
normal audit records, the system on which the audit trail runs will incur a significant perfor-
mance hit. Performance degradation due to auditing on Unix is reported (Nitzberg, 1994) to
range from negligible impact for the arithmetic component of the benchmark to 85% degradation
for system call overhead tests.
Therefore, we need to strike a balance between security and performance. (Bernaschi,
Gabrielli, & Mancini, 2002) divided the Linux system call set into nine groups, as shown in
Table 3.1 according to their intended functionalities. They assert that it is safe to disregard
10
Table 3.1: System call group
Group Functionality
1 File system and device
2 Process
3 Loadable kernel module
4 Memory
5 Time and Timer
6 Interprocess Communication
7 System Information
8 Reserved
9 Not Implemented
system calls from group 4 to 9 as they cannot be used to subvert a system. For example,
unimplemented system calls (group 9) obviously cannot be used to subvert a system. So are
reserved (group 8) and system information related (group 7) calls. More interestingly, system
calls that deal with memory management (group 4), timers (group 5) and interprocess commu-
nication (group 6) are useless to an attacker who seeks to control either the running system or
the running process. We are left with system calls from group 1 to group 3 where either the file
system, processes or kernel modules are involved.
They also classifed the system calls, as in Table 3.2 according to four levels of security
threats, from allowing full control of the system to harmless. The result is that a total of 102
system calls are identified as security sensitive.
In our implementation, we apply a set of kernel patches to a stock Linux kernel version
2.4 as shown in Figure 3.1. These patches are placed at critical system call points to capture
security related information to a buffer. The set of system calls chosen are based on the study
done by (Bernaschi et al., 2002), with the exception of signals and real-time signals.
11
Table 3.2: Threat level classification
Threat Level Description
1 Allows full control of the system
2 Used for a denial of service attack
3 Used for subverting the invoking process
4 Harmless
Figure 3.1: Audit records are generated at system calls
12
As can be seen from Figure 3.2, the system calls are categorized into three threat levels.
• Level 1 refers to system calls that allow full control of the system.
• Level 2 identifies those calls that can be used for a denial of service attack on the host
machine
• Level 3 includes system calls that can be used for subverting the invoking process.
Note that by convention, the smaller the threat level number, the more dangerous are the
system calls. Also, there exists a total order across the threat levels; for if a system call allows
the attacker to take full control of a system, he can subsequently launch a denial-of-service
attack on the local machine trivially.
Together, these system calls deal with file systems, devices, processes and module manage-
ments that are the core functionality of the Linux kernel. By auditing these specific system
calls, we can discover actions that lead to successful or attempts of security breaches. For ex-
ample, various attempts have been made in the past to effectively examine audit trails in order
to discover important events that lead to a security breach. Such a forensic discovery process
can be greatly assisted by two types of tools.
The first type of log analyzers, such as Xlogmaster(Greve, 1998) and Swatch(Hansen &
Atkins, 1993), focus mainly on the effective filtering of audit events according to user-defined
rules. The problems of these tools soon become obvious to practioners that they require
highly experienced professionals who know what they are looking for from audit trails. Se-
curity breaches are often the result of successful exploitation of unprecedented vulnerabilities
in operating systems or application programs. This calls for a tool that can automatically trace
from compromised objects to the initial break-ins.
The answer to that is the second type of audit trail analyzers, such as Tudumi(Takada &
Koike, 2002) amd BackTracker(King & Chen, 2003). In short, they employ techniques from
infomation visualization to present the chain of events in a graphical user interface (GUI). In
this thesis, we merely state that auditing security related system calls can lead to a successful
identification of security vulnerabilities.
13
Figure 3.2: Security sensitive system calls
14
3.2.2 Auditor Device
Since we have previously argued that write operations to the file system is expensive and should
be minimized, we create a character device in the kernel-space that writes audit records as they
are created by the hooks in the modified system calls to memory buffers. The reason that the
auditor is a character device and not a block device is that we do not require the nonsequential
access to the memory buffer. Nonsequential access to block devices such as floppy drives and
CD-ROM drives are not relevant to the auditor device. At run time, the auditor device will fill
up a memory buffer with audit records sequentially and a user-space audit daemon will clear
the audit records once the memory stream is full, also sequentially. Implementing it as a block
device would incur considerable effort with no apparent benefits. The other important task
of this auditor is to signal to the user-space audit daemon that the memory buffer is full and
the audit daemon should copy the contents in the buffer out. In this way, disk write is only
performed periodically and the periodicity can be adjusted via the size of the buffer based on
empirical results of performance impact on the system.
Storing audit records in a memory buffer solves the problem of frequent write but it creates
another problem. When the content of a full buffer is being copied to the user-space, some
audit records will be lost as the the buffer is in the read mode2. At first thought, it is tempting
to solve this problem by simply implementing the memory buffer as a queue data structure. In
this way, as soon as the buffer is full, we can start dequeuing from the other end of the buffer
to accommodate the incoming audit records. However the rate of clearing the content into
user-space might not equal or exceed the rate at which new audit records are generated. Hence,
audit loss becomes inevitable in some circumstances. In the security sensitive environment, we
must assume that an attacker has access to the mechanism with which auditing is done. He
will exploit this vunerability so that the record of the key events are lost during the so-called
buffer jams.
In our improved LinuxBSM, we apply the double-buffer strategy, shown in Figure 3.3, to
avoid audit record loss. Specifically, the kernel auditor device associate two memory buffers2From the perspective of the user-space daemon
15
and label them as the read and write buffer respectively. From the user-space daemon’s point
of view, read buffer refers to the daemon’s reading from this buffer. Write buffer is the memory
region that audit records are appended as they are generated. Once the write buffer is full, the
device driver sends a signal to the user-space daemon and immediately swaps the two buffers.
In this way, while the audit daemon writes the content of read buffer to the local file system,
fresh audit records are written to the newly appointed write buffer. The double-buffer strategy
is reminiscent to graphics rendering devices in that while the front buffer is rasterized onto
screen display the back buffer can be used to store the content of the next frame to render.
This design is due to the original LinuxBSM.
It might be argued that circumstance might occur where audit records are generated at a
rate that is faster than that of being cleared by the user-space audit daemon and hence propose
a triple-buffering strategy. We believe that while triple-buffering certainly adds a higher level
of assurance against audit record loss and in fact, a variable number of buffers can be used
in an adaptive manner depending on the rate of audit growth, empirical results indicate that
double-buffering is sufficient even at high-load situations, as we can see from Chapter 4.
16
Figure 3.3: Auditor device with double buffering
17
3.2.3 Audit Daemon
The communication between the kernel-space and user-space in the Linux operating system
is facilitated through the use of signals. Signals, in this case, are used as software interrupts
that allow asynchronous notifications from kernel-space to user-space about events being raised.
Specifically in our design, as soon as we reach the point where storing one more audit record
will overflow the write buffer, the kernel notifies a user-space audit daemon with a SIGIO signal
to swap the contents in the buffer out of the kernel-space. A daemon is similar to a service in
the Windows operating system that is an autonumous background process. The audit daemon
copies the content from the read buffer in the kernel and writes to the local file system partition
or sends the audit data via network to another computer.
Before the audit daemon is running, the kernel auditor device will not generate any audit
records. The audit daemon is started with a set of command line parameters including the range
of process numbers, user ids and group ids to audit. These allow the selective auditing of events
that are deemed useful by the system administrator. Besides setting the audit parameters, the
audit daemon also registers the signals and attaches signal handlers to them. In this way, it can
perform the necessary actions such as writing to filesystem and closing the audit device based
on the received signals. Once the initialization of opening the device, setting up the memory
buffers and registering signal handlers are done, the audit daemon informs the kernel with a
specific IOCTL command. As soon as the kernel is aware that the audit daemon is running, it
can start auditing predefined system calls.
18
Chapter 4
Evaluation
In this thesis, we benchmark the performance overhead of our improved LinuxBSM that audits
a total of 82 security related system calls. Following the standard practice of benchmarking
the kernel, for example as done for Linux Security Modules (LSM) (Wright et al., 2002), we
use lmbench (Larry W. McVoy, 1996) for microbenchmarking. lmbench is developed specifi-
cally to measure the performance of core kernel system calls and facilities such as file system
access, context switching and memory access. For macrobenchmarking, we follow the oft used
convention of measuring kernel compile times.
4.1 Test Environment
We compare the standard Linux 2.4.22 kernel against the 2.4.22 kernel incorporating the im-
proved LinuxBSM with the audit daemon running. The tests are run on a single-processor 1.6
GHz Pentium 4 computer with 256MB of SDRAM and one 20GB Ultra ATA/133 hard disk
drive. In most tests, the performance overhead of the improved LinuxBSM is within the bound-
ary of the inevitable experimental errors. In certain cases we even have better results with
auditing and that is almost certainly due to random errors that are part of any experiment.
In addition, the size of each buffer, which is used by the auditor as described in Chapter 3, is
configured to be 250 kilobytes. The results are shown in section 4.2 and 4.3.
19
4.2 Microbenmarks
4.2.1 Arithmetic Operations
Figure 4.1 shows that, for basic arithmetic opreations, besides empirical disrepancies, there is
no difference between the unpatched kernel and improved LinuxBSM. This is to be expected as
these CPU-bound tasks do not invoke any system calls that we audit.
20
Figure 4.1: Integer, float and double opreations
21
4.2.2 Memory and File System Operations
Just as auditing has no performance impact on arithmetic operations, so is its lack of overhead
on memory latencies, as shown in Figure 4.2. In our test machine we have both level 1 and
level 2 cache and access to these memory is effected through architecture specific assembly code
where no auditing is performed. Furthermore, main and random memory access are not audited
too because so far attackers are unable to manipulate kernel memory directly without crashing
the system. Since no auditing is done to memory operation, no consistent performance impact
could possibly be observed.
On the other hand, file system operations are fully audited due to their sensitive nature. In
the Linux operating system, most objects, such are directories and devices, can be considered
as part of the file system. As a result, we expect and observe a large proportion of filesystem
related audit records. Therefore, this is the most likely bottleneck of our auditing mechanism.
However, Figure 4.2 shows encouraging results as we fail to observe significant performance
differences.
22
Figure 4.2: Memory and file system results
23
Figure 4.3: Process related results
4.2.3 Process Related Benchmarks
The performance overhead of auditing context switch, which occurs when the scheduler calls
context switch() to run the next process in the run queue, is shown in Figure 4.3. Benchmarks
are performed on 2, 8 and 16 processes with 0, 16 and 64KB of data associated with the switching
processes. Again, except for random errors, we fail to observe audit-related performance impact.
Also shown in Figure 4.3 is process related benchmarks such as process creation fork() and
overlay exec() system calls. These system calls, among others, are audited but we do not see
any discernible performance difference.
24
4.2.4 Communication Benchmarks
Finally, we measure the bandwitdth and latency of memory related operations and interprocess
communications. Again, we fail to observe, as shown in Figure 4.4, any significant difference
between the unpatched kernel and the modified kernel with the auditor running.
25
Figure 4.4: Communication results
26
Table 4.1: Linux kernel compilation macrobenchmarks, time in seconds
unpatched auditd running % Overhead with auditd
313 316 1.0
4.3 Macrobenchmarks
We perform the widely used kernel compilation macrobenmark, measuring the time to build
the Linux kernel. As before, we run the compilations on a single-processor 1.6 GHz Pentium 4
computer with 256MB of SDRAM and 20GB Ultra ATA/100 disk.The command make bzImage
is executed with the results shown in Table 4.1.
4.4 Analysis
Our observation of the negligible performance overhead resulting from kernel auditing requires
some explanation. We present an approximate calculation of the performance impact here which
is in line with the observed results and therefore provides credence to it.
The memory size, B, used by the auditor is configured to be 250 kilobytes for each buffer.
The buffers are described in detail in Chapter 3. Empirical observation shows that the auditor-
induced disk write occurs at 5 second frequency during intensive I/O operations.
The hard disk drive running on the test PC has the following relevant parameters.
• Number of sectors per track, denoted by P: 63
• Number of bytes per sector, denoted by Q: 512
• Track-Track seek time, denoted by s: 0.8 ms
• Average seek time, denoted by S: 8.5 ms
• Rotational Delay, denoted by R: 4.2 ms
• Data Transfer time, denoted by D: 0.595 ms per track
27
4.4.1 Ideal Scenario
In the ideal scenario where the partition to which audit data is written is unfragmented, the
audit daemon approximately writes to consecutive free blocks and little time is lost because
of seek delay. Under such an assumption, we have T, the total time, required for the write
operation as
T =B
PQ∗ (S + R + D)
where B is the buffer size, P is the number of sectors per track, Q is the number of bytes per
sector, S is the average seek time, R is the rotational delay and D is data transfer time. The
first multiplier BPQ calculates the number of tracks written. The second multiplier, (S +R+D)
calculates the time required to write one full track. The result of this multiplication is the total
time, T, required for the write operation.
Hence, at every 5 second intervals we spend
25000063 ∗ 512
∗ (8.5 + 4.2 + 0.595) ≈ 106ms
in write operations.
This is about 2% (106 ∗ 10−3/5 ≈ 0.0212) and that is close to the performance overhead
observed in our benchmarks.
4.4.2 Worse Case Scenario
In the worse case scenario, the partition is heavily fragmented due to frequent read-write op-
erations. In this case, the disk access incurs a seek on every sector that is written. In this
scenario,
T =B
PQ∗ (S + R + D + s)
Hence, at every 5 second intervals we spend
25000063 ∗ 512
∗ (8.5 + 4.2 + 0.595 + 0.8) ≈ 109ms
in write operations.
This is again about 2% (109 ∗ 10−3/5 ≈ 0.0218) performance overhead.
28
4.4.3 Remarks
With these hardware parameters, we understand that the negligible audit performance overhead
is due to the rapid advances in hardware technology. It is also worth noting that lmbench
benchmarking suite emulates both I/O intensive and typical interactive sessions in its testing.
The result of it therefore covers both scenarios in which auditing is highly desirable.
While lmbench provides us with a set of performance measurements of typical system oper-
ations on the Linux kernel, we aim to measure the worst case scenario with the audit subsystem
turned on. In order to do that, we create a program that repeatedly calls open() and close()
on a file in the local file system 10,000,000 times. The process that executes this program does
virtually nothing except invoking audited system calls and it provides us with a satisfactory
upper-bound of the performance overhead of the audit subsystem.
int main()
{
int fd;
int counter = 0;
while (counter < 10000000)
{
fd = open("abc");
close(fd);
count++;
}
return 0;
}
Executing the above program on the same test machine takes about 74.2 and 45.7 seconds with
auditing turned on and off repsectively. This indicates an upper bound of 62.4% performance
overhead. In reality, processes invoke system calls in a manner that is more akin to lmbench’s
simulation than to our explicitly constructed upper bound. Across this spectrum of system call
29
intensities, system administrators will have to understand the opreating environment and make
a judgment call on whether to turn on kernel-level auditing.
30
Chapter 5
Conclusions and Future Work
5.1 Future Work
5.1.1 Tamper Proof Audit Trail
One requirement for forensic analysis on compromised computers is that the audit trails be
tamper proof. Such a framework is described in (Schneier & Kelsey, 1999), and we plan to
study and implement the framework to our improved LinuxBSM. This will remarkably increase
the reliability to LinuxBSM as a forensic tool.
5.1.2 LSM Integration
As described in earlier, we intend to port our auditing mechanism to the LSM framework. This
has two advantages.
1. As part of a standardized effort to provide enhanced security to the Linux operating
system, our auditing system can be more easily accepted by system administrators.
2. The current improved LinuxBSM is based on the Linux 2.4 kernel. By porting it to the
LSM framework, it automatically works with the latest Linux 2.6 kernel.
31
5.2 Conclusions
In this thesis, we provide a much more fine-grained auditing system for the Linux operating
system to the Linux 2.4 kernel. Also, we discover that given a well implementd auditing system,
performance overhead is negligible as compared to an unpatched kernel. With these, we believe
the scale of auditing adoption by system administrators will improve considerably and security
vulnerabilitiy exploits can be discovered and rectified in a much more timely fashion.
32
References
Banford, J. (2001). Linuxbsm. http://linuxbsm.sourceforge.net/, , July, 2001.
Bernaschi, M., Gabrielli, E., & Mancini, L. V. (2002). Remus: A security-enhanced operatingsystem. ACM Transactions on Information and System Security, 5 , February, 2002, 36–61.
Bishop, M. (1995). A standard audit trail format. Proceedings of the 18th NIST-NCSC NationalInformation Systems Security Conference (pp. 136–145), Baltimore, Maryland, October,1995.
CERT (2003). Cert advisory ca-2003-23 rpcss vulnerabilities in microsoft windows.http://www.cert.org/advisories/CA-2003-23.html, , September, 2003.
DoD (1985). Trusted computer system evaluation criteria. Washington, DC 20301-1000: De-partment of Defense.
Greve, G. C. F. (1998). The xlogmaster. http://www.gnu.org/software/xlogmaster/xlogmaster.html,, June, 1998.
Hansen, S. E., & Atkins, E. T. (1993). Automated system monitoring and notificationwith swatch. Proceedings of the Seventh Systems Administration Conference (LISA VII)(USENIX Association: Berkeley, CA), , November, 1993, 145.
King, S. T., & Chen, P. M. (2003). Backtracking intrusions. Proceedings of InternationalConference on Graph Theory, Ann Arbor, MI 48109-2122, October, 2003.
Larry W. McVoy, C. S. (1996). lmbench: Portable tools for performance analysis. USENIXAnnual Technical Conference (pp. 279–294), San Diego, CA, January, 1996: USENIXAssociation.
Leyden, J. (2004). Students find 44 unix flaws as homework.http://www.theregister.co.uk/2004/12/16/unix flaw homework/, , December, 2004.
Mathieu Blanc, Laurent Oudot, V. G. (2003). Global intrusion detection: Prelude hybrid ids(Technical report). Villeurbanne, France: ExaProbe.
Microsoft (2003). Windows server 2003 product documentation. Redmond,WA, 98052: Mi-crosoft Corporation.
NIST (1995). Introduction to computer security: The nist handbook. Gaithersburg, MD: NIST.
Nitzberg, S. (1994). Performance benchmarking of unix system auditing. Master’s thesis,Monmouth college, Monmouth, IL.
33
Ocelic, D. (2001). Linuxbsm2. http://linuxbsm2.sourceforge.net/, , December, 2001.
Schneier, B., & Kelsey, J. (1999). Secure audit logs to support computer forensics. ACMTransactions on Information and System Security, 2 (2), May, 1999, 159–176.
Schwarz, M. A. (2000). Take command: klogd: The kernel logging daemon. Linux Journal, ,August, 2000, 20.
Sekar, R., Bowen, T., & Segal, M. (1999). On preventing intrusions by process behaviormonitoring. Proceedings of the Workshop on Intrusion Detection and Network Monitoring(pp. 29–40), Santa Clara, CA, April, 1999.
Sun (2000). Sunshield basic security module guide. Palo Alto, CA: Sun Microsystems.
Sun (2005). Solaris 10 product documentation. Santa Clara, CA 95054: Sun Microsystems.
SuSE (2003). Linux audit-subsystem design documentation. Waltham, MA 02451: Suse Inc.
Takada, T., & Koike, H. (2002). Tudumi: Information visualization system for monitoring andauditing computer logs. Proceedings of Sixth International Conference on InformationVisualisation, London, England, July, 2002.
Veronique Abily, M. D. (2000). Benchmarking a distributed intrusion detection system basedon asax: Preliminary results. Lecture Notes in Computer Science, 1907 , April, 2000.
Wright, C., Cowan, C., Smalley, S., Morris, J., & Kroah-Hartman, G. (2002). Linux securitymodules: General security support for the linux kernel. Proceedings of the 11th USENIXSecurity Symposium (pp. 17–31), San Francisco, CA, August, 2002: USENIX Association.
34
Project Information Page
Project Type : Honours Year Project
Project Area : Software System
Project Title : Assessing the Performance Impact of the Linux
Kernel Audit Trail
Project No : H089020
Students Name : Qiu Jiang
Project Advisor : Prof. Sandeep Kumar
Date of Completion : April 2005
Deliverables : Report 1 Volume
Implementation
Software and Hardware : Linux Kernel 2.4
AbstractSecurity breach into information systems is almost always caused by unpatched or unknownvulnerabilities. To counter this threat, system administrators often rely on effective auditingto discover such security violations. The benefits of auditing include accountability,reconstruction of higher-level events, intrusion detection, and problem analysis. Furthermore,the mere presence of auditing is a powerful deterrence to potential attackers. Despite thesewell-understood benefits, system administrators are reluctant to turn on auditing on theirsystems because of the fear of performance overhead. In this thesis, we design, implement andbenchmark a low overhead auditing sub-system for the Linux 2.4 kernel.
Subject Descriptors:C5 Computer System ImplementationD.4.6 Security and Protection
Keywords:audit, kernel
35