Copyright 2004 CyberGuard Corporation. All rights reserved.

Secure to the Core: The Next Generation Secure Operating System from CyberGuardPaul A. Henry MCP+I, MCSE, CCSA, CCSE, CFSA, CFSO, CISSP, CISM, CISASenior Vice President CyberGuard Corp

A CyberGuard CorporationWhite PaperSeptember 2004

Secure to the Core: CyberGuard’s Next Generation Operating Environment Page 2

What is a Secure Firewall Operating System? While industry experts may debate which firewall architecture provides the optimum level of security, few would disagree about the critical importance of a secure firewall Operating System. Many vendors claim their network security products are built upon a “hardened OS.” What you will find in virtually all cases, is that the vendor simply turned off -- or removed -- unnecessary services, and then patched the OS for known vulnerabilities. Clearly, this is not a “hardened OS” but really a “patched OS.” A true hardened OS is one in which the vendor has modified the kernel source code to enforce a security perimeter between the operating system, firewall software and network stack. Correctly implemented, this eliminates the risk that a service running on the hardened OS could be exploited by a hacker to obtain root level privilege and then “highjack” the firewall. CyberGuard’s heritage in developing secure real-time operating systems for the US Department of Defense is evident today in our Mandatory Access Control (MAC) and Multi-Level Security (MLS) operating system technologies. Together, MAC/MLS completely insulate the operating system layers used to inspect and transport packets from those that allow the firewall to be configured and managed. A user who has been authenticated and authorized for administrative access can never leave the firewall and connect to the network. Likewise, a user seeking network connectivity can never gain access to the firewall’s management functions. Even of the firewall were to be compromised, no further network incursions would be possible. In this White Paper, we review the sophisticated techniques CyberGuard used to implement this same level of security in our next generation firewall product, and the resulting technical and operational benefits. The Evolution of CyberGuard’s Next Generation of Security Products CyberGuard’s experience building secure operating systems dates back to 1967, when the company -- then known as Datacraft -- began building mission-critical real-time simulation and control systems. In 1987, with 20 years of OS experience under its belt, CyberGuard – now a part of the Harris Corporation -- and AT&T Federal Systems began joint development of an Orange Book B1 MLS/OS and LAN solution. It has been estimated that 75 man years of development time was invested in this critical government project. Development concluded in 1991 and the NCSC B1 Evaluation Cycle began. In 1992 the evaluation concluded with the award by the United States Department of Defense for the world’s first TCSEC B1 OS and Network LAN certification. No other firewall vendor to date has ever been able to match this achievement.

Secure to the Core: CyberGuard’s Next Generation Operating Environment Page 3

In response to customer demands, CyberGuard decided to migrate to a more mainstream Operating System that would preserve the same levels of security while supporting a broader set of hardware platforms. CyberGuard purchased the source code rights to SCO UnixWare and ported many of the security mechanisms that enabled us to achieve our previous B1 certification. This purpose built Operating System has served our clients well since 1996. In fact not a single CERT bulletin has ever been written against our SCO UnixWare based firewall solution. The advent of 64 bit processing architectures and high performance platforms from Intel (among others), creates the foundation for CyberGuard to offer a new set of highly scalable purpose-built security solutions. This was our primary motivation in developing CG Linux™; a next generation secure Operating System based on the Linux kernel. CyberGuard’s CG-Linux Operating System takes full advantage of the security controls that helped us earn our original TC SEC B1 certification, while incorporating the additional security features available when u tilizing a Linux-based kernel. Collectively, these capabilities allo w us to offer the highest level of Operating System security available in a commercial firewall product. The table below summarizes the security features built into the respective UnixWare and CG-Linux OS.

Security Mechanisms Features Linux UnixWare Description Discretionary Access Control (DAC) Yes Yes Limit’s a user's access to a file or directory. Based

on owner/group IDs and permission bits. Multilevel Security (MLS) Yes Yes Creates a barrier between non-administrative users,

processes, and data, and the corresponding set of users, processes, and data of the firewall security systems. Based on a modified Bell-LaPadula security model.

Mandatory Access Control (MAC) Yes Yes Enforces mandatory system-wide policies that cannot be changed at the discretion of individual users. Based on a modified Bell-LaPadula security model.

Capability (Privileges) Yes Yes Divides the super user privilege into a number of discrete privileges that can be assigned to multiple users or programs.

Roles Yes Yes Organizes administrative duties in to roles that can be assigned to multiple administrative users. Used to provide separation of duties.

Auth Yes No RSBAC mechanism for restricting the ID to which a program may switch (setuid).

File Flags Yes No RSBAC model for providing fine-grained access control over file system objects (files, directories, symbolic links etc.).

PAM User Level Authentication Yes No Enables the use of longer passwords and more granular transaction logging.

Audit Yes Yes Audits security relevant events at a very granular level, enabling forensic analysis and accountability.

Secure to the Core: CyberGuard’s Next Generation Operating Environment Page 4

Discretionary Access Control (DAC) is an access control service that enforces a security policy based on the identity of system users (or groups of users) and their respective authorizations to access files and other system resources.

There are three categories of users:

1. Owner – The owner of the file 2. Group – Users in the same group as the owner 3. Other – Everyone else

There are three kinds of authorizations:

1. Read – Users may read the file or list the contents of a directory 2. Write - Users may write to the file or add a new file to the directory 3. Execute – Users may execute the file or lookup a specific file

DAC is used primarily to limit a user's access to a file or directory. This access is considered to be discretionary because the owner determines “at his or her discretion” who receives these read, write and execute access rights.

Multi Level Security

CyberGuard’s implementation of Multi Level Security (MLS) is based on a modified version of the Bell-LaPadula security model. MLS provides the security mechanisms and enforcement systems needed to allow data with different degrees of sensitivity to be securely maintained and accessed on the same system. Essentially, MLS provides a barrier between the non-administrative users, processes, and data, and the corresponding set of users, processes, and data of the firewall security systems.

A process inherits its sensitivity level from its respective use. Therefore the permissions for the process determine the level of sensitivity of the data that the process is permitted to act upon.

MLS enforcement enables an administrative user to run a process that reads or modifies a firewall configuration file, while preventing a non-administrator -- running the exact same process -- from accessing or modifying the firewall configuration data.

Mandatory Access Controls

CyberGuard’s implementation of Mandatory Access Controls (MAC) enhances and complements DAC by enforcing MLS rules within the CG-Linux kernel.

MAC enforces mandatory system-wide policies that cannot be changed at the discretion of individual users. Most commercial Operating Systems provide support for DAC only.

Discretionary Access Control

Secure to the Core: CyberGuard’s Next Generation Operating Environment Page 5

Many in the security community believe that MAC is inherently more secure than DAC, because it eliminates some of the most prevalent “incorrect permissions” mistakes made by administrators trying to implement DAC in security systems. In addition, a DAC-based OS can be exploited by a Trojan Horse program to alter the DAC security settings, thereby allowing an escalation of privileges for a malicious user.

Limiting Super User Privileges

In a typical UNIX or Linux operating system, the Super User -- otherwise known as the Root User -- has total control over all aspects of the operating system and the tasks and programs it is running.

CyberGuard’s CG-Linux divides Super User privileges across multiple users and system processes, effectively reducing the Super User’s “authority.” This increase security by reducing the dependence on a single entity that could otherwise assert total control over all security processes.

Should a malicious hacker ever achieve Super User sta tus, they would gain very limited control over the firewall and its security-related processing.

Role-Based Management

CyberGuard has always provided extremely granular control over the separation of administrative duties. This makes it possible to provide selective administrative access while ensuring that no one can gain complete control over the firewall and its security processing. CyberGuard has enhanced and extended this “role” functionality in its next generation firewall product by incorporating these features into the CG Linux OS.

This provides additional security by reducing the possibility that a firewall administra tive duty can be circumvented by an operating system administra tive function.

Auth Module

In a typical UNIX environment, a Super User can change the authority level at which a process operates within the OS. This is explicitly prevented by CG-Linux, which offers extensive controls over which privilege changes are permitted and by whom they can be applied.

This eliminates the common ploy of the Privilege Escalation Attack, in which a hacker alters the au thority level of a process in order to increase their privileges within the OS and gain control over the firewall.

Secure to the Core: CyberGuard’s Next Generation Operating Environment Page 6

To further enhance OS security, CG-Linux provides additional granularity in its standard file access controls. The file flags are complementary to the standard Linux file permissions and can only be altered or changed by an authenticated security officer.

Flag Checked for

execute_only FILE, FIFO, SYMLINK search_only DIR read_only FILE, FIFO, SYMLINK, DIRwrite_only FILE, FIFO, SYMLINK secure_delete FILE no_execute FILE no_delete_or_rename FILE, FIFO, SYMLINK, DIRappend_only FILE, FIFO, SYMLINK add_inherited FILE, FIFO, SYMLINK, DIR

By extending the access control capabilities of Linux standard file permission, CG-Linux provides a level of granular contro l far beyond what is available in a commercial OS.

Pluggable Authentication Module (PAM)

PAM is a UNIX programming interface that enables third-party security methods to be used. By using PAM, multiple authentication technologies, such as RSA, DCE, Kerberos, smart card and S/Key, can be added without changing any of the login services, thereby preserving existing system environments.

CyberGuard has incorporated PAM into CG-Linux, affording numerous security enhancements, including:

1. Support for longer passwords 2. Password and account expirations / verifications 3. Improved transaction logging (including information on the user and login

address) 4. RSBAC restrictions to provide more granular access control

The incorporation of an enhanced version of PAM into CG-Linux affords a great deal of flexibility and expandability for authentication-related servic es in current and future CyberGuard products.

Audit and Alert Systems

CyberGuard has always offered superior logging capabilities, and stored log files in binary format to preserve data integrity. CG-Linux provides additional security benefits

File Flags

Secure to the Core: CyberGuard’s Next Generation Operating Environment Page 7

by dramatically improving the performance and flexibility of the firewall Alert and Audit Systems.

Because these systems can reach deeper and wider into the CG Linux audit trail, the granularity and amount of data that can be logged is far more extensive and granular than ever before. The binary format dramatically improves search and query performance while the increased granularity and breadth of information allows you to drill down into the log file with increased precision during your queries.

The Alert system has also been dramatically enhanced and now includes fully user configurable alerts. This includes OS performance data that is typically absent with firewalls built upon a commercial OS.

Operating System Performance

The evolutionary move to Linux allows CyberGuard to leverage new OS efficiencies that significantly improve performance and throughput.

CG-Linux offers full support for 64 bit processors, for the Intel Itanium processor family, as well as non-Intel-based platforms. This provides compelling opportunities to deploy CG-Linux based security solutions on embedded devices.

The firewall architecture has changed from a UnixWare Stream model to a the faster and more efficient Socket model implemented in Linux

Linux fully supports threads. This significantly improves performance, memory management and overall resource efficiency.

In Conclusion

CyberGuard historically has provided the most secure and best performing application proxy based firewalls in the industry. The legacy continues with the evolution of our next generation CG-Linux OS. Providing unparalleled performance and security, CyberGuard is well positioned to remain as the preferred solution for securing the world’s most demanding networks.

Threads provide a useful programming technique for dividing work into separate pieces. Programs that correctly use threads can run on multiprocessor systems, with each thread running on a separate CPU. Any slow process running on a single-CPU system can theoretically execute on an N-way multiprocessor in 1/N of the time.

CyberGuard Corporate Headquarters

CyberGuard Europe Limited Asmec Centre, Eagle House The Ring, Bracknell Berkshire, RG12, 1HB United Kingdom Phone: +44 (0) 1344 382550 Fax: +44 (0) 1344 382551 E-mail: info@cyberguard.co.uk www.cyberguard.com

Copyright© 2004 by CyberGuard Corporation. All rights reserved. This publication is intended for use with CyberGuard Corporation products by CyberGuard's personnel, customers and end users of CyberGuard's products. It may not be reproduced in any form without the written permission of CyberGuard Corporation. CyberGuard® is a registered trademark of CyberGuard Corporation. UnixWare® is a registered trademark of Santa Cruz Operations, Inc. All other trademarks are the property of their respective owners.

Quadrant Business Center350 SW 12th AvenueDeerfield Beach, FL 33442Phone: 954-375-3500Fax: 954-375-3501E-mail: info@cyberguard.com