systems management in an untrusted network dealing with backups, monitoring, administration, and...

Systems Management in an Untrusted Network

Dealing with backups,monitoring, administration, and

logging in the DMZ

Cory L. Scott, Lead Security Consultant

Securify, Inc.

Introduction

Implementing systems management components in untrusted or semi-trusted networks is difficult…

…if you are concerned about security!

Outline of today’s talk:

Example of threats

DMZ Network Architectures in the Real World™

Two core designs and advanced design issues

System and network configuration for systems management

The threat is out there…

SNMP– Multiple Vendor SNMP World Writeable Community

Vulnerability– NAI Sniffer Agent SNMP Buffer Overflow

Vulnerability

Sniffers– Microsoft Network Monitor Multiple Buffer Overflow

Vulnerabilities– Solaris snoop (print_domain_name) Buffer

Overflow VulnerabilitySpecific Vulnerability Titles courtesy of SecurityFocus

The threat is out there… Remote Control Software (besides its

intended functionality)– AT&T VNC Weak Authentication Vulnerability– PCAnywhere32 Denial of Service Vulnerability

Administrative Interfaces (over intended functional protocols)– Allaire ColdFusion Server 4.5.1 Administrator

Login Password DoS Vulnerability– Cisco 7xx Series Router DoS Vulnerability– Cisco 675 Web Administration Denial of Service

Vulnerability

Specific Vulnerability Titles courtesy of SecurityFocus

The threat is out there…System logging– Age-old attacks:• log flood• log erase• selective log edit

– Linux syslogd Denial of Service Vulnerability– Solaris syslogd Unresolvable Address Remote

Denial of Service Vulnerability

Backup– Unauthorized restore/delete, unencrypted backups– Veritas Backup Denial of Service Vulnerability

Specific Vulnerability Titles courtesy of SecurityFocus

The Purpose of the DMZ

But… I’m filtering System Management Protocols at the perimeter! Isn’t that enough?

No. Why? Two words: Aggravated Penetration. Want two more? Privilege Escalation.

More? Insider attack.

DMZ hosts are bastion hosts or perimeter service hosts. Why do we spend all this time hardening our DNS servers and then leave a poor password on the ssh service listening on an untrusted interface?


The DMZ exists to mitigate risk by isolating certain services and functions in a separate segment of the network.

Segmentation by isolation is generally not enough. Ingress filtering is usually deployed, but typical designs need work past the “crunchy” outside layer. Defense in depth, along with proper protection of internal hosts from the DMZ, is required.


Example 1. Bastion hosts in a DMZ Segment.


Example 2. Perimeter service hosts in a flat network.


If it were only that simple…


Example 3. Segmented DMZs.


Example 4. Colocated DMZs.


Example 5. Partner DMZs.


Other problems in the DMZ– Constant change– Too many hands in the pot– Service protocols not designed with security in

mind– Systems management protocols not designed

with security in mind– Scalability mechanisms create additional

separation and obfuscation of a clean network design

– Collusion of disparate types of traffic going through the DMZ

System Management – Composite Sketch

Common sighting – the status quo:– No centralized logging.– SSH inbound from the internal network; often from

external network, too.– PCAnywhere, VNC, or SMS accessible from some

management hosts or worse...– Backup system non-existent or backups batch copied to

internal hosts.– Default administrative protocols and interfaces left

accessible within the DMZ and from the internal network:• SNMP on routers, web interfaces on servers• Openview or other monitoring system “pinging”

from the inside.

Dealing with the problem…

Securing Systems Management components requires a combination of network architecture and system configuration.

Two Core Designs

DMZ Network Architecture for System Management

Example 1. Combination of Management and Production Traffic on the same untrusted segment.

Definition of untrusted segment: Where untrusted users and/or processes can place packets on the segment.

Advantages:– Simple to manage, do not have deal with multiple

interfaces– Easier firewall rulesets and router ACLs to manage


Example 1. Combination of Management and Production Traffic on the same untrusted wire.

Disadvantages:– Bandwidth utilization– Failure to segment different types of traffic introduces security

risks– Must place loghosts, monitoring consoles, and control

components on the internal network to keep isolation– Harder to monitor for policy violations– Untrusted segment behind firewall will advertise management

services– For services that listen for input, must configure host-based

inclusion rather than interface/network inclusion– Compromised host on segment could spoof management

connection


Example 2. Separate Management LAN.Advantages:– Protects bandwidth on untrusted network segment– Introduces another hurdle for intruders to jump

interfaces, which can be locked down more aggressively

– Ability to monitor for violations in both segments improved

– Can place loghosts, monitoring hosts, and control components in management LAN with less risk, reducing internal network exposure and reliance

– Allows for more flexibility with private address space and less border firewall concerns


Example 2. Separate Management LAN.Disadvantages:– Need to make sure that forwarding is disabled;

routing must be configured correctly on each host; additional configuration and equipment needed

– Management LAN can still be used as a conduit to attack hosts if not properly secured and monitored

– Adds complexity to segmented DMZs and potential bypass mechanism between segments

Advanced Design Issues


Example 3. Management Aggregation Points based on natural segregation of the segmented DMZ.

Advantages:– Works well in segmented DMZs– Reduces management LAN bandwidth– All of the advantages of segmented DMZs

Disadvantages:– More equipment and more routes– Need to maintain ACLs and rulesets between

Management LANs– Additional points of failure– All of the disadvantages of segmented DMZs


Example 4. Pushing data versus pulling data. Pushing data from internal network to the DMZ/Admin

LAN. Good – but how much do you trust your internal users?

DMZ/Admin LAN pulling data from the internal network. Bad.

Degrees of push:– File / Data one-way with or without validation– Interactive transfer with restricted privilege– Remote control administration with full interactivity

Use the minimum amount of push whenever possible.When DMZ hosts need to push data for administrative

purposes, aggregate in the same trust boundary. Then pull from a more trusted environment.

Never have DMZ hosts pull or push from the Internet without appropriate risk analysis and mitigation.

The Need for Systems Management

BackupDiagnostic information and availability

monitoringRemote administrationSystem logging

Backup Solutions

Risks– Bandwidth utilization– Unauthorized restore / backup– Capture of backup traffic– Agent vulnerabilities – authentication– Procedures for restore offsite– Local backup devices unmanageable or

difficult to scale– Backup clients not necessarily designed with

security in mind

Backup Solutions

Securing Backup SolutionsProtect the backup server at all costs

– Place behind another firewall / filter– Backup server should initiate all backup / restore requests

to eliminate inbound connections– Consider the physical security of the server and the media– Implement tight security controls on server.

Encryption – examine the risks / benefits– Is the wire insecure? If so, client has burden of encrypting

the data. – Store the data encrypted or not? How is key management

performed? What happens if the key is lost?– Encrypt both on-site and off-site media?

Backup Solutions

Securing Backup SolutionsAdministrative LAN segment very beneficial for backup

solutions Implementing a Storage Area Network may provide

another means for backup that doesn’t use the LANOne example of a hard-to-secure product:

– Legato:• Server uses default ports 7937-9936/TCP&UDP• Client uses default ports 10001-30000/TCP&UDP• Runs its own portmapper• Ports can be restricted• Authentication client/server unclear• NAT not supported• Unable to determine which interface it listens on

Monitoring Solutions

SNMPAssume that anything sent over SNMP is readable by all.Community strings should be changed. If possible, limit the hosts that can query SNMP on the

queried device itself.Examine the type of information that your device gives via

SNMP – it may surprise you.Determine the criticality of the information when deciding

whether or not to use SNMP.Never allow reconfiguration of devices via SNMP. Disable

write privileges on any SNMP device.Traps should be used sparingly and there should be a

dedicated receiver in the DMZ.NT SNMP giveaway.Oftentimes, it is the lesser of another evil.

Monitoring Solutions

ICMPEcho reply/request is fine on an internal

interface.If possible, throttle your ICMP response

queue.

Remote Administration Solutions

Console accessTarget devices include: routers, switches, load

balancers, some UPSes, firewalls, and serversAggregate console connections into a terminal serverCan use a hardware terminal server with a serial or

network interface to a PC that maintains accessAlternatively, many newer terminal servers support

direct network connections via SSH, with RADIUS support and IP filtering

Can also connect out-of-band via dial-up modem with callback feature


Console accessAdvantages:– Console messages can be logged to a terminal server– Central point of authentication into console

management– Provides the ability to turn off telnet and other

administrative clear-text protocols on network equipment

– If ssh or other interactive interface fails to respond, administrator can directly connect to console without physically going to the DMZ


Console accessDisadvantages:– The unintentional <BREAK> problem– Additional hardware and cabling– Authentication and logging for console use (once

the user has accessed the terminal server) is difficult to implement with a hardware device


SSH bastion gatewayOne (hardened) point of entry via SSH to

other hostsCan use ssh-agent to eliminate interactivity

on the gateway, while maintaining only a single host that can SSH to the endpoints

Use RSA identity files Disable password authenticationDisable rhosts authentication and root loginBind ssh only to admin LAN interfaceWatch your patch levels – ssh is a popular

target


Windows GUI – 2 popular options:– PCAnywhere– Windows Terminal Services


Windows GUI – PCAnywhereRisks– Runs on well-known port – juicy target for attackers– Previous versions have been vulnerable to DoS

attacks and weak password encryption– Typical configuration binds to all interfaces– Should avoid exposing on an untrusted network

segment– Typical configuration bypasses Windows login

mechanism


Windows GUI – PCAnywhereSecuring PCAnywhere

– Make use of the allowed IP addresses feature – limit admin hosts– Enable TCPIPHostBindMode to only listen on admin interface– Change default port– Make sure the Windows NT user is logged off after session

disconnect (normal and abnormal)– Enable event logging and session recording (if disk space permits)– Utilize Symmetric encryption / Deny lower-level– If possible, use X.509 for host authentication– Disable response to PCAnywhere query broadcasts– Configure clients to only use TCP to connect (rather than a UDP

query – reduces firewall ruleset)– Use separate user account for each admin with strong passwords– Limit login attempts– Only use PCAnywhere user with PCAnywhere privileges


Windows GUI – Terminal ServicesRisks– Utilizes Windows authentication method– Runs on a well-known port– Should avoid exposing on an untrusted

network segment


Windows GUI – Terminal ServicesSecuring WTS (for administration use)

– Bind only to the administrative segment interface– Force all configuration parameters at the server level– Use a separate WTS login from Windows login and give each

administrator unique login with strong password– Take Administrators group out of connection permissions– Enable security auditing– Remove TsInternetUser account– Utilize High Encryption for RDP– Disconnect idle/broken connections aggressively– For those who are paranoid, change the WTS port.

Logging solutions

Log types

The system management need is to centralize logs for analysis.

Type Mode

Syslog – UNIX and Network Devices

Write to local filesystem or send over network (UDP 514)

Windows NT/2000 Event Logs

Write to local filesystem (network support for syslog available from 3rd parties)

Application / Service Log Files

Syslog, NT Event Log, flat file, binary file, database entry

Logging solutionsNetwork syslog If possible, limit which machines can send log entries to

a host.Heartbeat creation and detection is absolutely

imperative.Flood detection is also imperative.Syslog servers should sit on administrative LANs if at

all possible.Make sure that clients are sending the messages over

the administrative LAN interface. Initiatives are out there for secure syslog – not close to

implementation yet:– Log signing– Encrypted transfer– Insertion / deletion attacks

Take a look at syslog-ng. http://www.balabit.hu

Logging solutions

NT/2000 Event LogNeed to get those logs off each server posthaste.Two major options:

– Agent-based forwarding• Syslog• Commercial solutions

– Batch retrieval• Can use common resource kit utilities to pull logs out in binary

and text format• To push or to pull?• If log is cleared by an intruder, you better know about it! (Use a

perl script to check for Event ID 517.)

See my SANS NetSec 2000 presentation for many more details!

Logging solutions

Flat file logsCan always “syslog” them

– tail -f /var/log/mylog | logger– Ok… maybe not!

Need to get them off of the originator as soon as possible.

If they are too big and/or cumbersome, consider culling them during the push or pull process.

How often to push/pull? Determine the criticality of the logs and analyze the worst case scenario: where the attacker blows away the local copy of the log file and your mission is to figure out what happened.

When log disappears, you better know about it!

General tips

Watch out for administrative interfaces.

Follow best practices, especially in regards to:

Resource utilization

Segmentation

Authentication

Integrity

systems management in an untrusted network dealing with backups, monitoring, administration, and...

Documents

dmz slide

dmz example

dmz hosts

dmz segment

systems management slide

management connection

management hosts

system management example