Xinwen Fu

Linux Logging Mechanisms91.580.203 Computer & Network Forensics

CS@UML

OutlineLog filesWhat need to be loggedLogging policiesFinding log filesSyslog: the system event logger

CS@UML

Who logs data?The accounting systemThe kernel Various utilities All produce data that need to be loggedMost of the data has a limited useful lifetime, and needs to be summarized, compressed, archived and eventually thrown away

CS@UML

Logging policiesThrow away all data immediatelyReset log files at periodic intervalsRotate log files, keeping data for a fixed amount of timeCompress and archive to tape or other permanent media

CS@UML

Which policy to chooseDepends on:how much disk space you havehow security-conscious you areWhatever scheme you select, regular maintenance of log files should be automated using cron

CS@UML

1. Throwing away log filesNot recommendSecurity problems (accounting data and log files provide important evidence of break-ins)Helpful for alerting you to hardware and software problemsIn general, keep one or two monthsIn a real world, it may take one or two weeks for SA to realize that site has been compromised by a hacker and need to review the logs

CS@UML

2. Reset log files at periodic intervalsMost sites store each days log info on disk, sometimes in a compressed formatThese daily files are kept for a specific period of time and then deletedOne common way to implement this policy is called rotation

CS@UML

3. Rotating log filesKeep backup files that are one day old, two days old, and so on. logfile, logfile.1 , logfile.2, logfile.6Linux: /etc/logrotate.conf Specify the frequency with which the files are reused

Each day rename the files to push older data toward the end of the chain

CS@UML

#! /bin/shcd /var/logmv logfile.2 logfile.3mv logfile.1 logfile.2mv logfile logfile.1cat /dev/null > logfileSome daemons keep their log files open all the time, this script cant be used with them. To install a new log file, you must either signal the daemon, or kill and restart it.In Unix-like operating systems, /dev/null or the null device is a special file that discards all data written to it, and provides no data to any process that reads from it. In Unix programmer jargon, it may also be called the bit bucket or black hole.Script to archive 4 days files

CS@UML

4. Archiving log filesSome sites must archive all accounting data and log files as a matter of policy, to provide data for a potential auditLog files should be first rotated on disk, then written to tape or other permanent media

CS@UML

Finding log filesTo locate log files, read the system startup scripts : /etc/rc* or /etc/init.d/* If logging is turned on when daemons are runWhere messages are sentSome programs handle logging via syslog (syslogd or rsyslogd)Check /etc/syslog.conf (or rsyslog.conf on Fedora Core 9) to find out where this data goes

CS@UML

Finding log files (default configuration)Different operating systems put log files in different places:/var/log/*/var/cron/log/usr/adm/var/adm On Linux, all the log files are in /var/log directory

CS@UML

OutlineLog filesSyslog: the system event loggerhow syslog worksits configuration file debugging syslogthe software that uses syslog

CS@UML

What is syslogA comprehensive logging system, used to manage information generated by the kernel and system utilitiesAllow messages to be sorted by their sources and importance, and routed to a variety of destinations:Log files, users terminals, or even other machines

CS@UML

Syslog: three partsSyslogd: daemon that does the actual loggingConfiguration file: /etc/syslog.conf API: openlog, syslog, closelogLibrary routines that programs use to send data to syslogdloggerUser-level command for submitting log entries

CS@UML

Using syslog library routineswrite log entries to a special file /dev/logsyslogd /etc/syslog.conf readsconsultsdispatchesLogfilesUserssterminalsOther machines/dev/kloghttp://www.calpoly.edu/cgi-bin/man-cgi?syslogdMost system logging daemons listen on one or more Unix sockets, the most typical being /dev/log; /dev/klog is kernel log socket

CS@UML

Configuring syslogdThe configuration file /etc/syslog.conf controls syslogds behaviorIt is a text file with simple format, blank lines and lines beginning with # are ignored (comment).selector actionfor example mail.info/var/log/maillog

CS@UML

Configuration file - selectorIdentifies Program facility that is sending a log message Messagess severity level eg. mail.infoSyntaxfacility.levelFacility names and severity levels must be chosen from a list of defined values

CS@UML

Configuration file - Facility NamesFACILITYPROGRAMS THAT USE ITkernthe kerneluserUser process, default if not specifiedmailThe mail systemdaemonSystem daemonsauthSecurity and authorization related commandslprthe BSD line printer spooling systemnewsThe Usenet news system

CS@UML

Configuration file - Facility names (Cont.)FACILITYPROGRAMS THAT USE ITuucpReserved for UUCPcronthe cron daemonmarkTimestamps generated at regular intervalslocal0-7Eight flavors of local messagesyslogsyslog internal messagesauthprivPrivate or system authorization messagesftpthe ftp daemon, ftpd*All facilities except markUUCP stands for Unix to Unix CoPy.

CS@UML

Configuration file - Facility names (Cont.)Facility - Mark: Timestamps can be used to log time at regular intervals (by default, every 20 minutes), so you can figure out that your machine crashed between 3:00 and 3:20 am, not just sometime last night. This can be a big help if debugging problems occur on a regular basisStart at command line: syslogd m 1Use syslog.confStart syslog daemon: syslogdAdd the line to syslog.conf: mark.*/var/log/messages

CS@UML

Configuration file - severity levelLEVELAPPROXIMATE MEANINGemerg (panic)Panic situationalertUrgent situationcritCritical conditionerrOther error conditionswarningWarning messagesnoticeUnusual things that may need investigationinfoInformational messagesdebugFor debugging

severenot severe

CS@UML

Configuration file - selector Levels indicate the minimum importance that a message must have in order to be loggedmail.warning - would match all the messages from the mail system, at the minimum level of warningLevel of none will exclude the listed facilities regardless of what other selectors on the same line may say.*.info;mail.noneaction All the facilities, except mail, at the minimum level info will subject to action

CS@UML

Configuration file selector (Cont.) Can include multiple facilities separated with , commase.g., daemon,auth,mail.infoactionMultiple selectors can be combined with ;e.g. daemon.level1;mail.level2actionSelectors are | -- ORed together, a message matching any selector will be subject to the actionCan contain * - meaning all none - meaning nothing

CS@UML

Configuration file - action(Tells what to do with a message)ACTIONMEANING

filenameWrite message to a file on the local machine@hostnameForward messages to the syslogd on hostname@ipaddressForward messages to the host at IP address

user1, user2,Write messages to users screens if they are logged in*Write messages to all users logged in

CS@UML

Configuration file - action (Cont.)If a filename action used, the filename must be absolute path. The file must exist since syslogd will not create ite.g. /var/log/messagesIf a hostname is used, it must be resolved via a translation mechanism such as DNS or NISWhile multiple facilities and levels are allowed in a selector, multiple actions are not allowed.

CS@UML

Config file examples (1)# Small network or stand-alone syslog.conf file# emergencies: tell everyone who is logged on*.emerg*

# important messages*.warning;daemon,auth.info/var/adm/messages

# printer errorslpr.debug/var/adm/lpd-errs

CS@UML

# network client, typically forwards serious messages to # a central logging machine# emergencies: tell everyone who is logged on*.emerg;user.none*

#important messages, forward to central logger*.warning;lpr,local1.none@netloghostdaemon,auth.info@netloghost

# local stuff to central logger toolocal0,local2,local7.debug@netloghost

# card syslogs to local1 - to boulderlocal1.debug@ialab.cs.uml.edu

# printer errors, keep them locallpr.debug/var/adm/lpd-errs

# sudo logs to local2 - keep a copy herelocal2.info/var/adm/sudologConfig file examples (2)

CS@UML

Sample syslog outputMar 27 09:10:02 tcb-ia-lab-inst sshd[4100]: Accepted password for cis418 from ::ffff:216.254.235.105 port 61940 ssh2Mar 27 18:10:00 tcb-ia-lab-inst sshd[9332]: Failed password for root from ::ffff:216.254.235.105 port 62817 ssh2Mar 27 18:10:08 tcb-ia-lab-inst sshd[9332]: Accepted password for root from ::ffff:216.254.235.105 port 62817 ssh2Mar 27 20:08:27 tcb-ia-lab-inst sshd[10629]: Accepted password for root from ::ffff:10.0.0.111 port 42172 ssh2Mar 27 20:09:48 tcb-ia-lab-inst sshd[10649]: Failed password for root from ::ffff:10.0.0.111 port 48233 ssh2

CS@UML

SyslogdA hangup signal (HUP, signal 1) cause syslogd to close its log files, reread its configuration file, and start logging againIf you modify the syslog.conf file, you must HUP syslogd to make your changes take effectps -ef | grep syslogdKill -1 pid-of-syslogd

CS@UML

Software that uses syslogPROGRAMFACILITYLEVELSDESCRIPTIONamdautherr-infoNFS automounterdateauthnoticeDisplay and set dateftpddaemonerr-debugftp daemongateddaemonalert-infoRouting daemongopherdaemonerrInternet info serverhalt/rebootauthcritShutdown programslogin/rlogindauthcrit-infoLogin programslpdlprerr-infoBSD line printer daemon

CS@UML

Software that uses syslog PROGRAMFACILITYLEVELSDESCRIPTIONnameddaemonerr-infoName sever (DNS)passwdautherrPassword settingprogramssendmailmaildebug-alertMail transport systemrwhodaemonerr-noticeromote who daemonsuauthcrit, noticesubstitute UID prog.sudolocal2notice, alertLimited su programsyslogdsyslog, markerr-infointernet errors, timestamps

CS@UML

Syslog 's functionsLiberate programmers from the tedious mechanics of writing log filesPut SA in control of loggingBefore syslog, SA had no control over what information was kept or where it was storedCan centralize the logging for a network system

CS@UML

Debugging syslog -- loggerUseful for submitting log entries from shell scripts

Can also use it to test changes in syslogds configuration file.For example..

CS@UML

Add line to syslog.conf:local5.info/var/log/test.log

verify it is working, runlogger -p local5.info test messages

a line containing test messages should be written to /tmp/test.log

If this doesnt happen:forgot to create the test.log file or forgot to send syslogd a hangup signal

CS@UML

Remote loggingOn a central logging server: 10.0.0.192syslogd -rOn a local server: 10.0.0.45authpriv.*;auth.*@10.0.0.192Question: where are those events written?

CS@UML

Process Accountingaccton is used to turn on or turn off process accountinglastcomm tracks commands each user usestouch /var/log/pacct/sbin/accton /var/log/pacctlastcomm -f /var/log/pacct ac prints out statistics about users' connection times in hours based on the logins and logouts in the current /var/log/wtmp fileac -p -dsa summarizes accounting information from previously executed commands, software I/O operation times, and CPU times, as recorded in the accounting record file /var/log/pacctsa /var/log/pacct

CS@UML

Process Accounting (Cont.)last goes through the /var/log/wtmp file and prints out information about users' connection times

lastb is the same as last, except that by default it shows a log of the file /var/log/btmp, which contains all the bad login attempts.

CS@UML

Using syslog in programsopenlog( ident, logopt, facility);Messages logged with the options specified by logopt begin with the identification string ident. syslog( priority, messge, parameters);Send message to syslogd, which logs it at the sepecified priority levelclose( );

CS@UML

/ * c program: syslog using openlog and closelog */

#include main ( ) {openlog ( SA-BOOK, LOG_PID, LOG_USER);syslog ( LOG_WARNING, Testing . );closelog ( );}On the host, this code produce the following log entry:Apr 4 15:21:57 tcb-ia-lab-inst SA-BOOK[7762]: Testing ...

CS@UML

SummaryOn linux, check following files:/etc/syslog.conf : syslog configuration file/etc/logrotate.conf : logging policy, rotate/etc/logrotate.d/*/var/log/* : log filestry following commands to find out more...man logrotateman syslogd

CS@UML

ReferencesChris Prosise, Kevin Mandia, Matt Pepe, Incident Response and Computer Forensics, Second Edition (Paperback), ISBN: 007222696XBrian Hatch, Preventing Syslog Denial of Service attacks, http://www.hackinglinuxexposed.com/articles/20030220.htmlAlbert M.C. Tam, Enabling Process Accounting on Linux HOWTO, 02/09/2001, http://www.faqs.org/docs/Linux-mini/Process-Accounting.html Keith Gilbertson, Process Accounting, 12/01/2002, http://www.linuxjournal.com/article/6144

CS@UML

NotesChange host name/etc/hosts # add the host to the end of 127.0.0.1/etc/sysconfig/network

CS@UML

#! /bin/shcd /var/logmv logfile.2.Z logfile.3.Zmv logfile.1.Z logfile.2.Zmv logfile logfile.1cat /dev/null > logfilekill -signal pidcompress logfile.1

signal - appropriate signal for the program writing the log filepid - process id

CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1 CS140 - Winter 2002 - Handout #1