cis 191 - lesson 12

CIS 191 - Lesson 12

SystemMonitoring

1

CIS 191 - Lesson 12

System Monitoring

Monitoring Log Files

• /var/log

‒ Can be used as indication of systematic degradation

• log rotation

‒ logrotate

‒ /etc/logrotate.conf

2

CIS 191 - Lesson 12

System Monitoring

Many important logs (Red Hat family)• Kernel and system boot messages

• dmesg• boot.log (broken – see bugzilla)

• Security and authorization messages• secure, btmp, wtmp, lastlog, audit, …

• System module messages• messages (a good catch-all log), cron, maillog,

… Key Log File Locations:

/var/log directory /etc/syslog.conf

3

https://bugzilla.redhat.com/show_bug.cgi?id=151238

CIS 191 - Lesson 12

Logging

The syslog daemon, controlled by /etc/syslog.conf, is a central clearing house for handling all the log messages sent by various system programs

The klogd daemon handles kernel log messages. klogd does not have a configuration file and is controlled by command line switches.

[root@opus ~]# ps -e | grep log 2152 ? 00:00:07 syslogd 2155 ? 00:00:00 klogd[root@opus ~]#

4

CIS 191 - Lesson 12

System Monitoring

Most log files are ascii text

messages: ASCII English textmessages.1: ASCII textmessages.2: ASCII English textmessages.3: ASCII English textppp: directoryprelink: directoryrpmpkgs: ASCII textrpmpkgs.1: ASCII textrpmpkgs.2: ASCII textrpmpkgs.3: ASCII textsamba: directoryscrollkeeper.log: ASCII textsecure: ASCII textsecure.1: emptysecure.2: ASCII English textsecure.3: ASCII textspooler: emptyspooler.1: emptyspooler.2: emptyspooler.3: emptytallylog: emptyvbox: directorywtmp: datawtmp.1: dataXorg.0.log: ASCII English textXorg.0.log.old: ASCII English textyum.log: ASCII text[root@benji log]# 5

CIS 191 - Lesson 12

System Monitoring[root@opus ~]# ls -l /var/logtotal 153572-rw-r----- 1 root root 3665 Nov 11 13:36 acpid-rw------- 1 root root 527440 Jun 16 15:47 anaconda.log-rw------- 1 root root 22282 Jun 16 15:47 anaconda.syslog-rw------- 1 root root 58040 Jun 16 15:47 anaconda.xlogdrwxr-x--- 2 root root 4096 Nov 24 02:03 audit-rw------- 1 root root 0 Nov 23 04:02 boot.log-rw------- 1 root root 0 Nov 16 04:02 boot.log.1-rw------- 1 root root 0 Nov 9 04:02 boot.log.2-rw------- 1 root root 0 Nov 2 04:02 boot.log.3-rw------- 1 root root 0 Oct 26 04:03 boot.log.4-rw------- 1 root utmp 136987008 Nov 29 15:16 btmpdrwxr-xr-x 2 root root 4096 Jun 28 2007 conmandrwxr-xr-x 2 root root 4096 Jun 28 2007 conman.old-rw------- 1 root root 12817 Nov 29 16:01 cron-rw------- 1 root root 13860 Nov 23 04:02 cron.1-rw------- 1 root root 13706 Nov 16 04:02 cron.2-rw------- 1 root root 13843 Nov 9 04:02 cron.3-rw------- 1 root root 14117 Nov 2 04:02 cron.4drwxr-xr-x 2 lp sys 4096 Nov 27 04:02 cups-rw-r--r-- 1 root root 18903 Nov 11 13:35 dmesg-rw------- 1 root root 29256 Nov 11 08:11 faillogdrwxr-xr-x 2 root root 4096 Mar 28 2008 gdmdrwx------ 2 root root 4096 Oct 19 04:02 httpd-rw-r--r-- 1 root root 355948 Nov 29 16:34 lastlogdrwxr-xr-x 2 root root 4096 Jun 16 15:39 mail

-rw------- 1 root root 27520 Nov 29 08:51 maillog-rw------- 1 root root 38980 Nov 23 04:02 maillog.1-rw------- 1 root root 56964 Nov 16 04:02 maillog.2-rw------- 1 root root 74842 Nov 9 04:02 maillog.3-rw------- 1 root root 110136 Nov 2 04:02 maillog.4-rw------- 1 root root 9165 Nov 29 15:35 messages-rw------- 1 root root 11706 Nov 22 21:30 messages.1-rw------- 1 root root 35986 Nov 16 03:22 messages.2-rw------- 1 root root 12430 Nov 8 23:59 messages.3-rw------- 1 root root 6224 Nov 1 16:21 messages.4drwxr-xr-x 2 root root 4096 Jun 17 15:02 pmdrwx------ 2 root root 4096 Dec 1 2006 pppdrwxr-xr-x 2 root root 4096 Jun 27 2007 prelink-rw-r--r-- 1 root root 31559 Nov 29 04:03 rpmpkgs-rw-r--r-- 1 root root 31559 Nov 22 04:03 rpmpkgs.1-rw-r--r-- 1 root root 31559 Nov 15 04:03 rpmpkgs.2-rw-r--r-- 1 root root 31559 Nov 8 04:02 rpmpkgs.3-rw-r--r-- 1 root root 31559 Nov 1 04:02 rpmpkgs.4drwx------ 2 root root 4096 May 20 2008 samba-rw-r--r-- 1 root root 107169 Jun 17 15:07 scrollkeeper.log-rw------- 1 root root 1702726 Nov 29 16:34 secure-rw------- 1 root root 5069529 Nov 23 03:38 secure.1-rw------- 1 root root 1196200 Nov 16 03:30 secure.2-rw------- 1 root root 2404320 Nov 8 23:59 secure.3-rw------- 1 root root 6374517 Nov 1 19:52 secure.4drwxr-xr-x 2 root root 4096 Nov 23 04:02 setroubleshoot-rw------- 1 root root 0 Nov 23 04:02 spooler-rw------- 1 root root 0 Nov 16 04:02 spooler.1-rw------- 1 root root 0 Nov 9 04:02 spooler.2-rw------- 1 root root 0 Nov 2 04:02 spooler.3-rw------- 1 root root 0 Oct 26 04:03 spooler.4drwxr-x--- 2 squid squid 4096 Apr 1 2008 squid-rw------- 1 root root 0 Jun 17 14:57 tallylog-rw-r--r-- 1 root root 34140 Nov 29 16:34 up2date-rw-r--r-- 1 root root 37324 Nov 23 03:34 up2date.1-rw-r--r-- 1 root root 43305 Nov 16 03:34 up2date.2-rw-r--r-- 1 root root 32088 Nov 9 03:49 up2date.3-rw-r--r-- 1 root root 34650 Nov 2 03:49 up2date.4drwxr-xr-x 2 root root 4096 Nov 20 2007 vbox-rw-rw-r-- 1 root utmp 23040 Nov 29 16:34 wtmp-rw-rw-r-- 1 root utmp 1093632 Nov 27 02:13 wtmp.1-rw-rw-r-- 1 root cis90 59894 Oct 24 08:23 Xorg.0.log-rw-rw-r-- 1 root cis90 59894 Sep 16 12:58 Xorg.0.log.old-rw-r--r-- 1 root root 20546 Jun 17 19:32 yum.log

[root@opus ~]#

How many backups are there of each log? 4

How often are these log files rotated? weekly

From observing /var/log ….

Log files are owned by root and have restrictive permissions due to the sensitive information they contain

6

CIS 191 - Lesson 12

syslog.conf

7

CIS 191 - Lesson 12

/etc/syslog.conf

/etc/syslog.conf on Opus

Each entry is a selector followed by an action 8

CIS 191 - Lesson 12

/etc/syslog.confFacility Description

auth The authorization system. Ex.: login, su, ftpd, rshd

authpriv User access messages use this

cron Used by the cron facility

daemon Other daemon programs without a facility of their own

ftp Used by ftp applications

kern Kernel messages

lpr The line printer spooling system

mail Used by mail applications

mark Used by syslogd to produce timestamps in log files

news Used by news applications

security Same as auth. Should not be used anymore.

syslog

user Messages generated by random user processes. Default.

uucp UUCP messages

local0 – local7 Reserved for local use.

* For all

9http://www.linode.com/wiki/index.php/Syslog_Howto

CIS 191 - Lesson 12

/etc/syslog.conf

Security Level

Priority Keyword Description

0 emergencies emerg, panic A panic condition. This is normally broadcast to all users

1 alerts alert Inmmediate action required. e.g.: Corrupted system database

2 critical crit Critical condition. e.g.: Hard device errors

3 errors err, error Error conditions

4 warning warning, warn Warning conditions

5 notifications notice Normal but significant conditions that need attention

6 informational info Informational messages

7 debugging debug Debugging messages


CIS 191 - Lesson 12

/etc/syslog.confSelector Description

kernel.* kernel facility, any priority

mail.debug mail facility, debug or higher priority (same as *)

lpr,news.* all messages from printer or news

auth.warning all security messages of warning or higher priority

*.info all messages from any facility of level info or higher

ftp.=info ftp facility, info msgs only (and not higher)

*.!err any facility, pri <= err only

*.!=alert any facility, any priority except alert

*.info;mail,news,authpriv.none all msgs with info or higher priority except mail, news, and authpriv

mail.2all mail messages level critical and higher

mail.0 only critical mail messages


CIS 191 - Lesson 12

/etc/syslog.conf

Action Description

/complete/path/of/some/file Messages logged to a file

/dev/console This is a link to the system console

-/complete/path/of/some/file Don't flush (write to disk) file each time; better performance but risks loss of some log info

username1[,username2 ...] Users that will get the message

* All logged in users get the message

@remotehost Log to remote host. Start the remote syslogd with "-r" option

|/path/to/named/pipe

To send output to a command you must create a named pipe, say /var/lib/cmd.pipe with the mkfifo command. Then start the command with cmd </var/lib/cmd.pipe


CIS 191 - Lesson 12

Logging

Note: You can use the severity level to control where messages are sent, but you don't have control over the level a program assigns to a message.

13

CIS 191 - Lesson 12

/etc/syslog.conf

write messages to console

kernel facility (all messages) commented out

14

CIS 191 - Lesson 12

/etc/syslog.conf

All facilties with info (6) or higher priority except mail, authpriv or cron

write messages to this file

15

CIS 191 - Lesson 12

/etc/syslog.conf

authpriv facility, any priority write messages to this file

16

CIS 191 - Lesson 12

/etc/syslog.conf

mail facility, any priority write messages to this file (- means don't flush file each time)

17

CIS 191 - Lesson 12

/etc/syslog.conf

cron facility, any priority write messages to this file

18

CIS 191 - Lesson 12

/etc/syslog.conf

All emergency level (0) messages from any facility

All logged in users get the message

19

CIS 191 - Lesson 12

/etc/syslog.conf

Critical (2) or higher messages from uucp or news facilities

Messages are written to this file

20

CIS 191 - Lesson 12

/etc/syslog.conf

any messages from local7 (used by Red Hat family for boot messages)


21

CIS 191 - Lesson 12

/etc/syslog.conf

only notification level (5) messages from any facility


For Lab 10

In Lab 10 a new entry is added to /etc/syslog.conf for a custom notices log

22

CIS 191 - Lesson 12

/etc/syslog.conf

Must restart the logging service for the change in /etc/syslog.conf to take effect.

For Lab 10

Create a custom logfile

23

CIS 191 - Lesson 12

/etc/syslog.confFor Lab 10

Login as root on tty2 Login as cis191 on tty3, then su with bad password

The new log will hold root logins and login failures 24

CIS 191 - Lesson 12

logrotation

25

CIS 191 - Lesson 12

Log file rotation

logrotate is normally run out of cron once every day

26The actual program lives in /usr/sbin

This is actually a script that calls the logrotate program

CIS 191 - Lesson 12

[root@opus ~]# cat /etc/logrotate.conf# see "man logrotate" for details# rotate log files weeklyweekly

# keep 4 weeks worth of backlogsrotate 4

# create new (empty) log files after rotating old onescreate

# uncomment this if you want your log files compressed#compress

# RPM packages drop log rotation information into this directoryinclude /etc/logrotate.d

# no packages own wtmp -- we'll rotate them here/var/log/wtmp { monthly minsize 1M create 0664 root utmp rotate 1}

# system-specific logs may be also be configured here.[root@opus ~]#

/etc/logrotate.conf

applie

s to all

files

for

specific

files

logrotate.conf on Opus

27

CIS 191 - Lesson 12

/etc/logrotate.conf[root@benji ~]# cat /etc/logrotate.conf# see "man logrotate" for details# rotate log files weeklyweekly

# keep 4 weeks worth of backlogsrotate 4

# create new (empty) log files after rotating old onescreate

# uncomment this if you want your log files compressed#compress

# RPM packages drop log rotation information into this directoryinclude /etc/logrotate.d

# no packages own wtmp -- we'll rotate them here/var/log/wtmp { monthly create 0664 root utmp rotate 1}

# system-specific logs may be also be configured here.[root@benji ~]#

logrotate.conf on Benji

28

CIS 191 - Lesson 12

logins

29

CIS 191 - Lesson 12

/var/log/wtmp and var/log/btmp

[root@opus log]# ls -l /var/logtotal 153576-rw-r----- 1 root root 3665 Nov 11 13:36 acpid-rw------- 1 root root 527440 Jun 16 15:47 anaconda.log-rw------- 1 root root 22282 Jun 16 15:47 anaconda.syslog-rw------- 1 root root 58040 Jun 16 15:47 anaconda.xlogdrwxr-x--- 2 root root 4096 Nov 24 02:03 audit-rw------- 1 root root 0 Nov 23 04:02 boot.log-rw------- 1 root root 0 Nov 16 04:02 boot.log.1-rw------- 1 root root 0 Nov 9 04:02 boot.log.2-rw------- 1 root root 0 Nov 2 04:02 boot.log.3-rw------- 1 root root 0 Oct 26 04:03 boot.log.4

-rw------- 1 root utmp 136987008 Nov 29 15:16 btmpdrwxr-xr-x 2 root root 4096 Jun 28 2007 conmandrwxr-xr-x 2 root root 4096 Jun 28 2007 conman.old-rw------- 1 root root 13117 Nov 29 20:01 cron-rw------- 1 root root 13860 Nov 23 04:02 cron.1-rw------- 1 root root 13706 Nov 16 04:02 cron.2-rw------- 1 root root 13843 Nov 9 04:02 cron.3-rw------- 1 root root 14117 Nov 2 04:02 cron.4drwxr-xr-x 2 lp sys 4096 Nov 27 04:02 cups-rw-r--r-- 1 root root 18903 Nov 11 13:35 dmesg-rw------- 1 root root 29256 Nov 11 08:11 faillogdrwxr-xr-x 2 root root 4096 Mar 28 2008 gdmdrwx------ 2 root root 4096 Oct 19 04:02 httpd-rw-r--r-- 1 root root 355948 Nov 29 18:39 lastlogdrwxr-xr-x 2 root root 4096 Jun 16 15:39 mail-rw------- 1 root root 28085 Nov 29 19:56 maillog-rw------- 1 root root 38980 Nov 23 04:02 maillog.1-rw------- 1 root root 56964 Nov 16 04:02 maillog.2-rw------- 1 root root 74842 Nov 9 04:02 maillog.3-rw------- 1 root root 110136 Nov 2 04:02 maillog.4-rw------- 1 root root 9165 Nov 29 15:35 messages-rw------- 1 root root 11706 Nov 22 21:30 messages.1-rw------- 1 root root 35986 Nov 16 03:22 messages.2-rw------- 1 root root 12430 Nov 8 23:59 messages.3-rw------- 1 root root 6224 Nov 1 16:21 messages.4drwxr-xr-x 2 root root 4096 Jun 17 15:02 pmdrwx------ 2 root root 4096 Dec 1 2006 pppdrwxr-xr-x 2 root root 4096 Jun 27 2007 prelink-rw-r--r-- 1 root root 31559 Nov 29 04:03 rpmpkgs-rw-r--r-- 1 root root 31559 Nov 22 04:03 rpmpkgs.1-rw-r--r-- 1 root root 31559 Nov 15 04:03 rpmpkgs.2-rw-r--r-- 1 root root 31559 Nov 8 04:02 rpmpkgs.3-rw-r--r-- 1 root root 31559 Nov 1 04:02 rpmpkgs.4drwx------ 2 root root 4096 May 20 2008 samba-rw-r--r-- 1 root root 107169 Jun 17 15:07 scrollkeeper.log-rw------- 1 root root 1703877 Nov 29 19:59 secure-rw------- 1 root root 5069529 Nov 23 03:38 secure.1-rw------- 1 root root 1196200 Nov 16 03:30 secure.2-rw------- 1 root root 2404320 Nov 8 23:59 secure.3-rw------- 1 root root 6374517 Nov 1 19:52 secure.4drwxr-xr-x 2 root root 4096 Nov 23 04:02 setroubleshoot-rw------- 1 root root 0 Nov 23 04:02 spooler-rw------- 1 root root 0 Nov 16 04:02 spooler.1-rw------- 1 root root 0 Nov 9 04:02 spooler.2-rw------- 1 root root 0 Nov 2 04:02 spooler.3-rw------- 1 root root 0 Oct 26 04:03 spooler.4drwxr-x--- 2 squid squid 4096 Apr 1 2008 squid-rw------- 1 root root 0 Jun 17 14:57 tallylog-rw-r--r-- 1 root root 34818 Nov 29 19:34 up2date-rw-r--r-- 1 root root 37324 Nov 23 03:34 up2date.1-rw-r--r-- 1 root root 43305 Nov 16 03:34 up2date.2-rw-r--r-- 1 root root 32088 Nov 9 03:49 up2date.3-rw-r--r-- 1 root root 34650 Nov 2 03:49 up2date.4drwxr-xr-x 2 root root 4096 Nov 20 2007 vbox

-rw-rw-r-- 1 root utmp 26112 Nov 29 19:02 wtmp-rw-rw-r-- 1 root utmp 1093632 Nov 27 02:13 wtmp.1-rw-rw-r-- 1 root cis90 59894 Oct 24 08:23 Xorg.0.log-rw-rw-r-- 1 root cis90 59894 Sep 16 12:58 Xorg.0.log.old-rw-r--r-- 1 root root 20546 Jun 17 19:32 yum.log

[root@opus log]#

bad login attempts

good login attempts

30

CIS 191 - Lesson 12


[root@opus ~]# lastb | grep "cool.nju.edu.cn" | headbind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)bind ssh:notty cool.nju.edu.cn Sun Nov 30 06:35 - 06:35 (00:00)

[root@opus ~]# lastb | grep "cool.nju.edu.cn" | wc -l3104[root@opus ~]#

31

Shows break in attempt on 11/30/2008

CIS 191 - Lesson 12


[root@opus ~]# lastb | grep "Nov 2 17:45"webadmin ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)webadmin ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)retsu ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)retsu ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)sbear ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)sbear ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)sky ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)sky ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)harvey ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)harvey ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)harvey ssh:notty 211.96.97.179 Sun Nov 2 17:45 - 17:45 (00:00)[root@opus ~]#

[root@opus ~]# lastb -i | grep "211.96.97.179" | wc -l598[root@opus ~]#

32

Shows break in attempt by 211.96.97.179 on 11/2/2008

CIS 191 - Lesson 12

/var/log/lastlog

33

CIS 191 - Lesson 12

/var/log/lastlog

34

CIS 191 - Lesson 12


failed logins

successful logins

35

CIS 191 - Lesson 12


Either way prints successful login history

36

CIS 191 - Lesson 12


[root@opus log]# lastb | sort | cut -f1 -d' ' | grep -v ^$ | uniq –c > bad[root@opus log]# sort –g bad > bad.sort[root@opus log]# [root@opus log]# cat bad.sort | tail -50 471 ftp 472 public 490 test 490 tomcat 498 user 506 service 508 mike 508 username 524 cyrus 530 pgsql 532 test1 544 master 554 linux 554 toor 576 paul 584 support 590 testuser 604 irc

610 test 656 noc 686 www 690 postfix 723 john 734 testing 738 adam 746 alex 754 info 798 tester 832 library 935 guest 990 admin 1002 office 1022 temp 1070 ftpuser

1138 webadmin 1298 nagios 1332 web 1374 a 1384 student 1416 postgres 1690 user 1858 oracle 1944 mysql 2086 webmaste 5324 test 10803 root 10824 admin 18679 root 24064 root[root@opus log]#

Top 50 usernames used by the bad guys 37

CIS 191 - Lesson 12

logwatch

39

CIS 191 - Lesson 12

logwatch[root@opus ~]# mailMail version 8.1 6/6/93. Type ? for help."/var/spool/mail/root": 349 messages 349 new>N 1 [email protected] Mon Jun 16 17:04 43/1587 "Logwatch for opus.cabrillo.edu (Linux)" N 2 [email protected] Mon Jun 16 17:12 18/795 "Anacron job for 'opus.cabrillo.edu' cron." N 3 [email protected] Tue Jun 17 16:14 141/3966 "Logwatch for opus.cabrillo.edu (Linux)" N 4 [email protected] Wed Jun 18 04:02 728/32707 "Logwatch for opus.cabrillo.edu (Linux)" N 5 [email protected] Wed Jun 18 04:05 47/1877 "Cron <root@opus> run-parts /etc/cron.dail" N 6 [email protected] Thu Jun 19 04:02 1007/61932 "Logwatch for opus.cabrillo.edu (Linux)" N 7 [email protected] Thu Jun 19 04:02 47/1889 "Cron <root@opus> run-parts /etc/cron.dail" N 8 [email protected] Fri Jun 20 04:02 168/5533 "Logwatch for opus.cabrillo.edu (Linux)" N 9 [email protected] Fri Jun 20 04:02 47/1891 "Cron <root@opus> run-parts /etc/cron.dail" N 10 [email protected] Sat Jun 21 04:02 274/8886 "Logwatch for opus.cabrillo.edu (Linux)" N 11 [email protected] Sat Jun 21 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 12 [email protected] Sun Jun 22 04:02 156/4722 "Logwatch for opus.cabrillo.edu (Linux)" N 13 [email protected] Sun Jun 22 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 14 [email protected] Mon Jun 23 04:02 241/10770 "Logwatch for opus.cabrillo.edu (Linux)" N 15 [email protected] Mon Jun 23 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 16 [email protected] Tue Jun 24 04:02 3768/316984 "Logwatch for opus.cabrillo.edu (Linux)" N 17 [email protected] Tue Jun 24 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 18 [email protected] Wed Jun 25 04:02 3246/274685 "Logwatch for opus.cabrillo.edu (Linux)" N 19 [email protected] Wed Jun 25 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 20 [email protected] Thu Jun 26 04:02 1390/112446 "Logwatch for opus.cabrillo.edu (Linux)" N 21 [email protected] Thu Jun 26 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 22 [email protected] Fri Jun 27 04:02 72/2185 "Logwatch for opus.cabrillo.edu (Linux)" N 23 [email protected] Fri Jun 27 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 24 [email protected] Sat Jun 28 04:02 91/3228 "Logwatch for opus.cabrillo.edu (Linux)" N 25 [email protected] Sat Jun 28 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 26 [email protected] Sun Jun 29 04:02 150/6673 "Logwatch for opus.cabrillo.edu (Linux)" N 27 [email protected] Sun Jun 29 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 28 [email protected] Mon Jun 30 04:02 247/14351 "Logwatch for opus.cabrillo.edu (Linux)" N 29 [email protected] Mon Jun 30 04:02 47/1894 "Cron <root@opus> run-parts /etc/cron.dail" N 30 [email protected] Tue Jul 1 04:02 395/20660 "Logwatch for opus.cabrillo.edu (Linux)" N 31 [email protected] Tue Jul 1 04:02 47/1891 "Cron <root@opus> run-parts /etc/cron.dail" N 32 [email protected] Wed Jul 2 04:02 481/32664 "Logwatch for opus.cabrillo.edu (Linux)" N 33 [email protected] Wed Jul 2 04:02 47/1891 "Cron <root@opus> run-parts /etc/cron.dail" N 34 [email protected] Thu Jul 3 04:02 102/3197 "Logwatch for opus.cabrillo.edu (Linux)" N 35 [email protected] Thu Jul 3 04:02 47/1891 "Cron <root@opus> run-parts /etc/cron.dail"& 29

You have mail … from logwatch

40

CIS 191 - Lesson 12

logwatch

example email message from logwatch

41

& 11Message 11:From [email protected] Tue Dec 2 10:47:06 2008Date: Tue, 2 Dec 2008 10:47:06 -0800To: [email protected]: [email protected]: Logwatch for benji.localdomain (Linux)MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Type: text/plain; charset="iso-8859-1"

################### Logwatch 7.3 (03/24/06) #################### Processing Initiated: Tue Dec 2 10:47:06 2008 Date Range Processed: yesterday ( 2008-Dec-01 ) Period is day. Detail Level of Output: 0 Type of Output: unformatted Logfiles for Host: benji.localdomain ##################################################################

--------------------- Disk Space Begin ------------------------

Filesystem Size Used Avail Use% Mounted on /dev/sda1 2.9G 2.5G 272M 91% / /dev/sda5 289M 234M 41M 86% /opt /dev/sda3 487M 77M 385M 17% /var /dev/sda7 196M 5.6M 181M 3% /home

---------------------- Disk Space End -------------------------

###################### Logwatch End #########################

CIS 191 - Lesson 12

Configuring logwatch

42

This file shows all the defaults being used by logwatch

Level of detail is Low by default

CIS 191 - Lesson 12

Configuring logwatch

Edit /etc/logwatch/conf/logwatch.conf to modify defaults

43

This line was added to override the default level of Low

Read this for all options to set

CIS 191 - Lesson 12

logwatchMessage 14:From [email protected] Tue Dec 2 10:53:22 2008Date: Tue, 2 Dec 2008 10:53:21 -0800To: [email protected]: [email protected]: Logwatch for benji.localdomain (Linux)MIME-Version: 1.0Content-Transfer-Encoding: 7bitContent-Type: text/plain; charset="iso-8859-1"

################### Logwatch 7.3 (03/24/06) #################### Processing Initiated: Tue Dec 2 10:53:21 2008 Date Range Processed: yesterday ( 2008-Dec-01 ) Period is day. Detail Level of Output: 10 Type of Output: unformatted Logfiles for Host: benji.localdomain ##################################################################

--------------------- Cron Begin ------------------------

Commands Run: User root: /sbin/dump 0uf /backup/level0/backup-L0-`date +%Y-%d-%m`.dmp /home: 2 Time(s) /sbin/dump 1uf /backup/level1/backup-L1-`date +%Y-%d-%m`.dmp /home: 5 Time(s) /sbin/dump 2uf /backup/level2/backup-L2-`date '+: 4 Time(s) /sbin/dump 2uf /backup/level2/backup-L2-`date +: 5 Time(s) /sbin/dump 2uf /backup/level2/backup-L2-`date +%Y-%d-%m`.dmp /home: 14 Time(s) /sbin/dump 2uf /backup/level2/backup-L2.dmp /home: 2 Time(s) dump 1uf /backup/level1/backup-daily-$(date +: 1 Time(s) dump 2uf /backup/level2/backup-L2.dmp /home: 2 Time(s) dump 2uf /backup/level2/backup-daily-$(date +: 9 Time(s)

logwatch report using High level of detail

44

CIS 191 - Lesson 12

logwatchpersonal crontab deleted: 3 Time(s) personal crontab edited: 6 Time(s) personal crontab listed: 7 Time(s) personal crontab reloaded: 7 Time(s) personal crontab replaced: 11 Time(s) run-parts /etc/cron.daily: 1 Time(s) run-parts /etc/cron.hourly: 24 Time(s) run-parts /etc/cron.monthly: 1 Time(s)

---------------------- Cron End -------------------------

--------------------- sendmail Begin ------------------------

STATISTICS ----------

Bytes Transferred: 90737 Messages Processed: 92 Addressed Recipients: 92

Message recipients per delivery agent: Name # Rcpts local 46 --------------------- TOTAL: 46 in addition to 46 relay submission(s) from MSP

45

logwatch report using High level of detail continued

CIS 191 - Lesson 12

logwatch

Message Size Distribution: Range # Msgs KBytes 0 - 10k 92 88 ---------------------------------- TOTAL 92 88 Avg. Size 0

Top 10 Email Recipients ---------------------------------- [email protected] : 46 emails

Top relays (recipients/connections - min 10 rcpts, max 25 lines): 46/46: benji.localdomain [127.0.0.1] 46/46: root@localhost

---------------------- sendmail End -------------------------

--------------------- Syslogd Begin ------------------------

Syslogd started 1 Time(s)

---------------------- Syslogd End -------------------------

46


CIS 191 - Lesson 12

logwatch

--------------------- Disk Space Begin ------------------------

Filesystem Size Used Avail Use% Mounted on /dev/sda1 2.9G 2.5G 272M 91% / /dev/sda5 289M 234M 41M 86% /opt /dev/sda3 487M 77M 385M 17% /var /dev/sda7 196M 5.6M 181M 3% /home

---------------------- Disk Space End -------------------------

###################### Logwatch End #########################

&

47


CIS 191 - Lesson 12

logwatch

--------------------- SSHD Begin ------------------------ SSHD Killed: 1 Time(s)

SSHD Started: 1 Time(s)

Disconnecting after too many authentication failures for user: guest90 : 1 Time(s)

Failed logins from: 76.254.22.196 (adsl-76-254-22-196.dsl.pltn13.sbcglobal.net): 2 times 201.7.115.194 (201-7-115-194.spopa302.ipd.brasiltelecom.net.br): 2135 times 210.240.12.14: 20 times

Illegal users from: 201.7.115.194 (201-7-115-194.spopa302.ipd.brasiltelecom.net.br): 564 times 210.240.12.14: 42 times

Users logging in through sshd: guest: 76.254.22.196 (adsl-76-254-22-196.dsl.pltn13.sbcglobal.net): 2 times jimg: 70.132.20.25 (adsl-70-132-20-25.dsl.snfc21.sbcglobal.net): 7 times ordazedw: 76.254.22.196 (adsl-76-254-22-196.dsl.pltn13.sbcglobal.net): 1 time root: 63.249.86.11 (dsl-63-249-86-11.cruzio.com): 3 times 70.132.20.25 (adsl-70-132-20-25.dsl.snfc21.sbcglobal.net): 1 time rsimms: 63.249.86.11 (dsl-63-249-86-11.cruzio.com): 2 times

the bad boys trying to break in … this is why you need strong passwords

48

CIS 191 - Lesson 12

logwatch

49

http://ws.arin.net/whois/?queryinput=201.7.115.194

CIS 191 - Lesson 12

/var/log/secure

The bad boys trying to break in as root … this is why you need strong passwords

Nov 30 06:02:24 opus sshd[27486]: Failed password for root from 202.119.60.132 port 36322 ssh2Nov 30 06:02:24 opus sshd[27487]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:02:27 opus sshd[27488]: pam_unix(sshd:auth): authentication failure; logname= uid=0

euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn user=rootNov 30 06:02:29 opus sshd[27488]: Failed password for root from 202.119.60.132 port 36846 ssh2Nov 30 06:02:29 opus sshd[27489]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:02:32 opus sshd[27490]: pam_unix(sshd:auth): authentication failure; logname= uid=0





euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn user=root

50

CIS 191 - Lesson 12

/var/log/secure

The bad boys trying to break in, guessing usernames … this is why you need strong passwords

Nov 30 06:27:20 opus sshd[28166]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=cool.nju.edu.cn

Nov 30 06:27:20 opus sshd[28166]: pam_succeed_if(sshd:auth): error retrieving information about user shop

Nov 30 06:27:23 opus sshd[28166]: Failed password for invalid user shop from 202.119.60.132 port 40634 ssh2

Nov 30 06:27:23 opus sshd[28167]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:27:25 opus sshd[28168]: Invalid user lady from 202.119.60.132Nov 30 06:27:25 opus sshd[28169]: input_userauth_request: invalid user ladyNov 30 06:27:25 opus sshd[28168]: pam_unix(sshd:auth): check pass; user unknownNov 30 06:27:25 opus sshd[28168]: pam_unix(sshd:auth): authentication failure; logname= uid=0

euid=0 tty=ssh ruser= rhost=cool.nju.edu.cnNov 30 06:27:25 opus sshd[28168]: pam_succeed_if(sshd:auth): error retrieving information about

user ladyNov 30 06:27:28 opus sshd[28168]: Failed password for invalid user lady from 202.119.60.132

port 41408 ssh2Nov 30 06:27:28 opus sshd[28169]: Received disconnect from 202.119.60.132: 11: Bye ByeNov 30 06:27:30 opus sshd[28170]: Invalid user lady from 202.119.60.132Nov 30 06:27:30 opus sshd[28171]: input_userauth_request: invalid user ladyNov 30 06:27:30 opus sshd[28170]: pam_unix(sshd:auth): check pass; user unknownNov 30 06:27:30 opus sshd[28170]: pam_unix(sshd:auth): authentication failure; logname= uid=0

euid=0 tty=ssh ruser= rhost=cool.nju.edu.cnNov 30 06:27:30 opus sshd[28170]: pam_succeed_if(sshd:auth): error retrieving information about

user lady

51

cis 191 - lesson 12

Documents

root root

root utmp

ascii textmessages

ascii textrpmpkgs

ascii english textmessages

ascii textroot

ascii textspooler

ascii textsamba