suse advanced troubleshooting: the boot process · suse advanced troubleshooting: the boot process...

www.novel l .comNovell Training Services

AT T L I V E 2 0 1 2 L A S V E G A S

SUSE Advanced Troubleshooting: The Boot ProcessLecture

S U S 2 1

Novell, Inc. Copyright 2012-ATT LIVE-1-HARDCOPY PERMITTED. NO OTHER PRINTING, COPYING, OR DISTRIBUTION ALLOWED.

Proprietary StatementCopyright © 2011 Novell, Inc. All rights reserved.

Novell, Inc., has intellectual property rights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed on the Novell Legal Patents Web page (http://www.novell.com/company/legal/patents/) and one or more additional patents or pending patent applications in the U.S. and in other countries.

No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the publisher.

Novell, Inc.404 Wyman Street, Suite 500Waltham, MA 02451U.S.A.www.novell.comNovell TrademarksFor Novell trademarks, see the Novell Trademark and Service Mark list (http://www.novell.com/company/legal/trademarks/tmlist.html).Third-Party MaterialsAll third-party trademarks are the property of their respective owners.Software PiracyThroughout the world, unauthorized duplication of software is subject to bothcriminal and civil penalties.

If you know of illegal copying of software, contact your local Software Antipiracy Hotline. For the Hotline number for your area, access Novell’s World Wide Web page (http://www.novell.com) and look for the piracy page under “Programs.”Or, contact Novell’s anti-piracy headquarters in the U.S. at 800-PIRATES (747-2837) or 801-861-7101.

DisclaimerNovell, Inc., makes no representations or warranties with respect to the contents or use of this documentation, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose.

Further, Novell, Inc., reserves the right to revise this publication and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. Further, Novell, Inc., makes no representations or warranties with respect to any software, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to make changes to any and all parts of Novell software, at any time, without any obligation to notify any person or entity of such changes.

Any products or technical information provided under this Agreement may besubject to U.S. export controls and the trade laws of other countries. You agree to comply with all export control regulations and to obtain any required licenses or classification to export, re-export or import deliverables. You agree not to export or re-export to entities on the current U.S. export exclusion lists or to any embargoed or terrorist countries as specified in the U.S. export laws. You agree to not use deliverables for prohibited nuclear, missile, or chemical biological weaponry end uses. See the Novell International Trade Services Web page (http://www.novell.com/info/exports/) for more information on exporting Novell software. Novell assumes no responsibility for your failure to obtain any necessary export approvals.

This Novell Training Manual is published solely to instruct students in the use of Novell networking software. Although third-party application software packages are used in Novell training courses, this is for demonstration purposes only and shall not constitute an endorsement of any of these software applications.

Further, Novell, Inc. does not represent itself as having any particular expertisein these application software packages and any use by students of the same shall be done at the student’s own risk.

Front MatterProprietary StatementCopyright © 2011 Novell, Inc. All rights reserved.

Novell, Inc., has intellectual property r ights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed on the Novell Legal Patents Web page (http://www.novell.com/company/legal/patents/) and one or more additional patents or pending patent applications in the U.S. and in other countries.

No part of this publication may be reproduced, photocopied, stored on a retr ieval system, or transmitted without the express written consent of the publisher.

Novell, Inc.404 Wyman Street, Suite 500Waltham, MA 02451U.S.A.www.novell.comNovell TrademarksFor Novell trademarks, see the Novell Trademark and Service Mark list (http://www.novell.com/company/legal/trademarks/tmlist.html).Third-Party MaterialsAll third-party trademarks are the property of their respective owners.Software PiracyThroughout the world, unauthorized duplication of software is subject to bothcriminal and civil penalties.

If you know of illegal copying of software, contact your local Software Antipiracy Hotline. For the Hotline number for your area, access Novell’s World Wide Web page (http://www.novell.com) and look for the piracy page under “Programs.”Or, contact Novell’s anti-piracy headquarters in the U.S. at 800-PIRATES (747-2837) or 801-861-7101.

DisclaimerNovell, Inc., makes no representations or warranties with respect to the contents or use of this documentation, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose.

Further, Novell, Inc., reserves the right to revise this publication and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. Further, Novell, Inc., makes no representations or warranties with respect to any software, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to make changes to any and all parts of Novell software, at any time, without any obligation to notify any person or entity of such changes.

Any products or technical information provided under this Agreement may besubject to U.S. export controls and the trade laws of other countries. You agree to comply with all export control regulations and to obtain any required licenses or classification to export, re-export or import deliverables. You agree not to export or re-export to entities on the current U.S. export exclusion lists or to any embargoed or terrorist countr ies as specified in the U.S. export laws. You agree to not use deliverables for prohibited nuclear, missile, or chemical biological weaponry end uses. See the Novell International Trade Services Web page (http://www.novell.com/info/exports/) for more information on exporting Novell software. Novell assumes no responsibility for your failure to obtain any necessary export approvals.

This Novell Training Manual is published solely to instruct students in the use of Novell networking software. Although third-party application software packages are used in Novell training courses, this is for demonstration purposes only and shall not constitute an endorsement of any of these software applications.

Further, Novell, Inc. does not represent itself as having any particular expertisein these application software packages and any use by students of the same shall be done at the student’s own risk.


Copying all or part of this manual, or distributing such copies, is strictly prohibited. To report suspected copying, please call 1-800-PIRATES

Contents

SECTION 1: The Boot Process 4

SECTION 2: The Boot Process Overview 6

SECTION 3: Troubleshooting Big Picture 8

SECTION 4: Boot Process Details 10

SECTION 5: Troubleshooting Techniques 39

SECTION 6: Boot Process Labs 49

Table of Contents


4

SUSE Advanced Troubleshooting: The Boot Process


TroubleshootingSection 01: The Boot Process

The purpose of this course is to understand the boot process, learn troubleshooting techniques that will help you solve boot related issues, and apply those techniques in the lab.


5



Objectives

• Boot Process Overview• Troubleshooting Big Picture• Boot Process Details


6



The Boot ProcessSection 02: The Boot Process Overview



7



Graphical boot process overview

init

BIOS

/boot/grub/stage2

boot

rc

/etc/init.d/boot.d/S*/etc/init.d/boot.local

/etc/init.d/rc[prev_run_lvl].d/K*

/etc/init.d/rc[cur_run_lvl].d/S*

Initial RamdiskKernel

MBR/boot/grub/stage1

mingetty

xdminit

The boot process is linear by nature. Understanding the general process of how a server boots will help to narrow down boot related problems.

This graphical overview of the boot process will be used throughout the presentation to indicate which topic will be covered and when. The topic to be covered will be highlighted.

The troubleshooting table is based on this orderly boot process. An important troubleshooting technique is to be aware of what messages appear on the screen when each of these stages is reached. This will help you narrow down boot related problems.


8



The Boot ProcessSection 03: Troubleshooting Big Picture



9



Troubleshooting big picture

init

BIOS

MBR

Bootloader

Kernel

Initial Ramdisk

boot

boot.local

rc

mingetty

xdminit

Boot Installed System Chroot Installed System, init 1PROMPT_FOR_CONFIRM

You can apply the troubleshooting process to the sequential nature of the boot process itself. Boot Installed System (BIS) is a method where you use the boot loader, kernel and ram disk from the installation media to boot your installed system. BIS bypasses everything in the Boot Installed System bubble above. Consequently if BIS works, your boot problem is contained within that bubble. BIS lets you basically narrow your boot problem in half. If BIS still fails, your problem is most likely in the bubble on the right. There are three primary troubleshooting techniques used in this bubble to help narrow down problems, they are Chroot Installed System (CIS), administrative run levels like run level 1 (init 1), and using flow control from /etc/sysconfig/boot to slow down the loading of system daemons (PROMPT_FOR_CONFIRM).


10



The Boot ProcessSection 04: Boot Process Details



11



BIOS & MBR

init

BIOS

/boot/grub/stage2

boot

rc






mingetty

xdminit

Considering the BIOS and the transition from BIOS to the first stage boot loader.


12



BIOS

• The BIOS determines the order and location of hard drives

• The BIOS loads the first 512 bytes on the hard disk with the the boot flag set into memory

• Control is transferred to the MBR• What's on screen?‒ POST Messages

The BIOS boots from the master boot record (MBR). It basis the boot device based on boot order and the boot flag set on the disk. The BIOS banner and POST messages are clear on-screen indicators for your location in the boot process.


13



MBR (Master Boot Record)

• The first 512 bytes on the hard disk‒ Divided into three parts‒ Code Area: First 446 bytes, mostly used by boot loaders‒ Partition Table: 64 bytes, four 16 byte entries‒ Signature: 2 bytes, also known as the Magic Number. The BIOS

executes the Code Area if the Magic Number 0x55AA is found.• What's on screen?‒ Usually just a black screen, but you might see “No Operating

System Found”

An MBR as described in this course assumes an msdos disk type, as opposed to gpt, sun or others.


14



Troubleshooting BIOS/MBR

• Update BIOS and hardware component firmware• Mark a disk partition bootable

If there are no partitions marked bootable, you will need to set the bootable flag on the partition containing /boot. This is usually the root (/) partition or the /boot partition if it is separate. To see if a partition is marked bootable, run fdisk -l, which would show a partition with the boot flag like this:

Device Boot Start End Blocks Id System/dev/sda1 * 2048 417791 207872 83 Linux

If sda1 was not marked bootable, set the boot flag by running:

parted -s /dev/sda toggle 1 boot

If you don't have a boot partition at all, you may have a corrupted MBR. You should restore it from backup, or restore it from the supportconfig's fs-diskio.txt file's sfdisk entry. This process will be documented in the lab manual.


15



Bootloader: GRUB

init

BIOS

/boot/grub/stage2

boot

rc






mingetty

xdminit

Considering the GRUB second stage boot loader.


16



GRUB

• Loads the kernel and initrd into memory and boots• Consists of three stages: stage1, stage1.5 and stage2• /boot/grub/stage1 is written to the MBR‒ It's only purpose is to load stage2‒ stage2 location is hard coded into stage1

• /boot/grub/*stage1_5 provide file system drivers• /boot/grub/stage2 provides the main boot loader

functions‒ Reads /boot/grub/menu.lst‒ Loads kernel and initrd via direct filesystem access

Stage 1 is copied into the code section of the MBR. It's purpose is to launch the appropriate stage 1.5 for the filesystem, and then load stage 2. Stage 2 is the main boot loader code that reads the menu.lst configuration file and loads the kernel and ramdisk into memory.


17



GRUB

• Reads disk devices from the BIOS or other disk controllers

• Naming convention‒ (hd0,0) – hd0 is the first disk detected, “,0” is the first partition

on that disk‒ (hd1,2) – The second disk detected on the third partition‒ GRUB does not distinguish among IDE, SCSI or SATA devices

• The devices are saved in:‒ /boot/grub/device.map

• What's on screen?‒ GRUB menu or grub> prompt

You know you have reached the second stage boot loader if you see the GRUB menu or the grub> prompt.


18



Troubleshooting GRUB

• A grub RPM validation will not check /boot/grub/stage*.‒ The /usr/lib/grub/* files are owned by the RPM and copied to

/boot/grub with an RPM post install script• Reinstall the grub RPM package‒ yast -i grub

• Reinstall GRUB‒ grub-install

• Recreate or check the /boot/grub/menu.lst‒ YaST, System, Boot Loader, Other, Propose New

• grub> tab completion

If you are having boot problems, reinstalling grub and/or the grub package can help resolve the issue.

GRUB's tab completion is a quick trick to help find a file. The GRUB find command is used to identify the devices on which that file exists. It is not used to search for an unknown file.


19



vmlinuz & initrd

init

BIOS

/boot/grub/stage2

boot

rc






mingetty

xdminit

Considering the kernel and the initial ramdisk.


20



Kernel

• The GRUB kernel command loads the kernel file directly from the /boot directory

• The kernel file is /boot/vmlinuz-3.* and is owned by the kernel base RPM package

• Additional /lib/modules kernel drivers are owned by the kernel RPM package

• /boot/grub/menu.lst provides the default location for the kernel and menu screen for booting

• The kernel uses root= to help identify the root partition• What's on screen? kernel

The kernel is the heart of the SUSE Linux operating system. The kernel /boot/vmlinuz-3.* file is provided by the kernel RPM package and is loaded into memory by GRUB. GRUB will display “kernel” just prior to booting the kernel. Once the kernel is booting, you will see the familiar kernel time stamped messages with the format [ 0.0000000].


21



Troubleshooting the kernel

• Validate the kernel and kernel base packages‒ Boot Installed System‒ Run rpm -Vf /boot/vmlinuz-* to verify the kernel package

• If the kernel does not pass validation, reinstall the kernel or kernel base rpm

• Make sure root= in /boot/grub/menu.lst is pointing to the correct root device

• If you are at a grub> prompt try loading the kernel without any options

Make sure the root device matches in /boot/grub/menu.lst root= parameter, the /etc/fstab, and the ramdisk. Once you have checked the root device in menu.lst and fstab, you can run mkinitrd to make sure that device is put correctly into the ramdisk. It is good to remember that kernel drivers are part of the kernel RPM package. Verifying the kernel RPM package will not only validate the kernel itself, but all the drivers associated with it.


22



Initial ramdisk

• The initial ramdisk default file is /boot/initrd-3.*• It is owned by the kernel base package‒ The initrd file is not included in an rpm validation

• Provides an initial root filesystem for the kernel• Contains all kernel drivers, binaries or configuration

files needed to find and mount the root filesystem• The mkinitrd command creates the initial ramdisk• What's on screen? initrd

Once GRUB loads the kernel into memory, it then loads the ram disk into memory. The kernel needs a root filesystem to operate. When the kernel is first loaded, it is not aware of the root filesystem on the hard disk, so a temporary root filesystem is provided using the initial ram disk. The ram disk is a compressed cpio archive and contains all the drivers and binaries necessary to find and mount the root partition. Once the root partition is mounted, all additional drivers and filesystems can be loaded and mounted from the root filesystem on the hard disk.


23



Troubleshooting initial ramdisk

• Boot Installed System• Edit /etc/sysconfig/kernel‒ Make sure the INITRD_MODULES variables has the correct

drivers included to find the root partition• From the command prompt, run 'mkinintrd'• Check /boot and make sure the symbolic links below

are pointing to the correct kernel and initrd‒ /boot/vmlinuz‒ /boot/initrd

• Reboot to test

The best troubleshooting option for a problematic ram disk is to rebuild it. Make sure /etc/sysconfig/kernel INITRD_MODULES has a correct module list, and then run mkinitrd to rebuild the ram disk.

You can extract the contents of the ramdisk as follows:1. mkdir -p /boot/tmp2. cd /boot/tmp3. zcat /boot/initrd-3.* | cpio -id4. ls -l /boot/tmp


24



init (In the RAMDISK)

init

BIOS

/boot/grub/stage2

boot

rc






mingetty

xdminit

There are two “init” commands used in the boot process. The first is a script created by the nkinitrd command that gets embedded into the root directory of the ramdisk (/boot/initrd). The other is a dynamically linked executable installed on the system in /sbin/init.

Now considering the init in the ramdisk.


25



init (RAMDISK)

• Responsible for finding the root filesystem• Mounts the root filesystem• Establishes network connections if needed for root• Loads modules associated the root device• What's on screen?‒ Loading ‒ Mounting root partition

The init script is located in the root directory of the ram disk. It loads all necessary modules for mounting the root filesystem. The init script also runs all other binaries required to find and mount the root filesystem.


26



Troubleshooting init (RAMDISK)

• Boot Installed System• Edit /etc/sysconfig/kernel‒ Make sure the INITRD_MODULES variables has the correct

drivers included to find the root partition• From the command prompt, run 'mkinintrd'• Check /boot and make sure the symbolic links below

are pointing to the correct kernel and initrd‒ /boot/vmlinuz‒ /boot/initrd

• Reboot to test

The mkinitrd command creates the ram disk init script. The best troubleshooting technique to try is to check the /etc/sysconfig/kernel INITRD_MODULES variable and run mkinitrd. The init script will be recreated.


27



/sbin/init process

init

BIOS

/boot/grub/stage2

boot

rc






mingetty

xdminit

Considering /sbin/init


28



/sbin/init functionality

• /sbin/init is the master process• It is always PID 1• The kernel is programmed to launch /sbin/init by

default• All other processes are forked from init• It reads and processes /etc/inittab• What's on screen?‒ INIT: version 2.86 booting

Once the root filesystem is mounted, the kernel run /sbin/init. It is the master process and runs with a process ID of 1. It is responsible for starting system daemons. It's configuration file is /etc/inittab.


29



Troubleshooting /sbin/init

• Validate sysvinit, which owns /sbin/init• Validate aaa_base, which owns /etc/inittab• Review the man pages for inittab(5)• Modify /etc/sysconfig/boot to:‒ Enable PROMPT_FOR_CONFIRM‒ Try RUN_PARALLEL=”no”‒ Enable FLOW_CONTROL

• Bypass /sbin/init with Chroot Installed System (CIS), init=/bin/bash or init=/bin/sash

• Boot to single user mode to minimize running daemons‒ Boot Options: ' 1' or run 'init 1' at the prompt

Make sure the associated RPM packages pass validation. Confirm the /etc/inittab has been configured properly. The init= kernel parameter can be used to bypass /sbin/init.


30



init: boot & boot.local

init

BIOS

/boot/grub/stage2

boot

rc






mingetty

xdminit

Considering /etc/init.d/boot


31



init: boot & boot.local

• Runs all of the /etc/init.d/boot.d scripts• What's on screen when boot starts?‒ System Boot Control: Running /etc/init.d/boot‒ System Boot Control: The system has been setup

• What's on screen when boot.local starts?‒ Done‒ System Boot Control: Running /etc/init.d/boot.local

• init runs /etc/init.d/boot to run all boot level init scripts

The boot.d scripts setup the basic framework of the server, like mounting all additional filesystems. They are run first, before run level daemons. Generally they are run once, and do not keep running as system daemons. Once the boot level scripts are complete, /etc/init.d/boot runs the /etc/init.d/boot.local script. You can put any commands in this file you want to run prior to starting the run level scripts.


32



Troubleshooting init: boot & boot.local

• If a boot service fails, modify /etc/sysconfig/boot to help determine which one‒ PROMPT_FOR_CONFIRM‒ FLOW_CONTROL

• If the server will not boot, use‒ Chroot Installed System (CIS)

• Files needed and it's (RPM)‒ /bin/bash (bash)‒ /etc/init.d/boot (aaa_base)‒ /etc/init.d/boot.d/* (various packages)

The boot messages scroll pretty fast. You can slow down the messages using flow control or prompt for confirm. Flow control lets you press Ctrl-S to stop the screen long enough to read the error messages, and Ctrl-Q to resume the screen. Prompt for confirm tells the operating system to prompt you to load each individual service during the boot process. The settings are found in the /etc/sysconfig/boot file.


33



init: Resource Control (rc)

init

BIOS

/boot/grub/stage2

boot

rc






mingetty

xdminit

Considering /etc/init.d/rc


34



Resource Control (rc)

• Starts all of the daemons in a specified run level• init uses the /etc/inittab to determine the default runlevel‒ # The default runlevel is defined here‒ id:3:initdefault:

• What's on screen when rc starts?‒ Done‒ Master Resource Control:‒ Master Resource Control: runlevel 3 has been reached‒ Skipped services in runlevel 3:

• init uses /etc/init.d/rc to run all runlevel scripts

The resource control script controls the scripts in a specified run level. The run level is specified in the /etc/inittab file or on the kernel command line.


35



Troubleshooting Resource Control

• If a system daemon fails, modify /etc/sysconfig/boot to help determine which one‒ PROMPT_FOR_CONFIRM‒ FLOW_CONTROL

• If the server will not boot, use‒ Run Level S or 1‒ Chroot Installed System (CIS)

• Files needed and it's (RPM)‒ /bin/bash (bash)‒ /etc/init.d/rc (aaa_base)‒ /etc/init.d/rc?.d/* (various packages)

You can slow down the messages using flow control or prompt for confirm. Flow control lets you press Ctrl-S to stop the screen long enough to read the error messages, and Ctrl-Q to resume the screen. Prompt for confirm tells the operating system to prompt you to load each individual service during the boot process. The settings are found in the /etc/sysconfig/boot file.


36



init: User Login

init

BIOS

/boot/grub/stage2

boot

rc






mingetty

xdminit

Considering the user login


37



User Login

• mingetty is used for tty1 – tty6• xdm is used for tty7, the graphical login screen• init uses /etc/inittab to determine how the login

screens will behave‒ By default init respawns the mingetty and xdm processes

everytime they are terminated‒ This allows you to login again after logging out

• What's on screen?‒ Login prompt

The /sbin/init process reads /etc/inittab to determine which application to run on which terminal. It runs mingetty for virtual consoles 1 through 6 and an X display manager on terminal 7. The /etc/sysconfig/displaymanager is used for the display manager configuration.


38



Troubleshooting User Login

• Check /etc/inittab to make sure /sbin/init is managing logins correctly

• Files affecting logins with it's (RPM)‒ /etc/nologin (n/a)‒ /sbin/mingetty (mingetty)‒ /etc/pam.d/login (login)‒ /etc/sysconfig/displaymanager (n/a)‒ /etc/sysconfig/windowmanager (n/a)

• If you usually login from the X console, try logging into the virtual consoles or vise versa

• Try logging into a newly created test user

If you have a graphical display problem, you can change to run level 3 (init 3), run sax2 -a, and then change back to run level 5 (init 5). This allows sax2 to automatically probe the X configuration.


39



The Boot ProcessSection 05: Troubleshooting Techniques



40



Objectives

• Troubleshooting Procedure• Boot Installed System (BIS)• Administrative Run Levels• Chroot Installed System (CIS)• Flow Control• Troubleshooting Table


41



Troubleshooting Procedure

• Let the server boot until it fails• Write down verbatim what's on the screen• Match on-screen landmarks to the Troubleshooting

Table• Use Boot Installed System (BIS) to bypass GRUB,

kernel and ram disk issues• Use an administrative run level for daemon failure• Use Chroot Installed System (CIS) if all else fails• Address the issues and files associated with the

location of the boot failure (see Troubleshooting Table)

Notes:


42



Boot Installed System (BIS)

• Provided on the installation media• Bypasses issues relating to:‒ BIOS, boot loader, kernel and ram disk

• Allows the server to be up while troubleshooting

Boot Installed System (BIS) Procedure

1.Used mostly in lines 1-7 of the Troubleshooting Table.2.Boot from DVD13.Select “Installation”4.Accept the License Agreement5.Click “Next”, and “Next” to skip media checks6.Select “Repair Installed System” and “Next”7.Select “Expert Tools”8.Select “Boot Installed System”

NOTE: Select “Repair Installed System” directly from the DVD boot menu does not probe as thoroughly as the “Repair Installed System” from the Installation option.


43



Administrative Run Levels

• Run level S and 1 use the installed system• Minimal system daemons are started• Useful when attempting to isolate a misbehaving

system daemon

Run level S and 1 are very similar to chroot installed system (CIS), as far as run levels go. However, run levels S and 1 use the installed system's boot loader, kernel and ram disk to boot. It just doesn't start all the system processes like run level 3 or 5 do. So, run level S and 1 are preferred over CIS. There are a couple of ways to change to run level S or 1. You could just type init 1. However, if you are troubleshooting system processes that fail at boot time or cause the server to misbehave as a result; you will want to reboot the server, bypass the default run level and boot into the run level 1. To boot to run level 1, do the following:1) Boot the server normally2) Select the kernel you usually boot to3) Tab or click in the “Boot Options” field4) Append “ 1” (a space followed by the number 1) to the boot options line5) Type root's password

If you need network access, run “/etc/init.d/network start”


44



Chroot Installed System (CIS)

• CIS uses rescue mode to access the installed system• Boot Installed System (BIS) and Run Level 1 are

preferred• When is CIS used?‒ BIS and run level 1 fail‒ When the root user account does not work properly, use CIS

Notes:


45



Chroot Installed System (CIS) Procedure

1)Used mostly in lines 8 and above of the troubleshooting table.2)Boot from DVD13)Select “Rescue System”, Rescue login: root4)Your first goal is to find and mount the root “/” partition, so we can see /etc/fstab

1) Run cat /proc/partitions to find the disk devices the OS sees2) For each device, display the partition table

lsboot:~ # parted s /dev/sda printDisk geometry for /dev/sda: 0kB 2147MBDisk label type: msdosNumber Start End Size Type File system Flags1 32kB 214MB 214MB primary ext2 boot, type=832 214MB 535MB 321MB primary linuxswap type=823 535MB 2147MB 1612MB extended lba, type=0f5 535MB 1012MB 477MB logical reiserfs type=836 1012MB 1596MB 584MB logical reiserfs type=837 1596MB 2147MB 551MB logical reiserfs type=83

3) You can ignore type 82 swap and type 0f extended partitions4) To find the root partition, you may need to just guess. For example,

1) mount /dev/sda1 /mnt2) ls -l /mnt3) If the /mnt directory listing shows /etc and /root, then its the root partition4) Repeat these steps for each device until you find root. In this case, the root device is /dev/sda65) mount /dev/sda6 /mnt

5)Mount all additional file systems relative to /mnt1) Once you have mounted the root filesystem, run cat /mnt/etc/fstab to see all the other filesystem

mount points.2) Mount all file systems manually as shown in /mnt/etc/fstab.

mount /dev/sda1 /mnt/bootmount /dev/sda5 /mnt/varmount /dev/sda7 /mnt/usr

3) Rebind the /proc, /sys and /dev filesystems. mount rbind /proc /mnt/procmount rbind /sys /mnt/sysmount rbind /dev /mnt/dev

6)Chroot to the installed system: chroot /mnt7)To return to the rescue system, type exit.


46



Flow Control

• File: /etc/sysconfig/boot• FLOW_CONTROL‒ Use Ctrl-S to stop and Ctrl-Q to resume

• PROMPT_FOR_CONFIRM‒ Prompts to load each service at boot time

• SHIFT-PGUP‒ Scrolls up a couple of pages on the console boot screen

The normal boot messages display on the screen very fast. There are ways to slow it down and test each service as it loads. The boot messages are controlled by variables set in the /etc/sysconfig/boot file.

FLOW_CONTROL=”yes”Allows you to stop the boot process messages using Ctrl-S and resume them with Ctrl-Q.

PROMPT_FOR_CONFIRM=”yes”CONFIRM_PROMPT_TIMEOUT=”5”

This will display the prompt:

Enter interactive startup mode? y/[n](5s)

You will need to select “y” to enter interactive startup mode within the CONFORM_PROMPT_TIMEOUT period, otherwise the server will boot normally without prompting to load system daemons. After you enter interactive startup mode, you will be prompted to load each service with the following:

Start service , (Y)es/(N)o/(C)ontinue? [y]

The CONFIRM_PROMPT_TIMEOUT value also applies to each service start prompt. This was not true with earlier versions of SLES.

Once the server has booted up, you can use Shift-PgUp to scroll up about two screens worth of boot messages, regardless of the /etc/sysconfig/boot settings. However, if you switch to other consoles (ie tty2), you will not be able to use this keystroke.


47



Troubleshooting Table

• Troubleshooting Table's Purpose‒ Narrow down boot related issues faster

• Identifies location in the boot process• Associated files at each step• On-screen landmarks to help narrow down where you

are in the boot process• Troubleshooting suggestions at each step

Troubleshooting Table

# Location Files On-Screen Landmarks Troubleshooting/ Potential Fixes

1 BIOS N/A BIOS messages Update firmwareMake sure a disk device is marked bootable

2 MBR /boot/grub/stage1 GRUB loading stage2... BISgrub-install /dev/

3 GRUB /boot/grub/stage2/boot/grub/menu.lst

GRUB menu or grub> prompt BISgrub-install /dev/Check /boot/grub/menu.lst

4 GRUB /boot/vmlinuz/boot/initrd

root (hd?,?)kernel /initrd /

Reinstall kernel RPMmkinitrd

5 kernel /boot/vmlinuz Kernel driver info scrollingRAMDISK driver initialized:

BISReinstall the kernel RPM

6 initrd /boot/initrd/etc/sysconfig/kernel

RAMDISK: BISExtract and view the /boot/initrd

7 initrd:init /init in /boot/initrd/etc/sysconfig/kernel

Starting udevdCreating devicesLoad There will be “loading” for each module defined in /etc/sysconfig/kernel INITRD_MODULES and their dependent modules.

BISMkinitrd creates the initrd:ini file


48



Troubleshooting Table (cont)

# Location Files On-Screen Landmarks Troubleshooting/ Potential Fixes

8 sbin:init /sbin/init/etc/inittab

INIT: version 2.86 booting init 1, then CISUse boot options init=/bin/bash or init=/bin/sash to bypass running /sbin/init

9 sbin:init:boot /bin/bash/etc/init.d/boot/etc/init.d/boot.d/*/etc/sysconfig/boot

System Boot Control: Running /etc/init.d/bootEach service shows: done, failed or skippedSystem Boot Control: The system has been setup

init 1 starts the minimum servicesCIS starts no servicesPROMPT_FOR_CONFIRM=”yes”RUN_PARALLEL=”no”FLOW_CONTROL=”yes” (Ctrl-S stops, Ctrl-Q resumes)

10 sbin:init:boot /etc/init.d/boot.local System Boot Control: Running /etc/init.d/boot.local init 1 or CIS


INIT: Entering runlevel: 3 init 1 or CIS

12 sbin:init:rc /bin/bash/etc/init.d/rc/etc/init.d/rc?.d/*/etc/sysconfig/boot

Master Resource Control: previous runlevel:N, switching to runlevel: 3Master Resource Control: Running /etc/init.d/before.localEach service shows: done, failed or skippedMaster Resource Control: Running /etc/init.d/after.localMaster Resource Control: runlevel 3 has been reachedSkipped services in runlevel 3:

Init 1 or CISPROMPT_FOR_CONFIRM=”yes”RUN_PARALLEL=”no”FLOW_CONTROL=”yes”


N/A init 1 or CISinit refers to it's inittab file to know how to run the login programs

14 sbin:init:mingetty /etc/issue/etc/motd/etc/nologin/sbin/mingetty/etc/pam.d/login

login:

init 1 bypasses mingettyCIS

15 sbin:init:X /etc/sysconfig/ displaymanger/etc/sysconfig/ windowmanager

Graphical login screen init 1 bypasses X loginCIS


49



The Boot ProcessSection 06: Labs



50



LABS 1-10: Boot Process Labs

Summary: In these exercises, you will be simulating several boot failures and troubleshooting them using the troubleshooting techniques described above.

Special InstructionsComplete labs 1 through 10. If you have time, continue working on the

additional labs.

Duration: The remainder of classmachine

You are not expected to complete all the labs. Labs 25-28 may be broken in several ways, making the issue more complex. The purpose of many labs with varying degrees of difficulty is to meet the needs of a wide range of system administrators.


51




52



Unpublished Work of SUSE. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary, and trade secret information of SUSE. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.


SUSE_Cover_SlideSUSE_Legal_SlideSUSE_TOC_SlideSlide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40Slide 41Slide 42Slide 43Slide 44Slide 45Slide 46Slide 47Slide 48Slide 49SUSE_Lab_SlideSUSE_Splash_SlideSUSE_Legal_Final_SlideSUSE_Cover_SlideSUSE_Legal_SlideSUSE_TOC_SlideSlide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Slide 14Slide 15Slide 16Slide 17Slide 18Slide 19Slide 20Slide 21Slide 22Slide 23Slide 24Slide 25Slide 26Slide 27Slide 28Slide 29Slide 30Slide 31Slide 32Slide 33Slide 34Slide 35Slide 36Slide 37Slide 38Slide 39Slide 40Slide 41Slide 42Slide 43Slide 44Slide 45Slide 46Slide 47Slide 48Slide 49SUSE_Lab_SlideSUSE_Splash_SlideSUSE_Legal_Final_Slide

suse advanced troubleshooting: the boot process · suse advanced troubleshooting: the boot process...

Documents