basics of kernel crash dump analysis - red...
Post on 06-Feb-2018
254 Views
Preview:
TRANSCRIPT
-
BasicsofKernelCrashDumpAnalysis
-
Agenda: Basicsofkernelcrashdumpanalysis
Initialanalysisofkernelcrashdump
Initialanalysisofmemorysubsystem
-
Whatarethetools/datarequiredtoperformakernelcrashdumpanalysis?
a)crashutility(/usr/bin/crash)
Providedby"crash"package.Eg:crash6.0.42.el6.x86_64.rpm
Nocrossplatformsupport.oUse32bitversionofcrashforvmcorecapturefrom32bitkernel.oUse64bitversionofcrashforvmcorecapturedfrom64bitkernel.oSimilarlyfors390,s390xandPPC.
Howtoinstall?
Installitusingyumorrpmcommand.
#yuminstallcrash
OR
#rpmivhcrash6.0.42.el6.x86_64.rpm
Compilefromupstreamsourcecode(notsupportedbyRedHatSupport):
#gitclonehttps://github.com/crashutility/crash.git#cdcrash/#make#makeinstall
b)Kernelsymbolfile(vmlinux)ofthecrashedkernel
Providedby"kerneldebuginfo"package.Eg:kerneldebuginfo2.6.32431.el6.x86_64.rpm
Theversionandarchofkerneldebuginfopackagemustmatchwiththeversionofkernelfromwhichvmcorewascaptured.
-
Howtoinstall?
Subscribeto"rhel6serverdebugrpms"channel.
#yumconfigmanagerenablerhel6serverdebugrpms#subscriptionmanagerreposenablerhel6serverdebugrpms
Installitusingyumordebuginfoinstallcommand:
#yuminstallkerneldebuginfo2.6.32431.el6#debuginfoinstallkerneldebuginfo2.6.32431.el6
OR
DownloadkerneldebuginfocommonandkerneldebuginforpmpackagesfromRedHatCustomerPortalandinstallitusingrpmcommand.
#rpmivhkerneldebuginfocommon2.6.32431.el6.x86_64.rpm#rpmivhkerneldebuginfo2.6.32431.el6.x86_64.rpm
Howtouse(vmlinux)withoutinstallingkerneldebuginfopackageonthesystem?
Extractthekerneldebuginforpmpackageusingrpm2cpiocommand.
#rpm2cpiokerneldebuginfo2.6.32431.el6.x86_64.rpm|cpioidv
c)Amachineofsamearchitectureasofkernelfromwhichvmcorewascaptured.
Akernelcrashdump(vmcore)capturedfromax86_64machinecanonlybeviewedonax86_64machine.
Similarly,akernelcrashdump(vmcore)capturedfromas390machinecanonlybeviewedonas390machine.
d)kernelcrashdumpfile(vmcore)
Capturedusingkdump/diskdump/netdump/xendump/LKCD/vmss2coremechanisms.
-
e)Sourcecodeofthecrashedkernel(optional)
Providedby"src"rpmpackageofkernel.Eg:kernel2.6.32431.el6.src.rpm
Theversionof"src"rpmpackageofkernelmustmatchwiththeversionofkernelfromwhichvmcorewascaptured.
Howtoinstall?
Installitusingyumorrpmcommand.
#yuminstallkernelsrc#rpmivhkernel2.6.32431.el6.src.rpm
OR
Extractthekernelsrcrpmpackageusingrpm2cpiocommand.
#rpm2cpiokernel2.6.32431.el6.src.rpm|cpioidv
-
Howtoopenakernelcrashdump(vmcore)foranalysis?
a)Typicalpostmortemdebugging:[Offlinemode]
Syntax:#crash/path/to/vmlinux/path/to/vmcore
Kernelobjectfileandmemoryimagearesupplied,respectively.
#crashosreleasevmcore2.6.32431.el6.x86_64
#crashd1vmcore|grepRELEASEOSRELEASE=2.6.32431.el6.x86_64
#crash/var/crash/vmcore/usr/lib/debug/lib/modules/2.6.32431.el6.x86_64/vmlinux
b)Livememorydebugging:[Onlinemode]
Syntax:#crash/path/to/vmlinux
/dev/crashusedbydefaultforlivememoryimage.
#crash/usr/lib/debug/lib/modules/2.6.32431.el6.x86_64/vmlinux
c)Livememorydebugging(withvmlinuxsearch):[Onlinemode]
Syntax:#crash
Predefineddirectoriesaresearchedforpropervmlinux
Versionstringmatchedtotherunningkernel(/proc/version)
-
Whatarethebasiccommandsofcrashutility?
osys:Displayessentialsysteminformation.
olog:Displaythekernelringbufferlog.
obt:Displayakernelstackbacktrace.
obtf:Displayallstackdatacontainedinaframe.
odis:Disassembleafunctionorsymbol.
okmemi:Displaysinformationabouttheuseofkernelmemory.
okmems:Displaysbasickmalloc()slabdata.
oipcs:SystemVIPCfacilities.
ops:Displaysprocessesinthesystem.
orunq:Displaysthetasksontherunqueuesofeachcpu.
omount:Displaysinformationaboutthemountedfilesystems.
ofiles:Displaytheopenfilesofthecurrentcontext.
odev:Displaycharacterandblockdevicedata.
odevd:DisplaydiskI/Ostatistics.
onet:Displaythesystem'snetworkdevicelist.
oirq:DisplayIRQdata.
omod:Displaythecurrentlyinstalledmodules.
omodt:Displaymodulesthatare"tainted".
oexit:Exitthecrashsession.
oextend:Dynamicallyloadsorunloadscrashextensionsharedobject.
ohelp:Gethelp.
-
InitialAnalysisofKernelCrashDump
-
Howtocheckbasicsysteminformation?
"sys"commanddisplaysessentialsysteminformation.
crash>sysKERNEL:/usr/lib/debug/lib/modules/2.6.32431.el6.x86_64/vmlinuxDUMPFILE:/var/crash/127.0.0.12016012117:45:34/vmcoreCPUS:4DATE:ThuJan2117:45:302016UPTIME:00:02:01LOADAVERAGE:0.49,0.30,0.11TASKS:264NODENAME:hvyas.example.comRELEASE:2.6.32431.el6.x86_64VERSION:#1SMPSunNov1022:19:54EST2013MACHINE:x86_64(2790Mhz)MEMORY:5.8GBPANIC:"SysRq:Triggeracrash"
Howtocheckhardwareinformation?
Theoption(i)of"sys"commanddumpstheDesktopManagementInterface(DMI)stringdataifavailableinthekernel.
crash>sysiDMI_BIOS_VENDOR:SeabiosDMI_BIOS_VERSION:0.5.1DMI_BIOS_DATE:01/01/2007DMI_SYS_VENDOR:RedHatDMI_PRODUCT_NAME:KVMDMI_PRODUCT_VERSION:RHEL6.5.0PCDMI_PRODUCT_SERIAL:DMI_PRODUCT_UUID:3D51C070C307BD3A281DCA1A689C22C6DMI_CHASSIS_VENDOR:RedHatDMI_CHASSIS_TYPE:1DMI_CHASSIS_VERSION:DMI_CHASSIS_SERIAL:DMI_CHASSIS_ASSET_TAG:DMI_SMBIOS_VERSION:2.4
OR
crash>log|grepDMI:DMI:RedHatKVM,BIOS0.5.101/01/2007
-
Howtocheckkernelringbuffer(dmesg)?
"log"or"dmesg"commanddumpsthekernellog_bufcontents.
crash>log|tailn38SysRq:TriggeracrashBUG:unabletohandlekernelNULLpointerdereferenceat(null)IP:[]sysrq_handle_crash+0x16/0x20PGD140aff067PUD140a9f067PMD0Oops:0002[#1]SMPlastsysfsfile:/sys/devices/pci0000:00/0000:00:01.2/usb1/11/speedCPU2Moduleslinkedin:nfsdlockdnfs_aclauth_rpcgssexportfsautofs4sunrpcbnx2fccnicuiofcoe8021qlibfcoegarpstplibfcllcscsi_transport_fcscsi_tgtxt_NFQUEUEiptable_filterip_tablesip6t_REJECTnf_conntrack_ipv6nf_defrag_ipv6xt_statenf_conntrackip6table_filterip6_tablesipv6uinputmicrocodevirtio_balloonvirtio_netsnd_hda_intelsnd_hda_codecsnd_hwdepsnd_seqsnd_seq_devicesnd_pcmsnd_timersndsoundcoresnd_page_alloci2c_piix4i2c_coreext4jbd2mbcachevirtio_blkpata_acpiata_genericata_piixvirtio_pcivirtio_ringvirtiodm_mirrordm_region_hashdm_logdm_mod[lastunloaded:speedstep_lib]
Pid:2187,comm:bashNottainted2.6.32431.el6.x86_64#1RedHatKVMRIP:0010:[][]sysrq_handle_crash+0x16/0x20RSP:0018:ffff8801422bde18EFLAGS:00010096RAX:0000000000000010RBX:0000000000000063RCX:0000000000000000RDX:0000000000000000RSI:0000000000000000RDI:0000000000000063RBP:ffff8801422bde18R08:0000000000000000R09:203a207152737953R10:0000000000000000R11:0000000000000000R12:0000000000000000R13:ffffffff81b01a40R14:0000000000000286R15:0000000000000007FS:00007ffdf778f700(0000)GS:ffff880028280000(0000)knlGS:0000000000000000CS:0010DS:0000ES:0000CR0:0000000080050033CR2:0000000000000000CR3:0000000140815000CR4:00000000000006e0DR0:0000000000000000DR1:0000000000000000DR2:0000000000000000DR3:0000000000000000DR6:00000000ffff0ff0DR7:0000000000000400Processbash(pid:2187,threadinfoffff8801422bc000,taskffff88018d4e8ae0)Stack:ffff8801422bde68ffffffff8134b982ffff88018d4e8ae0ffff88010000000000000000000003000000000000000002ffff880140ace3c000007ffdf77990000000000000000002fffffffffffffffbffff8801422bde98ffffffff8134ba3eCallTrace:[]__handle_sysrq+0x132/0x1a0[]write_sysrq_trigger+0x4e/0x50[]proc_reg_write+0x7e/0xc0[]vfs_write+0xb8/0x1a0[]sys_write+0x51/0x90[]system_call_fastpath+0x16/0x1bCode:d08881a31cfe81c9c36666662e0f1f840000000000554889e50f1f440000c7050d077500010000000faef804250000000001c9c3554889e50f1f4400008d47RIP[]sysrq_handle_crash+0x16/0x20RSPCR2:0000000000000000
-
Howtodeterminethepanictask?
Theoption(p)of"set"commandsetsthecontexttothepanictask,orbacktothecrashtaskonalivesystem.
crash>setpPID:2187COMMAND:"bash"TASK:ffff88018d4e8ae0[THREAD_INFO:ffff8801422bc000]CPU:2STATE:TASK_RUNNING(SYSRQ)
Howtocheckthebacktraceofpanictask?
"bt"commanddisplaysakernelstackbacktrace.Ifnoargumentsaregiven,thestacktraceofthecurrentcontextwillbedisplayed.
crash>btPID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"#0[ffff8801422bd9e0]machine_kexecatffffffff81038f3b#1[ffff8801422bda40]crash_kexecatffffffff810c5d92#2[ffff8801422bdb10]oops_endatffffffff8152b510#3[ffff8801422bdb40]no_contextatffffffff8104a00b#4[ffff8801422bdb90]__bad_area_nosemaphoreatffffffff8104a295#5[ffff8801422bdbe0]bad_areaatffffffff8104a3be#6[ffff8801422bdc10]__do_page_faultatffffffff8104ab6f#7[ffff8801422bdd30]do_page_faultatffffffff8152d45e#8[ffff8801422bdd60]page_faultatffffffff8152a815[exceptionRIP:sysrq_handle_crash+22]RIP:ffffffff8134b6c6RSP:ffff8801422bde18RFLAGS:00010096RAX:0000000000000010RBX:0000000000000063RCX:0000000000000000RDX:0000000000000000RSI:0000000000000000RDI:0000000000000063RBP:ffff8801422bde18R8:0000000000000000R9:203a207152737953R10:0000000000000000R11:0000000000000000R12:0000000000000000R13:ffffffff81b01a40R14:0000000000000286R15:0000000000000007ORIG_RAX:ffffffffffffffffCS:0010SS:0018#9[ffff8801422bde20]__handle_sysrqatffffffff8134b982#10[ffff8801422bde70]write_sysrq_triggeratffffffff8134ba3e#11[ffff8801422bdea0]proc_reg_writeatffffffff811f328e#12[ffff8801422bdef0]vfs_writeatffffffff81188f78#13[ffff8801422bdf30]sys_writeatffffffff81189871#14[ffff8801422bdf80]system_call_fastpathatffffffff8100b072RIP:0000003e2c2db560RSP:00007fff425f6548RFLAGS:00010202RAX:0000000000000001RBX:ffffffff8100b072RCX:00000000004b5e34RDX:0000000000000002RSI:00007ffdf7799000RDI:0000000000000001RBP:00007ffdf7799000R8:000000000000000aR9:00007ffdf778f700R10:0000000000000000R11:0000000000000246R12:0000000000000002R13:0000003e2c58e7a0R14:0000000000000002R15:0000003e2c58e7a0
-
ORIG_RAX:0000000000000001CS:0033SS:002b
Howtodisplayparentalhierarchyofaprocess?
The(p)optionof"ps"commanddisplaystheparentalhierarchyofselected,orall,tasks.
crash>psp2187PID:0TASK:ffffffff81a8d020CPU:0COMMAND:"swapper"PID:1TASK:ffff880192d39500CPU:2COMMAND:"init"PID:1865TASK:ffff88014cf00040CPU:1COMMAND:"sshd"PID:1877TASK:ffff88018d606040CPU:2COMMAND:"sshd"PID:2161TASK:ffff8801420beaa0CPU:1COMMAND:"bash"PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"
Howtodisplaythechildtaskofaprocess?
The(c)optionof"ps"commanddisplaysthechildrenofselected,orall,tasks.
crash>psc2187PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"(nochildren)
crash>psc2161PID:2161TASK:ffff8801420beaa0CPU:1COMMAND:"bash"PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"
Howtochecklistofopenfilesbypanictask?
"files"commanddisplaystheopenfilesofthecurrentcontext.
crash>files2187PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"ROOT:/CWD:/rootFDFILEDENTRYINODETYPEPATH0ffff880140ac6d40ffff880141057500ffff880141037d58CHR/dev/pts/01ffff880140ace3c0ffff8801410738c0ffff880141043078REG/proc/sysrqtrigger2ffff880140ac6d40ffff880141057500ffff880141037d58CHR/dev/pts/010ffff880140ac6d40ffff880141057500ffff880141037d58CHR/dev/pts/0254ffff88018cc71500ffff8801434ef240ffff88014104c4b8REG/root/system_crash.sh255ffff880140ac6d40ffff880141057500ffff880141037d58CHR/dev/pts/0
-
Howtochecktaskpriorityandpolicy?
"task"commandcanusedtodeterminethetaskpriorityandpolicy.
"task"commanddisplaysthecontentsofatask'stask_structandthread_infostructures.
crash>setpPID:2187COMMAND:"bash"TASK:ffff88018d4e8ae0[THREAD_INFO:ffff8801422bc000]CPU:2STATE:TASK_RUNNING(SYSRQ)
crash>task2187Rpolicy,prio,rt_priorityPID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"policy=0,prio=120,rt_priority=0,
Schedulingpolicies:#defineSCHED_NORMAL0#defineSCHED_FIFO1#defineSCHED_RR2#defineSCHED_BATCH3#defineSCHED_IDLE5
OR
crash>setpPID:2187COMMAND:"bash"TASK:ffff88018d4e8ae0[THREAD_INFO:ffff8801422bc000]CPU:2STATE:TASK_RUNNING(SYSRQ)
crash>task_struct.policy,prio,rt_priority0xffff88018d4e8ae0policy=0prio=120rt_priority=0
-
Howtocheckthecommandlineargumentsandenvironmentstringsoftask?
Theoption(a)of"ps"commanddisplaystheargumentandenvironmentdataforthetask.
crash>psaautomountPID:3948TASK:f722ee30CPU:0COMMAND:"automount"ARG:/usr/sbin/automounttimeout=60/netprogram/etc/auto.netENV:SELINUX_INIT=YESCONSOLE=/dev/consoleTERM=linuxINIT_VERSION=sysvinit2.85PATH=/sbin:/usr/sbin:/bin:/usr/binLC_MESSAGES=en_USRUNLEVEL=3runlevel=3PWD=/LANG=ja_JP.UTF8PREVLEVEL=Nprevious=NHOME=/SHLVL=2_=/usr/sbin/automount
Note:Thisisinformationisonlyavailableifuserspacecontentsarenotfilteredfromkernelcrashdump.
-
Howtodetermineresourcelimits(rlimits)ofaprocess?
Theoption(r)of"ps"commanddisplaysresourcelimits(rlimits)ofselected,orall,tasks.
crash>set1PID:1COMMAND:"init"TASK:ffff880192d39500[THREAD_INFO:ffff880192d3a000]CPU:2STATE:TASK_INTERRUPTIBLE
crash>psr1PID:1TASK:ffff880192d39500CPU:2COMMAND:"init"RLIMITCURRENTMAXIMUMCPU(unlimited)(unlimited)FSIZE(unlimited)(unlimited)DATA(unlimited)(unlimited)STACK10485760(unlimited)CORE0(unlimited)RSS(unlimited)(unlimited)NPROC4533145331NOFILE10244096MEMLOCK6553665536AS(unlimited)(unlimited)LOCKS(unlimited)(unlimited)SIGPENDING4533145331MSGQUEUE819200819200NICE00RTPRIO00RTTIME(unlimited)(unlimited)
-
Howtodeterminetotalnumberoftasksindifferentstate?
Theoption(S)of"ps"commanddisplaysasummaryconsistingofthenumberoftasksinataskstate.
crash>psSRU:5IN:259
Howtodisplayonlyuserspaceprocess?
Theoption(u)of"ps"commanddisplaysonlyusertasks.
crash>psu|headPIDPPIDCPUTASKST%MEMVSZRSSCOMM102ffff880192d39500IN0.0193641500init48311ffff88018d76a040IN0.0112401320udevd133613ffff88014cf00aa0IN0.027640864auditd133710ffff88018e46f540IN0.027640864auditd136110ffff88018cde6080IN0.02490921628rsyslogd136213ffff880191be5540IN0.02490921628rs:mainQ:Reg136312ffff88018d49c080IN0.02490921628rsyslogd136413ffff88018e796aa0IN0.02490921628rsyslogd139612ffff88018d4d4080IN0.018976924rpcbind
Howtodisplayonlykernelthreads?
Theoption(k)of"ps"commanddisplaysonlykernelthreads.
crash>psk|headPIDPPIDCPUTASKST%MEMVSZRSSCOMM>000ffffffff81a8d020RU0.000[swapper]>001ffff880192d81540RU0.000[swapper]002ffff880192d8a040RU0.000[swapper]>003ffff880192dc2aa0RU0.000[swapper]201ffff880192d38aa0IN0.000[kthreadd]320ffff880192d38040IN0.000[migration/0]420ffff880192d67540IN0.000[ksoftirqd/0]520ffff880192d66ae0IN0.000[migration/0]620ffff880192d66080IN0.000[watchdog/0]
-
Howtocheckthetotaltimeofaprocessinaspecificstate?
Theoption(m)of"ps"commanddisplaysthetimestampintodays,hours,minutes,seconds,andmillisecondssincethetaskwaslastrunonacpu.
crash>psm|grepRU[000:00:00.000][RU]PID:2187TASK:ffff88018d4e8ae0CPU:2COMMAND:"bash"[000:02:01.177][RU]PID:0TASK:ffffffff81a8d020CPU:0COMMAND:"swapper"[000:02:01.409][RU]PID:0TASK:ffff880192d81540CPU:1COMMAND:"swapper"[000:02:01.408][RU]PID:0TASK:ffff880192d8a040CPU:2COMMAND:"swapper"[000:02:01.408][RU]PID:0TASK:ffff880192dc2aa0CPU:3COMMAND:"swapper"
HowtocheckrunqueueofeachCPU?
"runq"commanddisplaysthetasksonaCFSrunqueue:
crash>runqCPU0RUNQUEUE:ffff880028216840CURRENT:PID:0TASK:ffffffff81a8d020COMMAND:"swapper"RTPRIO_ARRAY:ffff8800282169c8[notasksqueued]CFSRB_ROOT:ffff8800282168d8[notasksqueued]
CPU1RUNQUEUE:ffff880028256840CURRENT:PID:0TASK:ffff880192d81540COMMAND:"swapper"RTPRIO_ARRAY:ffff8800282569c8[notasksqueued]CFSRB_ROOT:ffff8800282568d8[notasksqueued]
CPU2RUNQUEUE:ffff880028296840CURRENT:PID:2187TASK:ffff88018d4e8ae0COMMAND:"bash"RTPRIO_ARRAY:ffff8800282969c8[notasksqueued]CFSRB_ROOT:ffff8800282968d8[notasksqueued]
CPU3RUNQUEUE:ffff8800282d6840CURRENT:PID:0TASK:ffff880192dc2aa0COMMAND:"swapper"RTPRIO_ARRAY:ffff8800282d69c8[notasksqueued]CFSRB_ROOT:ffff8800282d68d8[notasksqueued]
-
Howtodeterminethevaluesofsysctlparameterfromvmcore?
Checkthe".data"fieldofdesiredsysctlparameterinkernelsocurcesysctl.cfile.
SourceFile:kernel2.6.32431.el6/kernel/sysctl.c
{.ctl_name=VM_SWAPPINESS,.procname="swappiness",.data=&vm_swappiness,.maxlen=sizeof(vm_swappiness),.mode=0644,.proc_handler=&proc_dointvec_minmax,.strategy=&sysctl_intvec,.extra1=&zero,.extra2=&one_hundred,},
crash>vm_swappinessvm_swappiness=$1=60
SourceFile:kernel2.6.32431.el6/kernel/sysctl.c
{.ctl_name=VM_PANIC_ON_OOM,.procname="panic_on_oom",.data=&sysctl_panic_on_oom,.maxlen=sizeof(sysctl_panic_on_oom),.mode=0644,.proc_handler=&proc_dointvec,},
crash>sysctl_panic_on_oomsysctl_panic_on_oom=$2=1
-
InitialAnalysisofMemorySubsystem
-
Howtocheckoverallmemoryusageonthesystem?
"kmemi"commanddisplaysgeneralmemoryusageinformation:
crash>kmemiPAGESTOTALPERCENTAGETOTALMEM12058384.6GBFREE11025064.2GB91%ofTOTALMEMUSED103332403.6MB8%ofTOTALMEMSHARED1448456.6MB1%ofTOTALMEMBUFFERS637724.9MB0%ofTOTALMEMCACHED39960156.1MB3%ofTOTALMEMSLAB1772569.2MB1%ofTOTALMEM
TOTALSWAP1638264MBSWAPUSED000%ofTOTALSWAPSWAPFREE1638264MB100%ofTOTALSWAP
COMMITLIMIT6167412.4GBCOMMITTED59755233.4MB9%ofTOTALLIMIT
Howtocheckperzonememorystatistics?
"kmemz"commanddisplaysperzonememorystatistics.
crash>kmemzNODE:0ZONE:0ADDR:ffff880000010000NAME:"DMA"SIZE:4095PRESENT:3831MIN/LOW/HIGH:42/52/63VM_STAT:NR_FREE_PAGES:3930NR_INACTIVE_ANON:0NR_ACTIVE_ANON:0NR_INACTIVE_FILE:0NR_ACTIVE_FILE:0NR_UNEVICTABLE:0NR_MLOCK:0NR_ANON_PAGES:0NR_FILE_MAPPED:0NR_FILE_PAGES:0NR_FILE_DIRTY:0NR_WRITEBACK:0NR_SLAB_RECLAIMABLE:0NR_SLAB_UNRECLAIMABLE:0NR_PAGETABLE:0NR_KERNEL_STACK:0NR_UNSTABLE_NFS:0
-
Howtodeterminememoryusageinuserspace?
The"psGu"commandcanbeusedtodeterminetheRSSofuserspacetasks.
crash>psGu|sed's/>//g'|awk'{total+=$8}END{printf"TotalRSSofusermode:%.02fGiB\n",total/2^20}'TotalRSSofusermode:30.74GiB
Perprocessmemoryusage:
crash>psG|sortk8,8nr|headPIDPPIDCPUTASKST%MEMVSZRSSCOMM>28882201091ffff880831e0e040RU92.86527242832120772rsession>698569842ffff88082e934040RU0.06416644680coda>639213ffff88082e934ab0RU0.01673366324perfd612312ffff88082c194ab0IN0.123848045732splunkd694669451ffff88082f804ab0RU0.08337449188opcmona2738710ffff88082d288040RU0.01299526472bgsagent627510ffff88082f7eaab0UN0.0362845184scopeux209820973ffff88082dbe7520IN0.014543285172python698169801ffff880831017520RU0.09035483236opcmsga215113ffff88082daed520IN0.07063202748EracentEUAServi
Specificprocess'smemoryusage:
crash>psGursession|tailn+2|cutb2|gawk'{mem+=$8}END{print"Total"mem/1048576"GB"}'Total30.6334GB
-
Howtodisplayinformationforeachconfiguredswapdevice?
"swap"commanddisplaysinformationforeachconfiguredswapdevice.
crash>swapSWAP_INFO_STRUCTTYPESIZEUSEDPCTPRIFILENAMEffff88014be681c0FILE65528k0k0%1/swapfile
Howtodeterminememoryusageinkernelspace?
"kmems"commanddisplaysbasickmalloc()slabdata.
crash>kmems|grepeNAMEeanon_vma_chainCACHENAMEOBJSIZEALLOCATEDTOTALSLABSSSIZEffff88043bf20b00anon_vma_chain4830204272030204281839226344k
Formula:(OBJSIZE*ALLOCATED)/2^30
crash>!bcqscale=2(48*302042720)/2^3013.50
Memoryallocatedto"anon_vma_chain"slabobjectis~13.50GiB.
-
Howtocheckmemoryallocatedforhugepages?
RHEL6:
"kmemh"commanddisplaystheaddressofhugepagehstatearrayentries,alongwiththeirhugepagesize,totalandfreecounts,andname.
crash>kmemhHSTATESIZEFREETOTALNAMEffffffff81fbb8c02MB1010hugepages2048kB
RHEL5:
a)TotalNumberofHugepages.
crash>pdnr_huge_pagesnr_huge_pages=$1=35845
crash>pdnr_huge_pages*(1pdfree_huge_pagesfree_huge_pages=$2=0
-
HowtocheckmemoryallocatedtoVMwareballooningdriver(RHEL6+)?
Determinetheaddressofsymbolballoon.crash>symballoonffffffffa002b600(b)balloon[vmware_balloon]
Determinethevalueof"size"variableusingtheaddressofsymbolballoon.crash>structvmballoon.size0xffffffffa002b600$4=2721049
crash>!bcqscale=22721049*4/2^2010.37
2721049pages=2721049x4=10884196KiB=10.37GiB
Note:ThevalueofsizevariableistheamountofmemoryallocatedbyVMwareBallooningdriverinpages.
Determinethevalueof"target"variableusingtheaddressofsymbolballoon.crash>vmballoon.target0xffffffffa002b600$5=3177311
crash>!bcqscale=23177311*4/2^2012.12
3177311pages=3177311x4=12709244KiB=12.12GiB
Note:ThevalueoftargetvariableistheamountofmemoryneededbyVMwareBallooningdriverinpages.
-
Questions?
top related