![Page 1: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/1.jpg)
JBoss Troubleshooting Essential장애 처리와 튜닝 프로세스(Thread Dump 분석)
![Page 2: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/2.jpg)
![Page 3: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/3.jpg)
시스템 구축 후 - 고민들
https://www.youtube.com/watch?v=jYjGoNEV-_E&index=12&list=PLaFPOkYzLL-_QWKsbT40cxwr_-cijU6L
![Page 4: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/4.jpg)
고객이 체험하는 것은?
https://www.youtube.com/watch?v=jMUpM-1hx7Q&index=6&list=PLaFPOkYzLL-_QWKsbT40cxwr_-cijeU6L
![Page 5: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/5.jpg)
엔터프라이즈 모니터링에서는 문제 없음
https://www.youtube.com/watch?v=eE7rPuksd1c&index=7&list=PLaFPOkYzLL-_QWKsbT40cxwr_-cijeU6L
![Page 6: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/6.jpg)
각 분야에서는 전문가 – 코끼리 만지기
https://www.youtube.com/watch?v=P5S0gI-4hZI&list=PLaFPOkYzLL-_QWKsbT40cxwr_-cijeU6L&index=10
![Page 7: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/7.jpg)
![Page 8: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/8.jpg)
Troubleshooting 공통 프로세스
![Page 9: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/9.jpg)
Incident & Problem Management
• 장애복구시간을 단축할 수 있는 도구는 ?
![Page 10: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/10.jpg)
증상과 원인 - 장애 내용 파악
![Page 11: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/11.jpg)
증상과 원인 - 현황 파악
![Page 12: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/12.jpg)
증상과 원인 - 수집 자료는?
![Page 13: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/13.jpg)
진단 및 모니터링 - 정보수집
![Page 14: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/14.jpg)
조치
![Page 15: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/15.jpg)
조치 - Patch
![Page 16: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/16.jpg)
방지 대책
![Page 17: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/17.jpg)
트러블 슈팅의 기본
![Page 18: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/18.jpg)
![Page 19: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/19.jpg)
Server Hang
![Page 20: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/20.jpg)
Server Hang이란?
![Page 21: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/21.jpg)
Server Hang의 원인
![Page 22: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/22.jpg)
Thread 상태
![Page 23: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/23.jpg)
Java Thread Dump의 이해
![Page 24: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/24.jpg)
스레드 덤프의 정보 구성
![Page 25: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/25.jpg)
Thread Dump의 Blocked의 의미
![Page 26: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/26.jpg)
Java Thread Dump의 BLOCKED의 의미
![Page 27: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/27.jpg)
Thread BLOCKED 상태가 길어지면 발생하는 상황
![Page 28: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/28.jpg)
![Page 29: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/29.jpg)
스레드 덤프
• 스레드 덤프란?
• JVM 내부에서 현재 수행중인 스레드의 SnapShot을 텍스트로 출력
• 서버 Hang, 서버 Slowdown등 서버에서 발생하는 문제를 해결하는 가장 유용한도구
• 약 3~5초간격으로 3~5회 연속해서 스레드 덤프를 받아야 분석이 가능하다
• 언제 받아야 하나?
• 운영 중인 서버에서도 수행가능(많은 시간이 걸리지 않는 작업임)
• 서버 Hang, Slowdown시, 서버가 Crash되거나 OutOfMemory가 발생한 직후에 받는 것이 좋음
Thread Dump #1
Thread Dump #3
. .
TimeThread Dump #2
.
![Page 30: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/30.jpg)
Take at least 3 times, with 3 to 5 seconds interval
스레드 덤프를 분석
![Page 31: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/31.jpg)
스레드 덤프 받는 방법은?• Linux / UNIX 시스템에서 스레드 덤프
• kill -3 <PID>
• 일반적을 nohup 명령을 이용하여 시작하기 때문에
nohup.out파일에 stdout 로그가 남게 됨
• 윈도우에서는 Ctrl + Break
• KHAN [apm] 화면에서 버튼 클릭(장애시 자동 생성)
• Docker와 Openshift 같은 환경
• 스레드 덤프을 어떻게 받나?
• docker exec -it c4343003c1 /bin/bash~# kill -3 <PID>
• 내용을 어떻게 복사? nohup 로그는 남기나?
• docker cp c4343003c1:/file/path/nohup.out /host/path/target
• KHAN [apm]은 Openshift, Docker를 지원 버튼 클릭 & 분석 UI 제공
![Page 32: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/32.jpg)
Server가 느려 지는 상황에서 Dashboard
![Page 33: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/33.jpg)
KHAN [apm] 스레드 덤프 분석 예시
Pending Reqeust가 Apache Commons Http
Client Pool에서 Connection을 가져오려고
기다리고 있음
스레드에서 어떤 URL이 얼마동안,
CPU 자원을 사용하며 실행중인지
표시함
![Page 34: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/34.jpg)
KHAN [apm] 스레드 덤프 분석 예시
Log4j Console Appender가 잡은 Lock이
풀리기를 기다리는 스레드가 대부분
default task-238 스레드에서 기다리는 Lock
default task-196 스레드에서 잡은 Lock
Lock의 Owner인 Log4j
Console Appender
스레드 덤프의 Lock 정보를 Hyper Link로
클릭하여 Root Cause 분석
![Page 35: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/35.jpg)
란
![Page 36: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/36.jpg)
란
쓰레드덤프와모니터링화면전달
고객사
오픈나루
시스템이슈발생
오픈나루서비스데스크접수
장애분석
장애분석보고서전달
![Page 37: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/37.jpg)
문제분석
문제해결
오픈나루
Knowledge
Base
문제발생문제보고
QuickService
ON Desk
자동 접수
기술지원엔지니어
고객사
![Page 38: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/38.jpg)
KHAN [apm] = Ah ! Provisioning + Monitoring
![Page 39: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/39.jpg)
“살아 남는 종(種)은 강한 종이 아니고,
또 우수한 종도 아니다.
변화에 적응하는 종이다. ”
- Charles Darwin, 1809
![Page 40: 장애처리와튜닝프로세스(Thread Dump 분석 - Opennaru, Inc.opennaru.com/wp-content/uploads/2018/02/KHAN_Thread_Dump... · 2018-02-27 · KHAN [apm] 스레드덤프분석예시](https://reader033.vdocuments.mx/reader033/viewer/2022060418/5f15e71393c9b830ad4e49f1/html5/thumbnails/40.jpg)
감사합니다.