prma - introduction
TRANSCRIPT
Graduation Project Plan
Cai Bowen
3/10/2014
PRMA
Program Runtime Monitoring and Analyzing
github.com/xkommando/PRMA
BackgroundDifferent logs generate by programs
log
Access logServer log
System log
…
Internal event log
Server log
System log
…
127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
Data Mining
logger.error("update dese user read_index failed, user={}", Jsons.toJson(user))
Debugging
Performance tuning
Basic Principle
Monitor program exception, failure and performance,
analyzing bottleneck, robustness, and reliability,
based on internal event logging.
Event Monitor DBcompress
analyze
BackgroundHow application logs are stored today
Running Program debug.log
warning.log
all-error.log
Production Server
Stepping Stone
Log entry stored as plain text file.
• Logs are printed to a file upon file system, possible IO failure or write lock.
• Log content are highly redundant yet uncompressed, file size often reach to several GB.
• Logs by different replications are stored on different VMs, hard to process.
• Logs are unindexed, hard to query (i.e., by “tail”, “grep”, “wc”) or analyze.
• Log files are managed manually or by some third party tool, e.g., Time Tunnel.
• Real-time log monitoring is done manually.
• Ugly interface, e.g., command line prompt.
Backgroundproblems of current fashion:
• Logs are printed to a file upon file system, possible IO failure or write lock.
• Log content are highly redundant yet uncompressed, file size often reach to several GB.
• Logs by different replications are stored on different VMs, hard to process.
• Logs are unindexed, hard to query (i.e., by “tail”, “grep”, “wc”) or analyze.
• Log files are managed manually or by some third party tool, e.g., Time Tunnel.
• Real-time log monitoring is done manually.
• Ugly interface, e.g., command line prompt.
Backgroundproblems of current fashion:
• No writable file system, or only one temporary file is writable, in PaaS.
• Demo process or long-time background thread not allowed.
• Security restrictions, e.g., customer “java.lang.SercurityManager”.
• JVM restrictions, e.g., no native interface, no JMX.
Backgroundpossible problem in the future:
Logging and storage:
the most popular logging tool is slf4j + logback, store logs to file.
Log storage: Logback built-in function to store logs to RDBMS,
Log4j to MongoDB tool: log4Mongo,
Logback to MongoDB tool: Logback-contrib-mongo
Monitor: A general purpose monitoring tool: www.nagios.org
Analyzing: No specific tool to analyzing logs for debugging and profiling.
Current State
ObjectivesPRMA Architecture
DB
API
External Systems
monitorStore
Analyzing
Web interface
ObjectivesWork mode:
Log Server App Server
Stepping Stone
Client – Server Mode Embedded Mode
Running Program Stepping
Stone
PRMA
• Non-blocking, highly reliable storage engine with built-in compression functionality.
• Log file management, e.g., removal, transfer.
• Advanced query.
• Real-time monitoring and making alert.
• Built-in analyzing algorithms as well as rich API.
• Embedded within app or working client-server mode.
• Easy configuration and optional web based user interface.
Features:
• Software architecture.
• Analyzing on semi-structured data.
• Other facilities: net interface, monitor and making alerts.
• Insufficient man-month.
Difficulties
1, divide project into several phases, and focus on core functionalities on first phase, i.e., monitoring and analyzing algorithm.
2, Allocate more time on the project, and plan carefully.
3, using existing technologies:
Logback, mongoDB or RDBMS
4, Previous knowledge and experience.
Method
Phase 1:
Goal: Compact / compressed and more structured log store on MongoDB
Steps:
1, getting familiar with mongoDB.
2, analyzing log event, find appropriate schemas.
3, adopting Logback to mongoDB and test.
Phase 2:
Goal: the monitor API and one or two referential monitor implementation (HTTP callback and email).
Steps:
4, analyzing log monitor requirement, design monitoring mechanism.
5, log monitor experimental implementation and testing.
6, integrate monitor into PRMA.
Brief Plan:
Phase 3:
Goal: analyzing on logs, aggregation, filtering API.
7, search for log analyzing algorithms, compare, test them and find suitable ones.
8, experimental implementation of analyzing algorithm and testing.
9, Integration and refactoring.
Phase 4:Goal: refactoring and documents.
10, logging API for other languages.
11, refactoring and testing.
11, documentation and project presentation, release version 1.0
Brief Plan:
Phase 5:
Goal: other facilities.
12, log file management.
14, fix bugs, benchmark and performance turning, release version 1.2.
13, web interface.
15, more improvements and bug fix, release version 2.0.
Brief Plan:
Project URL: https://github.com/xkommando/PRMA
Active Brach: https://github.com/xkommando/PRMA/tree/p2
Supporting Project: https://github.com/xkommando/Gplume
My Weekly Reports: https://github.com/xkommando/PRMA/tree/p2/doc/gp_weekly_report
Appendix