developing highly instrumented applications with minimal effort
DESCRIPTION
Presentation from Silicon Valley Code Camp 2013. Related code on github: * https://github.com/hoserdude/mvcmusicstore-instrumented * https://github.com/hoserdude/spring-petclinic-instrumented * https://github.com/hoserdude/nodecellar-instrumentedTRANSCRIPT
Tim HobsonPrincipal Engineer, Chief Caffeine Officer - Intuit
Developing highly instrumented applications with minimal effort
2
About Me
Work
Play
3
Agenda
Theory• Patterns• Supporting
Frameworks• Supporting
Components
Practice• .Net Example • Java Example• Node.js Example
Concepts• Quality Data• Application
Instrumentation
• Application Logging
GOAL: You should be able to apply this to your application today
Concepts
5
If You Do Nothing…
OR
6
There’s No Free Lunch
Garbage In, Garbage OutWe can
parse
anything!
Insight!
7
Quality In, Insight Out
Most apps start with only the framework or app server logging (or nothing!)
None of the above is interesting to the business or the developer.
You can’t get if you don’t give – there are many ways to give, and many classes of data to provide.
8
Classes of System Output
App Instrumentation
• Cross-cutting (free)• App Activity• Passively triggers
alerts• Source of
performance data• Source of usage data
App Logging• Intentional• Business
Transactions• Overtly triggers
alerts• Source of business
metrics• Aids in
troubleshooting failures, bugs
System Instrumentation• JMX/WMI/SNMP monitoring• Apache/IIS/nginx access logs
Theory and Best Practices
10
Best Practices (© splunk>)
Create human readable events
Clearly timestamp events
Use key-value pairs
Be aware of multi-value fields
Log unique identifiers
11
Best Practices (© Tim)
Global timestamps (UTC – 2013-08-21 22:43:31,990)
Context setting (who/what/where/when/how)
Categories/taxonomy (what tier, what component)
Timing (time everything!)
Security (never log sensitive data: password=***)
Consistency in naming – (action=purchase; sale=oct13;
productId=123123)– (action=buy; promo=oct13; sku=123123)
12
Example Output
2013-08-21 22:55:36,504; LogLevel=INFO; sid=q3prv41kt511vzojytnx1d42; rid=6500583; userLogin=(null); ipAddress=0.0.0.0; thread=249; category=Web.Controllers.BaseWebController; msg=RequestInfo; server=ws001prod; url=https://myapp.com/account/logon; method=GET; languages=en-US,en;q=0.8; referrer=https://myapp.com/members/dashboard; userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36; userId=123456; controller=Account; action=Logon;
2013-08-21 22:55:36,519; LogLevel=INFO; sid=q3prv41kt511vzojytnx1d42; rid=6500583; userLogin=(null); ipAddress=0.0.0.0; thread=249; category=TraceInterceptor; timeTracing=Service.Users.GetUser, time=3;
Context
Metrics
13
Patterns
Pipeline– Filter– Interceptor
Dependency injection– Proxy
Pointcuts/aspects
14
HTTP Pipelines
Java (from Oracle Docs)
ASP.Net (from MSFT Docs)
express.js
var app = express();app.use(func1);app.use(func2);app.use(func3);app.use(func4);app.use(func5);app.use(func6);app.use(func7);app.use(func8);app.listen(8080);
15
Dependency Injection
• Enforces interface-based programming
• Forbids circular references• Lends itself to testability• Flexibility in changing
implementations• For our purposes: cross-cutting
capabilities
Intercepting Calls With Dynamic Proxies
16
Implementation
Caller
DI Container
Dependency A
Dependency B
Proxy
Inte
rcep
tor
Inte
rcep
tor
Inte
rcep
tor
Proxy
Inte
rcep
tor
Inte
rcep
tor
Inte
rcep
tor
Proxy
Inte
rcep
tor
Inte
rcep
tor
Inte
rcep
tor
Supporting Frameworks
17
The Inner Sanctum• Unity Dynamic
Proxies• Spring
@AspectJ Pointcuts
• JavaScript Mixins
Annotations & Attributes• @Instrumenta
ble• [Instrumentabl
e]
The Front Door• ASP.Net
Modules/ActionFilters
• Java Servlet Filters/Spring Interceptors
• Node.js interceptors
Supporting Components
18
Java• LogBack• SLF4J• Log4J
node.js• SenchaLab
s Connect• Winston
.Net• Log4Net• NLog
Intervention!
20
Sample App Intervention
3 Sample Apps
3 Interventions3 Platforms
+ + +
21
Goals
Minimize impact on existing code
Apply best practices
Leverage aspects for interception
Leverage an HTTP pipeline for context
22
MVC Music Store Intervention (Demo)
Missing dependency injection/interception framework: UnityMissing logging library: Log4Net, Buche
1. Add DI framework (Unity)2. Configure aspects (Interception)3. Configure controller factory, container locator 4. Set up logger (Log4Net)5. Configure log pattern and targets (Log4Net.xml)6. Wire up logging interceptor7. Wire up LogActionFilter and BaseLoggingController8. Deploy!
23
Spring Pet Clinic Intervention (Demo)
1. Set up loggera. Configure log pattern (to support context data)b. Configure targets (rotating file)
2. Create LogAspect.java and @Instrumentable attributea. Capture calling contextb. Start/stop timing
3. Create LogInterceptor.javaa. Capture request metadatab. Set MDCc. Start/stop timing
4. Configure aspect bean
5. Configure interceptor bean
6. Annotate the methods we care about
24
NodeCellar Intervention (Demo)
Missing Interception Framework: ScarletMissing Logging Library: Winston
1. Set up Scarlet 2. Configure log pattern3. Configure method interception4. Create logging interceptor5. Create LoggingFilter6. Deploy!
Closing Thoughts
Got Log? Now Get Intimate With Your App
26
Ops Dashboards Pro-Active Service
Degradation Alerting
Quality Assurance
PerformanceMetricsBusiness
Dashboards
SLA Tracking
Pre-ReleaseSanity TestingSecurit
y Alertin
g
Customer
Support
27
Key Takeaways
• It is YOUR responsibility as a developer to provide useful operational and business data.
• It is not hard, and most of it is for free once you have the patterns in place.
• The same patterns and practices can be applied to practically any platform, and any type of application or service.
• When you provide consistent and predictable data others can build on your greatness
28
Get the Code@hoserdude
Spring Pet Clinic Intervention:https://github.com/hoserdude/spring-petclinic-instrumented
MVC Music Store Intervention:https://github.com/hoserdude/mvcmusicstore-instrumented
NodeCellar Intervention:https://github.com/hoserdude/nodecellar-instrumented
Intuit Speakers @ Silicon Code Camp 2013:SATURDAY9:45 a.m. - Ramakrishna Kollipara – “Complete Automation of Performance Testing” 1:45 p.m. - Joe Wells - “QBO: Journey From legacy Java app to a Client-side HTML5 app”3:30 p.m. - Naga Addagadde & Sangeeta Narang – “Intuit APIs for Financial Transaction Aggregation”5:00 p.m. Ted Drake –“Hitting the Accessibility High Notes with ARIA”
SUNDAY9:15 a.m. - Eugene Krivopaltsev –“Building Native Mobile Apps with Custom Views” 1:15 p.m. - Tim Hobson – “Developing Highly Instrumental Applications with Minimal Effort”
You don't want to miss out on a chance to win this cool headset. Stop by our booth to enter!
For more information about joining our organization visit our booth or connect with our onsite recruiter:
30
THANK YOU
Want to talk more?
I’ll be at the Intuit booth today 3-5 PM.