real-time event processing using linq for logs & traces
TRANSCRIPT
![Page 1: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/1.jpg)
![Page 2: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/2.jpg)
Real-time Event Processing using LINQ for Traces (TX)Gert Drapers (#DataDude)
Principle Software Design Engineer
![Page 3: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/3.jpg)
Agenda
•Rx (Reactive Extensions)
•Tx (LINQ to Logs and Traces)
•Case Study
Subtitel
![Page 4: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/4.jpg)
TX (LINQ to Logs and Traces)
•http://tx.codeplex.com/
•TX allows you use LINQ directly on raw event sources:• Ad-hoc query on past history in
trace and log files • Standing queries on real-time feeds,
such as Event Tracing for Windows (ETW) sessions
![Page 6: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/6.jpg)
Reactive Essential Interfaces
namespace System {
// Producers (.NET events, WinRT Events, Sensors, APM method etc.)
public interface IObservable<out T> {
IDisposable Subscribe(IObserver<T> observer);
}
// Events stream consumers
public interface IObserver<in T> {
void OnNext(T value);
void OnError(Exception error);
void OnCompleted();
}
}
![Page 7: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/7.jpg)
Event processing at all scales with Reactive Extensions Bart de SmetThrusday 17 april 13:15 - 14:30
![Page 8: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/8.jpg)
DemoUsing TX with LINQPad
![Page 9: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/9.jpg)
Event Sources
• Tx.Windows• Event Tracing for Windows (ETW)• Performance Counters - interop with the "Performance Data Helper"
(PDH) native API• Event Logs (.evtx)• IIS W3C text log files
• Tx.SqlServer• SQL Server Extended Events (XEvent) is the tracing technology used by
the SQL-Server engine.
• Extensibility
![Page 10: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/10.jpg)
Type Generation
![Page 11: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/11.jpg)
Playback
•Structured Mode• Lose analogy is that events of given type are like a "Table“• playback.GetObservable< T> is similar to dataContext.GetTable<T> in
LINQ to SQL• More precisely, IObservable is a sequence (i.e. order matters) • This is the same distinction as IEnumerable vs.. SQL Table
•Timeline Mode• This mode is useful to ask questions that are about the timeline of all
events, regardless of type
![Page 12: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/12.jpg)
Playback Features
•Multiplexed sequences
•Hiding the heterogeneity of the event sources
•Same API for real-time and past history
•Single-Read to answer multiple queries from file(s)
•Occurrence Time Scheduler
![Page 13: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/13.jpg)
DemoHttpSpew
![Page 14: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/14.jpg)
Case Study XblWatch
•Objective of XblWatch:• Replace existing rule based SCOM pack monitoring of web service• Provide near-realtime monitoring• Create Synthetic Performance Counters for success and failure rates
per API and service• Publish Synthetic Performance Counters to integrate with existing
monitoring and alerting system
![Page 15: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/15.jpg)
XblWatch/erxblwatcher.prod.live
XblWatcher.exe XblWatchCmd.exehttp://*:80
tcp://*:9000
Perf Counters Site/Service/API
Machine 1
XblWatch.exe
http.sys ETW trace
Perf CountersMachine/Service/API
Machine 2
XblWatch.exe
http.sys ETW trace
Perf CountersMachine/Service/API
Machine 3
XblWatch.exe
http.sys ETW trace
Perf CounterMachine/Service/API
Machine N
XblWatch.exe
http.sys ETW trace
Perf CountersMachine/Service/API
![Page 16: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/16.jpg)
DemoXblWatch
![Page 17: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/17.jpg)
Resources
•Tx (LINQ to Logs and Traces)
• (Rx) Reactive Extensions
•Event processing at all scales with Reactive Extensions Bart de SmetThrusday 17 april 13:15 - 14:30
![Page 18: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/18.jpg)
Laat ons weten wat u vindt van deze sessie! Vul de evaluatie in via www.techdaysapp.nl en maak kans op een van de 20 prijzen*. Prijswinnaars worden bekend gemaakt via Twitter (#TechDaysNL). Gebruik hiervoor de code op uw badge.
Let us know how you feel about this session! Give your feedback via www.techdaysapp.nl and possibly win one of the 20 prices*. Winners will be announced via Twitter (#TechDaysNL). Use your personal code on your badge.
* Over de uitslag kan niet worden gecorrespondeerd, prijzen zijn voorbeelden – All results are final, prices are examples
![Page 19: Real-time Event Processing using LINQ for Logs & Traces](https://reader033.vdocuments.mx/reader033/viewer/2022050614/58ecfafb1a28abc06a8b4569/html5/thumbnails/19.jpg)