Esper ReferenceVersion 8.3.0

by EsperTech Inc. [http://www.espertech.com]

Copyright 2006 - 2019 by EsperTech Inc.

http://www.espertech.comhttp://www.espertech.com

iii

Preface ......................................................................................................................... xxvii

1. Getting Started ............................................................................................................ 1

1.1. Introduction to Complex Event Processing ........................................................... 1

1.2. Introduction to the Architecture ............................................................................ 1

1.3. Introduction to EPL ............................................................................................. 2

1.4. Compiler Getting-Started ..................................................................................... 3

1.4.1. Compiler - Step One: Setting up the Compiler Classpath ............................ 3

1.4.2. Compiler - Step Two: Provide Information on Input Events .......................... 3

1.4.3. Compiler - Step Three: Compiling EPL ...................................................... 4

1.5. Runtime Getting-Started ...................................................................................... 5

1.5.1. Runtime - Step One: Setting up the Runtime Classpath .............................. 5

1.5.2. Runtime - Step Two: Obtain Runtime ........................................................ 6

1.5.3. Runtime - Step Three: Deploy EPL Compiled Module and Attach a Callback

.......................................................................................................................... 6

1.5.4. Runtime - Step Four: Send Events ............................................................ 7

1.6. Required 3rd Party Libraries ................................................................................ 8

1.6.1. Common Required 3rd Party Libraries ....................................................... 8

1.6.2. Compiler - Required 3rd Party Libraries ..................................................... 8

1.6.3. Runtime - Required 3rd Party Libraries ...................................................... 8

2. Basic Concepts ........................................................................................................... 9

2.1. Introduction ........................................................................................................ 9

2.2. Basic Select ..................................................................................................... 10

2.3. Basic Aggregation ............................................................................................. 11

2.4. Basic Filter ....................................................................................................... 12

2.5. Basic Filter and Aggregation .............................................................................. 13

2.6. Basic Data Window ........................................................................................... 14

2.7. Basic Data Window and Aggregation ................................................................. 15

2.8. Basic Filter, Data Window and Aggregation ........................................................ 17

2.9. Basic Where-Clause ......................................................................................... 17

2.10. Basic Time Window and Aggregation ............................................................... 19

2.11. Basic Partitioned Statement ............................................................................. 20

2.12. Basic Output-Rate-Limited Statement ............................................................... 21

2.13. Basic Partitioned and Output-Rate-Limited Statement ........................................ 22

2.14. Basic Named Windows and Tables .................................................................. 23

2.14.1. Named Windows .................................................................................. 23

2.14.2. Tables ................................................................................................. 25

2.15. Basic Aggregated Statement Types ................................................................. 27

2.15.1. Un-Aggregated and Un-Grouped ........................................................... 27

2.15.2. Fully Aggregated and Un-Grouped ........................................................ 27

2.15.3. Aggregated and Un-Grouped ................................................................ 28

2.15.4. Fully Aggregated and Grouped .............................................................. 28

2.15.5. Aggregated and Grouped ...................................................................... 28

2.16. Basic Match-Recognize Patterns ...................................................................... 29

2.17. Basic EPL Patterns ......................................................................................... 29

Esper Reference

iv

2.18. Basic Indexes ................................................................................................. 30

2.18.1. Index Kinds .......................................................................................... 30

2.18.2. Filter Indexes ....................................................................................... 31

2.18.3. Event Indexes ...................................................................................... 35

2.19. Basic Null ....................................................................................................... 35

3. Event Representations ............................................................................................... 37

3.1. Event Underlying Java Objects .......................................................................... 37

3.2. Event Properties ............................................................................................... 38

3.2.1. Escape Characters ................................................................................. 39

3.2.2. Expression as Key or Index Value ........................................................... 40

3.3. Dynamic Event Properties ................................................................................. 41

3.4. Fragment and Fragment Type ........................................................................... 43

3.5. Comparing Event Representations ..................................................................... 43

3.5.1. Incoming Events .................................................................................... 44

3.5.2. Outgoing Events .................................................................................... 45

3.5.3. Schema ................................................................................................. 45

3.5.4. Side-by-Side .......................................................................................... 46

3.6. Support for Generic Tuples ............................................................................... 47

3.7. Updating, Merging and Versioning Events .......................................................... 49

3.8. Coarse-Grained Events ..................................................................................... 49

3.9. Event Objects Instantiated and Populated by Insert Into ...................................... 49

3.10. Event Type Uniqueness .................................................................................. 51

4. Context and Context Partitions ................................................................................. 53

4.1. Introduction ....................................................................................................... 53

4.2. Context Declaration ........................................................................................... 55

4.2.1. Context-Provided Properties ................................................................... 55

4.2.2. Keyed Segmented Context ..................................................................... 56

4.2.3. Hash Segmented Context ....................................................................... 63

4.2.4. Category Segmented Context ................................................................. 66

4.2.5. Non-Overlapping Context ........................................................................ 68

4.2.6. Overlapping Context ............................................................................... 70

4.2.7. Context Conditions ................................................................................. 74

4.3. Context Nesting ................................................................................................ 78

4.3.1. Nested Context Sample Walk-Through .................................................... 79

4.3.2. Built-In Nested Context Properties ........................................................... 81

4.4. Partitioning Without Context Declaration ............................................................. 82

4.5. Output When a Context Partition Ends or Terminates .......................................... 82

4.6. Context and Named Window ............................................................................. 84

4.7. Context and Tables ........................................................................................... 86

4.8. Context and Variables ....................................................................................... 87

4.9. Operations on Specific Context Partitions ........................................................... 87

5. EPL Reference: Clauses ............................................................................................ 89

5.1. EPL Introduction ............................................................................................... 89

5.2. EPL Syntax ...................................................................................................... 90

v

5.2.1. Specifying Time Periods ......................................................................... 90

5.2.2. Using Comments .................................................................................... 91

5.2.3. Reserved Keywords ............................................................................... 92

5.2.4. Escaping Strings .................................................................................... 93

5.2.5. Data Types ............................................................................................ 93

5.2.6. Using Constants and Enum Types .......................................................... 96

5.2.7. Annotation ............................................................................................. 97

5.2.8. Expression Alias ................................................................................... 103

5.2.9. Expression Declaration ......................................................................... 104

5.2.10. Script Declaration ............................................................................... 107

5.2.11. Referring to a Context ........................................................................ 107

5.2.12. Composite Keys and Array Values as Keys .......................................... 107

5.3. Choosing Event Properties and Events: The Select Clause ................................ 108

5.3.1. Choosing the Event Itself: Select * ........................................................ 108

5.3.2. Choosing Specific Event Properties ....................................................... 109

5.3.3. Expressions ......................................................................................... 109

5.3.4. Renaming Event Properties ................................................................... 110

5.3.5. Choosing Event Properties and Events in a Join ..................................... 110

5.3.6. Choosing Event Properties and Events From a Pattern ........................... 112

5.3.7. Selecting Insert and Remove Stream Events .......................................... 112

5.3.8. Select Distinct ...................................................................................... 113

5.3.9. Transposing an Expression Result to a Stream ...................................... 114

5.3.10. Selecting EventBean Instead of Underlying Event ................................. 114

5.4. Specifying Event Streams: The From Clause .................................................... 115

5.4.1. Filter-Based Event Streams ................................................................... 116

5.4.2. Pattern-Based Event Streams ............................................................... 119

5.4.3. Specifying Data Windows ..................................................................... 120

5.4.4. Multiple Data Windows ......................................................................... 121

5.4.5. Using the Stream Name ....................................................................... 122

5.5. Specifying Search Conditions: The Where Clause ............................................. 124

5.6. Aggregates and Grouping: The Group-By Clause and the Having Clause ............ 125

5.6.1. Using Aggregate Functions ................................................................... 125

5.6.2. Organizing Statement Results into Groups: The Group-by Clause ............ 127

5.6.3. Using Group-By with Rollup, Cube and Grouping Sets ............................ 130

5.6.4. Specifying Grouping for Each Aggregation Function ................................ 134

5.6.5. Specifying a Filter Expression for Each Aggregation Function .................. 135

5.6.6. Selecting Groups of Events: The Having Clause ..................................... 136

5.6.7. How the Stream Filter, Where, Group By and Having-Clauses Interact ..... 137

5.6.8. Comparing Keyed Segmented Context, the Group By Clause and

#groupwin for Data Windows .......................................................................... 138

5.7. Stabilizing and Controlling Output: The Output Clause ....................................... 139

5.7.1. Output Clause Options ......................................................................... 139

5.7.2. Aggregation, Group By, Having and Output Clause Interaction ................. 145

5.8. Sorting Output: the Order By Clause ................................................................ 146

Esper Reference

vi

5.9. Limiting Row Count: the Limit Clause ............................................................... 147

5.10. Merging Streams and Continuous Insertion: The Insert Into Clause ................... 148

5.10.1. Transposing a Property to a Stream .................................................... 150

5.10.2. Merging Streams by Event Type .......................................................... 151

5.10.3. Merging Disparate Types of Events: Variant Streams ............................ 152

5.10.4. Decorated Events ............................................................................... 153

5.10.5. Event as a Property ............................................................................ 153

5.10.6. Instantiating and Populating an Underlying Event Object ....................... 154

5.10.7. Transposing an Expression Result ....................................................... 154

5.10.8. Select-Clause Expression and Inserted-Into Column Event Type ............ 154

5.10.9. Insert Into for Event Types Without Properties ...................................... 155

5.11. Subqueries .................................................................................................... 156

5.11.1. The 'Exists' Keyword ........................................................................... 159

5.11.2. The 'In' and 'Not In' Keywords ............................................................. 160

5.11.3. The 'Any' and 'Some' Keywords .......................................................... 160

5.11.4. The 'All' Keyword ................................................................................ 161

5.11.5. Subquery With Group By Clause ......................................................... 162

5.11.6. Multi-Column Selection ....................................................................... 162

5.11.7. Multi-Row Selection ............................................................................ 163

5.11.8. Hints Related to Subqueries ................................................................ 164

5.12. Joining Event Streams ................................................................................... 165

5.12.1. Introducing Joins ................................................................................ 165

5.12.2. Inner (Default) Joins ........................................................................... 167

5.12.3. Outer, Left and Right Joins ................................................................. 168

5.12.4. Unidirectional Joins ............................................................................. 170

5.12.5. Unidirectional Full Outer Joins ............................................................. 172

5.12.6. Hints Related to Joins ......................................................................... 172

5.13. Accessing Relational Data via SQL ................................................................ 173

5.13.1. Joining SQL Query Results ................................................................. 174

5.13.2. SQL Query and the EPL Where Clause ............................................... 176

5.13.3. Outer Joins With SQL Queries ............................................................ 177

5.13.4. Using Patterns to Request (Poll) Data .................................................. 178

5.13.5. Polling SQL Queries via Iterator .......................................................... 178

5.13.6. JDBC Implementation Overview .......................................................... 178

5.13.7. Oracle Drivers and No-Metadata Workaround ....................................... 179

5.13.8. SQL Input Parameter and Column Output Conversion ........................... 180

5.13.9. SQL Row POJO Conversion ............................................................... 180

5.14. Accessing Non-Relational Data via Method, Script or UDF Invocation ............... 180

5.14.1. Joining Method, Script or UDF Invocation Results ................................. 181

5.14.2. Polling Invocation Results via Iterator ................................................... 182

5.14.3. Providing the Method .......................................................................... 183

5.14.4. Using a Map Return Type ................................................................... 185

5.14.5. Using a Object Array Return Type ....................................................... 186

5.14.6. Using an EventBean Return Type ........................................................ 188

vii

5.14.7. Providing the Script ............................................................................ 188

5.14.8. Providing the UDF .............................................................................. 189

5.15. Declaring an Event Type: Create Schema ...................................................... 189

5.15.1. Declare an Event Type by Providing Names and Types ......................... 189

5.15.2. Declare an Event Type by Providing a Class Name .............................. 193

5.15.3. Declare a Variant Stream .................................................................... 194

5.16. Splitting and Duplicating Streams ................................................................... 194

5.16.1. Generating Marker Events for Contained Events ................................... 196

5.17. Variables and Constants ................................................................................ 198

5.17.1. Creating Variables: The Create Variable Clause ................................... 198

5.17.2. Setting Variable Values: The On Set Clause ........................................ 201

5.17.3. Using Variables .................................................................................. 202

5.17.4. Object-Type Variables ......................................................................... 203

5.17.5. Class and Event-Type Variables .......................................................... 203

5.18. Declaring Global Expressions, Aliases and Scripts: Create Expression .............. 204

5.18.1. Global Expression Aliases ................................................................... 205

5.18.2. Global Expression Declarations ........................................................... 205

5.18.3. Global Scripts ..................................................................................... 206

5.19. Contained-Event Selection ............................................................................. 207

5.19.1. Select-Clause in a Contained-Event Selection ...................................... 210

5.19.2. Where Clause in a Contained-Event Selection ...................................... 212

5.19.3. Contained-Event Selection and Joins ................................................... 213

5.19.4. Sentence and Word Example .............................................................. 214

5.19.5. More Examples .................................................................................. 216

5.19.6. Contained Expression Returning an Array of Property Values ................ 217

5.19.7. Contained Expression Returning an Array of EventBean ........................ 217

5.19.8. Generating Marker Events Such as a Begin and End Event ................... 218

5.19.9. Contained-Event Limitations ................................................................ 218

5.20. Updating an Insert Stream: The Update IStream Clause .................................. 219

5.20.1. Immutability and Updates .................................................................... 221

5.21. Controlling Event Delivery : The For Clause .................................................... 221

6. EPL Reference: Named Windows and Tables .......................................................... 225

6.1. Overview ........................................................................................................ 225

6.1.1. Named Window Overview ..................................................................... 225

6.1.2. Table Overview .................................................................................... 226

6.1.3. Comparing Named Windows and Tables ............................................... 226

6.2. Named Window Usage .................................................................................... 228

6.2.1. Creating Named Windows: The Create Window clause ........................... 228

6.2.2. Inserting Into Named Windows .............................................................. 231

6.2.3. Selecting From Named Windows ........................................................... 234

6.3. Table Usage ................................................................................................... 235

6.3.1. Creating Tables: The Create Table Clause ............................................. 235

6.3.2. Aggregating Into Table Rows: The Into Table Clause .............................. 238

6.3.3. Table Column Keyed-Access Expressions ............................................. 242

Esper Reference

viii

6.3.4. Inserting Into Tables ............................................................................. 244

6.3.5. Selecting From Tables .......................................................................... 245

6.4. Triggered Select: The On Select Clause ........................................................... 246

6.4.1. Notes on On-Select With Named Windows ............................................ 248

6.4.2. Notes on On-Select With Tables ........................................................... 249

6.4.3. On-Select Compared to Join ................................................................. 249

6.5. Triggered Select+Delete: The On Select Delete Clause ..................................... 249

6.6. Updating Data: The On Update Clause ............................................................ 250

6.6.1. Notes on On-Update With Named Windows ........................................... 254

6.6.2. Notes on On-Update With Tables .......................................................... 254

6.7. Deleting Data: The On Delete Clause ............................................................... 254

6.7.1. Using Patterns in the On-Delete Clause ................................................. 255

6.7.2. Silent Delete With Named Windows ...................................................... 256

6.7.3. Notes on On-Delete With Named Windows ............................................ 256

6.7.4. Notes on On-Update With Tables .......................................................... 256

6.8. Triggered Upsert Using the On-Merge Clause ................................................... 256

6.8.1. On-Merge Insert ................................................................................... 257

6.8.2. On-Merge Update, Delete, Select (Upsert) ............................................. 257

6.8.3. Notes on On-Merge With Named Windows ............................................ 262

6.8.4. Notes on On-Merge With Tables ........................................................... 262

6.9. Explicitly Indexing Named Windows and Tables ................................................ 262

6.10. Using Fire-and-Forget Queries With Named Windows and Tables ..................... 265

6.10.1. Inserting Data ..................................................................................... 265

6.10.2. Updating Data .................................................................................... 267

6.10.3. Deleting Data ..................................................................................... 267

6.11. Events as Property ........................................................................................ 268

7. EPL Reference: Patterns .......................................................................................... 271

7.1. Event Pattern Overview ................................................................................... 271

7.2. How to Use Patterns ....................................................................................... 272

7.2.1. Pattern Syntax ..................................................................................... 272

7.2.2. Patterns in EPL .................................................................................... 272

7.2.3. Subscribing to Pattern Events ............................................................... 273

7.2.4. Pulling Data From Patterns ................................................................... 273

7.2.5. Pattern Error Reporting ......................................................................... 274

7.2.6. Suppressing Same-Event Matches ........................................................ 274

7.2.7. Discarding Partially Completed Patterns ................................................. 275

7.3. Operator Precedence ...................................................................................... 276

7.4. Filter Expressions in Patterns .......................................................................... 278

7.4.1. Filter Expressions and Filter Indexes ..................................................... 280

7.4.2. Controlling Event Consumption ............................................................. 280

7.4.3. Use With Named Windows and Tables .................................................. 281

7.5. Pattern Operators ........................................................................................... 282

7.5.1. Every ................................................................................................... 282

7.5.2. Every-Distinct ....................................................................................... 287

ix

7.5.3. Repeat ................................................................................................. 289

7.5.4. Repeat-Until ......................................................................................... 291

7.5.5. And ..................................................................................................... 294

7.5.6. Or ........................................................................................................ 296

7.5.7. Not ...................................................................................................... 296

7.5.8. Followed-By ......................................................................................... 297

7.5.9. Pattern Guards ..................................................................................... 299

7.6. Pattern Atoms ................................................................................................. 304

7.6.1. Filter Atoms ......................................................................................... 304

7.6.2. Observer Atoms Overview .................................................................... 304

7.6.3. Interval (timer:interval) .......................................................................... 304

7.6.4. Crontab (timer:at) ................................................................................. 305

7.6.5. Schedule (timer:schedule) ..................................................................... 308

7.7. Event Pattern Walkthrough .............................................................................. 314

8. EPL Reference: Match Recognize ............................................................................ 317

8.1. Overview ........................................................................................................ 317

8.2. Comparison of Match Recognize and EPL Patterns ........................................... 317

8.3. Syntax ............................................................................................................ 318

8.3.1. Syntax Example ................................................................................... 319

8.4. Pattern and Pattern Operators ......................................................................... 321

8.4.1. Operator Precedence ............................................................................ 321

8.4.2. Concatenation ...................................................................................... 321

8.4.3. Alternation ............................................................................................ 322

8.4.4. Quantifiers Overview ............................................................................ 323

8.4.5. Permutations ........................................................................................ 323

8.4.6. Variables Can Be Singleton or Group .................................................... 324

8.4.7. Eliminating Duplicate Matches ............................................................... 325

8.4.8. Greedy or Reluctant ............................................................................. 326

8.4.9. Quantifier - One or More (+ and +?) ...................................................... 327

8.4.10. Quantifier - Zero or More (* and *?) ..................................................... 327

8.4.11. Quantifier - Zero or One (? and ??) ..................................................... 328

8.4.12. Repetition - Exactly N Matches ............................................................ 329

8.4.13. Repetition - N or More Matches ........................................................... 330

8.4.14. Repetition - Between N and M Matches ............................................... 330

8.4.15. Repetition - Between Zero and M Matches ........................................... 331

8.4.16. Repetition Equivalence ........................................................................ 332

8.5. Define Clause ................................................................................................. 332

8.5.1. The Prev Operator ............................................................................... 333

8.6. Measure Clause .............................................................................................. 334

8.7. Datawindow-Bound ......................................................................................... 334

8.8. Interval ........................................................................................................... 335

8.9. Interval-or-Terminated ..................................................................................... 336

8.10. Use With Different Event Types ..................................................................... 337

8.11. Limiting Runtime-Wide State Count ................................................................ 338

Esper Reference

x

8.12. Limitations .................................................................................................... 339

9. EPL Reference: Operators ....................................................................................... 341

9.1. Arithmetic Operators ....................................................................................... 341

9.2. Logical and Comparison Operators .................................................................. 341

9.2.1. Null-Value Comparison Operators ......................................................... 341

9.3. Concatenation Operators ................................................................................. 342

9.4. Binary Operators ............................................................................................. 342

9.5. Array Definition Operator ................................................................................. 343

9.6. Dot Operator ................................................................................................... 344

9.6.1. Duck Typing ......................................................................................... 345

9.7. The 'In' Keyword ............................................................................................. 345

9.7.1. 'In' for Range Selection ......................................................................... 346

9.8. The 'Between' Keyword ................................................................................... 347

9.9. The 'Like' Keyword .......................................................................................... 348

9.10. The 'Regexp' Keyword ................................................................................... 349

9.11. The 'Any' and 'Some' Keywords ..................................................................... 350

9.12. The 'All' Keyword .......................................................................................... 351

9.13. The 'New' Keyword ....................................................................................... 351

9.13.1. Using 'New' to Populate a Data Structure ............................................. 352

9.13.2. Using 'New' to Instantiate an Object ..................................................... 353

10. EPL Reference: Functions ..................................................................................... 355

10.1. Single-Row Function Reference ..................................................................... 355

10.1.1. The Case Control Flow Function ......................................................... 357

10.1.2. The Cast Function .............................................................................. 358

10.1.3. The Coalesce Function ....................................................................... 360

10.1.4. The Current_Evaluation_Context Function ............................................ 360

10.1.5. The Current_Timestamp Function ........................................................ 361

10.1.6. The Event_Identity_Equals Function .................................................... 361

10.1.7. The Exists Function ............................................................................ 361

10.1.8. The Grouping Function ....................................................................... 362

10.1.9. The Grouping_Id Function ................................................................... 362

10.1.10. The Instance-Of Function .................................................................. 363

10.1.11. The Istream Function ........................................................................ 364

10.1.12. The Min and Max Functions .............................................................. 365

10.1.13. The Previous Function ...................................................................... 365

10.1.14. The Previous-Tail Function ................................................................ 368

10.1.15. The Previous-Window Function ......................................................... 369

10.1.16. The Previous-Count Function ............................................................ 371

10.1.17. The Prior Function ............................................................................ 372

10.1.18. The Type-Of Function ....................................................................... 372

10.2. Aggregation Functions ................................................................................... 374

10.2.1. SQL-Standard Functions ..................................................................... 375

10.2.2. Event Aggregation Functions ............................................................... 377

10.2.3. Approximation Aggregation Functions .................................................. 385

xi

10.2.4. Additional Aggregation Functions ......................................................... 388

10.3. User-Defined Functions ................................................................................. 391

10.4. Select-Clause Transpose Function ................................................................. 395

10.4.1. Transpose with Insert-Into ................................................................... 396

11. EPL Reference: Enumeration Methods .................................................................. 397

11.1. Overview ...................................................................................................... 397

11.2. Example Events ............................................................................................ 401

11.3. How to Use .................................................................................................. 403

11.3.1. Syntax ............................................................................................... 403

11.3.2. Introductory Examples ........................................................................ 403

11.3.3. Input, Output and Limitations ............................................................... 404

11.4. Inputs ........................................................................................................... 405

11.4.1. Subquery Results ............................................................................... 405

11.4.2. Named Window .................................................................................. 407

11.4.3. Table ................................................................................................. 408

11.4.4. Event Property and Insert-Into With @eventbean .................................. 408

11.4.5. Event Aggregation Function ................................................................ 409

11.4.6. Prev, Prevwindow and Prevtail Single-Row Functions as Input ............... 410

11.4.7. Single-Row Function, User-Defined Function and Enum Types .............. 411

11.4.8. Declared Expression ........................................................................... 413

11.4.9. Variables ............................................................................................ 413

11.4.10. Substitution Parameters .................................................................... 414

11.4.11. Match-Recognize Group Variable ....................................................... 414

11.4.12. Pattern Repeat and Repeat-Until Operators ........................................ 414

11.5. Example ....................................................................................................... 414

11.6. Reference ..................................................................................................... 415

11.6.1. Aggregate .......................................................................................... 415

11.6.2. AllOf .................................................................................................. 416

11.6.3. AnyOf ................................................................................................ 417

11.6.4. Average ............................................................................................. 417

11.6.5. CountOf ............................................................................................. 418

11.6.6. DistinctOf ........................................................................................... 419

11.6.7. Except ............................................................................................... 419

11.6.8. FirstOf ................................................................................................ 420

11.6.9. GroupBy ............................................................................................ 420

11.6.10. Intersect ........................................................................................... 421

11.6.11. LastOf .............................................................................................. 422

11.6.12. LeastFrequent .................................................................................. 422

11.6.13. Max ................................................................................................. 423

11.6.14. MaxBy .............................................................................................. 423

11.6.15. Min .................................................................................................. 424

11.6.16. MinBy .............................................................................................. 425

11.6.17. MostFrequent ................................................................................... 425

11.6.18. OrderBy and OrderByDesc ................................................................ 426

Esper Reference

xii

11.6.19. Reverse ........................................................................................... 426

11.6.20. SelectFrom ....................................................................................... 427

11.6.21. SequenceEqual ................................................................................ 427

11.6.22. SumOf ............................................................................................. 428

11.6.23. Take ................................................................................................ 428

11.6.24. TakeLast .......................................................................................... 429

11.6.25. TakeWhile ........................................................................................ 429

11.6.26. TakeWhileLast .................................................................................. 430

11.6.27. ToMap ............................................................................................. 431

11.6.28. Union ............................................................................................... 431

11.6.29. Where .............................................................................................. 432

12. EPL Reference: Date-Time Methods ....................................................................... 433

12.1. Overview ...................................................................................................... 433

12.2. How to Use .................................................................................................. 437

12.2.1. Syntax ............................................................................................... 437

12.3. Calendar and Formatting Reference ............................................................... 438

12.3.1. Between ............................................................................................. 438

12.3.2. Format ............................................................................................... 439

12.3.3. Get (By Field) .................................................................................... 440

12.3.4. Get (By Name) ................................................................................... 440

12.3.5. Minus ................................................................................................. 441

12.3.6. Plus ................................................................................................... 441

12.3.7. RoundCeiling ...................................................................................... 442

12.3.8. RoundFloor ........................................................................................ 442

12.3.9. RoundHalf .......................................................................................... 442

12.3.10. Set (By Field) ................................................................................... 443

12.3.11. WithDate .......................................................................................... 443

12.3.12. WithMax ........................................................................................... 444

12.3.13. WithMin ............................................................................................ 444

12.3.14. WithTime .......................................................................................... 444

12.3.15. ToCalendar ...................................................................................... 445

12.3.16. ToDate ............................................................................................. 445

12.3.17. ToMillisec ......................................................................................... 445

12.4. Interval Algebra Reference ............................................................................ 445

12.4.1. Examples ........................................................................................... 446

12.4.2. Interval Algebra Parameters ................................................................ 446

12.4.3. Performance ....................................................................................... 447

12.4.4. Limitations .......................................................................................... 447

12.4.5. After .................................................................................................. 447

12.4.6. Before ................................................................................................ 448

12.4.7. Coincides ........................................................................................... 449

12.4.8. During ................................................................................................ 450

12.4.9. Finishes ............................................................................................. 451

12.4.10. Finished By ...................................................................................... 452

xiii

12.4.11. Includes ........................................................................................... 453

12.4.12. Meets ............................................................................................... 454

12.4.13. Met By ............................................................................................. 455

12.4.14. Overlaps .......................................................................................... 455

12.4.15. Overlapped By .................................................................................. 456

12.4.16. Starts ............................................................................................... 457

12.4.17. Started By ........................................................................................ 458

13. EPL Reference: Aggregation Methods ................................................................... 461

13.1. Overview ...................................................................................................... 461

13.2. How to Use .................................................................................................. 461

13.2.1. Syntax ............................................................................................... 461

13.2.2. Use with Aggregation Function ............................................................ 462

13.2.3. Use with Table Column ....................................................................... 462

13.2.4. Using Dot to Obtain Property Values ................................................... 463

13.2.5. Using Dot to Enumerate Events .......................................................... 463

13.3. Aggregation Methods for Sorted Aggregations ................................................ 463

13.3.1. Overview ............................................................................................ 463

13.3.2. Specifying Composite Keys ................................................................. 467

13.3.3. CeilingEvent, FloorEvent, HigherEvent, LowerEvent, GetEvent .............. 467

13.3.4. CeilingEvents, FloorEvents, HigherEvents, LowerEvents, GetEvents ...... 467

13.3.5. CeilingKey, FloorKey, HigherKey, LowerKey ......................................... 468

13.3.6. FirstEvent, LastEvent, MinBy, MaxBy ................................................... 469

13.3.7. FirstEvents, LastEvents ....................................................................... 469

13.3.8. FirstKey, LastKey ............................................................................... 469

13.3.9. ContainsKey ....................................................................................... 470

13.3.10. CountEvents ..................................................................................... 470

13.3.11. CountKeys ........................................................................................ 470

13.3.12. EventsBetween ................................................................................. 471

13.3.13. Submap ........................................................................................... 471

13.3.14. NavigableMapReference ................................................................... 471

13.4. Aggregation Methods for Window Aggregations .............................................. 472

13.4.1. Overview ............................................................................................ 472

13.4.2. CountEvents ....................................................................................... 473

13.4.3. ListReference ..................................................................................... 473

13.4.4. First, Last ........................................................................................... 474

13.5. Aggregation Methods for CountMinSketch Aggregations .................................. 474

13.6. Aggregation Methods for Custom Plug-In Multi-Function Aggregations .............. 474

14. EPL Reference: Data Windows .............................................................................. 475

14.1. A Note on Data Window Name and Parameters .............................................. 478

14.2. A Note on Batch Windows ............................................................................. 480

14.3. Data Windows ............................................................................................... 481

14.3.1. Length Window (length or win:length) .................................................. 481

14.3.2. Length Batch Window (length_batch or win:length_batch) ...................... 482

14.3.3. Time Window (time or win:time) .......................................................... 482

Esper Reference

xiv

14.3.4. Externally-timed Window (ext_timed or win:ext_timed) ........................... 482

14.3.5. Time batch Window (time_batch or win:time_batch) .............................. 483

14.3.6. Externally-timed Batch Window (ext_timed_batch or

win:ext_timed_batch) ...................................................................................... 485

14.3.7. Time-Length Combination Batch Window (time_length_batch or

win:time_length_batch) ................................................................................... 486

14.3.8. Time-Accumulating Window (time_accum or win:time_accum) ............... 487

14.3.9. Keep-All Window (keepall or win:keepall) ............................................. 488

14.3.10. First Length Window(firstlength or win:firstlength) ................................ 488

14.3.11. First Time Window (firsttime or win:firsttime) ....................................... 489

14.3.12. Expiry Expression Window (expr or win:expr) ..................................... 489

14.3.13. Expiry Expression Batch Window (expr_batch or win:expr_batch) ......... 492

14.3.14. Unique Window (unique or std:unique) ............................................... 494

14.3.15. Grouped Data Window (groupwin or std:groupwin) .............................. 495

14.3.16. Last Event Window (std:lastevent) ..................................................... 498

14.3.17. First Event Window (firstevent or std:firstevent) ................................... 499

14.3.18. First Unique Window (firstunique or std:firstunique) ............................. 499

14.3.19. Sorted Window (sort or ext:sort) ........................................................ 500

14.3.20. Ranked Window (rank or ext:rank) ..................................................... 500

14.3.21. Time-Order Window (time_order or ext:time_order) ............................. 502

14.3.22. Time-To-Live Window (timetolive or ext:timetolive) .............................. 503

14.4. Special Derived-Value Windows ..................................................................... 504

14.4.1. Size Derived-Value Window (size) or std:size) ...................................... 504

14.4.2. Univariate Statistics Derived-Value Window (uni or stat:uni) ................... 505

14.4.3. Regression Derived-Value Window (linest or stat:linest) ......................... 506

14.4.4. Correlation Derived-Value Window (correl or stat:correl) ........................ 508

14.4.5. Weighted Average Derived-Value Window (weighted_avg or

stat:weighted_avg) ......................................................................................... 509

15. Compiler Reference ............................................................................................... 511

15.1. Introduction ................................................................................................... 511

15.2. Concepts ...................................................................................................... 511

15.2.1. Module ............................................................................................... 511

15.2.2. EPL-objects ........................................................................................ 513

15.2.3. Dependencies .................................................................................... 513

15.2.4. Dependency Resolution ...................................................................... 514

15.2.5. Access Modifiers ................................................................................ 515

15.2.6. Bus Modifier for Event Types .............................................................. 516

15.3. Compiling a Module ...................................................................................... 517

15.4. Reading and Writing a Compiled Module ........................................................ 517

15.5. Reading Module Content ............................................................................... 517

15.6. Compiler Arguments ...................................................................................... 517

15.6.1. Compiler Configuration ........................................................................ 517

15.6.2. Compiler Path .................................................................................... 518

15.6.3. Compiler Options ................................................................................ 520

xv

15.7. Statement Object Model ................................................................................ 520

15.7.1. Building an Object Model .................................................................... 521

15.7.2. Building Expressions ........................................................................... 522

15.7.3. Building a Pattern Statement ............................................................... 523

15.7.4. Building a Select Statement ................................................................ 524

15.7.5. Building a Create-Variable and On-Set Statement ................................. 524

15.7.6. Building Create-Window, On-Delete and On-Select Statements .............. 525

15.8. Substitution Parameters ................................................................................. 526

15.9. OSGi, Class Loader, Class-For-Name ............................................................ 527

15.10. Authoring Tools ........................................................................................... 527

15.11. Testing Tools .............................................................................................. 528

15.12. Debugging .................................................................................................. 528

15.12.1. @Audit Annotation ............................................................................ 528

15.13. Ordering Multiple Modules ........................................................................... 530

15.14. Logging ....................................................................................................... 530

15.15. Debugging Generated Code ......................................................................... 530

15.16. Compiler Version and Runtime Version ......................................................... 531

15.17. Compiler Byte Code Optimizations ............................................................... 531

16. Runtime Reference ................................................................................................. 533

16.1. Introduction ................................................................................................... 533

16.2. Obtaining a Runtime From EPRuntimeProvider ............................................... 533

16.3. The EPRuntime Runtime Interface ................................................................. 534

16.4. Deploying and Undeploying Using EPDeploymentService ................................ 535

16.4.1. Substitution Parameters ...................................................................... 536

16.4.2. Atomic Deployment Management ........................................................ 537

16.5. Obtaining Results Using EPStatement ............................................................ 538

16.5.1. Receiving Statement Results ............................................................... 538

16.5.2. Setting a Subscriber Object ................................................................. 541

16.5.3. Adding Listeners ................................................................................. 546

16.5.4. Using Iterators .................................................................................... 547

16.5.5. Event and Event Type ........................................................................ 549

16.5.6. Interrogating Annotations .................................................................... 553

16.6. Processing Events and Time Using EPEventService ........................................ 554

16.6.1. Event Sender ..................................................................................... 556

16.6.2. Receiving Unmatched Events .............................................................. 557

16.7. Execute Fire-and-Forget Queries Using EPFireAndForgetService ..................... 557

16.7.1. Fire-and-forget Query Single Execution ................................................ 558

16.7.2. Fire-and-forget Query Prepared Unparameterized Execution .................. 559

16.7.3. Fire-and-forget Query Prepared Parameterized Execution ..................... 559

16.8. Runtime Threading and Concurrency .............................................................. 560

16.8.1. Advanced Threading ........................................................................... 562

16.8.2. Processing Order ................................................................................ 565

16.9. Controlling Time-Keeping ............................................................................... 566

16.9.1. Controlling Time Using Time Span Events ............................................ 569

Esper Reference

xvi

16.9.2. Time Resolution and Time Unit ........................................................... 570

16.9.3. Internal Timer Based on JVM System Time .......................................... 571

16.10. Exception Handling ...................................................................................... 571

16.11. Condition Handling ...................................................................................... 572

16.12. Runtime and Statement Metrics Reporting .................................................... 573

16.12.1. Runtime Metrics ................................................................................ 574

16.12.2. Statement Metrics ............................................................................. 574

16.13. Monitoring and JMX .................................................................................... 575

16.14. Event Rendering to XML and JSON ............................................................. 575

16.14.1. JSON Event Rendering Conventions and Options ............................... 577

16.14.2. XML Event Rendering Conventions and Options ................................. 577

16.15. Plug-In Loader ............................................................................................ 577

16.16. Context Partition Selection ........................................................................... 579

16.16.1. Selectors .......................................................................................... 581

16.17. Context Partition Administration .................................................................... 581

16.18. Test and Assertion Support .......................................................................... 582

16.18.1. EPAssertionUtil Summary ................................................................. 582

16.18.2. SupportUpdateListener Summary ....................................................... 583

16.18.3. Usage Example ................................................................................ 583

16.19. OSGi, Class Loader, Class-For-Name .......................................................... 584

16.20. When Deploying with J2EE .......................................................................... 584

16.20.1. J2EE Deployment Considerations ...................................................... 585

16.20.2. Servlet Context Listener .................................................................... 585

17. Configuration ......................................................................................................... 589

17.1. Overview ...................................................................................................... 589

17.2. Programmatic Configuration ........................................................................... 589

17.3. Configuration via XML File ............................................................................. 590

17.4. Configuration Common .................................................................................. 590

17.4.1. Annotation Class and Package Imports ................................................ 591

17.4.2. Class and Package Imports ................................................................ 591

17.4.3. Events Represented by Classes .......................................................... 592

17.4.4. Events Represented by java.util.Map ................................................... 597

17.4.5. Events Represented by Object[] (Object-array) ..................................... 598

17.4.6. Events Represented by JSON ............................................................. 600

17.4.7. Events Represented by Avro GenericData.Record ................................ 600

17.4.8. Events Represented by org.w3c.dom.Node .......................................... 602

17.4.9. Event Type Defaults ........................................................................... 608

17.4.10. Event Type Import Package (Event Type Auto-Name) ......................... 611

17.4.11. From-Clause Method Invocation ........................................................ 611

17.4.12. Relational Database Access .............................................................. 612

17.4.13. Common Settings Related to Logging ................................................ 620

17.4.14. Common Settings Related to Time Source ......................................... 621

17.4.15. Variables .......................................................................................... 622

17.4.16. Variant Stream ................................................................................. 622

xvii

17.5. Configuration Compiler .................................................................................. 623

17.5.1. Compiler Settings Related to Byte Code Generation ............................. 623

17.5.2. Compiler Settings Related to View Resources ...................................... 626

17.5.3. Compiler Settings Related to Logging .................................................. 627

17.5.4. Compiler Settings Related to Stream Selection ..................................... 627

17.5.5. Compiler Settings Related to Language and Locale .............................. 628

17.5.6. Compiler Settings Related to Expression Evaluation .............................. 629

17.5.7. Compiler Settings Related to Scripts .................................................... 631

17.5.8. Compiler Settings Related to Execution of Statements .......................... 631

17.5.9. Compiler Settings Related to Serializers and Deserializers .................... 633

17.6. Configuration Runtime ................................................................................... 633

17.6.1. Runtime Settings Related to Concurrency and Threading ...................... 633

17.6.2. Runtime Settings Related to Logging ................................................... 638

17.6.3. Runtime Settings Related to Variables ................................................. 639

17.6.4. Runtime Settings Related to Patterns ................................................... 640

17.6.5. Runtime Settings Related to Match-Recognize ..................................... 640

17.6.6. Runtime Settings Related to Time Source ............................................ 641

17.6.7. Runtime Settings Related to JMX Metrics ............................................. 642

17.6.8. Runtime Settings Related to Metrics Reporting ..................................... 642

17.6.9. Runtime Settings Related to Expression Evaluation .............................. 644

17.6.10. Runtime Settings Related to Execution of Statements ......................... 645

17.6.11. Runtime Settings Related to Exception Handling ................................. 648

17.6.12. Runtime Settings Related to Condition Handling ................................. 649

17.7. Passing Services or Transient Objects ........................................................... 649

17.7.1. Service Example ................................................................................ 650

17.7.2. Class-for-Name .................................................................................. 650

17.7.3. Class Loader ...................................................................................... 651

17.8. Type Names ................................................................................................. 652

17.9. Logging Configuration .................................................................................... 652

17.9.1. Log4j Logging Configuration ................................................................ 653

18. Script Support ........................................................................................................ 655

18.1. Overview ...................................................................................................... 655

18.2. Syntax .......................................................................................................... 655

18.3. Examples ...................................................................................................... 656

18.4. Built-In EPL Script Attributes .......................................................................... 657

18.5. Performance Notes ....................................................................................... 658

18.6. Additional Notes ............................................................................................ 658

19. EPL Reference: Spatial Methods and Indexes ....................................................... 659

19.1. Overview ...................................................................................................... 659

19.2. Spatial Methods ............................................................................................ 659

19.2.1. Point-Inside-Rectangle ........................................................................ 659

19.2.2. Rectangle-Intersects-Rectangle ........................................................... 660

19.3. Spatial Index - Quadtree ................................................................................ 661

19.3.1. Overview ............................................................................................ 661

Esper Reference

xviii

19.3.2. Declaring a Point-Region Quadtree Index ............................................. 662

19.3.3. Using a Point-Region Quadtree as a Filter Index .................................. 662

19.3.4. Using a Point-Region Quadtree as an Event Index ................................ 663

19.3.5. Declaring a MX-CIF Quadtree Index .................................................... 665

19.3.6. Using a MX-CIF Quadtree as a Filter Index .......................................... 666

19.3.7. Using a MX-CIF Quadtree as an Event Index ....................................... 667

19.4. Spatial Types, Functions and Methods from External Libraries ......................... 668

20. EPL Reference: Data Flow ..................................................................................... 671

20.1. Introduction ................................................................................................... 671

20.2. Usage ........................................................................................................... 671

20.2.1. Overview ............................................................................................ 671

20.2.2. Syntax ............................................................................................... 673

20.3. Built-In Operators .......................................................................................... 678

20.3.1. BeaconSource .................................................................................... 678

20.3.2. EPStatementSource ............................................................................ 680

20.3.3. EventBusSink ..................................................................................... 681

20.3.4. EventBusSource ................................................................................. 682

20.3.5. Filter .................................................................................................. 683

20.3.6. LogSink .............................................................................................. 684

20.3.7. Select ................................................................................................ 685

20.4. API ............................................................................................................... 686

20.4.1. Declaring a Data Flow ........................................................................ 686

20.4.2. Instantiating a Data Flow .................................................................... 687

20.4.3. Executing a Data Flow ........................................................................ 688

20.4.4. Instantiation Options ........................................................................... 689

20.4.5. Start Captive ...................................................................................... 690

20.4.6. Data Flow Punctuation With Markers ................................................... 691

20.4.7. Exception Handling ............................................................................. 691

20.5. Examples ...................................................................................................... 691

20.6. Operator Implementation ............................................................................... 693

20.6.1. Sample Operator Acting as Source ...................................................... 693

20.6.2. Sample Tokenizer Operator ................................................................. 695

20.6.3. Sample Aggregator Operator ............................................................... 696

20.6.4. Passing Operator Parameters ............................................................. 697

21. Integration and Extension ...................................................................................... 701

21.1. Overview ...................................................................................................... 701

21.2. Single-Row Function ..................................................................................... 702

21.2.1. Implementing a Single-Row Function ................................................... 702

21.2.2. Configuring the Single-Row Function Name .......................................... 703

21.2.3. Value Cache ...................................................................................... 703

21.2.4. Single-Row Functions in Filter Predicate Expressions ............................ 704

21.2.5. Single-Row Functions Taking Events as Parameters ............................. 705

21.2.6. Single-Row Functions Returning Events ............................................... 705

21.2.7. Receiving a Context Object ................................................................. 706

xix

21.2.8. Exception Handling ............................................................................. 707

21.3. Virtual Data Window ...................................................................................... 707

21.3.1. How to Use ........................................................................................ 708

21.3.2. Implementing the Forge ...................................................................... 710

21.3.3. Implementing the Factory-Factory ........................................................ 711

21.3.4. Implementing the Factory .................................................................... 712

21.3.5. Implementing the Virtual Data Window ................................................. 714

21.4. Data Window View and Derived-Value View ................................................... 715

21.4.1. Implementing a View Forge ................................................................. 716

21.4.2. Implementing a View Factory .............................................................. 717

21.4.3. Implementing a View .......................................................................... 718

21.4.4. View Contract ..................................................................................... 718

21.4.5. Configuring View Namespace and Name ............................................. 719

21.4.6. Requirement for Data Window Views ................................................... 720

21.4.7. Requirement for Derived-Value Views .................................................. 720

21.5. Aggregation Function ..................................................................................... 720

21.5.1. Aggregation Single-Function Development ........................................... 721

21.5.2. Aggregation Multi-Function Development ............................................. 728

21.6. Pattern Guard ............................................................................................... 737

21.6.1. Implementing a Guard Forge ............................................................... 738

21.6.2. Implementing a Guard Factory ............................................................ 739

21.6.3. Implementing a Guard Class ............................................................... 740

21.6.4. Configuring Guard Namespace and Name ........................................... 741

21.7. Pattern Observer ........................................................................................... 741

21.7.1. Implementing an Observer Forge ........................................................ 742

21.7.2. Implementing an Observer Factory ...................................................... 743

21.7.3. Implementing an Observer Class ......................................................... 744

21.7.4. Configuring Observer Namespace and Name ....................................... 745

21.8. Date-Time Method ......................................................................................... 745

21.8.1. Implement the DateTimeMethodForgeFactory Interface ......................... 746

21.8.2. Implement the Static Methods ............................................................. 748

21.8.3. Add the Date-Time Method Extension to the Compiler Configuration ....... 751

21.8.4. Use the new Date-Time Methods ........................................................ 751

21.9. Enumeration Method ..................................................................................... 751

21.9.1. Implement the EnumMethodForgeFactory Interface .............................. 752

21.9.2. Implement the EnumMethodState Interface .......................................... 755

21.9.3. Implement the Static Method for Processing ......................................... 756

21.9.4. Add the Enumeration Method Extension to the Compiler Configuration .... 757

21.9.5. Use the new Enumeration Method ....................................................... 757

21.9.6. Further Information to Lambda Parameters .......................................... 757

22. Examples, Tutorials, Case Studies ......................................................................... 759

22.1. Examples Overview ....................................................................................... 759

22.2. Running the Examples .................................................................................. 761

22.3. AutoID RFID Reader ..................................................................................... 762

Esper Reference

xx

22.4. Runtime Configuration ................................................................................... 762

22.5. JMS Server Shell and Client .......................................................................... 762

22.5.1. Overview ............................................................................................ 762

22.5.2. JMS Messages as Events ................................................................... 763

22.5.3. JMX for Remote Dynamic Statement Management ............................... 764

22.6. Market Data Feed Monitor ............................................................................. 764

22.6.1. Input Events ....................................................................................... 764

22.6.2. Computing Rates per Feed ................................................................. 764

22.6.3. Detecting a Fall-Off ............................................................................ 765

22.6.4. Event generator .................................................................................. 765

22.7. OHLC Plug-In Data Window .......................................................................... 765

22.8. Transaction 3-Event Challenge ...................................................................... 766

22.8.1. The Events ........................................................................................ 766

22.8.2. Combined Event ................................................................................. 766

22.8.3. Real-Time Summary Data ................................................................... 767

22.8.4. Find Problems .................................................................................... 767

22.8.5. Event Generator ................................................................................. 767

22.9. Self-Service Terminal .................................................................................... 768

22.9.1. Events ............................................................................................... 768

22.9.2. Detecting Customer Check-In Issues ................................................... 768

22.9.3. Absence of Status Events ................................................................... 769

22.9.4. Activity Summary Data ........................................................................ 769

22.9.5. Sample Application for J2EE Application Server ................................... 769

22.10. Assets Moving Across Zones - An RFID Example .......................................... 771

22.11. StockTicker ................................................................................................. 772

22.12. MatchMaker ................................................................................................ 772

22.13. Named Window Query ................................................................................. 772

22.14. Sample Virtual Data Window ........................................................................ 773

22.15. Sample Cycle Detection ............................................................................... 773

22.16. Quality of Service ........................................................................................ 773

22.17. Trivia Geeks Club ........................................................................................ 774

23. Performance ........................................................................................................... 775

23.1. Big O Notation .............................................................................................. 775

23.1.1. Big-O Complexity of Matching Events to Statements and Context

Partitions ....................................................................................