index activiti data on elasticsearch
TRANSCRIPT
Index Activiti data on Elasticsearch
Activiti User Day Paris 2015
Silvio dos Passos NetoCTO at iColabora
@silvioneto
“Don’t bridge the business-‐IT divide. Obliterate it!” (2003)
Smith & Fingar
?
?
@mike_dias
The big table problem
ID_ NAME_ VALUE_* …ACT_HI_VARINST
User form
ProcessInstance
ID_ NAME_ VALUE_* …
1 client_name Jonh …
2 client_tel 123456 …
3 due_date 01/06/2015 …4 demand_desc I have a problem… …
ACT_HI_VARINST
User form
ProcessInstance
ProcessInstance
User form
ID_ NAME_ VALUE_* …
1 client_name Jonh …
2 client_tel 123456 …
3 due_date 01/06/2015 …4 demand_desc I have a problem… …5 client_name Bob …6 client_tel 654321 …7 due_date 10/06/2015 …8 demand_desc My internet conn… …
ACT_HI_VARINST
85 fields x
~1000 Process per day =
~85.000 variables per day
~15 million variables
in 9 months
The Tool
Built on top of
Analytics
Distributed
Indexing the data
Historic Data
P
P PP
PP
PP
PP
P P PP
P P
PPP
Process Lake
P
P PP
PP
PP
PP
P P PP
P P
PPP
P
P
PP
P
PP
P
PP
P
P
CPU 1 CPU 2 CPU 3 CPU 4
Process Lake
P
P
P
CPU 1
V V
V V
V V
V V
VariablesT
T
T
Tasks
P
Process
{ }P V V V
V V V V
V T T T
JSON
REST API
Real-Time Data
E E E E
E E E E
E E E E
E E E E
E E E E
E E E E
E E E E
E E E E
Engine Events
E
E
E
E
E
E
E
E
E
E
E
E
E E E E
E E E E
E E E E
E E E E
E E E E
E E E E
E E E E
E E E E
Engine Events
E
E
E
E
E
E
E
E
E
E
E
E
E EE
E EE{ }JSON
REST API
Listeners
Playing with the data
Search
{ "query":{ "path":"variables", "nested":{ "query":{ "match":{ "text":"João Silva" } } } }}
Search results
Compare
SELECT * FROM ACT_HI_VARINSTWHERE NAME_ = 'passport'AND TEXT_ = '1234'
{ "filter":{ "nested":{ "path":"variables", "filter":{ "bool":{ "must":[ { “term": { "name":"passport" }}, { “term": { "text":"1234" }} ] } } } }}
Response Time
0 secs
45 secs
90 secs
135 secs
180 secs
MySQL Elasticsearch
0,08 secs
161 secs
Response Time
0 secs
45 secs
90 secs
135 secs
180 secs
MySQL Elasticsearch
0,08 secs
161 secs
CENSU
RED
Lessons learned
Full text search is a helpful feature
Reduce MySQL workload
ES is great for analytics
Next steps
Apache Spark
Lightning-Fast Cluster Computing
Java EE dependency
Open source
Thank you!@mike_dias @silvioneto
Questions?@mike_dias @silvioneto