![Page 1: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/1.jpg)
DISTRIBUTED TRACINGTAKING
TO THE NEXT LEVEL
![Page 3: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/3.jpg)
WHAT IS THIS TALK ABOUT
▸ The Fundamental Concepts
▸ The Current State
▸ The Next Level
TAKING DISTRIBUTED TRACING TO THE NEXT LEVEL
![Page 4: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/4.jpg)
LOGGINGMETRICS TRACING
OBSERVABILITY
![Page 5: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/5.jpg)
LOGGING
![Page 6: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/6.jpg)
LOGGING
![Page 7: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/7.jpg)
![Page 8: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/8.jpg)
![Page 9: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/9.jpg)
THERE IS ALWAYS AN ALTERNATIVE ;)
![Page 10: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/10.jpg)
LOGGING
![Page 11: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/11.jpg)
LOGGING
![Page 12: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/12.jpg)
LOGGING - MDC
Source: https://www.baeldung.com/mdc-in-log4j-2-logback
![Page 13: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/13.jpg)
TRACEABILITY OF SOFTWARE
MONOLITHIC ARCHITECTURE
![Page 14: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/14.jpg)
TRACEABILITY OF SOFTWARE
MICROSERVICES
![Page 15: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/15.jpg)
TRACEABILITY OF SOFTWARE
MDC CONTEXT PROPAGATION
TK TK TK
TKTK
TK
TK
TK TK
![Page 16: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/16.jpg)
DISTRIBUTED TRACING
![Page 17: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/17.jpg)
DISTRIBUTED TRACING
BRINGING EVENTS IN
CAUSAL ORDER IN
DISTRIBUTED ENVIRONMENT
![Page 18: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/18.jpg)
DISTRIBUTED TRACING
Span TracerTrace Context(span id, trace id)
Trace
![Page 19: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/19.jpg)
DISTRIBUTED TRACING
WHY DISTRIBUTED TRACING?
▸ Latency visualization
▸ Service dependency visualization
▸ Hidden architecture revealed
▸ Error analysis
▸ Infrastructure check
▸ Version check
![Page 20: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/20.jpg)
▸ Trace Context propagation
▸ in-process
▸ inter-process
▸ Clock skew / synchronization
▸ Performance and stability
▸ Data ingestion, storage, retention
▸ GDPA
DISTRIBUTED TRACING
PROBLEMS OF DISTRIBUTED TRACING
SERVICE B
SERVICE A
Latency measurement points
Host 1
Host 2
![Page 21: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/21.jpg)
CORRELATE LOG ENTRIES
TRACE COMMUNICATION BETWEEN SERVICES
USE DISTRIBUTED TRACING
USE TRACE-ID IN THE MDC
![Page 22: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/22.jpg)
THE CURRENT STATE OF DISTRIBUTED TRACING
![Page 23: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/23.jpg)
BOOKING PAYMENT
USER INTERFACE Live Reporting(Performance monitoring)
OFFER
DATASTORE BigData Store(High-Performance / Indexing)
WEBS
HOP
Context propagation(Service Instrumentation)
Common language(Specification / Trace-Context)
COLLECTOR Decoupled Reporter(Asynchronous collecting)
![Page 24: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/24.jpg)
APM - PROVIDERSSUPPORT FOR OPENTRACING ENABLES
![Page 25: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/25.jpg)
TOOLING LANDSCAPE
![Page 26: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/26.jpg)
GOAL: MAKE OBSERVABILITY EASY FOR MODERN APPLICATIONS
libraries used for collecting tracing and metrics dataready to use implementation for database engines, kubernetes etc.tightly coupled to the implementationsopinionated implementation for capturing observability signals
COMPLEMENTARY NOT CONTRADICTORY
vendor-neutral API with default implementations
+ =
vendor-neutral API wide vendor supporttracer implantation effort
![Page 27: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/27.jpg)
DEMO
https://github.com/trasiercom/springboot-example/branches
![Page 28: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/28.jpg)
TAKING DISTRIBUTED TRACING
TO THE NEXT LEVEL
![Page 29: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/29.jpg)
DISTRIBUTED TRACING
![Page 30: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/30.jpg)
DISTRIBUTED TRACING
credits to jaeger documentation
![Page 31: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/31.jpg)
THE NEXT LEVEL
BUSINESS TRACING EASY BUG TRIAGE
BUSINESS ANALYSIS PREDICTIVE ANALYSIS
MOCK SERVICES REPLAY CONVERSATIONS
![Page 32: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/32.jpg)
BUSINESS TRACING
CONVERSATION AS A WHOLE
![Page 33: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/33.jpg)
CUSTOMER CHARGED TOO MUCH?
INSPECT PAYLOAD OF TRANSACTIONS
![Page 34: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/34.jpg)
ON-THE-FLY ERROR DETECTION
https://ui.trasier.com/#/account/170522/space/nova-test/search/id/267cd7b7-2d6a-474b-ade2-bf0453881f72?spanId=f53e4982-a7c0-4117-b98c-91260eefbdbb&view=editor&editortab=response
![Page 35: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/35.jpg)
DEMO
https://github.com/trasiercom/springboot-example/branches
![Page 36: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/36.jpg)
THE NEXT LEVEL
EASY BUG TRIAGE
https://ui.trasier.com/#/account/123456/space/sys-test/search/id/267cd7b7-2d6a-474b-ade2-bf0453881f72
Allocating bugs to the responsible team can be tricks when there is only a functional description.
![Page 37: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/37.jpg)
THE NEXT LEVEL
MOCK SERVICES
OFFER THIRD PARTY SERVICE
TEST
- CL
IENT X-CID=ABC123
HEADER
Testing can be hard when you’re integrating with many services. Especially unreliable integration systems can increase the development costs unexpectedly.
TEST 1 | MOCK ID=ABC123
BUSINESS TRACING BACKEND
![Page 38: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/38.jpg)
THE NEXT LEVEL
MOCK SERVICESTesting can be hard when you’re integrating with many services. Especially unreliable integration systems can increase the development costs unexpectedly.
OFFER THIRD PARTY SERVICE
TEST
- CL
IENT MODE=READ
HEADER
X-CID=ABC123
TEST 1 | MOCK ID=ABC123
BUSINESS TRACING BACKEND
![Page 39: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/39.jpg)
THE NEXT LEVEL
MOCK SERVICES
Change on the integrator side (Bug, Error, API Change)
Probably bug in our system
Run tests against real backend and against mock at the same time
Test against mock50 (100%) SUCCESS, 0 (0%) ERROR
Test against real backend40 (80%) SUCCESS, 10 (20%) ERROR
Test against real backend40 (80%) SUCCESS, 10 (20%) ERRORTest against mock40 (80%) SUCCESS, 10 (20%) ERROR
![Page 40: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/40.jpg)
THE NEXT LEVEL
REPLAY CONVERSATIONS
OFFER THIRD PARTY SERVICE REPL
AY -
CLIEN
T
BUSINESS TRACING BACKEND
MODE=REPLAYHEADER
X-CID=ABC123
![Page 41: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/41.jpg)
THE NEXT LEVEL
BUSINESS ANALYSISThe data stored by the business tracing system is the perfect foundation for extensive research and data analysis. Gathering the equivalent information from your existing data stores is expensive and may not even be possible.
![Page 42: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/42.jpg)
THE NEXT LEVEL
BUSINESS ANALYSISThe data stored by the business tracing system is the perfect foundation for extensive research and data analysis. Gathering the equivalent information from your existing data stores is expensive and may not even be possible.
![Page 43: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/43.jpg)
THE NEXT LEVEL
PREDICTIVE ANALYSISThe chance is to finally take the step to predictive analysis on your actual business data and processes.
Offer creation errors during the day per product id.
Offer creation errors during the day.
![Page 44: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/44.jpg)
BUSINESS OBSERVABILITY IS AS IMPORTANT AS TECHNICAL TRACING
![Page 46: TAKING DISTRIBUTED TRACING - JAX London · 2019-11-05 · taking distributed tracing to the next level. metrics logging tracing. observability. logging . ... tooling landscape. goal:](https://reader034.vdocuments.mx/reader034/viewer/2022042414/5f2e4680852dec466934ad16/html5/thumbnails/46.jpg)
▸ OpenTracing
▸ https://opentracing.io
▸ OpenCensus
▸ https://opencensus.io
▸ OpenTelemetry
▸ https://opentelemetry.io
▸ Zipkin
▸ https://zipkin.io
▸ Jaeger
▸ https://jaegertracing.io
▸ Trasier
▸ https://trasier.com
THANK YOU