opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. opentracing otoh encourages...
TRANSCRIPT
![Page 1: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/1.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
OpencensusObservability
![Page 2: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/2.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
About me
•Name: Iliia Khaprov
•Twitter: dead_trickster
•Github: deadtrickster
•Other contributions: Prometheus.erl/ex, beam-dashboards
![Page 3: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/3.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
OpencensusObservability
![Page 4: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/4.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Observability
• Metrics
• Logs
• Distributed Tracing
• Analysis
• Visualization
![Page 5: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/5.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
OpencensusA single distribution of libraries that collect metrics and distributed traces from
your services
![Page 6: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/6.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Tracing• Span
• Name• SpanId• TraceId• Start/End time• …
• TraceContext• TraceId• ParentSpantId
• Sampling
![Page 7: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/7.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Tags
Key-value pairs to provide contextual information about request.
As request propagates throughdistributed system context enrichedwith tags.
Used my stats, routing etc.
![Page 8: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/8.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Propagation
• Trace Contexthttps://www.w3.org/TR/trace-context/Defines standard headers and value format to propagate context information that enables distributed tracing scenarios.
• Correlation Contexthttps://github.com/w3c/correlation-contextDefines event correlation using trace context information across systems
![Page 9: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/9.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Stats/Metrics
• OS/Runtime metrics• Stats:
• Multidimensional (tags or labels)• Measures decoupled from aggregations via views
• Count• Distribution• Sum• LastValue
• Views integrated with TagContext
![Page 10: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/10.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
ServiceSet of components that can collect traces and metrics, do
aggregation/smart sampling and export to different backends
![Page 11: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/11.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Observability
![Page 12: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/12.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
What’s more
• Opencensus.io
• Tutorials for various languages
• Blogs
• Community resources
• Github:census-ecosystem (additional exporters, demo, etc)
![Page 13: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/13.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Opencensus & Opentracing
![Page 14: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/14.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Opencesus & Opentracing
![Page 15: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/15.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Opencensus & Opentracing
• Opencensus is more decoupled – there are separate APIs for tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything
• Opentracing leaving implementation of tracers to vendors. This might be good but often behavior is different enough. Opencensus has implementations with exporters.
• Opencensus relies on standard propagation format.
![Page 16: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/16.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Opencensus & Opentracing
Still too similar?
![Page 17: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/17.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Opencensus & Opentracing
Merge as OpenTelemetry!
![Page 18: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/18.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
OpenTelemetry
• A new, unified set of libraries and specifications for observability telemetry.
• The Java reference implementation is available, and cross-language work will began on May 8th, 2019.
• By the September 2019, the plan is to reach parity with existing projects for Erlang, C#, Golang, Java, NodeJS, and Python. Please contribute!
• When each language reaches parity, the corresponding OpenTracing and OpenCensus project will be sunset. OC and OT support will continue for two years, via a backwards compatibility bridge.
![Page 19: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/19.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
How a system becomes observable
All connections, similarities are accidental (all possible small fonts follow).
![Page 20: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/20.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
A Monolith
• Supervisor process checks if pid alive
• Prometheus OS agents
• Basic alerts: disk space, RAM, etc
• Elasticsearch-Logstash-Kibana
• Logs are useless?
![Page 21: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/21.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Microservices
![Page 22: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/22.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Microservices + Opencensus
• We are fully comfortable with metrics, and dashboards.
• We started to see execution path. And more bugs and
bottlenecks.
• When we turned on Distributed Tracing everyone was so
excited so our tracing backend couldn’t keep up with data.
• Too many boring traces!
![Page 23: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/23.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
BEAM is amazing!
![Page 24: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/24.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Conclusions so far
• Operating is hard! We tried:• Jaeger/Cassandra• Zipkin/Elasticsearch
• Sampling is hard• Adaptive sampling?• Tail sampling?
• Managing configuration and data flows is hard• oc-agent may help
• Context is awesome!
![Page 25: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/25.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
It’s not only about microservices
Dependencies are important too!• ~50 dependencies
• ~4 have instrumentation hooks
• Cowboy has middlewares
• Elixir is in a better shape
![Page 26: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/26.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Community
• Opencensus.io• Spec: github:census-instrumentation/opencensus-proto• Gitter: census-instrumentation/Lobby• Library: github:census-instrumentation/opencensus-erlang• Hex.pm: opencensus• Integrations: github:opencensus-beam• Slack: #opencensus (erlanger, elixir-lang)• Dashboards: github:deadtrickster/beam-dashboards
![Page 27: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/27.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Opencensus BEAMIntegrations
• Elli• Cowboy• SpaceTime-IoT/pgo• tsloughter/grpcbox• Plugs • Phoenix• Ecto• Absinthe• Tesla• Telemetry• logger
Exporters/Reporters• Prometheus• Stackdriver• Zipkin• Jaeger• Datadog• Opencensus Service• InfluxDB
![Page 28: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/28.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
What’s missing?
• More integrations• More exporters• ZPages (maybe observer_cli integration?)• More patterns
• Context propagation• Pools• Web servers• gen_*
• Using span fields properly• Deploying
![Page 29: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/29.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Observability WG
![Page 30: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/30.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
EEF Observability WG: Objectives
• Improve runtime observability through integration with tools like Jaeger, LightStep, Stackdriver — without vendor lock-in
• Improve state of whitebox monitoring of BEAM applications• Review possibilities to advance blackbox monitoring of BEAM (dtrace, eBPF)
• Provide common interfaces to gathering and viewing VM and application statistics and traces
• Cooperate with major OAM providers like New Relic to provide proper commercial support for Erlang monitoring
![Page 31: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/31.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Observability WG: Participants
• Opencensus
• New Relic
• Spandex
• Telemetry
• Lager
• Prometheus.erl
Thanks to Tristan Sloughter for leading the effort
![Page 32: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/32.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
An ExampleA chat app
https://github.com/opencensus-beam/opencensus-chat
![Page 33: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/33.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Chat: diagram
![Page 34: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/34.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
![Page 35: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/35.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
![Page 36: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/36.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Chat: Dependencies
![Page 37: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/37.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Chat: configuration&integration
![Page 38: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/38.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Chat: goals
Create more microservices to show more integrations! • Http servers, frameworks interfaces (Elli, Rax)• More databases (Ecto, drivers)• Different http clients• Pools, queues, etc
Deployment patterns
• Kubernetes• Service Meshes
![Page 39: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/39.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Thank you!
• Evgeny Derevianko
• Benoit Chesneau
• Tristan Sloughter
• Erik Dahmen
![Page 40: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/40.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Background designed by vexels.com
Hashtag icon made by Freepik from www.flaticon.com
Standards XKCD https://xkcd.com/927/
Twitter and Github logos are from respective official sets
Diagrams made in draw.io
![Page 41: Opencensus - codesync.global€¦ · tracing, metrics and _soon_ logs. Opentracing OTOH encourages using the same tracing API for everything •Opentracing leaving implementation](https://reader033.vdocuments.mx/reader033/viewer/2022060522/6051f0e59ef6922059475738/html5/thumbnails/41.jpg)
opencensus-beamdead_trickster CodeBEAMSTO
Questions?