microservice monitoring

Download Microservice monitoring

Post on 12-Jul-2015

442 views

Category:

Software

6 download

Embed Size (px)

TRANSCRIPT

PowerPoint Presentation

1Microservice monitoringMarek Koniew

Reactive manifesto

0

http://www.reactivemanifesto.org/

Bulkheadshttps://github.com/Netflix/Hystrix/wiki

Cascading failures2

Circuit breaker, bulkhead pattern, cascading failures

Back pressure

4

https://github.com/Netflix/Hystrix/wikiCircuit breaker, back pressure, bulkhead pattern, cascading failures, http 429 too many requests

Hystrix9

Circuit BreakerRequest CollapsingRequest CachingMonitoring

Hystrix Dashboard12

Real time information. To store information use Graphite.

Hystrix Command Widget14

Hello World17

Migrating a Library to Hystrix19

External service call should be wrapped into a HystrixCommandConfigure thread poolsFine grained command names

Circuit breaker, Timeouts, queue size, thread pool size

Patterns and anti-patterns: naming convention22

https://wiki.hybris.com/display/prodandtech/Hystrix+commands+naming+convention+-+draftdocu-repo-v2.deleteType.dbService nameversionCommand nameExternal service

Patterns and anti-patterns: thread pools22

https://github.com/Netflix/Hystrix/wiki/Configuration#threadpool-properties

- Netflix API has 30+ of its threadpools set at 10, 2 at 20 and 1 at 25- Queue size affect request times !

Hystrix Dashboard Demo25

DEMO

Hystrix Dashboard Demo26

DEMO

DynatraceDynatrace architecturehttps://www.youtube.com/watch?v=judLcsVns-s

Memory analysis:https://www.youtube.com/watch?v=dUn4iBM6Hik28

Dynatrace31

Dynatrace35

Wiki: https://wiki.hybris.com/display/INFRA/OnDemand+performance+testing+setup

Configure java agent: java agentpath:libdtagent.so= name=app, server=dynatrace.yrd.fra.hybris.com,wait=45, storage=.

Dynatrace advantagesDirect access to agent machineMemory dumpsCPU samplingMuch moreJVM and system monitoringRequest tracing across servicesHot sensor placement class instrumentation can be changed without restart38

Dynatrace disadvantagesPerformance overheadSensor configuration affects all agentsWeak support for java 8 (although dynatrace version 6 is out)Steep learning curve41

Dynatrace patterns and anti-patternsDo NOT use it for monitoring utilize its profiling powerMuch better tools available for monitoring (Graphite, hystrix dashboard)Applications designed to be monitored does not need dynatraceOnly very simple alerts availableTurn off sensors by default to save performanceConfigure separate service instance with extended number of sensors on demandRoute part of the traffic through this service (canary testing)44

Dynatrace demo scenario47

Start application with agent.Observe allocated memory Observe GC suspension timesCreate memory dumpAnalyze memory dump

Dynatrace demo allocated memory49

Dynatrace demo GS suspentions51

Dynatrace demo application crash52

Dynatrace demo memory dump53

Dynatrace demo memory dump54

Dynatrace demo analyze memory55

Dynatrace demo root cause57

https://wiki.hybris.com/display/prodandtech/2014/08/26/Your+service+is+leaking+memory

Hystrix Dashboard Demo60

DEMO

Questions

Workshops agenda

Create hystrix command:Set command name.Set command group.Configure thread pool.Setup hystrix.stream endpoint.Setup hystrix dashboard.Connect application to dynatrace.Create dynatrace graph:Memory usage.GC suspension.Create memory dump.