![Page 1: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/1.jpg)
www.erlang-solutions.com
Orchestrating Mayhem Functional Chaos Engineering
Szymon [email protected]@szymonmetelgithub.com/mentels
![Page 2: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/2.jpg)
www.erlang-solutions.com
1.Chaos: WHATWHYHOW
![Page 3: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/3.jpg)
www.erlang-solutions.com
WHAT is Chaos: complex systems
Client
A
B C
D
E
![Page 4: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/4.jpg)
www.erlang-solutions.com
WHAT is Chaos: complex systems
Client
A
B C
D
E
Interactions compounded with real-world events
may lead to unpredictable outcomes
F
![Page 5: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/5.jpg)
www.erlang-solutions.com
“Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.
~ http://principlesofchaos.org/
![Page 6: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/6.jpg)
www.erlang-solutions.com
Chaos Engineering: experimenting
Client
A
B C
D
E
F
![Page 7: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/7.jpg)
www.erlang-solutions.com
WHY
Trust Being Proactive
Costeffectiveness
What is the rationale for Chaos Engineering?
![Page 8: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/8.jpg)
www.erlang-solutions.com
HOW: Steady State
![Page 9: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/9.jpg)
www.erlang-solutions.com
HOW: 4 steps
1. Define
Steady State
Steady Statewill continue
2. Hypothesize
3. Introduce variables
4. Try to disprove hypothesis
Real-WorldEvent
LearntBehaviour
ControlGroup
ExperimentalGroup
![Page 10: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/10.jpg)
www.erlang-solutions.com
HOW: Chaos Engineering at Netflix
How API handles failure of Rating service?
Client
Gateway
Ratings
APIAPI Exp.
ChAP
API Ctrl
99%0,5%0,5%compare
![Page 11: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/11.jpg)
www.erlang-solutions.com
2.Applying Chaos Engineering Principles
![Page 12: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/12.jpg)
www.erlang-solutions.com
Applying ChE: Injection Points
MongooseIM(XMPP)
Client
Client
MongooseIM(XMPP)
Erlang Distribution
Authentication Service
HTTP
Message Archive Database
Native protocol
Streaming Service
Native protocol
![Page 13: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/13.jpg)
www.erlang-solutions.com
Applying ChE: Comparing
Scenario
Load Generator MongooseIM
Load Generator MongooseIM
Control Load Test
Experimental Load Test with Hypothesis
Fault Injection
Compare
![Page 14: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/14.jpg)
www.erlang-solutions.com
Hypothesis 1: Database Failure
Failure to write to the database won’t disrupt the service
Pool Supervisor
Worker Worker Worker
Message Archive (DB)
ErlangVM
![Page 15: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/15.jpg)
www.erlang-solutions.com
Hypothesis 1: Database Failure
# setup
kill_pool = fn f ->
workers = Supervisor.which_children(PoolSup)
Enum.each(workers,
fn pid -> Process.exit(pid, :chaos) end)
Process.sleep(10)
f.(f)
end
# run
killer = spawn(fn -> kill_pool.(kill_pool) end)
# stop
Process.exit(killer, :kill)
![Page 16: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/16.jpg)
www.erlang-solutions.com
Hypothesis 1: Database Failure
Chaos Started Chaos Stopped
Request Response Time
![Page 17: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/17.jpg)
www.erlang-solutions.com
Hypothesis 2: Slow Network
Delay on the connection to the Authentication Service won’t prevent users from logging in
Auth Pool Supervisor
Worker Worker Worker
ErlangVM
Authentication Service
![Page 18: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/18.jpg)
www.erlang-solutions.com
Hypothesis 2: Slow Network
# setup
delay = 100
delayed_auth = fn user, pass ->
Process.sleep(delay)
AuthService.authenticate(user, pass)
end
:ok = :meck.new(AuthService, [:passthrough])
# run
:ok = :meck.expect(AuthService, :authenticate,
fn user, pass -> delayed_auth(user, pass) end)
# stop
:ok = :meck.unload(AuthService)
![Page 19: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/19.jpg)
www.erlang-solutions.com
Hypothesis 2: Slow Network
Session Count
Not affected nodesThe affected node
![Page 20: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/20.jpg)
www.erlang-solutions.com
Hypothesis 3: Network Glitch
Network glitch on the connection to Kafka won’t cause any data loss
Kafka Server
Kafka Service
IP Tables
MongooseIMMongooseIM server
![Page 21: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/21.jpg)
www.erlang-solutions.com
Hypothesis 3: Network Glitch
# setup
cmd = ["INPUT -m state --state NEW,ESTABLISHED,RELATED -p
tcp --dport 9092 -s #{mim_addr} -j DROP"]
enable_cmd = "-A" <> cmd
disable_cmd = "-D" <> cmd
# run
{_, 0} = System.cmd("iptables", enable_cmd)
# stop
{_, 0} = System.cmd("iptables", disable_cmd)
![Page 22: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/22.jpg)
www.erlang-solutions.com
M
Hypothesis 3: Network Glitch
Session Count
Application Retries
Network Recovery
M M M MKafka Driver Queue
MMM
![Page 23: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/23.jpg)
www.erlang-solutions.com
Fault Injection: no recompilation
▸ RPC
▸ Remote processes
Direct or Remote
Control Node
Affected Node
:rpc.call(:aff_node@localhost, PoolSup, :which_children, [])
Node.spawn(:aff_node, fn -> … end)
![Page 24: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/24.jpg)
www.erlang-solutions.com
3.MongooseIM Chaos Engineering Automation
![Page 25: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/25.jpg)
www.erlang-solutions.com
Automation of Chaos Engineering
Scenario
Load Generator MongooseIM
Load Generator MongooseIM
Control Load Test
Experimental Load Test with Hypothesis
Fault Injection
Compare
Chaos Experiment
Chaos Automation (Typhoon)
![Page 26: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/26.jpg)
www.erlang-solutions.com
Typhoon: Elixir Application
apps
├── typhoon
│ └── lib
│ └── typhoon
│ ├── chaos_manager
│ ├── fault_injection
│ └── infrastructure
└── typhoon_web
└── lib
└── typhoon_web
![Page 27: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/27.jpg)
www.erlang-solutions.com
Typhoon: Elixir Application
Fault Injection
Infrastructure
Chaos Manager
%TestSetup{}%TestTopology{}
Fault Protocol%MyFault{}
%Experiment{ :control_test_id, :experimental_test_id, :setup, :faults, :faults_offsets,}
![Page 28: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/28.jpg)
www.erlang-solutions.com
Load Testing Infrastructure
Typhoon: Elixir Application
Fault InjectionInfrastructure
Chaos Manager
Load Test Config: … Faults: …
%Topology{}[faults]
TyphoonWeb
%Experiment{}1. %Setup{} (Control Test)
2. %Setup{} (Exp. Test)
Steady State Learnt BehaviourCompare
![Page 29: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/29.jpg)
www.erlang-solutions.com
Typhoon: Fault Injection
defprotocol FaultInjection.Fault do
@doc "Applies the fault to the load test run by `test_id`"
@spec apply(struct(), test_id()) :: :ok | {:error, term()}
def apply(fault, test_id)
end
defimpl FaultInjection.Fault, for: MyFault do
def apply(fault, test_id), do: send self, %{fault: MyFault}
end
FaultInjection.Fault.apply(%MyFault{}, "test_id")
![Page 30: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/30.jpg)
www.erlang-solutions.com
Typhoon: Fault Injection
defmodule FaultInjection.Fault.MyFault do
embedded_schema do
field(:param1, :integer)
field(:param2, :string)
end
def changeset(struct, attrs) do
struct
|> cast(attrs, [:param1, :param2])
|> validate_required([:param1, :param2])
end
defimpl FaultInjection.Fault do
def apply(fault, test_id), do: send self, %{fault: MyFault}
end
end
![Page 31: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/31.jpg)
www.erlang-solutions.com
CHAOS ENGINEERING
APPLY
AUTOMATE
it to your system using the most basic techniques available
is for everyone - go and explore it
if it works for you add it to your continuous integration pipeline
![Page 32: Orchestrating Mayhem Functional...Chaos Engineering Principles . Applying ChE: Injection Points MongooseIM Client (XMPP) Client MongooseIM Erlang (XMPP) Distribution Authentication](https://reader033.vdocuments.mx/reader033/viewer/2022051605/60084519cfd9fa43fa6f6f27/html5/thumbnails/32.jpg)
www.erlang-solutions.com
THANK YOU!
[email protected]@szymonmetel
github.com/mentelsmedium.com/@mentels