how fast ist it really? benchmarking in practice
TRANSCRIPT
![Page 1: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/1.jpg)
The other day
![Page 2: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/2.jpg)
iex(1)> defmodule RepeatN do...(1)> def repeat_n(_function, 0) do...(1)> # noop...(1)> end...(1)> def repeat_n(function, 1) do...(1)> function.()...(1)> end...(1)> def repeat_n(function, count) do...(1)> function.()...(1)> repeat_n(function, count - 1)...(1)> end...(1)> end{:module, RepeatN, ...}
![Page 3: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/3.jpg)
iex(1)> defmodule RepeatN do...(1)> def repeat_n(_function, 0) do...(1)> # noop...(1)> end...(1)> def repeat_n(function, 1) do...(1)> function.()...(1)> end...(1)> def repeat_n(function, count) do...(1)> function.()...(1)> repeat_n(function, count - 1)...(1)> end...(1)> end{:module, RepeatN, ...}iex(2)> :timer.tc fn -> RepeatN.repeat_n(fn -> 0 end, 100) end{210, 0}
![Page 4: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/4.jpg)
iex(1)> defmodule RepeatN do...(1)> def repeat_n(_function, 0) do...(1)> # noop...(1)> end...(1)> def repeat_n(function, 1) do...(1)> function.()...(1)> end...(1)> def repeat_n(function, count) do...(1)> function.()...(1)> repeat_n(function, count - 1)...(1)> end...(1)> end{:module, RepeatN, ...}iex(2)> :timer.tc fn -> RepeatN.repeat_n(fn -> 0 end, 100) end{210, 0}iex(3)> list = Enum.to_list(1..100)[...]iex(4)> :timer.tc fn -> Enum.each(list, fn(_) -> 0 end) end{165, :ok}
![Page 5: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/5.jpg)
iex(1)> defmodule RepeatN do...(1)> def repeat_n(_function, 0) do...(1)> # noop...(1)> end...(1)> def repeat_n(function, 1) do...(1)> function.()...(1)> end...(1)> def repeat_n(function, count) do...(1)> function.()...(1)> repeat_n(function, count - 1)...(1)> end...(1)> end{:module, RepeatN, ...}iex(2)> :timer.tc fn -> RepeatN.repeat_n(fn -> 0 end, 100) end{210, 0}iex(3)> list = Enum.to_list(1..100)[...]iex(4)> :timer.tc fn -> Enum.each(list, fn(_) -> 0 end) end{165, :ok}iex(5)> :timer.tc fn -> Enum.each(list, fn(_) -> 0 end) end{170, :ok}iex(6)> :timer.tc fn -> Enum.each(list, fn(_) -> 0 end) end{184, :ok}
![Page 6: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/6.jpg)
Success!
![Page 7: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/7.jpg)
The End?
![Page 8: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/8.jpg)
![Page 9: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/9.jpg)
How many atrocities have we just committed?
![Page 10: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/10.jpg)
Atrocities● Way too few samples● Realistic data/multiple inputs?● No warmup● Non production environment● Does creating the list matter?● Is repeating really the bottle neck?● Repeatability?● Setup information● Running on battery● Lots of applications running
![Page 11: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/11.jpg)
n = 10_000range = 1..nlist = Enum.to_list rangefun = fn -> 0 end
Benchee.run %{ "Enum.each" => fn -> Enum.each(list, fn(_) -> fun.() end) end, "List comprehension" => fn -> for _ <- list, do: fun.() end, "Recursion" => fn -> RepeatN.repeat_n(fun, n) end}
![Page 12: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/12.jpg)
Name ips average deviation medianRecursion 6.83 K 146.41 μs ±15.76% 139.00 μsEnum.each 4.39 K 227.86 μs ±8.05% 224.00 μsList comprehension 3.13 K 319.22 μs ±16.20% 323.00 μs
Comparison: Recursion 6.83 KEnum.each 4.39 K - 1.56x slowerList comprehension 3.13 K - 2.18x slower
![Page 13: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/13.jpg)
How fast is it really? Benchmarking in Practice
Tobias Pfeiffer@PragTob
pragtob.info
![Page 14: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/14.jpg)
How fast is it really? Benchmarking in Practice
Tobias Pfeiffer@PragTob
pragtob.info
![Page 15: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/15.jpg)
![Page 17: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/17.jpg)
Concept vs Tool Usage
![Page 18: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/18.jpg)
Ruby?
![Page 19: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/19.jpg)
Profiling vs. Benchmarking
![Page 20: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/20.jpg)
Flame Graph
Elixir.Life.Board:map/2
El.. Elixir.Enum:-map/2-lc$^0/1-0-/2El.. El..El..
El.. Elixir.Life.Board:map/2Elixir.Life.Board:map/2 El..
El..
Elixir.Life.Board:map/2
Elixir.Enum:-map/2-lc$^0/1-0-/2
El..
El..Elixir.Enum:-map/2-lc$^0/1-0-/2Elixir.Enum:-map/2-lc$^0/1-0-/2
(0.47.0)
Eli..
eflame:apply1/3
Elixir.Life:run_loop/3
Elixir.Life.Board:map/2
Elixir.Enum:-map/2-lc$^0/1-0-/2
El..El.. El..
http://learningelixir.joekain.com/profiling-elixir-2/
![Page 21: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/21.jpg)
What to benchmark?
![Page 22: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/22.jpg)
● Runtime?● Memory?● Throughput?● Custom?
What to measure?
![Page 23: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/23.jpg)
The famous post
![Page 24: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/24.jpg)
What to measure?● Runtime!● Memory?● Throughput?● Custom?
![Page 25: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/25.jpg)
But, why?
![Page 26: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/26.jpg)
What's fastest?
![Page 27: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/27.jpg)
How long will this take?
![Page 28: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/28.jpg)
Enum.sort/1 performance
Name ips average deviation median10k 595.62 1.68 ms ±8.77% 1.61 ms100k 43.29 23.10 ms ±13.21% 21.50 ms1M 3.26 306.53 ms ±9.82% 291.05 ms5M 0.53 1899.00 ms ±7.94% 1834.97 ms
Comparison: 10k 595.62100k 43.29 - 13.76x slower1M 3.26 - 182.58x slower5M 0.53 - 1131.09x slower
![Page 29: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/29.jpg)
Enum.sort performance
![Page 30: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/30.jpg)
Enum.sort performance
![Page 31: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/31.jpg)
Did we make it faster?
![Page 32: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/32.jpg)
“Isn’t that the root of all evil?”
![Page 33: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/33.jpg)
“Programing Bumper Sticker”
![Page 34: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/34.jpg)
More likely, not reading the sources is the source
of all evil
![Page 35: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/35.jpg)
“We should forget about small efficiencies, say about 97% of the time: premature optimization
is the root of all evil.”Donald Knuth, 1974
(Computing Surveys, Vol 6, No 4, December 1974)
![Page 36: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/36.jpg)
“Yet we should not pass up our opportunities in that critical 3%.
A good programmer (…) will be wise to look carefully at the critical code but only after that
code has been identified.”Donald Knuth, 1974
(Computing Surveys, Vol 6, No 4, December 1974)
The very next sentence
![Page 37: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/37.jpg)
80 / 20
![Page 38: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/38.jpg)
What is critical?
Application Monitoring
![Page 39: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/39.jpg)
“In established engineering disciplines a 12 % improvement, easily obtained, is never
considered marginal; and I believe the same viewpoint should prevail in software engineering.”
Donald Knuth, 1974(Computing Surveys, Vol 6, No 4, December 1974)
Prior Paragraph
![Page 40: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/40.jpg)
“It is often a mistake to make a priori judgments about what parts of a program are really critical, since the universal experience of
programmers who have been using measurement tools has been that their intuitive guesses fail.”
Donald Knuth, 1974( Computing Surveys, Vol 6, No 4, December 1974 )
![Page 41: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/41.jpg)
What's the fastest way to sort a list of numbers
largest to smallest?
![Page 42: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/42.jpg)
list = 1..10_000 |> Enum.to_list |> Enum.shuffle
Benchee.run %{ "sort(fun)" => fn -> Enum.sort(list, &(&1 > &2)) end, "sort |> reverse" => fn -> list |> Enum.sort |> Enum.reverse end, "sort_by(-value)" => fn -> Enum.sort_by(list, fn(val) -> -val end) end}
![Page 43: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/43.jpg)
list = 1..10_000 |> Enum.to_list |> Enum.shuffle
Benchee.run %{ "sort(fun)" => fn -> Enum.sort(list, &(&1 > &2)) end, "sort |> reverse" => fn -> list |> Enum.sort |> Enum.reverse end, "sort_by(-value)" => fn -> Enum.sort_by(list, fn(val) -> -val end) end}
![Page 44: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/44.jpg)
list = 1..10_000 |> Enum.to_list |> Enum.shuffle
Benchee.run %{ "sort(fun)" => fn -> Enum.sort(list, &(&1 > &2)) end, "sort |> reverse" => fn -> list |> Enum.sort |> Enum.reverse end, "sort_by(-value)" => fn -> Enum.sort_by(list, fn(val) -> -val end) end}
![Page 45: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/45.jpg)
list = 1..10_000 |> Enum.to_list |> Enum.shuffle
Benchee.run %{ "sort(fun)" => fn -> Enum.sort(list, &(&1 > &2)) end, "sort |> reverse" => fn -> list |> Enum.sort |> Enum.reverse end, "sort_by(-value)" => fn -> Enum.sort_by(list, fn(val) -> -val end) end}
![Page 46: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/46.jpg)
Name ips average deviation mediansort |> reverse 596.54 1.68 ms ±6.83% 1.65 mssort(fun) 238.88 4.19 ms ±5.53% 4.14 mssort_by(-value) 146.86 6.81 ms ±8.68% 6.59 ms
Comparison: sort |> reverse 596.54sort(fun) 238.88 - 2.50x slowersort_by(-value) 146.86 - 4.06x slower
![Page 47: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/47.jpg)
Different types of benchmarks
![Page 48: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/48.jpg)
Feature
Integration
Unit
Testing Pyramid
![Page 49: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/49.jpg)
Application
Macro
Micro
Benchmarking Pyramid
![Page 50: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/50.jpg)
![Page 51: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/51.jpg)
Micro Macro Application
![Page 52: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/52.jpg)
Micro Macro
Components involved
Application
![Page 53: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/53.jpg)
Micro Macro
Setup Complexity
Components involved
Application
![Page 54: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/54.jpg)
Micro Macro
Setup Complexity
Execution Time
Components involved
Application
![Page 55: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/55.jpg)
Micro Macro
Setup Complexity
Execution Time
Confidence of Real Impact
Components involved
Application
![Page 56: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/56.jpg)
Micro Macro
Setup Complexity
Execution Time
Confidence of Real Impact
Components involved
Chance of Interference
Application
![Page 57: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/57.jpg)
Micro Macro
Setup Complexity
Execution Time
Confidence of Real Impact
Components involved
Chance of Interference
Golden MiddleApplication
![Page 58: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/58.jpg)
Micro Macro
Setup Complexity
Execution Time
Confidence of Real Impact
Components involved
Chance of Interference
Application
![Page 59: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/59.jpg)
Good Benchmarking
![Page 60: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/60.jpg)
What are you benchmarking for?
![Page 61: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/61.jpg)
Overly specific benchmarks & exaggerated results
![Page 62: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/62.jpg)
● Elixir 1.3.4● Erlang 19.1● i5-7200U – 2 x 2.5GHz (Up to 3.10GHz)● 8GB RAM● Linux Mint 18 - 64 bit (Ubuntu 16.04 base)● Linux Kernel 4.4.0-51
System Specification
![Page 63: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/63.jpg)
Interference free Environment
![Page 64: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/64.jpg)
[info] GET /[debug] Processing by Rumbl.PageController.index/2 Parameters: %{} Pipelines: [:browser][info] Sent 200 in 46ms[info] GET /sessions/new[debug] Processing by Rumbl.SessionController.new/2 Parameters: %{} Pipelines: [:browser][info] Sent 200 in 5ms[info] GET /users/new[debug] Processing by Rumbl.UserController.new/2 Parameters: %{} Pipelines: [:browser][info] Sent 200 in 7ms[info] POST /users[debug] Processing by Rumbl.UserController.create/2 Parameters: %{"_csrf_token" => "NUEUdRMNAiBfIHEeNwZkfA05PgAOJgAAf0ACXJqCjl7YojW+trdjdg==", "_utf8" => " ", "user" => ✓%{"name" => "asdasd", "password" => "[FILTERED]", "username" => "Homer"}} Pipelines: [:browser][debug] QUERY OK db=0.1msbegin [][debug] QUERY OK db=0.9msINSERT INTO "users" ("name","password_hash","username","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5) RETURNING "id" ["asdasd", "$2b$12$.qY/kpo0Dec7vMK1ClJoC.Lw77c3oGllX7uieZILMlFh2hFpJ3F.C", "Homer", {{2016, 12, 2}, {14, 10, 28, 0}}, {{2016, 12, 2}, {14, 10, 28, 0}}]
Logging & Friends
![Page 65: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/65.jpg)
Garbage Collection
![Page 66: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/66.jpg)
![Page 67: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/67.jpg)
Zoom in
![Page 68: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/68.jpg)
Correct & Meaningful Setup
![Page 69: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/69.jpg)
Warmup
![Page 70: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/70.jpg)
Inputs matter!
![Page 71: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/71.jpg)
Malformed inputs
![Page 72: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/72.jpg)
Where are your inputs
n = 10_000fun = fn -> 0 end
Benchee.run %{ "Enum.each" => fn -> Enum.each(Enum.to_list(1..n), fn(_) -> fun.() end) end, "List comprehension" => fn -> for _ <- Enum.to_list(1..n), do: fun.() end, "Recursion" => fn -> RepeatN.repeat_n(fun, n) end}
![Page 73: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/73.jpg)
Executed every time
n = 10_000fun = fn -> 0 end
Benchee.run %{ "Enum.each" => fn -> Enum.each(Enum.to_list(1..n), fn(_) -> fun.() end) end, "List comprehension" => fn -> for _ <- Enum.to_list(1..n), do: fun.() end, "Recursion" => fn -> RepeatN.repeat_n(fun, n) end}
![Page 74: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/74.jpg)
defmodule MyMap do def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) end
defp do_map_tco(acc, [], _function) do acc end defp do_map_tco(acc, [head | tail], func) do do_map_tco([func.(head) | acc], tail, func) end
def map_body([], _func), do: [] def map_body([head | tail], func) do [func.(head) | map_body(tail, func)] endend
TCO
![Page 75: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/75.jpg)
defmodule MyMap do def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) end
defp do_map_tco(acc, [], _function) do acc end defp do_map_tco(acc, [head | tail], func) do do_map_tco([func.(head) | acc], tail, func) end
def map_body([], _func), do: [] def map_body([head | tail], func) do [func.(head) | map_body(tail, func)] endend
TCO
![Page 76: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/76.jpg)
alias Benchee.Formatters.{Console, HTML}
map_fun = fn(i) -> i + 1 endinputs = %{ "Small (10 Thousand)" => Enum.to_list(1..10_000), "Middle (100 Thousand)" => Enum.to_list(1..100_000), "Big (1 Million)" => Enum.to_list(1..1_000_000), "Bigger (5 Million)" => Enum.to_list(1..5_000_000)}
Benchee.run %{ "tail-recursive" => fn(list) -> MyMap.map_tco(list, map_fun) end, "stdlib map" => fn(list) -> Enum.map(list, map_fun) end, "body-recursive" => fn(list) -> MyMap.map_body(list, map_fun) end}, time: 20, warmup: 10, inputs: inputs, formatters: [&Console.output/1, &HTML.output/1], html: [file: "bench/output/tco_small_sample.html"]
TCO
![Page 77: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/77.jpg)
alias Benchee.Formatters.{Console, HTML}
map_fun = fn(i) -> i + 1 endinputs = %{ "Small (10 Thousand)" => Enum.to_list(1..10_000), "Middle (100 Thousand)" => Enum.to_list(1..100_000), "Big (1 Million)" => Enum.to_list(1..1_000_000), "Bigger (5 Million)" => Enum.to_list(1..5_000_000)}
Benchee.run %{ "tail-recursive" => fn(list) -> MyMap.map_tco(list, map_fun) end, "stdlib map" => fn(list) -> Enum.map(list, map_fun) end, "body-recursive" => fn(list) -> MyMap.map_body(list, map_fun) end}, time: 20, warmup: 10, inputs: inputs, formatters: [&Console.output/1, &HTML.output/1], html: [file: "bench/output/tco_small_sample.html"]
TCO
![Page 78: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/78.jpg)
##### With input Small (10 Thousand) #####Comparison: body-recursive 5.12 Kstdlib map 5.07 K - 1.01x slowertail-recursive 4.38 K - 1.17x slower
##### With input Middle (100 Thousand) #####Comparison: body-recursive 491.16stdlib map 488.45 - 1.01x slowertail-recursive 399.08 - 1.23x slower
##### With input Big (1 Million) #####Comparison: tail-recursive 35.36body-recursive 25.69 - 1.38x slowerstdlib map 24.85 - 1.42x slower
##### With input Bigger (5 Million) #####Comparison: tail-recursive 6.93body-recursive 4.92 - 1.41x slowerstdlib map 4.87 - 1.42x slower
TCO
![Page 79: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/79.jpg)
##### With input Small (10 Thousand) #####Comparison: body-recursive 5.12 Kstdlib map 5.07 K - 1.01x slowertail-recursive 4.38 K - 1.17x slower
##### With input Middle (100 Thousand) #####Comparison: body-recursive 491.16stdlib map 488.45 - 1.01x slowertail-recursive 399.08 - 1.23x slower
##### With input Big (1 Million) #####Comparison: tail-recursive 35.36body-recursive 25.69 - 1.38x slowerstdlib map 24.85 - 1.42x slower
##### With input Bigger (5 Million) #####Comparison: tail-recursive 6.93body-recursive 4.92 - 1.41x slowerstdlib map 4.87 - 1.42x slower
TCO
![Page 80: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/80.jpg)
TCO
![Page 81: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/81.jpg)
Excursion into Statistics
![Page 82: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/82.jpg)
average = total_time / iterations
Average
![Page 83: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/83.jpg)
Why not just take the average?
![Page 84: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/84.jpg)
defp standard_deviation(samples, average, iterations) do total_variance = Enum.reduce samples, 0, fn(sample, total) -> total + :math.pow((sample - average), 2) end variance = total_variance / iterations :math.sqrt varianceend
Standard Deviation
![Page 85: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/85.jpg)
defp standard_deviation(samples, average, iterations) do total_variance = Enum.reduce samples, 0, fn(sample, total) -> total + :math.pow((sample - average), 2) end variance = total_variance / iterations :math.sqrt varianceend
Spread of Values
![Page 86: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/86.jpg)
Raw Run Times
![Page 87: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/87.jpg)
Histogram
![Page 88: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/88.jpg)
Outliers
![Page 89: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/89.jpg)
Low Standard Deviation
![Page 90: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/90.jpg)
Standard Deviation
![Page 91: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/91.jpg)
defp compute_median(run_times, iterations) do sorted = Enum.sort(run_times) middle = div(iterations, 2)
if Integer.is_odd(iterations) do sorted |> Enum.at(middle) |> to_float else (Enum.at(sorted, middle) + Enum.at(sorted, middle - 1)) / 2 endend
Median
![Page 92: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/92.jpg)
Average
![Page 93: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/93.jpg)
Median
![Page 94: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/94.jpg)
Average
![Page 95: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/95.jpg)
Median
![Page 96: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/96.jpg)
Minimum & Maximum
![Page 97: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/97.jpg)
Boxplot
![Page 98: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/98.jpg)
Surprise findings
![Page 99: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/99.jpg)
alias Benchee.Formatters.{Console, HTML}map_fun = fn(i) -> [i, i * i] end
inputs = %{ "Small" => Enum.to_list(1..200), "Medium" => Enum.to_list(1..1000), "Bigger" => Enum.to_list(1..10_000)}
Benchee.run(%{ "flat_map" => fn(list) -> Enum.flat_map(list, map_fun) end, "map.flatten" => fn(list) -> list |> Enum.map(map_fun) |> List.flatten end}, inputs: inputs, formatters: [&Console.output/1, &HTML.output/1], html: [file: "bench/output/flat_map.html"])
flat_map
![Page 100: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/100.jpg)
##### With input Medium #####Name ips average deviation medianmap.flatten 15.51 K 64.48 μs ±17.66% 63.00 μsflat_map 8.95 K 111.76 μs ±7.18% 112.00 μs
Comparison: map.flatten 15.51 Kflat_map 8.95 K - 1.73x slower
flat_map
![Page 101: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/101.jpg)
![Page 102: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/102.jpg)
![Page 103: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/103.jpg)
base_map = (0..50) |> Enum.zip(300..350) |> Enum.into(%{})
# deep maps with 6 top level conflictsorig = Map.merge base_map, some_deep_mapnew = Map.merge base_map, some_deep_map_2
simple = fn(_key, _base, override) -> override end
Benchee.run %{ "Map.merge/2" => fn -> Map.merge orig, new end, "Map.merge/3" => fn -> Map.merge orig, new, simple end,}, formatters: [&Benchee.Formatters.Console.output/1, &Benchee.Formatters.HTML.output/1], html: %{file: "bench/output/merge_3.html"}
merge/2 vs merge/3
![Page 104: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/104.jpg)
base_map = (0..50) |> Enum.zip(300..350) |> Enum.into(%{})
# deep maps with 6 top level conflictsorig = Map.merge base_map, some_deep_mapnew = Map.merge base_map, some_deep_map_2
simple = fn(_key, _base, override) -> override end
Benchee.run %{ "Map.merge/2" => fn -> Map.merge orig, new end, "Map.merge/3" => fn -> Map.merge orig, new, simple end,}, formatters: [&Benchee.Formatters.Console.output/1, &Benchee.Formatters.HTML.output/1], html: %{file: "bench/output/merge_3.html"}
merge/2 vs merge/3
![Page 105: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/105.jpg)
merge/2 vs merge/3
Is merge/3 variant about…– as fast as merge/2? (+-20%)– 2x slower than merge/2– 5x slower than merge/2– 10x slower than merge/2– 20x slower than merge/2
![Page 106: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/106.jpg)
merge/2 vs merge/3
Is merge/3 variant about…– as fast as merge/2? (+-20%)– 2x slower than merge/2– 5x slower than merge/2– 10x slower than merge/2– 20x slower than merge/2
![Page 107: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/107.jpg)
merge/2 vs merge/3
Is merge/3 variant about…– as fast as merge/2? (+-20%)– 2x slower than merge/2– 5x slower than merge/2– 10x slower than merge/2– 20x slower than merge/2
![Page 108: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/108.jpg)
merge/2 vs merge/3
Is merge/3 variant about…– as fast as merge/2? (+-20%)– 2x slower than merge/2– 5x slower than merge/2– 10x slower than merge/2– 20x slower than merge/2
![Page 109: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/109.jpg)
merge/2 vs merge/3
Is merge/3 variant about…– as fast as merge/2? (+-20%)– 2x slower than merge/2– 5x slower than merge/2– 10x slower than merge/2– 20x slower than merge/2
![Page 110: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/110.jpg)
merge/2 vs merge/3
Is merge/3 variant about…– as fast as merge/2? (+-20%)– 2x slower than merge/2– 5x slower than merge/2– 10x slower than merge/2– 20x slower than merge/2
![Page 111: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/111.jpg)
Name ips average deviation medianMap.merge/2 1.64 M 0.61 μs ±11.12% 0.61 μsMap.merge/3 0.0921 M 10.86 μs ±72.22% 10.00 μs
Comparison: Map.merge/2 1.64 MMap.merge/3 0.0921 M - 17.85x slower
merge/2 vs merge/3
![Page 112: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/112.jpg)
defmodule MyMap do def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) end
defp do_map_tco(acc, [], _function) do acc end defp do_map_tco(acc, [head | tail], function) do do_map_tco([function.(head) | acc], tail, function) end
def map_tco_arg_order(list, function) do Enum.reverse do_map_tco_arg_order(list, function, []) end
defp do_map_tco_arg_order([], _function, acc) do acc end defp do_map_tco_arg_order([head | tail], func, acc) do do_map_tco_arg_order(tail, func, [func.(head) | acc]) endend
![Page 113: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/113.jpg)
defmodule MyMap do def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) end
defp do_map_tco(acc, [], _function) do acc end defp do_map_tco(acc, [head | tail], function) do do_map_tco([function.(head) | acc], tail, function) end
def map_tco_arg_order(list, function) do Enum.reverse do_map_tco_arg_order(list, function, []) end
defp do_map_tco_arg_order([], _function, acc) do acc end defp do_map_tco_arg_order([head | tail], func, acc) do do_map_tco_arg_order(tail, func, [func.(head) | acc]) endend
![Page 114: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/114.jpg)
Does argument order make a difference?
![Page 115: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/115.jpg)
##### With input Middle (100 Thousand) #####Name ips average deviation medianstdlib map 490.02 2.04 ms ±7.76% 2.07 msbody-recursive 467.51 2.14 ms ±7.34% 2.17 mstail-rec arg-order 439.04 2.28 ms ±17.96% 2.25 mstail-recursive 402.56 2.48 ms ±16.00% 2.46 ms
Comparison: stdlib map 490.02body-recursive 467.51 - 1.05x slowertail-rec arg-order 439.04 - 1.12x slowertail-recursive 402.56 - 1.22x slower
##### With input Big (1 Million) #####Name ips average deviation mediantail-rec arg-order 39.76 25.15 ms ±10.14% 24.33 mstail-recursive 36.58 27.34 ms ±9.38% 26.41 msstdlib map 25.70 38.91 ms ±3.05% 38.58 msbody-recursive 25.04 39.94 ms ±3.04% 39.64 ms
Comparison: tail-rec arg-order 39.76tail-recursive 36.58 - 1.09x slowerstdlib map 25.70 - 1.55x slowerbody-recursive 25.04 - 1.59x slower
![Page 116: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/116.jpg)
##### With input Middle (100 Thousand) #####Name ips average deviation medianstdlib map 490.02 2.04 ms ±7.76% 2.07 msbody-recursive 467.51 2.14 ms ±7.34% 2.17 mstail-rec arg-order 439.04 2.28 ms ±17.96% 2.25 mstail-recursive 402.56 2.48 ms ±16.00% 2.46 ms
Comparison: stdlib map 490.02body-recursive 467.51 - 1.05x slowertail-rec arg-order 439.04 - 1.12x slowertail-recursive 402.56 - 1.22x slower
##### With input Big (1 Million) #####Name ips average deviation mediantail-rec arg-order 39.76 25.15 ms ±10.14% 24.33 mstail-recursive 36.58 27.34 ms ±9.38% 26.41 msstdlib map 25.70 38.91 ms ±3.05% 38.58 msbody-recursive 25.04 39.94 ms ±3.04% 39.64 ms
Comparison: tail-rec arg-order 39.76tail-recursive 36.58 - 1.09x slowerstdlib map 25.70 - 1.55x slowerbody-recursive 25.04 - 1.59x slower
![Page 117: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/117.jpg)
![Page 118: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/118.jpg)
![Page 119: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/119.jpg)
![Page 120: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/120.jpg)
![Page 121: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/121.jpg)
But… it can not be!
![Page 122: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/122.jpg)
“The order of arguments will likely matter when we generate the branching code. The order of
arguments will specially matter if performing binary matching.”José Valim, 2016
(Comment Section of my blog!)
A wild José appears!
![Page 123: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/123.jpg)
config|> Benchee.init|> Benchee.System.system|> Benchee.benchmark("job", fn -> magic end)|> Benchee.measure|> Benchee.statistics|> Benchee.Formatters.Console.output|> Benchee.Formatters.HTML.output
A transformation of inputs
![Page 125: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/125.jpg)
Always do your own benchmarks!
![Page 126: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/126.jpg)
alias Benchee.Formatters.{Console, HTML}map_fun = fn(i) -> [i, i * i] end
inputs = %{ "Small" => Enum.to_list(1..200), "Medium" => Enum.to_list(1..1000), "Bigger" => Enum.to_list(1..10_000)}
Benchee.run(%{ "flat_map" => fn(list) -> Enum.flat_map(list, map_fun) end, "map.flatten" => fn(list) -> list |> Enum.map(map_fun) |> List.flatten end}, inputs: inputs, formatters: [&Console.output/1, &HTML.output/1], html: [file: "bench/output/flat_map.html"])
Remember?
![Page 127: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/127.jpg)
● Elixir 1.4.0-rc.0● Erlang 19.1● i5-7200U – 2 x 2.5GHz (Up to 3.10GHz)● 8GB RAM● Linux Mint 18 - 64 bit (Ubuntu 16.04 base)● Linux Kernel 4.4.0-51
Mhm Upgrades
![Page 128: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/128.jpg)
alias Benchee.Formatters.{Console, HTML}map_fun = fn(i) -> [i, i * i] end
inputs = %{ "Small" => Enum.to_list(1..200), "Medium" => Enum.to_list(1..1000), "Bigger" => Enum.to_list(1..10_000)}
Benchee.run(%{ "flat_map" => fn(list) -> Enum.flat_map(list, map_fun) end, "map.flatten" => fn(list) -> list |> Enum.map(map_fun) |> List.flatten end}, inputs: inputs, formatters: [&Console.output/1, &HTML.output/1], html: [file: "bench/output/flat_map.html"])
flat_map
![Page 129: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/129.jpg)
alias Benchee.Formatters.{Console, HTML}map_fun = fn(i) -> [i, i * i] end
inputs = %{ "Small" => Enum.to_list(1..200), "Medium" => Enum.to_list(1..1000), "Bigger" => Enum.to_list(1..10_000)}
Benchee.run(%{ "flat_map" => fn(list) -> Enum.flat_map(list, map_fun) end, "map.flatten" => fn(list) -> list |> Enum.map(map_fun) |> List.flatten end}, inputs: inputs, formatters: [&Console.output/1, &HTML.output/1], html: [file: "bench/output/flat_map.html"])
flat_map
![Page 130: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/130.jpg)
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]Elixir 1.4.0-rc.0Benchmark suite executing with the following configuration:warmup: 2.0stime: 5.0sparallel: 1inputs: Bigger, Medium, SmallEstimated total run time: 42.0s
Benchmarking with input Bigger:Benchmarking flat_map...Benchmarking map.flatten...
Benchmarking with input Medium:Benchmarking flat_map...Benchmarking map.flatten...
Benchmarking with input Small:Benchmarking flat_map...Benchmarking map.flatten...
flat_map
![Page 131: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/131.jpg)
Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]Elixir 1.4.0-rc.0Benchmark suite executing with the following configuration:warmup: 2.0stime: 5.0sparallel: 1inputs: Bigger, Medium, SmallEstimated total run time: 42.0s
Benchmarking with input Bigger:Benchmarking flat_map...Benchmarking map.flatten...
Benchmarking with input Medium:Benchmarking flat_map...Benchmarking map.flatten...
Benchmarking with input Small:Benchmarking flat_map...Benchmarking map.flatten...
flat_map
![Page 132: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/132.jpg)
##### With input Bigger #####Name ips average deviation medianflat_map 1.76 K 569.47 μs ±26.95% 512.00 μsmap.flatten 1.02 K 982.57 μs ±25.06% 901.00 μs
Comparison: flat_map 1.76 Kmap.flatten 1.02 K - 1.73x slower
##### With input Medium #####Name ips average deviation medianflat_map 21.39 K 46.76 μs ±19.24% 48.00 μsmap.flatten 14.99 K 66.71 μs ±18.13% 65.00 μs
Comparison: flat_map 21.39 Kmap.flatten 14.99 K - 1.43x slower
##### With input Small #####Name ips average deviation medianflat_map 118.66 K 8.43 μs ±180.99% 8.00 μsmap.flatten 79.25 K 12.62 μs ±97.97% 12.00 μs
Comparison: flat_map 118.66 Kmap.flatten 79.25 K - 1.50x slower
The tables have turned!
![Page 133: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/133.jpg)
##### With input Bigger #####Name ips average deviation medianflat_map 1.76 K 569.47 μs ±26.95% 512.00 μsmap.flatten 1.02 K 982.57 μs ±25.06% 901.00 μs
Comparison: flat_map 1.76 Kmap.flatten 1.02 K - 1.73x slower
##### With input Medium #####Name ips average deviation medianflat_map 21.39 K 46.76 μs ±19.24% 48.00 μsmap.flatten 14.99 K 66.71 μs ±18.13% 65.00 μs
Comparison: flat_map 21.39 Kmap.flatten 14.99 K - 1.43x slower
##### With input Small #####Name ips average deviation medianflat_map 118.66 K 8.43 μs ±180.99% 8.00 μsmap.flatten 79.25 K 12.62 μs ±97.97% 12.00 μs
Comparison: flat_map 118.66 Kmap.flatten 79.25 K - 1.50x slower
> 2x faster
![Page 134: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/134.jpg)
![Page 135: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/135.jpg)
How did that happen?
![Page 136: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/136.jpg)
![Page 137: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/137.jpg)
18 minutes later...
![Page 139: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/139.jpg)
![Page 140: How fast ist it really? Benchmarking in practice](https://reader035.vdocuments.mx/reader035/viewer/2022070516/5870717d1a28ab48378b7a71/html5/thumbnails/140.jpg)
Enjoy Benchmarking! ❤
Tobias Pfeiffer@PragTob
pragtob.infogithub.com/PragTob/benchee