Download - Yapc eu 2011_perlformance-net
![Page 1: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/1.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Benchmark::Perl::Formance for the masses
Ste�en "renormalist" Schwigon
Dresden Perl Mongers
16 August 2011
![Page 2: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/2.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Abstract
Motivation
Measuring
Workloads
Perl::Formance
Challenges
Infrastructure
Summary
![Page 3: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/3.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Abstract
Abstract
![Page 4: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/4.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Abstract
Perl lacks benchmarking
I work on a full-coverage solution
benchmark implementation
complete "build - execute - evaluate" lifecycle
benchmark result databasegraphically present numbers and details
Present the vision and state
![Page 5: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/5.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Motivation
Motivation
![Page 6: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/6.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl standard benchmark suite?
![Page 7: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/7.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl standard benchmark suite?
![Page 8: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/8.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl standard benchmark suite?
![Page 9: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/9.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Measuring
Measuring
![Page 10: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/10.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
The most important benchmark question
What do you want to measure?
Primary goal: perl5, the interpreter
real world workloadsmicro benchmarksusethreads vs. no usethreadsuse64bitall vs. no use64bitall
Secondary goal: misc Perl5 topics
subs vs. methodsMooseregex engines
![Page 11: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/11.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Workloads
Workloads
![Page 12: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/12.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Where is Perl?
Everywhere and nowhere
Benchmarks are a corporate topic
Perl's usual corporate visibility problem
![Page 13: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/13.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
"The One Workload"
No dedicated single heavy applications
Except SpamAssassin
important app
heavy for a reason
hit by 5.12 deprecations
handholding on new CPAN release
![Page 14: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/14.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Self-referential - Perl for Perl
Refer to Perl itself
CPAN.pm
Perl::CriticPerl6 STD.pm (gimme5, viv)
![Page 15: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/15.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Nothing else!
Not much more
![Page 16: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/16.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Writing my own benchmark suite
Goals
Have benchmark workloads
"server" workloads (MEM! + TIME!)
"desktop" workloads (small + fast)
"micro" benchmarks (language features)
"multi purpose" workloads (average, not �tting in others)
Provide general benchmarking umbrella
![Page 17: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/17.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl::Formance
Perl::Formance
![Page 18: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/18.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
The Perl::Formance benchmark suite
Collect existing heavy code
Collect existing benchmark snippets
Write new benchmark code from scratch
Measure execution time ("smaller is better")
http://xrl.us/cpanperlformance
![Page 19: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/19.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl::Formance features (1)
Heavy workloads
The one: SpamAssassin
Type: "server"
Example corpus from spamassassin.org
Run sa-learn
![Page 20: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/20.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl::Formance features (2)
Wrap existing benchmarks
"Language Shootout" on alioth.debian.org
Type: "multi purpose"
binarytrees
fastanbody
pidigits
regexdna
revcomp
spectralnorm
fannkuch (!)
knucleotide (!)
mandelbrot (!)
Mostly copy'n'paste into Perl::Formance plugins
![Page 21: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/21.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl::Formance features (3)
Known heavy code from CPAN
Heavy test suites
Type: "desktop"
Moose test suiteRegexp::Common
Perl6 STD.pm
Type: "desktop"
gimme5 STD.pm6
viv STD.pm6
Data::DPath
![Page 22: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/22.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl::Formance features (4)
Stress of language features
Stress recursion (use Fibonacci numbers as vehicle)
Type: "micro"
subs - plain Perl
methods - plain Perl
methods - Moosemethods - MooseX::Declare
![Page 23: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/23.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl::Formance features (5)
Pathological Regular Expressions
Type: "micro"
Known pathological issues
my $re = ("a?" x $n) . ("a" x $n);
See
http://swtch.com/~rsc/regexp/regexp1.html
![Page 24: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/24.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Perl::Formance features (6)
Compare di�erent regex engines
Pluggable regex engines since Perl 5.10
Again pathological regular expressions
Type: "micro"
Regex engines
nativePOSIX::Regex
re::engine::Plan9
re::engine::PCRE
re::engine::Lua
re::engine::LPeg (confusion bonus)
re::engine::Oniguruma
![Page 25: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/25.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Di�erent output styles (1)
Human readable
$ benchmark-perlformance --fastmode [...]
Rx.regexes.fieldsplit1 : 1.267907
Rx.regexes.fieldsplit2 : 2.106220
Rx.regexes.pathological : 1.000129
Shootout.binarytrees : 1.046751
Shootout.fasta : 2.270553
Shootout.nbody : 1.685537
Shootout.spectralnorm : 1.855935
![Page 26: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/26.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Di�erent output styles (2)Evaluation friendly (�outstyle=yaml)
---
perlformance:
config:
fastmode: 1
results:
Shootout:
fasta:
Benchmark:
- 0.263123989105225
- 0.25
- 0
- 0
- 1
count: 1
goal: 5000
...
![Page 27: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/27.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Output style
Augment YAML with surrounding TAP
--tapdescription="some description"
Additional Codespeed data structure
--codespeed
Why TAP?
TAP is my hammer
Later embed into Tapper infrastructure
![Page 28: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/28.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Plugin API
Very lax, just namespace + sub main()
Benchmark::Perl::Formance::Plugin::Skeleton
Benchmark::Perl::Formance::Plugin::*
sub main($options) {
...
return { Benchmark => $t,
...
};
}
![Page 29: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/29.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Challenges
Challenges
![Page 30: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/30.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Automatic building
![Page 31: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/31.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Automatic building
sigh
![Page 32: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/32.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Automatic building
![Page 33: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/33.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Automatic building
Working throughout Perl's git history (5.8..blead)
Cherry-pick �xes from the future (5.8.x)
Inconsistent version tags
perlbrew?
No git
No bootstrapping CPAN, distroprefs (ANDK++), etc.
![Page 34: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/34.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
The Art of Benchmarking (1)
Stable benchmark environment
Perl <-> OS interaction
Flush CachesAddress Space Layout Randomisation (ASLR)
Disable Core Performance Boost
Provide stable set of CPAN dependencies
Own CPAN mirrorOnly sync once in a time
Stable execution environment
Machine, Memory, Harddisk
Operating System
Compile toolchain
Lazily load benchmark plugins+dependencies after fork
![Page 35: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/35.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
The Art of Benchmarking (2)
I/O everywhere
depends on the workload
SPECcpu 2006 (Perl 5.8.7) avoids I/O
I do not �ght that battle
![Page 36: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/36.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Infrastructure
CPAN module obviously not enough
I should solve all other challenges, too
Set up complete infrastructure
![Page 37: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/37.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Infrastructure
Infrastructure
![Page 38: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/38.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Infrastructure
perl64.org - Benchmark machine
Rent dedicated machine6-core AMD Opteron 4180
Without any running services
No private data, for relaxed access
perlformance.net - "Tapper" application
speed.perlformance.net - "Codespeed" application
![Page 39: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/39.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Single Run
Builds Perl
from git
inject CPAN toolchain
Perl 5.8 .. blead
Installs Benchmark::Perl::Formance
Installs Tapper::TestSuite::Benchmark::Perl::Formance
Runs tapper-testsuite-benchmark-perl-formance
runs benchmark-perlformance
�outstyle=yaml
�tapdescription="perlformance"
�codespeed
adds meta information for Tapper
sends report to Tapper server
![Page 40: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/40.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Multiple runs
Tapper
![Page 41: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/41.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Tapper
Test infrastructure, open sourced by AMD in 2011
Overview:
http://developer.amd.com/zones/opensource/AMDTapper
Source:
http://github.com/amd
Productized:
Starterkit/Deployment, Docs, Wiki
![Page 42: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/42.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Tapper
Why Tapper?
Tapper == TAP database + automation + scheduler
Query API for detailed data forensics
see my YAPC::EU 2009 presentation
"Cinderella 'TAP - The lazy evaluation sisters of TAP::Parser"
http://xrl.us/lazytap (PDF)
![Page 43: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/43.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Tapper
Why Tapper?
Schedule / intermix di�erent use-cases
auto-rerun for blead/threads
auto-rerun for blead/nothreads
speci�c commits
completely di�erent stu� I add later (Perl6 benchmarks?,smoke tests?)
Have bandwidth ratio for each,
eg. more non-threaded than threaded
Advanced automation support, timeout handling, etc.
Frontends to start single runs
cmdline + webGUI
with requested options (�commitid, �threads, �64bit)
Pass-through data chunks to Codespeed application
![Page 44: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/44.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Codespeed application
http://github.com/tobami/codespeed
Web application
Render benchmark graphs
Show commit/meta information
Comparison graphs, baselines, etc.
Can handle git repos
![Page 45: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/45.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Own CPAN mirror
http://perlformance.net/CPAN
To guarantee no unexpected changes
slower/faster/break
![Page 46: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/46.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Summary
Summary
![Page 47: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/47.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Summary
I have benchmarks
I have automatic Perl + CPAN setup
I have applications for complete infrastructure
I have dedicated benchmarking server
I have dedicated CPAN mirror
Polished in many details
TODO :
1. Re-setup after server went mental
2. Allow users to request benchmark runs
![Page 48: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/48.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Bonus Screenshots
Bonus Screenshots
![Page 49: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/49.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Tapper at http://perlformance.net
![Page 50: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/50.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Tapper - Overview lists
![Page 51: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/51.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Tapper - Metainfo
![Page 52: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/52.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Tapper - Metainfo
![Page 53: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/53.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Tapper - Tracked data
![Page 54: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/54.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Codespeed at http://speed.perlformance.net
![Page 55: Yapc eu 2011_perlformance-net](https://reader033.vdocuments.mx/reader033/viewer/2022052823/55583242d8b42acb078b4608/html5/thumbnails/55.jpg)
Abstract Motivation Measuring Workloads Perl::Formance Challenges Infrastructure Summary
Thanks.
Thanks.