perl in testing
TRANSCRIPT
-
8/8/2019 Perl in Testing
1/89
Automatic Testing with Perl
Gabor Szabo
-
8/8/2019 Perl in Testing
2/89
Automatic Testing with Perl
by Gabor Szabo
1.05 Edition
Published Mon May 28 07:11:15 2007
Copyright 2004, 2005, 2006, 2007 PTI Ltd. Perl Training Israelhttp://www.pti.co.il/
-
8/8/2019 Perl in Testing
3/89
Table of Contents1. About Perl Training Israel .................................................................................................................... 1
2. Introduction............................................................................................................................................ 2
2.1. Self Introduction - Who am I ? ................................................................................................... 2
2.2. Self Introduction - Who are you ?............................................................................................... 2
3. Preface .................................................................................................................................................... 3
3.1. Objectives.................................................................................................................................... 3
3.2. Plan of the seminar...................................................................................................................... 3
4. Manual testing........................................................................................................................................ 4
4.1. Manual testing............................................................................................................................. 4
4.2. Web site testing ........................................................................................................................... 4
4.3. CLI testing................................................................................................................................... 4
4.4. Database testing .......................................................................................................................... 4
4.5. GUI testing .................................................................................................................................. 5
5. Basic Testing Framework in Perl ......................................................................................................... 6
5.1. Testing a simple command line tool............................................................................................ 65.2. Calculator test ............................................................................................................................. 6
5.3. Error ! .......................................................................................................................................... 6
5.4. Calculator test with expected results........................................................................................... 7
5.5. More difficult output ................................................................................................................... 7
5.6. Print only ok/not ok..................................................................................................................... 7
5.7. Write the ok function .................................................................................................................. 8
5.8. Introducing Test::Simple............................................................................................................. 9
5.9. Add names to the tests .............................................................................................................. 10
5.10. Enlarge our test suit ................................................................................................................. 10
5.11. Load Test::Simple at run time ................................................................................................. 11
5.12. Forget about your "plan", use "no_plan"................................................................................. 12
5.13. Put the test cases in an external file......................................................................................... 135.14. Harness.................................................................................................................................... 14
5.15. Move external call into function ............................................................................................. 15
5.16. Exercises: MyCalc .................................................................................................................. 15
5.17. Solution: MyCalc .................................................................................................................... 15
5.18. Test::Simple............................................................................................................................. 17
6. Test::More ............................................................................................................................................ 18
6.1. Moving over to Test::More ....................................................................................................... 18
6.2. Test::More ok( trueness, name);................................................................................................ 18
6.3. Test::More is( value, expected_value, name); ........................................................................... 19
6.4. diag(just_a_message ); .............................................................................................................. 19
6.5. like(value, qr/expected regex/, name); ...................................................................................... 20
6.6. cmp_ok( this, op, that, name);................................................................................................... 20
6.7. is_deeply( complex_structure, expected_complex structure, name); ....................................... 21
6.8. TODO........................................................................................................................................ 22
6.9. TODO with Harness.................................................................................................................. 23
6.10. Platform dependent tests ......................................................................................................... 23
6.11. SKIP some tests ...................................................................................................................... 24
iii
-
8/8/2019 Perl in Testing
4/89
6.12. My own test functions ............................................................................................................. 24
6.13. My own test functions - improved .......................................................................................... 25
6.14. Create a test module ................................................................................................................ 25
6.15. Test::Builder............................................................................................................................ 26
6.16. Number of tests ....................................................................................................................... 27
6.17. Early Abnormal Exit ............................................................................................................... 27
6.18. Less than planned tests - Early Normal Exit ........................................................................... 286.19. More tests than planned .......................................................................................................... 29
6.20. Multiply................................................................................................................................... 29
6.21. Error in the test........................................................................................................................ 30
6.22. Multiply - fixed ....................................................................................................................... 30
6.23. Multiple expected values......................................................................................................... 30
6.24. Exercises ................................................................................................................................. 31
7. Command line application .................................................................................................................. 33
7.1. bc - An arbitrary precision calculator language ........................................................................ 33
7.2. Normal operation ...................................................................................................................... 33
7.3. Expect.pm ................................................................................................................................. 33
7.4. Simple computation - adding two values .................................................................................. 337.5. Results ....................................................................................................................................... 34
7.6. Simple computation - separate send commands ....................................................................... 34
7.7. Simple computation - is it really working ? .............................................................................. 35
7.8. Results ....................................................................................................................................... 35
7.9. Reduce output ........................................................................................................................... 36
7.10. Output...................................................................................................................................... 36
7.11. More than one test ................................................................................................................... 36
7.12. Output...................................................................................................................................... 37
7.13. External test file ...................................................................................................................... 37
7.14. Random regression tests.......................................................................................................... 38
7.15. Random and regression testing ............................................................................................... 39
7.16. Random and regression testing - slight improvement ............................................................. 407.17. Results ..................................................................................................................................... 41
7.18. Recording session ................................................................................................................... 41
7.19. Capturing both STDOUT and STDERR ................................................................................. 41
7.20. Capturing both STDOUT and STDERR manually ................................................................. 42
7.21. Capturing both STDOUT and STDERR using IPC::Run3 ..................................................... 42
8. Networking devices .............................................................................................................................. 44
8.1. Introduction - pick the right abstraction level ........................................................................... 44
8.2. Socket level programming using Socket.pm ............................................................................. 44
8.3. Socket level programming using IO::Socket............................................................................. 45
8.4. Newline ..................................................................................................................................... 46
8.5. Net::Telnet................................................................................................................................. 468.6. Net::Telnet for HTTP ................................................................................................................ 46
8.7. Net::Telnet configure VLAN..................................................................................................... 47
8.8. ftp using Net::FTP..................................................................................................................... 49
8.9. ssh using Net::SSH.................................................................................................................... 49
8.10. LWP::Simple........................................................................................................................... 50
8.11. LWP......................................................................................................................................... 51
Copyright 2007, Gabor Szabo @ Perl Training Israel iv
-
8/8/2019 Perl in Testing
5/89
8.12. WWW::Mechanize.................................................................................................................. 51
8.13. WWW::GMail......................................................................................................................... 52
8.14. Exercise: CNN Developing story ............................................................................................ 52
8.15. Exercise: Search on Financial Times ...................................................................................... 52
8.16. Exercise: Compare exchange rates.......................................................................................... 53
8.17. Telnet to Unix machines.......................................................................................................... 53
9. Servers .................................................................................................................................................. 54
9.1. Net::Server ................................................................................................................................ 54
9.2. Skeleton Server ......................................................................................................................... 54
9.3. Simple Echo Server ................................................................................................................... 54
9.4. Echo Server ............................................................................................................................... 55
9.5. Complex network servers .......................................................................................................... 56
10. Command Line Interface .................................................................................................................. 58
10.1. Introduction ............................................................................................................................. 58
10.2. Connect to the device .............................................................................................................. 58
10.3. Reduce timeout ....................................................................................................................... 59
10.4. Exercise: Telnet....................................................................................................................... 60
10.5. Our test script .......................................................................................................................... 60
11. Testing networking devices ............................................................................................................... 64
11.1. Elements.................................................................................................................................. 64
11.2. Hardware setup........................................................................................................................ 64
11.3. Access the administrative interface......................................................................................... 64
11.4. Configure devices on all sides of our box ............................................................................... 64
11.5. Run tests .................................................................................................................................. 64
11.6. Check results ........................................................................................................................... 64
11.7. Expect.pm ............................................................................................................................... 65
11.8. Other modules ......................................................................................................................... 65
11.9. Networking.............................................................................................................................. 65
11.10. Network devices .................................................................................................................... 6511.11. Devices connected to Serial or Parallel port ......................................................................... 66
11.12. X10 protocol ......................................................................................................................... 66
12. Web Applications............................................................................................................................... 67
12.1. What can be tested ?................................................................................................................ 67
12.2. Tools ........................................................................................................................................ 67
12.3. Small test HTTP server ........................................................................................................... 67
12.4. Fetching a static page .............................................................................................................. 67
12.5. Fetching a not-existing static page .......................................................................................... 68
12.6. Checking good HTML ............................................................................................................ 68
12.7. Checking bad HTML .............................................................................................................. 69
12.8. What is this bad HTML ? ........................................................................................................ 69
12.9. HTML::Tidy and Test::HTML::Tidy ...................................................................................... 70
12.10. Test using W3C validator ...................................................................................................... 70
12.11. LWP::Simple and LWP ......................................................................................................... 71
12.12. WWW::Mechanize................................................................................................................ 71
12.13. Web based Calculator............................................................................................................ 71
12.14. More things to test................................................................................................................. 73
Copyright 2007, Gabor Szabo @ Perl Training Israel v
-
8/8/2019 Perl in Testing
6/89
12.15. Test page with JavaScript ...................................................................................................... 73
13. Database access using Perl DBI ........................................................................................................ 74
13.1. Architecture of a DBI Application.......................................................................................... 74
13.2. Create Sample Database.......................................................................................................... 74
13.3. Connect to database ................................................................................................................ 74
13.4. SELECT with one result ......................................................................................................... 7513.5. SELECT with more results ..................................................................................................... 75
13.6. SELECT, prepare with placeholders ....................................................................................... 76
13.7. SELECT, using hashref........................................................................................................... 76
13.8. INSERT ................................................................................................................................... 76
13.9. Sample database ...................................................................................................................... 77
13.10. Other methods ....................................................................................................................... 78
13.11. Attributes............................................................................................................................... 78
13.12. Error handling ....................................................................................................................... 78
13.13. Debugging (Trace levels) ...................................................................................................... 79
14. Database access using Class::DBI .................................................................................................... 80
14.1. Class::DBI ............................................................................................................................... 80
14.2. INSERT using Class::DBI ...................................................................................................... 8014.3. SELECT using Class::DBI...................................................................................................... 80
14.4. MyDBI and MyUsers .............................................................................................................. 80
Copyright 2007, Gabor Szabo @ Perl Training Israel vi
-
8/8/2019 Perl in Testing
7/89
List of Examples5-1. mycalc on the command line................................................................................................................ 6
5-2. examples/intro/t01_calc.t ..................................................................................................................... 6
5-3. examples/intro/t01_calc.out ................................................................................................................. 6
5-4. examples/intro/t02_calc.t ..................................................................................................................... 7
5-5. examples/intro/t02_calc.out ................................................................................................................. 75-6. examples/intro/t03_calc.t ..................................................................................................................... 8
5-7. examples/intro/t03_calc.out ................................................................................................................. 8
5-8. examples/intro/t04_calc.t ..................................................................................................................... 9
5-9. examples/intro/t04_calc.out ................................................................................................................. 9
5-10. examples/intro/t05_calc.t ................................................................................................................... 9
5-11. examples/intro/t05_calc.out ............................................................................................................... 9
5-12. examples/intro/t09_calc.t ................................................................................................................. 10
5-13. examples/intro/t09_calc.out ............................................................................................................. 10
5-14. examples/intro/t10_calc.t ................................................................................................................. 11
5-15. examples/intro/t10_calc.out ............................................................................................................. 11
5-16. examples/intro/t11_calc.t ................................................................................................................. 11
5-17. examples/intro/t12_calc.t ................................................................................................................. 125-18. examples/intro/t12_calc.out ............................................................................................................. 12
5-19. examples/intro/t13_calc.t ................................................................................................................. 13
5-20. examples/intro/calc.txt ..................................................................................................................... 13
5-21. examples/intro/t13_calc.out ............................................................................................................. 13
5-22. examples/harness.pl.......................................................................................................................... 14
5-23. examples/intro/t13_calc.harness.out ................................................................................................ 14
5-24. examples/intro/t14_calc.t ................................................................................................................. 15
5-25. examples/intro/mycalc_test.t............................................................................................................ 16
6-1. examples/intro/t21_calc.t ................................................................................................................... 18
6-2. examples/intro/t22_calc.t ................................................................................................................... 19
6-3. examples/intro/like.t ........................................................................................................................... 20
6-4. examples/intro/cmp_ok.t .................................................................................................................... 20
6-5. examples/intro/is_deeply.t .................................................................................................................. 21
6-6. examples/intro/is_deeply.out .............................................................................................................. 22
6-7. examples/intro/t23_calc.t ................................................................................................................... 23
6-8. examples/intro/without_skip.t ............................................................................................................ 24
6-9. examples/intro/skip.t .......................................................................................................................... 24
6-10. examples/intro/Test/MyTest.pm ....................................................................................................... 26
6-11. examples/intro/early-abnormal-exit.t ............................................................................................... 27
6-12. examples/intro/early-normal-exit.t ................................................................................................... 28
6-13. examples/intro/too-many-tests.t ....................................................................................................... 29
6-14. examples/intro/t24_calc.t ................................................................................................................. 29
6-15. examples/intro/t25_calc.t ................................................................................................................. 306-16. examples/intro/multiple_choice.t ..................................................................................................... 31
7-1. examples/bc/bc1.pl ............................................................................................................................. 34
7-2. examples/bc/bc2.pl ............................................................................................................................. 34
7-3. examples/bc/bc3.pl ............................................................................................................................. 35
7-4. examples/bc/bc4.pl ............................................................................................................................. 36
7-5. examples/bc/bc5.pl ............................................................................................................................. 36
vii
-
8/8/2019 Perl in Testing
8/89
-
8/8/2019 Perl in Testing
9/89
Chapter 1. About Perl Training Israel
PTI has been providing training and development services since 2000. Our courses include the following:
Perl Courses
Fundamentals of Perl
References, Modules and Objects (Advanced)
Web Application Development with Perl (Advanced)
Debugging Perl Scripts and Applications (Advanced)
Perl Quick Start (Beginner)
QA Automation using Perl (Advanced)
Perl for QA Professionals (Beginner)
Non-Perl Courses
Regular Expressions for Java programmers
Regular Expressions for .NET programmers
Version Control with Subversion
Details can be found on our web site http://www.pti.co.il
1
-
8/8/2019 Perl in Testing
10/89
Chapter 2. Introduction
2.1. Self Introduction - Who am I ? Gabor Szabo http://www.pti.co.il
First program in 1983
In the field since 1993
Perl trainer since 2000
Perl developer
CPAN author
System Administrator
Web application developer
Testing Automation
http://www.szabgab.com
2.2. Self Introduction - Who are you ?
Name
Workplace
Background in computers (e.g. Operating Systems)
Background in programming (which languages? How many years?)
Background in Perl?
Why do you take this course ?
What do you expect to accomplish ?
2
-
8/8/2019 Perl in Testing
11/89
Chapter 3. Preface
3.1. Objectives
All tests successful.
Files=10, Tests=2078, 50 wallclock secs
Why and when to test ?
Learn about the tools Perl provides for automated testing
Use the test framework provided by Perl
3.2. Plan of the seminar1. Introduction
2. Manual testing, What, when how to test ?
3. Introduce the Perl modules for basic testing framework
4. Command line applications
5. Devices with CLI access
6. Web applications
7. Testing Databases
8. X Windows
9. Microsoft Windows GUI Applications
10. Internet Explorer based applications
11. Appendix
3
-
8/8/2019 Perl in Testing
12/89
Chapter 4. Manual testing
4.1. Manual testing
Before we dig in into the world of automated testing we should first see that we are speaking about the
same thing.
4.2. Web site testing
You type in a URL
see if the site comes up correctly,
click on the link to the registration form,
fill in the fields (youll have to play with this and fill in the fields with good values, bad values, find theedge cases etc.)
Check if you get back the correct response page.
Maybe check if the database contains the correct information
4.3. CLI testing
Here you have a device like a router, or some other box connected to the network. Normally you would
telnet to it and then interactively test the various commands to see if they work. In addition you might be
able to fetch the raw configuration information where you can validate if the configuration values were
written correctly.
Going even further after you configured the device somehow you can test it if the new behaviour of the
device really can be observed: You connect other devices and ping this box or try to send packets and see
if they get to the correct location.
Telnet to device
Use SNMP to monitor/configure the device
Prepare external entities on 2 or more sides of the device
Send packets
Check if the packets were received correctly
4
-
8/8/2019 Perl in Testing
13/89
Chapter 4. Manual testing
4.4. Database testing
This can be considered as part of any application as there is some kind of a database used by every
application. In the simple case the database might be a flat file but it can also be some csv file or xml
file or an RDBMS that you can access using SQL. In this case you would like to test what are the
consequences on the database of certain operations of the application ?
Prepare a database
Execute some code
Check if the database was updated correctly
4.5. GUI testing
Launch the application
Find various controls Type in values/check if they show up correctly
Push buttons
Check if results are correct
Copyright 2007, Gabor Szabo @ Perl Training Israel 5
-
8/8/2019 Perl in Testing
14/89
Chapter 5. Basic Testing Framework in Perl
5.1. Testing a simple command line tool
First well talk about a simple application with simple tests and from there youll be able to move on to
test more complex applications. I have a new calculator program called "mycalc". It expects a
mathematical expression on the command line and prints the result on the screen. e.g.:
Example 5-1. mycalc on the command line
> mycalc 3 + 4
7
Lets look at it first an then write a couple of tests for it.
5.2. Calculator test
Example 5-2. examples/intro/t01_calc.t
#!/usr/bin/perl
use strict;
use warnings;
system "./mycalc 1 + 1";
print "\n";
system "./mycalc 2 + 2";
print "\n";
system "./mycalc 2 + 2 + 2";
print "\n";
Output:
Example 5-3. examples/intro/t01_calc.out
2
4
4
5.3. Error !
Did you notice the bad answer ?
6
-
8/8/2019 Perl in Testing
15/89
Chapter 5. Basic Testing Framework in Perl
The third line should have been 6.
5.4. Calculator test with expected results
So we should write the expected value next to the correct one:
Example 5-4. examples/intro/t02_calc.t
#!/usr/bin/perl
use strict;
use warnings;
system "./mycalc 1 + 1";
print " 2\n";
system "./mycalc 2 + 2";
print " 4\n";
system "./mycalc 2 + 2 + 2";
print " 6\n";
Output:
Example 5-5. examples/intro/t02_calc.out
2 2
4 4
4 6
Now it is better.
5.5. More difficult output
But what if the expected output is:
This is the expected output of my program that should be checked.
This is the expected output of my program that should be checked.
Yes, in this case they are actually the same. What if you had 300 such line pairs ? And what if 3000 ?
It would be probably much better if our testing program already compared the expected value with the
actual results and would only print OK or NOT OK depending on success or failures.
OK
OK
NOT OK
Copyright 2007, Gabor Szabo @ Perl Training Israel 7
-
8/8/2019 Perl in Testing
16/89
Chapter 5. Basic Testing Framework in Perl
5.6. Print only ok/not ok
Example 5-6. examples/intro/t03_calc.t
#!/usr/bin/perl
use strict;
use warnings;
my $result;
$result = ./mycalc 1 + 1;
if ( $result == 2 ) {
print "ok\n";
}
else {
print "not ok\n";
}
$result = ./mycalc 2 + 2;
if ( $result == 4 ) {print "ok\n";
}
else {
print "not ok\n";
}
$result = ./mycalc 2 + 2 + 2;
if ( $result == 6 ) {
print "ok\n";
}
else {
print "not ok\n";
}
# We replaced the "system" calls with backtick in order to catch the STDOUT
# It is extreamly verbose and we are repeating the same code a lot of times
Output:
Example 5-7. examples/intro/t03_calc.out
ok
ok
not ok
Copyright 2007, Gabor Szabo @ Perl Training Israel 8
-
8/8/2019 Perl in Testing
17/89
Chapter 5. Basic Testing Framework in Perl
5.7. Write the ok function
Example 5-8. examples/intro/t04_calc.t
#!/usr/bin/perl
use strict;
use warnings;
ok(./mycalc 1 + 1 == 2);
ok(./mycalc 2 + 2 == 4);
ok(./mycalc 2 + 2 + 2 == 6);
sub ok {
my ($ok) = @_;
print $ok ? "ok\n" : "not ok\n";
}
Output:
Example 5-9. examples/intro/t04_calc.out
ok
ok
not ok
But why reinvent the wheel ?
Besides, if there are lots of tests, we would need some way to easily recognise which test(s) fail. So we
should put a counter on our tests.
5.8. Introducing Test::Simple
Example 5-10. examples/intro/t05_calc.t
#!/usr/bin/perl
use strict;
use warnings;
# tell how many tests you are going to write. This is our "plan"
use Test::Simple tests => 3;
# the ok function of Test::Simple prints "ok" or "not ok"
ok ./mycalc 1 + 1 == 2;
ok ./mycalc 2 + 2 == 4;
ok ./mycalc 2 + 2 + 2 == 6;
Output:
Copyright 2007, Gabor Szabo @ Perl Training Israel 9
-
8/8/2019 Perl in Testing
18/89
Chapter 5. Basic Testing Framework in Perl
Example 5-11. examples/intro/t05_calc.out
1..3
ok 1
ok 2
not ok 3
# Failed test in t05_calc.t at line 11.
# Looks like you failed 1 test of 3.
It is more verbose, it has a couple of additional useful piece of information: 1..3 says how many tests we
were planning then we get the tests numbered and we even get a small explanation when the test fails.
5.9. Add names to the tests
So Test::Simple makes our life a bit more simple in that we dont have to write our testing expression. In
addition this new "ok" function can actually do some more favour. It can get two arguments. The first one
indicates success or failure of the test and the second one is the name of the test. When running a test
with these additional names they get printed on the same line where the "ok" or "not ok" is printed. In
case of lots of tests it will make it easier to locate the test and if the names were written carefully they
can provide an immediate hint what went wrong. Sometimes you wont even need to look at the test
script itself, right from this comment youll know where to look for the bug.
Example 5-12. examples/intro/t09_calc.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::Simple tests => 3;
ok ./mycalc 1 + 1 == 2, small sum: 1+1;
ok ./mycalc 2 + 2 == 4, small sum: 2+2;
ok ./mycalc 2 + 2 + 2 == 6, two operators: 2+2+2;
Output:
Example 5-13. examples/intro/t09_calc.out
1..3
ok 1 - small sum: 1+1
ok 2 - small sum: 2+2
not ok 3 - two operators: 2+2+2
# Failed test two operators: 2+2+2
# in t09_calc.t at line 9.
# Looks like you failed 1 test of 3.
Copyright 2007, Gabor Szabo @ Perl Training Israel 10
-
8/8/2019 Perl in Testing
19/89
Chapter 5. Basic Testing Framework in Perl
5.10. Enlarge our test suit
Example 5-14. examples/intro/t10_calc.t
#!/usr/bin/perl
use strict;
use warnings;
my %tests = (
1 + 1 => 2,
2 + 2 => 4,
2 + 2 + 2 = > 6 ,
1+1 => 2,
0+ -1 => -1,
0-1 => -1,
-1+1 => 0,
);
use Test::Simple tests => 7;
foreach my $t ( keys %tests ) {
ok( ./mycalc $t == $tests{$t}, $t );
}
Example 5-15. examples/intro/t10_calc.out
1..7
o k 1 - 0 - 1
ok 2 - 1 + 1
ok 3 - -1+1
o k 4 - 0 + - 1
o k 5 - 1 + 1
n o t o k 6 - 2 + 2 + 2
# Failed test 2 + 2 + 2
# in t10_calc.t at line 18.
ok 7 - 2 + 2
# Looks like you failed 1 test of 7.
There is a small problem though. When you add a new test to the hash, you also have to remember to
update the tests => 7 line.
There are a number of solution to this problem
5.11. Load Test::Simple at run time
Example 5-16. examples/intro/t11_calc.t
#!/usr/bin/perl
Copyright 2007, Gabor Szabo @ Perl Training Israel 11
-
8/8/2019 Perl in Testing
20/89
Chapter 5. Basic Testing Framework in Perl
use strict;
use warnings;
my %tests = (
1 + 1 => 2,
2 + 2 => 4,
2 + 2 + 2 = > 6 ,
1+1 => 2,
0+ -1 => -1,
0-1 => -1,
-1+1 => 0,
);
require Test::Simple;
import Test::Simple tests => scalar keys %tests;
foreach my $t ( keys %tests ) {
ok( ./mycalc $t == $tests{$t}, $t );
}
5.12. Forget about your "plan", use "no_plan"
Example 5-17. examples/intro/t12_calc.t
#!/usr/bin/perl
use strict;
use warnings;
my %tests = (
1 + 1 => 2,
2 + 2 => 4,
2 + 2 + 2 = > 6 ,
1+1 => 2,
0+ -1 => -1,
0-1 => -1,
-1+1 => 0,
);
use Test::Simple "no_plan";
foreach my $t ( keys %tests ) {
ok( ./mycalc $t == $tests{$t}, $t );
}
Example 5-18. examples/intro/t12_calc.out
o k 1 - 0 - 1
ok 2 - 1 + 1
ok 3 - -1+1
Copyright 2007, Gabor Szabo @ Perl Training Israel 12
-
8/8/2019 Perl in Testing
21/89
Chapter 5. Basic Testing Framework in Perl
o k 4 - 0 + - 1
o k 5 - 1 + 1
n o t o k 6 - 2 + 2 + 2
# Failed test 2 + 2 + 2
# in t12_calc.t at line 18.
ok 7 - 2 + 2
1..7
# Looks like you failed 1 test of 7.
The 1..7 is now at the end.
5.13. Put the test cases in an external file
Example 5-19. examples/intro/t13_calc.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::Simple "no_plan";
open my $fh, "
-
8/8/2019 Perl in Testing
22/89
Chapter 5. Basic Testing Framework in Perl
Example 5-21. examples/intro/t13_calc.out
ok 1 - 1 + 1
ok 2 - 2 + 2
n o t o k 3 - 2 + 2 + 2
# Failed test 2 + 2 + 2
# in t13_calc.t at line 16.
o k 4 - 1 + 1
o k 5 - 0 + - 1
o k 6 - 0 - 1
ok 7 - -1+1
1..7
# Looks like you failed 1 test of 7.
5.14. Harness
This is a module that can analyse the ok / not ok printouts with the numbers. In particular it can analysethe output of Test::Simple, Test::More and all the Test::Builder based modules.
Example 5-22. examples/harness.pl
#!/usr/bin/perl
use strict;
use warnings;
use Test::Harness qw(runtests);
runtests @ARGV;
Run the previous test file using Test::Harness
$ perl ../harness.pl t13_calc.t
Example 5-23. examples/intro/t13_calc.harness.out
t13_calc....
# Failed test 2 + 2 + 2
# in t13_calc.t at line 16.
# Looks like you failed 1 test of 7.
dubious
Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 3
Failed 1/7 tests, 85.71% okay
Failed Test Stat Wstat Total Fail List of Failed
-------------------------------------------------------------------------------
t13_calc.t 1 256 7 1 3
Failed 1/1 test scripts. 1/7 subtests failed.
Files=1, Tests=7, 1 wallclock secs ( 0.12 cusr + 0.05 csys = 0.17 CPU)
Copyright 2007, Gabor Szabo @ Perl Training Israel 14
-
8/8/2019 Perl in Testing
23/89
Chapter 5. Basic Testing Framework in Perl
Failed 1/1 test programs. 1/7 subtests failed.
5.15. Move external call into function
Example 5-24. examples/intro/t14_calc.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::Simple "no_plan";
open my $fh, "
-
8/8/2019 Perl in Testing
24/89
Chapter 5. Basic Testing Framework in Perl
5.17. Solution: MyCalc
Example 5-25. examples/intro/mycalc_test.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::Simple tests => 5+2*3;
use MyCalc;
# tests, one by one
ok(add(1, 1) == 2);
ok(add(1, -1) == 0);
ok(sum(1) == 1);
ok(sum() == 0);
ok(sum(1, 1, 1, 1) == 4);
# tests listed in an array
my @tests = (
{
func => add,
in => [2, 3],
out => 5,
},
{
func => sum,
in => [1, 2, 3],
out => 6,
},
);
foreach my $t (@tests) {
if ($t->{func} eq add) {
ok(add( @{ $t->{in} } ) == $t->{out}, "add @{ $t->{in} }");
}
if ($t->{func} eq sum) {
ok(sum( @{ $t->{in} } ) == $t->{out}, "sum @{ $t->{in} }");
}
}
# The same but Danger! Danger! Danger!
# Using symbolic references here!foreach my $t (@tests) {
no strict refs;
ok(&{ $t->{func} }( @{ $t->{in} } ) == $t->{out}, "$t->{func} @{ $t->{in} }");
# the same with helper variables:
my $func = $t->{func};
Copyright 2007, Gabor Szabo @ Perl Training Israel 16
-
8/8/2019 Perl in Testing
25/89
Chapter 5. Basic Testing Framework in Perl
my @in = @{ $t->{in} };
ok(&$func(@in) == $t->{out}, "$func @in");
}
5.18. Test::Simple
This is all very nice and Simple.
What if you want More ?
Copyright 2007, Gabor Szabo @ Perl Training Israel 17
-
8/8/2019 Perl in Testing
26/89
Chapter 6. Test::More
6.1. Moving over to Test::More
Test::Simple is really a very simple module. Its sole exported function is the "ok" function.
Test::More has the same "ok" function - so it is a drop-in replacement - but it also has lots of
other functions and tools:
ok
is
isnt
diag
like
cmp_ok
is_deeply
SKIP
TODO
6.2. Test::More ok( trueness, name);
A drop-in replacement of Test::Simple.
Example 6-1. examples/intro/t21_calc.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests => 3;
ok ./mycalc 1 + 1 == 2, 1+1;
ok ./mycalc 2 + 2 == 4, 2+2;
ok ./mycalc 2 + 2 + 2 == 6, 2+2+2;
Result
$ perl t6_calc.t
1..3
o k 1 - 1 + 1
o k 2 - 2 + 2
not ok 3 - 2+2+2
18
-
8/8/2019 Perl in Testing
27/89
Chapter 6. Test::More
# Failed test (t6_calc.t at line 7)
# Looks like you failed 1 tests of 3.
6.3. Test::More is( value, expected_value, name);
It would be much better to see the expected value and the actually received value. This usually helps in
locating the problem.
Example 6-2. examples/intro/t22_calc.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests => 3;
is ./mycalc 1 + 1, 2, 1+1;
is ./mycalc 2 + 2, 4, 2+2;
is ./mycalc 2 + 2 + 2, 6, 2+2+2;
Result
$ perl t7_calc.t
1..3
o k 1 - 1 + 1
o k 2 - 2 + 2
not ok 3 - 2+2+2
# Failed test (t7_calc.t at line 7)
# got: 4
# expected: 6# Looks like you failed 1 tests of 3.
See, in this case we can already guess that it cannot add 3 values.
compares using eq
6.4. diag(just_a_message );
diag prints out a message along with the rest of the output.
Use it for whatever extra output in order to ensure that your printouts will not interfere with future
changes in the test environment modules (such as Test::Harness).
diag "We are going to test the Foo-Bar device now";
Copyright 2007, Gabor Szabo @ Perl Training Israel 19
-
8/8/2019 Perl in Testing
28/89
Chapter 6. Test::More
# We are going to test the Foo-Bar device now
6.5. like(value, qr/expected regex/, name);
It is especially important when you dont want or cant realisticly expect an exact match with the result.
compares with =~
Example 6-3. examples/intro/like.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests => 2;
like( foo(), qr/\d+/, "there are some digits in the result" );
like( bar(), qr/\d+/, "there are some digits in the result" );
sub foo {
return "This is a long text with a number 42 in it";
}
sub bar {
return "This is another string with no number in it";
}
$ perl like.t
1..2
ok 1 - there are some digits in the resultnot ok 2 - there are some digits in the result
# Failed test there are some digits in the result
# in like.t at line 7.
# This is another string with no number in it
# doesnt match (?-xism:\d+)
# Looks like you failed 1 test of 2.
6.6. cmp_ok( this, op, that, name);
compares with anything
Copyright 2007, Gabor Szabo @ Perl Training Israel 20
-
8/8/2019 Perl in Testing
29/89
Chapter 6. Test::More
Example 6-4. examples/intro/cmp_ok.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests => 2;
my $start = time;
wait_for_input_with_timeout(3);
my $end = time;
cmp_ok $end - $start, ">=", 2, "process was waiting at least 2 secs";
cmp_ok $end - $start, "
-
8/8/2019 Perl in Testing
30/89
Chapter 6. Test::More
my %a = fetch_data_from_bug_tracking_system(0);
is_deeply( \%a, \%expected, "Query 0" );
my %b = fetch_data_from_bug_tracking_system(1);
is_deeply( \%b, \%expected, "Query 1" );
my %c = fetch_data_from_bug_tracking_system(2);
is_deeply( \%c, \%expected, "Query 2" );
sub fetch_data_from_bug_tracking_system {
my @sets = (
{ bugs => 3,
errors => 6,
failures => 8,
warnings => 1,
},
{ bugs => 3,
errors => 9,
failures => 8,
warnings => 1,
},
{ bogs => 3,
erors => 9,
failures => 8,
warnings => 1,
},
);
my $h = $sets[shift];
return %$h;
}
Example 6-6. examples/intro/is_deeply.out
1..3
ok 1 - Query 0
not ok 2 - Query 1
# Failed test Query 1
# in is_deeply.t at line 19.
# Structures begin differing at:
# $got->{errors} = 9
# $expected->{errors} = 6
not ok 3 - Query 2
# Failed test Query 2
# in is_deeply.t at line 22.
# Structures begin differing at:
# $got->{errors} = Does not exist
# $expected->{errors} = 6
# Looks like you failed 2 tests of 3.
Copyright 2007, Gabor Szabo @ Perl Training Israel 22
-
8/8/2019 Perl in Testing
31/89
-
8/8/2019 Perl in Testing
32/89
Chapter 6. Test::More
6.10. Platform dependent tests
Example 6-8. examples/intro/without_skip.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More "no_plan";
like( /sbin/ifconfig, qr/eth0/ );
like( ipconfig, qr/Windows IP Configuration/ );
$ perl without_skip.t
ok 1
not ok 2
# Failed test (without_skip.t at line 5)
# undef
# doesnt match (?-xism:Windows IP Configuration)
1..2# Looks like you failed 1 tests of 2.
6.11. SKIP some tests
Example 6-9. examples/intro/skip.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests => 2;
like( /sbin/ifconfig, qr/eth0/ );
SKIP: {
skip "Windows related tests", 1 if $^O !~ /Win/i;
like( ipconfig, qr/Windows IP Configuration/ );
}
$ perl skip.t
ok 1
ok 2 # skip Windows related tests
1..2
Copyright 2007, Gabor Szabo @ Perl Training Israel 24
-
8/8/2019 Perl in Testing
33/89
Chapter 6. Test::More
6.12. My own test functions
After writing lots of tests, youll start to re-factor your code and create functions for specific parts of the
test:
You will have some code like this:
use Test::More tests => 2;
is(my_test(2, +, 3), 5);
is(my_test(4, +, 4), 10);
sub my_test {
my ($x, $op, $z) = @_;
# do stuff
return $result;
}
6.13. My own test functions - improved
It would be better if we could eliminate the multiple occurance of is
use Test::More tests => 2;
my_test(2, +, 3, 5);
my_test(4, +, 4, 10);
sub my_test {
my ($x, $op, $z, $expected) = @_;
# do stuff
is($result, $expected);
}
6.14. Create a test module
You need the above my_test subroutine in several of your test files. So you create a module for this.
use Test::More tests => 2;
use Test::MyTest "my_test";
my_test(2, +, 3, 5);
Copyright 2007, Gabor Szabo @ Perl Training Israel 25
-
8/8/2019 Perl in Testing
34/89
Chapter 6. Test::More
my_test(4, +, 4, 10);
package Test::MyTest;
require Exporter;
our @ISA = qw(Exporter);
our @EXPORT_OK = qw(my_test);
sub my_test {
my ($x, $op, $z, $expected) = @_;
# do stuff
is($result, $expected);
}
Not good !
We need the is function from the Test::More package. so well have to use it in our module as well but
Test::More needs a "plan".
6.15. Test::Builder
Example 6-10. examples/intro/Test/MyTest.pm
package Test::MyTest;
use strict;
use warnings;
use base Exporter;
our @EXPORT_OK = qw(my_test);
use Test::Builder;
my $Test = Test::Builder->new;
sub my_test {
my ($x, $op, $y, $expected) = @_;
my $result;
if ($op eq +) {
$result = $x + $y;
} else {
die "Not yet implemented";
}
$Test->is_num($result, $expected);
Copyright 2007, Gabor Szabo @ Perl Training Israel 26
-
8/8/2019 Perl in Testing
35/89
Chapter 6. Test::More
}
1;
Test modules created using Test::Builder all work nicely together. Among other things, they dont get
confused with the counting of the tests.
Test::Simple, Test::More, Test::Exception, Test::Differences all use Test::Builder as a back-end.
6.16. Number of tests
It is always good to have a plan when you start working even if you are going to change your plans
within a short time. If for nothing else if you use some version control youll be able to see your
progress. (What do you mean you dont have version control ?) Checking if the right number of tests
were successfully executed is a test in itself. Some things that can go wrong if you dont check it:
Test script dies before running all the tests.
Test script dies after running all the tests.
You run too many tests - in itself it is not a problem but later when for some reason your test script
stops after the correct number of test you wont notice the missing tests.
It is always good to have some plan.
6.17. Early Abnormal Exit
Example 6-11. examples/intro/early-abnormal-exit.t
#!/usr/bin/perl
use strict;
use warnings;
# example of tests where the script dies before running all the tests
use Test::More tests => 4;
is ./mycalc 1 + 1, 2, 1+1;
is ./mycalc 2 + 2, 4, 2+2;
exit;
is ./mycalc 3 + 3, 6, 3+3;
is ./mycalc 4 + 4, 8, 4+4;
Directly:
Copyright 2007, Gabor Szabo @ Perl Training Israel 27
-
8/8/2019 Perl in Testing
36/89
Chapter 6. Test::More
1..4
o k 1 - 1 + 1
o k 2 - 2 + 2
# Looks like you planned 4 tests but only ran 2.
With Harness:
early-abnormal-exit....ok 2/4# Looks like you planned 4 tests but only ran 2.
early-abnormal-exit....dubious
Test returned status 2 (wstat 512, 0x200)
DIED. FAILED tests 3-4
Failed 2/4 tests, 50.00% okay
Failed Test Stat Wstat Total Fail Failed List of Failed
----------------------------
early-abnormal-exit.t 2 512 4 4 100.00% 3-4
Failed 1/1 test scripts, 0.00% okay. 2/4 subtests failed, 50.00% okay.
6.18. Less than planned tests - Early Normal Exit
Example 6-12. examples/intro/early-normal-exit.t
#!/usr/bin/perl
use strict;
use warnings;
# example of too few tests (less than planned)
use Test::More tests => 4;
is ./mycalc 1 + 1, 2, 1+1;is ./mycalc 2 + 2, 4, 2+2;
is ./mycalc 3 + 3, 6, 3+3;
Directly:
1..4
o k 1 - 1 + 1
o k 2 - 2 + 2
o k 3 - 3 + 3
# Looks like you planned 4 tests but only ran 3.
With Harness:
early-normal-exit....ok 2/4# Looks like you planned 4 tests but only ran 3.
early-normal-exit....dubious
Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 4
Failed 1/4 tests, 75.00% okay
Copyright 2007, Gabor Szabo @ Perl Training Israel 28
-
8/8/2019 Perl in Testing
37/89
Chapter 6. Test::More
Failed Test Stat Wstat Total Fail Failed List of Failed
-----------------------------
early-normal-exit.t 1 256 4 2 50.00% 4
Failed 1/1 test scripts, 0.00% okay. 1/4 subtests failed, 75.00% okay.
6.19. More tests than planned
Example 6-13. examples/intro/too-many-tests.t
#!/usr/bin/perl
use strict;
use warnings;
# example of too many tests (more than planned)
use Test::More tests => 2;
is( ./mycalc 1 + 1, 2, 1+1 );
is( ./mycalc 2 + 2, 4, 2+2 );
is( ./mycalc 3 + 3, 6, 3+3 );
Directly:
1..2
o k 1 - 1 + 1
o k 2 - 2 + 2
o k 3 - 3 + 3
# Looks like you planned 2 tests but ran 1 extra.
With Harness:
too-many-tests....ok 3/2# Looks like you planned 2 tests but ran 1 extra.
too-many-tests....dubious
Test returned status 1 (wstat 256, 0x100)
DIED. FAILED test 3
Failed 1/2 tests, 50.00% okay
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------
too-many-tests.t 1 256 2 1 50.00% 3
Failed 1/1 test scripts, 0.00% okay. -1/2 subtests failed, 150.00% okay.
6.20. Multiply
Example 6-14. examples/intro/t24_calc.t
#!/usr/bin/perl
Copyright 2007, Gabor Szabo @ Perl Training Israel 29
-
8/8/2019 Perl in Testing
38/89
Chapter 6. Test::More
use strict;
use warnings;
use Test::More tests => 1;
is ./mycalc 2 * 2, 4, 2*2;
Results
$ perl t10_calc.t
1..1
Bareword found where operator expected at (eval 1) line 1, near "2 IE"
(Missing operator before IE?)
n o t o k 1 - 2*2
# Failed test (t10_calc.t at line 5)
# got:
# expected: 4
# Looks like you failed 1 tests of 1.
6.21. Error in the test
User error
When you find a problem it is not always the problem in the application you are testing.
If it was a human who did the work we would call it a "user error" but in our case the
user is a program itself. As the developer of the application can make errors the
person who writes the tests can also make errors.
6.22. Multiply - fixed
Example 6-15. examples/intro/t25_calc.t
#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests => 1;
is ./mycalc "2 * 2", 4, 2*2;
Results
$ perl t25_calc.t
1..1
ok 1 - 2*2
Copyright 2007, Gabor Szabo @ Perl Training Israel 30
-
8/8/2019 Perl in Testing
39/89
Chapter 6. Test::More
6.23. Multiple expected values
Someone asked me this question:
Q: Is it possible instead ofis(foo(), 42) to have a fixed set of expected values ?
A: With is there is no such option but you have a number of solutions anyway.
Example 6-16. examples/intro/multiple_choice.t
#!/usr/bin/perl
use strict;
use warnings;
# run this script several times to see the (random) error message
use Test::More tests => 2;
like( foo(), qr/^(23|42|68)$/ );
ok( grep { foo() eq $_ } ( 23, 42, 68 ), "name" );
sub foo {
return ( ( 23, 42, 68, 100, 200 )[ rand(5) ] );
}
$ perl multiple_choice.t
not ok 1
# Failed test (multiple_choice.t at line 4)
# 100
# doesnt match (?-xism:^(23|42|68)$)
not ok 2# Failed test (multiple_choice.t at line 5)
1..2
# Looks like you failed 2 tests of 2.
$ perl multiple_choice.t
ok 1
ok 2
1..2
6.24. Exercises* take the ifconfig/ipconfig test script and fix it so
there will be a skip block on the ifconfig part as well.
* create a test function is_any that will get
a scalar and and a reference to an array and
Copyright 2007, Gabor Szabo @ Perl Training Israel 31
-
8/8/2019 Perl in Testing
40/89
Chapter 6. Test::More
check if the scalar is one of the values in
the array.
write test script that uses this function.
* create a test module that exports the is_any function.
Write test script that uses this module.
* Change the test script you wrote for MyCalc.pm
to use the is function of Test::More.
Copyright 2007, Gabor Szabo @ Perl Training Israel 32
-
8/8/2019 Perl in Testing
41/89
Chapter 7. Command line application
7.1. bc - An arbitrary precision calculator language
It is much more than a calculator, it is a language. Luckily we dont need to learn the whole language in
order to to do simple calculations. Normally you execute bc from the command line and then you type
in your calculations. Pressing ENTER will do the calculation.
7.2. Normal operation
> bc
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type warranty.
23+7
30
quit
>
Try it ....
7.3. Expect.pm
Provides a way to describe user behaviour in a command line environment
Can send information as if it was typed in the keyboard
Can wait for some Expect-ed value and based on this value do something
Originally an extension of Tcl
Ported to Perl
Can be used in environments such as:
Command line application like bc
Telnet to another box and type in things
Anything usually a person would do on the command line.
33
-
8/8/2019 Perl in Testing
42/89
Chapter 7. Command line application
7.4. Simple computation - adding two values
Example 7-1. examples/bc/bc1.pl
#!/usr/bin/perl
use strict;
use warnings;
use Expect;
my $e = Expect->new;
$e->raw_pty(1);
$e->spawn("bc") or die "Cannot run bc\n";
$e->expect(1, "warranty") or die "no warranty\n";
$e->send("23+7\n");
$e->expect(1, 30) or die "no sum\n";
print "Success\n";
raw_pty turns off echo
spawn starts the external program
expect(timeout, regex) return undef if failed
timeout is in seconds, 0 means check once, undef means wait forever
send - as if the user was typing at the keyboard
7.5. Results
>perl examples/bc/bc1.pl
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type warranty.
30
Success
7.6. Simple computation - separate send commands
Instead of calling send once, we will call it now 4 times
Copyright 2007, Gabor Szabo @ Perl Training Israel 34
-
8/8/2019 Perl in Testing
43/89
Chapter 7. Command line application
Example 7-2. examples/bc/bc2.pl
#!/usr/bin/perl
use strict;
use warnings;
use Expect;
my $e = Expect->new;
$e->raw_pty(1);
$e->spawn("bc") or die "Cannot run bc\n";
$e->expect(1, "warranty") or die "no warranty\n";
$e->send("23");
$e->send("+");
$e->send("7");
$e->send("\n");
$e->expect(1, 30) or die "no sum\n";
print "Success\n";
7.7. Simple computation - is it really working ?
The same computation but we expect the wrong value ...
This is not that clever but I could not find a real bug in bc to show here as an example and Id like to
show what happens when the computation fails.
Example 7-3. examples/bc/bc3.pl
#!/usr/bin/perluse strict;
use warnings;
use Expect;
my $e = Expect->new;
$e->raw_pty(1);
$e->spawn("bc") or die "Cannot run bc\n";
$e->expect(1, "warranty") or die "no warranty\n";
$e->send("23+7\n");
$e->expect(1, 29) or die "no sum\n";
print "Success\n";
7.8. Results
The same computation but we expect the wrong value ... and fail
Copyright 2007, Gabor Szabo @ Perl Training Israel 35
-
8/8/2019 Perl in Testing
44/89
Chapter 7. Command line application
>perl examples/bc/bc3.pl
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type warranty.
30
no sum
7.9. Reduce output
We dont want to see all the output bc generates and then try to look for the correct responses or the error
messages. Wed prefer just see ok or not ok
Example 7-4. examples/bc/bc4.pl
#!/usr/bin/perl
use strict;
use warnings;
use Expect;
use Test::More qw(no_plan);
$Expect::Log_Stdout = 0;
my $e = Expect->new;
$e->raw_pty(1);
$e->spawn("bc") or die "Cannot run bc\n";
$e->expect(1, "warranty") or die "no warranty\n";
$e->send("23+7\n");
ok($e->expect(1, 30));
$Expect::Log_Stdout = 0; - turn off the printing to the screen
7.10. Output
>perl examples/bc/bc4.pl
ok 1
1..1
7.11. More than one test
We can then setup lots of tests and run them through one invocation of bc.
Copyright 2007, Gabor Szabo @ Perl Training Israel 36
-
8/8/2019 Perl in Testing
45/89
Chapter 7. Command line application
Example 7-5. examples/bc/bc5.pl
#!/usr/bin/perl
use strict;
use warnings;
use Expect;
use Test::More qw(no_plan);
$Expect::Log_Stdout = 0;
my @sets = (
[23+7, 30],
[23+7, 30],
[11+1, 10],
[2*21, 42],
);
my $e = Expect->new;
$e->spawn("bc") or die "Could not start bc\n";
$e->expect(undef, "warranty") or die "no warranty\n";
foreach my $set (@sets) {
$e->send("$$set[0]\n");
ok($e->expect(1, $$set[1]), $$set[0]);
}
$e->send("quit\n");
7.12. Output
>perl examples/bc/bc5.pl
# You named your test 30. You shouldnt use numbers for your test names.# Very confusing.
o k 1 - 3 0
ok 2 - 23+7
not ok 3 - 11+1
# Failed test (examples/bc/bc5.pl at line 20)
ok 4 - 2*21
1..4
# Looks like you failed 1 tests of 4.
The first entry in the @sets array is not good as the operation will be executed by Perl and not by bc.
7.13. External test file
Separating the test cases from the code.
Copyright 2007, Gabor Szabo @ Perl Training Israel 37
-
8/8/2019 Perl in Testing
46/89
Chapter 7. Command line application
Example 7-6. examples/bc/bc5a.pl
#!/usr/bin/perl
use strict;
use warnings;
use Expect;
use Test::More qw(no_plan);
$Expect::Log_Stdout = 0;
open my $fh, "
-
8/8/2019 Perl in Testing
47/89
Chapter 7. Command line application
Either save the new results as the new expectation or discard it and discard the current version of the
application
7.15. Random and regression testing
Example 7-7. examples/bc/bc6.pl
#!/usr/bin/perl
use strict;
use warnings;
use File::Compare;
use Expect;
$Expect::Log_Stdout = 0;
if (not @ARGV or $ARGV[0] ne "random" and $ARGV[0] ne "regress") {
die "Usage: $0 [random|regress]\n";
}
if ($ARGV[0] eq "regress" and not -e "tests.txt") {
die "Cannot run regression before running random tests!\n";
}
if ($ARGV[0] eq random) {
my $e = Expect->new;
$e->raw_pty(1);
$e->log_file("random.log", "w");
$e->spawn("bc") or die "Could not start bc\n";
$e->expect(1, "warranty") or die "no warranty\n";
open my ($test_file), ">tests.txt" or die "Cannot open tests file for writing\n"
foreach (1..3) {
my ($a, $b) = (rand, rand);
my $op = qw(+ * - /)[int rand 4];
my $line = "$a $op $b\n";
print $test_file $line;
$e->send($line);
$e->expect(1,[qr/\d+/]);
}
$e->send("quit\n");
}
if ($ARGV[0] eq regress) {
my $e = Expect->new;
Copyright 2007, Gabor Szabo @ Perl Training Israel 39
-
8/8/2019 Perl in Testing
48/89
Chapter 7. Command line application
$e->raw_pty(1);
$e->log_file("regress.log", "w");
$e->spawn("bc") or die "Could not start bc\n";
$e->expect(1, "warranty") or die "no warranty\n";
open my ($test_file), "tests.txt" or die "Cannot open tests file for reading\n";
while (my $line = ) {
$e->send($line);
$e->expect(1, [qr/\d+/]);
}
$e->send("quit\n");
$e->log_file(undef); # close the log file
if (compare("random.log", "regress.log")) {
print "Regression failed\n";
} else {
print "Regression successful\n";
}
}
# Two parts
# - random tests
# - regression tests
#
#
# run random tests
# save test cases in a test cases file
# save all the results in a log file
#
#
# run all the tests from the test cases file and log results to a new log file
# compare original log with new log and complain if they are not the same.
#
7.16. Random and regression testing - slight
improvement
Example 7-8. examples/bc/bc7_diff.pl
#!/usr/bin/perl
use strict;
use warnings;
use Text::Diff; # instead of File::Compare
# and diff instead of compare
Copyright 2007, Gabor Szabo @ Perl Training Israel 40
-
8/8/2019 Perl in Testing
49/89
Chapter 7. Command line application
if (my $diff = diff ("random.log", "regress.log")) {
print "Regression failed\n\n";
print $diff;
} else {
print "Regression successful\n";
}
7.17. Results
~/work/training/testing/examples/bc>perl bc7.pl regress
Regression failed
--- random.log Mon Apr 19 23:09:51 2004
+++ regress.log Mon Apr 19 23:38:54 2004
@@ -3,5 +3,5 @@
This is free software with ABSOLUTELY NO WARRANTY.
For details type warranty.
1
-.9114942335378392
+.8114942335378392
.8404963860314223
7.18. Recording session
A simple example how to "record" a session for later analysing and changing into a real script.
Example 7-9. examples/expect/record.pl
#!/usr/bin/perl
use strict;
use warnings;
use Expect;
my $exp = Expect->spawn("bash") or die "Cannot spawn bash child\n";
$exp->log_file($ARGV[0] || "record.log");
#$exp->raw_pty(1);
$exp->interact();
Copyright 2007, Gabor Szabo @ Perl Training Israel 41
-
8/8/2019 Perl in Testing
50/89
Chapter 7. Command line application
7.19. Capturing both STDOUT and STDERR
Write out the expected STDIN to a file called "in"
Run the app system "$app < in >out 2>err";
read in the out and err files an examine them
7.20. Capturing both STDOUT and STDERR manually
Example 7-10. examples/io/capture.pl
#!/usr/bin/perl -w
use strict;
use Test::More tests => 2;
my $app = "./examples/io/application.pl";
my @in = (10, 21, hello, 3x);
my $in = join "\n", @in;
my @expected_out = (20, 42);
my @expected_err = (
"The input hello contains no numeric values",
"The input 3x contains no numeric values",
);
{
open my $fh, ">", "/tmp/in" or die $!;
print $fh $in;
}
system "$app < /tmp/in > /tmp/out 2> /tmp/err";
{
open my $fh, "
-
8/8/2019 Perl in Testing
51/89
Chapter 7. Command line application
7.21. Capturing both STDOUT and STDERR using
IPC::Run3
Example 7-11. examples/io/capture_ipc.pl
#!/usr/bin/perl -wuse strict;
use Test::More tests => 2;
use IPC::Run3;
my $app = "./examples/io/application.pl";
my @in = (10, 21, hello, 3x);
my $in = join "\n", @in;
my @expected_out = (20, 42);
my @expected_err = (
"The input hello contains no numeric values","The input 3x contains no numeric values",
);
{
my $out;
my $err;
run3 [$app], \$in, \$out, \$err;
my $expected_out = join("\n", @expected_out) . "\n";
is($out, $expected_out, "IPC Output");
my $expected_err = join("\n", @expected_err) . "\n";
is($err, $expected_err, "IPC Error");
}
Copyright 2007, Gabor Szabo @ Perl Training Israel 43
-
8/8/2019 Perl in Testing
52/89
Chapter 8. Networking devices
8.1. Introduction - pick the right abstraction level
When trying to connect some network device using Perl you have a number of choices.
See the full stack of HTTP connections:
built in socket function and the Socket module
IO::Socket::INET using IO::Socket
Net::Telnet, Net::FTP, Net::SSH (wrapping ssh), Net::SSH::Perl, Net::*
LWP::Simple, LWP
WWW::Mechanize
WWW::GMail
At the lowest level you can use the built in socket function.
Using the Socket library provides several extra functions
and constants that will make your code cleaner and more portable.
See also perlipc
8.2. Socket level programming using Socket.pmExample 8-1. examples/network/socket.pl
#!/usr/bin/perl
use strict;
use warnings;
use Socket qw(:DEFAULT :crlf);
# Using the built in "socket" function with various helper variables
# and functions from the standard Socket.pm module
# get the protocol id (on Linux from /etc/protocols)my $protocol_id = getprotobyname(tcp);
# build C structure in_addr from hostip
# if hostname is given it tries to resolve hostname to ip first (and returns
# undef if not successful)
#my $host = 127.0.0.1;
my $host = localhost;
44
-
8/8/2019 Perl in Testing
53/89
Chapter 8. Networking devices
#my $host = 66.249.85.99;
#my $host = www.google.com;
my $host_struct = inet_aton($host);
# inet_ntoa($host_struct) returns the resolved ip address
my $port = 80;
socket(my $socket, PF_INET, SOCK_STREAM, $protocol_id) or die $!;
my $sockaddr_in = pack_sockaddr_in($port, $host_struct);
connect($socket, $sockaddr_in) or die $!;
# turn off buffering on the socket
{
my $old = select($socket);
$ | = 1 ;
select($old);
}
print $socket "GET /$CRLF";
while (my $line = ) {
print $line;
}
8.3. Socket level programming using IO::Socket
Example 8-2. examples/network/io_socket.pl
#!/usr/bin/perl
use strict;
use warnings;
use IO::Socket;
# IO::Socket is a higher level abstraction
# Hides many of the ugly part we had to know in case of the socket() function.
# Provides an OOP interface.
#my $host = 127.0.0.1;
my $host = localhost;
#my $host = www.google.com;
#my $host = 209.85.135.103;
#my $host = www.perl.org.il;
my $port = 80;
my $CRLF = "\015\012";
my $socket = IO::Socket::INET->new(
PeerAddr => $host,
PeerPort => $port,
Copyright 2007, Gabor Szabo @ Perl Training Israel 45
-
8/8/2019 Perl in Testing
54/89
Chapter 8. Networking devices
Proto => tcp,
) or die $!;
$socket->send("GET /$CRLF") or die $!;
my $SIZE = 100;
my $data = ;
while ($socket->read($data, $SIZE, length $data) == $SIZE) {};
print $data;
8.4. Newline
\n is a newline on our current system (is NOT always ASCII LF)
\r is (is NOT always ASCII CR)
use \015\012 to say CR+LF on networking applications
8.5. Net::Telnet
Example 8-3. examples/telnet/telnet.pl
#!/usr/bin/perl
use strict;
use warnings;
use Net::Telnet ();
my $t = Net::Telnet->new();
$t->open(localhost);
$t->login(smoke, 123456);
my @lines = $t->cmd("who");
print @lines;
print "\n";
print "Who am i: ", $t->cmd("whoami"), "\n\n";
8.6. Net::Telnet for HTTP
Example 8-4. examples/telnet/telnet_http.pl
#!/usr/bin/perl
use strict;
use warnings;
use Net::Telnet ();
Copyright 2007, Gabor Szabo @ Perl Training Israel 46
-
8/8/2019 Perl in Testing
55/89
Chapter 8. Networking devices
my $t = Net::Telnet->new(
Timeout => 10,
Host => localhost,
Port => 80,
);
$t->print("GET /\n") or die $!;
while (my $line = $t->getline) {
print $line;
}
8.7. Net::Telnet configure VLAN
Example 8-5. examples/telnet/configure_vlan.pl
#!/usr/bin/perl
use strict;
use warnings;
use Net::Telnet;
open my $out, ">>", "out.log" or die $!;
my $t = Net::Telnet->new(
Timeout => 2,
#Prompt => />/,
input_log => "input.log",
);
$t->open("172.30.40.146");
$t->waitfor(/User:.*$/);
$t->print("admin");
$t->waitfor(/Password:/);
$t->print("");
$t->waitfor(/>/);
$t->prompt(/\(Switching\) >/);
my @lines = $t->cmd("show vlan 5");
if (grep /VLAN ID: 5/, @lines) {
print "VLAN is already configured\n";
print "Please remove it manually and rerun the program\n";
exit;
#$t->cmd("logout");
}
$t->print("enable");
$t->waitfor(/Password:/);
Copyright 2007, Gabor Szabo @ Perl Training Israel 47
-
8/8/2019 Perl in Testing
56/89
Chapter 8. Networking devices
$t->prompt(/\(Switching\) #/);
$t->print("");
$t->prompt(/\(Switching\) \(Vlan\) #/);
@lines = $t->cmd("vlan database");
@lines = $t->cmd("vlan 5");
#print @lines;
#if (grep /VLAN already/, @lines) {
# print "QQ 1\n";
#}
@lines = $t->cmd("vlan 5000");
#print @lines;
@lines = $t->cmd("vlan 5");
#print @lines;
#if (grep /VLAN already/, @lines) {
# print "QQ 2\n";
#}
#@lines = $t->cmd("no vlan 5");
$t->prompt(/\(Switching\) #/);
$t->cmd("exit");
$t->prompt(/--More-- or \(q\)uit/);
@lines = $t->cmd("show ?");
$t->output_record_separator("");
push @lines, $t->cmd(" ");
$t->prompt(/\(Switching\) #show/);
push @lines, $t->cmd(" ");
#print @lines;
$t->output_record_separator("\n");
$t->prompt(/\(Switching\) #/);
@lines = $t->cmd(" vlan 5"); # show was left on the promt line !
#print @lines;
@lines = $t->cmd("show vlan 7");
#print @lines;
@lines = $t->cmd("show slot");
#print @lines;
print $out @lines;
$t->prompt(/\(Switching\) \(Vlan\) #/);
@lines = $t->cmd("vlan database");
@lines = $t->cmd("no vlan 5");
Copyright 2007, Gabor Szabo @ Perl Training Israel 48
-
8/8/2019 Perl in Testing
57/89
Chapter 8. Networking devices
$t->prompt(/\(Switching\) #/);
$t->cmd("exit");
print "done: $_\n";
print $out "done: $_\n";
8.8. ftp using Net::FTP
Example 8-6. examples/network/upload.pl
#!/usr/bin/perl
use strict;
use warnings;
use Net::FTP;
use File::Basename qw(dirname);
use File::Spec;
my $DEBUG = 1;
if (not @ARGV) {
print "Usage:\n";
print " $0 FILE [FILES]\n";
exit;
}
my $ftp = Net::FTP->new(192.168.1.100) or die $!;
$ftp->login(gabor, the password of gabor) or die $!;
my $pwd = $ftp->pwd;
foreach my $file (@ARGV) {
my $dir = dirname $file;
$ftp->cwd($pwd);
$ftp->cwd($dir);
$ftp->put($file);
}
8.9. ssh using Net::SSH
Wrapping the external ssh command. Therefore working only in UNIX/Linux.See also Net::SSH::Perl.
Copyright 2007, Gabor Szabo @ Perl Training Israel 49
-
8/8/2019 Perl in Testing
58/89
Chapter 8. Networking devices
Example 8-7. examples/network/ssh.pl
#!/usr/bin/perl
use strict;
use warnings;
use Net::SSH qw(sshopen2);
use IO::File;
my $output = IO::File->new;
my $input = IO::File->new;
sshopen2("localhost", $output, $input) or die $!;
print $input "set\n";
print $input "echo DONE\n";
print $input "who\n";
print $input "echo DONE\n";
print $input "date\n";
print $input "echo DONE\n";
print $input "cat README\n";
print $input "exit\n";
my @out = ;
my $c=0;
my @section;
while (my $line = shift @out) {
if ($line =~ /^DONE$/) {
$c++;
next;
}
push @{$section[$c]}, $line;
}
foreach my $sect (@section) {
print @$sect;
print "--------------------\n";
}
8.10. LWP::Simple
Example 8-8. examples/network/lwp_simple.pl
#!/usr/bin/perl
use strict;
use warnings;
my $url = http://localhost/;
if (defined $ARGV[0]) {
$url = $ARGV[0];
}
Copyright 2007, Gabor Szabo @ Perl Training Israel 50
-
8/8/2019 Perl in Testing
59/89
Chapter 8. Networking devices
use LWP::Simple qw(get);
my $page = get($url);
if (defined $page) {
print $page;
} else {
print "Could not fetch $url\n";
}
8.11. LWP
Example 8-9. examples/network/lwp.pl
#!/usr/bin/perl
use strict;
use warnings;
my $url = http://localhost/;
if (defined $ARGV[0]) {
$url = $ARGV[0];
}
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent("Internet Explorer/17.1");
my $req = HTTP::Request->new(GET => $url);
my $res = $ua->request($req);
if ($res->is_success) {
print $res->content;
} else {
print $res->status_line, "\n";
}
8.12. WWW::Mechanize
Example 8-10. examples/network/mechanize.pl
#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;
my $w = WWW::Mechanize->new();
$w->get(http://www.google.com/);
$w->submit_form(
Copyright 2007, Gabor Szabo @ Perl Training Israel 51
-
8/8/2019 Perl in Testing
60/89
Chapter 8. Networking devices
fields => {
q => perl israel
},
);
$w->follow_link( n => 5 );
print $w->title;
8.13. WWW::GMail
Example 8-11. examples/network/gmail.pl
#!/usr/bin/perl
use strict;
use warnings;
use WWW::GMail;
my $w = WWW::GMail->new(
username => "USERNAME",
password => "PASSWORD",
);
my $ret = $w->login();
if ($ret == -1) {
die "password incorrect\n";
} elsif ($ret == 0) {
die "unable to login $w->{error}\n";
}
my @messages = $w->get_message_list(inbox);
foreach my $msg (@messages) {
print "Subject: $msg->[6]\n";
}
8.14. Exercise: CNN Developing story
Write a script that will print the title of the current "DEVELOPING STORY" on
CNN (if there is one). That is click on the link titled "DEVELOPING STORY"
and then fetch the TITLE tag from the new page.
Copyright 2007, Gabor Szabo @ Perl Training Israel 52
-
8/8/2019 Perl in Testing
61/89
Chapter 8. Networking devices
8.15. Exercise: Search on Financial Times
Go to the Financial Times (www.ft.com) and search for the workd "perl".
Print out how many items match.
8.16. Exercise: Compare exchange rates
Go to various sites, check the AAA/BBB exchnage rate on each one of them.
Print out the current exchange rate and also send an e-mail to yourself
with the information.
Also put a ARBITRAGE warning if there is a difference between the rates.
AAA and BBB can be any two currencies you are interested in.
Try the following sites:
http://www.xe.com/
http://www.oanda.com/
http://www.bankofcanada.ca/en/rates/exchform.html
8.17. Telnet to Unix machines
Pick a Unix/Linux machine your are using and write a script that will telnet
to it, execute report what network cards does it have (ifconfig on Linux)
and what is its routing table (route -n on Linux).
Copyright 2007, Gabor Szabo @ Perl Training Israel 53
-
8/8/2019 Perl in Testing
62/89
Chapter 9. Servers
9.1. Net::Server
We are going to use the Net::Server module to create various server processes.
9.2. Skeleton Server
First we create