matthias leich omer barnir...– based on ‘mysqltest’ test cases and scripts – high level...

1Copyright 2007 MySQL AB The World’s Most Popular Open Source DatabaseMySQL Confidential

Writing Advanced Test Case

Matthias Leich

&

Omer BarNir


Agenda

• Upgrade/Downgrade Tests (Omer BarNir)

• Concurrency/Longevity Tests (Omer BarNir)

• Best Practices and Advance Mysqltest Features(Matthias Leich)


Upgrade/Downgrade Tests

• Overview (1/2)

– Shell based framework to test MySQL feature compatibility

between different MySQL Server versions.

– Based on ‘mysqltest’ test cases and scripts

– High level execution path:

• Starts an 'old' version of the server

• Executes initial ‘setup’ tests

• Export the database

• Stops the old server and starts the new one (using the existing

copy of the database or creates a fresh on -- as configured) – If fresh database loads the old exported database

• Runs the upgrade program

• Executes ‘upgrade’ tests

• Stops the new server and starts the old one (using the existing

copy of the database or creates a fresh on -- as configured) – If fresh database loads the new exported database

• Executes ‘downgrade’ tests



• Overview (2/2)

– Execution path is repeated running combination of ‘live’and ‘import/expert based upgrade and downgrade cycles.

– Execution path can be limited to a single test in a test scenario

– Execution path can be run in a ‘no upgrade’ mode

• To isolate if a problem is upgrade related

– Execution can be run in a ‘debug’ mode

• Where the script is stopped in key steps and the mysql

command line client can be invoked



• Script files• ‘run_updown’

– Main script running the upgrade/downgrade test

• Environment file (*.env)– A configuration file containing information related to where binaries are installed and

other system related parameters

• Scenario file (*.tst)– A configuration file containing information related to how the tests should be run

and other test related parameters

./run_updown --config=local.env --scenario=50to51.tst

• Test Files• Each test is made up of three ‘mysql-test’ files, one run before

upgrade, one after the upgrade and the third after the downgrade.

• Test file names are in the format of{initial|upgrade|downgrade}<nn>.test

where nn is a sequential number given to that test.

• All tests for a given ‘scenario’ (i.e. ’50minor’, ‘50to51’ etc.) are stored in ‘scenario name’ directory.



• Location, Files and Directory Structure

– Located in the mysql-test-extra-5. n trees in the */mysql-test-/qa-suite/updown directory.

– Structure of the updown directory:

• ‘bin’ directory– Holds execution scripts, and configuration files

• ‘scenarios’ directory– Holds sub-directory for each of the ‘scenarios’

• Each of the ‘scenario’ directories includes– /t for test files

– /r for result files



• How to run the tests:– Install ‘old’ and ‘new’ binaries

– Clone the ‘test-extra’ tree

– In new_bin/mysql-test/suite and old_bin/mysql-test/suitecreate links to the test_extra/mysql-test/qa-suite/updowndirectory:

• ln –s test_extra/mysql-test/qa-suite/updown updown

– Configure the environment file

– Modify the ‘scenario file (if needed)

– From test-extra/mysql-test/qa-suite/updown/bin, Execute:

• ./run_updown –scenario=file –config=file

• Optional command line options– --pause - for debug stops and invoking the client

– --no-upgrade - to run all tests against the ‘old’ binaries

– --test-num=n - to run a specific test



• Environment file example

###########################################################################SUITE="updown"

TOP_DIR_OLD_VER=/home/omer/ent-5.0.46-com # Top directory of old MySQL versionTOP_DIR_NEW_VER=/home/omer/beta-5.1.21 # Top directory of new MySQL versionTOP_TEST_EXTRA=/home/omer/source/tst51_801 # Top directory for mysql-test-extra

WINDOWS=0 # Set to '1' if running on Windows

SERVER_HOST="localhost" # Server hostname:# <localhost> for local connections# <some-host-name> for remote connections

SERVER_PORT="9306" # Port for remote connections

VAR_DIR=/tmp/systest/var # Variable pointing to the var'directory

# The following variables are used for storing result files fir the different# up/down permutations during the test runs. Names are relative to the location# of the 'scenario' directory in test-extraDD_DIR=res_dd # Upgrade/downgrade using a dump fileDL_DIR=res_dl # Upgrade using dump file, live downgradeLD_DIR=res_ld # Live upgrade, using dump file for downgradeLL_DIR=res_ll # live upgrade and downgrade

# TZ=GMT-3 # Set timezone variable if needed

################################################################################



• Scenario file example (1/2)

###########################################################################SCENARIO=50to51 # Scenario name

UPGRADE_TYPE="live" # Determines the upgrade method to use,# Can be set to:# 'live' - Use the existing database after upgrade# 'dump' - Export the db and import it after upgrade

DOWNGRADE_TYPE="dump" # Determines the downgrade method to use,# Can be set to:# 'live' - Use the existing database after downgrade# 'dump' - Export the db and import it after downgrade

UPDN_TYPE="major" # Indicates the differences bewtten the two releases# tested (since 'live' downgrade is not supported# between 'major' releases). Can be set to:# 'minor' - allow 'live' downgrade testing# 'major' - bypass 'live' downgrade testing,# -> If UPDN_MODE="single" set DOWNGRADE_TYPE to 'dump'# -> If UPDN_MODE="all", bypass 'live' DOWNGRADE_TYPE.

UPDN_MODE="all" # Indicates which of the combinations of# UPGRADE_TYPE and DOWNGRADE_TYPE will be used.# Can be set to:# 'single' - Run one round using $UPGRADE_TYPE for# the upgrade method and $DOWNGRADE_TYPE# for the downgrade method.# 'all' - Run four rounds using all combinations# of 'live' and 'dump' (ignoring the above# values of UPGRADE_TYPE and DOWNGRADE_TYPE




# TEST_NUM= # If set limits the run to a specific test scenario# Values are in the form of 00, 01,...

BATCH_MODE="one-pass" # Determines the order in which the test cases will# be executed. Can be set to:# 'one-pass - run all 'initial' then all 'opgrade' then# all 'downgrade' (i1,i2...,u1,u2...,d1,d2...)# 'none' - run each test in its own uprade/downgrade# cycle (i1,u1,d1,i2,u2,d2,...)# Note: BATCH_MODE is ignored if --test-num is specified# on the command line or if $TEST_NUM is set.

NEW_UPG_TOOL=mysql_upgrade # Determines what tool should be used during upgrade# can be set to 'fix_privs' or 'mysql_upgrade'# If new/old versions prior to 5.0.19 (*nix)# and 5.0.23 (Windows) should be 'fix_privs'

OLD_UPG_TOOL=mysql_upgrade # Determines what tool should be used during downgrade# can be set to 'fix_privs' or 'mysql_upgrade'# If new/old versions prior to 5.0.19 (*nix)# and 5.0.23 (Windows) should be 'fix_privs'

OLDVER_DUMP_OPT="" # Additional command line parameters to pass to# 'mysqldump' when dumping the 'old version' database# during a 'dump' upgrade.

NEWVER_DUMP_OPT="--skip-events" # Additional command line parameters to pass to# 'mysqldump' when dumping the 'new version' database# during a 'dump' downgrade.

###########################################################################



• More information:– https://inside.mysql.com/wiki/QA_UpDown_Howto

• Credits/Owner

– Initial Scripts: Matthias Leich

– Major Update and 5.0 test cases: Eric MaLossi

– Current revision, 5.1 test cases and current maintainer: Omer BarNir

– Other contributions: Serge Kozlov


Questions?


Concurrency/Longevity Tests

• Overview– Perl/shell based framework to test MySQL features in

concurrency for long durations of time.

– Based on ‘mysqltest’ test cases and scripts

– High level execution path:

• Starts a MySQL server (using mysql-test-run.pl)

• Executes initial ‘setup’ tests

• Executes subsequent tests concurrently for a specified

duration

• Stops the clients after duration has been reached



• Script files• ‘run_systest’

– Parent shell script controls the overall execution of the test.

• ‘mysql-stress-test.pl’– ‘low level script executing parallel executions of mysqltest with different test files as

specified in command line arguments

• Environment file (*.env)– A configuration file containing information related to where binaries are installed and

other system related parameters

• Scenario file (*.tst)– A configuration file containing information related to how the tests should be run

and other test related parameters

./run_systems --config=local.env --scenario=50to51.tst

• Test Files• Test files are grouped into ‘lists’.

• List files are text based

• ‘mysql-test’ test files are picked at random / in order from the list file and are executed against the server.



• Location, Files and Directory Structure

– Located in the mysql-test-extra-5.n trees in the */mysql-test-/qa-suite/systems directory.

– Structure of the systems directory is similar to other ‘test’ suites.



• How to run the tests:

– Clone the ‘test-extra’ tree

– Create links to the test_extra/mysql-test/qa-suite/updowndirectory:

• ln –s test_extra/mysql-test/qa-suite/updown updown


– Modify the ‘scenario file (if needed)

– From test-extra/mysql-test/qa-suite/updown/bin, Execute:

• ./run_systest –scenario=file –config=file

• Optional command line options (override scenario file):– --duration=m - time to run the test (seconds)

– --threads=n - number of threads (1st client)



• Environment file example

###########################################################################STRESS_SUITE="systems"

MYSQL_BASEDIR="/home/qauser/bin64_5122rc“ # Top location of MySQL binariesMYSQL_BINDIR="$MYSQL_BASEDIR/bin" # Location of the MySQL BinariesMYSQL_TEST="$MYSQL_BINDIR/mysqltest“ # Full path to mysqtest binarySTRESS_TEST_BASEDIR="/home/qauser/extra51_0827“ # Location of test-extra treeSTRESS_TEST_WORKDIR="/data0/qa_test_data/5122“ # Location of test log files

WINDOWS=0 # Set to '1' for Wimdows environmentsSERVER_HOST="localhost" # Server hostname:

# <localhost> for local connections through socket# <some-host-name> for remote connections

SERVER_PORT="9306" # Port for remote connections

MONITOR_FILE="sys_data.out" # Name of system monitor fileMONITOR_INTR=10 # Sleep time between system monitoringCLEANUP_INTR=30 # Sleep time between log file cleanup

CLEANUP=1 # Set to '0' if running as a client against a remote server

VAR_DIR="/data0/qa_test_data/5122_64bit-rhel5/var"# If running as a client of a remote server# Set to the value of the VARDIR on the server

export VAR_DIR # Used by log_cleanup.sh

SERVER_SOCKET="$VAR_DIR/tmp/master.sock"STRESS_TEST_SUITEDIR="$STRESS_TEST_BASEDIR/mysql-test/qa-suite/$STRESS_SUITE"

################################################################################




###########################################################################DURATION=86400 # 24 hour run - Duration in secondsTHREADS=50 # Number of test threadsLOG_TIME=30 # Defaut sleep time

LOAD_LINES=5000 # Initial # of lines loaded to tablesNUM_VAL=15 # Random values/segments to useIUDS_UNIT=10 # Number of rows used in some routines.

LOG_UPPER=5000 # Max Number of rows in log tableLOG_LOWER=4000 # Number of rows to truncate log

ENG1=innodb # Engine type used for the first tableENG1_ALT=myisam # Engine type used for altering the first tableENG2=innodb # Engine type used for the second tableENG_LOG=myisam # Engine type used for the log table

PROGRAM_INIT="initdb.txt" # Testprogram (collection of scripts) which# should be executed before the stress test# clients and the monitor are started

PROGRAM_TASK_1="log_tests.txt" # Testprogram (Collection of scripts) which# should monitor the ongoing stress test

LOG_TIME_1=$LOG_TIME # Sleep time between logging for the first task

PROGRAM_CLIENT_1="iuds3_tests.txt" # Testprogram (Collection of scripts) which# should be executed by the first stress test client

THREADS_CLIENT_1=25 # Number of threads for the first client




# The following can be used to specify a command to run once the test has completed# and all clients were terminatedPOST_TEST_COMMAND="$MYSQL_BINDIR/mysql --user=root --protocol=tcp --port=$SERVER_PORT \

-e'SET GLOBAL event_scheduler=OFF;'"

# Settings for the 2nd client run (ignored if PROGRAM_CLIENT_2 is unset)PROGRAM_CLIENT_2="iuds1_tests.txt" # Testprogram (Collection of scripts) which

# should be executed by the 2nd stress test clientTHREADS_CLIENT_2=10 # Number of threads for the 2nd client

# Settings for the 3rd client run (ignored if PROGRAM_CLIENT_3 is unset)PROGRAM_CLIENT_3="iuds0_tests.txt" # Testprogram (Collection of scripts) which

# should be executed by the 3rd stress test clientTHREADS_CLIENT_3=15 # Number of threads for the 3rd client

# Settings for the 2nd task run (ignored if PROGRAM_TASK_2 is unset)PROGRAM_TASK_2="tb1_mgr_mon.txt" # Testprogram (Collection of scripts) which

# should be executed by the 2nd test taskLOG_TIME_2=35 # Sleep time for the 2nd task

# Settings for the 3rd task run (ignored if PROGRAM_TASK_3 is unset)PROGRAM_TASK_3= # Testprogram (Collection of scripts) which

# should be executed by the 3rd test task

LOG_TIME_3= # Sleep time for the 3rd task

###########################################################################



• How to Add Tests:– Copy the test files you want to use in to the ‘/t’ directory under

‘systems’

– Copy the result files you want to use in to the ‘/r’ directory under ‘systems’

– In the ‘/t’ directory create: • a text file listing the ‘test’ file(s) you want to be run once for the test

setup.

• Test files listing ‘test’ file(s) you want to be be run concurrently during the test.

– In the ‘systems’ directory create (or copy an existing) ‘scenario’(*.tst) file. • Set the ‘PROGRAM_CLIENT_n‘ and ‘PROGRAM_TASK_n’ to point to

test files created above

• Set ‘THREADS_CLIENT_n’ for required number of concurrent threads

• Set ‘LOG_TIME_n’ for required task sleep time

• Set required values for ‘DURATION’, ‘THREADS’ and ‘LOG_TIME’


– From test-extra/mysql-test/qa-suite/updown/bin, Execute:• ./run_systest –scenario=file –config=file



• Test Results

– Test run result files including test log files, 'reject' files, screen outputs and error files are located under the 'STRESS_TEST_WORKDIR' location as defined in the 'environment file.

– These files need to be reviewed during/after the test run in order to analyze the test results



• Quick Run Option:

For running a single test case, a simplified ‘mysql-test-run.pl’

wrapper can be used:

perl ./mysql-test-run.pl --stress --stress-init-file=init.txt

--stress-test-file=run.txt –stress-test-duration=<seconds>

--stress-threads=<number of threads>

where –

init.txt includes the name of the test file to run once for the setup

run.txt includes the name(s) of the test files to run by the concurrent threads



• More information:– https://inside.mysql.com/wiki/QA_Systems_Howto

• Credits/Owner

– mysql-stress-test.pl: Alexey Stroganov (Ranger)

– run_systems: Omer BarNir / Matthias Leich

– Current maintainer: Omer BarNir


Questions?


Best Practices and Advance

Mysqltest Features

• Details and Examples :– https://inside.mysql.com/wiki/Uni_How_to_Create_good_Tests


Questions?

matthias leich omer barnir...– based on ‘mysqltest’ test cases and scripts – high level...

Documents