automate legacy-system testing: easy, reliable, and extendible
TRANSCRIPT
THE UGLYTHE GOOD THE BAD
GREENFIELD vs. BROWNFIELD
GREENFIELD PROJECT
BROWNFIELD PROJECT
BROWNFIELD PROJECT
WHY INVEST IN LEGACY SYSTEM?
TEAM HAPPINESS
LOW TEAM MORALE
Regression Bugs
Fragile Software
Slow Feedback
Stupid Errors
Repetitive Work
Slow Progress
Quality software is team effort. It needs system thinking.
SHIFT LEFT
THE THREE PILLARS OF AUTOMATED TESTING
Static Code Analysis
Unit Tests
Black Box Tests
WHAT TO DO ABOUT IT
Start with basic acceptance tests
Functionality that makes money
Must have functionality - compliance, security
Repeating Manual Tests - Save Time
Pareto Principle - 80/20
Your Tests
Fast
Reliable
Maintainable
Do not test through the UI. (if possible)
result = RestClient.post( REGISTER_URL, user_details.to_json, {:content_type => :json} )
800 test x 10 seconds = 2h 13min
This saved us:
Set test data via API or DB.
Limit external dependencies calls. (talked about this last year)
Need to Call External System
Automation Test?
Talk to the real system
No
Fake the response
Yes
Test should create the data they need.
Scenario: Client admin should not be able to view master’s agencies Given а master user And master creates agency And a client admin When client admin views master's agency Then client admin should get an error
Poll for results from API/DB operations.
sleeping(1).seconds.between_tries.failing_after(10).tries do result = some_operation raise 'No Data' if result['data'] == []end
Run a test 20 times consecutively. Commit only if the test does not fail.
for i in {1..20}; do your_test; done
Automatically rerun failed tests.
Same static code checks for tests code as for production code.
CODE CHANGES
First Order of Business: Remove Unused Code
Remove the commented code
Remove the code you know it’s not used
Instrument the code to check what’s really used
Second Order of Business: Stop The Rot
CONTINUOUS INTEGRATION
Run on every commit
Max execution time: 5 min.
Hook one by one all the checks
Run longer tests periodically
Developers need to receive feedback about their new code within 5 minutes.
CHECKS ON COMMIT
The PHP Case
LINTER
php -l api/models/mobile_push_model.php
PHP Parse error: api/models/mobile_push_model.php on line 61 Errors parsing api/models/mobile_push_model.php
HHVM
UnknownObjectMethod in file: api/models/mobile_push_model.php, line: 55, problem entry:
$pusher->reallyUnsubscribeDevice ($params['user_id'], $params['device_id'], $actions)
STATIC CODE QUALITY
CYCLOMATIC COMPLEXITY
function testPrint() { echo('Hello World'); }
Complexity: 1
function testPrint($parameter) {if($parameter) {
echo('Hello World'); }
}
Complexity: 2
Method complexity should be less than 10.
Complexity 82
Complexity 10
Constantly refactor to decrease complexity
Method size should be less than 100 lines (ideally less than 50).
Improve the code - then lower the threshold on commit check.
Then repeat.
FIGHT LEGACY CODE
WRITE UNIT TESTS
Written by Developers
Fast, Independent
Test Technical Aspects
Cooperation between QA & Developers
[Demo]
100% test coverage is not sufficient!
Missing assertions
Missing handling unlikely conditions
Don’t aim for specific coverage number
SECURITY TESTS
SQL Injection Detection (PHP and ADOdb)
$dbConn->GetRow(“SELECT * FROM users WHERE id = $user_id”)
$dbConn->GetRow(“SELECT * FROM users WHERE id = ?”, array(‘$user_id’))
Those errors can be caught with code analysis.
No need to run slow whole application security scan.
There was no such tool.
So we developed one.
github.com/emanuil/php-reaper
Scans PHP ADOdb code for SQL injections
Command line
Suitable for CI, on-commit tests
Plans to expand, pull requests welcomed
MONITORING
Your second line of defence.
Show a lot with TV and Raspberry Pi.
Live Graphs + Deploys
Live Graphs + Deploys
CONCLUSION
Аutomatе the most important functionalities
Continuously improve static code quality
Write unit tests for changed/new code
Expand checks on commit
Enable monitoring
RECOMMENDED READING
