testing business-logic-in-dsls
TRANSCRIPT
Testing Business Logic Using DSLs in Clojure
Mayank Jain Test Engineer
A TALK, IN 6 PARTS
1. What is the problem?
2. Real World Example.
3. Demo of the actual code.
4. Other features tested using same ideas.
5. Advantages/Disadvantages of writing DSLs for testing.
6. QA.
WHAT IS THE PROBLEM?
Testing real world stateful business logic is hard
Microwave Oven State Machine
Business logic is Stateful
Microwave Oven State Machine
Large number transition states
Difficult to enumerate all possible cases
Microwave Oven State Machine
Tests become unreadable
Microwave Oven State Machine
HELPSHIFT
• Embeddable support desk for native apps
• Main Features:
• Frequently Asked Questions which customers can search
• File issues/tickets from within the app
Helpshift
Supercell
Clash of Clans
Gaana
General Billing
Domain
App
Section
FAQ 1 FAQ 2FAQs
Boom Beach Gaana App
Translations English Content Hindi Content
….
….
….
….
….
Top Down View of FAQs
Example of Gaana FAQ Page
Customers can search FAQs
Available FAQs FAQ Sections
Domain
App
FAQ Title
FAQ Body
FAQ is Visible?
PROBLEM
You cannot share FAQs across apps.
Big Customers have multiple apps which have same FAQ translations content example “Privacy Policy”
FEATURE: LINKED FAQS
{:published? false :id “faq-id-1” :app_id “app-1” :section_id “section-1” :translations {:en {:published? false :stags [] :body “Privacy Body” :title “Privacy Title”}} :linked_faq_ids [“faq-id-2”] :publish_id “1”}
App-1
{:published? false :id “faq-id-2” :app_id “app-2” :section_id “section-2” :translations {:en {:published? false :stags [] :body “Privacy Body" :title “Privacy Title"}} :linked_faq_ids [“faq-id-1”] :publish_id “2”}
App-2
faq 1
(Sync faq-1 to App-2)General
faq 2
General
{:published? false :id “faq-id-1” :app_id “app-1” :section_id “section-1” :translations {:en {:published? false :stags [] :body “Update Body” :title “Privacy Title”}} :linked_faq_ids [“faq-id-2”] :publish_id “1”}
{:published? false :id “faq-id-2” :app_id “app-2” :section_id “section-2” :translations {:en {:published? false :stags [] :body “Privacy Body" :title “Privacy Title"}} :linked_faq_ids [“faq-id-1”] :publish_id “2”}
faq 1Sync
faq 2
Update FAQ 1’s Body
{:published? false :id “faq-id-1” :app_id “app-1” :section_id “section-1” :translations {:en {:published? false :stags [] :body “Update Body” :title “Privacy Title”}} :linked_faq_ids [“faq-id-2”] :publish_id “1”}
{:published? false :id “faq-id-2” :app_id “app-2” :section_id “section-2” :translations {:en {:published? false :stags [] :body “Update Body" :title “Privacy Title"}} :linked_faq_ids [“faq-id-1”] :publish_id “2”}
faq 1Sync
faq 2
Update FAQ 1’s Body
EXAMPLE TEST CASE
1. Add 1st App with only English languages.
2. Add 2nd App with only English languages
3. Add 1st FAQ under 1st App
4. Link 1st FAQ to 2nd App to create 2nd FAQ
5. Check -> translations of 2nd FAQ == 1st FAQ
6. Update English title of FAQ-1
7. Check -> translations of 2nd FAQ == 1st FAQ
8. Delete FAQ-1
9. Check if 1st FAQ is deleted in DB
10.Assert 2nd FAQ should remain as it is in database.
Simulation Verification
PARTS OF EACH ACTION
“Add APP-1 with only English language”
1. Type of Action Add an App
2. Names App-1
3. Arguments English Language
4. Expected Result As per Spec/Modal of the system
5. Actual Result Database
EXPRESS ACTIONS AS CLOJURE DATA
“Add app APP-1 with English language”:add-app :app-1 {:langs-config [:en]}
ONE UNIT OF ACTION
[:add-app :app-1 {:langs-config [:en]}]
SERIES OF ACTIONS[[:add-app :app-1 {:langs-config [:en]]
[:add-app :app-2 {:langs-config nil}]
[:add-faq :faq-1 {:app-var :app-1 ...}]
[:link-faq :faq-2 {:faq-var :faq-1 :app-var :app-2 ...}]
[:update-faq nil {:app-var :app-1 :faq var :faq-1 ...}]
[:update-faq nil {:app-var :app-2 :faq-var :faq-2 ...}]
[:delete-faq nil {:faq-var :faq-1 ...}]]
REDUCE ON ACTIONS
Reduce
{}
[:add-app :app-1 {:langs-config nil}])
{:env {:apps {:app-1 {…}}} :result [{result-1…}]} …..}
Reduce
[:add-app :app-2 {:langs-config nil}])
{:env {:apps {:app-1 {…} :app-2 {…}}} :result [{result-1…} {result-2…}]} …..}
{:env {:apps {:app-1 {…}}} :result [{result-1…}]} …..}
And so on…
RESULT HASH-MAP COMPRISES OF
{:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}}} :result [{:action-type :add-faq :expected {….} :actual {….}} {…more}] “112” {…state…} …more}
Environment (:env) - Contains bindings of vars
RESULT HASH-MAP COMPRISES OF
{:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}}} :result [{:action-type :add-faq :expected {….} :actual {….}} {…more}] “112” {…state…} …more}
Result (:result) - Contains Actual And Expected Result
RESULT HASH-MAP COMPRISES OF
{:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}}} :result [{:action-type :add-faq :expected {….} :actual {….}} {…more}] “112” {…state…} …more}
Current Generated Expected State
Dispatch On Action Type
Update The Variables in arguments to its
bindings
Call args with relevant function
Bind the result to the given var in Global Data
Store Expected current state
Store Actual current database state
What happens
inside the
reducer?
Dispatch On Action Type
Update The Variables in arguments to its
bindings
Call args with relevant function
Bind the result to the given var in Global Data
Store Expected current state
Store Actual current database state
[:add-app :app-1 {:langs-config nil}])
[:add-app :app-1 {:langs-config nil}])
(add-app {:langs-config nil})
{:env {:apps {:app-1 {:app-id "970"}}}}
Dispatch On Action Type
Update The Variables in arguments to its
bindings
Call args with relevant function
Bind the result to the given var in Global Data
Store Expected current state
Store Actual current database state
[:add-app :app-2 {:langs-config nil}])
[:add-app :app-2 {:langs-config nil}])
(add-app {:langs-config nil})
{:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142"}}}}
Dispatch On Action Type
Update The Variables in arguments to its
bindings
[:add-faq :faq-1 {:app-var :app-1 ….}])
{:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}}
Dispatch On Action Type
Update The Variables in arguments to its
bindings
[:add-faq :faq-1 {:app-var :app-1 ….}])
[:add-faq :faq-1 {:app-id “970” …}])
{:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}}
Dispatch On Action Type
Update The Variables in arguments to its
bindings
Call args with relevant function
Bind the result to the given var in Global Data
[:add-faq :faq-1 {:app-var :app-1 ….}])
[:add-faq :faq-1 {:app-id “970” …}])
(add-faq {:app-id “970” ….})
Dispatch On Action Type
Update The Variables in arguments to its
bindings
Call args with relevant function
Bind the result to the given var in Global Data
[:add-faq :faq-1 {:app-var :app-1 ….}])
[:add-faq :faq-1 {:app-id “970” …}])
(add-faq {:app-id “970” ….}){:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}} :faqs {:faq-1 {:faq-id “112”}}}}
Dispatch On Action Type
Update The Variables in arguments to its
bindings
Call args with relevant function
Bind the result to the given var in Global Data
Store Expected current state
[:add-faq :faq-1 {:app-var :app-1 ….}])
[:add-faq :faq-1 {:app-id “970” …}])
(add-faq {:app-id “970” ….}){:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}} :faqs {:faq-1 {:faq-id “112”}}}}{:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}}} :result [{:action-type :add-faq :expected {….}}
Dispatch On Action Type
Update The Variables in arguments to its
bindings
Call args with relevant function
Bind the result to the given var in Global Data
Store Expected current state
Store Actual current database state
[:add-faq :faq-1 {:app-var :app-1 ….}])
[:add-faq :faq-1 {:app-id “970” …}])
(add-faq {:app-id “970” ….}){:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}} :faqs {:faq-1 {:faq-id “112”}}}}{:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}}} :result [{:action-type :add-faq :expected {….} :actual {….}}
VERIFICATION:COMPARE EXPECTED VS ACTUAL FOR
EACH STEP
COMPARE RESULT FORFIRST ACTION
{"112" {:translations {:en {:published? false, :stags [], :body "Temp body", :title “Title 1"}}, :linked-faq-ids #{}}}
Actual Data in DBExpected Data
from my Generated result
{"112" {:translations {:en {:published? false, :stags [], :body "Temp body", :title “Title 1"}}, :linked-faq-ids #{}}}
==PASS
{"112" {:translations {:en {:published? false, :stags [], :body "Temp body", :title "Title 1"}}, :linked-faq-ids #{}}}
Actual Data in DBExpected Data
from my Generated result
{"112" {:translations {:en {:published? false, :stags [], :body "Temp body", :title "Title 2"}}, :linked-faq-ids #{}}}
==FAIL
COMPARE RESULT FORSECOND ACTION
DEMO TIME
APPROACHES TO VERIFY
Actions : HardcodedExpected Output : Hardcoded
Actions : SimulatedExpected Output : Hardcoded
Actions : SimulatedExpected Output : Generated
Actions : GeneratedExpected Output : Generated
Hardcoded Actions, Hardcoded Expected Output
ADVANTAGES DISADVANTAGES
Requires no extra knowledge to understand
Very cumbersome to enumerate
Anyone can add/edit tests Modification is hard
False negatives are not possible
Simulate list of Actions, Hardcoded Final Expected
Output
ADVANTAGES DISADVANTAGES
Easy to enumerateRequires knowledge of
the DSL
Very readableMaintenance overhead of
DSL
Shareable with dev to simulate bugs
Does not check intermediate state
False negatives are not possible
Expected output may have data which is
available only at runtime like faq-ids
Simulate list of actions, Generate Expected Output
ADVANTAGES DISADVANTAGES
Easy to enumerateRequires knowledge of
the DSL
Very readableMaintenance overhead of
DSL
Shareable with dev to simulate bugs
Maintenance overhead of Expected Modal
Checks intermediate stateFalse negatives are
possibleRuntime data is available
like faq-ids
Generate Actions, Generate Expected Output
ADVANTAGES DISADVANTAGES
Possible to generate large number of tests
Requires knowledge of the DSL
Possible to Shrink failed test case using test.check library
Maintenance overhead of DSL
Shareable with dev to simulate bugs
Maintenance overhead of Expected Modal
Checks intermediate stateFalse negatives are
possible
Maintenance overhead of generative code for list of
actions.
Generate Actions, Generate Expected Output
Feature - Issue Audit Trail
Issue Audit Trail• Maintains a log of
• Who Took an action
• What Action they took
Who - Types of Users
App User Support User
Agents
Admins
Helpshift
What - Types of Actions
App User
• Create Issue
• Reply Issue
• etc…
Admin User
• Create Issue
• Reply Issue
• Resolve Issue
• Edit Tags
• etc…
Agent User
• Create Issue
• Reply Issue
• Resolve Issue
• Edit Tags
• etc…
App User
Admin“Mayank
”
Agent“Agent-2”
Example Issue
Example Issue
Issue Audit Logs
DEMO - ISSUE AUDIT TRAIL
Generate Actions, Generate Expected Output
ADVANTAGES OF WRITING DSL
Discovering very hard to find bugs, for example we found:
Duplicate issue messages being rendered only if the number of messages were "just right".
Finding ordering bugs.
Increase in developer/tester productivity
DISADVANTAGES OF WRITING DSL
Cost of maintaining DSLs is high.
If your DSL is just data being evaluated at run time, changes in feature code will not throw any compile time errors like function parameters being changed in DSL code.
You have to educate your team members to learn your specific DSL for that feature to be able to understand the tests.
CONCLUSION
DSL can be used as a workflow for writing tests.
Separating simulation vs verification of tests.
A step towards ability to generate tests instead of writing them.
Further Resources• Clojure Made Simple - Rich Hickey
• Growing a DSL with Clojure
• Jeanine Adkisson - Design and Prototype a Language In Clojure
• John Hughes - Testing the Hard Stuff and Staying Sane
• Reid Draper - Powerful Testing with test.check
• Clojure Tutorials on DSLs with Tim Baldridge (Paid)
Any Questions?@firesofmay
facebook.com/firesofmay/