guidelines on facilitating testability - final

GuidelinesonFacilitatingTestability

Copyright©2016Anti-MalwareTestingStandardsOrganization,Inc.Allrightsreserved.Nopartofthisdocumentmaybereproducedinanyform,inanelectronicretrievalsystemorotherwise,withouttheprior

writtenconsentofthepublisher.

2

NoticeandDisclaimerofLiabilityConcerningtheUseofAMTSODocuments

ThisdocumentispublishedwiththeunderstandingthatAMTSOmembersaresupplyingthisinformationforgeneraleducationalpurposesonly.Noprofessionalengineeringoranyotherprofessionalservicesoradvice isbeingofferedhereby. Therefore,youmustuseyourownskillandjudgmentwhenreviewingthisdocumentandnotsolelyrelyontheinformationprovidedherein.

AMTSObelievesthattheinformationinthisdocumentisaccurateasofthedateofpublicationalthoughithasnotverifieditsaccuracyordeterminedifthereareanyerrors.Further,suchinformationissubjecttochangewithoutnoticeandAMTSOisundernoobligationtoprovideanyupdatesorcorrections.

Youunderstandandagreethat thisdocument isprovidedtoyouexclusivelyonanas-isbasiswithoutanyrepresentationsorwarrantiesofanykindwhetherexpress, impliedorstatutory. Without limitingthe foregoing, AMTSO expressly disclaims all warranties of merchantability, non-infringement,continuousoperation,completeness,quality,accuracyandfitnessforaparticularpurpose.

InnoeventshallAMTSObeliableforanydamagesorlossesofanykind(including,withoutlimitation,any lost profits, lost data or business interruption) arising directly or indirectly out of any use of thisdocument including, without limitation, any direct, indirect, special, incidental, consequential,exemplary and punitive damages regardless of whether any person or entity was advised of thepossibilityofsuchdamages.

Thisdocument isprotectedbyAMTSO’s intellectualpropertyrightsandmaybeadditionallyprotectedbytheintellectualpropertyrightsofothers.



3

GuidelinesonFacilitatingTestabilityIntroduction

Thisdocumentcoverswaysinwhichtestersandvendorscancollaborateandshareinformationinorderto make testing more efficient and accurate, and to enable external verification of results. Thedocumentoutlinesadditional issues involved inbestpracticetesting,aboveandbeyondotherAMTSOguidelinesandbestpractices.Thisdocumentisnotacomprehensivelistingofallsuchissues.

Unless otherwise defined herein, all terms included in this document are used with their commonmeaning.

AMTSOdocumentsarebest read inconjunctionwith theFundamentalPrinciplesofTestingandotherdocumentsontheAMTSOdocumentspageatwww.amtso.org.

Logging

Moredetailedandaccessible logginghelpsbothtestersandvendors.Althoughsometypesoftestwillneed to minimize all interaction with vendors, and run products exactly as normal users would, forlarger-scaleautomatedtestingitisoftenarequirementtohaveaccesstofinedetailabouttheactivitiesandobservationsof solutionsunder test, andwhat they reportaboutevents takingplaceon the testmachine.Inmanyexistingsolutions,loggingmaybeincomplete,orprovidedinaformatunsuitableforexternal processing. In order to have their solutions properly and accurately tested, vendors shouldensuretheirproductsprovideadequateinformationtofullyrecordalleventstakingplaceduringatest.In situationswhere logging systemsmust remain in a non-standard, encrypted or proprietary formatwhichisnotbestsuitedtoexportingandparsing,vendorsshouldprovidetoolstoconvertlogdataintoamoreusableformat.

Formore sophisticated stylesof testing itwill oftenbeuseful tohave considerablymore informationrecorded than is currently kept by most solutions, such as information on all pop-ups and alertsdisplayedbyproducts, and all responsesprovidedbyusers to requests for confirmationor decisions.Detailsofproducts'internalcommunicationssuchascloudlookupswillalsoprovideusefulinformationfor testers when analysing and interpreting test results. A list of all log entries, prompts and othermessagesandwhattheymeanwouldbeuseful,butitwouldnotnormallybenecessarytocovereverypossiblelanguage.

Havingallthisadditionalinformationavailableinaformatprovidedbythesolutionsundertestwillalsobe useful to vendors when investigating issues that emerge during third-party tests. Testers areencouraged to provide vendors takingpart in their testswith adequate information to diagnose and,ideally,torectifyanyproblemsreportedintests–forexamplefailuretodetectorblockattacks–andhavingadequatelydetailedinternalloggingsimplifiesthisprocessconsiderably.

Muchof the information requiredby testers,andusedbyvendors todiagnoseproblems,maynotbeneededbyreal-worldusersaspartoftheireverydayuseofsolutions.Vendorsmayfindthat includingoptions in their products to enablemore detailed logging, or providing testerswith external tools to



4

gatheror collate the requireddata,will facilitatebetter testingandmayevenbeofuse in real-worldsupportdiagnostics.

LogEntryExamples

Herearesomeexamplesofthesortofevent-relatedcontentthatmayberequiredinlogs:

• Aneventoccurred

• Timeofevent

• AuniqueeventIDorreference

• Eventcategoryordescription

• Sourceororiginatoroftheevent(file,URL...)

• ThreatID/classification

• Action/staken[automatedaction,prompttouser,cloudlookup]

• Timetakenbetweeneventandresponse/action

Andherearesomeexamplesofproductrelatedcontentforlogging:

• Initializationtime

• Updatetime/version

• Versioninformation

Vendor/TesterCommunications

Inordertoproperlytestasolution’sfullrangeofcapabilities,automationisavitaltoolforenablingtestsexecutedona largerscaleandthuspotentiallymoreaccurate.Whendevelopingautomationsystems,testersneedtoanalyseproductbehaviourindetailandtunetheirautomationtooperateandmonitorthesolutionsproperly:forexample,byrespondingtopromptsforuserinteractionorcapturingpop-upsandotheralertsasevidenceofresponsestothreatevents.Tominimisetheadditionallabourrequiredindevelopingandmaintaining suchautomationsystems, itmaybenecessary forvendorsand testers tocommunicateopenlybothonhowsolutionsoperate,andonhowtestsarebeingrun.

Itisparticularlyimportantthatvendorskeeptestersinformedofanychangestohowproductsoperatewhichmayaffecttherunningofongoingtests.Significantchangeswouldinclude,butarenotlimitedto,areassuchasloggingformat,thestyleandpositionofpromptsorpop-ups,defaultconfigurations,andsystemrequirements.Forexample,whenanumberofproductswentfrompop-upstotoasterswhileanextensivetestwasbeingcarriedout, theneedtore-engineerthetest inmidstreamcausedsignificantdisruption.

AMTSOstronglyencouragesopenandtimelycommunicationsbetweentestersandvendors,particularlyon issueswhichmayaffecthowtestscanberun,and theorganizationexists inpartwith theexpresspurposeoffacilitatingsuchcommunications.



5

AutomationorScriptabilityofSolutionsDuringTesting

Toenabletesterstorunfully-automatedtestsinasefficientamanneraspossible,itwouldbeusefulforsolutiondeveloperstoprovideoptionsormethodstoruntheirproductsinfully-automatedmode,withnouserinteractionrequired.Thismightinvolveaconfigurationoptionwheretheproductalwaysappliesadefaultactionwhenauserpromptwouldnormallyberequested,andtestersmight find ituseful torun tests with various approaches to user response, comparing the protection offered by a solutionwhenauseralwayschoosesthemostparanoidoptionversusthemostpermissive.

Someproductsmayincludefulloptionsforautomatingbehaviourasstandard,butwheresuchcontrolsare not provided vendors should assist testers by providing them with ways of making the productoperate in a an automatedway. Exactly how this kindof functionality canbest be providedwill varyfrom solution to solution, but possible approaches would include command line switches for use intestingonly,externalconfigurationfiles,ortemplateswhichcanbeselectedbythetester.Itmayalsobenecessarytoprovidetesterswithspecialbuildsofsolutionssettooperateinpredefinedways.

When using such non-standard products or configurations, testers should be aware of the possibilitythatproductsmaynotaccuratelyreflectreal-worldbehaviour.

Herearesomeexamplesofstandarduserprofilesandhowtheymightbeapplied:

• paranoiduser–defaultstoblockingordenyinganyactivityqueriedbytheirsecuritysolution

• “confident”(power-user,super-user)–assumesallactivitiesareastheyintended,soallowsorwhitelistsbydefault

• homeuser–lesswell-informedandlessinterestedinspendingtimedecipheringmessages,andmore interested in getting on with what they were doing; generally permits actions despitealerts,butmaysometimesdenyifthealertisstronglyworded

• corporateuser –worried about getting into trouble for breaking corporatepolicies; generallyclicks‘deny’,butmayoccasionallystretchtherulesifawarningdoesn’tsoundtooserious

• privilegeduserwhomaybestretchingasysteminwaysthatanunprivilegeduserwouldn’tthinkof trying,perhaps forevaluationor to testahypothetical situation–mayhaveaveryspecifictestingobjectiveinmindthatrequiresunusualdetailinspecificconfigurations.

• drunkenuser–selectsarandomoptioneachtimeasolutionpresentarequestforadecision

SolutionFlexibility

Whilemanysolutiondevelopersmayprefertokeeptheirsolutionsassimpleaspossible,tolessentheriskofconfusingnon-expertusers,manytypesoftestingmayrequireconsiderablymoreflexibilityandcontrolinproductsinordertoproducequalitytestresultswithoutexcessiveeffort.

Someexamples of areaswhichmaynot usually be configurable in some typesof solution, butwhichmaycausesignificantproblemsfortesting,mightinclude:

• thesize,locationanddetailoflogging,asmentionedabove



6

• theabilitytoscannon-standardareas,suchasnetworkdrives

• theoptiontorunscanswithoutapplyinganycleaning,onlyloggingdetectionresults

• thescaleandlocationofquarantinefacilities–iflargequantitiesofsamplesarebeingmovedtoaquarantinefolderbyasolutionduringatest , itmaybenecessarytorelocate itsomewhereotherthanthestandardordefault location.Sometestersmayalso find itusefultohaveeasyaccesstothecontentsofquarantinefolders,tohelpdiagnosethebehaviourofsolutionsduringtests.

ToolSharing

AMTSOencouragesvendorstobeopentorequestsforchangesfromtesters,andtodowhatevertheycantoimprovethetestabilityoftheirsolutions.

Forexample,vendorsmayalreadyhave,ormaybeabletoeasilyproduce,toolswhichwouldbeusefulto testers. These could include automation tools used in QA testing, log processing utilities, and fileanalysisorsystemmonitoringtoolsforuse insamplevalidationorclassification.Testersmayalsofindcommand-line versions of malware scanners useful for sample classification purposes. Vendors areencouragedtoassisttesterswherevertheycanbymakingsuchtoolsandutilitiesavailable.

ThisdocumentwasadoptedbyAMTSOonMay4,2011



7

AppendixAProposedXMLSchemaForLogFilesproductinfo company information product-specific information version signature-version (or versions, where multiple engines used) updates(s) information: last applied, last asked [true/false attribute?] environment (optional: tester should know this anyway) OS CPU RAM timestamp start finish object url domain ip-address path&file @md5 (and other hashes) process registry network (object or event? Product-specific?) ?items within archive Classification/ threatID malicious-file malicious-url phishing dynamic spam IM Etc. [vendor to classify] action-taken blocked quarantined disinfected deleted user-interaction



8

action-result success failed [Threat category (if not specified above)] XMLdraft<product> <Name>...</Name> <Update>...</Update> <OS>...</OS> <object> <name>...</name> <type>File/URL/Firewall</type> <date>...</date> <detection>...</detection> <actions> <action> <date>...</date> <name>...</name> <result>...</result> </action> ... ... ... <action> </action> </actions> </object> </product>