software validation of applications deployed on windows azure platform

7/31/2019 Software Validation of Applications Deployed on Windows Azure Platform

1/29

Software validation of applicationsdeployed on Windows Azure

Sidharth Subhash Ghag, Divya Sharma, Trupti Sarang

May 2011


2/29

Abstract

As a very popular Infosys adage goes, In God we trust, rest all we test , itreflects the importance of testing in the software development life cycle.

According to Wikipedia1

, Software Testing can be stated as the process of validating and verifying whether a software program/application/product - one, itmeets the business and technical requirements that guided its design anddevelopment; two, works as expected and three, can be implemented with the samecharacteristics.

As software programs get complex, testing becomes vital and error prone.

Testing an application on cloud is quite different from testing an application in thetraditional on-premise environment. Microsoft Azure TM is one such cloudplatform that runs applications on the cloud.

From a testing perspective, Test environment setup on Azure TM is quick andeasy which can save considerable time and effort as compared to testinfrastructure setup and administration. However by design, the Azure TM

platform imposes certain restrictions on services deployed on it. Thus testing anapplication hosted on Azure TM becomes difficult and complex as compared withtheir traditional on-premise counterparts. In this paper we shall explainchallenges along with approaches to handle different aspects of testing onAzure TM . Also we shall provide an insight into the end-to-end testing lifecycleinvolved for testing applications to be deployed on Windows Azure TM .

2 | Infosys - White Paper


3/29

Table of Content

Introduction ......................................................................................................................................3

Role of testing in development of Azure cloud applications ......................................................... 5

Comparison between practices of testing traditional on-premise vis-a- vis Azure applications .... 5

Testing Azure applicati ons ............................................................................................................ 11

Unit Testing ................................................................................................................................................................ 11

Integra on Testing ..................................................................................................................................................... 12

System Testing ........................................................................................................................................................... 13

Performance Testing .................................................................................................................................................. 15

Security Testing ......................................................................................................................................................... 17

Compliance & Regulatory Testing .............................................................................................................................. 19

User Acceptance Testing ........................................................................................................................................... 20

Benefits realized a er applica on valida on with Azure ............................................................... 22

Challenges of testing applications on Azure ................................................................................ 23

Organization Policies Are Not Implicit For Hosted Applications ................................................................................. 23

Development Challenges ............................................................................................................................................. 23

Connectivity ................................................................................................................................................................ 25

Summary .......................................................................................................................................... 26

Reference ......................................................................................................................................... 28

Infosys - White Paper | 3


4/29

Introduction

Testing an application is a very important aspect of software development and so is the case with the applications deployed

on Microsoft Azure TM. Similar to the development and testing tools available for on-premise applications, Microsoft

offers tools for application development and testing on Azure TM Platform as well. Microsoft offers a simulation of the cloud

environment for developing applications locally through the development fabric. Unit and integration testing can be

performed in a manner similar as it is performed for an on-premise application. But, testing after deploying an application on

Azure TM is different from the approach of testing an on-premise application. Applications deployed on Azure TM are managed,

governed and controlled by the Windows Azure TM Fabric controllers. The controllers impose certain infrastructure

restrictions, defined in the form of role definitions, at different levels of the platform and which limit flexibility otherwise

available with on-premise systems. The levels of flexibility available to account owners on Azure TM would differ based on

the Azure TM role definitions - Web, Worker or VM (Virtual Machine) Roles. Broadly these allow Azure TM to offer cloud

service model capabilities such as Infrastructure as a Service (IaaS) and Platform as a Service (PaaS).

To offer the IaaS service model, Microsoft announced VM role in PDC 2010 with enhanced control and flexibility on a

virtual machine instance. VM role is offered to support development and migration of Windows server applications which would

require the creation of a custom Windows server image, installing required software, applications and managing it as well. The VM

role can be used to realize scenarios where legacy applications built on design practices more suited fo r onpremise deployments that

needed to be moved to Azure TM Cloud .

Whereas In the PaaS service model offering, Azure TM offers a ready platform for directly deploying and hosting applications

without having to worry about installing or managing the software required to run these applications. In this model, the

platform provides an abstraction layer comprising of application runtime services and tools over the underlying

infrastructure. This layer offers a more managed and standard environment on which applications that would run with the

available IaaS service models. One can make use of the web and worker roles to leverage the PaaS service models offered by

AzureTM

. However one must note that with high abstraction levels offered by the web and worker roles, Az ure has accesscontrols in place which may restrict the free movement of applications from on-premise to cloud. These restrictions would also

apply while validating applications deployed under this service model on Azure TM.

When one needs to test an Azure application, it should be tested on both development and test environments. In the

development environment users can test an application on the development fabric and development storage, using Visual

Studio test suite capabilities. Once the application is hosted on the Microsoft Azure TM Platform, the scope of testing

becomes quite restrictive, which is enforced by the design of Azure TM Platform. At the same time, a ready-to-use platform

saves time and additional cost in setting up various test environments. Setting up environment and scaling it as desired is only

a few clicks away.

In this paper, we shall discuss the methodology and approach to test next generation cloud based applications on the

Windows Azure TM platform. The scope of the discussion would be limited to software validation of applications developed on PaaS

specific offering of the Azure TM platform i.e. apps leveraging the web/worker role models. We highlight the key challenges faced

with developing PaaS applications along with some approaches to overcome them. IaaS - style

applications, developed using Azure TM Virtual Machine roles, will not be in the scope of this paper.

http://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspx


5/29


6/29

As traditional application testing requires various test environments to set up e.g. development, system test, performance test,quality assurance, user acceptance test, production etc. Whereas, since Microsoft Azure TM platform offers Platform as aService , it gives liberty to application developers to host applications on a ready to use hosted platform without worryingabout infrastructure management. However, they get limited privileges on infrastructure. These features affect the approachof testing Azure TM applications as compared to that with the traditional application testing methods. The following diagramshows different deployment environments used and roles involved in the typical testing life cycle of a traditional on-premiseapplication compared with that of an Azure TM application. These figures are representative and actual scenarios may differbased on project requirements.

Developers

UnitTes ng

UnitTes ng

UnitTes ng

Func onal Tes ng Non - Func onal Tes ng

Per ormanceTesters

SystemTesters

Developers PerformanceTes ng

SecurityTesters

Integra on SystemTes ng Tes ng

Security & ComplianceTes ng

Development and Test

Client End User

User Produc onAcceptance Environment

Tes ng

Produc on

Figure 3: Testing approach for an on-premise application

The above figure shows the environment setup required for various stages of an end-to-end testing lifecycle for anytraditional on-premise application. As you can observe one would be required to set up separate test environments withrequired hardware and software installations for the various types of testing. This adds as overhead in terms of efforts and costinvolved in managing various test environments.



7/29

Figure 4 : Testing approach for an Azure TM application

There are two deployment regions in one hosted Azure TM instance, i.e., Azure TM service viz. staging and production. Any of thesetwo deployment regions of an Azure TM service can be used for testing the application based on the availability. Productionand staging are exactly identical with respect to hardware and installed software. In the above diagram, we have considered ascenario where we have used two Azure TM services in a project life cycle - one for testing and another for release. As depictedin the illustrative scenario above, testing on Azure TM environment saves considerable effort and time by eliminating the overheads intest environment setup and administration.

In the traditional testing approach, test environment setup would require setting up several machines with required software,

whereas in AzureTM

it will only require provisioning an AzureTM

account. Creating new AzureTM

service will implicitly allocatehardware resources from a shared pool along with the required software installations required to run your applicatio n in a matter of minutes.

A comparative view highlighting the differences of the effort required for testing on traditional on-premise development vis-vis Azure TM cloud development is shown below:

Traditional Application Azure Application

Prepare test strategy Prepare test strategyTest plan

Plan for test environment Plan for test environment

Procure hardware of desired configuration andrequired software licenses

Test Identify test lab (Physical Location)environmentsetup Infrastructure management team will setup

hardware and install software

On-demand provisioning of Azure account

Not Applicable (NA)

Not Applicable (NA)

Perform a quality check of environment setup Not Applicable (NA)



8/29


9/29

Test steps mentioned in the test environment setup service 1 (procured for testing)

Deploy the application on performance test Deploy the application on productionenvironment deployment region on Azure TM service 1

Develop Performance Test Scripts using the Develop Performance Test Scripts using theidentified Performance Test Tool. identified Performance Test Tool.

Perform Dry Runs Perform Dry Runs

Setup Performance Test Data Setup Performance Test Data

Run Performance Tests Run Performance Tests

Capture performance metrics Capture performance metrics through AzureTM

diagnostics

Analyze output and find out parameters to Analyze output and find out parameters toimprove improve

Modify application and deploy again Modify application and deploy again

Run Performance Tests run post performance Run Performance Tests run post performancedefect fix defect fix

Setup security test environment. Repeat steps Use staging deployment region of Azure TM

mentioned in the test environment setup service 1 (procured for testing)

Deploy the application on security test Deploy the application on staging deploymentenvironment region of Azure TM service 1

Security Test Security Testing Security Testing

Find out any security threats, system Find out any security threats, systemvulnerabilities vulnerabilities


Security Tests run post defect fix Security Tests run post defect fix

Compliance Testing Compliance Testing

Find out any non-compliances as per the Find out any non-compliances as per theCompliance organization and Government policies such as organization and government policies such asTest data location, security audit etc. data location, security audit etc.


Conduct Audits post defect fix Conduct Audits post defect fix

Setup user acceptance test environment Use staging deployment region of AzureTM

service 2 (procured for release)

UserAcceptanceTesting

Deploy the application on user acceptance testenvironment

Smoke Testing

User Acceptance Testing

Deploy the application on staging deploymentregion of Azure TM service 2

Not Applicable (NA)

User Acceptance Testing

Find out any defects Find out any defects




10/29

Conduct tests post defect fix Conduct tests post defect fix

Setup production environment Use production deployment region of AzureTM

service 2 (procured for release)

Setup datacenter - real estate, utility services,communication services, security management, Not Applicable (NA)government and environmental approvals, etc.

Align teams- infrastructure, database, Align teams - developers, applicationapplication administrators, developers administrators

Procure and install hardware and network resources- switches, routers, servers, cables, Not Applicable (NA)racks, storage, H/W load balancers

Upgrade to Procure and setup software licenses - operatingProduction system, application server, database, firewalls, Not Applicable (NA)

antivirus, patches, etc.

Validate the production environment for security

threats and compliances

Not Applicable (NA)

Harden production environment setup Not Applicable (NA)

Install operations tools for infrastructure Subscribe to cloud management vendors formonitoring and management application management and monitoring

Prepare deployment scripts for application to be Prepare deployment package for application todeployed - MSI s be deployed on Azure TM - cspkg

Deploy application to production environment Swap deployment from staging to production

Table 1 : Comparison of efforts in testing on-premise and Azure TM applications

Apart from test environment setup, there are dissimilarities in the processes of testing life cycles as well. The table belowdescribes the differences in each phase of the testing life cycle for a traditional on-premise application and an Azure TM

application.

Traditional Application Azure Application

Test Plan Prepare the test plan Prepare the test plan

Test Design Define test strategy, goals and approach Define test strategy, goals and approach

Test Execution Separate test environment setup needs to bedone for development / Test / QA / Production

Hardware and Software are to bemaintained

Desired software has to be installed

Exploring database for test verification iscomparatively simple

Production size instances to be provisioned fordifferent test environments

Hardware and Software details are abstracted and acomputation and storage services are provided.

Standard MS software components such as IIS, .NETframework etc... are pre-installed with the Azure TM

compute services However installing third partycomponents would have to be done by the testengineer by enabling Administrator privileges on thecloud services.

Although in latest releases MS has provided manyaiding tools, accessing and exploring Azure TM

10| Infosys - White Paper


11/29

Storage o r SQL Azure is complex as compared totheir on- premise counterpart. For SQL Azurecertain features of SQL Server and managementstudio are not available and for Azure storage,developers may need to use third party tools such asCloudber ry, Azure storage explorer, Cerebrata sCloud storage studio etc.

Scaling the environment will require extra Azure can be scaled dynamically (both vertically hardware and software setup taking either as well horizontally) by modifying number of the vertical or horizontal scaling route. running instances without any overhead

Test Analysis Debugging is easier than cloud applications Debugging cloud applications is a challenge

Functionality rich tools are available for Being a nascent technology, the testing toolsdefect analysis, debugging, trouble shooting available are limited

Table 2: Testing life cycle for on- premise and Azure applications

Testing Azure applicationsThis section explains how various testing cycles are performed on Azure applications . Testing is carried out in bothdevelopment as well as the hosted Azure instance; following sections describe the overall process of testing an Azureapplication:

Unit Testing

Unit testing is a verification and validation process for a unit of source code, carried out by the developers to ensure the code isworking as intended. It is performed for an Azure application in the same way as it is done for a traditional application. T hediagram below shows the steps followed in each phase of unit testing.

Figure 5 : Steps in Unit Testing

Azure applications can leverage the test feature of Microsoft Visual Studio to auto generate unit test cases from theAzure source code. Testers can provide input and expected result values to run tests and verify the application logic.

http://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspx


12/29

Figure 6: Comparative View for unit testing

These test cases are executed on development fabric which is a simulation environment of the Windows Azure tosimulate the Azure fabric locally. Unit testing of Azure applications doesn t require any additional setup. The diagram belowshows a comparative view of people, environment and tools required for unit testing an on-premise application vis--vis that of an Azure application.

Integration TestingIntegration Testing is performed on integrated modules comprising of individual unit tested modules. Integration testing

ensures that the modules communicate right with each other and integrated modules work as specified in high level design. Thediagram below shows the steps followed in integration testing.

Figure 7: Steps in integration testing



13/29

The high level design is input of integration testing and integration test plan is prepared based on this. The individual unittested modules are integrated based on the approach chosen to integrate, i.e., top down (the top module is tested first and then itssubordinate modules), bottom up (the low level modules are tested first and then the top level ones calling them) andtested to ensure conformance with high level design.

For an Azure application, various unit tested modules will be integrated on development fabric and integration testing isperformed on development fabric using the visual studio web tests and/or manual tests. It ensures that application interfaces areworking properly and integration did not cause any issues in different modules. This is a part of functional testing of application.Once it is confirmed that application is working fine on development environment then it is moved to Azure cloud instance.The diagram below shows a comparative view of people, environment and tools required for integration testing an on-premiseapplication vis--vis that of an Azure application.

Figure 8: Comparative View for integration testing

System TestingSystem testing is performed on an application with all its disparate modules integrated to form the system as a whole.Systems tests are carried out to ensure that the system fulfills all the specified functional requirements which the applicationis expected to meet. In the traditional on-premise model, for system testing of an application, a separate system testenvironment is commissioned, pre-requisite hardware and software infrastructure setup is installed. . Whereas for systemtesting of applications deployed on Azure Cloud, test environments need not be commissioned separately. A single cloudcomputation service provisioned on Azure comprises of a production and a staging environment and the system test isperformed in the staging region of this service. The diagram below illustrates the steps followed in system testing of an

Azure application.

http://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdfhttp://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdfhttp://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdfhttp://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdfhttp://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdf


14/29

Figure 9: Steps in system testing

System testing is to extensively test the functionality of application. Once integration testing of the application is perfor med ondevelopment fabric, it is then hosted on Azure staging environment and application functionality is tested again.

Manual system test cases are created against high level design of the application (Use cases).Web tests created using Microsoft Visual Studio web test features to test the functionality of the application ondevelopment environment during integration test can be reused.

The web tests created on development environment can be utilized by changing the application URL in test cases tothe hosted application URL. The path of application will be modified using context parameters and tests can beexecuted for application hosted in staging deployment region of Azure account.

Since hosted environment provides restricted privilege access to users, system metrics need to be accessed through the Azure diagnostics feature.

Proper code instrumentation will be required for trouble shooting of hosted application. Developers will have towrite tracing and logging code in the application for writing out information through Azure logging APIs forapplication monitoring and analyzing defects (if any).Similar to investigating on-premise, applications developers have access to the event viewer, system logs, IISlogs etc., to analyze application issues. However in Azure such facilities are not directly accessible and hence

Azure diagnostics API (Microsoft.WindowsAzure.Diagnostics) would have to be relied upon. Users can alsoconnect to an instance of their application deployed on Azure using remote desktop, monitor and trouble shootthe running instance.

The diagram below shows a comparative view of people, environment and tools required for system testing an on-premiseapplication vis--vis that of an Azure application.



15/29

Figure 10: Comparative View for system testing

Performance TestingPerformance testing is conducted to confirm how the system would perform under certain workload. Application behavior istested in terms of various performance aspects under different workload conditions. This includes scalability, reliability,availability and response time of the application, ability to handle load, spikes and throughput. The diagram below lists the stepsfollowed in performance testing of an Azure application.

Figure 11: Steps in performance testing

In Azure applications, performance testing plays a vital role. Since the infrastructure is abstracted and a standard platformis offered, it becomes essential to gauge the performance of application deployed on the Azure platform. The number of compute instances and VM size will also affect the application performance. Since customers have to pay for number of



16/29

specified sized compute instances running at a time, the optimum use of Azure compute instances becomes important forcost-effective utilization of the Azu re owned resources. Performance evaluation of application will help derive the factorswhich can aid in optimizing these resources with respect to cost and performance measures. Azure supports dynamicscalability which makes it easy to test application performance at varying scales which is made available on-demand.Procuring the compute instances and releasing it based on the load is a manual process. But once requested for resources it will beavailable in no time. It can be automated using some third party applications which continuously monitors the load of deployedapplications and does the resource management.

Performance testing is performed on hosted Azure instance because application performance should be tested in anenvironment closely similar to the production environment. Based on the performance requirements, test plan is prepared,performance scripts developed and the approach to execute performance test scripts and capture performance metrics isdecided. Performance metrics are captured with performance test scripts execution and analyzed. Since Azure platform is amanaged environment, capturing of performance metrics in not easy. Performance parameters are captured using Azurediagnostics with support of diagnostic monitor. The diagram below shows a comparative view of people, environment andtools required for performance testing an on-premise application vis--vis that of an Azure application.

Figure 12 : Comparative View for performance testing

Though performance testing is usually performed after system test, the performance goal should be well defined right from theapplication architecture, design and build.



17/29

Security TestingSecurity testing involves testing the security at two levels -

Platform Security Testing - To test the platform where application is hosted, is not easily vulnerable to security attacks.Application Security Testing - To test that application is not easily vulnerable to security attacks.

The Azure platform offered by Microsoft is a managed and standardized environment which is protected by the Azure

fabric controllers, network security and personnel level checks and controls making them highly secure. As per Microsoft 6

, Windows Azure operates in the Microsoft Global Foundation Services (GFS) infrastructure, portions of which are

ISO27001-certified. ISO27001 is recognized worldwide as one of the premiere international information securitymanagement standards. Also Azure being offered in a shared model, Microsoft today does not allow performing anyplatform security tests so as to avoid it affecting other tenants hosted on the same host. However the same cannot be saidabout the application hosted on Azure . Hence as a part of this phase the focus should primarily be on application securitytesting.

Application security testing is to ensure that application provides the right data and operations to the right person. Theconfidentiality of data should not be compromised with the functionalities of application. Whatever functionalities are added tothe application, should allow secure access to data and operations. Exposing the applications over internet involves moreconcerns of security since any flaws within the application can make the application vulnerable to security attacks. Thismakes security testing essential for Azure applications.

Figure 13 : Accessibility for on-premise and Azure applications

Following are the security aspects to verify in an application security test -

Confidentiality: Protect against the disclosure of information to parties other than the intended recipient

Integrity: Ensure received information is correct

Authentication: Ensuring received information is from intended known user

Authorization: Allow access to intended users

Availability: Availability of application in case of denial of service attacks and distributed denial of service attacks

Non-repudiation: measures should be taken before the concerned parties deny about an action occurred, a message sent by

sender, or received by receiver, etc.



18/29

Figure 14 : Steps in security testingIn traditional security testing almost all security measures can be implemented through firewall. Once the application isdeployed, all requests are passed through the firewall and application maintenance team can restrict the application serverfrom users accessing the application. Using this application, the deploying team has control over the users accessing theapplication. Azure is a common platform for deploying the application. The user can access the application hosted oncloud ubiquitously.

There are firewalls installed on Azure platform but it is managed by Microsoft . The application developer and deployingteam has the least access to infrastructure and they cannot control the firewalls behavior. Application security testing mus t

be conducted on hosted Azure instance to make sure that the application in production will work as intended. Theprocesses of testing an on- premise and Azure application are quite similar but since the application and data do not remain

within an organ ization s conrol, application security testing becomes a major concern. The diagram below shows acomparative view of people, environment and tools required for security testing an on -premise application vis--vis anAzure application.

Figure 15 : Comparative View for security testing



19/29

Compliance & Regulatory TestingCompliance & Regulatory testing is conducted to ensure that an application does not violate any organization, industry orgovernment regulations that might lead to legal and possible financial implications. Compliance and regulatory check becomes imperative for the applications hosted on cloud since an application hosted on the public cloud is operated andmanaged by a third party. The cloud service providers should provide the required compliance such as information about datalocation, security audits. Hence compliance and regulatory testing is required to ensure that the systems and processes of the

cloud service provider have to be compliant with that as is expected by the domain in which the application would operate in.The system must have supportive documentation and/or certification compliance to be handed over to the regulatory agencies(if applicable).

Figure 16: Steps in compliance and regulatory testing

Below is the list of common compliance requirements of IT systems -

Sarbanes-Oxley Act (SOX) 3 - The Sarbanes -Oxley Act of 2002 sets new or enhanced standards for all U.S. publiccompany boards, management and public accounting firms. The legislation affects both the financial side of corporations and the IT departments.

Section 404 of SOX requires management and the external auditor to report on the adequacy of thecompany's internal control over financial reporting (ICFR).

Section 802(a) of the SOX states, whoever knowingly alters, destroys, mutilates, conceals, covers up,falsifies, or makes a false entry in any record, document, or tangible object with the intent to impede,obstruct, or influence the investigation or proper administration of any matter will be panelized.

Similar to SOX act in US other countries also have similar act - Japan - J-SOX

Canada - Bill 198 Germany - German Corporate Governance Code Australia - Corporate Law Economic Reform Program Act 2004 (CLERP-9) France - Financial Security Law of France Italy - L262/2005 South Africa - King Report

http://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspxhttp://www.infosys.com/cloud/partners/Pages/microsoft.aspx


20/29

Health Insurance Portability and Accountability Act (HIPAA) 4 - Title II of HIPAA, known as theAdministrative Simplification (AS) provisions, requires the establishment of national standards for electronic health caretransactions. This is intended to help people keep their information private and secure.

ISO/IEC 270014- ISO/IEC 270015 is an Information Security Management System (ISMS) standard whichrequires that management follows the required measures to keep the information secure in case of security threats andvulnerabilities.

Payment Card Industry Data Security Standard (PCI DSS)6

is a worldwide information security standarddefined by the Payment Card Industry Security Standards Council. The standard was created to help organizationsthat process card payments prevent credit card fraud through increased controls around data and its exposure tocompromise.

Most of the compliances require that data security and privacy must be maintained and to ensure this they need to conductsecurity audits in a timely manner. The application must follow the required regulatory compliances which client needs tofollow as per their legislation. The developers would need a test to ensure that it follows required regulatory compliances. Thediagram below shows a comparative view of people, environment and tools required for regulatory compliance testing anon-premise application vis--vis that of an Azure application.

Figure 17: Comparative View for compliance and regulatory testing

User Acceptance TestingUser Acceptance testing is similar to system testing but this is performed by the application end -user before accepting thesystem. Acceptance testing for Azure applications must be performed on the application deployed on the hosted Azureinstance. The user acceptance test environment should be closely similar to the actual production environment.



21/29

Figure 18: Steps in user acceptance testing

User acceptance testing approach is quite similar for an on-premise application and Azure application. The diagram belowshows a comparative view of people, environment and tools required for user acceptance testing an on-premise applicationvis--vis that of an Azure application.

Figure 19: Comparative View for user acceptance testing

Once all the defects are fixed and system is tested, it is deployed on the production deployment region on Azure for users.



22/29

Benefits realized after application validation with AzureThe platform provided by Microsoft Azure offers the capabilities which ease testing applications on Azure. Following are thebenefits derived -

Testing environment setup made easy - using Microsoft Azure one can easily set up the kind of environmentrequired to test an application. The application can be easily configured to use specified number of compute or storage

instances on Azure as well as the specified VM size. Apart f rom this, various test environments would not requireany additional efforts such as system hardening, resource access lock down, etc., to ensure that the platform has to bemade ready to be hosting applications. This will also eliminate administrative overhead of managing multiple testenvironments.

Abstraction for infrastructure - As compared to traditional on-premise applications where setting up testenvironment would mean procuring server class hardware and installing software licenses, Azure provides aplatform where a user can directly host applications and not worry about setting up the infrastructure. Infrastructure leveldetails are abstracted and are managed by Microsoft . Since acquiring test and production environments would be easy andtime to market of the applicationwould be significantly faster.

Standard environment for all test environments on Azure - Microsoft Azure offers a standard platform,therefore all test environments setup on Azure will be identical. In traditional testing approach this is a commonissue that application works fine in testing but fails in production. One major and common reason is that testing andproduction environments are n ot same. In Azure the staging and production environments are exactly same.Hardware configuration and software installed are consistent throughout and it helps avoiding this kind of issues.

Effective test environment management - The Azure platform is well managed and has robust mechanism forensuring security, recovery and high availability in place out-of-the-box. Providing these features in the traditional set up willrequire considerable planning, effort and specialized skills. But with Azure the QoS criteria mentioned and demanded byany enterprise application is delivered implicitly as a part of the Azure services offerings.

Dynamic scalability - While developing or testing an application if user wants to scale the application, this willrequire setting up new servers along with possible modifications in the application. On Azure scaling is just a matter of modifying number of compute and storage instances required by application.

Control over Operating System Updates and Upgrades - Customers can control which operating system updates toapply to their hosted Azure instances. With this customers can ensure that any patches or updates that break their application are not rolled out or the required changes are made in application beforehand.

Cost effective - When setting up environment for a traditional testing approach, acquiring hardware, installingsoftware, maintaining infrastructure will add the same cost as setting up actual production environment. Whereas inAzure char ges are based on utilization, thus saving a lot of time and manpower and resulting in a cost effectivesolution.



23/29

Challenges of testing applications on AzureDespite the fact that Microsoft Azure platform has got potential for hosting applications without worrying aboutinfrastructure level details, it has certain limitations also. With the abstraction comes the lack of control. Microsoft provides restrictedaccess privileges to users on the Azure platform. Some of these limitations cause difficulties in performingapplication tests on Azure. These challenges are described as follows -

Organization Policies Are Not Implicit For Hosted ApplicationsIn a traditional approach, all incoming and outgoing network traffic will have to pass the organization firewall and certainsecurity measures and policies are implicit with this. Organizations apply many security policies through firewalls like thekind of response data than can be sent outside or the type of incoming requests. When an application is hosted on Azure, itis no longer in the confinement of the organization, so enforcing the organization policies through firewalls is not possible.Testers will have to perform proper testing to ensure these otherwise implicit parameters are validated and function asexpected.

Challenge Workaround/ Supportive Tools/ Recommendation

Firewall restriction Though Microsoft Azure provides a highly secure e nvironment, any othersecurity aspect such as imposing site access restrictions, secured transfer of confidential enterprise data, etc., will have to be taken care of by the

application and testers will need to test it.

Policy enforcement Organization policies can t be enforced implicitly since applications run out of the boundaries of an organization s IT system. Application will have to take care of

and the testers will need to test it.

Table 3: Challenges in enforcing organization policies and their workarounds

Development Challenges

For an on-premise application, analyzing a bug, debugging and trouble shooting is comparatively easy with support of numerous Microsoft tools, open source tools and third party tools available today. Since Microsoft Azure is a newplatform; this kind of support is currently not available. SQL server provides a proficient way to access database, run profiler,check indexing, database tuning, checking query execution plan etc. SQL Azure doesn t support that level of featurestoday.

Azure as of today has very limited tools available that can support testing Azure applications. Generation of test data, ca pturing

results, capturing performance matrices, analyzing test results, etc., are very easy and convenient in on-premise applications using

various tools. Unavailability of such tools hampers the overall productivity and amount of efforts by developers and testers.



24/29

Challenge

Less support of debuggingand troubleshooting

Limited tools available forsecurity testing

Limited tools available forperformance testing

Database access and analysis

Test data creation

Interop APIs

Workaround/ Supportive Tools/ Recommendation

Windows Azure diagnostics feature can be used for code instrumentation of Azure applications.

Developers can add additional functionality in the application to view systemevent logs, IIS logs and performance monitor logs using Windows AzureDiagnostics feature.

Platform level security is in the control of Microsoft . Application levelsecurity measures such as role and membership based access will have to betaken care of by the application developers.

Windows Azure diagnostics feature along with Diagnostic Monitor (typeperf utility) of Windows Azure can be leveraged to collec tperformance parameters of a hosted application.

SQL Azure doesn t support full fledged features of SQL server. e.g. profiler,

query execution plan, etc. Apart from SQL 2008 R2, Microsoft

provides anonline database management tool with project name Houston for DBoperations on SQL Azure.

Test data can be created manually or programmatically for Azure storageand SQL Azure; inbuilt tools are not available. Microsoft SQL AzureData Sync tool can be used to replicate database from SQL server to SQLAzure.

The applications using interop APIs are not supported on Azure since theserequire APIs to be registered. This kind of scenario cannot be supported onAzure. Such applications will have to look at hybrid approaches (mix o f on-

premise and Azure deployment) to build applications on Azure.

Table 4: Development challenges and their workarounds

Troubleshooting as a Tester

Debugging and trouble shooting an Azure application in the development environment works the same way as that of anon- Azure application. One can attach debugger to the executing code and debug it in Visual Studio. Or one can useTrace.Write() to get trace related information,

If an applicati on hosted on Azure platform has to be debugged, one can use below methods -

1. Use IntelliTrace feature of Visual Studio 2010a. Enable IntelliTrace for roles in Azure application

b. Publish application on Azure subscription

c. After deployment, open server explorer and expand add current deployment

d. Open IntelliTrace in Windows Azure Compute for deployed application

e. View IntelliTrace logs and call stack

f. Disable IntelliTrace once debugging is done and before deploying the Azur e application in production.



25/29

a. Write traces in application or modify application configuration (.cscfg) to capture performance counters,Windows event logs, IIS logs, application crash dumps, infrastructure logs etc.

b. Logs will be written into blob and table storage

c. Use tools such as Azure Storage Explorer, Windows Azure Service Management CmdLets,Windows Azure MMC, etc. to view logs

3. Use Remote Desktop

a. Before publishing the application, configure remote desktop connections and enable connections for allroles

b. Provide certificate, user name and password for connection

c. Go to Azure management portal, enable remote access and connect to a role

d. Download .rdp file and open it. Provide user name, password and connect to the role with remote access.

Connectivity

Internet availability is the basic requirement for developing and testing an application on Azure. If Internet is down, test erswon t be able to perform cloud testing. Hence, it is mandatory to have internet connectivity all the ti me to carry out testing on

the cloud.

There are several services of Microsoft Azure such as Windows Azure Platform for hosted application and Azure storage(Blob, Queue, and Table), AppFabric service bus, Windows Identity Framework, Access Control Service, SQL Azure etc.,which an application may be using as building blocks. If an Azure application is using any of the above mentioned servicesthen the connectivity to those services and availability of the services needs to be checked before initiating tests. Say inenterprises, owing to security threats access to several external sites are controlled and more often blocked. They have a lockeddown environment where many of the ports are also disabled. If an application uses AppFabric service bus, then, ensuring thatrequired ports to connect to the service are open is necessar y. Apart from this the requests going to and responses coming fromapplication hosted on Azure will pass through the organization firewall. Testers will have to take care of this aspect also.Corresponding ports will have to be opened and firewall policies might have to be amended for sending requests and receiving

responses from an organization.

Challenge

Internet connectivity

Connectivity to Microsoft Azure services

Workaround/ Supportive Tools/ Recommendation

The tester needs to ensure the proper internet connectivity for smooth testingoperations.

Connectivity to the Azure blocks e.g. Azure compute, Azure storage(blob, queue, table), SQL Azure, AppFabric service bus, Access controlservice etc. used by application needs to be ensured. It may require openingsome of the ports and few modifications in the firewall policies.

Table 5: Ch allenges in connecting to Azure services from an organization and their workaround



26/29

Summary Azure cloud computing offerings in the PaaS space offers developers a platform to not only host and run applications butalso a ready-to-use test bed which can help projects reduce the overheads with setting up any world class test facility whichhas been discussed in this paper and also further explained later in this section. On the application validation front, Azurebased projects will benefit in terms of reduced time, effort and cost involved in setting up the various test environments.However there are challenges on the other end, more so since the technology is new and the space still lacks tools andutilities for testing, debugging and troubleshooting and which we have highlighted in this paper along with strategies tomitigate them. The figure shown below summarizes the pros and cons of testing an application on Azure which developersneed to be aware of.

Figure 20: Pros and cons of application testing on Azure

Further to support our views on Azure capabilities of accelerating the testing lifecycle, the graph drawn below show a relative

comparison of total cost required for testing an on-premise application vis--vis to testing an Azure application. The graph isillustrative and may differ for different projects.

http://www.infosys.com/cloud/offerings/Pages/index.aspxhttp://www.infosys.com/cloud/offerings/Pages/index.aspxhttp://www.infosys.com/cloud/offerings/Pages/index.aspxhttp://www.infosys.com/cloud/offerings/Pages/index.aspx


27/29


28/29

Reference

Description

[1] Wikipedia

[2] Windows AzureSecurity Overview

[3] SOX Compliance

[4] HIPPA

[5] ISO/IEC 27001

[6] PCI Data SecurityStandard

MSDN

Testing in MSDN

Unit testing

Web test

Validation rule

SQL Azure Firewall

Azure Storage Explorer

OS versioning on WindowsAzure

Microsoft SQL Azure DataSync

Azure ApplicationTroubleshooting

Remote Desktop on Azure application


URL

http://en.wikipedia.org/wiki/Systems_Development_Life_Cycle http://en.wikipedia.org/wiki/Security_testing

http://www.microsoft.com/Windows Azure /whitepapers/papers/default.aspx

http://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act http://searchcio.techtarget.com/sDefinition/0, ,sid182_gci920030,00.html

http://en.wikipedia.org/wiki/HIPPA

http://en.wikipedia.org/wiki/ISO/IEC_27001

http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard

http://msdn.microsoft.com/en- us/library/microsoft.WindowsAzure.diagnostics.aspx

http://msdn.microsoft.com/en-us/library/dd286680(VS.100) .aspx

http://msdn.microsoft.com/en-us/library/ms182524.aspx

http://msdn.microsoft.com/en-us/library/aa337591.aspxhttp://msdn.microsoft.com/en-us/library/ms182539.aspx

http://msdn.microsoft.com/en-us/library/ms404670.aspx

http://www.Azuresupport.com/2009/12/sql -Azure -firewall-tutorial/ http://msdn.microsoft.com/en-us/library/ee621783.aspx

http://Azure storageexplorer.codeplex.com/

http://blogs.msdn.com/WindowsAzure /archive/2010/01/11/operating -system-versioning-in-Windows- Azure .asp x

http://www.microsoft.com/WindowsAzure /sql Azure /datasyn c/

http://msdn.microsoft.com/en-us/library/ff966484.aspx

http://msdn.microsoft.com/en-us/library/gg443832.aspx
http://en.wikipedia.org/wiki/systems_development_life_cycle/http://en.wikipedia.org/wiki/systems_development_life_cycle/http://en.wikipedia.org/wiki/security_testing/http://en.wikipedia.org/wiki/security_testing/http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%B7hitepapers/papers/default.aspx/http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%B7hitepapers/papers/default.aspx/http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%B7hitepapers/papers/default.aspx/http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%B7hitepapers/papers/default.aspx/http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%B7hitepapers/papers/default.aspx/http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%B7hitepapers/papers/default.aspx/http://en.wikipedia.org/wiki/sarbanes%e2%80%93oxley_act/http://en.wikipedia.org/wiki/sarbanes%e2%80%93oxley_act/http://searchcio.techtarget.com/sdefinition/0/http://en.wikipedia.org/wiki/hippa/http://en.wikipedia.org/wiki/iso/iec_27001/http://en.wikipedia.org/wiki/payment_card_industry_data_security_standard/http://msdn.microsoft.com/en-us/library/microsoft.windows%EE%A1%BAure%E9%AE%A4iagnostics.aspx/http://msdn.microsoft.com/en-us/library/microsoft.windows%EE%A1%BAure%E9%AE%A4iagnostics.aspx/http://msdn.microsoft.com/en-us/library/microsoft.windows%EE%A1%BAure%E9%AE%A4iagnostics.aspx/http://msdn.microsoft.com/en-us/library/dd286680(vs.100/http://msdn.microsoft.com/en-us/library/ms182524.aspx/http://msdn.microsoft.com/en-us/library/aa337591.aspx/http://msdn.microsoft.com/en-us/library/ms182539.aspx/http://msdn.microsoft.com/en-us/library/ms404670.aspx/http://www.azuretmsupport.com/2009/12/sql-azure%E9%AD%A6irewall-tutorial/http://www.azuretmsupport.com/2009/12/sql-azure%E9%AD%A6irewall-tutorial/http://www.azuretmsupport.com/2009/12/sql-azure%E9%AD%A6irewall-tutorial/http://www.azuretmsupport.com/2009/12/sql-azure%E9%AD%A6irewall-tutorial/http://msdn.microsoft.com/en-us/library/ee621783.aspx/http://azuretm/http://azuretm/http://blogs.msdn.com/windows%EE%A1%BAure%E9%AF%80http://blogs.msdn.com/windows%EE%A1%BAure%E9%AF%80http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%80http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%80http://msdn.microsoft.com/en-us/library/ff966484.aspx/http://msdn.microsoft.com/en-us/library/gg443832.aspx/http://msdn.microsoft.com/en-us/library/gg443832.aspx/http://msdn.microsoft.com/en-us/library/ff966484.aspx/http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%80http://blogs.msdn.com/windows%EE%A1%BAure%E9%AF%80http://azuretm/http://msdn.microsoft.com/en-us/library/ee621783.aspx/http://www.azuretmsupport.com/2009/12/sql-azure%E9%AD%A6irewall-tutorial/http://msdn.microsoft.com/en-us/library/ms404670.aspx/http://msdn.microsoft.com/en-us/library/ms182539.aspx/http://msdn.microsoft.com/en-us/library/aa337591.aspx/http://msdn.microsoft.com/en-us/library/ms182524.aspx/http://msdn.microsoft.com/en-us/library/dd286680(vs.100/http://msdn.microsoft.com/en-us/library/microsoft.windows%EE%A1%BAure%E9%AE%A4iagnostics.aspx/http://en.wikipedia.org/wiki/payment_card_industry_data_security_standard/http://en.wikipedia.org/wiki/iso/iec_27001/http://en.wikipedia.org/wiki/hippa/http://searchcio.techtarget.com/sdefinition/0/http://en.wikipedia.org/wiki/sarbanes%e2%80%93oxley_act/http://www.microsoft.com/windows%EE%A1%BAure%E9%AF%B7hitepapers/papers/default.aspx/http://en.wikipedia.org/wiki/security_testing/http://en.wikipedia.org/wiki/systems_development_life_cycle/


29/29

AuthorsSidharth Subhash Ghag ([email protected] ) is a Senior Technology Architect with the Microsoft TechnologyCenter (MTC) in Infosys. With over twelve years of industry experience, he currently leads solutions in Microsoft Technologies in the area of Cloud Computing. In the past he has also worked in the areas of SOA and service-enablingmainframe systems. He has been instrumental in helping Infosys clients with the service orientation of their legacymainframe systems. Currently he helps customers adopt Cloud computing within their Enterprise. He has authored papers onCloud computing and service-enabling mainframe systems. Sidharth blogs at http://www.infosysblogs.com/cloudcomputing

Divya Sharma ([email protected] ), Technology Analyst with Microsoft Technology Centre (MTC), Infosysand has more than 4 years of experience of developing and designing solutions in Microsoft Technologies such asWindows applications, ASP.net, web services, WCF services etc. Her current work includes exploring Windows Azureplatform and implementing applications using technology stack of Azure and deploying those on Azure platform. Divyablogs at http://www.infosysblogs.com/cloudcomputing

Trupti Sarang ([email protected] ), Senior Systems Engineer with Microsoft Technology Center (MTC) inInfosys. She has nearly 3 years of industry experience on Microsoft .NET. She currently works on implementing and

deploying projects on the Windows Azure platform.

AcknowledgementAuthors would like to acknowledge Vijayanathan Naganathan , Senior Technology Architect, Independent ValidationServices (IVS), Infosys Technologies Ltd. and Bhavin Jayantilal Raichura , Senior Technology Architect, Manufacturing IBU,Infosys Technologies Ltd. for their help in reviewing this paper.

Source: http://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdf

www.infosys.com
http://www.infosysblogs.com/cloudcomputing/http://www.infosysblogs.com/cloudcomputing/http://www.infosysblogs.com/cloudcomputing/http://www.infosysblogs.com/cloudcomputing/http://www.infosysblogs.com/cloudcomputing/http://www.infosysblogs.com/cloudcomputing/http://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdfhttp://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdfhttp://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdfhttp://www.infosys.com/http://www.infosys.com/http://www.infosys.com/http://www.infosys.com/cloud/resource-center/Documents/software-validation-applications.pdfhttp://www.infosysblogs.com/cloudcomputing/http://www.infosysblogs.com/cloudcomputing/