business objects calculation engine changes & regression testing

Track Calculation Engine Changes & Regressions in SAP BusinessObjects

Bruno Masek, VP, GB&SMITHAmy Betten, Director of Sales, GB&SMITH

• How do you track calculation engine changes? Alt+Tab ? Print Reports and put on top of each other 2 screens Asking Business Or no checking

STATE OF THE ART?

REGRESSION TESTING

WHEN?• Migration• Service Pack / Patch• Univers Update (UNV —> UNX)• ETL / Data-warehouse changes• Etc

3.1 Webi 4.2 Webi

3.1 Deski 4.2 Webi

4.1 SP74.1 SP5

ROI

• In house checking => Safety• No more manual checking • Automated and Scheduled process • Can be used by Business, QA & IT

ROI

USE CASE

While POC, a service pack was implemented 4.1 SP4 to 4.1 SP6. SP6 bugs + regressions were found due to Sybase DB upgrade.Upgrade was stopped, until Bind checked all reports for calculation engine changes/regressions & all issues fixed.Risk management was included in the loop/process and permanent automated regression testing became mandatory.

BI4.1 SP6 and Sybase upgrade

CENTRAL BANK

USE CASE

While POC, administrator ran the automated regression tool live on a few financial reports that had been checked by consultants. Admin found on some reports with data coming from same database : $1Billion margin in 3.1 and $1.2Billion in 4.1. Customer estimated 40+hours were needed to check certain reports each quarter due to service packs.

BI4.1 Quality check

REAL-ESTATE COMPANY

USE CASE

During a migration with very tight budget, City of Chicago implemented automated calculation engine/regression testing after report clean-up and conversion. As a result $200,000 were saved and regressions were found even on reports 100% converted.

Migration 3.1 to 4.2

CITY OF CHICAGO

USE CASE

500 reports (Webi-Deski-Crytal) complex reports needed to be checked. 250 man days were needed with the Alt+tab method. Automated process was implemented taking 15 days and finding 5% Calculation engine changes/regression.

Quality check of 500 reports

INSURANCE COMPANY

USE CASE

One of the largest company in the world. 10% Regressions found.

4.1 to 4.2 Migration

LARGE OIL & GAS COMPANY

USE CASE

4.1 had been implemented for a while. Automated calculation engine changes was implemented. They found out that for years they were regressions in 3.1 that were fixed in 4.1.

3.1 to 4.2

HUGE TELECOM COMPANY

USE CASE

Organization checks regressions randomly, part of QA control. Regressions found between Dev / Prod due to a table change.

DEV and PROD

HEALTHCARE ORGANIZATION

12,000 Documents730 Users

70 Universes

DETAILED REGRESSION TESTING USE CASE

SAP BusinessObjects XI3.1 SP7to

SAP Business Intelligence 4.2 SP2

Federal Agency

1. RUN THE 360BIND EXPORTS & COMPARISONS

2. IDENTIFY ALL KINDS OF REGRESSIONS

3. ANALYZE ROOT CAUSE OF REGRESSIONS

4. RUN IMPACT ANALYSIS ON ALL PLATFORM DOCUMENTS

1. Run Exports and Comparisons: 1.5 hours

EXPORT 1

Run Exports and Comparisons 1.5 hours

Create an Export Task

Select Documents

Run the 360Bind Exports and Comparisons 1.5 hours

Add Documents

Run the 360Bind Exports and Comparisons 1.5 hours

Select Frequency

Run the 360Bind Exports and Comparisons 1.5 hours

Select an Export Categoryor Create a New one

Run the 360Bind Exports and Comparisons 1.5 hours

Populate the Prompts

and Schedule!

Run the 360Bind Exports and Comparisons 1.5 hours

EXPORT 2

Export TaskXI 3

Run the 360Bind Exports and Comparisons 1.5 hours

Select the first export task

Export it into a Zip file

Import TaskBI 4

Run the 360Bind Exports and Comparisons 1.5 hours

Import the task saved

and Run it

COMPARISON

Schedule Comparison Job

Run the 360Bind Exports and Comparisons 1.5 hours

Select Export Category

Run the 360Bind Exports and Comparisons 1.5 hours

Working folder you exported the reports

Select Export Results

Run the 360Bind Exports and Comparisons 1.5 hours

Select Comparison Options

Run the 360Bind Exports and Comparisons 1.5 hours

1. RUN THE 360BIND EXPORTS AND COMPARISONS

2. IDENTIFY ALL KINDS OF REGRESSIONS

3. ANALYZE ROOT CAUSE OF REGRESSIONS

4. RUN IMPACT ANALYSIS ON ALL PLATFORM DOCUMENTS

2. Identify all Kinds of Regressions: 1 hour

Data - Structure - Images - Style28/28 documents with regressions

Identify all Kinds of Regressions1 hour

1. Vertical Alignment & Font Families

Identify all Kinds of Regressions1 hour

XI 3.1 BI 4.2

Data - Structure - Images - Style24/28 documents with regressions

Identify all Kinds of Regressions1 hour

2. Image: Label font, Scales and Layout

Identify all Kinds of Regressions1 hour

XI 3.1 BI 4.2

2. Image: Label font, Scales and Layout

Identify all Kinds of Regressions1 hour

XI 3.1 BI 4.2

Data - Structure - Images - Style17/28 documents with regressions

Identify all Kinds of Regressions1 hour

3. Data: Missing Decimals (48.32 48.00)

Business Expenses Details (in %)

Percentage Total< 100%!!!

Identify all Kinds of Regressions1 hour

XI 3.1 BI 4.2

3. Data: Error DIV/0

Business Expenses Details (in %)

Identify all Kinds of Regressions1 hour

XI 3.1 BI 4.2

1. RUN THE 360BIND EXPORTS AND COMPARISONS

2. IDENTIFY ALL KINDS OF REGRESSIONS

3. ANALYZE ROOT CAUSE OF REGRESSIONS

4. RUN IMPACT ANALYSIS ON ALL PLATFORM DOCUMENTS

3. Analyze root cause of Data Regressions: 1 hour• Document: “State A14 - Expenses Details”• Variable Analyzed: “C Fch Ttc”

• Variable is using “Round()” function included in “FormatNumber()” function: FormatNumber(Round(…))

Analyze root cause of Data Regressions1 hour

Analyze root cause of Data Regressions1 hour

Working fine in XIR3.1 SP7 but no longer in BI 4.2 SP2 due to Calculation Engine Modification

1. RUN THE 360BIND EXPORTS AND COMPARISONS

2. IDENTIFY ALL KINDS OF REGRESSIONS

3. ANALYZE ROOT CAUSE OF REGRESSIONS

4. RUN IMPACT ANALYSIS ON ALL PLATFORM DOCUMENTS

4. Run impact analysis on all the Documents: 0.5 hour

Identify all documents using “FormatNumber(Round(…))” formulas in variables or cells

596 Documents probably impacted by the Regressions

IMPACT ANALYSIS

Run impact analysis on all the Documents0.5 hour

• Identify which one are really used:

262 Documents to Fix for the Migration

28 documents troubleshot between versions:

• Service proposal from a BI consulting company= 3 days to test the 28 documents with manual comparison

• 360Bind and 360Eyes:1. Run the 360Bind exports and comparisons : 1.5 hours2. Identify all kinds of regressions : 1 hour3. Analyze root cause of regressions : 1 hour4. Run impact analysis on all platform documents : 0.5 hour= 4 hours to identify all the 28 document regressions and the impact on the 12,000 platform documents

FIND CALCULATION ENGINE CHANGES

EASIER, FASTER, CHEAPER & SAFER MIGRATION

Backup, promote & version BO content

Schedule & burst your BO reports

Explore & analyze BO metadata and licenses

Automate report qualificationNon-regression testing

XIR2

XI3

BI4

Secure your BO deployment

Streamline and Extend SAP BusinessObjects

STREAMLINE SAP BUSINESSOBJECTS

ABOUT GB&SMITH

• 1,000,000 administrated users• Won EY entrepreneur of the

year and other famous prices• Continuously evolving

company and tools

• More than 500 customers• In more than 30 countries• Small to big deployment

(100-92000 users)

• 70 employees• Very reactive support team

(98% renewal rate)• 50% in R&D

THEY TRUST US

Take a look at our solutions on: www.youtube.com/360suite

Let’s work together

contact@gbandsmith.comwww.gbandsmith.com

QUESTIONS?