a step by-step process to design and manage a successful sap bi implementation or upgrade for large...

How to: Manage Large SAP BI Implementations & Upgrades for Success

This presentation we walk the audience through some vital steps how to approach building, maintaining, scaling or integrating a globally distributed SAP BI4.x deployment. We also look at some examples and solutions shared by our specialists.

• For New deployments we look at a multi-tiered architecture design strategy, as well as a detailed implementation strategy involving: pre-requisites, supportability, connectivity, SAP BI4.x installation and post-configuration/customization, monitoring.

• For existing deployments we explore a number of approaches: dynamic monitoring, SAP BI4.x administration/governance, Tune/scale approach, Software integration approach, Upgrade/patch approach

The resource Library at the end of the Presentation includes the links mentioned in the slides as well as additional useful sources for further information.

Introduction

New deployments

Agenda

Existing deployments Real-life examples

Closing Remarks Xoomworks BI

New deployments

A SAP BI4.x implementation depends on everything!

• Not only on:• Networking components (Network connectivity/hops/bandwidth, Active Directory, DNS, network card

drivers and configuration, etc.)• OS and its settings• Type of authentication• Type of access required• Security and application behaviour/stability/availability

• But also from all the connected data sources and from human behaviour (if left uncontrolled)

• We are going to look at:• The Design strategy: How to reach the goal of an unbreakable, dynamically reconfigurable, and error-free BI

solution• The Implementation strategy

New deployments

Design Strategy

• Discuss and clarify the customer’s objectives and document the business requirements to define the correct architecture design strategy

• Understand the position of, and build relationships with all people involved in the design and implementation: sponsors, program managers, architects, developers, consultants, DBAs, etc.

• Document all details about hardware, OS, network, possible virtualization, back-up procedures, security rules, content management, disaster recovery procedure, and monitoring tools of the customer’s internal solution offering

• Understand the capabilities of SAP BI4.x and the features that can lead to multiple possible technical solutions and weigh up which solution best meets the business requirements• This needs to include pros, cons, risks, mitigations, implementation roadmap, and deliverable timelines

New deployments

Design Strategy (cont.)

• List all technical checks needed for software supportability/compatibility with all third-party applications.• Client connectivity, all data sources, AD/LDAP servers, virtualization platforms, reverse proxies, browsers,

OS client/server, smartphones, Citrix, languages, application/portal servers• See link to SAP BusinessObjects Supported Platforms document in resource library

• Create proposals for an SAP BI4.x multi-tier implementation that match the content development strategy: its testing from a formatting, data, and performance point of view, its promotion to the production system for its final consumption. • Consider future implementations, as well as maintenance and operational costs

• Create a load and performance test plan to confirm the system/reporting performances expected by the customer using worst case scenario (usually month end and year end).

New deployments

Design Strategy (cont.)

• Five-tier architecture diagram example and related mapping of data source tiers

Data flows

Content Promotion

Sandbox Data Sources

Development Data Sources

Acceptance Data Sources

Quality Data Sources

Production Data Sources

BI4.x Sandbox Distributed Cluster

BI4.x Development Distributed Cluster

BI4.x AcceptanceDistributed Cluster

BI4.x Quality Distributed Cluster

BI4.x Production Distributed Cluster

Software changes/ upgrades

Content Developers

User Acceptance

testing

Quality check and load tests

End-user consumption Key:

New deployments

Design Strategy: Multi-Tier Design Approach

Sandbox:

• The Sandbox environment is used to perform testing and exploratory investigation into issues experienced within other environments, for software upgrades and patches/private fix regression testing, new integration with other software, or new SAP BI4.x functionality• This landscape should be architected like a minimal simplified distributed Production landscape (server

separation: CMS/Audit DBs, web-app server, Core server, Proc server, etc.)

Development:

• The Development environment is used by developers to create BI Content• The development data sources will be connected to this landscape. They need not contain valid production

data, only the same structure as Production data sources

New deployments

Design Strategy: Multi-Tier Design Approach (Cont.)

Acceptance:

• The Acceptance environment should be used to validate the configuration of builds against the connected Acceptance databases prior to promoting the content to Quality, then Production• The data sources should contain an old copy of Production data to allow validation of formatting, calculation

results, and single report performances

Quality:

• The Quality environment should be used to validate, by an internal BI COE, the quality of the generated BI content, which should adhere to reporting best practices before promotion to the Production landscape for its final consumption• The Quality environment will also be used for load and performance tests. This needs to be carried out

following a real-life scenario on production data volumes, testing concurrent usage, scheduling and publications under a worst-case scenario

New deployments

Design Strategy: Multi-Tier Design Approach (Cont.)

Production:

• The Production environment is used for the final reporting consumption/job processing against the connected Production data sources

New Deployments Existing Deployments Real Life Examples Concluding Remarks Xoomworks BI

New deployments

Implementation Strategy

• Define and run a PoC to prove that the agreed subset of business requirements can be met, and troubleshoot/fix/document any technical issues which arise during this phase

Before running the BI4.x installation:

• Check all OS settings and prerequisites needed by the application: Local Security Policies, the Service Account to be used in SAP BI4.x (which is set up in AD and OS Groups rights), SSL certificates, setspncommands, installation path rights, etc.

• Install all connectivity drivers required before installing the BI4.x software

New deployments

Implementation Strategy (Cont.)

BI4.x installation:

• Select the right SAP BI4.x software components to be sure you can dynamically change from CMC only (Core-Proc nodes) in case the load needs to be redistributed across nodes without performing a modification of the installation

• For installation on multiple servers, follow the same patching level approach (e.g., install BI4.1 SP3 Full install on all servers first, then SP3.6 upgrade on all servers) • You can run parallel installs, but avoid installing/upgrading nodes with a CMS at the same time (leave few

minutes of delay between nodes with a CMS) - Conflicts in the CMS DB might occur if placeholders are getting updated by 2 or more CMS processes at the same time

• Set up/test all connectivity, documenting the network hops/bandwidth (where applicable) between them and SAP BI4.x

New deployments

Implementation Strategy (Cont.)

BI4.x configuration post-install :

• Follow the BI4.x servers configuration post-install• CMC, OS registry customizations, environment variables, etc.

• Hand over to support or continue a new possible content migration project from older landscape version (phased or big-bang migration)• Make sure to implement a flexible security model based on CALs that matches the security requirements,

but that can be adjusted/expanded in the future to allow multiple independent projects running on the same deployment

• Set up monitoring to have an immediate idea of usage, performances, and assess any optimisations to improve your deployment (BI4.x monitoring, Wily Introscope, SAP Solution Manager, Tivoli, etc.• Make sure you avoid duplicate monitoring tools, which will only waste system resources

New deployments

Existing deployments

For Existing Deployments

• SAP BI4.x growth/usage can change dynamically every day depending on the type and level of controls which are in place• Make sure you’re in a position to monitor and collect the root metrics of the entire SAP BI4.x system usage

over time. This will enable you to review scalability, resizing, tuning, and cleanup processes. This ensures you keep a stable, secure, and available system.


Key Points to Monitor

CMS and Audit DB:

• CPU, RAM, DB cache size, query length, concurrent parallel connections open by the CMS (up to 14 for each CMS by default), and all related administration operations that can impact both schemas

Operating System:

• Total CPU/RAM utilisation overtime, number of running processes, OS warning/error logs, process dumps, disk size utilization, I/O throughput (disks and network cards), administrative access to the server, recurring scheduled tasks for maintenance and anti-virus activity (where applicable)


Key Points to Monitor (Cont.)

SAP BI4.x processes:

• CMS server: Amount of objects in the CMS DB and amount of cached objects in the CMS process, size of the process in memory, CPU usage overtime, automated tasks done in the background • SAP/AD/LDAP aliases update, Instance Limit cleanup, cleanup of audit events older than X days, etc.

• Input/Output Filestore: Check their growth/usage overtime, linking growth peaks with users’ utilisation, content promotions, running jobs/publications

• Processing servers: Depending on their nature and usage (parent/child vs. single multi-threaded processes, 32 vs. 64-bit processes and their limit on memory that can be allocated on each of them), monitor CPU used, size in memory of the process (to spot possible Java memory leaks or Java out of memory), PID, number of user sessions inside the process, number of opened documents, number of successful/failed jobs


• Is the customer happy with the current solution? (Yes, No, Yes but)

• Which new features or other integrated products can be used?

• Does the business case still hold true with the current requirements?

On-Going Maintenance

• The questions to the right need to be answered when projects involve:• Tuning or scaling an existing deployment• Integrating other software components into the

existing landscape (e.g., Explorer, Design Studio, Lumira, etc.) or enabling new, currently unused features (e.g., AOLAP, Federation, etc.)

• Planning and running an SAP BI4.x software upgrade (SAP BI4.x Support Packs, Patches)


Tuning and Scaling Process

• Explore the system’s historical metrics, current customer performance issues (recurring and/or random ones), system errors/instabilities and analyse/reverse engineer the impacted BI4.x processes depending on workflows

• Isolate the issue until you can reproduce the bottleneck consistently in a simplified system, possibly at a specific time of the day or under certain specific load conditions. Only then will you be able to consider possible solutions• DBs: CPU, RAM, cache size, amount of parallel queries, query planning optimization• BI4.x processes: Xmx memory of each APS/APJ, number of duplicated processes and

their distribution for failover and high availability, add nodes, add CPU/RAM, etc.

• For more information on trouble shooting see link to guide in resource library.


Integration Process

• An existing SAP BI4.x system can be integrated with other SAP software when required by the customer. It’s important to have the correct approach to avoid having an impact on the existing live system • Follow the same steps mentioned in the NEW DEPLOYMENTS – Design Strategy • Run an “integration PoC” to understand the integration process to follow while installing

the software, simulating the minimal deployment scenario when dealing with distributed deployments (web-app nodes + Core nodes + Processing nodes + Job server nodes, etc.)

• Allow developers to creat some content using the new integrated software to see if it matches customer expectations in terms of reporting functionality, performances, etc.


Integration Process (cont.)

• Address the possible impact on existing SAP BI4.x system sizing, system resource utilisations to make sure that the newly integrated software won’t impact the current stability and performance of your production system

• Test the new integrated software in isolation mode to understand the resources needed to run it concurrently on the same SAP BI4.x landscape to avoid generating possible system overloads• E.g. deploying Explorer or Design Studio web-apps on the same Tomcat which already

serves the BI4.x applications (BILaunchpad, OpenDoc, CMC, WebServices, etc.)

• Install the new software on existing non-production landscapes and plan/run a load test in worst-case scenario in terms of data retrieved and user concurrency• If there is an negative impact, then a scaling exercise needs to happen before installing the

new software on the production tier and going live the business users


SAP BI4.x Software Upgrade Process

• First, start an “upgrade” PoC process on a simplified distributed cluster to know:• The time needed for the entire software upgrade which needs a full system downtime• The installation sequence to be followed on distributed landscapes: First, upgrade all

nodes with a CMS process, leaving the other nodes stopped. Then upgrade all Proc nodes and web-app nodes in parallel

• See process webinar in resource library• Check for the possible loss of system customizations (mostly web-apps and CMC-

Application settings) and document how to re-apply them (e.g., custom scripts)• Also cover the upgrade process of all BI4.x client tools which might be used on a shared

Citrix environment or packaged and distributed down to end users/developers


SAP BI4.x Software Upgrade Process (cont.)

• Involve the developers/customer users to test their content on the upgraded PoClandscape• This is a vital step to avoid the introduction of unforeseen software regressions in terms

of functionality or single report/job/publication performances

• Allow enough time to troubleshoot/escalate to SAP support, through Service Marketplace, any regressions found while testing existing content• Depending on customer impact, a private fix might be released before the official

patch/support pack in case of regressions in the higher software version

• Plan/run a load and performance test on the QUA cluster (which should be sized as close as possible to the PRD cluster) before upgrading the PRD system. This avoids the possible introduction of unforeseen regressions in terms of overall system performance and process behaviour


SAP BI4.x Software Upgrade Process (cont.)

• This needs to be done within a specific timeframe, agreed upon with the customer, as it involves full system downtime• Consider how you will deal with systems running 24x7• Assess the timescales required to install the new software • Plan/run a load test in worst-case scenario to confirm the expected performances after

the upgrade

• PRD upgrade• Take a full system backup• Pause all recurring jobs and/or disable all BI4.x servers except CMS and Input/

Output FRS• Run a reposcan to fix all system inconsistencies• Perform the upgrade following the same steps applied on non-PRD clusters


BI4.x Satellite Clusters for Globally Distributed Solutions Network Communication for Fully Independent Solutions

AME users BI4.x Launchpad –Mobile-Client tools

BI4.x GLOBAL MULTITIERCLUSTERS (SBE/DEV/TEST/QUA/PRD)

BI4.x PRDSatellite Cluster

AMEAD Server

AM-EU-AME-APJGLOBAL AD

Servers

AM users BI4.x Launchpad –Mobile-Client tools

BI4.x PRDSatellite Cluster

AM AD Server

Global user access – BI4.x Launchpad – Mobile-Client tools

Data Sources

Data Sources

Data Sources

Audit/ CMSDB

Audit/ CMSDB

Audit/ CMS DB

Key:

AD SSO logon

SAP SSO logon

Federation/Replication

BI4.x CENTRAL GLOBAL INSTANCE

BI4.x SATELLITES

Data flows AM = AMERICASEU = EUROPEAME = AFRICA-MEAPJ = ASIAPAC


Real-life examples

Real-life examples

Example 1: Hardware Design Mistakes and How They Were Solved

Problem

• The runtime when refreshing/calculating a WebIreport is 20 seconds on a TEST CPU at 1.8GHz; while on a PRD CPU at 3.6 GHz, the same WebI refreshes in 40 seconds

• The query runtime in the DB is exactly the same, as well as the user/report/universe/network

Troubleshooting

• Once the query has sent all data to the WebI process for its calculation, and before it is sent to the web-app server, the time is spent in the CPU/memory of the WebI process

• The TEST CPU at 1.8GHz has a Front-Side Bus (FSB) at 400MHz

• The PRD CPU at 3.6 GHz has a Front-Side Bus (FSB) at 100MHz

Mistake/Lesson learned

• Missing detailed information about chosen hardware

• For more information about detailed hardware information see resource library

Real-life examples

Example 2: Hardware Configuration Issue and How It Was Solved

Problem

Tomcat and BI4.x processes are taking too much time (20 minutes) when BI4.x is installed on an attached iSCSI disk which is running separately from the disk where the OS runs: application performance is unacceptable for the customer

Troubleshooting

• iSCSI disk configuration looks correct and there’s no I/O bottleneck when trying to transfer files from it

• At every server reboot, there’s a warning about attached iSCSI disks

• The iSCSI initiator is relaying on DNS name resolution for its storage


• incorrect configuration of the iSCSI disk where BI4.x runs

• For examples of iSCSI; or SCSI over TCP name construction and discussion of discovery of iSCSI resources see link in resource library

Real-life examples

Example 3: Operational Mistake and How It Was Solved

Problem

A PRD running landscape suddenly runs in an unreliable state

Troubleshooting

20 GB of new content has been produced in a few hours overnight in the Input FRS, there’s been an overload in the Destination Job servers, a delay in recurring scheduling jobs causing the loss of cluster connectivity between Core and Processing nodes, and each user in the system has 20 new reports in his BI4.x Inbox


• Missing governance in the BI4.x security model allowed a single end user the freedom to send 20 reports to the “Everyone” group composed of 30,000 users

• This misuse caused a system overload, impacting its stability/availability and the correct start time/output of other recurring jobs

Real-life examples

Example 4: Design Mistake and How It Was Solved

Problem

The number of issues/showstoppers raised during a SAP BI4.x PoC phase was too high and an entire new BI4.x project was jeopardized

Troubleshooting

• SAP BI4.x architecture proposal has been reviewed and looks correct on paper

• Security and encryption requirements are in place, but cause issues due to software bugs around content promotion

• Questioned the requirements which should explain where the end users will have access to the reporting solution (only BI Launchpad, no Client Tools)


• SSL encryption was not needed on the CMS process, but only on web-app server

• Therefore, the architecture proposal was more complex than needed and caused redundant issues to be raised to SAP to meet project goals

Real-life examples

Example 5: Design Mistake and How It Was Solved

Problem

The AD SSO log-on process on BI Launchpad takes more than 10 seconds on a Windows distributed deployment

Troubleshooting

• All web-app deployments and web-app servers are correctly set up as per best practice

• Tracing Kerberos traffic, BI Launchpad, and CMS traces reveals the delay in the NetBIOS packets while querying for the computer name

• Using Nbtstat tool, concluded that WINS was not configured and the NetBIOS request was pending for each of all the network cards on the server until it timed out


• A default setting of an OS/network card doesn’t constitute the correct/best performing setting for SAP BI4.x

• After disabling WINS on all NICs, the network tracing didn’t show any NetBIOS packets when accessing BI Launchpad, and the AD SSO logon was completed in seconds

Real-life examples

Closing Remarks

1. The correct approach leads to the right solution

2. Don’t worry about not knowing the answer, leave the question open, get information, test and prove things yourself before making decisions based on inaccurate facts which will only lead to the wrong conclusion

3. Record and share your knowledge in clear technical documentation for any type of process (installation, customization, integration, upgrade, etc.) adding who does what, when, where, and how, and keep it up to date

4. Keep yourself up to date on new features and new integrated software, as this increases the opportunities to achieve the project goals and drastically reduces the risk of project failures

7 Take Home Messages

5. Always put yourself in the customer’s shoes, as one single report might have more value than all others; be sure to explore/test/prove all project variables/dependencies

6. Be curious when looking for RCA of any issue, starting from OS issues up to app issues

7. Lastly but not least. Just because things look right for the user, it does not neccesarilymean it is right in the back end

7 Take Home Messages Cont.

“Having the strong technical capability of Xoomworksbehind us has enabled us to prove what SAP BusinessObjects can achieve, and that’s how we’ve managed to build up and maintain a single platform concept for all our management information.”The Solution Delivery Manager for a major petro-chemical company.

“Their consultants were very diligent, spending a lot of time working on the material for the training so that they were very well prepared for the sessions.”Raf Wayne, CTO, Neural Technologies Limited.

THANK YOUFind out more:

On our website: www.xoomworks.comOr drop us a line: [email protected]

And follow us on LinkedIn (Xoomworks BI) and Twitter (@Xoomworks_BI) for regular updates

Resource Library: Mentioned links• Design and Manage a Successful Implementation

- Slide 7: http://scn.sap.com/docs/DOC-54443• Upgrade an Existing Implementation

- Slide 20: http://wiki.scn.sap.com/wiki/display/BOBJ/Troubleshooting+by+Scenario- Slide 23: www.slideshare.net/SAPanalytics/webinar-5-bi-41-upgrade-bi-installation-and-

configuration• Real-Life Examples

- Slide 28: http://en.wikipedia.org/wiki/Front-side_bus- Slide 29: www.ietf.org/rfc/rfc3721.txt

Resource Library: Further information• Design and Manage a Successful Implementation:

- www.sapbi.com/bi-strategy/- www.sapbi.com/implement/- https://apps.support.sap.com/sap/support/pam *- https://service.sap.com/sizing *- http://help.sap.com/bobip41/- http://service.sap.com/sap/support/notes/1694041 *

• Upgrade an Existing Implementation:- www.sapbi.com/upgrade/- https://scn.sap.com/community/businessobjects-web-intelligence/blog/2014/12/18/revisit-

the-sizing-for-your-deployment-of-bi-4x-web-intelligence#- Ted Ueda, “Revisit the Sizing for your deployment of BI 4.x Web Intelligence Processing Servers!” (SCN, December

2014).

* Requires login credentials to the SAP Service Marketplace