solman:monitoring best practice

Central System Monitoring for mySAP.com Best Practice for Solution Management

Version Date: May 2002. The newest version of this Best Practice can always be

obtained through the SAP Solution Manager

Contents

Applicability, Goals, and Requirements....................................................................................................2 Best Practice Procedure...........................................................................................................................2

Requirements for a Monitoring and Optimization Concept .........................................................2 Requirements from the User Viewpoint .................................................................................3 From System Monitoring to Solution Monitoring....................................................................4 Central Monitoring and Enhanceability ..................................................................................4 Summary of Monitoring Requirements ..................................................................................4

Tools and Methods for Monitoring and Optimization Concept.....................................................5 Central Monitoring..................................................................................................................5 Continuous Monitoring ...........................................................................................................5 Performance Indicators..........................................................................................................5 Working with the CCMS Monitoring Tool ...............................................................................6 CCMS Monitoring Tree setup.................................................................................................6 Monitoring Availability and Performance in mySAP.com Components ..................................8 Monitoring the Performance of Selected Transactions ..........................................................9 Monitoring Error Situations in mySAP.com Components.......................................................9 Automatic Alert Notification Setup........................................................................................10 Using the Graphical User Interface in the SAP Solution Manager ......................................11

Further Information .................................................................................................................................15

Best Practice: Central System Monitoring for mySAP.com 2

Applicability, Goals, and Requirements To ensure that this Best Practice is the one you need, consider the following goals and requirements.

Goal of Using this Service Use this service to lay the foundation for setting up the correct procedures for monitoring a mySAP solution. The recommended technique is to define an explicit monitoring concept for central system monitoring using the Computer Center Management System (CCMS) in conjunction with the SAP Solution Manager. This form of business process oriented system monitoring starting with individual business processes and monitoring all operations-relevant system components is known an inside-out approach. The complementary outside-in approach, that is, system monitoring using tools from third-party manufacturers is described in the Best Practice "E-Business Availability Monitoring". To achieve comprehensive monitoring, you should use both approaches.

Staff and Skills Requirements You need:

1. A monitoring team that has all basic skills related to mySAP Technology 2. An SAP-Certified Technical Consultant with a basic understanding of performance analysis 3. CCMS agents are running on all computers with software without SAP basis

System Requirements You need:

1. To have the SAP Solution Manager installed 2. SAP Basis 3.0 or higher for all components that are to be monitored (such as SAP R/3, SAP

BW, SAP APO, and so on)

Duration and Timing Apply this Best Practice before the start of production, that is, before going live with the relevant mySAP.com solution.

You will need around one week to set up your monitoring concept and implement it for the first software component. You will need additional two or three days for each subsequent component, depending on the size of the respective component.

Best Practice Procedure Applying this Best Practice has two parts:

Creating your monitoring concept •

• Implementing the monitoring concept in the CCMS monitoring architecture and in the Solution Manager

The details are explained below.

Requirements for a Monitoring and Optimization Concept To meet the expectations of a user of a mySAP.com application, it is necessary to create a monitoring and optimization concept. As SAP has established itself as a solution provider and is no longer just an Enterprise Resource Planning (ERP) software vendor, monitoring tasks have changed. Instead of the

© 2001 SAP AG


classic system monitoring, we now talk about solution monitoring, which instead of monitoring just individual system components monitors the business process as a whole across multiple components.

SAP AG 2001, Title of Presentation, Speaker Name 1

...

Requirements for Monitoring a mySAP.com Solution

Availability• Availability • Redundancy

Performance• Response times • Throughput• Workload/resource

Management

Accuracy• Data integrity• Data currency• Recoverability• Scheduled maintenance

Security• System users• System resources• Access control• Intrusion detection

Expectations

Responsibilities / Viewpoints

Systems Business Processes Interfaces

Time Schedulehourly tasks daily weekly monthly

Alert Monitoring “Keep the business running” tasks

Service Level Management & ReportingContinuous improvement, planning tasks

Business Scenarios and Components

mySAP Logistics,mySAP Financials

mySAP Customer Relationship Management

mySAP Supply Chain Management

mySAP Business Intelligence

SAP Basis System Databases SAP

ITSSAP

LiveCacheBusiness Connector

...

Figure 1: Requirements for monitoring a mySAP.com Solution

Requirements from the User Viewpoint The users of a mySAP.com application are employees, customers, or partners (for example, suppliers) of the company that owns the application. The satisfaction of these users is one of the main goals that must be achieved. Accordingly, the monitoring concept must cover the performance indicators that correspond to the expectations and requirements of users.

You can ask, for example, "What will users expect from an Internet application that they use to place a goods order or make a bank transaction?" The answer will probably identify four main expectations:

• The application should be available when the user needs it

• The application should run with an acceptable performance

• The application should run correctly (in other words, users want to receive the product as displayed; in addition, the price on the bill should match the price on the order)

• The application should be secure, that is, no one should be able to manipulate user data

Availability To ensure the availability – the first expectation of a user of an application – most system management platforms on the market offer monitoring for hardware and software components. However, this is not enough to ensure that the business process is available. A user will also consider an application to be "unavailable" if there is a disruption in communication between components or if a serious application error makes it impossible to enter or request data. As a result, availability monitoring must guarantee that the business process is available, not just that individual components are available.

Performance Poor performance in an e-business application easily aggravates a customer. Performance affects the dialog part of the application, that is the part in which users enter and save data, as well as automatic background processing, which continues to process data even if the customer is no longer online. Poor performance in the dialog part of the application affects the customer immediately during data

© 2001 SAP AG


• The concept must record all business processes and components, if possible in a central tool.

To manage all of these requirements for monitoring mySAP.com solutions, SAP has created the Solution Management Program whose central tools are the SAP Solution Manager and the Computer Center Management System (CCMS). These are described in the following sections. This document is restricted to the aspects of system monitoring. (For business monitoring see best practice “General Business Process Monitoring”.)

Tools and Methods for Monitoring and Optimization Concept The SAP Basis system contains a series of efficient tools for monitoring and performance analysis, which are continuously improved by SAP's experienced performance specialists. These tools include a central monitoring architecture, which performs the continual system monitoring of all hardware and software components.

Central Monitoring Continuous system monitoring ensures that all components are available and are working efficiently. If this is not the case, an alert is triggered. You can automate continuous monitoring through the central monitoring architecture. For continuous system monitoring, use the central monitoring tool of transaction RZ20: the Computer Center Management System (CCMS).

In the CCMS you define an SAP R/3 system as the central monitoring system, in which the error messages are received from all other mySAP.com components. This central monitoring tool collects monitoring information about the SAP instances, databases, operating system and other mySAP.com components, such as the SAP Internet Transaction Server (ITS). In addition, data supplying tools are available for non-SAP components (planned).

Continuous Monitoring The CCMS (Computing Center Management System) contains a tool with which you can monitor all aspects of your mySAP.com solution. The monitor uses the object-based technology of the CCMS monitoring architecture. The monitor offers:

Complete, detailed monitoring of performance indicators for mySAP.com software components, computers, databases and external components

•

•

•

•

Status flags (green, yellow, or red alerts) for performance indicators whose values are lower or higher than their thresholds

Tracking and administration for alerts

Unlimited enhancement possibilities through open structures. This enables you to cover even non-SAP software components.

The central monitoring tool is available as of SAP Basis 4.0 (the features described in this Best Practice Document are based on SAP Basis 4.6). The systems to be monitored require at least SAP Basis 3.0.

The following sections contains concrete recommendations on, for example:

• How to customize the tree in the monitoring tool to suit your mySAP.com solution

• How to set an escalation procedure, for example, using the monitoring tool to contact the appropriate person through e-mail, by pager, and so on

• How to connect the monitoring tool to the graphical interface of the SAP Solution Manager

Together, these measures provide you with a complete monitoring solution.

Performance Indicators The status of an IT solution is determined by performance indicators. Performance indicators can be: Numeric values (counters), such as average response times, throughput key figures, process activity levels, or fill levels of memory areas

© 2001 SAP AG


Text information such as error messages. Thus, the error message "Document processing for document X was terminated" would be an indicator for poor performance in a system. In this sense, performance means not only the pure runtime performance but also availability and error situations.

The task of solution monitoring includes recording, saving, aggregating and evaluating performance indicators. SAP's central monitoring tool integrates performance indicators from multiple expert monitors and combines them in a monitoring tree. These expert monitors serve the monitoring tool as a data supplier. The advantage of a central monitoring tool is that you only have to use the expert monitors in exceptional situations and not to regularly monitor components. In the central monitoring tool, the expert monitors that belong to the indicators are stored as analysis methods. You can go to the specific expert monitor directly from the monitoring tool (including via logon to another system).

The CCMS monitoring tool generates alerts automatically when an indicator is lower or higher than the threshold values. Threshold values are delivered with default values that can be customized individually.

Not all performance indicators have the same weight, when evaluating whether your mySAP.com solution is running efficiently. As you gather experience, you will find that some indicators prove to be particularly important. These indicators are called key performance indicators (KPIs). Therefore, one goal when creating your monitoring concept is to filter out from the large amounts of information those KPIs that are relevant for you.

Working with the CCMS Monitoring Tool Prerequisite for this Best Practice is knowledge of the Online Documentation for the central Alert Monitor. To access the online documentation, choose Computing Center Management System (BC-CCM) >> Monitoring in CCMS >> Alert Monitor.

CCMS Monitoring Tree setup The CCMS monitor can be used directly after installation. However, for optimal monitoring, you should modify it to suit your solution. This involves redefining monitoring objects and monitor trees to suit your needs. These changes can also be transported to other systems.

The most important performance indicators that you must consider in your daily monitoring and for which you should customize the CCMS monitor are described in this section.

© 2001 SAP AG


Figure 2: Component monitoring in the central monitor as set up for a mySAP Customer Relationship Management (mySAP CRM)

Figures 2 (above) and 3 (below) show that key performance indicators are divided into three groups.

Availability and Performance of Components: The monitoring goal in this part of the tree is to ensure the availability and system performance of the hardware and software components in your mySAP.com solution. In other words, an alert in this area indicates that a component is either not functioning at all or only very slowly. Monitoring covers all components regardless of whether they are from SAP or not

•

• Performance of Business Transactions: Monitoring in this part of the tree analyzes the performance of SAP online transactions. Transaction-specific monitoring of response times enables you to react flexibly and individually to performance problems. For example, you can

© 2001 SAP AG


set a higher reaction priority for a transaction in the sales area than for a transaction in financials. You should particularly monitor those transactions for which you have Service Level Agreements (for details see Best Practice Document “Service Level Management”).

Error Situations: This is where production operation is monitored for errors. •

Monitoring Availability and Performance in mySAP.com Components By default, the CCMS alert monitor monitors the mySAP.com components in which it is started. However, you can also monitor multiple mySAP.com components with one single monitor. To do this, you must define a mySAP.com component as your central system for monitoring. This system should have an up-to-date version of SAP Basis, which ensures that the newest monitoring functions are available. Depending on the size of your installation, you should use a dedicated SAP System. When you use the SAP Solution Manager, set both it and central CCMS monitoring up on one system.

Make the other mySAP.com components known to the CCMS alert monitor. The detailed description of how to include an SAP System in the central CCMS alert monitor is described in the SAP Online Documentation.

To monitor remote components, the CCMS alert monitor uses an RFC connection. From a single system, you can monitor an unlimited amount of mySAP.com components. Technical factors, such as the speed of your network and network traffic limit the number of systems that can be monitored. This is particularly noticeable when a component fails in its landscape or the performance is particularly poor. In this case it is not necessary to determine all the performance indicators. To be able to react in such a case, the central CCMS alert monitor offers a special form of availability monitoring, with which you can monitor the availability of remote mySAP.com components. Monitoring availability involves determining whether a component and its server are running and are available for work.

Monitoring availability uses the alerts and display functions in the monitoring architecture. However for data collection, to determine whether the remote system is active and available, availability monitoring uses an agent. An agent is a dedicated program that runs outside of mySAP.com components. Using agents enable the monitoring tool to check the availability of multiple remote systems from a central system without having the risk that an inactive system blocks the display in the monitoring tool. Thus, agents make it possible to efficiently monitor the availability of multiple components. It is not necessary to establish an RFC connection to the remote systems. It is also not necessary for the monitoring tool to logon to remote components to determine its availability. This means that the data collection procedure

http://service.sap.com/systemmanagement


Monitoring Third-Party Software Components The open structure of CCMS monitoring architecture enables you to write your own data suppliers for the monitoring tool, so that you can also monitor software components that do not originate from SAP. For more information, see the SAP Service Marketplace at the Internet address http://service.sap.com/systemmanagement.

Summary of Monitoring mySAP.com Components Set up component monitoring in the central CCMS alert monitor as follows:

1. Define an SAP System as your central system for monitoring. This system should have an up-to-date version of SAP Basis, which ensures that the newest monitoring functions are available. Depending on the size of your installation, you should use a dedicated SAP System. When you use the SAP Solution Manager, set both it and central CCMS monitoring up on one system.

2. Set up the availability monitoring (monitoring with an external agent program) for all software components with SAP Basis (such as SAP R/3, SAP BW, SAP APO, SAP CRM/EBP, SAP Workplace 2). For details see SAP Note 209834.

3. Set up the availability monitoring (monitoring with an external agent program) for all hardware components that run software components without SAP Basis - SAP components (such as SAP ITS) or non-SAP-components. For details see SAP Note 209834.

4. Set up the availability monitoring (monitoring with an external agent program) for all SAP software components without SAP Basis (such as SAP ITS, SAP liveCache, SAP BC) For details see SAP Note 420213.

5. (Planned:) Set up the availability monitoring (monitoring with an external agent program) for all non-SAP software components. A partner portal that offers you extensions to the open SAP Monitoring Architecture for third-party partner products will be available in the future.

As a result of the monitoring setup, Figure 2 displays component monitoring in the central monitor, as it was set up for a mySAP Customer Relationship Management (mySAP CRM). In the tree structure, "Availability and Performance of Components", "Software Components" contains the monitoring for three SAP instances in the Systems CTQ (R/3 backend) and FCC (CRM server), as well as other components of the solution for Internet Transaction Server (ITS), the Internet Pricing and Configurator (IPC) and the Index Management Server (IMS). The monitoring tree area "Hardware Components" contains the performance indicators for the computers on which the SAP Systems CTQ and FCC are running, as well as a computer on which the SAP ITS is running (this component is monitor by the agent as described above).

Monitoring the Performance of Selected Transactions The CCMS monitoring tool enables you to monitor the response times of certain clients or SAP transactions. This is particularly interesting for the transactions that you have included in your Service Level Agreement (see Best Practice Document “Service Level Management” ). In SAP Basis 4.6C, the monitoring tool contains the monitor "Transaction-Specific Dialog Monitor" among the SAP CCMS monitors for optional components. For more information, see the SAP Online Documentation. In our example (in figure 3), the transaction "Create Sales Order" is included in the monitoring process.

Monitoring Error Situations in mySAP.com Components The most important error situations, which you should continually monitor for all mySAP.com components, are:

Terminated updates: terminated or not executed updates lead to the situation that documents that are created or change by users are not saved to the allocated application tables and are therefore "non-existent" for users. Daily checks on update requests are therefore an important task for an SAP system administrator. If terminated updates are not tended to immediately, after several days it is virtually impossible to track the source of the error.

•

• Terminated background processes: Background processes are critical for smooth processing of inbound requests. Background process ensure, for example, for the replication of data

© 2001 SAP AG

http://service.sap.com/systemmanagement.


between software components. Terminated background processes must also be tended to immediately, otherwise there is a danger that the business process can hang.

Terminated interface processes (transactional RFC, queued RFC, IDocs): For terminated interface processes, the same information is valid as for updates and background processes. For important alerts for terminated interface processes, refer to the topic "Transactional RFC" in the monitoring area. For details on the Monitoring of interfaces, see Best Practice Document “Interface Monitoring”.

•

Figure 3: The monitoring tree area "Error Situations" shows which error situations should be monitored.

Automatic Alert Notification Setup As of SAP Basis 4.6 you can allocate auto-reaction methods to critical performance indicators so that you receive an alert by e-mail, fax or pager, even if you are not currently working in the monitoring tool. The monitoring tool can automatically dial a pager or send a message by e-mail or fax to the following recipients:

© 2001 SAP AG


A business workplace user in client 000. The e-mail is sent 0-5 minutes after the alert is triggered and delivered immediately.

•

• A distribution list or an external e-mail address The e-mail is sent 0-5 minutes after the alert is triggered; however, depending on the settings in SAPconnect, there may be a delay before mail to external e-mail addresses are sent (that is addresses of uses, who are not defined in client 000 or in the SAP System.) You should set the time frame for the SAPconnect send process to take less than one hour.

The message text of the e-mail contains the same information as in the monitoring display, that is the problem at hand, where and when it occurred, and how level of the alert (a red alert points to a problem or an error, a yellow alert is a warning). For more information, see the SAP Online Documentation and SAP Note 176492.

Using the Graphical User Interface in the SAP Solution Manager The structure of the alert tree can quickly become unmanageable, especially with complex component landscapes. When you use the SAP Solution Manager, you can allocate the alerts to their business processes and summarize them in graphical form. Business process graphics simplify orientation and enable you to see at first glance whether a specific business process is affected or not.

Sample Scenario The following example shows you how to setup your System Monitoring in the Solution Manager.

Prerequisites • SAP Solution Manager is up and running

• SAP Systems to be monitored are connected to the Solution Manager (RFC connection)

• The solution landscape (that is, Systems, hardware etc.) has been specified in the Solution Scope area of the Solution Manager by answering the questionnaire “Define Solution Landscape”

• For non-SAP components agents are running which report their data into a CCMS which is connected to the solution manager

Monitoring In the following example we created a productive CRM landscape named “High Tech Shop”. In order to setup the system monitoring for your landscape you have to perform the session “Setup System Monitoring” in the system monitoring area of the Solution Manager (Monitoring -> System Monitoring). The following screen shot shows the starting point.

© 2001 SAP AG


In the session “Setup System Monitoring” you are asked to specify

• The systems you want to monitor, that is,

o Components with SAP basis

o Non-SAP systems

• For the SAP systems the alerts that you want to have displayed in the alert graphic, that is

o alerts which are defined by default

o user defined alerts, which have to be connected to the CCMS via the MTE names of the respective alerts

The MTE names can be found in transaction RZ20 via the menu points Edit -> Nodes (MTE) -> Display MTE description. In order to get access to the alerts of satellite systems you have to maintain RFC connections to these systems in the check “Setup RFC connections for Monitoring”. A detailed description of the maintenance procedure can be found in the check description. For additional systems, that means systems, which have been specified during the Scope Session as:

• Further SAP application systems

• SAP technology components

• Non SAP application components or non SAP technology components

you also can propagate alerts by maintaining the checks “Additional Software Components” and “Additional Hardware Components” respectively. In the following screenshot you see the SAP technology component “Business Connector” for which we have integrated the alert with MTE name “CTQ\SAPBC P37221 5555\...\Hardware Properties\...”. Of course this alert has to be accessible in an alert monitor of the system the solution manager is running on. (For details on the alert monitor see: http://help.sap.com/ -> SAP R/3 -> SAP R/3 Release 4.6C -> Basis Components -> Computing Center Management System -> CCMS Monitoring -> The Alert Monitor).

It is important to notice the following rules for the rating of the checks by which you can customize the alerts for the respective system:

• If all alerts are not accessible now or if all alerts are deactivated the rating for the check is undefined

• If all activated alerts are accessible then the rating for check is green

• If at least one alert was never accessible yet then the rating for the check is red

• If at least one of the activated alerts is not accessible now, but was accessible before, and no never-accessible alert is activated, then the rating for the check is yellow

The following screen shot shows the session “Setup System Monitoring”.

© 2001 SAP AG

http://help.sap.com/


After having finished the setup of the system monitoring the corresponding CCMS alerts are connected to the solution manager. The solution manager performs a graphical correlation between the alert and the corresponding system in the landscape. To get the view onto the alert screens click on “Display Alerts” in the Monitoring area. The result for the CRM Solution High Tech Shop can be seen in the following screen shot (here we have chosen the software and the business process viewpoints; you can choose alternatively the hardware viewpoint).

Alerts for software components:

Alerts for business process:

© 2001 SAP AG


On the top of the boxes, that are representing the systems you can see some alert icons. Possible alerts are green, yellow or red. If the alert, which you have maintained, is not accessible you will get a gray icon for the corresponding component. In the check “Copy customizing” you have the possibility to deactivate all non-accessible alerts, so that they will not appear in the graphics.

Clicking on the alert icons will display the alert in detail. The following screen shot shows the alerts for the system CTD:

Clicking on the components brings you to the next “alert level” in order to analyze the alert in greater detail:

The ultimate level could be, for example, a corresponding monitoring transaction, which has to be maintained as a method assigned to the MTE or MTE class, that corresponds to the alert. For more information on this topic see http://help.sap.com/ -> SAP R/3 -> SAP R/3 Release 4.6C -> Basis Components -> Computing Center Management System -> CCMS Monitoring -> The Alert Monitor -> Method Assignments.

Summary of Best Practice: Monitoring Tools and Methods 1. The CCMS monitor can theoretically be used directly after installation. However, in practice,

you should customize it to your mySAP.com solution. Monitoring includes:

Using the monitoring agent for central monitoring of availability and performance of all the hardware and software components that are involved in a mySAP.com solution. (The agent can be expanded to include non-SAP software.)

•

•

•

Monitoring performance of important SAP online transactions

Monitoring error situations in online transactions, updates, background processing and interfaces

© 2001 SAP AG

http://help.sap.com/


2. For alerts, that you identify as particularly critical, define automatic reaction procedures, for example sending e-mails or SMS messages, to ensure continuous monitoring is performed even when you are not watching it.

3. Set up the SAP Solution Manager to provide you with a graphical interface to display your business processes and the allocated alert situations. This simplifies monitoring especially for complex processes and software landscapes and allows you to visualize the connections between the business processes and software.

Further Information Troubleshooting If executing this Best Practice did not produce the desired results, you may have tried to implement System Monitoring too quickly, too superficially, or without the full cooperation of all the company employees involved. If you encounter problems with the SAP Solution Manager, send feedback to SAP.

Background Information and References For more information, use the following link. SAP Solution Manager: www.service.sap.com/solutionmanager

Background Information and References Best Practice Document “System Monitoring” •

• Best Practice Document “Interface Monitoring”

Feedback and Questions Send any feedback by formulating an SAP customer message to component SV-GST-SMC. You can do this at http://service.sap.com/message. © Copyright 2001 SAP AG. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® and SQL Server® are registered trademarks of Microsoft Corporation. IBM®, DB2®, OS/2®, DB2/6000®, Parallel Sysplex®, MVS/ESA®, RS/6000®, AIX®, S/390®, AS/400®, OS/390®, and OS/400® are registered trademarks of IBM Corporation. ORACLE® is a registered trademark of ORACLE Corporation. INFORMIX®-OnLine for SAP and Informix® Dynamic Server

TM are registered trademarks of Informix Software

Incorporated. UNIX®, X/Open®, OSF/1®, and Motif® are registered trademarks of the Open Group. HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. JAVA® is a registered trademark of Sun Microsystems, Inc. JAVASCRIPT® is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. SAP, SAP Logo, R/2, RIVA, R/3, ABAP, SAP ArchiveLink, SAP Business Workflow, WebFlow, SAP EarlyWatch, BAPI, SAPPHIRE, Management Cockpit, mySAP.com Logo and mySAP.com are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other products mentioned are trademarks or registered trademarks of their respective companies.

© 2001 SAP AG

http://service.sap.com/solutionmanager

http://service.sap.com/message


Disclaimer: SAP AG assumes no responsibility for errors or omissions in these materials. These materials are provided “as is” without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall not be liable for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. SAP does not warrant the accuracy or completeness of the information, text, graphics, links or other items contained within these materials. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third party Web pages nor provide any warranty whatsoever relating to third party Web pages.

© 2001 SAP AG

solman:monitoring best practice

Documents