ibm svc advanced copyservices

Draft Document for Review February 10, 2008 5:17 pm SG24-7574-00

ibm.com/redbooks

Front cover

SVC 4.2.1 Advanced Copy Services

Jon TatePall Thorir Beck

Tom JahnDan Rumney

Learn about the new Advanced Copy Services functionality

Server integration examples

Scripting examples

http://www.redbooks.ibm.com/


International Technical Support Organization


November 2007

Draft Document for Review February 10, 2008 5:17 pm 7574edno.fm

SG24-7574-00

© Copyright International Business Machines Corporation 2007. All rights reserved.Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP ScheduleContract with IBM Corp.

7574edno.fm Draft Document for Review February 10, 2008 5:17 pm

First Edition (November 2007)

This edition applies to the IBM System Storage SAN Volume Controller v4.2.1.

This document created or updated on February 10, 2008.

Note: Before using this information and the product it supports, read the information in “Notices” on page v.

Draft Document for Review February 10, 2008 5:17 pm 7574spec.fm

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.

© Copyright IBM Corp. 2007. All rights reserved. v

7574spec.fm Draft Document for Review February 10, 2008 5:17 pm

Trademarks

The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

Redbooks (logo) ®AIX®DS4000™

DS8000™FlashCopy®IBM®

Redbooks®System Storage™Tivoli®

The following terms are trademarks of other companies:

QLogic, and the QLogic logo are registered trademarks of QLogic Corporation. SANblade is a registered trademark in the United States.

Snapshot, and the Network Appliance logo are trademarks or registered trademarks of Network Appliance, Inc. in the U.S. and other countries.

Java, and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Windows Server, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

vi SVC 4.2.1 Advanced Copy Services

Draft Document for Review February 10, 2008 5:17 pm 7574TOC.fm

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .vTrademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiThe team that wrote this book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiBecome a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xvComments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xv

Chapter 1. Introduction to SVC 4.2.1 Advanced Copy Services . . . . . . . . . . . . . . . . . . . 11.1 What is new in FlashCopy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.1 Incremental FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.1.2 Cascaded FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.1.3 Dynamically configurable replication services space . . . . . . . . . . . . . . . . . . . . . . . 21.1.4 Grain size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.1.5 Bitmap space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1.6 Cleaning mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 What is new in Remote Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Other new functionality in 4.2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3.1 Increased storage addressing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3.2 Volume management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3.3 Managed disks test before added to MDisk Groups . . . . . . . . . . . . . . . . . . . . . . . . 31.3.4 Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Chapter 2. Planning for Advanced Copy Services implementation . . . . . . . . . . . . . . . . 52.1 First questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Reasons for copying data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 Deciding what data to copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 Ensuring copied data sets are usable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.4.1 Understanding caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.4.2 Understanding consistency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.5 Accessing the copied data sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.6 Determining how much data can be replicated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.6.1 VDisk capacity limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.7 Understanding your current environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.7.1 Storage controller information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.7.2 Server information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192.7.3 Rate of transfer of data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.7.4 Intersite latency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.8 Administration of Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.8.1 SSH Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.8.2 Command Auditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.8.3 Authorization Roles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.9 Removing workarounds with 4.2.1 new functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Chapter 3. Remote Copy: Metro Mirror and Global Mirror . . . . . . . . . . . . . . . . . . . . . . 233.1 Remote Copy services overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.1.1 Metro Mirror . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.1.2 Cluster partnership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.1.3 Intercluster communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

© Copyright IBM Corp. 2007. All rights reserved. vii

7574TOC.fm Draft Document for Review February 10, 2008 5:17 pm

3.1.4 Maintenance of the intercluster link. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.1.5 I/O channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273.1.6 Background copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273.1.7 Remote Mirror relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.1.8 Copy relationship naming convention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283.1.9 Supported methods for synchronizing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.1.10 Remote mirror consistency groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3.2 State overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.2.1 Connected or disconnected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.2.2 Consistent or inconsistent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.2.3 Consistent or synchronized. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.2.4 Detailed Information about copy states. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.3 Metro Mirror and Global Mirror configuration limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.4 Remote Copy internals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.4.1 Read stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.4.2 Write ordering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.4.3 Write consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.4.4 Write completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.4.5 Freeze/Run on consistency groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.4.6 Management of difference map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413.4.7 Non-volatile journal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423.4.8 Quick Recovery Copy or Reconstruct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Chapter 4. Implementing Remote Copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.2 Implementation using the GUI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

4.2.1 Creation of cluster partnership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.2.2 Changing a cluster partnership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.2.3 Deleting a cluster partnership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514.2.4 Creating a copy relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524.2.5 Changing copy relationship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564.2.6 Starting a copy relationship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574.2.7 Stopping a copy relationship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 594.2.8 Deleting a copy relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604.2.9 Creating consistency groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614.2.10 Starting a consistency group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634.2.11 Stopping a consistency group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.2.12 Renaming a consistency group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664.2.13 Reversing copy directions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

4.3 Implementation using the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684.3.1 Listing cluster candidates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694.3.2 Create cluster partnership. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694.3.3 Change cluster partnership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704.3.4 Deleting a cluster partnership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704.3.5 Creating a copy relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714.3.6 Starting a copy relationship. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 724.3.7 Stopping a copy relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744.3.8 Change a copy relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744.3.9 Delete copy relationship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.3.10 Creating a consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.3.11 Start consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 774.3.12 Stopping the consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 784.3.13 Changing a consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

viii SVC 4.2.1 Advanced Copy Services


4.3.14 Deleting a consistency group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.3.15 Reversing copy directions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

4.4 Example of site failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 824.4.1 Site failover when production site loses back-end storage . . . . . . . . . . . . . . . . . . 834.4.2 Site failover using the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 844.4.3 Site failover using the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

4.5 Copy commands using the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Chapter 5. FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 915.1 Introduction to FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

5.1.1 What is FlashCopy and what is it useful for? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925.1.2 Usage cases of FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

5.2 FlashCopy functions and features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 935.2.1 FlashCopy Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 935.2.2 FlashCopy Consistency Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945.2.3 FlashCopy background copy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955.2.4 FlashCopy autodelete option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 965.2.5 Multiple Target FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 965.2.6 Cascaded FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 975.2.7 Combined Multiple Target FlashCopy and Cascaded FlashCopy . . . . . . . . . . . . 985.2.8 Incremental FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.3 FlashCopy internals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1005.3.1 What is the indirection layer and what is it for?. . . . . . . . . . . . . . . . . . . . . . . . . . 1015.3.2 What is a grain? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1015.3.3 What is a bitmap? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025.3.4 Usable bitmap space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025.3.5 Number of FlashCopy Mappings in a cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . 1035.3.6 How can we specify and view the bitmap space on an I/O group. . . . . . . . . . . . 1045.3.7 Where is the bitmap stored? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1045.3.8 FlashCopy characteristics listed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1055.3.9 FlashCopy maximum configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

5.4 FlashCopy events and states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1065.4.1 FlashCopy Mapping states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1065.4.2 FlashCopy events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1075.4.3 FlashCopy consistency group states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

5.5 Dependencies between FlashCopy Mappings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1105.5.1 Linked lists of mappings and resulting dependencies. . . . . . . . . . . . . . . . . . . . . 1105.5.2 Stopping state and cleaning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.5.3 Command Line commands for information on dependencies. . . . . . . . . . . . . . . 116

Chapter 6. Implementing FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1176.1 Example 1: basic FlashCopy using the GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

6.1.1 Creating a FlashCopy mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1186.1.2 Starting a FlashCopy mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1246.1.3 Modify a mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1256.1.4 Stopping a mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1266.1.5 Creating a FlashCopy Consistency Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1276.1.6 Creating a FlashCopy mapping and adding it to the Consistency Group . . . . . . 1306.1.7 Preparing a FlashCopy Consistency Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1346.1.8 Starting a FlashCopy Consistency Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1366.1.9 Stopping a Consistency Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1396.1.10 Renaming a Consistency Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1406.1.11 Issuing a command to a mapping within a Consistency Group . . . . . . . . . . . . 141

Contents ix


6.2 Example 2: Multiple Target FlashCopy using the GUI . . . . . . . . . . . . . . . . . . . . . . . . 1426.2.1 Creating FlashCopy Mappings and a Consistency Group . . . . . . . . . . . . . . . . . 1436.2.2 Starting a Consistency Group in an MTFC tree . . . . . . . . . . . . . . . . . . . . . . . . . 146

6.3 Example 3: Cascaded and Incremental FC using the GUI . . . . . . . . . . . . . . . . . . . . . 1486.3.1 Creating FlashCopy Consistency Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1496.3.2 Create Incremental FlashCopy Mappings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

6.4 Example 4: FlashCopy of Metro Mirror VDisks using the CLI . . . . . . . . . . . . . . . . . . . 1576.4.1 Checking VDisks and Metro Mirror relationship to be used . . . . . . . . . . . . . . . . 1586.4.2 Create FlashCopy mappings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1616.4.3 Creating a FlashCopy Consistency Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1636.4.4 Adding FlashCopy mappings to the Consistency Group. . . . . . . . . . . . . . . . . . . 1646.4.5 Preparing a FlashCopy mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1666.4.6 Starting a FlashCopy Consistency Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1666.4.7 Modify a mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1696.4.8 Stopping a mapping and a Consistency Group. . . . . . . . . . . . . . . . . . . . . . . . . . 1706.4.9 Deleting a mapping and a Consistency Group . . . . . . . . . . . . . . . . . . . . . . . . . . 170

Chapter 7. Server integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1737.1 What data is copied?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1747.2 What data is on the copied VDisk? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1747.3 AIX specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

7.3.1 Creating useful copies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1757.3.2 Accessing useful copies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

7.4 Windows 2000/2003 and Copy Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1887.4.1 Creating useful copies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1887.4.2 Accessing useful copies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194

7.5 Linux Specifics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2007.5.1 Creating useful copies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2007.5.2 Accessing useful copies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203

7.6 Other Operating Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Chapter 8. Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2078.1 Running commands on the cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

8.1.1 Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2088.1.2 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2098.1.3 Submission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2108.1.4 Authorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2108.1.5 Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

8.2 Creating connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2118.2.1 Transient connections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2118.2.2 Persistent connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

8.3 SVC command line scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2148.3.1 Submitting command line scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2158.3.2 An example SVC Command Line script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216

8.4 Server side scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2178.4.1 An example script for creating persistent SVC connections . . . . . . . . . . . . . . . . 2178.4.2 Handling the SVC response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

8.5 SVC CIMOM programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2248.5.1 CIM concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2258.5.2 How SVC concepts map to CIM concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2268.5.3 Example script to create and start a FlashCopy mapping. . . . . . . . . . . . . . . . . . 227

x SVC 4.2.1 Advanced Copy Services


8.6 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2338.7 Auditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

Other resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237Referenced Web sites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238

How to get Redbooks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Contents xi


xii SVC 4.2.1 Advanced Copy Services

Draft Document for Review February 10, 2008 5:17 pm 7574pref.fm

Preface

This book describes the new features that have been added with the release of the IBM® System Storage™ SAN Volume Controller 4.2.1 code. Readers are expected to have an advanced knowledge of the SVC and SAN environment, and we recommend these books as background reading:

� IBM System Storage SAN Volume Controller, SG24-6423� Introduction to Storage Area Networks, SG24-5470� SAN Volume Controller: Best Practices and Performance Guidelines, SG24-7521� Using the SVC for Business Continuity, SG24-7371

The team that wrote this book

This book was produced by a team of specialists from around the world working at the International Technical Support Organization, San Jose Center. The team is pictured in Figure 0-1.

Figure 0-1 L-R, Dan, Pall, Tom , and Jon

Jon Tate is a Project Manager for IBM System Storage SAN Solutions at the International Technical Support Organization, San Jose Center. Before joining the ITSO in 1999, he worked in the IBM Technical Support Center, providing Level 2 support for IBM storage

© Copyright IBM Corp. 2007. All rights reserved. xiii

7574pref.fm Draft Document for Review February 10, 2008 5:17 pm

products. Jon has 22 years of experience in storage software and management, services, and support, and is both an IBM Certified IT Specialist and an IBM SAN Certified Specialist. He is also the UK Chair of the Storage Networking Industry Association (SNIA).

Pall Thorir Beck is a Technical Team Lead in IBM ITD Denmark. He has 11 years of experience working with storage, and joined the IBM Denmark team in 2005. Prior to working for IBM in Denmark he worked as an IBM CE performing hardware installaions and repairs for IBM iSeries, pSeries and zSeries in Iceland. His current position involves the administration, design, and support of large SAN environments in Denmark. He is also part of a Nordic storage team that covers storage for open systems in Sweden, Norway and Finland. Pall has a diploma as an Electronics Technician from Odensen Tekniske Skole in Denmark, and IR in Reykjavik, Iceland.

Tom Jahn is a Senior I/T Specialist for IBM TSS Storage Systems Sales in Germany. He has 10 years of experience providing technical sales support in IBM. Tom provided technical sales support for networking and server consolidation on OS/390® UNIX for IBM and its customers before he joined the storage brand eight years ago. He is engaged in providing technical support for open systems storage solutions across multiple platforms and a wide customer base, currently with a focus on storage virtualization. He holds a Dipl. Ing. degree in Computer Science from the Staatliche Studienakademie Sachsen.

Dan Rumney is the IBM Global Support Manager for the JPMC Account. Prior to this role, he has worked in development and support of storage products since joining IBM in 2002. Dan worked in System Verification Test for IBM before moving to support. He has been providing Level 2 and Level 3 support of SAN Volume Controller for the past 2 years..

We extend our thanks to the following people for their contributions to this project.

There are many people that contributed to this book. In particular, we thank the development and PFE teams in Hursley. In particular Matt Smith was also instrumental in moving any issues along and ensuring that they maintained a high profile.

We would also like to thank the following people for their contributions:

John AgombarChris BeekenIain BethuneTrevor BoardmanCarlos FuenteGary JarmanColin JewellAndrew MartinPaul MerrisonSteve RandleBill ScalesMatt SmithBarry WhyteIBM Hursley

Bill WiegandIBM Advanced Technical Support

Evelyn WickIBM Beaverton

Dorothy FaurotIBM Raleigh

xiv SVC 4.2.1 Advanced Copy Services

Draft Document for Review February 10, 2008 5:17 pm 7574pref.fm

Marci NagelIBM Rochester

Chris SaulIBM San Jose

Sharon WangIBM Chicago

Tom CadyDeanna PolmSangam RacherlaBill TrimbleIBM ITSO

Tom and Jenny ChangGarden Inn Hotel, Los Gatos, California

Become a published author

Join us for a two- to six-week residency program! Help write a book dealing with specific products or solutions, while getting hands-on experience with leading-edge technologies. You will have the opportunity to team with IBM technical professionals, Business Partners, and Clients.

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you will develop a network of contacts in IBM development labs, and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at:

ibm.com/redbooks/residencies.html

Comments welcome

Your comments are important to us!

We want our books to be as helpful as possible. Send us your comments about this book or other IBM Redbooks® in one of the following ways:

� Use the online Contact us review Redbooks form found at:

ibm.com/redbooks

� Send your comments in an e-mail to:

[email protected]

� Mail your comments to:

IBM Corporation, International Technical Support OrganizationDept. HYTD Mail Station P0992455 South RoadPoughkeepsie, NY 12601-5400

Preface xv

http://www.redbooks.ibm.com/residencies.html

http://www.redbooks.ibm.com/residencies.html



http://www.redbooks.ibm.com/contacts.html

7574pref.fm Draft Document for Review February 10, 2008 5:17 pm

xvi SVC 4.2.1 Advanced Copy Services

Draft Document for Review February 10, 2008 5:17 pm 7574 Introduction.fm

Chapter 1. Introduction to SVC 4.2.1 Advanced Copy Services

In this book we will discuss what is new in v4.2.1 code, how to plan for it, focus on the new functions and features, including implementation examples. We also describe server integration and finally some some useful scripts to simplify some of the SVC tasks.

We do not intend to go into any great depth about Business Continuity and Disaster Recovery theories and recommend these books as sources for information:

� IBM System Storage Business Continuity: Part 1 Planning Guide, SG24-6547� IBM System Storage Business Continuity: Part 2 Solutions Guide, SG24-6548� Using the SVC for Business Continuity, SG24-7371

1

Note: IBM System Storage SAN Volume Controller Software v4.2.1 can only be installed on the IBM 2145-8G4, IBM 2145-8F4, 2145-8F2, and 2145-4F2 SAN Volume Controller Storage Engines.

© Copyright IBM Corp. 2007. All rights reserved. 1

7574 Introduction.fm Draft Document for Review February 10, 2008 5:17 pm

1.1 What is new in FlashCopy

There are some considerable changes made to FlashCopy® functionality in this release.

The major changes are to FlashCopy as follows:

� Incremental FlashCopy� Cascaded FlashCopy� Dynamically configurable replication services space� Capability to address storage up to eight petabytes� Grain size� Bitmap Space� Cleaning mode

1.1.1 Incremental FlashCopy

This feature allows us to copy only the portions of the source VDisk that have been updated at either the source or target since the last time that the mapping was started, instead of copying all the content of that disk. The quantity of data that needs copying also depends on the grain size, the smaller the grain size the less data there may be to copy. This will lead to shorter copy time and reduce the load on the SVC. To be able to see the difference between source and target a “difference” value has been added.

1.1.2 Cascaded FlashCopy

This feature allows us to use a target VDisk of a FlashCopy mapping as the source VDisk of another FlashCopy mapping. The total number of targets from the “original” source can be a maximum of 16.

1.1.3 Dynamically configurable replication services space

This feature allows us to configure up to 1 PB of address space per I/O group. The default bitmap space is 40 MB per I/O group, which is evenly divided by FlashCopy and Metro/Global Mirror. SVC 4.2.1 introduces a new feature which can “borrow” some read/write cache to serve as copy service bitmap space. The bitmap space per I/O group can be up to 512 MB, which can support up to 1 PB replication service capacity.

We can dynamically configure this space between FlashCopy, Metro Mirror and Global Mirror relationships in customized sizes. This new feature has the option to allocate the cache memory address space when needed to achieve copy services requirements. The exact amount of copy services space available to users will depend upon the copy services relationships used, and the resulting copy services space allocated to each VDisk.

1.1.4 Grain size

Now we can specify a smaller grain size of 64KB. This smaller grain size is not restricted to incremental mappings. If a mapping is being created which will be added to a tree of cascaded and/or multiple target mappings, the mapping will use the grain size used by the other mappings in the tree. The cascading of mappings provides the possibility of joining two separate trees of mappings together. If these two trees do not use the same grains size then it will not be possible to create the mapping which will join them.

2 SVC 4.2.1 Advanced Copy Services

Draft Document for Review February 10, 2008 5:17 pm 7574 Introduction.fm

1.1.5 Bitmap space

Bitmap space for mappings is allocated on a per I/O group basis. For a single mapping, the default I/O group from which to take bitmap space is that of the source VDisk for the mapping. For a mapping which is being added to an existing tree of mappings, the mapping will use the I/O group being used by the other mappings in the tree.

The I/O group can be set as a parameter when creating the mapping to allow the user to force the I/O group from which to consume bitmap space. This ability can be useful if the user knows that trees of mappings may be joined in the future, or for reconstructing a tree of mappings in a different order to their original creation during configuration restore.

1.1.6 Cleaning mode

Cleaning mode is a method of copying grains from the mapping that is to be stopped when it is in copying state (and when its target VDisk is still accessible to the host). This is achieved by adding a new “cleaning rate” parameter for a mapping which, if combined with a zero background copy rate, causes the mapping to copy grains to a downstream mapping. A new “cleaning progress” field in the query of a mapping informs the user of progress. When cleaning is complete, the mapping can be stopped and will transition to stopped state immediately. If the mapping is stopped before cleaning is complete, the remainder of the cleaning will occur whilst in the existing stopping state.

1.2 What is new in Remote Copy

Metro Mirror now uses the same inter-node communication technology as Global Mirror. That means only one round trip delay for the remote I/O traffic instead of two. This improves the robustness, and could increase performance on the exciting applications, and could even increase the distance where Metro Mirror latency impact does not play a significant role.

1.3 Other new functionality in 4.2.1

This is an overview of other new functions in SVC 4.2.1.

1.3.1 Increased storage addressing

This feature has increased the storage addressing capability from two petabytes to eight petabytes per cluster. This capability is based on an increased extent size (up to 2 GB), and applies regardless of the number of nodes in the cluster. MDisk and VDisk relations and use are unaffected by this increase.

1.3.2 Volume management

This feature offers the possibility to assign or change the preferred node upon VDisk reallocation to new I/O group.

1.3.3 Managed disks test before added to MDisk Groups

This feature offers a built-in test to check MDisks before they are added to an MDisk Group.

Chapter 1. Introduction to SVC 4.2.1 Advanced Copy Services 3

7574 Introduction.fm Draft Document for Review February 10, 2008 5:17 pm

1.3.4 Cache

In this version of software the cache space is partitioned by managed disk group. Each MDisk group that qualifies is limited to a certain amount of the cache, depending on how many MDisk groups there are running in the cluster.


Draft Document for Review February 10, 2008 5:17 pm 7574 Planning for ACS implementation.fm

Chapter 2. Planning for Advanced Copy Services implementation

Implementing Advanced Copy Services requires considered planning in order to be effective. In this chapter, we discuss the issues that should be considered during your planning phase and provide guidance on how to resolve some of the questions.

2


7574 Planning for ACS implementation.fm Draft Document for Review February 10, 2008 5:17 pm

2.1 First questions

Copy Services come in two types:

� Remote Copy, comprising:

– Metro Mirror

– Global Mirror

� FlashCopy, comprising:

– Single Target FlashCopy (FC)

– Multiple Target FlashCopy (MTFC)

– Cascaded FlashCopy (CFC)

– Incremental FlashCopy (IFC)

In the early phases of planning, it is tempting to select one of these Copy Services functions as the very first step; we strongly recommend that you wait until later in the process, By identifying the business aspects first, you will be better positioned to select the most appropriate technical solution.

Prior to implementing Copy Services, there are a number of questions to ask yourself.

� Why do I need multiple copies of my data?� What data do I want to copy?� How do I ensure that my copies are usable?� Under what circumstances will I need to access this data?� How much data can I copy?� How are my current systems’ workloads characterized?� Will Copy Services be controlled by server or storage administrators?

2.2 Reasons for copying data

There are many reasons why you may want to copy your data. Several business cases are discussed in the later chapters, but at a high level, these may include:

Disaster Recovery Data are copied to prevent loss of information due to a disaster and to allow the continuation of business operations until the disaster has passed

Testing and Assurance Data are copied to provide test and assurance systems with a copy of real production data to allow for meaningful testing

Business Operations Data are copied prior to system configuration changes to provide snapshots in case rollback is required

The reason for copying data dictates which data will be copied. For instance, the data copied for a Disaster Recovery solution may differ significantly from the data copied for Testing, even if the same servers are involved.

2.3 Deciding what data to copy

Complex systems consume and produce vast amounts of data during day to day operations. If the decision was made to mirror every byte of data in an environment then storage



utilization would never exceed 50%. It’s clear that decisions need to be made to select precisely which data need copying to balance the cost of copying with its benefits.

There are costs associated with Copy Services beyond the cost of the technology. These include:

� The capital and maintenance cost of the additional storage needed� The management cost of administering the Copy Services� The opportunity cost of the bandwidth used to mirror the data

These costs are offset by the benefit of having multiple copies of your data. Business cases for which these Copy Services can be used are listed in later chapters. The benefits of these copies are often calculated by determining the cost of having to recreate this data or the cost of losing it forever. The benefit is considered to be the fact that you do not have to meet this cost, if you implement the Copy Services solution. In extreme cases, the data may be critical to your business’s core function and the cost of losing it is your business.

We call groupings of files and other information ‘data sets’. A data set could be multiple VDisks which contain the information that you are concerned with. It could be that a data set is smaller than a VDisk, for instance, a single partition on a larger VDisk. However, the smallest entity that SVC can replicate is a VDisk and we will not discuss file or filesystem replication in this Redbook.

Some data sets have no business value, but is generated during day to day operations. Here are some examples of data sets that you may not wish to copy:

� Temporary file space� Temporary database tables� Non-business data, such as employee’s personal files

Some data sets are easy to reproduce or, more relevantly, cheaper to reproduce than they are to copy. As such, you may decide that some data should be left uncopied. Examples of such data sets may include:

� Development sandboxes� Test systems� SAN Boot volumes (reinstallation may be simpler and cheaper than replication)

Undoubtedly, even after removing data sets such as those above, you will be left with data sets that are too expensive to lose or recreate. It is the VDisks upon which these data sets reside that you will want to copy.

2.4 Ensuring copied data sets are usable

Copying the contents of a VDisk to another VDisk is all very well, but for the copy to be worth something, you must be able to access and use it at a later date. In order to design a system that ensures this, you must understand how write-caching comes into play and what consistency is.

2.4.1 Understanding caching

Caching has been used in computing for over three decades to increase the performance of access requests to storage devices. In many computer systems, there are multiple layers of cache and it is important to understand some of these layers when implementing Copy Services.

Chapter 2. Planning for Advanced Copy Services implementation 7


While information is cached on both reads and writes, we are only concerned with write caching, when we consider Copy Services

Timeline of a write cacheFigure 2-1 on page 8 shows a basic diagram, outlining the timeline of a write to a storage device when a write cache is in use.

Using a cache speeds up write operations in the following way. Without the cache, the host will send a write request directly to the storage device. The storage device will respond and data will travel from client to device. Once all of the data is in place on the storage device it will signal that the process is complete. The transfer is limited by the speed at which the disks can provide data. For a storage subsystem. this operation could take 10s of milliseconds to complete, in addition to the transit time over the network.

With a cache in place, the storage client need only transfer the data to the cache. The cache works much more quickly than the disks themselves, so this transfer takes less time to complete. This corresponds to the time interval between T0 and T1 on Figure 2-1. At some later time (T2), the cache performs a destaging operation and saves the cached data to the physical disk. This completes at time T3. The host, however, is completely unaware of this operation. For a storage subsystem with cache, the network transit time becomes the limiting factor and the operation time could be less than a millisecond.

Figure 2-1 Simple write operation with caching

Figure 2-1 is a simplification. In a production environment, you will have cache above cache above cache. Consider a system with SAN Volume Controller (SVC) installed, as displayed in Figure 2-2 on page 10. This shows how a production environment can have many caches between the application that end users see and the disks in a storage subsystem.

The numbering in the list below refers to the numbering of the caches in Figure 2-2 on page 10:

1. Database cache: Queries to the database may well be cached by the database process to allow for rapid completion of application processes. This may be done in the form of storing queries for later execution.

2. Filesystem cache: IO commands to the filesystem may be cached in memory to allow faster completion to processes running on the server. These commands will be passed to underlying storage at a later time or when the cache needs to make space for subsequent IO commands.



3. SVC cache: SVC has a redundant write-cache which allows for fast completion of IO commands to the server. The data from these commands are destaged at a later time.

4. Storage controller cache: Most modern storage controllers have write cache of their own. Once data is stored in these caches, the system will report a successful conclusion. At a later date, this information will be written to the physical disks themselves.

The diagram shows 4 layers representing a function in the system. As data flows from layer to layer, it passes through a cache. Each cache makes a commitment to the layer above that any data that it accepts will eventually be destaged. However, there is no commitment as to when the data will be committed. This is important to remember when planning a solution that uses Copy Services. It should also be understood that each layer has no idea that there is data in the caches owned by higher layers.

Remember:

� A cache makes a commitment that completed writes will be written to the layer below, but with no commitment as to when

� A cache has no idea about the existence of caches above it.



Figure 2-2 Layers of caching in a production environment

Impact of write caches to Copy ServicesCopy Services are provided by SAN Volume Controller at the storage layer. However, they are used to solve business challenges at the application or hosting layer. As a result, it is important to be aware of the data flow between the layers when planning your Copy Services solution.

Consider the FlashCopy function. This provides a Point In Time (PIT) copy of your data. When using the FlashCopy function, it is important to ensure that the VDisk that FlashCopy is



copying is identical to the volume that your application is seeing. Because of write-caching, this is not always the case.

In Figure 2-3 you can see where FlashCopy exists with respect to caches. The server sees a SCSI LUN presented by the SVC, corresponding to a VDisk. When an application performs certain actions, data is written to this SCSI LUN. Because of the caching, data written at some time t0 may not actually hit the VDisk until a later time t1. Normally this is not a problem, because:

� The writes will eventually get there� All reads go through the same cache, so there is never any noticeable inconsistency

between what is on the SCSI LUN that the server sees and what is on the VDisk that SVC presents.

Figure 2-3 Position of FlashCopy with respect to caches

When using Copy Services, this does become a problem. The business problem that is being solved demands that the data on the LUN be copied. As a result, every block of data that the system has written to its LUNs must be captured by the FlashCopy. However, as commented above, SVC works with VDisks and has no idea that there is data in the caches above it.

In order to ensure that the FlashCopy Target VDisk is an identical match to the SCSI LUN that the server sees, data must be purged from the caches at all layers and the system must be configured to place no more data in the caches until the FlashCopy has been started. This can be done by

� Quiescing IO operations



� Setting caches to a write-through mode � Vendor-specific processes, such as setting applications to special backup modes that

allow for consistent copies to be made.

The same phenomenon exists for Global and Metro Mirror. In these cases, the Primary and Secondary VDisks remain in synch with one another, but, as with FlashCopy VDisks, there will be a time lag between the SVC VDisks and the SCSI LUN that the server sees. Thus, when the relationship is broken or stopped, the data on the VDisk will not be identical to the data that application sees on the LUN.

Dealing with write cachesAll of the above is a property of write caches and not a limitation of SVC. Once data is in the SVC write-cache, it will be handled appropriately; write commands that have been reported as having completed successfully will be completed.

You must be aware of all the write caches on systems that you implement Copy Services on. This may involve discussions with vendors or systems experts to understand what data is cached, when it is cached and how long it is cached for. In addition, you should find out if there are methods for purging the cache, disabling the cache and checking the cache state. This information will be critical for planning your end to end solution.

2.4.2 Understanding consistency

Consistency is a critical concept when it comes to Copy Services. By ensuring consistency in its Copy Services, SVC allows systems running on attached hosts to access copied VDisks effectively. It does this by ensuring Read Stability

If two reads are submitted for the same area and those reads complete successfully and there was no intervening (in time) or overlapping (in time) write activity then the two reads must return the same data

This may seem obvious, but it’s useful to state it explicitly. In addition to this, the following behavior is also displayed:

If a write to a certain area completes successfully then any future reads from that area will return the data that was written (until a further write to the same area is completed successfully)

Again, this may seem obvious, but these two characteristics define how the ordering of reads and writes behave. They allow us to define our expectations for what is on a disk at any single point in time.

Take note that writes must complete successfully before you can be certain of what a read operation will return. If you send a read (R1) to an area of disk, then send a write (W1) to the same area of disk and then send another read (R2) to the same area of disk, before W1 completes, the only guarantee is that R2 will match R1 or W1.

Finally, we discuss write ordering. Many applications rely on write ordering in order to survive failures. Write ordering means:

If a write (W1) is sent and then reported as complete, and then a second write (W2) is sent and then reported as complete, then the cache will submit these writes to the next level in the same order.



However, if W2 is sent before W1 completes, then there is no guarantee as to the order in which the cache will submit these to the layer below. In fact, the cache may well batch these and submit them at the same time.

Consistency for a single copy relationshipLike all SCSI Storage Devices, an SVC VDisk is made up of blocks. Consider a VDisk at a time t0. A process starts writing data to numerous blocks on the VDisk until time t1, at which point it stops. If you now read from all of the blocks, you will have a set of data the represents the VDisk’s state at time t1. If at some later point in time, t2, you read from all of the blocks again, because of Read Stability, you will get an identical set of data. Therefore, you can say that the data set from t2 is the same data set as that from t1. Since each block from t2 matches each block from t1 we say that it is a consistent copy.

To demonstrate this concept further, take a look at Figure 2-4 on page 14. This figure demonstrates the idea of consistency in the context of Copy Services. Consider two brand new VDisks: a Source and a Target (or a Primary and a Secondary in the case of Metro/Global Mirror)



Figure 2-4 Explaining Copy Services Consistency

At t0, we have 2 VDisks which are about to be put into a copy relationship. They are very small VDisks with only 4 blocks each. The precise type of copy relationship is not relevant as the concept is the same. At this point in time, the relationship is inconsistent because the Target VDisk data set does not match any version of the Source VDisk data set. The subsequent points in time are described in the following list:

1. At this point, block A is copied from the Source to the Target. The relationship is still inconsistent.

2. At this point, block B is copied from the Source to the Target. The relationship is still inconsistent.

3. At this point, block C is copied from the Source to the Target. The relationship is still inconsistent.

4. At this point, block A is overwritten by the server with block E. The relationship is still inconsistent.



5. At this point, block D is copied from the Source to the Target. The relationship is now consistent as the Target VDisk data set represents the same data set as the Source VDisk at t3.

6. At this point, block E is copied from the Source to the Target. The relationship is still consistent and is now synchronized

The difference in time between t3 and t5 represents the Recovery Point Objective1 (RPO) for this VDisk.

Consistency for a group of copy relationshipsHaving described the concept of consistency for a single copy relationship, we go on to define the concept of consistency for a group of copy relationships.

A single relationship is consistent if the data set of the Target VDisk matches the data set of the Source VDisk at some point in time in the past. In the same way, a group of relationships is consistent if each of the relationships in the group are themselves consistent and each relationship represents the same point in time as all of the others.

Once consistency has been attained, it is maintained until specific user actions are taken to disrupt it. Actions that result in consistency being broken are explained later in this Redbook.

Consistency for FlashCopyOnce a FlashCopy mapping started, the Source and Target VDisks are, by definition, consistent. The mapping represents a single point in time and every FlashCopy mapping in a consistency group represents the same point in time.

Bear in mind that the Target VDisks are consistent with the Source VDisks that SVC is presenting. Because of write caches above the Storage Layer, it is possible that these differ from the Logical Unit that the application sees. This is why the caches should be purged prior to starting FlashCopy mappings.

Consistency for Metro and Global MirrorWhen Metro/Global Mirror relationships are started, they are inconsistent. They must go through a synchronization stage before they become consistent (much like the process outlined in Figure 2-4 on page 14). Once this process is complete, the relationships will stay consistent throughout normal running.

At this point, we introduce the concept of synchronized relationships. As outlined above, a relationship is consistent if the data set on the Secondary VDisks represent a data set on the Primary VDisks at some point in time.

A relationship is synchronized if it is consistent and the point in time that the Secondary VDisks represent is the current point in time. Put another way, the Secondary VDisks contain exactly the same data as the Primary VDisks. This is represented by t6 on Figure 2-4 on page 14. This definition is stretched a little when it comes to Global Mirror due to the asynchronous nature of the function. The Global Mirror design is such that the lag is kept as low as possible. If the lag grows too long for an extended period of time, then SVC will stop the relationship to prevent performance on the Primary VDisk from getting too poor.

1 See Using the SVC for Business Continuity, SG24-7371 to learn more about the language of Business Continuity



2.5 Accessing the copied data sets

The decision of when to access the data sets that have been copied is very much a business decision. It stems from the rationale behind copying the data in the first place. We do not discuss that aspect here.

Then technical practicalities of presenting replicated VDisks from the SVC is straightforward and are discussed in Chapter 6, “Implementing FlashCopy” and Chapter 4, “Implementing Remote Copy”. Accessing these data sets from your servers is more complex and will be discussed in Chapter 7, “Server integration”.

2.6 Determining how much data can be replicated

A major part of Copy Services planning is determining how much data you actually want to replicate. As discussed in “Deciding what data to copy” on page 6, there are many costs associated with implementing Copy Services and some of these may act to limit the amount of data that you wish to replicate.

In addition to these business limitations, there are some technical limitations that you should be aware of when planning your solution. Copy Services require the use of SVC resources and so are not limitless. However, Table 2-1 shows that these limitations are enough to allow all but the most expansive Copy Services solutions. As new versions of SVC have been released, these limitations have become less restraining and we list here the limitations as they have changed over past releases. In each column, we highlight the item that has changed since the previous release

Table 2-1 SVC Copy Services Limitations

Objects 3.1.0.xa

a. Global Mirror not supported until SVC 4.1.1.x

4.1.0.xa 4.1.1.x 4.2.0.x 4.2.1.x

FlashCopy Mappings per cluster 2048 2048 2048 3855 3855

FlashCopy Mappings per consistency group

512 512 512 512 512

FlashCopy consistency groups 128 128 128 128 128

FlashCopy VDisk per IO Group 16 TB 16 TB 16 TB 40 TB 1024 TBb

b. Default is 40 TB. The bitmap space that is used to control Copy Services is shared between FlashCopy and Metro/Global Mirror.

FlashCopy targets per source 1 1 1 16 16

Metro/Global Mirror relationships per cluster

1024 1024 1024 1024 1024

Metro/Global Mirror consistency groups

256 256 256 256 256

Metro/Global Mirror VDisk per IO Group

16 TB 16 TB 16 TB 40 TB 1024 TBb

Metro/Global Mirror round-trip latency (ms)

68c/20d

c. Fibre Channel Extendersd. SAN Routers

68c/20d 80 80 80



You should refer to the latest SVC documentation to find out what limitations apply to the level of SVC that you are running with.

2.6.1 VDisk capacity limit

All Copy Services use bitmaps to track which VDisk data need to be copied from the Source/Primary to the Target/Secondary. The amount of space made available to these bitmaps is variable. This feature allows the user to trade off memory usage between FlashCopy, Metro Mirror and Global Mirror.

Copy Services use ‘grains’ to track what needs to be copied. Each grain represents 256 KB worth of data that may need copying for Metro/Global Mirror. For FlashCopy, a grain is either 64KB or 256KB at the user’s choice. Each bit in the bitmap represents 1 grain.

Bitmap space is owned by an IO Group. When a mapping or relationship is created, you can choose which IO Group will provide the bitmap space. This can be a different IO Group to the ones providing the associated VDisks.

The amount of VDisk capacity that can be copied depends on the following:

� The amount of the memory allocated to hold the bitmap (bitmap space)� The grain size� If FlashCopy is being used, then, whether it is normal or incremental

The maximum size of the bitmap space for one IO group is 512MB, with 20MB being the default. The bitmap space for an IO Group must be shared between FlashCopy and Metro/Global Mirror. If all of the space is used for FlashCopy, for instance, then there would be no space available for Metro/Global Mirror.

Bitmap space is allocated to FlashCopy mappings and Metro/Global Mirror relationships in increments of 4KB. This corresponds to 32,768 bits of bitmap space and so 32,768 grains. Thus, the smallest unit of VDisk capacity that can be copied is 8GB (for 256KB grains) or 2GB (for 64KB grains). Because bitmap space is assigned in fixed increments, it takes the same amount of bitmap space to copy a 1GB VDisk as it takes to copy an 8GB VDisk.

With Incremental FlashCopy, there are two bitmaps per FlashCopy mapping to be stored, which makes it consume twice the bitmap space of a normal FlashCopy mapping.

Table 2-2 on page 17 shows the relationship between VDisk capacity and bitmap space depending on the size of the grain and the kind of copy service being used.

Table 2-2 Bitmap space required for Copy Services

Copy Service Grain size (KB)

VDisk capacity provided by bitmap space

1 Byte 4KB 1MB 20MB 512MB

Metro/Global Mirror 256 2MB 8GB 2TB 40TB 1024TB

FlashCopy 256 2MB 8GB 2TB 40TB 1024TB

FlashCopy 64 512KB 2GB 512GB 10TB 256TB

Incremental FlashCopy 256 1MB 4GB 1TB 20TB 512TB

Incremental FlashCopy 64 256KB 1GB 256GB 5TB 128TB



2.7 Understanding your current environment

Prior to implementing Copy Services, you should have a good understanding of your storage environment. Implementing Copy Services leads to increased data transfers, so you must understand your current configuration before implementing a solution that will affect it.

Copy Services operations are driven by two things:

� Background copy processes� Foreground application processes

Both of these processes will generate traffic on your SAN and so it’s important to understand what your SAN can take, without disruption and how much traffic your application processes are generating.

2.7.1 Storage controller information

Controllers are designed to take as much IO as possible, before latency starts to creep up. Naturally, as a system with finite resources, you cannot increase the workload on a controller indefinitely without consequences. Figure 2-5 shows an illustration of how latency is affected as you increase the workload on a storage controller. All commands will have some latency due to transit time to the storage controller and service time of the commands. As the number of commands sent to a controller each second increases, there will be an increased wait due to queuing, which will be of the order of the service time. The dotted line on Figure 2-5 shows the maximum workload that should be drive to this storage controller.

Note: For the FlashCopy mapping bitmap space, the amount of mappings has to be taken into account.

Each mapping target for a FlashCopy uses up bitmap space. This means for Multiple Target FlashCopy, even though there is only one source vdisk involved, multiple targets will consume the same amount of bitmap space as the same number of targets of independent FlashCopy mappings would use up.



Figure 2-5 Latency of commands to a storage controller as workload increases

If you have a well equipped test lab, you may be in a position to create this kind of graph on the same equipment as that in your production environment. Failing that, your storage controller vendor should be able to advise the maximum workload that your storage controllers are capable of without significant increases in latency.

Once you have done this, you should investigate your storage controllers to see what kind of workload they are currently handling. The method by which you do this depends on the controller that you are using. If you use Tivoli® Productivity Center (TPC) for Disk and it is configured to monitor your storage controllers, then you can gather this information that way. Alternatively, if you have TPC installed and it is monitoring your SVC, you can gather performance information about your MDisks and work it out that way. Bear in mind that any performance statistics gathered about your MDisks only tell part of the storage if your storage controller is also presenting volumes directly to other servers.

2.7.2 Server information

It is important to know what kind of workload your servers are driving to the system. This include information about

� Data rate (MB/s) read and written� Workload patterns, e.g. sequential, random� Response requirements, i.e. maximum acceptable latency

It is important to get specific information and not settle for requirements such as “As fast as possible” and “No latency”. There is no way to meet these requirements and so they are of no



use to anyone. That’s not to say that the servers don’t have demanding requirements, but you will need real figures in order to plan your Copy Services system.

When a VDisk is being copied (either by FlashCopy or Metro/Global Mirror) writes to the VDisks will drive data copies. In the case of FlashCopy, the first write to a block on the source VDisk will result in data being written to the target VDisk; further writes to the same block will generate no extra IO. In the case of Metro/Global Mirror, each write to a block on the primary VDisk will be replicated to the secondary VDisk.

You must ensure that the storage controllers that the target/secondary VDisks are on are up to the workload that will be driven to them. This is particularly important for Metro/Global Mirror relationships. For instance, you may have a set of VDisks that are residing on DS8000™ storage controllers; these VDisks are comfortably handling a high write workload. You decided to replicate these VDisks to a remote site, using Metro Mirror. If the remote VDisks are residing on DS4000™ storage controllers, then you could expect to see problems. The remote VDisks must be able to handle the same write workload as the local storage or else you will start to see performance problems.

Under most workload patterns, Metro/Global Mirror does not cause serialization of writes that are submitted at the same time. Consider a server that sends 5 write operations to 5 different locations on a VDisk, all at the same time. Obviously, this server is not concerned about the write order as it has not waited for completion from the first before sending the second, third, fourth or fifth. Thus for Metro Mirror, these 5 writes will be sent to the secondary disk in any order and completion will be reported to the server as and when the operations are complete. For Global Mirror, these writes will be completed once they are in the cache and then written to the remote cluster as soon as possible.

There is one workload pattern which can lead to performance issues with Global Mirror. In order to maintain write ordering, the Global Mirror code only allows a single write to be active on any given block of a VDisk. Thus, if a write comes in from a host to a block, and that block still has data waiting to be written to the remote cluster, the new write will be delayed until the copy has been completed. Applications that deliver this kind of workload will not achieve the performance that Global Mirror is intended to support.

The Per VDisk Cumulative total number of overlapping writes (gwo) statistic provided by the SVC gives a count of colliding writes. If bad performance is being seen, checking this statistic can be used to determine if you are hitting this limitation.

2.7.3 Rate of transfer of data

As mention at the start of this section, Copy Services are driven, in part, by foreground application processes. The following operations will lead to data being copied:

� Every write to a grain on the primary VDisk of a Metro/Global Mirror relationship� The first write to a grain on the source or target VDisk in a FlashCopy mapping.� The first read from a grain on the target VDisk in a FlashCopy mapping.

The rate at which your applications change their data will give you an indication of the rate at which data will be transferred as part of Copy Services mapping/relationship.

Copy Services are also driven by background copy. The rate at which data is transferred is configured on a per-mapping basis for FlashCopy. For Metro/Global Mirror, the rate of data transfer is controlled by the intercluster bandwidth setting.



2.7.4 Intersite latency

Table 2-1 on page 16 shows the latency requirements for an intercluster connection. This latency requirement should be met in all situations. At the very least, the peak workload on the intersite link should not result in the latency requirement being missed.

If the intersite link is made up of multiple links trunked together, then the link should be architected such that the failure of a single link does not result in the latency requirement being missed.

2.8 Administration of Copy Services

In order to administer Copy Services, users needs access to the SVC Cluster. SVC provides multiple user roles to limit the actions that can be taken via the GUI and the command line. By doing this, system administrators can access the SVC Cluster in a limited capacity to administer the copy relationships associated with their servers’ storage. These commands are all logged in the Audit Log which can be used for problem resolution and to satisfy auditing requirements

2.8.1 SSH Keys

Access to the SVC Command Line is managed through SSH Keys. To provide a user with access to the SVC Command Line you should create a public-private key pair. The private key should be given to the user and kept safe. The public key should be uploaded to the SVC Cluster and given a descriptive label that uniquely describes that user. This label is used in the audit log (see 2.8.2, “Command Auditing”)

Access to the SVC GUI is managed through ICAT users.

2.8.2 Command Auditing

SVC keeps an audit log of all successfully executed commands. With appropriate configuration of SSH keys and ICAT users, you can use the audit logs to track which users made what changes to the cluster and when they made the changes.

2.8.3 Authorization Roles

The following roles are available

� Service� Monitor� CopyOperator� Administrator

These roles are applied to the public keys which have been uploaded to the cluster. When a user connects to the cluster, they do so with a specific private key. This corresponds to a public key on the cluster. Each command that they submit is checked against that public key’s authorization role before being executed and may fail if it does not have the required level of authority.

The commands that each role is authorized to execute are give in Table 2-3.



Table 2-3 Commands available to each Role

Up to 100 different keys can be stored on a cluster, so there is ample scope for assigning individual keys (and so roles) to systems or system administrators.

Currently, there is no way to limit which copy relationships a user can affect. A CopyOperator can execute commands against any FlashCopy mapping or Metro/Global Mirror relationship. However, the audit log will track all commands that are executed so rogue operators cannot act with impunity.

Chapter 8, “Automation” goes into more details about authorization and auditing.

2.9 Removing workarounds with 4.2.1 new functions

If you are already running SVC Copy Services then it’s quite possible that you implemented processes to work around the limitations of the FlashCopy and Metro/Global Mirror limitations. For instance, you may have a requirement for FlashCopies to be taken of the same VDisk in a short period of time. Before Incremental FlashCopy and Multiple Target FlashCopy, this may have been impossible as the FlashCopy mappings may have needed a too large period of time to complete. With the new functions that SVC brings, you have a lot more flexibility as to when you can start new mappings involving VDisks already in a FlashCopy mapping. Chapter 5, “FlashCopy” and Chapter 6, “Implementing FlashCopy” explain the new functionality.

The changes to Metro/Global Mirror in 4.2.1 are mainly in the restrictions; they have become less restrictive. With fewer constraints, Copy Services should become simpler to manage.

Role Allowed commands

Service The following commands:� svcinfo: all commnads� svcservicetask: all commands

Monitor The following commands:� svcinfo: all commands.� svctask : finderr, dumperrlog, dumpinternallog� svcservicetask : finderr, dumperrlog� svcconfig: backup

CopyOperator All commands allowed for Monitor role plus:� svctask : � prestartfcconsistgrp� startfcconsistgrp, stopfcconsistgrp� chfcconsistgrp� prestartfcmap� startfcmap, stopfcmap� chfcmap� startrcconsistgrp, stoprcconsistgrp� switchrcconsistgrp� chrcconsistgrp� startrcrelationship, stoprcrelationship� switchrcrelationship� chrcrelationship� chpartnership

Administrator All commands can be submitted by users of this role


Draft Document for Review February 10, 2008 5:17 pm 7574 Remote Copy.fm

Chapter 3. Remote Copy: Metro Mirror and Global Mirror

In this chapter we will introduce Remote Copy services, Remote Copy comprises of two methods of copying: one is Metro Mirror and the other is Global Mirror . Metro Mirror is designed used for shorter distances and is a synchronous copy; Global Mirror is used for long distances and is asynchronous.

We will discuss the functionality and how we can use Remote Copy services in our daily operations and for business continuity.

3


7574 Remote Copy.fm Draft Document for Review February 10, 2008 5:17 pm

3.1 Remote Copy services overview

The terminology Metro Mirror and Global Mirror are IBM branded terms for the functions Synchronous Remote Copy and Asynchronous Remote Copy respectively. Throughout this book the term Remote Copy is used to refer to both functions where the text applies to either equally.

In the topics that follow we discuss some of the many features and functionality of Metro Mirror and Global Mirror.

3.1.1 Metro Mirror

Metro Mirror works by establishing a synchronous copy relationship between two VDisks of equal size. This can be an intracluster relationship, that is within the same I/O group of one SVC cluster, or intercluster, which means a relationship between two SVC clusters that are separated by distance. Those relationships can be standalone or in a Consistency Group.

Figure 3-1 shows the function of Metro Mirror relationship.

Figure 3-1 Shows the functionality of Metro Mirror write operation.

The Metro Mirror functionality ensures that I/O’s are committed to both the primary and secondary VDisks before sending confirmation of the completion to the server. This ensures that the secondary VDisk is synchronized with the primary VDisk in case of a failure. The secondary VDisk is in a read-only state and manual intervention is required to change that access to read/write state. The server administrator will also need to mount the secondary disk so the application can start to use that VDisk.

Global MirrorGlobal Mirror copy relationships work in a similar way as Metro Mirror does but by establishing an asynchronous copy relationship between two VDisks of equal size. This is mostly intended for intercluster relationships over long distances.

(1)Write

(2) Mirror write

Remote copy CACHE

Remote copy CACHE

(4) Ack

(3) Ack write

Metro Mirror Relationship

Vdisk(Primary)

Vdisk(Secondary)

24 SVC Advanced Copy Services for Business Continuity


Global Mirror functionality is that a confirmation is sent to the server before it has received good completion at the secondary VDisk. When a write is sent to a primary VDisk it is given a sequence number. Mirror writes sent to the secondary VDisk are committed in sequence number order. If a write is issued, while another write is outstanding, it may be given the same sequence number.

Figure 3-2 shows the sequence.

Figure 3-2 Global mirror write operation

Therefore, if a host has submitted concurrent writes, the SVC will batch them together (because the host does not care in what order they are written). The secondary VDisk is therefore consistent with the primary. If a host is writing to the same block, before the previous write is committed, the next write is held until the previous write is completed.

3.1.2 Cluster partnership

When creating an SVC cluster copy partnership, we are connecting two SVC clusters together which are separated by distance. After the SVC cluster partnership creation has been configured on both clusters, further communication between the nodes in each of the clusters is established and maintained via the SAN network. The intercluster link, see 3.1.3, “Intercluster communication” on page 26, controls the traffic between the nodes and helps to co-ordinate activity between the cluster. All intercluster communication goes through the Fibre Channel network.

The SVC does not require a control network or fabric to be installed to manage Remote Copy services. The control link controls the copy states and co-ordinates updates at either end. The control link is implemented on top of the same FC connection as the SVC uses for Remote Copy I/O’s. If communication between the SVC clusters is interrupted or lost, an error is logged and all copy relationships will stop automatically.

(1)Write

(3) Mirror write

Remote copy CACHE

Remote copy CACHE

(2) Ack

(4) Ack write

Global Mirror Relationship

Vdisk(Primary)

Vdisk(Secondary)

Chapter 3. Remote Copy: Metro Mirror and Global Mirror 25


3.1.3 Intercluster communication

Each SVC cluster can be configured with one other SVC cluster as its partner. When both clusters are correctly configured and can communicate with each other over the fabric, they establish further communication facilities between the nodes on each of the clusters.This is called the intercluster link.

The intercluster link keeps information about the total bandwidth available from the local cluster to the remote cluster that is to be assigned to background copy process. This is used to govern the rate at which the background copy is performed.

The intercluster link can be changed during normal operation.The default value for this parameter is 50 MBps.

3.1.4 Maintenance of the intercluster link

All SVC nodes maintain a database of the other devices that are visible on the fabric. This is updated as devices appear and disappear.

Devices that are identified as SVC nodes are categorized according to the SVC cluster to which they belong. SVC nodes that belong to the same cluster establish communication channels between themselves and begin to exchange messages to implement the clustering and functional protocols of the SVC.

Nodes that are in different clusters do not exchange messages after the initial discovery is complete, unless they have been configured in a Remote Copy relationship.

The intercluster link carries the control traffic to coordinate activity between the two clusters. Through this intercluster link there is a message sent between the nodes that just checks if a connection is established, and that is referred to as a heartbeat between the clusters. It is formed between one node in each cluster which is termed the focal point. The traffic between the focal point nodes is distributed among the logins that exist between those nodes.

If the focal point node should fail (or all its logins to the remote cluster fail), then a new focal point is chosen to run the control traffic.

The software that controls the intercluster link monitors for link errors and will exclude the link if the error rate gets too high:

� An intermittent hardware problem on a non-redundant link that leads to a link reset for the SVC cluster.

� Lost frames or delay exceeding 1.5 seconds affecting the heartbeats between the two clusters.

� A problem leading to excessive data frames being dropped (even where non-data frames are passed successfully).

� A series of failures that mean that progress is not made, such that a message is not delivered for more than 15 seconds.

If there are errors that lead to the heartbeat timeout being detected on one of the clusters, that cluster will close all process logins between the focal point nodes.

The SVC cluster monitors for errors on the intercluster link and if an error reoccurs within 30 seconds, it is ignored but if there are more than three 30 second periods, that experience an

Note: If we change the focal point node, we would cause I/O to pause, but copy relationships will not enter the consistent stopped state.



error event, the intercluster link is excluded. Then, the focal point node will deliberately prevent an intercluster link being created between the local cluster and the remote cluster. The other cluster will report partially_configured in its configuration state for the remote cluster. It is likely the other cluster will also have detected the same set of errors and excluded the link on the same basis; in which case both clusters will report partially_configured. If only one cluster detects an error and excludes the link, the cluster that does not detect the error will continue to report fully_configured.

3.1.5 I/O channel

The I/O channel is the term used to describe the connections between clusters which are used to deliver mirror writes between the clusters when using Remote Copy (intercluster).

Intracluster Remote Copy uses connectivity within a node to deliver I/O to secondary virtual disks. I/Os are never routed between nodes in different I/O groups.

The I/O channel uses the same protocols for communication between the nodes in different clusters as is used between the nodes within a local cluster, This requires only on round trip to deliver write and receive its completion.

The I/O channel traffic is monitored using the same scheme as described in 3.1.4, “Maintenance of the intercluster link” on page 26.

3.1.6 Background copy

The bandwidth that background copy can use is configured on the intercluster link as stated in 3.1.3, “Intercluster communication” on page 26. That bandwidth is divided evenly between the nodes that are performing background copy for active copy relationships. This bandwidth allocation is independent from the number of disks that a node is responsible for.

Each node, in turn, divides its bandwidth evenly between the multiple relationships performing background copy. The default value for this feature is 50MBps and this value should be set to less or equal to the bandwidth that can be sustained by the intercluster link.

For intracluster relationships each node is assigned 25 MBps for background copy.

Bandwidth impact on foreground I/O latencyThe background copy bandwidth determines the rate at which the background copy will be attempted. The background copy bandwidth can affect foreground I/O latency in one of three ways:

� The following results can occur if the background copy bandwidth is set too high compared to the intercluster link capacity:

– There is a delay in the synchronous secondary writes of foreground I/Os.– The foreground I/O latency will increase as perceived by applications.

� If the background copy bandwidth is set too high for the storage at the primary site, background copy read I/Os overload the primary storage and delay foreground I/Os.

� If the background copy bandwidth is set too high for the storage at the secondary site, background copy writes at the secondary overload the secondary storage and again delay the synchronous secondary writes of foreground I/Os.

Note: Background copy will proceed from low LBA to high LBA in sequence.



In order to set the background copy bandwidth optimally, make sure that you take into consideration all aspects of your environments. The three biggest contributing resources are the primary storage, the intercluster link bandwidth, and the secondary storage. Provision the most restrictive of these three resources between the background copy bandwidth and the peak foreground I/O workload.

3.1.7 Remote Mirror relationship

A Remote Mirror relationship is a relationship between two individual VDisks of the same size and if both VDisks are in the same cluster they must both be within the same I/O group. These disks are called a master VDisk and an auxiliary VDisk, for more details see 3.1.8, “Copy relationship naming convention” on page 28.

A primary VDisk contains a valid copy of host data and is accessible for host write I/O’s. A secondary VDisk, if consistent, contains a valid copy of host data but is not available for host write I/O’s.

In an ordinary relationship, (not idling) with one primary VDisk and one secondary VDisk, Remote Copy will try to make the data on the secondary VDisk match the data on the primary VDisk by performing writes to the secondary virtual disk.

The following types of VDisk make a copy relationship.

� Primary VDisk - enabled for read and write I/O’s� Secondary VDisk - enabled only for read I/O’s

When a Remote Copy relationship is first created, the master VDisk is always assigned the role of primary, and the auxiliary VDisk is assigned the role of secondary. Hence, the copying direction is from master to auxiliary.

These roles might be reversed and the copying direction will then be from auxiliary to master. The direction of copying is reflected in the state of the system. A Global Mirror and Metro Mirror copy relationship have the same characteristics and use the same set of commands to create and control the relationship. The relationship between the VDisks persists until it is deleted.

3.1.8 Copy relationship naming convention

The SVC introduces the following terms in Remote Copy. An explanation of those commonly encountered terms follows.

� Master VDisk is the one that is created as master and it will always be the master. When creating a master VDisk by default it will become the master/primary VDisk open for write I/O’s from the application.

� Auxiliary VDisk is a partner vdisk to the master vdisk. When the copy relationship is created all previous data on that VDisk is destroyed. This disk will always be the auxiliary VDisk in this relationship.

� Primary VDisk is the production VDisk and allows read/writes. Updates to this VDisk are mirrored to the secondary VDisk. This state can be changed when changing copy directions. When changing the copy direction we are enabling write I/O’s to the secondary VDisk and the primary will be set to read-only.

Note: When the copy relationship is in the IDLE state, both primary and secondary are enabled for read and write I/O’s.



� Secondary VDisk in an active relationship is always in read-only mode when it contains consistent image of the primary VDisk. This role changes when you change the copy direction.

Figure 3-3 shows how the role name follows the copy direction.

Figure 3-3 Naming conventions for VDisks in copy relationship.

When you create a copy relationship you need to state which VDisk is the master VDisk, and the other VDisk in that relationship will become the auxiliary. This relationship does not change even if you change copy directions.

3.1.9 Supported methods for synchronizingThis section describes three methods that can be used to establish a relationship.

Full synchronization after CreateThis is the default method. It provides full copy of the primary VDisk to the secondary VDisk. It is the simplest in that it requires no administrative activity apart from issuing the necessary commands. However, in some environments the bandwidth available will make this method unsuitable.

The sequence for a single relationship is as follows:

� A mkrcrelationship is issued without specifying the -sync flag.� A startrcrelationship is issued without the -clean flag.

Note: When creating the SVC cluster partnership neither cluster is master/primary or auxiliary/secondary. They are both equal in this partnership and when you use the mkpartnership command you are using the only Remote Copy command that needs to be issued to both clusters, all other Remote Copy commands need be issued from only one of the clusters in the relationship.

Master VDisk

Auxiliary VDisk

Role: Primary

Role:Primary

Role:Secondary

Role: Secondary

´Copy direction

´Copy direction



Synchronized before CreateIn this method, the administrator must ensure that the primary and secondary virtual disks contain identical data before creating the relationship. There are two ways in which this might be done:

� Both disks are created with the -fmtdisk feature so as to make all data zero.� A complete tape image (or other method of moving data) is copied from one disk to the

other.

In either technique, no write I/O must take place to either primary or secondary before the relationship is established.

Then, the administrator must ensure that:

� A mkrcrelationship is issued with the -sync flag.� A startrcrelationship is issued without the -clean flag.

If these steps are not performed correctly, the relationship will be reported as being consistent, when it is not. This is likely to make any secondary disk useless. This method has an advantage over the full synchronization, in that it does not require all the data to be copied over a constrained link. However, if the data needs to be copied, the master and auxiliary disks cannot be used until the copy is complete, which might be unacceptable.

Quick synchronization after CreateIn this method, the administrator must still copy data from primary to secondary. But it can be used without stopping the application at the primary. The administrator must ensure that:

� A mkrcrelationship is issued with the -sync flag.� A stoprcrelationship is issued with the -access flag.� A tape image (or other method of transferring data) is used to copy the entire primary disk

to the secondary disk.

Once the copy is complete, the administrator must ensure that:

� A startrcrelationship is issued with the -clean flag.

With this technique, only the data that has changed since the relationship was created is copied from the primary and secondary. As with “Synchronized before Create” on page 30, the copy step must be performed correctly, or else the secondary will be useless, though the copy will report it as being synchronized.

3.1.10 Remote mirror consistency groups

A consistency group consist of one or more copy relationships. By grouping the relationships we ensure that these relationship are managed so that data within the group is in a consistent state. If the consistency group is empty, it acquires the type of the first relationship that is added to it, Metro Mirror or Global Mirror. Therefore, each relationship that you add to an already in-use consistency group must be of the same type.

Consistency groups address the issue where the objective is to preserve data consistency across multiple Remote Mirrored VDisks because the applications have related data which spans multiple VDisks. A requirement for preserving the integrity of data being written is to ensure that “dependent writes” are executed in the application's intended sequence.

Note: In an SVC cluster relationship it is a good rule to use the same SVC cluster for your master VDisks, even if the master SVC cluster contains the secondary VDisk.



Copy commands can be issued to either Metro Mirror or Global Mirror consistency groups, which affects all Remote Mirror relationships in that consistency group, or to a single standalone Remote Mirror relationship, if it is not part of a consistency group.

In summary:

� Remote Mirror relationships can be part of a consistency group, or be stand-alone and therefore handled as single instances.

� A consistency group can contain zero or more relationships. An empty consistency group, with zero relationships in it, has little purpose until it is assigned its first relationship, except that it has a name.

� All the relationships in a consistency group must have matching master and auxiliary SVC clusters.

� All relationships must be of the same type.

The rules behind consistency mean that certain configuration commands are prohibited, where this would not be the case if the relationship was not part of a consistency group.

3.2 State overview

After creation of a Remote Copy relationship, that relationship has a state at any given time. Stand- alone relationships and consistency groups share a command configuration and state modes. We will discuss those states and start with the three most common ones as shown in Figure 3-4 on page 32 :connected versus disconnected, consistent versus inconsistent and consistent versus synchronized.



Figure 3-4 State overview.

3.2.1 Connected or disconnected

This distinction can arise when a Remote Copy relationship is created with two virtual disks in different clusters. It is possible that the communication line between those clusters gets broken, and therefore communication between the two clusters is lost.

When the two clusters can communicate, the clusters and the relationships spanning them are described as connected. When they cannot communicate, the clusters and the relationships spanning them are described as disconnected.

If the cluster cannot communicate, each of the clusters is left with half the relationship and has only a portion of the information that was available to it before. Some limited configuration activity is possible, but not all.

When the clusters can communicate again the relationships become connected again. Remote Copy automatically reconciles the two state fragments, taking into account any configuration or other event that took place while the relationship was disconnected.

inconsistent

INCONSISTENT STOPPED

INCONSISTENT COPYING

Start

create_unsync

IDLING

Forced Startout of sync

Stop

Legend

REMOTE COPY STATE

Remote Copy Event

consistent

CONSISTENT STOPPED

StartCONSISTENT

SYNCHRONIZED

Stop

Forced Start Backgroundcopy complete

create_sync

Stopenable access

Stopenableaccess

Startin sync



As a result, the relationship can either return to the state it was in when it became disconnected, or it can enter a different connected state.

3.2.2 Consistent or inconsistent

A copy relationship between primary VDisk and a secondary VDisk can be described as being consistent or inconsistent. Consistency Groups containing copy relationships can also be described as being consistent or inconsistent. The Consistent/Inconsistent attribute describes the relationship of the data which exists on the secondary VDisk to that which exists on the primary VDisk. It can be considered a property of the secondary VDisk itself.

A secondary VDisk is described as consistent if it contains data that could have been read by a host system from the primary if, for example, power had failed at some point in time while I/O was in progress and the power was later restored.

This imaginary point in time is defined as the recovery point. The requirements for consistency are expressed with respect to activity at the primary up to the recovery point:

� The secondary VDisk contains the data from all writes to the primary for which the host had received good completion and that data had not been overwritten by a subsequent write (before the recovery point)

If we look at it from the host site then consistency means that a secondary VDisk contains the same data as the primary VDisk at the recovery point (the time at which the example power failure occurred).

If an application is designed to cope with unexpected power failure, this guarantee of consistency means that the application will be able to use the secondary and begin operation just as if it had been restarted after the hypothetical power failure.

Again, the application is dependent on the key properties of consistency:

� Write ordering � Read stability for correct operation at the secondary

If a relationship, or set of relationships, is inconsistent and an attempt is made to start an application using the data in the secondaries, a number of outcomes are possible:

� The application might decide that the data is corrupt and crash or exit with an error code.� The application might fail to detect the data is corrupt and return erroneous data.� The application might work without a problem.

Because of the risk of data corruption, and in particular undetected data corruption, Metro Mirror strongly enforces the concept of consistency and prohibits access to inconsistent data.

Consistency as a concept can be applied to a single relationship or a set of relationships in a consistency group. Write ordering is a concept that an application can maintain across a number of disks accessed through multiple systems and therefore consistency must operate across all those disks.

When deciding how to use consistency groups, the administrator must consider the scope of an application’s data, taking into account all the interdependent systems which communicate and exchange information.

Note: Relationships that are configured between virtual disks in the same cluster will never be described as being in a disconnected state since only one cluster is involved.



If two programs or systems communicate and store details as a result of the information exchanged, then either of the following actions might occur:

� All the data accessed by the group of systems must be placed into a single consistency group.

� The systems must be recovered independently (each within its own consistency group). Then, each system must perform recovery with the other applications to become consistent with them.

3.2.3 Consistent or synchronized

A copy which is consistent and up-to-date is described as synchronized. In a synchronized relationship, the primary and secondary VDisks are only different in regions where writes are outstanding from the host.

Consistency does not mean that the data is up-to-date. A copy can be consistent and yet contain data that was frozen at some point in time in the past. Write I/O might have continued to a primary and not have been copied to the secondary. This state arises when it becomes impossible to keep up-to-date and maintain consistency. An example is a loss of communication between clusters when writing to the secondary.

When communication is lost for an extended period of time, Metro Mirror tracks the changes that happen at the primary, but not the order of such changes, nor the details of such changes (write data). When communication is restored, it is impossible to make the secondary synchronized without sending write data to the secondary out-of-order, and therefore losing consistency.

Two policies can be used to cope with this:

� Take a point-in-time copy of the consistent secondary before allowing the secondary to become inconsistent. In the event of a disaster before consistency is achieved again, the point-in-time copy target provides a consistent, though out-of-date, image.

� Accept the loss of consistency during resync, and loss of useful secondary, while making it synchronized.

3.2.4 Detailed Information about copy states.

The available states are described in detail in this section, and they are:

� Consistent Stopped� Consistent Synchronized� Inconsistent Stopped� Inconsistent Copying� Idling� Idling disconnected� Inconsistent disconnected� Consistent disconnected� Empty (applies only to Consistency Groups)

Consistent StoppedThis is a connected state. In this state, the secondary contains a consistent image, but it might be out-of-date with respect to the primary.

This state can arise when a relationship was in consistent synchronized state and suffers an error which forces a consistency freeze. It can also arise when a relationship is created with a sync parameter set.



Normally, following an I/O error, subsequent write activity cause updates to the primary and the secondary is no longer synchronized. In this case, to re-establish synchronization, consistency must be given up for a period. A Start command with the Force option must be used to acknowledge this, and the relationship or consistency group transitions to inconsistent copying. Do this only after all outstanding errors are repaired.

In the unusual case where the primary and secondary are still synchronized (perhaps following a user stop, and no further write I/O was received), a Start command takes the relationship to consistent synchronized. The No Force option is required. Also in this unusual case, a Switch command is permitted which moves the relationship or consistency group to consistent synchronized and reverses the roles of the primary and secondary (switches copy directions).

If the relationship or consistency group becomes disconnected, then the secondary side transitions to consistent disconnected. The primary side transitions to idling disconnected.

An informational status log is generated every time a relationship or consistency group enters the consistent stopped with a status of Online state.

Consistent SynchronizedThis is a connected state. In this state, the primary VDisk is accessible for read and write I/O and the secondary VDisk is accessible for read-only I/O. This state is also called “normal copy state” and indicates that now your relationship is both consistent and synchronized.

Writes that are sent to the primary VDisk are sent to both primary and secondary VDisks. Either good completion must be received for both writes, or the write must be failed to the host, or a state transition out of consistent synchronized must take place before a write is completed to the host.

A stop command takes the relationship to consistent stopped state. A stoprcrelationship command with the -access argument takes the relationship to the Idling state.

A switch copy direction command leaves the relationship in the consistent synchronized state, but reverses the primary and secondary roles and therefore the copy direction.

A Startrcrelationship or startrcconsistgrp command is accepted, but has no effect.

If the relationship or consistency group becomes disconnected, the same transitions are made as for consistent stopped.

Inconsistent StoppedThis is a connected state. In this state, the primary is accessible for read and write I/O but the secondary is not accessible for either. The command “start copy process” needs to be given to make the secondary consistent.

This state is entered when the relationship or consistency group was inconsistent copying and has either suffered a persistent error or received a stop command which has caused the copy process to stop.

Note: It is possible to create some form of automation that monitors the status log. The SVC cluster has the possibility to send an SNMP trap that triggers an automation script that issues the start command in a case of loss of synchronization when certain log entries appear.



A Start command causes the relationship or consistency group to move to the inconsistent copying state again. A stop command is accepted, but has no effect.

If the relationship or consistency group becomes disconnected, the secondary side transitions to inconsistent disconnected. The primary side transitions to idling disconnected.

Inconsistent CopyingThis is a connected state. In this state, the primary is accessible for read and write I/O but the secondary is not accessible for either read or write I/O.

Inconsistent copying runs as a background copy process, which copies data from the primary to the secondary VDisk.

This state is entered after a start command is issued to an inconsistent stopped relationship or consistency group. It is also entered when a forced start is issued to an idling or consistent stopped relationship or consistency group.

In the absence of errors, an inconsistent copying relationship is active, and the copy progress increases until the copy process completes. In some error situations, the copy progress might freeze or even return to a known state.

A persistent error or “stop copy process” command places the relationship or consistency group into an inconsistent stopped state. A start command is accepted, but has no effect.

If the background copy process completes on a standalone relationship, or on all relationships for a consistency group, the relationship or consistency group transitions to consistent synchronized.

If the relationship or consistency group becomes disconnected, then the secondary side transitions to inconsistent disconnected. The primary side transitions to idling disconnected.

IdlingThis is a connected state. Both master and auxiliary disks are operating in the “primary” role. Consequently both are accessible for write I/O.

In this state, the relationship or consistency group accepts a start command. Metro Mirror maintains a record of regions on each disk which received write I/O while Idling. This is used to determine what areas need to be copied following a start command.

The Start command must specify the new copy direction. A Start command can cause a loss of consistency if either VDisk in any relationship has received write I/O. This is indicated by the synchronized status. If the start command would lead to a loss of consistency, then the Force parameter must be specified.

Following a Start command, the relationship or consistency group transitions to consistent synchronized if there is no loss of consistency, or to inconsistent copying if there is such a loss.

Also, while in this state, the relationship or consistency group accepts a Clean option on the start command. If the relationship or consistency group becomes disconnected, then both sides change their state to idling disconnected.

IdlingDisconnectedThis is a disconnected state. The VDisk or disks in this half of the relationship or consistency group are all in the “primary” role and accept read and write I/O.



The main priority in this state is to recover the link and make the relationship or consistency group connected once more.

No configuration activity is possible (except for deletes or stops) until the relationship becomes connected again. At that point, the relationship transition to a connected state. The exact connected state which is entered depends on the state of the other half of the relationship or consistency group, which depends on:

� The state when it became disconnected� The write activity since it was disconnected� The configuration activity since it was disconnected

If both halves are idling disconnected, then the relationship becomes idling when reconnected.

While idling disconnected, if a write I/O is received which causes loss of synchronization and the relationship was not already stopped (either through user stop or a persistent error), then an error log entry is raised to notify this. This error log is the same as that raised when the same situation arises when consistent synchronized.

Inconsistent DisconnectedThis is a disconnected state. The VDisks in this half of the relationship or consistency group are all in the secondary role and do not accept read or write I/O.

No configuration activity except for deletes is permitted until the relationship becomes connected again.

When the relationship or consistency group becomes connected again, the relationship becomes inconsistent copying automatically unless either:

� The relationship was inconsistent stopped when it became disconnected� The user issued a stop command while disconnected

In either case, the relationship or consistency group becomes inconsistent stopped.

Consistent DisconnectedThis is a disconnected state. The VDisks in this half of the relationship or consistency group are all in the secondary role and accept read I/O but not write I/O.

This state is entered from consistent synchronized or consistent stopped when the secondary side of a relationship becomes disconnected.

In this state, the relationship or consistency group displays an attribute of FreezeTime ( see 3.4.5, “Freeze/Run on consistency groups” on page 40) which is the point in time that consistency was frozen. When entered from consistent stopped, it retains the time it had in that state. When entered from consistent synchronized, the FreezeTime shows the last time at which the relationship or consistency group was known to be consistent. This corresponds to the time of the last successful heartbeat to the other cluster.

A stop command with enable write access to secondary transitions the relationship or consistency group to an idling disconnected state. This allows write I/O to be performed to the VDisks and is used as part of a disaster recovery scenario.

When the relationship or consistency group becomes connected again, the relationship or consistency group becomes consistent synchronized only if this does not lead to a loss of consistency under these criteria:

� The relationship was consistent synchronized when it became disconnected.



� No writes received successful completion at the primary while disconnected.

Otherwise the relationship become consistent stopped. The FreezeTime setting is retained.

EmptyThis state only applies to consistency groups. It is the state of a consistency group which has no relationships and no other state information to show.

It is entered when a consistency group is first created or a all relationship are removed from the consistency group.It is exited when the first relationship is added to the consistency group at which point the state of the relationship becomes the state of the consistency group.

3.3 Metro Mirror and Global Mirror configuration limits

Table 3-1 shows the Metro Mirror and Global Mirror configuration limits. All the limits in this table apply to the aggregate of the whole Remote Copy configuration, regardless of whether an individual relationship, or group, is Metro or Global Mirror.

Table 3-1 Metro Mirror and Global Mirror configuration limits.

3.4 Remote Copy internals

The following describe some aspects of the internal functions of Metro Mirror and/or Global Mirror.

3.4.1 Read stability

As mentioned in Chapter 2, “Planning for Advanced Copy Services implementation” on page 5, read stability can by defined as:

If two reads are submitted for the same area and those reads complete successfully and there was no intervening (in time) or overlapping (in time) write activity then the two reads must return the same data.

A number of applications that perform recoveries following power failure or crashes depend on this property for correct behavior.

Many implementations of mirroring systems fail to maintain read stability. Following a power loss or reset scenario this non-volatile record is used to ensure that both mirrors are made consistent by copying data from one disk to the other.

Reads are only allowed when stability is assured.

Parameter Value

Number of consistency groups 256 per SVC cluster

Number of relationships 1024 per SVC cluster

Total VDisk size per I/O group on both clusters

1024 TB is the per I/O group limit on the quantity of primary and secondary VDisk address spaces that can participate in relationships, the default is 40TB



This definition of read stability makes no guarantee of data returned to reads while writes are outstanding, only where reads are issued after writes have completed. It is possible to conceive of an application that issues writes and without waiting for write completion, issues reads to the same region. Such an application, on seeing new data being returned for its read, might conclude that the Write data is indeed committed. Such an application is not satisfied by this definition of Read Stability. Remote Copy only guarantees Read Stability following a Write Completion.

3.4.2 Write ordering

Many applications that use block storage have a requirement to survive failures such as loss of power, or a software crash, and not lose data that existed prior to the failure. Since many applications need to perform large numbers of update operations in parallel to that storage, maintaining write ordering is key to ensuring the correct operation of applications following a disruption.

If an application is performing a large set of updates, it needs to be designed with the concept of allowing dependent writes. These are writes where it is important to ensure that an earlier write has completed before a later write is started. Reversing the order of dependent writes can undermine the applications algorithms and can lead to problems such as detected, or undetected, data corruption.

3.4.3 Write consistency

Remote Copy maintains write consistency where it ensures that, while primary VDisk and secondary VDisk are synchronized, the VDisks stays in sync even in the case of failure in the primary cluster or other failures which cause the results of writes to be uncertain. After such a failure it is important to ensure that any data reads at the primary VDisk will also be read at the secondary VDisk, since subsequent writes will be dependent on the results of the read data for correctness.

It has been argued that Remote Copy does not require this support since during normal operation as only the primary disk is “active”, and the secondary is in a read only state. Only if an unusual event, forces us to use the secondary VDisk and the content of that disk is examined.

SVC Remote Copy Services uses mirrored non-volatile record marks to protect against this. This will lead to a very small increase in processor work, since extra protocol is involved, but should have no impact on I/O latency (unless I/O throughput is constrained by SVC), or bandwidth.

3.4.4 Write completion

To ensure read stability (see 3.4.1, “Read stability” on page 38) Remote Copy must not allow reads to complete successfully to areas which are out of sync except where there is currently an outstanding write to that region.

During normal operation, it is only regions that have outstanding writes that are out of sync and so reads do not need to be delayed and do not need to be tracked. When an I/O receives an error, then processing of new read I/Os is suspended, and this is achieved before the failed

Note: Read stability is also known as Mirror Write Consistency.



write is completed. This situation is resolved before any read is released, either by forcing the relationship ConsistentStopped or by failing new and pending I/O.

3.4.5 Freeze/Run on consistency groups

The algorithm that consistency is based on is called Freeze/Run. This algorithm freezes I/O in the consistency group before allowing the next I/O to commit, and when restarted it will only allow I/O’s to the primary VDisk.

A Freeze/Run cycle is required when the following has occurred:

� A ‘Stop’ command received from the User� An I/O command to either Primary or Secondary fails such that a host write is not known to

be copied to both disks and the Primary disk has not become “path offline”. (In such circumstances a Freeze is performed on the primary of the affected relationships)

� New host write I/O is suspended without issuing writes to either Primary or Secondary� Host writes where the data has not been written, with successful completion, to both

Primary and Secondary are Stalled. That is, completion to the host is delayed � All ongoing write activity is allowed to complete until it has either become Stalled or

completed to the host only if both Primary and Secondary writes have completed� New Read I/O is suspended

FreezeTimeWhen the freeze has been achieved across all relationships in a consistency group and any system transients (such as path offline status associated with connectivity to the controllers) have completed, then a decision is made as to how to proceed. An attempt might be made to perform a recovery copy which, if successful, allows I/O to resume without loss of synchronization. Otherwise, assuming the primary is still online , then a Run is performed with the relationship consistentstopped. As a result of a run I/O is resumed which means:

� Host I/Os are allowed to complete� New I/Os are allowed to proceed issuing writes only to the primary� Read I/Os are allowed to proceed

When a consistency group is in the state of Consistent Stopped, there is a parameter called freeze_time, see Example 3-1.

It shows the time in YY/MM/DD/HH/MM format, and the time it shows is the last known time when the state between primary and secondary was consistent synchronized.

Example 3-1 The freeze time parameter.

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp 254id 254name ITSO_SERVER_GRPmaster_cluster_id 000002006B603A38master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primary masterstate consistent_stoppedrelationship_count 2freeze_time 2007/11/08/20/14/54status onlinesync in_synccopy_type metroRC_rel_id 2



RC_rel_name ITSO_SERVER_01RC_rel_id 9RC_rel_name ITSO_SERVER_02

The most useful value of the FreezeTime is on the secondary VDisk since the primary sets its value when it first finds out that the relationship is consistent

3.4.6 Management of difference map

The difference map performs change recording. This records which areas of the disk were not copied from primary to secondary or were changed since they were copied, as well as recording which areas have writes in flight when synchronized or copying.

Space managementA difference map is maintained on all online nodes that are members of the I/O group of a primary VDisk. Space for a different map is permanently reserved for the secondary VDisk for use when it needs to act as a primary

The difference map is implemented as a non-volatile bitmap, one bit corresponding to a ‘grain’ of data, each grain corresponding to 256kB of aligned disk space. This is the unit of background copy (as well as recovery copy).

The difference map space is allocated in units corresponding to 8GB of disk space. (4KB * 8 bits * 256kB). Whole units are allocated to each VDisk, with any excess being lost. The limit is expressed in terms of VDisk space in each I/O group.

Within each I/O group it includes both primary and secondary VDisks

Difference map stateThe difference map may be in one of three global states, recorded in cluster state. The grain state at a specific moment can be clean, out of sync, or dirty.

Clean means that there is no I/O in flight and the primary and secondary are synchronized.

Out of sync means that the primary and secondary are synchronized, but I/O is in flight and must complete before being marked clean. It could also mean that after a disruption or failure, RecoveryCopy is required to complete successfully before declaring it clean.

If a grain is dirty, the primary and secondary are not synchronized.

Each grain may be in a state no worse than the difference map (that is to say, dirty difference maps can have grains that are clean or dirty, but a clean difference map has all grains clean, and hence no I/O in flight). Both OutOfSync and Dirty are recorded the same in the non-volatile bitmap. They are distinguished by the global state.

MessagingWhen two nodes are present in an I/O group they will exchange messages to ensure that each is kept up-to-date with respect to changes required in the difference map:

Note: When creating a copy relationship with VDisks that are less than 8GB we are still reserving bitmap space for 8GB, so when making a copy relationship it is a good practice to scale the size of the VDisks in increments of 8GB so as not to waste any bitmap space.



� When a write takes place to a grain that has been synchronized the map must be marked out of sync on both nodes before the I/O is allowed to proceed. The Unlock message is not sent until both primary and secondary I/Os have completed successfully. If either write fails an UnlockDoneFailed message is sent.

� When a background copy or recovery copy operation has successfully synchronized a grain, the grain must be marked clean on both nodes. The node that owns the grain sends a message to the partner node to mark the grain Clean. This is replied to by using a CleanDone message.

� When an I/O touches a grain that is synchronized but it is not going to perform a write to the secondary (e.g. it is in ConsistentStopped state) then the grain must be marked Dirty on both nodes. A node sends a DirtyGrain message and receives a DirtyDone reply from the partner.

When multiple write I/Os are active in the same synchronized grain each requires its own Lock/LockDone/Unlock exchange with the partner node. Only when the last I/O completes within a grain and no I/O that was active in that grain suffered a write failure is the grain marked Clean in each node.

Effect of node failureWhen nodes appear and disappear the difference maps must be resynchronized so that both nodes start (or restart) I/O operation with identical maps. When a node disappears for a long period of time and is survived by its partner node, the partner node must delete all the failed node’s maps, because they are invalid. This invalidation must complete before allowing any I/O to progress on the assumption that it has an assured difference map mark in place.

An I/O only proceeds when all up-to-date difference maps have been marked as required. That is just the local difference map if only one node is operational, but both difference maps if two nodes are operational. For instance, an I/O that dirties a grain must mark both maps dirty before allowing any write I/O to take place.

It is possible that through a sequence of node appearance and disappearance, an up-to-date copy of the difference map is not accessible though there is a node in the I/O group which has an online path to the VDisk. In this case, Remote Copy will hold the path offline and reject host I/O. Either the missing node must be recovered or the Remote Copy relationship must be deleted.

3.4.7 Non-volatile journal

The non-volatile journal data structure is used to maintain details of all writes that are active on consistent synchronized relationships. These details include the VDisk, LBA and count of sectors, and for Global Mirror the sequence number that was assigned. These details are used by the Quick Recovery Copy and Reconstruct processes, see 3.4.8, “Quick Recovery Copy or Reconstruct” on page 43.

The contents of the non-volatile journal are maintained on the nodes in the I/O group of the primary VDisk for the relationship, and are updated at the same time as the bitmap bit in the difference map. The same messages that maintain the locks are used to drive the maintenance of the non-volatile journal. The non-volatile journal has the same availability as any of the difference maps associated with the virtual disks in the same I/O group. In the event that the nodes of the I/O group are offline, then all relationships using the non-volatile journal will change from consistent_synchronized to consistent_stopped, at which point the non-volatile journal is deallocated and will be re-established when the nodes reappear. No administrative intervention is required.



3.4.8 Quick Recovery Copy or Reconstruct

Reconstruct and Quick Recovery Copy are used to attempt to maintain synchronization where some disruption has taken place and it is not certain whether I/Os have completed successfully to both primary and secondary.

They operate during the freeze state described previously. If successful, then I/O can be resumed without having to change to ConsistentStopped state. These processes are essential to making the ConsistentSynchronized state fault tolerant.

� Quick Recovery Copy is used for Metro Mirror.� Reconstruct is used for Global Mirror.

Each uses a record of outstanding I/O, the record of I/Os that must be copied to perform QuickRecoveryCopy or Reconstruct are maintained in the non-volatile journal. Each entry details the range of sectors that were subject to a write I/O that was in flight at the point that the disruption happened. This record is used to drive a process to copy data from the primary to secondary disks. This copy is similar to the process that is used for background copy.

While either Reconstruct or Quick Recovery Copy is in progress, I/O is suspended for the Relationship. Essentially Recovery Copy forms part of the Freeze described above. Recovery Copy completes when either all areas in doubt are copied or either Primary or Secondary are Offline or a persistent error occurs during a read.

Quick Recovery Copy or Reconstruct is performed by the owner node if two nodes within the I/O Group have online paths to the VDisk, or the one node with an online path within the IO Group if there is only one.

Recovery Copy uses the non-volatile difference map to record which grains were subject to an outstanding write I/O. Each grain is copied completely from primary to secondary. This process will only be performed by the current software if a recovery is required in the short period after a software upgrade completes, but before the conversion to use the non-volatile journal is complete.


Draft Document for Review February 10, 2008 5:17 pm 7574 Implementing Metro Mirror and Global Mirror.fm

Chapter 4. Implementing Remote Copy

In this chapter we will show you how to implement Remote Copy using the GUI and the command line interface. We will demonstrate basic usage and some of the advanced functionality of Remote Copy services.

4


7574 Implementing Metro Mirror and Global Mirror.fm Draft Document for Review February 10, 2008 5:17 pm

4.1 Implementation

In this chapter we will show how we implemented Remote Copy in our environment. We will use both the GUI and the command line interface to assist us in creating Metro and Global Mirror relationships and consistency groups. Our environment consists of the equipment shown in Table 4-1

Table 4-1 SVC and Subsystem

We will be using Metro Mirror intercluster relationship between the clusters as shown in Figure 4-1.

Figure 4-1 Our demo setup

Our SVC clusters are ITSOCL2 and ITSOCL3, and we are going to create a copy relationship between those two clusters. Since Metro Mirror and Global mirror are using the same set of commands we will show you in detail how we implement Metro Mirror and we will add additional comments when Global Mirror differs from Metro Mirror.

To be able to start the copy relationship there should been established connection between the two clusters through the SAN network and in this chapter we assume that this has been done. For information on how to install the SVC refer to the IBM System Storage SAN Volume Controller, SG24-6423.

Production site SVC Subsystem

Devices ITSOCL2 DS4500

Disaster site SVC Subsystem

Devices ITSOCL3 DS4700



4.2 Implementation using the GUI

When using the GUI to establish a copy relationship between two VDisks, we will have to logon to both SVC clusters (as stated, we will be using intercluster copy relationship). The only copy relationship command that has to be submitted from both clusters, is the create partnership command. After submitting the create partnership command we can perform all our copy functions from one cluster since this is an active/active partnership and both clusters can control the copy process.

4.2.1 Creation of cluster partnership

We start by logging on to the cluster called ITSOCL2, and press the Create button to create the new partnership, as in Figure 4-2.

Figure 4-2 Creating the partnership between the clusters.

After choosing to create our new partnership we will see our cluster candidates as in Figure 4-3 on page 48.

Chapter 4. Implementing Remote Copy 47


Figure 4-3 Selecting ITSOCL3 as our cluster partner

The candidates are the clusters that are already configured on the SAN and are zoned to our production cluster ITSOCL2.

When we are creating the partnership we need to configure the link between the clusters. If we are creating a Metro Mirror Partnership we only need to think of the bandwidth for the link. If we are creating a Global Mirror partnership we need to be aware of the link tolerance between the two SVC clusters. These parameters are described as follows:

� Bandwidth

– Specifies the bandwidth that is used by the background copy process between the clusters. It adjusts the bandwidth that is used by Metro Mirror or Global Mirror for the initial background copy process. The bandwidth is set to default to 50 MBps (megabytes per second) if you do not specify it. Set the bandwidth to a value that is less than or equal to the bandwidth that can be sustained by the intercluster link.

� Link Tolerance ( Use for Global mirror only)

– Defines how sensitive the cluster is to inter-link overload tolerance. The value specified are the seconds that the cluster will tolerate continued difficulties before the cluster will suspend the copy function to prevent host I/O impact. The parameter accepts values from 60 to 86400 seconds in steps of 10 seconds. The default is 300 seconds (5 minutes). You can disable the link tolerance by entering a value of zero (0) for this parameter.

� Inter cluster delay ( Use for Global mirror only)

– An optional feature for Global Mirror permits a delay simulation to be applied on writes sent to secondary virtual disks. This allows testing to be performed and can be used to



test an application before full deployment of the feature. A value of 0 disables this feature.

� Intra cluster delay ( Use for Global mirror only)

– An optional feature for Global Mirror permits a delay simulation to be applied on writes sent to secondary virtual disks. This allows testing to be performed and can be used to test an application before full deployment of the feature. A value of 0 disables this feature.

We will use the default values in our creation of the partnership and in Figure 4-4 on page 49, we have established a partial partnership between ITSOCL2 and ITSOCL3.

Figure 4-4 Partially configured partnership

We logon to ITSOCL3 to complete the creation of the partnership. See Figure 4-5 on page 50.

Note: When creating a partnership be aware of setting the copy rate settings to a value appropriate to the link and secondary back-end storage.



Figure 4-5 Completing the creation of the partnership on the remote cluster.

After selecting OK, we can see that the partnership is fully configured as in Figure 4-6.

Figure 4-6 Fully configured cluster partnership.

We close that window and now we have a fully configured partnership between the clusters and all copy commands from now on only need to be entered from either of those clusters, not both. We will continue using ITSOCL2 as our production SVC cluster.

4.2.2 Changing a cluster partnership

When changing a cluster partnership we select our partnership from the main menu as shown in Figure 4-7 on page 51.



Figure 4-7 Modifying a cluster partnership

If any changes are made on the partnership link it will not stop any active mirroring. In Figure 4-8 we can change the same settings that we inserted when we created the partnership, as in Figure 1-2 on page 5.

Figure 4-8 Parameter that can be changed for the cluster partnership.

4.2.3 Deleting a cluster partnership

To delete a partnership we will have to run the delete command on both clusters, similar to what we did when we created the partnership. See Figure 4-9 on page 52.



Figure 4-9 Deleting a cluster partnership

Figure 4-10 shows the partnership delete.

Figure 4-10 Confirming deletion of an partnership

After we have clicked the delete button the cluster partnership will be partially configured as shown in Figure 4-11. We can only see that state on the remote cluster and on that cluster we complete the deletion of the partnership.

Figure 4-11 After deleting the local partner the deletion is partially done

As we did when we created the partnership we will have to logon to the remote cluster to finish the deletion. And as shown in Figure 4-11 we have a partially configured link that we will permanently delete by clicking delete.

4.2.4 Creating a copy relationship

We have a VDisk on our production SVC cluster and we are going to mirror its contents to the remote SVC cluster. After creating the VDisks on both clusters we will create the relationship



between those VDisks as shown in Figure 4-12, then we will import that relationship into the consistency group.

Figure 4-12 Creating copy relationship.

After we have selected the creation of the relationship and pressed Go, we will get an overview of the steps ahead, see Figure 4-13 on page 53.

Figure 4-13 Overview of selections for creating the relationship

We name the relationship with a name that gives a meaning to what it is used for as shown in Figure 4-14.



Figure 4-14 Name the relationship

We choose the copy type to be Metro Mirror, since we want the copy to be synchronous. We can also select the relationship to be inter or intra cluster. We select intercluster since we are mirroring from production to disaster/recovery clusters that are separated by distance.

Figure 4-15 Search for Master VDisk candidates

If we know the name of the VDisk we are going to user as our Master VDisk, we can filter for that name, or to see all VDisk candidates on the Master cluster we can use an asterisk to see all the available VDisk candidates as in Figure 4-15 on page 54.

After sorting our available candidates, we choose ITSO_MM_S001 as our Master Vdisk as in Figure 4-16 on page 55.



Figure 4-16 Selecting our Master VDisk

Now we need to choose our Auxiliary VDisk from the other SVC cluster. The SVC clusters are already in a partnership and therefore it provides the cluster a list of candidates for the auxiliary VDisk, without requiring us to login to the remote cluster.

The GUI will only show VDisks that are of the same size as the Master Vdisk and are available for a copy relationship as in Figure 4-17 on page 55.

Figure 4-17 Selecting VDisk as our auxiliary VDisk

We are creating a copy relationship with unused VDisks, we know the status of the disks, and that is both VDisks exist and have not been mapped to any host.

From an operating system point of view they are not initialized and empty. Therefore we can chose to select them as synchronized. This is the most common way to perform this action if you are creating a new relationship between unused VDisks. as shown in Figure 4-18 on page 56.



Figure 4-18 Select synchronized so the primary and secondary are in sync when the copy process starts.

We will not add this relationship to any consistency group at this time, we will do that later in this chapter. But this feature to be able to add a relationship directly to a consistency group is excellent to use when adding disks to a host that already has VDisks in an consistency group.

Our relationship is called ITSO_SERVER_01 and it is now created as shown in Figure 4-19 on page 56.

Figure 4-19 Verifying the relationship.

4.2.5 Changing copy relationship

If we need to make any modifications to our relationship we can do that by selecting the relationship and from the drop down menu select modify relationship as shown in Figure 4-20 on page 57



Figure 4-20 Modifying a relationship.

When we are modifying a relationship we can change the name of the relationship, add/remove from a consistency group as shown in Figure 4.2.6.

Figure 4-21 Modifying a relationship.

4.2.6 Starting a copy relationship

After the successful creation of the relationship, the relationship is in a consistent stopped state as shown in Figure 4-22.

Figure 4-22 The relationship is created and ready to be started.

In Figure 4-22, notice that even though we have created the relationship, and that although we have not started any copy process the relationship is in a consistent stopped state. It is consistent because we selected the synchronized parameter when creating the relationship.



The final step towards a fully configured relationship is to start the copy process between the VDisks.

We select to start the copy process from the drop down menu as shown in Figure 4-23.

Figure 4-23 Start the copy process

In Figure 4-24 on page 58 we do not have to mark forced start or mark as clean since we are starting a new relationship and we have already selected it to be synchronous from the start.

Figure 4-24 Starting the Copy process

And now we see the final state of our relationship in Figure 4-25. Our relationship is in a consistent synchronized state so both primary and secondary VDisks should have the same content.

Figure 4-25 The final state of our relationship creation



Now we have created a relationship between two VDisks on two SVC clusters that are in an intercluster partnership. Our Primary VDisk is the Master VDisk and is therefore our production disk.

4.2.7 Stopping a copy relationship.

We can only stop a copy relationship when it is a stand alone relationship. If the relationship is in a consistency group we must stop the consistency group and therefore stop all relationships member in that group.

As shown in Figure 4-27 we select our relationship and from the drop down menu we select stop copy process.

Figure 4-26 Stopping copy relationship.

In Figure 4-27 we can select that we want to enable write access to the secondary VDisk. Since we are only stopping the copy process and do not want to enable write access to the secondary, we select OK without setting a check mark to enable write access.

Figure 4-27 Stopping copy process

In Figure 4-28 on page 60 we can see the result of stopping the copy process. Our copy state is now consistent stopped.



Figure 4-28 Copy state is now consistent stopped

4.2.8 Deleting a copy relationship

When we need to delete a relationship we select the relationship that we want to delete, and from the drop down menu we select Delete a Relationship as shown in Figure 4-29.

Figure 4-29 Deleting relationship

In Figure 4-30 we can choose to remove the relationship from the consistency group before we delete it. This will still delete the relationship but it removes the relationship from the consistency group before deleting it.

Figure 4-30 Removing from consistency group before it is deleted



4.2.9 Creating consistency groups

When we are creating our relationship we have the opportunity to add the relationship to a consistency group at the same time. If we have not already created the consistency group we need to create one and import the relationship to that consistency group. Many administrators will start with making the consistency group, and then add the relationship to it during creation.

We will also show you how you create a consistency group and how to perform different tasks with it, such as adding/removing the relationship, and stopping/starting the copy process.

In Figure 4-31 we select Manage Copy Services from the main menu, and from the main menu we select Metro & Global Mirror Consistency Groups.

Figure 4-31 Select Create a Consistency Group

After we have selected to create a new consistency group we click Go..

In Figure 4-32 we name our consistency group and select if we want to have it as an intracluster or intercluster consistency group.

Figure 4-32 Naming and configuring Consistency Group



We have given our consistency group the name ITSO_SERVER_GRP and it will be an intercluster relationship, since we know the relationship is between two SVC clusters.

When we have configured the consistency group we also have the opportunity to add already configured relationships to it.

The relationship and the consistency group need to have the same type of configuration and that is intercluster or intracluster.

In Figure 4-33 on page 62 we already have a relationship that is not in a consistency group and we select that we want to add it as a member in our ITSO_SERVER_GRP consistency group.

Figure 4-33 Selecting relationship to the consistency group

If we do not select any relationship for our consistency group it will be defined as empty, we can then add relationship(s) to it at our convenience.

In Figure 4-34 we see a summary of our consistency group.

Figure 4-34 Summary over the consistency group

After clicking Finish our consistency group is created, but as we can see in Figure 4-35 on page 63, the state of the consistency group is consistent stopped.

This is because the relationship we added to the consistency group was in a consistent stopped state. Consistency groups enter the state of the first relationship that is added to the



group, and after that you cannot add a relationship of a different state than the consistency group already has.

Figure 4-35 Consistency group created in consistent stopped state

4.2.10 Starting a consistency group

As the nature of consistency groups is to maintain consistency in all member relationships, we are therefore affecting all relationship in the given group we are working on.

Figure 4-36 shows where we can select to start the copy process from the drop down menu.

Figure 4-36 Starting a copy process

We start by selecting the consistency group that we want to start the copy process on and the options as in Figure 4-37 on page 64.



Figure 4-37 Select options on how we will start the copy process

As before, this copy relationship and the VDisks in that relationship are in sync.

Figure 4-38 shows successful start of copy process on the ITSO_SERVER_GRP consistency group.

Figure 4-38 Consistency group is now successfully started

4.2.11 Stopping a consistency group

When we stop a copy process, we are not effecting the primary VDisk in any way. There is a possibility to enable write access to the secondary VDisk when we select the stop copy process (we discuss this in 4.4.2, “Site failover using the GUI” on page 84).

In this scenario we are only stopping the copy process on all relationships in the consistency group leaving the secondary in read only mode.

In Figure 4-39 on page 65 from the Metro and Global Mirror consistency groups main menu we select the consistency group that we would like to stop.



Figure 4-39 Selecting consistency group to stop copy process on

We select stop the copy process, see Figure 4-40.

Figure 4-40 Stopping the copy process

In this case we do not enable write access to the secondary, but we select Go to stop the copy process.

Figure 4-41 Our consistency group is now consistent stopped



4.2.12 Renaming a consistency group

When we need to rename a consistency group it will not effect in any way the operation of the relationships that are within that group.

In Figure 4-42 we start by selecting the group we would like to rename and then from the drop down menu we select rename a consistency group.

Figure 4-42 Renaming a consistency group

After we have clicked on Go we enter the new name into the new name field, as shown in Figure 4-43.

Figure 4-43 New name selected for the consistency group

4.2.13 Reversing copy directions

If we have the consistency group in a consistent synchronized state we can switch copy directions. This is a full reverse of copy direction, that is primary will become secondary and vice versa.

In Figure 4-44 we choose the Metro and Global mirror consistency groups, to see what consistency groups are available and in what state they are.



Figure 4-44 Metro & global mirror consistency groups

We verify that the status is consistent synchronized and we can also see that the Primary is the Master VDisk, so the copy direction is from master to auxiliary.

We select the consistency group that we want to change the copy direction on and highlight Switch Copy Direction and click on Go as shown in Figure 4-45 on page 67.

Figure 4-45 Selecting switch copy directions

The next screen (Figure 4-46) shows us that this is the Primary VDisk and asks us if we would like to change that copy direction. If we choose the same current copy direction this command has no effect on the copy state. We select to change the Primary VDisk from Master to auxiliary and therefore select the Primary VDisk to be the auxiliary under new copy direction as shown in Figure 4-46.



Figure 4-46 Making Auxiliary the Primary VDisk

We have now changed our Remote copy direction. Primary is now auxiliary as in Figure 4-47 on page 68. This command does not check if the secondary VDisk is mapped to a host or the host is actually sending I/O’s to the primary VDisk when we make the change. So it is recommended to have the system administrator verify that all I/O is stopped when we issue the switch copy direction command.

Figure 4-47 Primary VDisk is now auxiliary.

4.3 Implementation using the CLI

When we are implementing Remote Copy using the CLI we do not have to use all the parameters of the command. Many of the parameters are optional and only if it is in our interest will we need to use them.

In Chapter 8, “Automation” on page 207 in this book we discuss some of the commands to use when creating a relationship and importing to a consistency group.

More detailed descriptions on each command can be found in IBM Total Storage SAN Volume Controller Command Line interface User’s Guide SC26-7903-02.

As we did using the GUI our demo environment consists of two SVC clusters that are separated by distance. We have already done the installation of both SVC’s, following instructions in IBM System Storage SAN Volume Controller, SG24-6423.

We start by checking for available candidates, and we are using SVC clusters ITSOCL2 and ITSOCL3. ITSOCL2 will be our production cluster and ITSOCL3 is our disaster/recovery.



4.3.1 Listing cluster candidates

We start by checking for if we have a cluster candidate for our partnership as shown in Example 4-1.

Example 4-1 List the cluster candidate.

IBM_2145:ITSOCL2:admin>svcinfo lsclustercandidateid configured cluster_name0000020068203A42 no ITSOCL3

We can see that this SVC cluster is not configured so our SVC cluster candidate is not already configured as our partner. Only one cluster partnership is available for each cluster.

4.3.2 Create cluster partnership

In Example 4-2 we make the partnership.

Example 4-2 The mkpartnership command.

>>- svctask -- -- mkpartnership -- ----------------------------->

>--+-----------------------------------+-- ---------------------> '- -bandwidth -- bandwidth_in_mbps -'

>--+- remote_cluster_id ---+----------------------------------->< '- remote_cluster_name -'

This creates a one-way partnership between our production cluster and the disaster/recovery cluster. To complete the two-way partnership, the equivalent mkpartnership command must be issued on the disaster/recovery cluster.

We create the partnership using the default bandwidth , that is 50 MBps, and our remote cluster is called ITSOCL3. Example 4-3 shows the command used on the production SVC cluster.

Example 4-3 Creation of partnership with ITSOCL3.

IBM_2145:ITSOCL2:admin>svctask mkpartnership ITSOCL3

Now we have partially created the partnership. We can see that this is partially created by issuing the command lscluster -delim : ITSOCL3 . See Example 4-4.

Example 4-4 Partially configured partnership.

IBM_2145:ITSOCL2:admin>svcinfo lscluster -delim : ITSOCL3id:0000020068403A42name:ITSOCL3location:remotepartnership:partially_configured_localbandwidth:50

We logon to ITSOCL3 ( the remote cluster) and complete the creation. We issue the same command as before svctask mkpartnership ITSOCL2 and then the svcinfo lscluster -delim : ITSOCL2 to see the status of the partnership as in Example 4-5.



Example 4-5 Creation of partnership with ITSOCL2

IBM_2145:ITSOCL3:admin>svctask mkpartnership ITSOCL2IBM_2145:ITSOCL3:admin>svcinfo lscluster -delim : ITSOCL2id:000002006B603A38name:ITSOCL2location:remotepartnership:fully_configuredbandwidth:50

Now we have created our fully configured partnership between ITSOCL2 and ITSOCL3 with a bandwidth of 50 MBps.

4.3.3 Change cluster partnership

In Example 4-6 we change the cluster partnership.

Example 4-6 The chpartnership command.

>>- svctask -- -- chpartnership -- ----------------------------->

>-- -bandwidth -- bandwidth_in_mbps -- ------------------------->


We can change the parameters of the cluster link without stopping the active copy process. We might want to increase the bandwidth that background copy is allowed to use from the default value of 50 Mbps to 75 MBps. This is shown in Example 4-7.

Example 4-7 Changing bandwidth between clusters from 50 to 75 MB/s.

IBM_2145:ITSOCL2:admin>svctask chpartnership -bandwidth 75 ITSOCL3

IBM_2145:ITSOCL2:admin>svcinfo lscluster ITSOCL3id 0000020068403A42name ITSOCL3location remotepartnership fully_configuredbandwidth 75

4.3.4 Deleting a cluster partnership

In Example 4-8 we show how to delete the cluster partnership.

Example 4-8 The rmpartnership command.

>>- svctask -- -- rmpartnership -- ----------------------------->


When deleting a copy partnership we need to delete the partnership from both the local and the remote clusters. From the cluster we are logged on we issue the rmparntership



command, we use the remote clusters name in the command as shown in Example 4-9 on page 71.

After issuing the rmpartnership command we logon to the remote cluster to complete the deletion of the partnership

Example 4-9 Removing partnership.

IBM_2145:ITSOCL2:admin>svctask rmpartnership ITSOCL3

IBM_2145:ITSOCL3:admin>svctask rmpartnership ITSOCL2

4.3.5 Creating a copy relationship

After we have created the intercluster partnership between our clusters, we need to start a copy relationship between two VDisks. We have already created VDisks on both clusters of the same size, which is required for a copy relationship to be created. The disks we have created are as follows:

� On our production cluster ITSOCL2

– ITSO_MM_S01

– ITSO_MM_S02

� On our Disaster/Recovery cluster ITSOCL3

– ITSO_MM_T01

– ITSO_MM_T02

In Example 4-10 we create the relationship.

Example 4-10 The mkrcrelationship command.

>>- svctask -- -- mkrcrelationship -- -------------------------->

>-- -master --+- master_vdisk_id ---+-- ------------------------> '- master_vdisk_name -'

>-- -aux --+- aux_vdisk_id ---+-- ------------------------------> '- aux_vdisk_name -'

>-- -cluster --+- cluster_id ---+-- ----------------------------> '- cluster_name -'

>--+------------------------+-- --------------------------------> '- -name -- new_name_id -'

>--+-----------------------------------------+-- ---------------> '- -consistgrp --+- consist_group_id ---+-' '- consist_group_name -'

>--+---------+-- --+-----------+------------------------------->< '- -sync -' '- -global -'



When using the mkrelationship command we need to have names and/or id’s of the VDisks and clusters we are using. For more detailed information about certain aspects of the command it can be found in the CLI command guide1. Our command will look like that shown in Example 4-11.

Example 4-11 Creating the relationship.

IBM_2145:ITSOCL3:admin>svctask mkrcrelationship -master ITSO_MM_S01 -aux ITSO_MM_T01 -cluster ITSOCL3 -name ITSO_SERVER_01 -sync

RC Relationship, id [0], successfully created

Now we have created a fully configured relationship between ITSO_MM_S01 and ITSO_MM_T01, but the copy process is not started. The VDisks are in sync and are only waiting for the copy process to be started, see Example 4-12.

Example 4-12 State of the relationship.

IBM_2145:ITSOCL3:admin>svcinfo lsrcrelationship 0id 0name ITSO_SERVER_02master_cluster_id 000002006B603A38master_cluster_name ITSOCL2master_vdisk_id 1master_vdisk_name ITSO_MM_S02aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3aux_vdisk_id 0aux_vdisk_name ITSO_MM_T02primary masterconsistency_group_idconsistency_group_namestate consistent_stoppedbg_copy_priority 50progressfreeze_timestatus onlinesync in_synccopy_type metro

4.3.6 Starting a copy relationship

In Example 4-13 we start the copy relationship.

Example 4-13 The startrcrelationship command.

>>- svctask -- -- startrcrelationship -- ----------------------->

>--+--------------------------+-- --+----------+-- ------------->

Note: If there is a consistency group already created we can add a relationship to a consistency group in the same step, using the -consistgrp parameter.

1 IBM Total Storage SAN Volume Controller Command Line Interface User’s Guide, SC26-7903-02



'- -primary --+- master -+-' '- -force -' '- aux ----'

>--+----------+-- --+- rc_rel_id ---+-------------------------->< '- -clean -' '- rc_rel_name -'

To start the copy process for our relationship, containing the VDisks ITSO_MM_S01 and ITSO_MM_T01, we will use the startrcrelationship command. If we have already put the relationship into a consistency group we would have to start the consistency group with all the relationships in that group by using the startrcconsistgrp command.

We have created the relationship and now we want to start copying, but we need to apply the -force parameter, because this parameter is required if consistency would be lost by starting a copy operation. Since we do not know if an I/O has occurred to the primary VDisk after we created the relationship we need to issue the command with the -force parameter. In general, the -force parameter is required if the relationship is in one of the following states:

� ConsistentStopped but not synchronized� Idling but not synchronized

The -force parameter is not required if the relationship is in one of the following states:

� InconsistentStopped� InconsistentCopying � ConsistentSynchronized

Example 4-14 shows where our relationship ITSO_SERVER_01 is started.

Example 4-14 Starting the copy process within the relationship.

IBM_2145:ITSOCL2:admin>svctask startrcrelationship -force ITSO_SERVER_01

After we have started the relationship it will become consistent synchronized as shown in Example 4-15.

Example 4-15 State of the relation ship is now consistent synchronized.

IBM_2145:ITSOCL2:admin>svcinfo lsrcrelationship ITSO_SERVER_01id 2name ITSO_SERVER_01master_cluster_id 000002006B603A38master_cluster_name ITSOCL2master_vdisk_id 2master_vdisk_name ITSO_MM_S001aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3aux_vdisk_id 4aux_vdisk_name ITSO_MM_T001primary masterconsistency_group_idconsistency_group_namestate consistent_synchronizedbg_copy_priority 50progressfreeze_timestatus onlinesync



copy_type metro

4.3.7 Stopping a copy relationship

In Example 4-16 we show how to stop a copy relationship.

Example 4-16 The stoprcrelationship command.

>>- svctask -- -- stoprcrelationship -- --+-----------+-- ------> '- -access -'>--+- rc_rel_id ---+------------------------------------------->< '- rc_rel_name -'

The stoprcrelationship command applies only to a stand-alone relationship. If the relationship is part of a consistency group, the group needs to be stopped and therefore all relationships within that group too. We can issue this command to stop a relationship that is copying from primary to secondary VDisks.

If the relationship is in a consistent state, that is; consistent stopped, consistent synchronized, or consistent disconnected state, we can use the -access parameter to enable write access to the secondary VDisk.

4.3.8 Change a copy relationship

In Example 4-17 on page 74 we show how to change a copy relationship.

Example 4-17 The chrcrelationship command

>>- svctask -- -- chrcrelationship -- -------------------------->

>--+- -name -- new_name_arg -----------------+------------------> +- -force --------------------------------+ '- -consistgrp --+- consist_group_id ---+-' '- consist_group_name -'

>--+- rc_rel_id ---+------------------------------------------->< '- rc_rel_name -'

When we need to change relationship we use the command svctask chrcrelationship

We can change name and add/remove relationships from the consistency group using this command. We can remove a relationship from a consistency group by specifying the -force parameter and the name or ID of the relationship.

We start by adding a relationship to an existing consistency group with the name of ITSO_SERVER_GRP as in Example 4-18.

Example 4-18 adding a stand alone relationship to consistency group.

IBM_2145:ITSOCL2:admin>svctask chrcrelationship -consistgrp ITSO_SERVER_GRP ITSO_SERVER_01

Then we remove the relationship from the consistency group again as in Example 4-19.



Example 4-19 Removing a relationship from a consistency group.

IBM_2145:ITSOCL2:admin>svctask chrcrelationship -force ITSO_SERVER_01

This form of the modify relationship command succeeds in the connected or disconnected states. If the clusters are disconnected the relationship is only removed from the consistency group on the local cluster at the time the command is issued. When the clusters are reconnected the relationship is automatically removed from the consistency group on the other cluster. Alternatively, you can issue an explicit modify (chrcrelationship) command to remove the relationship from the group on the other cluster while it is still disconnected.

4.3.9 Delete copy relationship

In Example 4-20 we show how to delete the copy relationship.

Example 4-20 The rmrcrelationship command.

>>- svctask -- -- rmrcrelationship -- --+- rc_rel_id ---+------>< '- rc_rel_name -'

When we delete a relationship we are only deleting the relationship between those two VDisks, it does not affect the VDisks themselves. It is not possible to delete a relationship if it is a member of a consistency group. We would have to remove it from the consistency group first and then delete it as shown in Example 4-21.

Example 4-21 Deleting a copy relationship

IBM_2145:ITSOCL2:admin>svctask chrcrelationship -force ITSO_SERVER_01

IBM_2145:ITSOCL2:admin>svctask rmrcrelationship ITSO_SERVER_01

4.3.10 Creating a consistency group

In Example 4-22 we show how to create a consistency group.

Example 4-22 The mkrcconsistgrp command

>>- svctask -- -- mkrcconsistgrp -- --+---------------------+---> '- -name -- new_name -'

>-- --+--------------------------------+----------------------->< '- -cluster --+- cluster_id ---+-' '- cluster_name -'

We will create a consistency group between ITSOCL2 and ITSOCL3 and implement the copy relationship (that we have already created) to that consistency group. We created the consistency group as shown in Example 4-23.

Note: If we make a Metro Mirror or Global Mirror consistency group not using the -cluster (name or ID of the remote cluster) the consistency group will only be created on the local cluster.



Example 4-23 Creating the consistency group

IBM_2145:ITSOCL2:admin>svctask mkrcconsistgrp -cluster ITSOCL3 -name ITSO_SERVER_GRPRC Consistency Group, id [255], successfully created

This consistency group is now created but it has no relationship in it, and we can see that by looking closer at the relationship in Example 4-24.

Example 4-24 Showing the content of the consistency group

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRPid 255name ITSO_SERVER_GRPmaster_cluster_id 000002006B603A38master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primarystate emptyrelationship_count 0freeze_timestatussynccopy_type empty_group

To move a relationship between two consistency groups, you must issue the chrcrelationship command twice. Use the -force parameter to remove the relationship from its current group, and then use the -consistgrp parameter with the name of the new consistency group.

Since our relationship was created as a standalone relationship it is not a member of a consistency group, so we only need to use the -consistgrp parameter as in Example 4-25.

Example 4-25 Adding relationship to consistency group.

IBM_2145:ITSOCL2:admin>svctask chrcrelationship -consistgrp ITSO_SERVER_GRP ITSO_SERVER_01

The content of our consistency group has changed as shown in Example 4-26.

Example 4-26 The consistency group now has a relationships member.

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRPid 255name ITSO_SERVER_GRPmaster_cluster_id 000002006B603A38master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primary masterstate consistent_stoppedrelationship_count 1freeze_timestatus onlinesync in_synccopy_type metro



RC_rel_id 0RC_rel_name ITSO_SERVER_01

4.3.11 Start consistency group

In Example 4-27 we show how to start the consistency group.

Example 4-27 The startrcconsistgrp command.

>>- svctask -- -- startrcconsistgrp -- ------------------------->

>--+--------------------------+-- --+----------+-- -------------> '- -primary --+- master -+-' '- -force -' '- aux ----'>--+----------+-- --+- rc_consist_group_id ---+---------------->< '- -clean -' '- rc_consist_group_name -'

When we start or stop a consistency group we are effecting all relationships in that group, hence the term “consistency”. It is possible to make changes to the consistency group within the start and stop commands, and this will be explained later.

By issuing the startrcconsisgrp command we are given the following possibilities as parameters:

� -primary

– Which VDisk will become the primary VDisk in this relationship? Is it the master or the auxiliary that we want to be the active disk.

� -force

– Should we force the copy process to start? Do we know the state of the secondary VDisk, and can we start the copy process even if we know that we will loose consistency while the secondary is catching up with primary?

� -clean

– Is it a clean start? Clean in this sense means that any changes that have been made at the secondary are ignored, and only changes made at the primary are considered when synchronizing the primary and secondary VDisks.

In our example we are starting the copy process by issuing the startrcconsistgrp command and stating that the primary will be the master VDisk. Example 4-28.

Example 4-28 Starting the copy process in the consistency group with master as primary disk.

IBM_2145:ITSOCL2:admin>svctask startrcconsistgrp -primary master ITSO_SERVER_GRP

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRPid 255name ITSO_SERVER_GRPmaster_cluster_id 000002006B603A38

Note: If we try to add a relationship to a consistency group which has a different copy state, such as the relationship is stopped and the consistency group is synchronized, we will get an error message saying: The relationship cannot be added because the states of the relationship and the consistency group do not match.



master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primary masterstate consistent_synchronizedrelationship_count 1freeze_timestatus onlinesynccopy_type metroRC_rel_id 0RC_rel_name ITSO_SERVER_01

Now we have started the ITSO_SERVER_01 copy relationship in the ITSO_SERVER_GRP consistency group and the VDisks are in the state consistent synchronized.

4.3.12 Stopping the consistency group

In Example 4-29 we show how to stop the relationship.

Example 4-29 The stoprcrelationship command.

>>- svctask -- -- stoprcrelationship -- --+-----------+-- ------> '- -access -'>--+- rc_rel_id ---+------------------------------------------->< '- rc_rel_name -'

By issuing the stoprcconsistgrp command with the -access parameter, we will stop the copy process and enable write access to the secondary, This command will not change the copy directions, but the state of the consistency group will be idling, and that means that both VDisks have read/write access enabled, and no copy process is active.

If we only issue the command stoprcconsistgrp the consistency group will stop mirroring between VDisks, but the secondary will still be in a read-only state.

Example 4-30 is an example of how we used the stoprcconsistgrp with the -access command to stop the copy process and enable write access to the secondary VDisk.

IBM_2145:ITSOCL2:admin>svctask stoprcconsistgrp -access ITSO_SERVER_GRP

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRPid 255name ITSO_SERVER_GRPmaster_cluster_id 000002006B603A38master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primarystate idlingrelationship_count 1freeze_timestatussync in_synccopy_type metroRC_rel_id 0



RC_rel_name ITSO_SERVER_01

To change the copy direction follow the process in 4.3.15, “Reversing copy directions” on page 81.

4.3.13 Changing a consistency group

Example 4-30 shows how we change the consistency group.

Example 4-30 The chrcconsistgrp command

>>- svctask -- -- chrcconsistgrp -- -- -name -- new_name_arg --->

>-- --+- rc_consist_group_name -+------------------------------>< '- rc_consist_group_id ---'

When changing the consistency group, all we are changing is the name. A consistency group is only a name until it has got a relationships member.

In Example 4-31 we are changing the consistency group name from ITSO_SERVER_GRP to ITSO_SILENT.

Example 4-31 Changing the name of the consistency group

IBM_2145:ITSOCL2:admin>svctask chrcconsistgrp -name ITSO_SILENT ITSO_SERVER_GRP

Now our consistency group has the name ITSO_SILENT.

4.3.14 Deleting a consistency group

In Example 4-32 we show how to delete the consistency group with active relationships in it, and how the relationships are not effected by this deletion.

Example 4-32 The rmrcconsistgrp command.

>>- svctask -- -- rmrcconsistgrp -- --+----------+-- -----------> '- -force -'>--+- rc_consist_group_id ---+--------------------------------->< '- rc_consist_group_name -'

If we are deleting a consistency group that is not empty we can use the -force parameter to delete the consistency group. This will change all relationships in that consistency group to a standalone relationship. The state of the relationship is not changed by deleting the consistency group.

In the examples that follow we delete a consistency group with two consistent synchronized relationships in it.

First we check the status of the consistency group called ITSO_SILENT, and in this group we have two relationships called ITSO_SERVER_01 and ITSO_SERVER_02. As shown in Example 4-33 the copy status is consistent synchronized.

Example 4-33 Checking status of the relationships within the consistency group.

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SILENTid 254



name ITSO_SILENTmaster_cluster_id 000002006B603A38master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primary masterstate consistent_synchronizedrelationship_count 2freeze_timestatus onlinesynccopy_type metroRC_rel_id 2RC_rel_name ITSO_SERVER_01RC_rel_id 9RC_rel_name ITSO_SERVER_02

Now we delete the consistency group as in Example 4-34.

Example 4-34 Deleting the consistency group.

IBM_2145:ITSOCL2:admin>svctask rmrcconsistgrp -force ITSO_SILENT

And by checking the status of the relationships we now see that they have not changed state, both are consistent synchronized. Example 4-35 and Example 4-36.

Example 4-35 Status of ITSO_SERVER_01 copy relationship.

IBM_2145:ITSOCL2:admin>svcinfo lsrcrelationship ITSO_SERVER_01id 2name ITSO_SERVER_01master_cluster_id 000002006B603A38master_cluster_name ITSOCL2master_vdisk_id 2master_vdisk_name ITSO_MM_S001aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3aux_vdisk_id 4aux_vdisk_name ITSO_MM_T001primary masterconsistency_group_idconsistency_group_namestate consistent_synchronizedbg_copy_priority 50progressfreeze_timestatus onlinesynccopy_type metro

Example 4-36 Status of ITSO_SERVER_02 copy relationship.

IBM_2145:ITSOCL2:admin>svcinfo lsrcrelationship ITSO_SERVER_02id 9name ITSO_SERVER_02master_cluster_id 000002006B603A38



master_cluster_name ITSOCL2master_vdisk_id 9master_vdisk_name ITSO_GLOBALaux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3aux_vdisk_id 5aux_vdisk_name ITSO_GLOBALprimary masterconsistency_group_idconsistency_group_namestate consistent_synchronizedbg_copy_priority 50progressfreeze_timestatus onlinesynccopy_type metro

4.3.15 Reversing copy directions

When a consistency group is in a consistent synchronized state, we can reverse the copy direction for that group by using the switchrcconsistgrp command by using the -primary parameter.See Example 4-37.

Example 4-37 The state of the consistency group before switching copy directions

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRPid 255name ITSO_SERVER_GRPmaster_cluster_id 000002006B603A38master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primary masterstate consistent_synchronizedrelationship_count 1freeze_timestatus onlinesynccopy_type metroRC_rel_id 0RC_rel_name ITSO_SERVER_01

And now we issue the switch command (Example 4-38).

Example 4-38 Issuing the switch copy directions command

IBM_2145:ITSOCL2:admin>svctask switchrcconsistgrp -primary aux ITSO_SERVER_GRP

And then check the status after issuing the command, see Example 4-39.

Example 4-39 The state of the consistency group after the switch command.

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp ITSO_SERVER_GRPid 255



name ITSO_SERVER_GRPmaster_cluster_id 000002006B603A38master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primary auxstate consistent_synchronizedrelationship_count 1freeze_timestatus onlinesynccopy_type metroRC_rel_id 0RC_rel_name ITSO_SERVER_01

Now we have switched copy directions and the primary disk is now the auxiliary Vdisk.

4.4 Example of site failover

When a disaster occurs it is often the case that we are not very well prepared for it. It is always good practice to have a detailed plan of what needs to be done, how, and by whom. In a storage environment we might need to take decisions on whether and when to declare a disaster or not, and when/if we do so it is important to have a detailed action plan, Often it is the same people that are trying to fix the failure and getting the customer up and running again.

The detailed action plan should be written in such a way that a storage administrator could carry out those tasks without having the person who wrote the plan explaining each line.

When controlling the copy functionality, either with Metro Mirror or Global Mirror, it can be controlled from either cluster in the copy relationship, and that means if one SVC cluster is taken down, then the other SVC cluster can stop the mirroring process and can enable read/write access to the secondary VDisks.

If we need to perform a site failover there are some things that will help us to minimize downtime of servers caused by the disaster.

� Make sure that all VDisks that are in a copy relationship in consistency groups.

� Make sure that you have created a script that will help you perform a site failover. Chapter 8, “Automation” on page 207 shows script examples.

� All servers on the recovery site should be zoned and created, but not mapped to the target SVC cluster VDisks.

� If disaster is declared and you are going to perform a site failover stop all copy processes if not already stopped. This is because the secondary VDisk cannot by set to read/write without stopping the copy process.

Note: Using Metro Mirror or Global Mirror the secondary VDisks are always in a read only mode and should not be mapped to a server until the copy direction is changed or the copy process is stopped and the VDisk is set to read/write mode.



4.4.1 Site failover when production site loses back-end storage

If we take an example where two SVC clusters are doing Metro Mirror copy.

In Figure 4-48 we have server A and server B. Both servers are connected to the SAN and defined on both SVC clusters, but only primary server A has allocated VDisks from SVC cluster 1. SVC cluster 2 is the remote SVC and therefore we create the disks and Metro Mirror relationship, but we do not allocate the VDisks to server B. The target VDisks are left un allocated.

Figure 4-48 Overview of environment

The server loses access to its LUNs and the SVC cannot see its MDisks from that subsystem as shown in Figure 4-49.



Figure 4-49 Subsystems on production site fails

If the disks on Server 1 become unavailable, we can perform a failover to the standby server at the disaster/recovery site and start using the secondary VDisks.

Here are some guidelines which we can follow:

1. Terminate all connections from Server 1 to SVC cluster 1 VDisks

If we can access SVC cluster 1 it is best to terminate all connections between server 1 and the SVC cluster. When the repair is complete we do not have to worry about conflicts with the “old” disks.

2. Stop all copy process that involve this server

a. Log on to the remote SVC cluster and stop all copy services.

i. By doing that we are able to grant write access to the secondary VDisk.

b. If we lose connection to our MDisks the copy services automatically stops but that is because the SVC sees this as a hardware failure.

i. You could also reverse the mirror and by that give read/write access to the target LUNs, but if the source Managed Disk Group (MDG) is offline, or has a hardware failure, you cannot change the copy direction.

3. Map VDisks from SVC cluster 2 to server B

4. Mount VDisks and restart the application.

4.4.2 Site failover using the GUI

This is our scenario: we have a dual site environment, production and disaster/recovery, the production site consists of server, subsystem and SVC. The disaster/recovery site has the identical hardware, and all disks for the production server are Metro Mirrored across to the disaster/recovery site. The production site we call site A, and the disaster/recovery site we

Note: The automation using CLI is mention in chapter 9 in this Redbook.



call site B. In site B we have standby server B and a Metro Mirror partner to our production SVC.

In Figure 4-50 we can see a Metro Mirror consistency group called ITSO_SERVER_GRP, that is running on our production SVC. In this consistency group we have two active relationships and the state is consistent synchronized.

Figure 4-50 A consistency group in healthy state.

In our scenario the subsystem on site A went offline so all our MDisks were unavailable at that site. Our copy process will then automatically stop as shown in Figure 4-51 on page 85.

Figure 4-51 Consistency groups in status consistent stopped.

At this point the server cannot see its disks at the production site and we want to perform a failover to the disaster/recovery site. We will now log on to the SVC cluster located at the disaster/recovery site and make the secondary VDisks available for read/write access.

Even though the status of the relationship is consistent stopped we need to stop the copy process to be able to enable read/write access to the VDisks in the consistency group. The state of the consistency group prevents us from using the Switch Copy Directions command.

In Figure 4-52 we have clicked the Metro & Global Mirror consistency groups from the main menu and there we can see the current consistency groups this cluster partnership has. From the drop down menu we can select to stop the copy process on the selected consistency group.



Figure 4-52 Selecting to stop copy process.

When we have selected to stop the copy process and clicked Go, we will get the chance to change the mode of the secondary VDisks to enable write access, as shown in Figure 4-53 on page 86.

Figure 4-53 Stopping the copy process and enabling write access to secondary.

By setting a checkmark in the enable write access checkbox, and selecting OK, we are enabling write access to all VDisks in this consistency group. All relationships in the consistency group will become idle and therefore the consistency group itself.

Figure 4-54 shows the status of the consistency group after we have enabled write access.



Figure 4-54 The consistency group is now in Idling state.

Now we can map the secondary VDisks to our disaster/backup server. After mapping the disk a server administrator needs to check if his SDD driver can see the disks and then restart the application.

After fixing the problem on the production site all errors regarding the copy relationship need to be cleared before you can start the copy process again. When we have enabled write access to the secondary VDisk, and all errors are fixed, we can copy from the disaster/recovery site to the production site, and have the auxiliary as our primary disk.

When the production environment is stabilized there is a need for a service window for the server to switch the copy directions and run production on the production site.

4.4.3 Site failover using the CLI

When we are performing a site failover using the CLI we can implement those commands in a script, see the examples in Chapter 8, “Automation” on page 207 to make it easier.

We will show the same steps as we did in 4.4.2, “Site failover using the GUI” on page 84 but using the CLI.

The CLI sometimes provides more detailed information about what is going on in our environment.

Our production cluster is called ITSOCL3 and our consistency group is called senegal.

By checking the status of the consistency group we can see that the state is consistent synchronized and the status is online, we can also see that our primary is our master VDisks. See Example 4-40 for status.

Example 4-40 Status of consistency group called Senegal.

IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp 255id 255name senegalmaster_cluster_id 0000020068203A42master_cluster_name ITSOCL3aux_cluster_id 000002006B403A38aux_cluster_name ITSOCL2primary masterstate consistent_synchronizedrelationship_count 2



freeze_timestatus onlinesynccopy_type metroRC_rel_id 0RC_rel_name senegal_r01RC_rel_id 1RC_rel_name senegal_r02

On the server everything is normal but now something happens to our subsystem and we can see that state and status has changed. The CLI even shows us which VDisk (primary or the secondary) has a problem. And as soon as something happens to either disk the copy process stops. See Example 4-41.

Example 4-41 Copy stops

IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp 255id 255name senegalmaster_cluster_id 0000020068203A42master_cluster_name ITSOCL3aux_cluster_id 000002006B403A38aux_cluster_name ITSOCL2primary masterstate consistent_stoppedrelationship_count 2freeze_time 2007/10/23/14/03/02status primary_offlinesync in_synccopy_type metroRC_rel_id 0RC_rel_name senegal_r01RC_rel_id 1RC_rel_name senegal_r02

When this happens we need to enable write access to the secondary VDisk in this relationship, so we logon to the disaster/recovery cluster, in our case ITSOCL2. The main command to use for that is stoprcconsistgrp with the -access parameter.

When a consistency group is in a consistent state (for example, in the consistent stopped, consistent synchronized, or consistent disconnected state) you can issue the -access parameter with the stoprcconsistgrp command to enable write access to the secondary virtual disks within that group. See Table 4-2 for examples when to use the -access option.

Table 4-2 Show us the criteria needed to use the -access option.

Note: stoprcconsistgrp with -access option is the command to be used to enable write access to the secondary VDisks in the group, if the group is in a consistent state.

Initial state Final state Notes

InconsistentStopped InconsistentStopped -

InconsistentCopying InconsistentStopped -

ConsistentStopped ConsistentStopped -access permitted



in our case we use svctask stoprcconsistgrp -access senegal and the outcome is shown in Example 4-42. Notice that it is in idling state.

Example 4-42 Idling state

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp 255id 255name senegalmaster_cluster_id 0000020068203A42master_cluster_name ITSOCL3aux_cluster_id 000002006B403A38aux_cluster_name ITSOCL2primarystate idlingrelationship_count 2freeze_timestatussync in_synccopy_type metroRC_rel_id 4RC_rel_name senegal_r01RC_rel_id 22RC_rel_name senegal_r02

Now we have enabled write access to the secondary VDisk so we can map the VDisk to the disaster/recovery server with the svctask mkvdiskhostmap and the server administrator can check if he can se the disks via the SDD driver.

4.5 Copy commands using the CLI

When we are using the CLI there are several commands you can use and we will list them here.

ConsistentSynchronized ConsistentStopped -access permitted

Idling ConsistentStopped -access permitted

IdlingDisconnected Unchanged A relationship may move to stopped state when reconnected.

InconsistentDisconnected InconsistentStopped On the cluster issuing the svctask stoprcconsistgrp command

InconsistentDisconnected Unchanged On the disconnected cluster

ConsistentDisconnected ConsistentStopped On the cluster issuing the svctask stoprcconsistgrp command, -access permitted

ConsistentDisconnected Unchanged On the disconnected cluster, -access permitted.

Initial state Final state Notes



All CLI commands can be found, along with a detailed description in the IBM System Storage SAN Volume Controller Command-Line Interface User’s Guide, SC26-7903-02.

� chpartnership� chrcconsistgrp� chrcrelationship� mkpartnership� mkrcconsistgrp� mkrcrelationship� rmpartnership� rmrcconsistgrp� rmrcrelationship� startrcconsistgrp� startrcrelationship� stoprcconsistgrp� stoprcrelationship� switchrcconsistgrp� switchrcrelationship


Draft Document for Review February 10, 2008 5:17 pm 7574 Discussing FlashCopy.fm

Chapter 5. FlashCopy

In this chapter we discuss SVC FlashCopy, which is the Point-in-Time copy capability of the SVC. FlashCopy is used to create instant copies of VDisks. We cover the characteristics of FlashCopy, and how to implement it to solve different problems to ensure we meet our Business Continuity objectives.

5


7574 Discussing FlashCopy.fm Draft Document for Review February 10, 2008 5:17 pm

5.1 Introduction to FlashCopy

FlashCopy creates a rapid, complete, and consistent copy from a source VDisk to a target VDisk. Often this functionality is called Time-Zero copy, Point-In-Time copy or Snapshot™ copy. Its main characteristic is to present a clone of a VDisk for a specific point in time, very fast. That is, it is being presented as complete before the copy is “really” created.

5.1.1 What is FlashCopy and what is it useful for?

Without a function such as FlashCopy, for a self-consistent copy of data, the I/O of the application which manipulates the data has to be quiesced for the whole time the physical copy process takes place. In such a case the time that the copy process needs is defined by the amount of data to be copied, and the capabilities of the infrastructure to copy the data. Only after the copy process is finished can the application which manipulates the data start to access the VDisk that was involved. Only then is it ensured that the data on the copy target is self-consistent and identical to the data on the source for a given point in time.

With FlashCopy this is different. FlashCopy enables the creation of a copy of a source VDisk to a target VDisk in a very short time. This way, the application only has to be prevented from changing the data for a short period of time.

After the FlashCopy has been started, the target VDisk represents the contents of the source VDisk for the point-in-time when the FlashCopy was invoked. The target VDisk does not yet contain all the data of the source VDisk physically. It can be seen as a “virtual” copy, created using bitmaps.

After the FlashCopy has been has been started but has not yet finished physically copying the data to the target, the copy can be accessed in read/write mode. From now on, data which gets changed on the source VDisk (by the applications now again manipulating the source VDisk) will be written to the target VDisk, thus ensuring that the representation of the data on the target VDisk for that point-in-time is still valid.

It is also possible to copy all the data of the source VDisk to the target VDisk through a background copy process. While not copied fully, the target VDisk represents the clone of the source VDisk as long as the relationship between the source and the target exists. Once all data has been copied to the target VDisk, the relationship between the VDisks can be removed and the both VDisks become normal VDisks again. The former target VDisk is now a physical clone of the source VDisk for the point-in-time that the FlashCopy was invoked.

To create consistent copies of data which spans multiple VDisks, consistency groups can be used. Consistency groups are sets of FlashCopy Mappings, which get copied at the same point-in-time, thus creating a consistent snapshot of the data across all VDisks involved.

FlashCopy is very flexible. It is possible to have a VDisk be a source VDisk in one FlashCopy Mapping, and to be the target VDisk in another FlashCopy Mapping. Also, one VDisk can have the role as source VDisk in multiple FlashCopy Mappings with different target VDisks. Another feature is that it is possible to incrementally “update” a fully copied target VDisk with only the changes that have been made to the source VDisk of the same mapping.

5.1.2 Usage cases of FlashCopy

It is possible to create multiple target VDisks from one source VDisk. While it is the same source VDisk and different target VDisks, for every source-target VDisk combination, a dedicated mapping for each combination is required.



FlashCopy can have many uses. One obvious use is the backup of a consistent set of data without the need for a long backup window. The application manipulating the data only needs to make sure the data is consistent; and only needs to be suspended for a short period of time. Once the copy is started the backup application can access the target while the applications can resume manipulating the live data. No full volume copy is needed.

One very useful case for FlashCopy in our Business Continuity strategy is to create a full consistent copy of production data for a given point-in-time at a remote location. Here, we combine Remote Copy and FlashCopy, where we take a FlashCopy from the Remote Copy secondary VDisks. The usage case could be either to take a consistent backup of our production data on the second location or to have a clone of the data for a given point-in-time available in case our production data gets invalidated due to a logical error.

FlashCopy can also be used as a safety net for operations which could make copies of data inconsistent for a longer than normal period of time. For example, if Global Mirror gets out-of-synchronization for some reason, the auxiliary VDisk is still consistent in itself, but the process of resynchronizing renders the auxiliary VDisk inconsistent as long as it is not finished. To retain a consistent copy of the data of the auxiliary VDisk while it is being synchronized and thus is inconsistent, a FlashCopy of this VDisk can be created.

Another use is the creation of clones of data for application development testing, or application integration testing. Here, full copies of data would be useful.

One other use would be if a set of data has to be used for different purposes, for example a FlashCopy database could be used for data mining.

5.2 FlashCopy functions and features

There are characteristics and features of the FlashCopy function of the SVC which are important to understand to be able to successfully use it.

Some features available increase the flexibility of FlashCopy. These are FlashCopy Consistency Groups, Multiple Target FlashCopy (MTFC), Cascaded FlashCopy (CFC) and Incremental FlashCopy (IFC).

5.2.1 FlashCopy Mappings

A FlashCopy Mapping is created when two ordinary VDisks get “mapped” together for the purpose of creating a point-in-time copy, shown in Figure 5-1 on page 94.

Chapter 5. FlashCopy 93


Figure 5-1 Single VDisk mapping

Once associated with the mapping, the VDisks get a new responsibility, one VDisk becomes a source VDisk and one VDisk becomes a target VDisk. This mapping of VDisks gets a FlashCopy Mapping Name, which can be seen as an alias that we use to manage the source and target VDisk pairs for FlashCopy tasks, instead of using the names of the VDisks.

The source VDisk provides the data to be copied to the target VDisk. The target VDisk is used to present the data of the source VDisk for the point-in-time when the FlashCopy was started, as long as the mapping exists and not all data of the source VDisk got copied to the target VDisk.

Additionally, the entire source VDisk can get copied to the target VDisk using a background copy process. Once this is finished the mapping is no longer needed for the target VDisk to present the data.

For a single VDisk there is a limit of 16 mappings to be shared between the use of this VDisk in mappings for MTFC and CFC. Any combination of mappings representing MTFC and CFC in a “cascading tree“ may not exceed 16.

5.2.2 FlashCopy Consistency Groups

A consistency group consists of a set of FlashCopy Mappings and has a FlashCopy Consistency Group Name. This name can be seen as an alias as well. As such, FlashCopy actions can now be directed at the consistency group which enables the execution of commands on multiple mappings while ensuring consistency across data which spans multiple source VDisk for all mappings the consistency group contains. A Consistency Group consisting of 2 mappings is illustrated in Figure 5-2 on page 95.

SourceVdisk m1

TargetVdisk m1

time

Mapping m1

T0



Figure 5-2 FlashCopy Consistency Group

Data which has to be consistent can span multiple VDisks. For instance, database management system (DBMS) logs reside on a different VDisk than the database itself. The problem to take care of is called “dependent writes”. These occur, for instance, when database logs are written before (to indicate a database update is about to take place) and after (to log a successful write operation to the database) the database itself gets updated.

If we were to start a FlashCopy of the involved VDisks without ensuring consistency across VDisks, it could happen that the FlashCopy target contains both written logs but not the database update. The logs in the FlashCopy target would indicate that the write I/O has occurred, but the actual write I/O would be missing from the target VDisk which holds the database, thus rendering the FlashCopy target data inconsistent from a DBMS point of view.

Mappings which are not put into a consistency group are reported to be members of the pseudo consistency group 0.

5.2.3 FlashCopy background copy

FlashCopy background copy is a feature which enables you to copy all the data of a source VDisk to the corresponding target VDisk. Without it, only data which changed on the source VDisk would be copied to the target VDisk. The result of a FlashCopy Mapping with a

timeT0

SourceVdisk m1

TargetVdisk m1

SourceVdisk m2

TargetVdisk m2

ConsistencyGroup cg1

Mapping m1 Mapping m2

StartConsistencyGroup cg1



background copy enabled would be that the target VDisk would become a real (independent from the source VDisk) clone of the FlashCopy Mapping source VDisk.

The background copy rate is a property of a FlashCopy Mapping which is expressed as a value between 0 and 100 percent. It can be changed in any FlashCopy Mapping state and can be different in the Mappings of one Consistency Group. With a value of 0, background copy is disabled.

Table 5-1 shows the relationship of the background copy rate value to the attempted number of grains to be split per second. Grains are discussed in 5.3.2, “What is a grain?” on page 101.

Table 5-1 Background copy rate/copy speed relationship

The SVC will try to achieve these copy rates. The background copy rate and the foreground copies share the same infrastructure. If the bandwidth needed (to copy data to the underlying managed disks provided by back end storage subsystems) is used up, a latency increase and a reduction of throughput on both the foreground copies and the background copy can occur. The degradation will be graceful. The background copy is performed by both nodes of the I/O group providing the source VDisk.

5.2.4 FlashCopy autodelete option

There is a new option which enables the automatic deletion of a FlashCopy Mapping once the background copy has finished copying all data to the target VDisk. This is especially useful for use in scripts.

5.2.5 Multiple Target FlashCopy

Multiple Target FlashCopy (MTFC) is when there is a VDisk which serves as the source VDisk in multiple mappings, where in each mapping the target is a different VDisk, shown in Figure 5-3 on page 97.

Background copy rate value in %

Data copied/sec 256KB Grains split/sec

64KB Grains split/sec

1-10 128KB 0.5 2

11-20 256KB 1 4

21-30 512KB 2 8

31-40 1MB 4 16

41-50 2MB 8 32

51-60 4MB 16 64

61-70 8MB 32 128

71-80 16MB 64 256

81-90 32MB 128 512

91-100 64MB 256 1024



Figure 5-3 Multiple Target FlashCopy

For each source VDisk, up to 16 different mappings are possible. These mappings are independently controllable from each other. For instance, the background copy priority can be different for these mappings. The implementation of MTFC introduces some dependencies, discussed in 5.5, “Dependencies between FlashCopy Mappings” on page 110.

MTFC Mappings can be members of the same or different consistency groups. In a case where all the mappings are in the same consistency group, the result of starting the consistency group will be multiple identical target VDisks.

5.2.6 Cascaded FlashCopy

A new feature of FlashCopy is Cascaded FlashCopy (CFC). With CFC a VDisk is a source for one FlashCopy mapping, and the target for another FlashCopy Mapping, as shown in Figure 5-4 on page 98 and Figure 5-4 on page 98. These FlashCopy Mappings are generally independent of each other, nonetheless, dependencies of how to perform actions on them are in place, as discussed in 5.5, “Dependencies between FlashCopy Mappings” on page 110. For each cascade, 16 mappings are possible.

SourceVdisk

m1 - m16

TargetVdisk

m1

Order of FlashCopy Start (time)

Mapping 1

TargetVdisk

m2

Mapping 2

... TargetVdiskm16

Mapping 16



Figure 5-4 Cascaded FlashCopy

The use of consistency groups is restricted when using CFC. A consistency group serves the purpose of invoking FlashCopy Mappings at the same point-in-time.

Within the same consistency group it is not possible to have mappings where:

� The source VDisk of one mapping is the target of another mapping.� The target VDisk of one mapping is the source VDisk for another mapping.

These combinations would not be useful because within a consistency group there is no way to have the mappings established in a certain order, which renders the content of the target VDisks undefined (for instance, was the first mapping established before the target VDisk of the first mapping that acts as a source VDisk for the second mapping?).

Even if there was a way to ensure an order in which the mappings are established within a consistency group, the result would be equal to MTFC (two VDisks which hold the same target data for one source VDisk).

In other words, a cascade is useful when we want to copy VDisks in a certain order (and copying the changed content targets of FlashCopies) rather than at the same time in an undefined order (from within one single consistency group).

5.2.7 Combined Multiple Target FlashCopy and Cascaded FlashCopy

MTFC and CFC can be combined in any way, as long as combinations of both do not exceed 16 mappings in one tree. A simple illustration of combining MTFC and CFC is shown in Figure 5-5.

Vdiskt m3-16


Vdiskt m1s m2

...Vdiskt m2

s m3-16

Mapping m1 Mapping m2 Mappings m3-16

SourceVdisk m1



Figure 5-5 A Tree of combined MTFC and CFC

As we can see in this picture, what we call a “target VDisk” or a “source VDisk” is in reality a VDisk which can be in the role of being the source, the target or a source and a target, if it is a member of one or more FlashCopy Mappings.

5.2.8 Incremental FlashCopy

Without Incremental FlashCopy (IFC), once all data in a FlashCopy Mapping has been copied to a target VDisk, and the FlashCopy Mapping is restarted, all data has to be copied again to create an independent copy.

By creating a FlashCopy Mapping with the “incremental” flag, only the data which has been changed since the last FlashCopy was started gets written to the target VDisk, illustrated in Figure 5-6 on page 100.

SourceVdisk

m1 – m2

TargetVdisk m1


Mapping m1

TargetVdisk m2 /

SourceVdisk m3

Mapping m2

TargetVdisk m3

Mapping m3



Figure 5-6 Incremental FlashCopy

The amount of data to be copied depends on the grain size as well: if the grain size is 64KB (as compared to 256KB), there might be less data to copy in order to get a full independent copy of the source again.

A “difference“ value is provided in the query of a mapping, which makes it possible to know how much data has changed, which would need to be copied once the IFC is restarted. The difference value is the percentage (0% - 100%) of how many grains have been changed and would need to be copied to the target VDisk in order to get a fully independent copy of the source VDisk again.

This functionality is needed in cases where we want to have, for example, a full copy of a VDisk for disaster tolerance, and/or application testing and/or data mining. It greatly reduces the time to establish a full copy of the source data again as a new snapshot, once the first background copy completed. In cases where customers maintain full independent copies of data as part of their disaster tolerance strategy, using IFC can be of benefit for their Business Continuity goals.

5.3 FlashCopy internals

Understanding how FlashCopy works internally helps us to configure it the way we need it, and to get the most benefit from it.

TargetVdisk m1

time

Start Mapping m1:copy all data

T0 (1)

Data change

SourceVdisk m1

SourceVdisk m1

TargetVdisk m1

Restart Mapping m1:copy changed data only

T0 (2)



5.3.1 What is the indirection layer and what is it for?

For FlashCopy to function, the I/O path of the SVC is being altered, with the introduction of a facility called the “indirection layer“. It is an additional layer of functionality in the I/O path which causes the I/O to make a detour to enable the FlashCopy function.

The indirection layer becomes active in the I/O path at the start of a FlashCopy Mapping and intercepts I/Os directed to both the source VDisk and the target VDisk. It decides for each I/O what actions are to be performed. It either:

� Allows the I/O to go directly to the VDisk� Redirects the I/O from the target VDisk to another VDisk� Stalls the I/O until it arranges the copying of data from the source VDisk to the target

VDisk

The indirection layer is placed below the cache, and this means any latency introduced by its function does not affect the interaction between the layer above it (the storage client and the SVC cache). If latency occurs because of I/Os which are stalled, this only affects the I/O generated by destaging of the cache. The decision on how to handle I/Os is based upon three attributes:

� The destination of the I/O (VDisk and its Logical Block Address (LBA))� Its direction (either it is a write or a read operation)� The state of the FlashCopy “bitmap”

How read operations are handledRead operations targeted to the source VDisk are always passed directly to the source VDisk. Read operations from the target VDisk use the information stored in the FlashCopy bitmap. In a case where the data to be read has not yet been copied to the target VDisk, it will be read from the source VDisk, or from another target VDisk in the case of MTFC. In a case where it has been copied, the data is read from the target VDisk.

How write operations are handledWrite operations for which the grain containing the data to be written to has been split already will go to the VDisk. Write operations to source VDisks and target VDisks which target an area which has not yet been copied will result in the write operation being stalled. Then a copy operation is started to copy the data from the source VDisk to the target VDisk, and after this completes, the stalled write operation resumes.

5.3.2 What is a grain?

Data gets copied between VDisks in a FlashCopy Mapping in units of the FlashCopy Mapping address space called “grain” (the granularity of the data to be copied from the source VDisk to the target VDisk is one grain). The grain size can be 256KB (default) or 64KB.

The grain size can affect the amount of data needed to be copied for Incremental FlashCopy, as with IFC, the smaller grain size of 64KB can lead to less data to be copied. If a new FlashCopy Mapping gets added to a tree, it will use the grain size which is already in use for the mappings in the tree. It is also possible to join two existing trees of FlashCopy Mappings. To be able to create the mapping used to join the trees, both trees must use the same grain size.



5.3.3 What is a bitmap?

The bitmaps are an internal data structure, used to track which grains in FlashCopy mappings have been copied from the source VDisk to the target VDisk (which grains have been “split“). One bit in each bitmap represents the state of one grain and either it is split or it is not.

The FlashCopy bitmap uses up bitmap space in the memory of the SVC. The maximum size of the bitmap space is 512MB per I/O Group, which has to be shared between FlashCopy bitmaps and Remote Copy bitmaps. How we assign the bitmap space to the copy functions is a variable.

The default is 20MB for FlashCopy and 20MB for Remote Copy (Metro/Global Mirror). The maximum shared bitmap space is 512MB per I/O group, that is to say 256MB for FlashCopy and 256 for Remote Copy, or 512 for FC and 0 RC or any similar combination.

This feature allows us to trade off memory between the cache, FlashCopy and Remote Copy (Metro Mirror/Global Mirror).

5.3.4 Usable bitmap space

Table 5-2 on page 102 shows the bitmap space / FlashCopy address space relationship depending on the size of the grain and the kind of copy service being used.

Table 5-2 Bitmap space / FlashCopy address space relationship for the specified I/O group

Copy Service

Grain size in KB:

1Byte of bitmap space gives a total of:

4KB of bitmap space gives a total of:

1MB of bitmap space gives a total of:

20MB of bitmap space gives a total of:

512MB of bitmap pace gives a total of:

Metro Mirror/Global Mirror

256 2MB of VDisk capacity

8GB of VDisk capacity

2TB of VDisk capacity



FlashCopy 256 2MB of target VDisk capacity

8GB of target VDisk capacity

2TB of target VDisk capacity



FlashCopy 64 512KB of target VDisk capacity





Incremental FlashCopy

256 1MB of target VDisk capacity





Incremental FlashCopy

64 256KB of target VDisk capacity







5.3.5 Number of FlashCopy Mappings in a cluster

The new maximum number of FlashCopy Mappings is 3855 for the whole cluster. Each single mapping (which might be part of a Multiple Target FlashCopy or Cascaded FlashCopy) contributes to the maximum number of mappings.

Before Multiple Target FlashCopy and Cascaded FlashCopy, when a VDisk could only take part in one 1-on-1 relationship, the maximum number of mappings was the maximum number of VDisks per cluster (4096) divided by two (2048).

The limit of 4096 VDisks per cluster still exists and it is still responsible for the maximum number of FlashCopy mappings. This time though, the maximum number depends on how we use FlashCopy, and to what extent we use MTFC and CFC. With the introduction of Multiple Target FlashCopy the maximum limit for mappings went up even though the maximum number of VDisks per cluster stayed the same. This is because with MTFC, one VDisk (which obviously only takes one spot in the maximum VDisk limit of 4096) can be the source VDisk in up to 16 mappings, and with CFC a VDisk can be a member of two mappings as well as increasing the possible number of mappings in a cluster.

Although it is good to have the increased flexibility of MTFC and CFC and the possible higher number of mappings, using this flexibility comes at an expense. That is the more VDisks we use up as target VDisks for MTFC, or the more we use CFC, the number of VDisks in a cluster left to be used as source VDisks for MTFC, or a cascade (as a “root VDisk“), declines accordingly.

For instance, in the extremely unlikely case (and this would be the “worst case”), that we try to use every VDisk to be copied as a source VDisk in as many mappings as possible (mapped with 16 different target VDisks), we could only have 240 VDisks to be source VDisks mapped to 16 target VDisks and 1 remaining VDisk mapped to 15 target VDisks.

The calculation here is as follows: 17 VDisks (1 source VDisk and 16 target VDisks) multiplied by 240 equals 4080 VDisks. The maximum number of VDisks supported by an SVC cluster is 4096, which leaves 16 VDisks to be used in mappings, which could be 1 VDisk as the source VDisk and 15 acting as target VDisks.

So, as we can see, the practical number of VDisks to act as source (root) VDisks in FlashCopy Mappings is somewhere between 241 (worst case, maximum usage of MTFC) and 2048 (no usage of MTFC and CFC).

Note: For the FlashCopy Mapping address space, the amount of mappings has to be taken into account.

Each mapping target for a FlashCopy uses up bitmap space. This means for MTFC, that even if there is only one source VDisk involved, multiple targets will consume the same amount of bitmap space as the same number of targets of independent FlashCopy Mappings would use up.

Note: These numbers apply to a cluster consisting of 4 I/O Groups supporting 4096 VDisks.

A cluster consisting of 2 nodes in one I/O Group supports 1024 VDisks. This cluster would only support between 60 “root” source VDisks, each in 16 mappings and 1 source VDisk in 3 mappings (worst case) and 512 source VDisks (no use of MTFC and CFC).



5.3.6 How can we specify and view the bitmap space on an I/O group

The user can increase and decrease the amount of memory used on nodes for keeping FlashCopy and Remote Copy bitmaps at runtime.

For example, for FlashCopy we specify the amount of bitmap space of 20MB for io_grp0, shown in Example 5-1 on page 104 and list it.

Example 5-1 Specifying and viewing the bitmap space located on an I/O group

admin>svctask chiogrp -feature flash -size 20 io_grp0admin>admin>svcinfo lsiogrp 0id 0name io_grp1_rnode_count 2vdisk_count 382host_count 4flash_copy_total_memory 20.0MBflash_copy_free_memory 20.0MBremote_copy_total_memory 20.0MBremote_copy_free_memory 19.9MB

5.3.7 Where is the bitmap stored?

The bitmap for FlashCopy Mappings is stored in a particular I/O group. When we create a new FlashCopy Mapping we can force which I/O group we want the bitmap to reside in. Using the CLI this can be achieved with the -iogrp parameter. This feature can be useful if changes are planned, for example creating, joining or re-organizing trees.

This ability can be useful if the user knows that trees of mappings may be joined in the future, or for reconstructing a tree of mappings in a different order to their original creation order during a configuration restore.

� If we specify the I/O group of the “target vdisk” when creating the mapping, the bitmap space will go towards the I/O group of the target VDisk.

� If we do not specify the I/O group for the bitmap space, the bitmap will be placed on the I/O group which owns the source VDisk.

� If we add a mapping to a Cascaded FlashCopy or a Multiple Target FlashCopy or a ’”cascaded tree” (a combination of both), discussed in 5.3.8, “FlashCopy characteristics listed” on page 105, the bitmap will be stored on the I/O group where the bitmap for the other mappings is stored already.

Note: This feature can lead to FlashCopy Mapping behavior which should be taken into consideration.

If, for example, we assume the bitmap of a tree is stored in I/O group 0, and some VDisks which are members of mappings in that tree are owned by I/O group 1, when I/O group 0, where the bitmap for the tree is stored is taken offline, the FlashCopy Mappings for the VDisks owned by I/O group 1 (which is still online, and whose VDisks are still online) will stop.



5.3.8 FlashCopy characteristics listed

Following is a list of FlashCopy characteristics:

� FlashCopy is a copy process involving two VDisks associated with a FlashCopy Mapping.� Once used in a FlashCopy Mapping, a VDisk becomes a source or target VDisk.� A FlashCopy copies a whole source VDisk to a target VDisks within one cluster.� The VDisks chosen as the source and target VDisks can be provided by different I/O

groups.� A VDisk can be the source VDisk in multiple FlashCopy Mappings.� A VDisk can be the target VDisk in one FlashCopy Mapping.� A VDisk can be the source in one mapping and the target in another at the same time.� The VDisks used for one FlashCopy Mapping must be of the same size.� The size of VDisks which are members in a FlashCopy Mapping can not be changed.� The content of the source VDisk does not get changed by the FlashCopy process.� The original content of the target VDisk gets destroyed by the FlashCopy process.� After the FlashCopy is established, the target VDisk represents a clone of the source

VDisk.� After the FlashCopy is established, the target VDisk can be accessed as read/write.� Until all the data is copied, the target VDisk presents the data as long as the mapping

exists.� Once all the data got copied, the mapping can be deleted and the VDisks are independent

again.� Once all the data got copied, the former target VDisk holds the data even without the

mapping.

5.3.9 FlashCopy maximum configurations

To plan for and to implement FlashCopy, some limits have to be checked and adhered to. The limits for an SVC cluster are shown in Table 5-3 on page 105.

Table 5-3 Configuration limits for FlashCopy

Property Maximum Notes

FlashCopy targets per source 16 The maximum number of FC Mappings that can exist with the same source VDisk

FlashCopy Mappings per cluster

3855 The maximum number of FC Mappings is calculated by noticing that 240 source VDisks with 16 FC Mappings plus one more with 15 mappings uses up all 4096 VDisks per cluster

FlashCopy consistency groups per cluster

128 An arbitrary limit policed by the software

FlashCopy VDisk space per I/O group

1024TB This is a limit on the quantity of FC Mappings using bitmap space from this I/O Group. This maximum configuration will consume all 512MB of bitmap space for the I/O Group and allow no Metro and Global Mirror bitmap space.The default is 40TB



5.4 FlashCopy events and states

The state of a FlashCopy Mapping is managed on a per FlashCopy Mapping basis (commands which prepare and start a FlashCopy Mapping are targeted to a consistency group).

5.4.1 FlashCopy Mapping states

A FlashCopy Mapping can have states, and it is put in these states by FlashCopy Mapping events.

CopyingThe FlashCopy has been started. From now on, the target VDisk presents the content of the source VDisk for the point in time when the FlashCopy was started (but is not holding all the data yet).

Idle/CopiedVDisks are members of a mapping but behave independently. The target represents a full copy of the source for the point in time when the FlashCopy was started. On both VDisks, write- and read caching is enabled.

StoppingThe mapping is in the process of transferring data to a mapping which depends on it.

StoppedThe data on the target VDisk is lost and the target VDisk is put offline. To regain access to the target VDisk, the mapping must be either deleted or it must be started again.

SuspendedThe FlashCopy Mapping was in the Copying or the Stopping state and the target has been “flashed” from the source. The access to the metadata has been lost resulting in both the source VDisk and the target VDisk going offline.

PreparingThe FlashCopy Mapping gets prepared for a FlashCopy start while I/O still continues to the source VDisk. This includes flushing the modified write data of the source VDisk from the cache, placing the cache for the source VDisk into write through mode, and discarding any data associated with the target VDisk from the cache.

PreparedThe FlashCopy Mapping is ready to be started.

FlashCopy Mappings per consistency group

512 Due to the time taken to prepare a consistency group with a large number of mappings.

Property Maximum Notes



Table 5-4 on page 107 summarizes the FlashCopy Mapping states and the corresponding cache states for the VDisks in a FlashCopy Mapping.

Table 5-4 FlashCopy Mapping states

5.4.2 FlashCopy events

In Figure 5-7 on page 109 we show the events alter the state a FlashCopy Mapping. We describe them here.

CreatePlaces the FlashCopy Mapping into the Idle/Copied state. It creates a new FlashCopy Mapping between two VDisks, assigning roles to them, one now being a source VDisk and one being the target VDisk for the new mapping.

PreparePlaces the FlashCopy Mapping in the Preparing state. The prepare command is directed to a consistency group (single mappings are members of consistency group 0, in this case the command is targeted at the mapping name for the given FlashCopy Mapping).

Flush DonePlaces the FlashCopy Mapping in the Prepared state. This is done once all cached data from the source VDisk has been flushed and all cached data of the target VDisk has been invalidated.

StartPlaces the FlashCopy Mapping in the Copying state. The FlashCopy Mapping is being started. The start of the FlashCopy Mapping is synchronized (ensuring consistency across all involved mappings in the consistency group).

Source VDisk Target VDisk

State Mapping State Cache State Mapping State Cache State

Idling/Copied online write-back online write-back

Copying online write-back online write-back

Stopped online write-back offline n/a

Stopping online write-back Online if copy complete, offline if not

n/a

Suspended offline write-back offline n/a

Preparing online write-through online, but not accessible

n/a

Prepared online write-through online, but not accessible

n/a

Note: Even though the FlashCopy Mapping has not been started yet, the target VDisk has to be considered corrupt from now on, since beginning with the act of preparing, cached writes are discarded.



StopPlaces the FlashCopy Mapping in either the Stopped or Stopping state. A FlashCopy Mapping can be stopped by a user command or by an I/O error.

Stop CompletePlaces the FlashCopy Mapping in the Stopped state. It occurs as a transition from Stopping to Stopped as a result of a complete copy operation to break dependencies between FlashCopy Mappings.

Flush FailedPlaces the FlashCopy Mapping in the Stopped state. This happens when the flush of data from the cache fails.

Copy CompletePlaces the FlashCopy Mapping in the Idle/Copied state. This happens once every grain has been split. With the autodelete feature being used, the FlashCopy Mapping will be deleted once it completes the copy. Without it, the FlashCopy Mapping will remain and can be prepared and started again. The copy complete event will not occur while there are still mappings which depend on it in the copying state.

Bitmap Offline/Bitmap OnlineIf all nodes in the I/O group that is storing the bitmap for a FlashCopy mapping go offline, then the bitmap will go offline. When the last node to go offline comes back online the bitmap will come back online. Bitmap Offline places the FlashCopy Mapping in the Suspended state. Bitmap Online places the FlashCopy Mapping in the Copying or Stopping state.

ModifyThis does not place the FlashCopy Mapping in another state. Many properties can now be modified - name, consistency group, background copy rate, cleaning rate,and the autodelete attribute.

DeleteThe Delete event deletes the FlashCopy Mapping.

Figure 5-7 on page 109 is the FlashCopy Mapping state diagram. It illustrates which state a mapping can be in, and what events are responsible for a state change.



Figure 5-7 FlashCopy Mapping State diagram

5.4.3 FlashCopy consistency group states

For reference, the FlashCopy consistency group state diagram is shown in Figure 5-8 on page 110. This diagram explains the state transitions for consistency groups, and the possible states of the mappings within the consistency groups.

Legend

FLASHCOPY MAPPING STATE

FlashCopyEvent

PREPARINGIDLE_OR_COPIED PREPARED

STOPPED STOPPING

Delete,Autodelete

Create

Prepare FlushDone Start

Copy Complete

Delete(Force)

FlushFailedPrepare Stop Bitmap

Offline

COPYING

SUSPENDED

BitmapOnline

BitmapOnline

BitmapOffline

StopCompletes(target is

complete)

StopCompletes

(target is notcomplete)

Stop

StopCompletes

(target is notcomplete)

Stop

Stop Completes(target is complete)



Figure 5-8 FlashCopy Consistency Group state diagram

5.5 Dependencies between FlashCopy Mappings

FlashCopy Mappings, which are members in a tree, can depend on each other. A dependency is when one or more mappings are responsible to provide grains for another mapping. In such a case, actions like the deletion of one mapping can affect other mappings which depend on the mapping being deleted.

A dependency between mappings only exists as long as a source VDisk of a mapping does not physically holds all grains. This can happen only when there are preceding mappings in a tree which have not yet completed the background copy process to establish an independent clone of the source VDisk).

5.5.1 Linked lists of mappings and resulting dependencies

Multiple Target FlashCopy and Cascaded FlashCopy (as well as single FlashCopy) are implemented using the same method, with structures called linked lists. This means that the restrictions dictated by this implementation apply to both, MTFC and CFC. Hence, the linked list implementation of FlashCopy is the cause for dependencies between mappings when using MTFC as well as using CFC.

COPYING

COPYINGSTOPPINGSTOPPEDIDLE_OR_COPIED

PREPARING

STOPPEDPREPAREDPREPARING

SUSPENDED

COPYING SUSPENDEDSTOPPINGSTOPPEDIDLE_COPIED

IDLE_OR_COPIED

IDLE_OR_COPIED

PREPARED

PREPARED

STOPPED

STOPPEDIDLE_OR_ COPIED

STOPPING

STOPPINGSTOPPEDIDLE_OR_COPIED

Legend

CONSISTENCY_GROUP STATE

VALID MAPPING STATE

IDLE_OR_COPIED



What is a linked listLinked lists are dynamic internal data structures which hold informations about active FlashCopy Mappings in a MTFC setup, a FlashCopy cascade or a combination of both, a tree. A mapping gets inserted into a linked list of mappings when it is being started, not at the time of the creation of the mapping. A cycle of mappings is possible to be set up, but it is not allowed to have them all active at the same time, no cycle is allowed in a linked list.

Dependencies and effect on mappingsDependencies in a dependency chain exist as long as there are unsplit grains in a single mapping. When a target VDisk in a mapping has received all grains from the source VDisk, the target VDisk is no longer dependent on the source VDisk. Similarly, when in a linked list of mappings all grains from a target VDisk (acting as a source) have been copied to a dependent target VDisk, the dependency ceases to exist.

Dependencies between mappings have effects such as the dependent mappings will stop if a mapping they depend on stops (either because it is force-stopped or has an error). Also, the target VDisks of the dependent mappings go offline if the target VDisk of the mapping they depend on goes offline. This happens if the source VDisk goes offline whilst the background copy is incomplete.

Dependencies with Cascaded FlashCopyIn the case of CFC, the dependency exists because the source VDisk of a mapping can be the target VDisk of another mapping which does not hold all the data physically, because the background copy process of the preceding mapping is not complete. Thus grains could still reside on the source VDisk of the preceding mapping in the cascade. A cascaded mapping cannot be prepared in case active mappings exist further down the cascade. The command to display these dependent mappings is shown in 5.5.3, “Command Line commands for information on dependencies” on page 116.

For Cascaded Mappings, if there are mappings in the linked list below the current mapping that are active (not stopped or idle-copied) then this mapping can not be started. In order to start this mapping, the downstream mappings must be stopped.

Dependencies with Multiple Target FlashCopyWith MTFC there is no obvious “preceding” mapping, since all target VDisks share the same source VDisk, which holds all data physically. Nonetheless, because of the implementation of FlashCopy using linked lists, an order is being established, which makes mappings precede other mappings. To avoid too much load on the source VDisk in case of many mappings with the same source, MTFC is implemented in a way that mappings try to get grains from preceding mappings in the list, which are newer members of the list (mappings started later).

Position of mappings and resulting dependenciesThe first mapping in a linked list just maps one source VDisk to a target VDisk (first member, oldest mapping). New mappings are inserted into the linked List (when the mapping is being started) using the following rules:

� A new Cascaded Mapping is inserted behind the mapping whose target VDisk is being used as the source VDisk.

� A new Multiple Target Mapping is inserted in front of the next oldest mapping of the same source VDisk.

� If multiple mappings are started at the same time, because they are in the same Consistency Group, an unspecified ordering will be chosen.



The mapping being in the first position, at the “head” of the inked list (not necessarily the one being inserted first) does not depend on other mappings. Every other succeeding mapping depends on the preceding mappings in the list.

The following illustrations show how mappings get inserted into a linked list, depending upon the being set up as MTFC or CFC or both.

A simple MTFC is shown in Figure 5-9 on page 112.

Figure 5-9 Multiple Target FlashCopy - m2 started after m1

Both mappings, sharing the same source VDisk result in an MTFC which results in the linked list consisting of two mappings, shown in Figure 5-10. These lists start with the leftmost mapping, which is called the head of the list.

The arrow originates from the mapping which is dependent on the mapping to which the arrow points to.

Figure 5-10 Linked list of MTFC Mappings m1 and m2

The mapping m2, which was started after mapping m1, precedes mapping m1 in the list. Mapping m1 now depends on mapping m2. This means, for any grains not located in mapping m1, the target VDisk of mapping m2 gets asked. Figure 5-11 on page 113 shows which VDisks provide grains for VDisk 2.

Vdisk 1

s m1, s m2

Vdisk 2

t m1

time

Vdisk 3

t m2

Mapping m1started at t1


m1started at t1

m2started at t2



Figure 5-11 Vdisks providing grains

A simple CFC is shown in Figure 5-12.

Figure 5-12 Cascaded FlashCopy - m2 started after m1

Figure 5-13 on page 113 shows the resulting linked list.

Figure 5-13 Linked list of CFC Mappings m1 and m2

Vdisk 1provides grains to

Vdisk 3

Vdisk 3

t m2

Vdisk 2

t m1

Vdisk 1

s m1, s m2


Vdisk 2

Vdisk 1 provides grains to Vdisk 2in case Vdisk 3 could not provide the grain

Vdisk 1

s m1

Vdisk 2

t m1s m2

time

Vdisk 3

t m2



m2started at t2

m1started at t1



Figure 5-14 shows which VDisks provide grains.

Figure 5-14 Vdisks providing grains

The rules about inserting mappings to the list also apply when combining MTFC and CFC. An example of such a tree is shown in Figure 5-15.

Figure 5-15 Combined MTFC and CFC

The resulting linked list is shown in Figure 5-16 on page 115.


Vdisk 2

Vdisk 2

t m1s m2

Vdisk 3

t m2

Vdisk 1

s m1


Vdisk 3

Vdisk 1 provides grains to Vdisk 3in case Vdisk 2 could not provide the grain

Vdisk 4

t m3s m4

time

Vdisk 5

t m4



Vdisk 1

s m1, s m2

Vdisk 2

t m1



Vdisk 3

t m2s m3



Figure 5-16 Linked List of combined MTFC and CFC

Writes to target VDisks in a linked listFor MTFC, writes to a target VDisk as well cause writes to target VDisks of the next dependent mapping. When a target VDisk gets written to by an application, its content changes compared to the content of the source VDisk for the specified point-in-time.

This change, however, is not supposed to happen on other mappings. To retain the data of the point-in-time for other dependent mappings, the target VDisk of the preceding mapping copies the grains to be changed to the target VDisk of the dependent mapping. For CFC, a write to a target VDisk causes a write to the target VDisk of the next mapping in the cascade. This write to an unsplit grain on this target VDisk merges the data from the target VDisk of the previous mapping in the cascade with the storage client write data and writes the result to the next in the cascade.

5.5.2 Stopping state and cleaning

One case where dependencies are to be coped with is when a mapping is to be stopped and deleted. This would have an impact on dependent mappings for which a VDisk in the deleted mapping held grains which were still not split. This, of course, needs to be prevented from happening.

To avoid impacting mappings which depend on the mapping to be stopped, the Stopping state is used. During the Stopping state, a copy process finds and copies all grains, which are unique on the mapping to be stopped, to another mapping in the tree. After this has been done, the mapping can enter the Stopped state and can be deleted. The target VDisk of a FlashCopy Mapping in the Stopping state is offline if the target is not an independent copy, and is therefore not accessible to storage clients for an what may be an unacceptably long time. If it is an independent copy then it is online whilst the mapping is stopping.

To tackle this, a new method of copying grains is being implemented, and this is the Cleaning mode. In this mode, grains are copied from the mapping to be stopped while it is still in the Copying state (and while its target VDisk is still accessible to the storage client) to an older mapping.

This is implemented using a new Cleaning Rate parameter. If this parameter is combined with a zero copy background rate, it causes the mapping to copy the grains to a downstream mapping.

Information about the progress is contained in a new Cleaning Progress field introduced into the query output of the mapping.

After the cleaning is complete (all unique grains are copied) the mapping can be stopped, and it will be in a stopped state immediately. If the mapping is stopped before the cleaning process completed, the mapping enters the Stopping state first, and the remaining grains will get copied. After this completes, the mapping will enter the Stopped state.

m3started at t3

m2started at t2

m1started at t1

m4started at t4



5.5.3 Command Line commands for information on dependencies

Using the command svcinfo lsvdiskdependentmaps we can display all mappings with target VDisks that are dependent upon data held on the specified VDisk. Cascaded Mappings can not be prepared (and then started) in case active mappings exist further down the cascade, which would be displayed using this command. The syntax is shown in Example 5-2 on page 116.

Example 5-2 svcinfo lsvdiskdependentmaps

>>- svcinfo -- -- lsvdiskdependentmaps --+- vdisk_id ---+------>< '- vdisk_name -'

We can as well specify the mapping using the command is svcinfo lsfcmapdependentmaps, which displays all the FlashCopy Mappings that are dependent on the user specified mapping. The syntax is shown in Example 5-3.

Example 5-3 svcinfo lsfcmapdependentmaps

>>- svcinfo -- -- lsfcmapdependentmaps -- --+----------+-- -----> '- -nohdr -'

>--+-----------------------+-- --+- fc_id ---+----------------->< '- -delim -- delimiter -' '- fc_name -'


Draft Document for Review February 10, 2008 5:17 pm 7574 Implementing FlashCopy.fm

Chapter 6. Implementing FlashCopy

In this chapter we show how to use FlashCopy in three different cases using the SVC GUI, and in one case using the CLI.

First we show a basic FlashCopy, secondly a Multiple Target FlashCopy and the third a Cascaded FlashCopy which uses the Incremental FlashCopy feature as well.

The example using the CLI is FlashCopy using Metro Mirror secondary VDisks as the FlashCopy source VDisks.

6


7574 Implementing FlashCopy.fm Draft Document for Review February 10, 2008 5:17 pm

6.1 Example 1: basic FlashCopy using the GUI

The first example we show is a basic FlashCopy consisting of one mapping where we do not need all data of the VDisk to be copied physically.

Goal: The goal is to backup the production data using a very small backup window, which is one of the most basic problems. As the data to backup gets larger, the time required to backup the data increases. In cases where a full consistent copy is needed, the application changing this data would have to be suspended from changing the data for the whole time the backup is being created.

Solution: Using FlashCopy to create a snapshot of the data. This is achieved through the use of FlashCopy to create a ’virtual’ copy of the data for the given point in time. Since it only needs a short time to complete its copy, the application has to be prevented from changing the data only for a short time.

Of course, the data should be consistent, all involved caches need to be flushed and written to the VDisk. The target copy can instantly be accessed by the backup application while the source is ready to be accessed and used (and to be changed) again.

6.1.1 Creating a FlashCopy mapping

We can start to create a single FlashCopy as shown in Figure 6-1 on page 118, on the Viewing FlashCopy Mappings screen. Here, existing mappings are displayed (none for now). The only action possible right now is to create a new mapping.

Figure 6-1 Viewing FlashCopy mappings - no mapping exists

The first screen of the Create a Mapping wizard is shown in Figure 6-2 on page 119. It gives an overview over the steps we go through to create the mapping.



Figure 6-2 Create a FlashCopy Mapping - Introduction

The wizard is very straight forward and consists of setting the properties of the FlashCopy Mapping, choosing the VDisks to be used and to verify the settings. The first step, to set the properties of the FlashCopy mapping, is shown in Figure 6-3 on page 120.

Chapter 6. Implementing FlashCopy 119


Figure 6-3 Create a FlashCopy fcm- Set the Properties

We can choose the FlashCopy Mapping Name, or let the system decide. We have not defined a Consistency Group yet, so the related input field Consistency Group Name is empty. Therefore, this mapping will be associated with Consistency Group 0, which is used for stand-alone mappings.

The Background Copy Priority specifies the speed of the background copy the SVC attempts to achieve, in case we would want to have the source copied to the target in fully. The possible speed is from 128 KBps at 10% priority, up to 64 MB/s at 100% priority. Priority 50 has a target copy rate of 2 MBps. Refer to 5.2.3, “FlashCopy background copy” on page 95 for a discussion about background copy.

For our purpose, we chose NOCOPY, which means no data gets copied by the background copy process. The Cleaning Priority does not need to be set because there is no mapping yet which would depend on this mapping (this new parameter is discussed in 5.5, “Dependencies between FlashCopy Mappings” on page 110).

The type of mapping will be Standard, as opposed to Incremental. We do not need it to be incremental because we do not want a full copy of the source VDisk which could benefit from being incrementally refreshed. Also, incremental FlashCopy would use up to twice the amount of bitmap space compared to standard FlashCopy (Incremental FlashCopy is discussed in 5.2.8, “Incremental FlashCopy” on page 99).

The grain size we use is 256KB. The new option of a grain size of 64KB uses four times the amount of bitmap space compared to a grain size of 256KB. A Grain Size of 64KB can be used to decrease the amount of data to be copied when restarting an Incremental FlashCopy mapping and thus the time it takes to re-establish a full independent clone of a source VDisk.



The last feature which can be used on this screen is that we can choose to have the system automatically delete the FlashCopy mapping when the background copy completes. We are not creating a full copy here, so we do not use this option.

Once we click on the Next button, we get a screen where we can filter the source VDisk candidates we want to see in the next step, shown in Figure 6-4 on page 121.

Figure 6-4 Create a FlashCopy mapping - Filtering Source VDisk Candidates

We can filter by Virtual Disk (VDisk) Name and Managed Disk Group Name (using a wildcard), by Virtual Disk Type (any, sequential, striped, image), I/O Group and State (offline, online, degraded).

In our example we have chosen to filter by VDisk name, since our naming scheme indicates the production VDisks to be used as source VDisk with the characters “prod“ as pat of the VDisk name.

The next screen shows us the filtered VDisks, as shown in Figure 6-5 on page 122.



Figure 6-5 Create a FlashCopy mapping - Select the Source VDisk

We select the VDisk “vd-prod-0001” to be the source VDisk for our FlashCopy mapping. Pressing Next, we get to the screen where we can choose the target VDisk, this time without a filter screen before, therefore displaying all VDisks in the system.

To get the target VDisk candidates filtered, we select Show Filter Row from the pull down menu, then click on filter on top of the Name column to get to the filter properties, shown in Figure 6-6 on page 122.

Figure 6-6 Create a FlashCopy mapping - Select the Target VDisk - Filter

Here we do not need to use the wildcard as we did within the filter screen for the source VDisk candidates. After selecting the VDisk to be the target VDisk and pressing Next, we verify the settings of the FlashCopy mapping as shown in Figure 6-7 on page 123.



Figure 6-7 Create a FlashCopy mapping - Verify FlashCopy mapping

Pressing Finish creates the mapping and takes Viewing FlashCopy screen, shown in Figure 6-8 on page 123.

Figure 6-8 Viewing FlashCopy mappings - one mapping created, state: Idle or Copied

We have successfully created the first FlashCopy mapping whose properties we can view, clicking on the FlashCopy mapping Name, shown in Figure 6-9 on page 124.



Figure 6-9 View FlashCopy mapping Details for fcm-001

We see the state of the mapping is Idle_or_Copied, therefore it is ready to be prepared and started. The consistency group related fields hold no information, internally this mapping belongs to Consistency Group 0.

6.1.2 Starting a FlashCopy mapping

Given that the application accessing and altering the data on the source VDisk has been stopped from changing the data, we can now start the FlashCopy mapping. To start it, we need to prepare and then start the mapping. This can be done either by selecting Prepare a Mapping from the pull down menu or by selecting Start a Mapping , as shown in Figure 6-10 on page 124.

Figure 6-10 Starting a FlashCopy Mapping



We get to the screen used to chose to prepare the mapping, shown in Figure 6-11 on page 125.

Figure 6-11 Starting FlashCopy Mapping fcm-001

Pressing OK first prepares (puts the mapping in the state Preparing) and then starts the mapping. The state of the mapping is now Copying, shown in Figure 6-12 on page 125. The application altering the data residing on the source VDisk can resume to do so, the target VDisk is brought online and ready to be used.

Figure 6-12 Viewing FlashCopy Mappings - one mapping started, state: Copying

6.1.3 Modify a mapping

It is possible to modify the properties of the mapping we created and started. Selecting the mapping and choosing Modify a Mapping from the pull down menu takes us to the screen shown in Figure 6-13 on page 126.



Figure 6-13 Modifying FlashCopy Mapping fcm-001

Here, the name of the mapping, the Consistency group it belongs to, the background copy priority, the cleaning priority and the autodelete option can be modified.

What cannot be modified is the grain size and the type of mapping (standard, incremental) and the I/O group the bitmap resides on.

6.1.4 Stopping a mapping

After we have our data from the target VDisk backed up we stop the mapping, shown in Figure 6-14 on page 126. This is needed in the next step, when we use this mapping as a member in a new Consistency Group.

Figure 6-14 Stop a FlashCopy Mapping



Now we get a warning screen, shown in Figure 6-15 on page 127, telling us that the content of the target VDisk will be unusable after we stop the mapping.

Figure 6-15 Stopping a FlashCopy Mapping fcm-001 - Stop or Forced Stop

This is because most of the data of the target VDisk is still located on the source VDisk, only data which would have been overwritten on the source VDisk has been copied to the target VDisk (we had chosen not to enable the background copy process). What this means is that the mapping will still exist, but whatever data was written to the target VDisk will be invalidated.

Also, any mapping which depends on this mapping would be affected. When we use the Forced Stop, no further cleaning occurs and dependent mappings will as well be stopped. We do not have any dependent mappings at that point in time, so we press the Stop button (we do not need to use the Forced Stop button). Now the status of the mapping is Stopping, as seen in Figure 6-16 on page 127, and shortly thereafter in the state Stopped.

Figure 6-16 FlashCopy Mapping, state: Stopping

6.1.5 Creating a FlashCopy Consistency Group

Our application has related data spanning multiple VDisks. To be able to produce a consistent snapshot of the data we put them into a Consistency Group to ensure not only



consistency of the data within one VDisk, but also across data spanning all the involved VDisks. First we choose FlashCopy Consistency Groups on the My Work pane as in Figure 6-17 on page 128.

Figure 6-17 Viewing FlashCopy Consistency Groups, no groups exist

There are no Consistency Groups set up yet. To create one we choose Create a Consistency Group from the pull down menu. The next screen, shown in Figure 6-18 on page 128, lets us choose the name of the Consistency Group and what FlashCopy mappings we want to be members in it.

Figure 6-18 Creating FlashCopy Consistency Groups - including the one existing mapping

We want to include the mapping FlashCopy fcm-001 as the first member in the Consistency Group, therefore we select it and press OK. If we had created some more mappings, they would be displayed here and could be included in the Consistency Group at once.



The next screen, shown in Figure 6-19 on page 129, displays the result of the creation, which in our case was a success. If, for instance, the mapping put into the Consistency Group was in the Copying state, there would be an error message indicating this. Nonetheless, an empty Consistency Group would have been created.

Figure 6-19 Creating FlashCopy Consistency Groups - Success

Closing this screen brings us back to the Viewing FlashCopy Consistency Group screen seen in Figure 6-20 on page 129.

Figure 6-20 Viewing FlashCopy Consistency Groups - one group created

Clicking on the Consistency Group Name shows us the properties of the Consistency Group. The first panel, General, shows just the name, ID and the state of the Consistency Group, as shown in Figure 6-21 on page 130.



Figure 6-21 View FlashCopy Consistency Group Details for fcg-001 - General

The Consistency Group state is Stopped, since its member, the FlashCopy Mapping fcm-001 is in the Stopped state. The Relationship panel shows the details for the relationship as shown in Figure 6-22 on page 130.

Figure 6-22 View FlashCopy Consistency Group Details for fcg-001 - Relationships

6.1.6 Creating a FlashCopy mapping and adding it to the Consistency Group

We can create a new mapping without choosing a Consistency Group, thus associating it with Consistency Group 0, as we did in 6.1.1, “Creating a FlashCopy mapping” on page 118. We can then later place it into a Consistency Group from within Viewing FlashCopy Mappings screen with using Modify a Mapping from the pull down menu.

If we already know which Consistency Group the mapping should be a member of, we can put the mapping into the Consistency Group from within the Create a FlashCopy Mapping wizard in one step, shown in Figure 6-23 on page 131.



Figure 6-23 Create a FlashCopy Mapping - Set Properties - Include in existing Consistency Group

The next screen lets us again filter the VDisks to be displayed as source VDisk candidates. This time we filter using the Managed Disk Group name (the backend storage system which holds the production VDisks is the DS4500), shown in Figure 6-24 on page 131.

Figure 6-24 Create a FlashCopy Mapping - Filtering Source VDisk Candidates - filter by mdgroup



Now we choose the source VDisk for the mapping from the filtered list, shown in Figure 6-25 on page 132.

Figure 6-25 Create a FlashCopy Mapping - Select the Source VDisk

We now choose the target VDisk for the new mapping shown in Figure 6-26.

Figure 6-26 Create a FlashCopy Mapping - Select the Target VDisk

The last step before the mapping is created is to verify the settings, as shown in Figure 6-27 on page 133.



Figure 6-27 Create a FlashCopy Mapping - Verify FlashCopy Mapping

Now we return to the Consistency Group screen, shown in Figure 6-28, and we see that both mappings are now members of the Consistency Group fcg-001.

Figure 6-28 Viewing FlashCopy Mappings - two mappings are members of one Consistency Group

The state of both mappings is Idle_or_Copied and therefore also the state of the Consistency Group, shown in Figure 6-29 on page 134.



Figure 6-29 Viewing FlashCopy Consistency Groups - fcg-001, state Idle or Copied

When clicking on the Name of the Consistency Group, we see the details of the Consistency Group. Clicking on Relationships now, as shown in Figure 6-30, we now see both FlashCopy Mappings as members of the Consistency Group.

Figure 6-30 View FlashCopy Consistency Group Relationship Details for fcg-001

6.1.7 Preparing a FlashCopy Consistency Group

The FlashCopy Mapping states are described in 5.4.3, “FlashCopy consistency group states” on page 109. There we can see that a Consistency Group (as well as a FlashCopy Mapping) needs to be prepared before it is started.

We can do this manually before we start the Consistency Group. Even if we do not prepare the Consistency Group (and therefore the mappings within), it will be prepared when choosing Start a Consistency Group. Doing it manually has some advantages.

One is that, depending on the amount of data associated with the VDisks to be copied still in the cache and the load on the system, it can take some time to prepare a mapping.



A command needs to be completed within a certain period of time. The prepare command is asynchronous and if it takes more than two minutes for the maps to get to the "prepared" state that will not cause a problem. The start command is synchronous and must complete within two minutes.

In a case where the start command is being prevented from being executed within two minutes, the event flush failed takes the mapping into the Stopped state. As it is in the Stopped state, it can be prepared again. The advantage of preparing the mappings first is that in the case just described, the mappings are already prepared, thus the time does not run out for the completion of the Start process.

Another advantage is that when preparing it first, we can specify the point-in-time to start the Consistency Group or the mapping, whereas with an unprepared Consistency Group, the (variable) time to prepare the Consistency Group decides the exact point-in-time it is started.

From the Consistency Group screen we choose Prepare a Consistency Group, as shown in Figure 6-31 on page 135.

Figure 6-31 Prepare a Consistency Group

A Consistency Group can only be put into the Preparing state, shown in Figure 6-32 when it is either in the Idle_or_Copied or Stopped state.



Figure 6-32 Prepare a Consistency Group - state: Preparing

After a refresh, the state changes to Prepared, as shown in Figure 6-33 on page 136.

Figure 6-33 Prepare a Consistency Group - state: Prepared

6.1.8 Starting a FlashCopy Consistency Group

To start our Consistency Group without preparing it first, we use the pull-down menu from the Viewing FlashCopy Consistency Group screen, as shown in Figure 6-34 on page 137.



Figure 6-34 Viewing FlashCopy Consistency Groups - Start a Consistency Group

As with starting the standalone mapping, when starting the Consistency Group, the next screen asks us to prepare the Consistency Group, as shown in Figure 6-35. This only applies if we did not prepare the Consistency Group before; if it has been done before the checkbox will be greyed out.

Figure 6-35 Starting FlashCopy Consistency Group fcg-001 - preparing the mappings within the group

There is a checkbox to prepare the FlashCopy Consistency group (which is checked by default, but can be unchecked).

As stated before, a Consistency Group or a FlashCopy Mapping always needs to be prepared before being started. The checkbox serves no other purpose than telling us that the Consistency Group will be prepared before it is being started. If we uncheck it, the start of the Consistency Group will fail as shown in Figure 6-36.



Figure 6-36 Starting FlashCopy Consistency Group fcg-001 - error when using the checkbox

Clicking the OK button, with the checkbox left checked, The Consistency Group first enters the Preparing, then Prepared and finally Copying state, in one uninterrupted sequence, as shown in Figure 6-37 on page 138.

Figure 6-37 Viewing FlashCopy Consistency Groups, fcg-001 in state: Copying

Looking at the details of the Consistency Group fcg-001 we see that both mappings, which are members of the Consistency Group, are in the Copying state, shown in Figure 6-38.



Figure 6-38 Viewing FlashCopy Consistency Group Relationship Details for fcg-01, state: Copying

6.1.9 Stopping a Consistency Group

Because of the NOCOPY option we have chosen, the target VDisks will be a valid snapshot of the source VDisks as long as the mappings exist. All we can do to manipulate the Consistency Group now is to stop it, which will end the relationship and which renders all data on the target VDisk invalid. To stop the Consistency Group, we choose Stop a Consistency Group from the pull-down menu, seen in Figure 6-39.

Figure 6-39 Stop a Consistency Group

Figure 6-40 on page 140 shows the confirmation screen where we can just stop the Consistency Group or chose to force the Consistency Group to be stopped in case there are mappings within which have dependencies to other mappings outside the Consistency Group to be stopped.



Figure 6-40 Stopping FlashCopy Consistency Group fcg-001

Before reaching the Stopped state, the Consistency Group enters the Stopping state, as shown in Figure 6-41.

Figure 6-41 Stopping a Consistency Group, state: Stopping

6.1.10 Renaming a Consistency Group

The only property of a Consistency Group to be changed from within the Consistency Group screen is to rename it. Again, we use the pull-down menu to chose the action as shown in Figure 6-42 on page 141.



Figure 6-42 Rename a Consistency Group - pull-down menu

The next screen shows is where we can change the name of the Consistency Group, this can be done in the Copying state, as shown in Figure 6-43 on page 141.

Figure 6-43 Renaming FlashCopy Consistency Group fcg-001

6.1.11 Issuing a command to a mapping within a Consistency Group

From within the FlashCopy Mapping screen, we can stop a FlashCopy Mapping and if this mapping is a member of a Consistency Group, we stop the whole Consistency Group while stopping the mapping.

We will do this with the mapping FlashCopy fcm-001. This also puts the other mappings, which are members of that Consistency Group in the Stopping state, shown in Figure 6-44 on page 142, and then into the Stopped state.



Figure 6-44 Viewing FlashCopy Mappings, one Mapping in state: Stopped, the other in state: Stopping

The same applies to preparing and starting a FlashCopy Mapping. When we prepare or start a mapping which is a member of a Consistency Group, we get an additional confirmation screen which tells us that since the FlashCopy Mapping is a member of a Consistency Group, we will issue the command to all mappings which are members of the Consistency Group, as shown in Figure 6-45 on page 142.

Figure 6-45 Starting FlashCopy Mappings Confirmation

6.2 Example 2: Multiple Target FlashCopy using the GUI

Here we use FlashCopy where one VDisk is acting as a source VDisk and it can be in multiple mappings with different VDisks, acting as target VDisk.

Goal: The goal is to create multiple consistent copies of the same data. If we need copies of VDisks for different purposes (and with different properties), it could happen that we require these copies of the source at the same time.

Solution: The solution is using MTFC to create multiple snapshots of one source. For instance, with MTFC we can create FlashCopies for our daily backup and others for application testing using the same sources.

The snapshots for backup could be set up as described in 6.1, “Example 1: basic FlashCopy using the GUI” on page 118. The snapshots to be used for application testing we set up with different properties, for instance, we would like to have the VDisks copied fully using the background copy process to establish real independent clones.



6.2.1 Creating FlashCopy Mappings and a Consistency Group

We already have the FlashCopy Mappings established for our backup purposes, shown in Figure 6-46.

Figure 6-46 Viewing FlashCopy Mappings - two mappings for backup in Consistency Group fcg-001

Clicking the name of the mapping gives us the properties of the mapping as shown in Figure 6-47.



Figure 6-47 View FlashCopy Details for fcm-001

Now we are going to establish two new mappings which use the same source VDisk but have a different target VDisk, shown in Figure 6-48 on page 144.

Figure 6-48 Viewing FlashCopy Mappings - two more Mappings using the same sources



The properties of the mapping FlashCopy fcm-test-001 are shown in Figure 6-49.

Figure 6-49 View FlashCopy Mapping Details for FlashCopy fcm-test-001

The background copy priority is 50% for these mappings. This tells the SVC to perform a background copy of the data of the source VDisk to the target VDisk with 2 MBps.

As we do not need the mapping after the clone of the source VDisks is established, the option to delete the mapping when the background copy has finished is enabled.

The mappings are not yet in a Consistency Group. As we want the timing of the FlashCopy to be independent from the Consistency Group we use for backup, we put the mappings in a new Consistency Group. We now create the Consistency Group and while doing so, put both new FlashCopy Mappings into the Consistency Group, as shown in Figure 6-50 on page 146.



Figure 6-50 Creating FlashCopy Consistency Groups - including both Mappings for test data

To check if all settings are correct, we look up the FlashCopy relationship details of the Consistency Group, as shown in Figure 6-51.

Figure 6-51 FlashCopy Consistency Group Details for fcg-test-001

6.2.2 Starting a Consistency Group in an MTFC tree

After we set up the Consistency Group, we can prepare and start it. As we can see in Figure 6-52 on page 147, the mappings in Consistency Group fcg-test-001 are in the Copying state while the mappings in fcg-001 are in the Idle_Copyied state.



Figure 6-52 Viewing FlashCopy Mappings - mappings in fcg-test-001 in state: Copying

Now we start (and prepare in one step) the FlashCopy Mapping fcg-001 again for our backup purposes.

Both of our Consistency Groups which have mappings which share the same source VDisks, are in the Copying state. Selecting a mapping in fcg-001 and choosing View FC Mappings in Dependency Order from the pull-down menu, shown in Figure 6-53 on page 147, lets us obtain information about the dependencies the mappings have.

Figure 6-53 View FC Mappings in Dependency Order

The list of mappings which are dependent on the mapping fcm-001 are shown in Figure 6-54.



Figure 6-54 Viewing FlashCopy Mappings in Dependency Order for mapping FlashCopy fcm-001

The only mapping which depends on the chosen mapping is fcm-test-001. The mapping fcm-001 itself is not shown in the list. The mapping fcm-test-001 depends on the mapping fcm-001 because fcm-001 is newer (started later) than fcm-test-001.

After the SVC finished copying the data of the source VDisks to the target VDisks for the mapping used to establish a clone of the production VDisks for test purposes, it deleted the related mapping as we specified.

Now, we only have 2 mappings existing which are the ones we use for our backups,as shown in Figure 6-55.

Figure 6-55 Viewing FlashCopy Mappings - only the mappings in Consistency Group fcg-001 left

6.3 Example 3: Cascaded and Incremental FC using the GUI

In this example we implement FlashCopy where a VDisk can be a target VDisk in one mapping, and a Source VDisk in another mapping.

Also, we implement a FlashCopy which can be incrementally refreshed. This will be implemented using Consistency Groups.



Goal 1: The first goal is to take a snapshot of a VDisk, which is already the target VDisk in a mapping. We create and start a FlashCopy Mapping and, additionally, want to establish a FlashCopy where the target VDisk of the first FlashCopy Mapping is the source of the second FlashCopy Mapping.

The first copy we set up with background copy to get a fully copied clone of a production VDisk. This first copy is to be used for application testing. The application working with the first copy might change the data residing on the target VDisk of the first FlashCopy Mapping. We now want to have a copy of the target VDisk of the first FlashCopy Mapping. With a FlashCopy function where a VDisk can be a member of only one FlashCopy Mapping, we would have to wait until the first copy has finished to fully copy the data to the target VDisk. Then we would have to delete the mapping (or have the autodelete option enabled). This can be a time-consuming procedure.

Solution1 : We use CFC to create a FlashCopy where a source VDisk of the second mapping is still the target VDisk in the first FlashCopy Mapping. This enables us to create the second copy before the first FlashCopy Mapping has finished copying all the data.

Goal 2: The second goal is to refresh a FlashCopy target. We would like to have the first copy (the copy of the production VDisk) to be re-established within a short period of time and without much impact on the production VDisk.

Solution 2: To re-establish the first copy in a short period of time and without much impact on the production volume, we use IFC for the first FlashCopy Mapping.

6.3.1 Creating FlashCopy Consistency Groups

First we create two FlashCopy Consistency Groups, named in such a way so that we know that the second Consistency Group is supposed to hold mappings for the ’copy of the copy’, as shown in Figure 6-56.

Figure 6-56 Viewing FlashCopy Consistency Groups - two empty Consistency Groups created

6.3.2 Create Incremental FlashCopy Mappings

We put the mapping to be created into the first Consistency Group. Also, we set it up to be incremental, which is shown in Figure 6-57 on page 150.



Figure 6-57 Create FlashCopy Mapping - Incremental

We now associate VDisks to be the source VDisk and the target VDisk with the mapping and finish the wizard. The properties of the mapping we created are shown in Figure 6-58 on page 151.



Figure 6-58 View FlashCopy Mapping Details for FlashCopy fcm-test-001

We set up the second mapping, for the second production VDisk the same way we did with the first. Both mappings are in the Idle_or_Copied state and are shown in Figure 6-59 on page 151.

Figure 6-59 Viewing FlashCopy Mappings - two mappings in the first Consistency Group



We can now switch to the Consistency Group screen and start the first Consistency Group, as shown in Figure 6-60 on page 152.

Figure 6-60 Start a Consistency Group

This puts the Consistency Group in the Copying state, as shown in Figure 6-61.

Figure 6-61 View FlashCopy Consistency Groups

Now the target VDisks of the first mapping are ready to be used for our application testing. the application testing will need to be temporarily stopped (and any host caches flushed) before the cascaded mappings are started. While this is ongoing, we create the next two mappings, where the target VDisks of the now Copying first mapping are to be the source VDisks.

We put the mappings in another Consistency Group and specify them to be incremental as well, as shown in the Verify FlashCopy Mapping screen at the final stage of creating the mapping, as shown in Figure 6-62 on page 153.



Figure 6-62 Verify FlashCopy Mapping

Now we have set up four mappings in two Consistency Groups, shown in Figure 6-63.

Figure 6-63 Viewing FlashCopy Mappings - 1 new Consistency Group in state: Idle_or_Copied

While the first Consistency Group and with it the mappings associated with it are still in the Copying state, we start the second Consistency Group, shown in Figure 6-64 on page 154.



Figure 6-64 Start a Consistency Group

We now have four mappings in two Consistency Groups in a Copying state, where VDisks act as both, as source VDisks and target VDisks, shown in Figure 6-65.

Figure 6-65 View FlashCopy Mappings - VDisks in the role of Source VDisk and Target VDisk

The mapping FlashCopy fcm-test-003 depends upon FlashCopy fcm-test-001, shown in Figure 6-66 on page 155. The same applies to the other two mappings.



Figure 6-66 Viewing FlashCopy Mappings in Dependency Order for mapping FlashCopy fcm-test-001

After some time, the progress indicates that the copy process for the first FlashCopy Mapping is almost complete, shown in Figure 6-67.

Figure 6-67 Viewing FlashCopy Mappings - first mapping copy process almost complete

We can see the progress using the Manage Progress screens, shown in Figure 6-68 on page 156, where we see that the first mapping successfully finished copying all the data to the target VDisk.



Figure 6-68 Viewing FlashCopy Progress

After both copies are copied fully, the state changes back to Idle_or_Copied for both Consistency Groups, shown in Figure 6-69 on page 156.

Figure 6-69 Viewing FlashCopy Consistency Groups - both in state Idle_or_Copied

We now restart the first Consistency Group, which has the production VDisks as source VDisks, shown in Figure 6-70 on page 157.



Figure 6-70 (Re-)Start a Consistency Group

The screenshot in Figure 6-71 was taken soon after we started the FlashCopy Mapping. Since the mapping is incremental, it just had to copy the grains which were changed, which was about 200 MB, instead of the whole 10 GB VDisk.

Figure 6-71 Viewing FlashCopy Mappings - mappings in fcg-test-001 refreshed

6.4 Example 4: FlashCopy of Metro Mirror VDisks using the CLI

Here we implement FlashCopy where the source VDisks are the secondary VDisks in a remote copy relationship.

Goal: The goal is to create a full consistent copy of production data for a given point-in-time at a remote location. We have a Remote Copy Relationship set up to copy VDisks to a second site using intercluster Metro Mirror, for disaster tolerance. We want to have a consistent



backup of our production data taken at the remote site. Also, we want to have a full copy (a clone) of our production VDisks on a daily basis, residing at the second location. This copy serves the purpose of providing a consistent point-in-time copy of the production data in case the data of both the primary and secondary of the Metro Mirror relationship get invalidated (for example, due to a logical error which affects both the primary and the secondary due to them being in a synchronized copy relationship). This clone is a building block for our Business Continuity strategy.

Solution: The solution is to setup FlashCopy where the source VDisks for the mappings are the secondary VDisks of the one MetroMirror Relationship. The target VDisks are then used to take a backup from. We establish the clone of the Metro Mirror secondary using the background copy process and we use IFC to refresh this clone once created to minimize load on the system, and to establish the clone quickly. To have the clone established in a short time decreases the time where no full clone of the VDisks is available, thus enhancing our Recovery Time Objectives.

6.4.1 Checking VDisks and Metro Mirror relationship to be used

To check the VDisks to be used for our FlashCopy Mapping we use the command svcinfo lsvdisk -delim :, shown in Example 6-1. To eliminate the white space between the output columns we use the -delim switch.

Example 6-1 svcinfo lsvdisk -delim : - VDisks to be used

IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim :id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type:FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count0:vd-prod-l2-0002:0:io_grp0:online:1:DS4700:25.0GB:striped:::0:MetroMirror Relationship-002:6005076801A100E90800000000000003:01:vd-prod-l2-0001:0:io_grp0:online:1:DS4700:25.0GB:striped:::1:MetroMirror Relationship-001:6005076801A100E90800000000000002:02:vd-back-l2-0001:0:io_grp0:online:1:DS4700:25.0GB:striped:::::6005076801A100E90800000000000004:03:vd-back-l2-0002:0:io_grp0:online:1:DS4700:25.0GB:striped:::::6005076801A100E90800000000000005:0IBM_2145:ITSOCL3:admin>

The VDisks which have the characters prod in the name we are going to use them as the source VDisk for the FlashCopy Mapping and the VDisks with the characters back in the name we will use as the target VDisks.

As an example, we display the properties of the VDisk with the id 2, using the command svcinfo lsvdisk -delim : 2, shown in Example 6-2.

Example 6-2 svcinfo lsvdisk -delim : 2 - details of VDisk 2

IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim : 2id:2name:vd-back-l2-0001IO_group_id:0IO_group_name:io_grp0

Note: Even though Global Mirror is “asynchronous”, this use case applies to GM as well, because the data gets copied over to the secondary VDisk as soon as possible and thus is object to get invalidated by logical errors.



status:onlinemdisk_grp_id:1mdisk_grp_name:DS4700capacity:25.0GBtype:stripedformatted:nomdisk_id:mdisk_name:FC_id:FC_name:RC_id:RC_name:vdisk_UID:6005076801A100E90800000000000004throttling:0preferred_node_id:2fast_write_state:emptycache:readwriteudid:0fc_map_count:0IBM_2145:ITSOCL3:admin>

To display the one MetroMirror Relationship we use the command svcinfo lsrcconsistgrp, as shown in Example 6-3, to display it.

Example 6-3 svcinfo lsrcconsistgrp -delim - existing remote copy consistency groups

IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp -delim :id:name:master_cluster_id:master_cluster_name:aux_cluster_id:aux_cluster_name:primary:state:relationship_count:copy_type255:mmg-001:000002006B603A38:ITSOCL2:0000020068403A42:ITSOCL3:master:consistent_synchronized:2:metroIBM_2145:ITSOCL3:admin>

As we can see, only one relationship exists. To show the properties of only this relationship, we specify the id with the command, shown in Example 6-4.

Example 6-4 svcinfo lsrcconsistgrp -delim : 255 - properties of the Consistency Group with id 255

IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp -delim : 255id:255name:mmg-001master_cluster_id:000002006B603A38master_cluster_name:ITSOCL2aux_cluster_id:0000020068403A42aux_cluster_name:ITSOCL3primary:masterstate:consistent_synchronizedrelationship_count:2freeze_time:status:onlinesync:copy_type:metroRC_rel_id:0RC_rel_name:MetroMirror Relationship-002RC_rel_id:1RC_rel_name:MetroMirror Relationship-001



IBM_2145:ITSOCL3:admin>

All important information is shown to verify that we have successfully set up the remote copy consistency group, which is associated with two Metro Mirror relationships.

We can also verify the properties of the two Metro Mirror relationships, with one of the two shown in Example 6-5, using the command svcinfo lsrcrelationship -delim : 0.

Example 6-5 .svcinfo lsrcrelationship -delim : 0 - properties of the Relationship with id 0

IBM_2145:ITSOCL3:admin>svcinfo lsrcrelationship -delim : 0id:0name:MetroMirror Relationship-002master_cluster_id:000002006B603A38master_cluster_name:ITSOCL2master_vdisk_id:1master_vdisk_name:vd-prod-l1-0002aux_cluster_id:0000020068403A42aux_cluster_name:ITSOCL3aux_vdisk_id:0aux_vdisk_name:vd-prod-l2-0002primary:masterconsistency_group_id:255consistency_group_name:mmg-001state:consistent_synchronizedbg_copy_priority:50progress:freeze_time:status:onlinesync:copy_type:metroIBM_2145:ITSOCL3:admin>

The second relationship we use VDisks from for our FlashCopy Mapping is shown in Example 6-6.

Example 6-6 svcinfo lsrcrelationship -delim : 0 - properties of the Relationship with id 1

IBM_2145:ITSOCL3:admin>svcinfo lsrcrelationship -delim : 1id:1name:MetroMirror Relationship-001master_cluster_id:000002006B603A38master_cluster_name:ITSOCL2master_vdisk_id:0master_vdisk_name:vd-prod-l1-0001aux_cluster_id:0000020068403A42aux_cluster_name:ITSOCL3aux_vdisk_id:1aux_vdisk_name:vd-prod-l2-0001primary:masterconsistency_group_id:255consistency_group_name:mmg-001state:consistent_synchronizedbg_copy_priority:50progress:freeze_time:



status:onlinesync:copy_type:metroIBM_2145:ITSOCL3:admin>

The VDisks of our MetroMirror Relationship in location 1 (Production Location) are named:

� vd-prod-l1-0001� vd-prod-l1-0002

In location 2 (Business Continuity Location) they are named:

� vd-prod-l2-0001� vd-prod-l2-0002

The two VDisks which serve as the secondary VDisks for the MetroMirror Relationship in location 2 are the ones we use as the source for the FlashCopy Mapping.

6.4.2 Create FlashCopy mappings

We set up the FlashCopy mappings and Consistency Group in the following order.

1. Create FlashCopy mappings2. Create a FlashCopy Consistency Group3. Add FlashCopy mappings to the Consistency Group

Mappings are to be created using the command svctask mkfcmap, and the syntax is shown in Example 6-7.

Example 6-7 svctask mkfcmap - syntax

>>- svctask -- -- mkfcmap -- ----------------------------------->

>-- -source --+- src_vdisk_id ---+-- ---------------------------> '- src_vdisk_name -'

>-- -target --+- target_vdisk_id ---+-- ------------------------> '- target_vdisk_name -'

>--+-------------------------+-- -------------------------------> '- -name -- new_name_arg -'


>--+------------------------+-- --+---------------+-------------> '- -copyrate -- percent -' '- -autodelete -'

>--+-------------------------+--+----------------+--------------> '- -grainsize --+- 64 --+-' '- -incremental -' '- 256 -'

>--+-----------------------------+------------------------------>

Note: Another method would be to create the Consistency Group first and then create the mappings and add them to the Consistency Group in the same step.



'- -cleanrate ---- percent ---'

>--+------------------------------+---------------------------->< '- -iogrp --+- iogroup_name -+-' '- iogroup_id --'

We do not use all the optional parameters, for instance we leave the Grain Size to default at 256KB. In our example we create the first mapping as shown in Example 6-8.

Example 6-8 svctask mkfcmap - create first mapping

IBM_2145:ITSOCL3:admin>svctask mkfcmap -source vd-prod-l2-0001 -target vd-back-l2-0001 -name fcm-back-001 -copyrate 70 -incrementalFlashCopy Mapping, id [0], successfully createdIBM_2145:ITSOCL3:admin>

We then create the second mapping, shown in Example 6-9.

Example 6-9 svctask mkfcmap - create second mapping

IBM_2145:ITSOCL3:admin>svctask mkfcmap -source vd-prod-l2-0002 -target vd-back-l2-0002 -name fcm-back-002 -copyrate 70FlashCopy Mapping, id [1], successfully createdIBM_2145:ITSOCL3:admin>

We review the creation of both mappings using the command svcinfo lsfcmap -delim :, shown in Example 6-10.

Example 6-10 svcinfo lsfcmap -delim : - both mappings

IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim :id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name:group_id:group_name:status:progress:copy_rate:clean_progress:incremental0:fcm-back-001:1:vd-prod-l2-0001:2:vd-back-l2-0001:::idle_or_copied:0:70:100:on1:fcm-back-002:0:vd-prod-l2-0002:3:vd-back-l2-0002:::idle_or_copied:0:70:100:offIBM_2145:ITSOCL3:admin>

Reviewing the mappings we realize that we forgot the -incremental parameter in our second mapping. We cannot modify this mapping to be incremental (modifying a mapping would be done using the command svctask chfcmap), so we have to delete the mapping and re-create it. We delete the mapping using the command svctask rmfcmap, shown in Example 6-11.

Example 6-11 svctask refcmap - syntax

>>- svctask -- -- rmfcmap -- --+----------+-- ------------------> '- -force -'

>--+- fc_map_id ---+------------------------------------------->< '- fc_map_name -'

After we recreated the second mapping, this time using the flag -incremental, we view the properties of that mapping using the command svcinfo lsfcmap 1, shown in Example 6-12.

Example 6-12 svcinfo lsfcmap 1 - properties of the mapping with id 1

IBM_2145:ITSOCL3:admin>svcinfo lsfcmap 1id 1



name fcm-back-002source_vdisk_id 0source_vdisk_name vd-prod-l2-0002target_vdisk_id 3target_vdisk_name vd-back-l2-0002group_idgroup_namestatus idle_or_copiedprogress 0copy_rate 70start_timedependent_mappings 0autodelete offclean_progress 100clean_rate 50incremental ondifference 100grain_size 256IO_group_id 0IO_group_name io_grp0IBM_2145:ITSOCL3:admin>

Using svcinfo lsvdisk again, shown in Example 6-13, we see that all four VDisks are now part of FlashCopy Mappings.

Example 6-13 svcinfo lsvdisk -delim : - all four VDisks are now part of FlashCopy mappings

IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim :id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type:FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count0:vd-prod-l2-0002:0:io_grp0:online:1:DS4700:25.0GB:striped:1:fcm-back-002:0:MetroMirror Relationship-002:6005076801A100E90800000000000003:11:vd-prod-l2-0001:0:io_grp0:online:1:DS4700:25.0GB:striped:0:fcm-back-001:1:MetroMirror Relationship-001:6005076801A100E90800000000000002:12:vd-back-l2-0001:0:io_grp0:online:1:DS4700:25.0GB:striped:0:fcm-back-001:::6005076801A100E90800000000000004:13:vd-back-l2-0002:0:io_grp0:online:1:DS4700:25.0GB:striped:1:fcm-back-002:::6005076801A100E90800000000000005:1IBM_2145:ITSOCL3:admin>

6.4.3 Creating a FlashCopy Consistency Group

Now we go on and create a FlashCopy Consistency Group to put the mappings into. The command to be used is svctask mkfcconsistgrp, and the syntax is shown in Example 6-14.

Example 6-14 svctask mkfcconsistgrp - syntax

>>- svctask -- -- mkfcconsistgrp -- ---------------------------->

>--+-------------------------------+--------------------------->< '- -name -- consist_group_name -'

The actual command we used to create our Consistency Group is shown in Example 6-15 on page 164.



Example 6-15 svctask mkfcconsistgrp - Consistency Group successfully created

IBM_2145:ITSOCL3:admin>svctask mkfcconsistgrp -name fcg-back-001FlashCopy Consistency Group, id [1], successfully createdIBM_2145:ITSOCL3:admin>

Example 6-16 shows the syntax of the command svcinfo lsfcconsistgrp, which can be used to display the Consistency Groups which have been created.

Example 6-16 svcinfo lsfcconsistgrp - syntax

>>- svcinfo -- -- lsfcconsistgrp -- ---------------------------->

>--+-----------------------------------+-- --+----------+-- ----> '- -filtervalue -- attribute=value -' '- -nohdr -'

>--+-----------------------+-- --+---------------+-- -----------> '- -delim -- delimiter -' +- object_id ---+ '- object_name -'

>--+-----------------+----------------------------------------->< '- -filtervalue? -'

We use the command now to verify the creation of the Consistency Group, as shown in Example 6-17.

Example 6-17 svcinfo lsfcconsistgrp - empty Consistency Group

IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrpid name status1 fcg-back-001 emptyIBM_2145:ITSOCL3:admin>

For now, what we have is two FlashCopy mappings and one empty FlashCopy Consistency Group.

6.4.4 Adding FlashCopy mappings to the Consistency Group

As the final step before we are able to start the FlashCopy, we include the two mappings into the Consistency Group we just created. This is done using the command svctask chfcmap, and the syntax is shown in Example 6-18.

Example 6-18 svctask chfcmap - syntax

>>- svctask -- -- chfcmap -- --+-------------------------+-- ---> '- -name -- new_name_arg -'

>--+----------+-- ----------------------------------------------> '- -force -'


>--+------------------------+-- --+-------------------------+---> '- -copyrate -- percent-' '- -autodelete --+-on--+-'



'-off-'

>--+-----------------------------+--+- fc_map_id ---+---------->< '- -cleanrate ---- percent ---' '- fc_map_name -'

The command we used to put the mappings into the Consistency Group is shown in Example 6-19.

Example 6-19 svctask chfcmap - associating both mappings with the same Consistency Group

IBM_2145:ITSOCL3:admin>svctask chfcmap -consistgrp fcg-back-001 fcm-back-001IBM_2145:ITSOCL3:admin>svctask chfcmap -consistgrp fcg-back-001 fcm-back-002IBM_2145:ITSOCL3:admin>

To view the Consistency Group we use the command svcinfo lsfcconsistgrp again, shown in Example 6-20 on page 165.

Example 6-20 svcinfo lsfcconsistgrp - Consistency Group in state: Idle_or_Copied

IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrpid name status1 fcg-back-001 idle_or_copiedIBM_2145:ITSOCL3:admin>

The Consistency Group state changed from empty to idle_or_copied. The properties, including the FlashCopy mappings which are now associated with the Consistency group we get when specifying the id of the Group with the command, as shown in Example 6-21.

Example 6-21 svcinfo lsfcconsistgrp - properties of the Consistency Group with id 1

IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrp 1id 1name fcg-back-001status idle_or_copiedFC_mapping_id 0FC_mapping_name fcm-back-001FC_mapping_id 1FC_mapping_name fcm-back-002IBM_2145:ITSOCL3:admin>

Example 6-22 shows the two FlashCopy mappings, using the command svcinfo lsfcmap, which shows the fact that both mappings are now members of the same Consistency Group.

Example 6-22 svcinfo lsfcmap - both mappings associated with the same Consistency Group

IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim :id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name:group_id:group_name:status:progress:copy_rate:clean_progress:incremental0:fcm-back-001:1:vd-prod-l2-0001:2:vd-back-l2-0001:1:fcg-back-001:idle_or_copied:0:70:100:on1:fcm-back-002:0:vd-prod-l2-0002:3:vd-back-l2-0002:1:fcg-back-001:idle_or_copied:0:70:100:onIBM_2145:ITSOCL3:admin>



6.4.5 Preparing a FlashCopy mapping

We chose to prepare the FlashCopy Consistency Group (and thus the FlashCopy mappings associated with it) prior to issuing the start of the Group. This is done using the command svctask prestartfcconsistgrp, and the syntax is shown in Example 6-23

Example 6-23 .svctask prestartfcconsistgrp - syntax

>>- svctask -- -- prestartfcconsistgrp -- ---------------------->

>--+- fc_consist_group_id ---+--------------------------------->< '- fc_consist_group_name -'

We prepare the Group and immediately afterwards we issue the command to see that the state of the Consistency Group is Prepared, which is shown in Example 6-24.

Example 6-24 svctask prestartfcconsistgrp, svcinfo lsfcconsistgrp - prepare and view Group

IBM_2145:ITSOCL3:admin>svctask prestartfcconsistgrp fcg-back-001IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrpid name status1 fcg-back-001 preparedIBM_2145:ITSOCL3:admin>

This time we have chosen the Consistency Group name to specify which Group to start, not the id. We can also use the command again, to see the state of the mappings, shown in Example 6-25.

Example 6-25 svcinfo lsfcmap - both mappings in state: Prepared

IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim :id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name:group_id:group_name:status:progress:copy_rate:clean_progress:incremental0:fcm-back-001:1:vd-prod-l2-0001:2:vd-back-l2-0001:1:fcg-back-001:prepared:0:70:100:on1:fcm-back-002:0:vd-prod-l2-0002:3:vd-back-l2-0002:1:fcg-back-001:prepared:0:70:100:onIBM_2145:ITSOCL3:admin>

6.4.6 Starting a FlashCopy Consistency Group

Example 6-26 shows the syntax of the command svctask startfcconsistgrp.

Example 6-26 svctask startfcconsistgrp - syntax

>>- svctask -- -- startfcconsistgrp -- --+---------+-- ---------> '- -prep -'


We could prepare the Consistency Group first using the -prep option, but since we have already prepared the Group we do not need to do it now.

Example 6-27 on page 167 shows how we start the Consistency Group (using the id to specify it) and immediately after we view the status of the Consistency Group.



Example 6-27 svctask startfcconsistgrp, svcinfo lsfcconsistgrp - start and view Group

IBM_2145:ITSOCL3:admin>svctask startfcconsistgrp 1IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrpid name status1 fcg-back-001 copyingIBM_2145:ITSOCL3:admin>

The state of the Consistency Group is now Copying, as expected. In Example 6-28 we see the state of both of the mappings is Copying as well.

Example 6-28 svcinfo lsfcmap - both mappings in state: Copying

IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim :id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name:group_id:group_name:status:progress:copy_rate:clean_progress:incremental0:fcm-back-001:1:vd-prod-l2-0001:2:vd-back-l2-0001:1:fcg-back-001:copying:34:70:100:on1:fcm-back-002:0:vd-prod-l2-0002:3:vd-back-l2-0002:1:fcg-back-001:copying:34:70:100:onIBM_2145:ITSOCL3:admin>

Example 6-29 shows the details of the FlashCopy mapping with the id 0.

Example 6-29 svcinfo lsfcmap -delim : 0

IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim : 0id:0name:fcm-back-001source_vdisk_id:1source_vdisk_name:vd-prod-l2-0001target_vdisk_id:2target_vdisk_name:vd-back-l2-0001group_id:1group_name:fcg-back-001status:copyingprogress:51copy_rate:70start_time:071107011320dependent_mappings:0autodelete:offclean_progress:100clean_rate:50incremental:ondifference:48grain_size:256IO_group_id:0IO_group_name:io_grp0IBM_2145:ITSOCL3:admin>

We see that the state of the mapping is Copying and that the progress is 51%, using a background copy priority of 70%.

The last output we look at are the details of one of the VDisks involved, shown in Example 6-30 on page 168.



Example 6-30 svcinfo lsvdisk -delim : 1

IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim : 1id:1name:vd-prod-l2-0001IO_group_id:0IO_group_name:io_grp0status:onlinemdisk_grp_id:1mdisk_grp_name:DS4700capacity:25.0GBtype:stripedformatted:nomdisk_id:mdisk_name:FC_id:0FC_name:fcm-back-001RC_id:1RC_name:MetroMirror Relationship-001vdisk_UID:6005076801A100E90800000000000002throttling:0preferred_node_id:2fast_write_state:emptycache:readwriteudid:fc_map_count:1IBM_2145:ITSOCL3:admin>

The VDisk vd-prod-l2-0001 is a member of both a Remote Copy Relationship and a FlashCopy mapping.

After the background copy process has finished copying all the data to the target VDisk, the state changes to Idle_or_Copied, as shown in Example 6-31.

Example 6-31 svcinfo lsfcconsistgrp 1 - state: Idle_or_Copied

IBM_2145:ITSOCL3:admin>svcinfo lsfcconsistgrp 1id 1name fcg-back-001status idle_or_copiedFC_mapping_id 0FC_mapping_name fcm-back-001FC_mapping_id 1FC_mapping_name fcm-back-002IBM_2145:ITSOCL3:admin>

We can also look at the details of one of the mappings, shown in Example 6-32.

Example 6-32 svcinfo lsfcmap -delim : 0

IBM_2145:ITSOCL3:admin>svcinfo lsfcmap -delim : 0id:0name:fcm-back-001source_vdisk_id:1source_vdisk_name:vd-prod-l2-0001target_vdisk_id:2target_vdisk_name:vd-back-l2-0001



group_id:1group_name:fcg-back-001status:idle_or_copiedprogress:100copy_rate:70start_time:071107011320dependent_mappings:0autodelete:offclean_progress:100clean_rate:50incremental:ondifference:0grain_size:256IO_group_id:0IO_group_name:io_grp0IBM_2145:ITSOCL3:admin>

Since the mappings are set up to be incremental, we can prepare and start the Consistency Group again to refresh it, and it would only copy the grains which have changed since the last FlashCopy process. This puts less load on the SVC and makes the full clone of the VDisks available faster.

6.4.7 Modify a mapping

To modify a mapping we use the command svctask chfcmap, which uses the syntax shown in Example 6-33.

Example 6-33 svctask chfcmap - syntax

>>- svctask -- -- chfcmap -- --+-------------------------+-- ---> '- -name -- new_name_arg -'

>--+----------+-- ----------------------------------------------> '- -force -'


>--+------------------------+-- --+-------------------------+---> '- -copyrate -- percent-' '- -autodelete --+-on--+-' '-off-'

>--+-----------------------------+--+- fc_map_id ---+---------->< '- -cleanrate ---- percent ---' '- fc_map_name -'

We want to change both mappings to a lower background copy priority. Example 6-34 shows how we change the background copy priority to 50% and verify the settings for the mapping with the id 0. We do the same with the second mapping.

Example 6-34 svctask chfcmap -copyrate 50 0 - changing the copy priority to 50%

IBM_2145:ITSOCL3:admin>svctask chfcmap -copyrate 50 0IBM_2145:ITSOCL3:admin>svcinfo lsfcmap 0id 0



name fcm-back-001source_vdisk_id 1source_vdisk_name vd-prod-l2-0001target_vdisk_id 2target_vdisk_name vd-back-l2-0001group_id 1group_name fcg-back-001status idle_or_copiedprogress 100copy_rate 50start_time 071107021634dependent_mappings 0autodelete offclean_progress 100clean_rate 50incremental ondifference 0grain_size 256IO_group_id 0IO_group_name io_grp0IBM_2145:ITSOCL3:admin>

6.4.8 Stopping a mapping and a Consistency Group

Example 6-35 shows the syntax of the command svctask stopfcmap, which is used to stop a FlashCopy mapping.

Example 6-35 svctask stopfcmap - syntax

>>- svctask -- -- stopfcmap -- --+---------+-- -----------------> '- -force-'


The Consistency Group can be stopped with the command svctask stopfcconsistgrp for which the syntax shown in Example 6-36.

Example 6-36 svctask stopfcconsistgrp - syntax

>>- svctask -- -- stopfcconsistgrp -- --+---------+-- ----------> '- -force-'

>--+- fc_consist_group_id --+--------------------------------->< '- fc_consist_group_name -'

6.4.9 Deleting a mapping and a Consistency Group

Example 6-37 shows the syntax of the command svctask rmfcmap, which is used to delete a FlashCopy mapping.

Example 6-37 svctask rmfcmap - syntax

>>- svctask -- -- rmfcmap -- --+----------+-- ------------------> '- -force -'




Example 6-38 shows the syntax of the command svctask rmfcconsistgrp, used to delete a FlashCopy Consistency Group.

Example 6-38 svctask rmfcconsistgrp - syntax

>>- svctask -- -- rmfcconsistgrp -- --+----------+-- -----------> '- -force -'



Draft Document for Review February 10, 2008 5:17 pm 7574 Server integration.fm

Chapter 7. Server integration

In this chapter, we discuss the steps that need to be taken to access copied VDisks from servers running various Operating Systems.

In addition, we discuss some of the steps that need to be taken on the server to ensure that the copies that are generated will be easily accessed when needed.

As far as servers are concerned, there is no difference between a VDisk which was copied using FlashCopy and one that was copied using Metro/Global Mirror. As a result, this chapter will focus on accessing the VDisks with no concern as to how they were created.

7


7574 Server integration.fm Draft Document for Review February 10, 2008 5:17 pm

7.1 What data is copied?

When a VDisk is copied using SVC’s Copy Services, a full block-by-block replica is created. This means that everything that is on the source/primary VDisk is copied to the target/secondary VDisk. For many systems, this means that it isn’t just application data which is copied. Many systems write configuration data onto the disks to allow their volume managers to uniquely identify them and determine their place in volume groups.

This is can be an advantage or a disadvantage. When presenting the copied VDisks to a brand new server, it make bringing the volumes back online much simpler. However, when presenting these VDisks to the same server which accesses the original VDisks, there are challenges involved, because the metadata on the VDisks are often expected to be unique and presenting the copied VDisk breaches that requirement.

There is no way to selectively copy data from one VDisk to another; all data is copied. This chapter will discuss the techniques required to handle this fact.

7.2 What data is on the copied VDisk?

As discussed in Chapter 2, “Planning for Advanced Copy Services implementation”, the data which is on the original VDisk may not match what your applications and filesystems believe are on it. This is because of the caches between that layer and the SVC layer.

It is important to ensure one of the following:

� That data in the caches above the SVC are fully flushed before initiating the copy, or� That the systems that will access the copied VDisks will be able to recover from the fact

that the data is not an identical copy of the system’s view

The second option can be done by using journalled filesystems and having applications that protect their actions with a transaction log.

There is critical moment in a Copy Services mapping/relationship’s life cycle that affects the contents of the target/secondary VDisk. After this critical moment, the readable contents of the VDisk are no longer updated.

This critical moment depends on which Copy Service is in use.

� For FlashCopy, the critical moment is when the mapping is started� For Metro/Global Mirror, the critical moment is when the relationship is stopped (either

manually or due to errors)

In this chapter, the action is referred to as freezing and the moment is referred to as the freeze point.

At the freeze point, the target/secondary VDisk will be an image of the source/primary VDisk at that moment in time (plus or minus a few seconds for Global Mirror). Any data in the caches above SVC will not be on the target/secondary VDisk.

In the case of manual actions, it is relatively straightforward to ensure that the target/secondary VDisk is a useful copy. For Metro/Global Mirror relationships that are stopped due to SAN errors, this isn’t possible, so it’s important that the systems accessing these VDisks are configured to allow recovery.



Some things are not copied. Persistent Reserves are made against a specific Logical Unit and are not copied with FlashCopy or Metro/Global Mirror.

7.3 AIX specifics

In this section, we describe the steps needed to create and access copied VDisks on AIX® hosts.

SVC supports the use of SDD and MPIO+SDDPCM to provide multipathing. When SDD is in use, AIX takes hdisks and creates vpaths. When SDDPCM is in use, AIX creates hdisks and each one has multiple paths.

For the sake of clarity, the following section will assume the use of SDD and so talk about vpaths. The methods here apply just as well with SDDPCM. However, in those instances, the term hdisk should replace vpath.

7.3.1 Creating useful copies

Prior to the freeze point, steps must be taken to ensure that any information in the filesystem cache gets written to all of the relevant VDisks. The simplest way to do this is to unmount the filesystem, as in Example 7-1

Example 7-1 Unmounting a filesystem prior to making the break

#umount <source filesystem>

If unmounting filesystems is not an option, an alternative is to force an update of the node table and a flush of buffered files. The AIX freeze/thaw functionality does precisely this. This is only available on JFS2 filesystems. Freezing a file system writes all dirty file system metadata and user data to the disk. While frozen, the file system is read-only, and anything that attempts to modify the file system or its contents must wait for the freeze to end. At this point, the break can be made and the resulting copy can be easily picked up by another server. Example 7-2 shows how this is done

Example 7-2 Showing a simple freeze/thaw process

#chfs -a freeze=60 <source filesystem>

<freeze the VDisk(s)>

#chfs -a freeze=0 <source filesystem>

Of course, sometimes the freeze point occurs in an unplanned way, such as a data center outage. So long as the mapping/relationships are consistent, your filesystems should be recoverable and, in the case of a journalled filesystem, this recovery should not result in any data loss.

Example 7-3 shows the initial configuration of the SVC cluster presenting VDisks to this AIX server. The two VDisks represent a data drive and a log drive.

Example 7-3 Cluster configuration prior to FlashCopying two VDisks

IBM_2145:ITSOCL2:admin>svcinfo lsvdisk -delim :id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type:FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count

Chapter 7. Server integration 175


5:kanagaFS:0:io_grp0:online:1:DS4700:36.0GB:striped:::::6005076801AD80E8E000000000000005:06:kanagaLog:0:io_grp0:online:0:DS4500:512.0MB:striped:::::6005076801AD80E8E000000000000006:0

IBM_2145:ITSOCL2:admin>svcinfo lshost KANAGAid 2name KANAGAport_count 2type genericmask 1111iogrp_count 4WWPN 10000000C932A7FBnode_logged_in_count 2state activeWWPN 10000000C932A800node_logged_in_count 0state active

Example 7-4 shows how these VDisks have been configured on an AIX server to hold a JFS2 filesystem. The vdisk_UID property of a VDisk matches the SERIAL property of a vpath. Thus you can use these two properties to determine which vpath represents which VDisk.

Example 7-4 AIX configuration prior to FlashCopying two VDisks

#datapath query device

Total Devices : 2

DEV#: 0 DEVICE NAME: vpath0 TYPE: 2145 POLICY: OptimizedSERIAL: 6005076801AD80E8E000000000000005==========================================================================Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk3 OPEN NORMAL 209 0 1 fscsi0/hdisk5 OPEN NORMAL 220 0 2 fscsi0/hdisk7 OPEN NORMAL 0 0 3 fscsi0/hdisk9 OPEN NORMAL 0 0


#lspvhdisk0 0009cddaea97bf61 rootvg activehdisk1 0009cdda43c9dfd5 rootvg activehdisk2 0009cddabaef1d99 rootvg activehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none None



hdisk7 none Nonehdisk8 none Nonehdisk9 none Nonehdisk10 none Nonevpath0 0009cdda1b5058e4 st7s99 activevpath1 0009cdda1b55c48c st7s99 active

#lsvg -l st7s99st7s99:LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINTitsoLV jfs2 32 32 1 open/syncd /itsoFSitsoLog jfs2log 4 4 1 open/syncd N/A

#lsfsName Nodename Mount Pt VFS Size Options Auto Accounting/dev/hd4 -- / jfs2 196608 -- yes no/dev/hd1 -- /home jfs2 65536 -- yes no/dev/hd2 -- /usr jfs2 19005440 -- yes no/dev/hd9var -- /var jfs2 65536 -- yes no/dev/hd3 -- /tmp jfs2 393216 -- yes no/proc -- /proc procfs -- -- yes no/dev/hd10opt -- /opt jfs2 2031616 -- yes no/dev/itsoLV -- /itsoFS jfs2 4194304 rw no no

#mount node mounted mounted over vfs date options-------- --------------- --------------- ------ ------------ --------------- /dev/hd4 / jfs2 Nov 07 09:55 rw,log=/dev/hd8 /dev/hd2 /usr jfs2 Nov 07 09:55 rw,log=/dev/hd8 /dev/hd9var /var jfs2 Nov 07 09:55 rw,log=/dev/hd8 /dev/hd3 /tmp jfs2 Nov 07 09:55 rw,log=/dev/hd8 /dev/hd1 /home jfs2 Nov 07 09:56 rw,log=/dev/hd8 /proc /proc procfs Nov 07 09:56 rw /dev/hd10opt /opt jfs2 Nov 07 09:56 rw,log=/dev/hd8 /dev/itsoLV /itsoFS jfs2 Nov 07 11:17 rw,log=/dev/itsoLog

The Volume Group (VG) which is being copied is st7s99, containing vpath0 and vpath1.

Is this is a JFS2 filesystem, we can freeze the filesystem prior to starting the FlashCopy. The applications that access this filesystem will see write requests delayed until the filesystem is thawed. With appropriate planning and scripting, this freeze can be kept to a few seconds at most.

1. Create the FlashCopy target VDisks

IBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -size 36 -unit gb -mdiskgrp 1 -iogrp 0 -name kanagaFS-FCVirtual Disk, id [4], successfully created

IBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -size 512 -unit mb -mdiskgrp 0 -iogrp 0 -name kanagaLog-FCVirtual Disk, id [3], successfully created

2. Create a FlashCopy consistency group

IBM_2145:ITSOCL2:admin>svctask mkfcconsistgrp -name kanagaFC



FlashCopy Consistency Group, id [2], successfully created

3. Create FlashCopy mappings for the VDisks and add them to the new FlashCopy consistency group

IBM_2145:ITSOCL2:admin>svctask mkfcmap -source kanagaFS -target kanagaFS-FC -consistgrp kanagaFCFlashCopy Mapping, id [0], successfully created

IBM_2145:ITSOCL2:admin>svctask mkfcmap -source kanagaLog -target kanagaLog-FC -consistgrp kanagaFCFlashCopy Mapping, id [1], successfully created

4. Prepare the FlashCopy consistency group.

IBM_2145:ITSOCL2:admin>svctask prestartfcconsistgrp kanagaFC

5. Wait until the consistency group is in the prepared state.

IBM_2145:ITSOCL2:admin>svcinfo lsfcconsistgrpid name status2 kanagaFC preparing...IBM_2145:ITSOCL2:admin>svcinfo lsfcconsistgrpid name status2 kanagaFC prepared

6. Freeze the filesystem on the AIX server.

#chfs -a freeze=60 /itsoFS

7. Start the FlashCopy consistency group. This is the freeze point.

IBM_2145:ITSOCL2:admin>svctask startfcconsistgrp kanagaFC

8. Thaw the filesystem on the AIX server.

#chfs -a freeze=0 /itsoFS

At this point, the FlashCopy operation has been started and VDisks 3 and 4 are exact copies of VDisks 5 and 6 at the time of the filesystem freeze.

7.3.2 Accessing useful copies

If the source/primary VDisk is defined to the AIX Logical Volume Manager (LVM) then all of its data structures and identifiers are copied to the target/secondary VDisk, as well. This includes the Volume Group Descriptor Area (VGDA), which contains the Physical Volume Identifier (PVID) and Volume Group Identifier (VGID).

For AIX LVM, it is currently not possible to activate a volume group with a physical volume (vpath) that contains a VGID and a PVID that is already used in a volume group existing on the same server. The restriction still applies even if the vpath PVID is cleared and reassigned with the two commands listed in Example 7-5.

Example 7-5 Clearing PVIDs

chdev -l <vpath#> -a pv=clearchdev -l <vpath#> -a pv=yes

Therefore, it is necessary to redefine the volume group information on the FlashCopy target VDisk using special procedures or the recreatevg command. This will alter the PVIDs and



VGIDs in all the VGDAs of the FlashCopy target VDisks, so that there are no conflicts with existing PVIDs and VGIDs on existing volume groups that reside on the source VDisks. If you do not redefine the volume group information prior to importing the volume group, then the importvg command will fail.

Accessing target/secondary VDisks from the same AIX hostIn this section, we describe a method of accessing the FlashCopy target VDisks on a single AIX host while the source VDisks are still active on the same server. The procedure is intended to be used as a guide and may not cover all scenarios.

AIX recreatevg commandCopying the source volume's content using FlashCopy causes all of the data structures and identifiers used by AIX's Logical Volume Manager to be duplicated to the target volume. The duplicate definitions (PVID and VGID) in turn cause conflicts within LVM. This problem is solved by using the AIX command recreatevg.

The recreatevg command is officially available in all levels of AIX that SVC supports

The recreatevg command overcomes the problem of duplicated LVM data structures and identifiers caused by a disk duplication process such as FlashCopy. It is used to recreate an AIX Volume Group (VG) on a set of target volumes that are copied from a set of source volumes belonging to a specific VG. The command will allocate new physical volume identifiers (PVIDs) for the member disks and a new volume group identifier (VGID) to the volume group. The command also provides options to rename the logical volumes with a prefix you specify, and options to rename labels to specify different mount points for file systems. To use this command, you must have root user authority.

The steps below assume that the AIX server and SVC cluster are in the state following the process in the previous section.

Perform these tasks to access the FlashCopy target VDisks:

1. Map the FlashCopy target VDisks to the host

IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host KANAGA kanagaFS-FCVirtual Disk to Host map, id [2], successfully createdIBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host KANAGA kanagaLog-FCVirtual Disk to Host map, id [3], successfully created

2. Run Configuration Manager to pick up the new devices on the AIX server

#cfgmgr -vl fscsi0

(skipped the debug output from this command)

3. Run Configuration Manager to create vpaths with the new hdisks

#cfgmgr -vl dpo

At this point, the configuration of the AIX server has been changed. Example 7-6 shows the new status.

Example 7-6 AIX configuration prior to FlashCopying two VDisks


Attention: Running recreatevg will change the PVID on the VDisks. In the case of a Metro/Global Mirror relationship, if you decide to reverse the direction of the relationship, you will change the PVID of the original primary VDisk during the synchronization process.



Total Devices : 4



DEV#: 2 DEVICE NAME: vpath2 TYPE: 2145 POLICY: OptimizedSERIAL: 6005076801AD80E8E000000000000007==========================================================================Path# Adapter/Hard Disk State Mode Select Errors 0 fscsi0/hdisk11 CLOSE NORMAL 0 0 1 fscsi0/hdisk13 CLOSE NORMAL 0 0 2 fscsi0/hdisk15 CLOSE NORMAL 0 0 3 fscsi0/hdisk17 CLOSE NORMAL 0 0


#lspvhdisk0 0009cddaea97bf61 rootvg activehdisk1 0009cdda43c9dfd5 rootvg activehdisk2 0009cddabaef1d99 rootvg activehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none Nonehdisk7 none Nonehdisk8 none Nonehdisk9 none Nonehdisk10 none Nonevpath0 0009cdda1b5058e4 st7s99 activevpath1 0009cdda1b55c48c st7s99 activehdisk11 0009cdda1b5058e4 st7s99 active



hdisk12 0009cdda1b55c48c st7s99 activehdisk13 0009cdda1b5058e4 st7s99 activehdisk14 0009cdda1b55c48c st7s99 activehdisk15 0009cdda1b5058e4 st7s99 activehdisk16 0009cdda1b55c48c st7s99 activehdisk17 0009cdda1b5058e4 st7s99 activehdisk18 0009cdda1b55c48c st7s99 activevpath2 none Nonevpath3 none None

The new vpaths, vpath2 and vpath3 appear with no PVID whereas the hdisks that make up those vpaths do. This is as expected. Note also that vpath2 and vpath3 are in a CLOSE state, because nothing is currently accessing them. The following steps will recreate the VG with a new name and recover the Logical Volumes (LV)

1. Create the new VG and prefix all file system path names with /fc and prefix all AIX LVs with fc:

#recreatevg -y st7s99fc -L /fc -Y FC vpath2 vpath3st7s99fc

2. Run fsck on the new filesystem. Strictly speaking this is not necessary as the steps taken while creating the copy ensure that the filesystem is client, but it is a prudent defensive step.

#fsck -y /fc/itsoFS

The current volume is: /dev/FCitsoLVPrimary superblock is valid.J2_LOGREDO:log redo processing for /dev/FCitsoLVPrimary superblock is valid.*** Phase 1 - Initial inode scan*** Phase 2 - Process remaining directories*** Phase 3 - Process remaining files*** Phase 4 - Check and repair inode allocation map*** Phase 5 - Check and repair block allocation mapFile system is clean.

3. Mount the new filesystem

#mount /fc/itsoFSReplaying log for /dev/FCitsoLV.

At this point, the FlashCopy target VDisks are ready for access. Example 7-7 shows the final server state. This new filesystem contains all of the files that were in the original filesystem at the time of the freeze.

Example 7-7 AIX configuration after recreating the VG


Restriction: The LV prefix that is given via the -Y parameter cannot be “fc”. The reason for this is that “fc” is already defined as a prefix in the PdDv class of the Device Configuration Database and so is not a valid LV prefix.



Total Devices : 4





#lspvhdisk0 0009cddaea97bf61 rootvg activehdisk1 0009cdda43c9dfd5 rootvg activehdisk2 0009cddabaef1d99 rootvg activehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none Nonehdisk7 none Nonehdisk8 none Nonehdisk9 none Nonehdisk10 none Nonevpath0 0009cdda1b5058e4 st7s99 activevpath1 0009cdda1b55c48c st7s99 activehdisk11 none None



hdisk12 none Nonehdisk13 none Nonehdisk14 none Nonehdisk15 none Nonehdisk16 none Nonehdisk17 none Nonehdisk18 none Nonevpath2 0009cdda1bf46d57 st7s99fc activevpath3 0009cdda1bf46f3b st7s99fc active

#lsvgrootvgst7s99st7s99fc


#lsvg -l st7s99fcst7299fc:LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINTFCitsoLV jfs2 32 32 1 open/syncd /fc/itsoFSFCitsoLog jfs2log 4 4 1 open/syncd N/A

#lsfsName Nodename Mount Pt VFS Size Options Auto Accounting/dev/hd4 -- / jfs2 196608 -- yes no/dev/hd1 -- /home jfs2 65536 -- yes no/dev/hd2 -- /usr jfs2 19005440 -- yes no/dev/hd9var -- /var jfs2 65536 -- yes no/dev/hd3 -- /tmp jfs2 393216 -- yes no/proc -- /proc procfs -- -- yes no/dev/hd10opt -- /opt jfs2 2031616 -- yes no/dev/itsoLV -- /itsoFS jfs2 4194304 rw no no/dev/FCitsoLV -- /fc/itsoFS jfs2 4194304 rw no no

#mountnode mounted mounted over vfs date options------ --------------- --------------- ------ ------------ ---------------

/dev/hd4 / jfs2 Nov 07 09:55 rw,log=/dev/hd8/dev/hd2 /usr jfs2 Nov 07 09:55 rw,log=/dev/hd8/dev/hd9var /var jfs2 Nov 07 09:55 rw,log=/dev/hd8/dev/hd3 /tmp jfs2 Nov 07 09:55 rw,log=/dev/hd8/dev/hd1 /home jfs2 Nov 07 09:56 rw,log=/dev/hd8/proc /proc procfs Nov 07 09:56 rw/dev/hd10opt /opt jfs2 Nov 07 09:56 rw,log=/dev/hd8/dev/itsoLV /itsoFS jfs2 Nov 07 11:17 rw,log=/dev/itsoLog/dev/FCitsoLV /fc/itsoFS jfs2 Nov 07 13:15 rw,log=/dev/FCitsoLog



Accessing target/secondary VDisks from a different AIX hostAccessing target/secondary VDisks from a different server to the one accessing the source/primary VDisks is much simpler.

The following procedure makes the data of the FlashCopy target VDisk available to another AIX host that has no prior definitions of the target VDisk in its configuration database (ODM). Example 7-8 on page 184 shows the state of the AIX server, prior to presenting the target VDisks.

Example 7-8 AIX configuration prior presenting FlashCopy target VDisks


No device file found

#lsvgrootvg

#lspvhdisk0 0009cddaea97bf61 rootvg activehdisk1 0009cdda43c9dfd5 rootvg activehdisk2 0009cddabaef1d99 rootvg active

#lsfsName Nodename Mount Pt VFS Size Options Auto Accounting/dev/hd4 -- / jfs2 196608 -- yes no/dev/hd1 -- /home jfs2 65536 -- yes no/dev/hd2 -- /usr jfs2 19005440 -- yes no/dev/hd9var -- /var jfs2 65536 -- yes no/dev/hd3 -- /tmp jfs2 393216 -- yes no/proc -- /proc procfs -- -- yes no/dev/hd10opt -- /opt jfs2 2031616 -- yes no

The initial process for accessing the VDisks is as before.


IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host UMNAK kanagaFS-FCVirtual Disk to Host map, id [2], successfully createdIBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host UMNAK kanagaLog-FCVirtual Disk to Host map, id [3], successfully created

2. Run Configuration Manager to pick up the new devices on the AIX server

#cfgmgr -vl fscsi0

(skipped the debug output from this command)

3. Run Configuration Manager to create vpaths with the new hdisks

#cfgmgr -vl dpo

At this point, the configuration of the AIX server has been changed. Example 7-9 shows the new status.

Example 7-9 AIX configuration after presenting FlashCopy target VDisks

datapath query device



Total Devices : 2



#lspvhdisk0 0009cddaea97bf61 rootvg activehdisk1 0009cdda43c9dfd5 rootvg activehdisk2 0009cddabaef1d99 rootvg activehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none Nonehdisk7 none Nonehdisk8 none Nonehdisk9 none Nonehdisk10 none Nonevpath2 0009cdda1b5058e4 st7s99vpath3 0009cdda1b55c48c st7s99

Note that, this time, it’s the new vpaths that have PVIDs and the hdisks have none. We use the importvg command to import the information about the VG into this server’s ODM:

1. Import the target volume group:

#importvg -y st7s99 vpath2st7s99

2. Vary on the Volume Group (the importvg command should varyon the volume group):

#varyonvg st7s99

3. Verify consistency of all file systems on the FlashCopy target VDisk:

#fsck -y /itsoFS

The current volume is: /dev/itsoLVPrimary superblock is valid.J2_LOGREDO:log redo processing for /dev/itsoLVPrimary superblock is valid.*** Phase 1 - Initial inode scan



*** Phase 2 - Process remaining directories*** Phase 3 - Process remaining files*** Phase 4 - Check and repair inode allocation map*** Phase 5 - Check and repair block allocation mapFile system is clean.

4. Mount all the target file systems:

#mount /itsoFS

At this point, the FlashCopy target VDisks are ready for access. Example 7-10 shows the final server state. This new filesystem contains all of the files that were in the original filesystem at the time of the freeze.

Example 7-10 AIX configuration after importing the VG


Total Devices : 2



#lspvhdisk0 0009cddaea97bf61 rootvg activehdisk1 0009cdda43c9dfd5 rootvg activehdisk2 0009cddabaef1d99 rootvg activehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none Nonehdisk7 none Nonehdisk8 none Nonehdisk9 none Nonehdisk10 none Nonevpath2 0009cdda1b5058e4 st7s99 activevpath3 0009cdda1b55c48c st7s99 active

#lsvgrootvgst7s99




#lsfsName Nodename Mount Pt VFS Size Options Auto Accounting/dev/hd4 -- / jfs2 196608 -- yes no/dev/hd1 -- /home jfs2 65536 -- yes no/dev/hd2 -- /usr jfs2 19005440 -- yes no/dev/hd9var -- /var jfs2 65536 -- yes no/dev/hd3 -- /tmp jfs2 393216 -- yes no/proc -- /proc procfs -- -- yes no/dev/hd10opt -- /opt jfs2 2031616 -- yes no/dev/itsoLV -- /itsoFS jfs2 4194304 rw no no

#mount node mounted mounted over vfs date options-------- --------------- --------------- ------ ------------ --------------- /dev/hd4 / jfs2 Nov 07 14:04 rw,log=/dev/hd8 /dev/hd2 /usr jfs2 Nov 07 14:04 rw,log=/dev/hd8 /dev/hd9var /var jfs2 Nov 07 14:04 rw,log=/dev/hd8 /dev/hd3 /tmp jfs2 Nov 07 14:04 rw,log=/dev/hd8 /dev/hd1 /home jfs2 Nov 07 14:05 rw,log=/dev/hd8 /proc /proc procfs Nov 07 14:05 rw /dev/hd10opt /opt jfs2 Nov 07 14:05 rw,log=/dev/hd8 /dev/itsoLV /itsoFS jfs2 Nov 07 14:36 rw,log=/dev/itsoLog



7.4 Windows 2000/2003 and Copy Services

Windows® 2000 and Windows 2003 incorporate a stripped-down version of the VERITAS Volume Manager, called the Logical Disk Manager (LDM) for handling their disks.

With the LDM, you are able to create logical partitions, perform disk mounts, and create dynamic volumes. There are five types of dynamic volumes: simple, spanned, mirrored, striped, and RAID-5.

On earlier versions of Windows, the information relating to the disks was stored in the Windows registry. With Windows 2000/2003, this information is stored on the disk drive itself in a partition called the LDM database, which is kept on the last few tracks of the disk. Each volume has its own 128-bit Globally Unique Identifier (GUID) and belongs to a disk group. This is similar to the concept of Physical Volume Identifier (PVID) and Volume Group in AIX. As the LDM is stored on the physical drive itself, with Windows 2000/2003 it is possible to move disk drives between different computers.

Copy Services limitations with Windows 2000/2003It is not possible to have FlashCopy source and target VDisks mapped t0 the same Windows host when they were created as dynamic volumes. The reason is that each dynamic volume has to have its own 128-bit GUID. As its name implies, the GUID must be unique on one system. When you perform FlashCopy, the GUID gets copied as well, so this means that if you tried to mount the source and target volume on the same host system, you would have two volumes with exactly the same GUID. This is not allowed, and you will not be able to mount the target volume.

7.4.1 Creating useful copiesAs with AIX, the way to create useful copies is to ensure that there is no data in the system write cache when you freeze the VDisks. The methods for doing this are conceptually similar. Both of these methods can be done by accessing the Computer Management console on your Windows host.

One option is to remove the drive letter associated with the disk. Figure 7-1 on page 189 shows a VDisk which is currently being used by a Windows server. Selecting “Change Drive Letter and Paths...” opens up the dialog box shown in Figure 7-2 on page 189. Doing this will result in all outstanding writes to the disk to be completed. Like unmounting a filesystem in AIX, no applications will be able to access data on this drive, without the drive letter.



Figure 7-1 Choosing to change the Drive Letter of a Windows Disk

Figure 7-2 Removing a Drive Letter from a Windows Disk

An alternative method is to disable the write cache for the disk in question. Figure 7-3 shows how to access the Properties of a Windows Disk. This will bring up a dialog box shown in Figure 7-4 on page 190.



Figure 7-3 Setting the properties of a Windows Disk

By selecting “Optimize for quick removal”, the write cache is disabled, so you can make the break and be sure that the VDisk contents matches the image that the host has.

Figure 7-4 Disabling the write cache of a Windows Disk



Example 7-11 shows the initial configuration of the SVC cluster presenting VDisks to this Windows server. The two VDisks represent a data drive and a log drive.

Example 7-11 Cluster configuration prior to FlashCopying two VDisks

IBM_2145:ITSOCL2:admin>svcinfo lsvdisk -delim :id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type:FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count4:senegalDynamic2:0:io_grp0:online:1:DS4700:36.0GB:striped:2:fcmap2:::6005076801AD80E8E00000000000000C:15:senegalDynamic1:0:io_grp0:online:1:DS4700:36.0GB:striped:1:fcmap1:::6005076801AD80E8E00000000000000B:16:senegalBasic:0:io_grp0:online:1:DS4700:36.0GB:striped:0:fcmap0:::6005076801AD80E8E00000000000000A:1

IBM_2145:ITSOCL2:admin>svcinfo lshost SENEGALid 1name SENEGALport_count 2type genericmask 1111iogrp_count 4WWPN 210000E08B89B9C0node_logged_in_count 2state activeWWPN 210000E08B89CCC2node_logged_in_count 0state active

Example 7-12 shows the SDD configuration of the Windows 2003 server to which these VDisks are mapped. Figure 7-5 shows how these VDisks have been configured on a Windows 2003 server.

Example 7-12 SDD Configuration of Windows 2003 server

C:\Program Files\IBM\Subsystem Device Driver>datapath query device

Total Devices : 3

DEV#: 0 DEVICE NAME: Disk1 Part0 TYPE: 2145 POLICY: OPTIMIZEDSERIAL: 6005076801AD80E8E00000000000000C============================================================================Path# Adapter/Hard Disk State Mode Select Errors 0 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 8 0 1 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 10 0 2 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 0 0 3 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 0 0 4 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 0 0 5 Scsi Port3 Bus0/Disk1 Part0 OPEN NORMAL 0 0

DEV#: 1 DEVICE NAME: Disk2 Part0 TYPE: 2145 POLICY: OPTIMIZEDSERIAL: 6005076801AD80E8E00000000000000B============================================================================Path# Adapter/Hard Disk State Mode Select Errors 0 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 0 0



1 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 0 0 2 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 3 0 3 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 5 0 4 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 5 0 5 Scsi Port3 Bus0/Disk2 Part0 OPEN NORMAL 5 0

DEV#: 2 DEVICE NAME: Disk3 Part0 TYPE: 2145 POLICY: OPTIMIZEDSERIAL: 6005076801AD80E8E00000000000000A============================================================================Path# Adapter/Hard Disk State Mode Select Errors 0 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 11 0 1 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 9 0 2 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 0 0 3 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 0 0 4 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 0 0 5 Scsi Port3 Bus0/Disk3 Part0 OPEN NORMAL 0 0

Figure 7-5 Showing how VDisks appear on a Windows 2003 host, prior to mapping target/secondary VDisks



Figure 7-6 Identifying a Windows Disk as a VDisk

The vdisk_UID property of a VDisk matches the SERIAL property of a vpath. Thus you can use these two properties to determine which vpath represents which VDisk. The Device Name of the vpath matches the Disk property shown under the Volumes tab as shown in Figure 7-6. Thus you can use these two properties to determine which Windows Disk represents which vpath. Subsequently, you can match a Windows Disk to a VDisk.

The process below shows how to make sure that the FlashCopy target VDisks contain useful copies

1. Create the FlashCopy target VDisks

IBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -unit gb -size 36 -iogrp 0 -mdiskgrp DS4700 -name senegalBasic-FCVirtual Disk, id [3], successfully createdIBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -unit gb -size 36 -iogrp 0 -mdiskgrp DS4700 -name senegalDyn1-FCVirtual Disk, id [8], successfully createdIBM_2145:ITSOCL2:admin>svctask mkvdisk -vtype striped -unit gb -size 36 -iogrp 0 -mdiskgrp DS4700 -name senegalDyn2-FCVirtual Disk, id [7], successfully created

2. Create a FlashCopy consistency group

IBM_2145:ITSOCL2:admin>svctask mkfcconsistgrp -name senegalFCFlashCopy Consistency Group, id [1], successfully created

3. Create FlashCopy mappings for the VDisks and add them to the new FlashCopy consistency group

IBM_2145:ITSOCL2:admin>svctask mkfcmap -source senegalBasic -target senegalBasic-FC -consistgrp senegalFCFlashCopy Mapping, id [0], successfully created



IBM_2145:ITSOCL2:admin>svctask mkfcmap -source senegalDynamic1 -target senegalDyn1-FC -consistgrp senegalFCFlashCopy Mapping, id [1], successfully createdIBM_2145:ITSOCL2:admin>svctask mkfcmap -source senegalDynamic2 -target senegalDyn2-FC -consistgrp senegalFCFlashCopy Mapping, id [2], successfully created

4. Prepare the FlashCopy consistency group

IBM_2145:ITSOCL2:admin>svctask prestartfcconsistgrp senegalFC

5. Wait until the consistency group is in the prepared state

IBM_2145:ITSOCL2:admin>svcinfo lsfcconsistgrpid name status1 senegalFC preparing...IBM_2145:ITSOCL2:admin>svcinfo lsfcconsistgrpid name status1 senegalFC prepared

6. Disable the write cache for the Windows Disks as shown in Figure 7-4 on page 190

7. Start the FlashCopy consistency group. This is the freeze point.

IBM_2145:ITSOCL2:admin>svctask startfcconsistgrp senegalFC

8. Re-enable the write cache for the Windows Disks

At this point, the FlashCopy operation has been started and VDisks 3, 8 and 7 are exact copies of VDisks 6, 5 and 4 at the time of the FlashCopy consistency group being started.

7.4.2 Accessing useful copiesBasic disks relatively straightforward to access from a Windows 2000/2003 host. Dynamic disks are a little more complicated because of the restrictions imposed on duplicate GUIDs.

Accessing target/secondary VDisks from the same Windows hostIn this section, we describe a method of accessing the FlashCopy target VDisks on a single Windows host while the source VDisks are still active on the same server.

The steps below assume that the Windows server and SVC cluster are in the state following the process in the previous section.

Perform these tasks to access the FlashCopy target VDisks:


IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SENEGAL senegalBasic-FCVirtual Disk to Host map, id [3], successfully createdIBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SENEGAL senegalDyn1-FCVirtual Disk to Host map, id [4], successfully createdIBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SENEGAL senegalDyn2-FCVirtual Disk to Host map, id [5], successfully created

2. Scan for new Hardware Changes as shown in Figure 7-7 on page 195 to pick up the newly presented VDisks



Figure 7-7 Scanning for hardware changes to pick up target/secondary VDisks

At this point, the configuration of the Windows host has been changed. Figure 7-8 on page 196 shows the new status. The new Disks, Disk 4 and Disk 6 appear as unallocated. These are the Dynamic Disks (as is reported by the LDM). Disk 5, the Basic Disk has already been recognized as one. All that needs to be done is to assign it a drive letter as show in Appendix 7-1, “Choosing to change the Drive Letter of a Windows Disk” on page 189. Clicking on “Add” brings up the dialog box as shown in Figure 7-9 on page 197

Once a drive letter has been assigned to your Basic Drives, you should run chkdsk to ensure filesystem consistency as shown in Example 7-13. This is particularly important if the break was made due to an unplanned Metro/Global Mirror stoppage.

Example 7-13 Running chkdsk on a newly mapped target/secondary VDisk

C:\Program Files\IBM\Subsystem Device Driver>chkdsk /f e:The type of the file system is NTFS.Volume label is basicVol.

CHKDSK is verifying files (stage 1 of 3)...File verification completed.CHKDSK is verifying indexes (stage 2 of 3)...Index verification completed.CHKDSK is verifying security descriptors (stage 3 of 3)...Security descriptor verification completed.Windows has checked the file system and found no problems.



37744685 KB total disk space. 20 KB in 2 files. 4 KB in 9 indexes. 0 KB in bad sectors. 67137 KB in use by the system. 65536 KB occupied by the log file. 37677524 KB available on disk.

4096 bytes in each allocation unit. 9436171 total allocation units on disk. 9419381 allocation units available on disk.

Target/secondary VDisks which are Dynamic Disks should never be presented to a host which has visibility of the source/primary VDisk as there is a risk of data loss.

Figure 7-8 Windows configuration after discovering target/secondary VDisks



Figure 7-9 Adding a drive letter to a Basic Drive

Accessing taget/secondary VDisks from the same Windows host

Accessing target/secondary VDisks from a different Windows server to the one accessing the source/primary VDisks is much simpler. In the following example, server has no VDisks mapped to it, to begin with. In Example 7-14, the VDisks are mapped to the server

Example 7-14 Mapping FlashCopy targets to a new Windows Server®

IBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SIAM senegalBasic-FCVirtual Disk to Host map, id [2], successfully createdIBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SIAM senegalDyn1-FCVirtual Disk to Host map, id [3], successfully createdIBM_2145:ITSOCL2:admin>svctask mkvdiskhostmap -host SIAM senegalDyn2-FCVirtual Disk to Host map, id [4], successfully created

As in Figure 7-7 on page 195, the next step is to scan for hardware changes. The result of this is shown in Figure 7-10. As is shown, the Basic Disk is picked up normally and just needs a Drive Letter assigned to it.

Figure 7-10 on page 198 also shows the state of the Dynamic Disks. They register as Foreign. By selecting “Import Foreign Disks...”, you will see the dialog box shown in Figure 7-11 on page 198. This is followed by Figure 7-12 on page 199 which lists the drives that you will import.

At this point, all of the drives letter have been assigned to your VDisks; you should run chkdsk to ensure filesystem consistency as shown in Example 7-13. This is particularly important if the freezing was due to an unplanned Metro/Global Mirror stoppage.



Figure 7-10 Results of hardware scan on a Windows 2003 server

Figure 7-11 Importing Foreign Disks



Figure 7-12 Importing Foreign Disk Volumes



7.5 Linux Specifics

In this section, we describe the steps needed to create and access copied VDisks on Linux® Hosts. SVC currently supports Red Hat and Suse Linux, but the steps are distribution independent.

SVC supports the use of multiple multi-pathing solutions when accessed via a Linux server. The examples below focus on the use of SDD.

7.5.1 Creating useful copies

As with AIX and Windows, it is necessary to ensure that any information in the filesystem cache gets written to the VDisks, prior to freezing the VDisks. The simplest way to do this is to unmount the filesystem, as was shown in Example 7-1 on page 175.

The XFS filesystem includes support for freezing filesystems in a similar manner to AIX. Details on this functionality is beyond the scope of this redbook, but more details can be found in an IBM Developerworks article: http://www.ibm.com/developerworks/linux/library/l-fs9.html.

Example 7-15 shows the initial configuration of the SVC cluster presenting VDisks to a Red Hat Linux server.

Example 7-15 Configuration of clusters prior to copying a VDisk with Metro Mirror

IBM_2145:ITSOCL2:admin>svcinfo lsvdisk -delim :id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type:FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count8:diomede-1ary:0:io_grp0:online:0:DS4500:36.0GB:striped:::::6005076801AD80E8E000000000000014:0

IBM_2145:ITSOCL2:admin>svcinfo lshost DIOMEDEid 0name DIOMEDEport_count 2type genericmask 1111iogrp_count 4WWPN 210000E08B0548BCnode_logged_in_count 2state activeWWPN 210000E08B0541BCnode_logged_in_count 2state active

IBM_2145:ITSOCL3:admin>svcinfo lsvdisk -delim :id:name:IO_group_id:IO_group_name:status:mdisk_grp_id:mdisk_grp_name:capacity:type:FC_id:FC_name:RC_id:RC_name:vdisk_UID:fc_map_count6:diomede-2dary:0:io_grp0:online:0:DS4500:36.0GB:striped:::::6005076801A100E90800000000000008:0

Example 7-16 on page 201 shows the configuration of an Linux server prior to copying a VDisk with Metro Mirror.


http://www.ibm.com/developerworks/linux/library/l-fs9.html


Example 7-16 Linux host configuration prior to copying 2 VDisks with Metro Mirror

[root@diomede ~]# datapath query device

Total Devices : 1

DEV#: 0 DEVICE NAME: vpatha TYPE: 2145 POLICY: Optimized SequentialSERIAL: 6005076801ad80e8e000000000000014============================================================================Path# Adapter/Hard Disk State Mode Select Errors 0 Host0Channel0/sda OPEN NORMAL 1 0 1 Host0Channel0/sdb OPEN NORMAL 41815 0 2 Host1Channel0/sdc OPEN NORMAL 0 0 3 Host1Channel0/sdd OPEN NORMAL 41679 0

[root@diomede ~]# pvdisplay --- Physical volume --- PV Name /dev/hda2 VG Name VolGroup00 PV Size 74.41 GB / not usable 0 Allocatable yes PE Size (KByte) 32768 Total PE 2381 Free PE 3 Allocated PE 2378 PV UUID Xq0UQs-94ID-tCq7-4jK2-KRt7-4iYF-lCf9wf

--- Physical volume --- PV Name /dev/vpatha VG Name itso PV Size 36.00 GB / not usable 0 Allocatable yes PE Size (KByte) 4096 Total PE 9215 Free PE 4607 Allocated PE 4608 PV UUID dAZQYq-VsI1-n2xp-vp0c-hfSE-glCx-MjnPV3

[root@diomede ~]# vgdisplay itso --- Volume group --- VG Name itso System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1



VG Size 36.00 GB PE Size 4.00 MB Total PE 9215 Alloc PE / Size 4608 / 18.00 GB Free PE / Size 4607 / 18.00 GB VG UUID 0PsMcJ-nDr8-b3ds-YyCM-Mlde-OegZ-LZFddT

[root@diomede /]# df -TFilesystem Type 1K-blocks Used Available Use% Mounted on/dev/mapper/VolGroup00-LogVol00 ext3 74699952 11571300 59334120 17% //dev/hda1 ext3 101086 22817 73050 24% /bootnone tmpfs 1033144 0 1033144 0% /dev/shm/dev/mapper/itso-mmPrimary ext3 18578172 77888 17556568 1% /mmSource

[root@diomede /]# mount/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)none on /proc type proc (rw)none on /sys type sysfs (rw)none on /dev/pts type devpts (rw,gid=5,mode=620)usbfs on /proc/bus/usb type usbfs (rw)/dev/hda1 on /boot type ext3 (rw)none on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)/dev/mapper/itso-mmPrimary on /mmSource type ext3 (rw)

The VG which is being copied is itso which contains a single vpath, vpatha.

This is an ext3 filesystem, There are no user commands for freezing this filesystem, so the only way to freeze the VDisk cleanly is to unmount the filesystem and deactivate the LV. However, with appropriate planning and scripting, the disruption can be minimized.

1. Create the Metro Mirror consistency group

IBM_2145:ITSOCL2:admin>svctask mkrcconsistgrp -name diomedeMM -cluster ITSOCL3RC Consistency Group, id [253], successfully created

2. Create the Metro Mirror relationship connecting the two VDisks and add it to the consistency group

IBM_2145:ITSOCL2:admin>svctask mkrcrelationship -master diomede-1ary -aux diomede-2dary -cluster ITSOCL3 -name diomedeMM1 -consistgrp diomedeMMRC Relationship, id [8], successfully created

3. Start the consistency group

IBM_2145:ITSOCL2:admin>svctask startrcconsistgrp diomedeMM

4. Monitor the consistency group and wait until it attains consistency and synchronization

IBM_2145:ITSOCL2:admin>svcinfo lsrcconsistgrp diomedeMMid 253name diomedeMMmaster_cluster_id 000002006B603A38master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primary master



state consistent_synchronizedrelationship_count 1freeze_timestatus onlinesynccopy_type metroRC_rel_id 8RC_rel_name diomedeMM1

5. Unmount the filesystem to flush all the data to the primary VDisk

[root@diomede ~]# umount /mmSource

6. Stop the Metro Mirror consistency group. This is the freeze point.

IBM_2145:ITSOCL3:admin>svctask stoprcconsistgrp -access diomedeMM

IBM_2145:ITSOCL3:admin>svcinfo lsrcconsistgrp diomedeMMid 253name diomedeMMmaster_cluster_id 000002006B603A38master_cluster_name ITSOCL2aux_cluster_id 0000020068403A42aux_cluster_name ITSOCL3primarystate idlingrelationship_count 1freeze_timestatussync in_synccopy_type metroRC_rel_id 6RC_rel_name diomedeMM1

7. Remount the filesystem

[root@diomede ~]# mount /dev/itso/mmPrimary /mmSource

At this point the Metro Mirror relationship has been stopped and diomede-2ary is an exact copy of diomede-1ary.

7.5.2 Accessing useful copies

As with AIX, if the source/primary VDisk is defined to the AIX Logical Volume Manager (LVM) then all of its data structures and identifiers are copied to the target/secondary VDisk, as well. This includes the Universal Unique Identifiers (UUIDs) that are used to identify PVs, VGs and LVs.

It is currently not possible to activate a volume group with a physical volume (vpath) that contains a VG UUID and a PV UUID that is already used in a volume group existing on the same server.

Accessing target/secondary VDisks from the same Linux hostOur research into method for bringing copied volume groups online on a Linux server have have shown up bespoke solutions that users have published on various newsgroups. There appears to be no clear method for doing this in a non-disruptive manner. Most strategies involve manually altering backups of the LVM configuration. As a result, we recommend that



you do not present copied VDisks back to their original servers. If you determine that this is a necessary process, you should consult with LVM experts to determine a suitable process.

Accessing target/secondary VDisks from a different Linux host

In this example, we show how to recover a VDisk which was not frozen cleanly. This can happen if a Metro Mirror relationship stops while a filesystem is in use.

1. Map the target VDisk to the new server

IBM_2145:ITSOCL3:admin>svctask mkvdiskhostmap -host PALAU diomede-2daryVirtual Disk to Host map, id [0], successfully created

2. Discover the LU on the Linux server. QLogic® provide a tool to perform this step. More information can be found at http://download.qlogic.com/ms/53595/readme_dynamic_lun_util_18.html

3. Run cfgvpath to create the relevant vpath

[root@palau ~]# cfgvpathc--------- 1 root root 254, 0 Nov 14 22:32 /dev/IBMsddAdded vpatha 252 0 path sda 8 0....Added vpatha 252 0 path sdb 8 16....Added vpatha 252 0 path sdc 8 32....Added vpatha 252 0 path sdd 8 48....Writing out new configuration to file /etc/vpath.confWaiting for hotplug/udev system to configure all vpath device nodes...

4. Run pvscan to pick up the copied PV

[root@palau ~]# pvscan PV /dev/hda2 VG VolGroup00 lvm2 [74.41 GB / 96.00 MB free] PV /dev/vpatha VG itso lvm2 [36.00 GB / 18.00 GB free] Total: 2 [110.40 GB] / in use: 2 [110.40 GB] / in no VG: 0 [0 ]

5. Run vscan to make sure that you have your VG

[root@palau ~]# vgscan Reading all physical volumes. This may take a while... Found volume group "VolGroup00" using metadata type lvm2 Found volume group "itso" using metadata type lvm2

6. Run lvscan to make sure that you have your expected LVs

[root@palau ~]# lvscan ACTIVE '/dev/VolGroup00/LogVol00' [72.38 GB] inherit ACTIVE '/dev/VolGroup00/LogVol01' [1.94 GB] inherit inactive '/dev/itso/mmPrimary' [18.00 GB] inherit

7. You will need to activate your newly discovery volume group

[root@palau ~]# vgchange -ay itso 1 logical volume(s) in volume group "itso" now active

8. Run fsck on the LV and then mount it for use



7.6 Other Operating Systems

The principles outlined above are equally relevant to other operating systems. You must be aware of the following when determining operating procedures

� What metadata is written to the disk (and so is copied by Copy Services)

� How does the operating system handle duplicated metadata

� How does the operating system handle ‘foreign’ metadata, i.e. disks that were originally mapped to a different server

� How does the operating system handle VDisks that were not frozen cleanly?


Draft Document for Review February 10, 2008 5:17 pm 7574 Automation.fm

Chapter 8. Automation

In this chapter, we discuss methods for automating your Copy Services solution. There are a number of methods open to you, including

� SVC Command Line Scripting� Server side scripting� SVC CIMOM Programming

Automation is the removal of human decision making from system operating processes. As a general rule, system designs will require a human operator to be involved at some stage, for instance:

� Deciding when to stop and start FlashCopy mappings or Remote Copy relationships� Deciding when to failover Remote Copy relationships (or fail back).

This chapter will also discuss the importance of logging as part of automation.

8


7574 Automation.fm Draft Document for Review February 10, 2008 5:17 pm

8.1 Running commands on the cluster

In order to automate Copy Services processes, you will need to connect to the cluster. In normal operations, this is done via the GUI or via the command line. The GUI is not an appropriate interface for automating processes, so we will not be discussing that further. All automation techniques will be via the SVC command line or via the Common Information Model Object Manager (CIMOM) (which, currently, acts as a proxy to the command line).

In this section we will refer to the ‘user agent’. This may be the CIMOM (which connects to the cluster via SSH) or a user connecting directly with an SSH client, either in an interactive mode or by means of a script.

Running commands to the cluster follow these steps:

1. Connection2. Authentication3. Submission4. Authorization5. Execution

8.1.1 Connection

Commands are submitted to the cluster during a connection session to the cluster. User agents make connections via the SSH protocol. SVC has a number of security features that affect how often you can attempt connections. These are to prevent attacks (malicious or accidental) from bringing an SVC node down. Although. at first glance, these features seem restrictive, they are relatively simple to work around in order to attain a valid connection.

Any automation system should ensure that it behaves responsibly and does not attempt to breach the connection rules. At the very least, any automation system should ensure that it can gracefully handle rejected connection attempts.

Figure 8-1 on page 209 shows how SVC’s connection restrictions work. There are 2 ‘queues’ in action: pending connections and active connections. The connection process follows this process:

1. Connection request comes into the SVC. If the Pending Connections queue has a free position, the request is added to it. Otherwise there are two possibilities; prior to version 4.2.1.0, the connection request times out; as of version 4.2.1.0, the connection is explicitly rejected.

2. Pending connections are handled in one of 2 ways:a. If any of the following are true, the connection request is rejected:

• No key is provided or the provided key is incorrect• The provided username is not admin or service• The Active Connections queue is full

In this case, a warning is returned to the SSH client as shown in Example 8-1 on page 209

b. If none of the above are true, then the connection request is accepted and is moved from the Pending Connections queue to the Active connections queue

3. Active connections are ended after any of the following events– User logs off manually– SVC’s SSH daemon recognizes that the connection has grown idle– Network connectivity fails



– The configuration node fails overIn this case, both queues will be cleared as the SHH daemon will stop and restart on a different node.

Figure 8-1 SVC’s SSH restrictions

Example 8-1 SVC command line warning due to too many logins

Using username "admin".Authenticating with public key "rsa-key-20071010"Too many logins for 'admin'.Last login: Thu Oct 25 14:15:30 2007 from 9.67.163.247

8.1.2 Authentication

This stage is represented by step 2 of the Connection process (8.1.1, “Connection”). The only authentication process available is public-private key authentication. IBM System Storage SAN Volume Controller, SG24-6423 provides detailed instructions on creating keys and uploading them to the SVC. There are a few points that we strengthen here to help make automation simpler to implement

Usernames and keysThe SVC command line only has two users defined:

� admin

Chapter 8. Automation 209


� service

Any attempt to log on as a different user will result in a rejection, regardless of whether a valid key is provided. In this way, keys and usernames are wholly independent.

The SVC GUI can have multiple usernames defined: the default username, superuser, and any others that are created. The usernames have no connection to the SVC cluster. They are, in fact, usernames for logging in to the CIMOM which runs on the Master Console. When the CIMOM logs into the SVC cluster, it will still use the admin username and the icat.ppk private key.

The SVC cluster allows you to store up to 100 keys. Any of these can be used to authenticate with the cluster, regardless of which username you use on the command line.

Once the user agent provide a valid username, the cluster receives the private key. If this matches with one of the cluster’s public keys then that key’s label is remembered and used for auditing purposes.

If the connection is made via that CIMOM, then the key label will always be the same (that being the public key to match the icat.ppk file). However, the username that was used to log on to the CIMOM is sent to the SVC cluster and is also remember for auditing purposes.

“Auditing” on page 234 discusses the auditing of commands on the SVC cluster.

8.1.3 Submission

Once connected to a cluster, the user agent may start submitting commands. The syntax is checked first of all. If this fails, an appropriate error message is returned. Any automation implementation must ensure that all commands that are submitted have correct syntax. If they do not, they must be designed to handle syntax errors. It’s much easier to design a solution that does not generate invalid syntax than one to handle all potential syntax errors.

8.1.4 Authorization

Commands that have valid syntax are next checked to see if the user agent has the authority to submit this command. Section 2.8.3, “Authorization Roles” on page 21 outlines the Authorization Roles that SVC supports. The key that was used to authenticate the connection has a role associated with it. SVC will check the submitted command against the authorization role. If the user agent is not authorized to execute this command, the following error will be returned: CMMVC6253E The command failed authorization because the session ssh key does not have the requisite role. If the user agent is authorized, the command is sent of execution.

8.1.5 Execution

When a command is executed, one of the following will occur

� The command will fail and an error message will be written to STDERR� The command will succeed and a warning will be written to STDERR � The command will succeed and a warning will be written to STDERR and information will

be sent to STDOUT� The command will succeed and information will be written to STDOUT� The command will succeed and nothing will be written.



Data written to STDOUT and STDERR by the cluster should be written to STDOUT and STDERR by your SSH client, but you will need to check this yourself.

8.2 Creating connections

Connecting to the cluster is the first step to running commands. Any automation solution will require a connection component. This component should be as robust as possible as it forms the foundation of your solution.

There are two forms of connection solution:

Transient One command is submitted per connection and the connection is closed once the command is completed.

Persistent Connection is made and stays open. Multiple commands are submitted via this single connection. These include interactive sessions and the CIMOM.

8.2.1 Transient connections

Transient connections are simple to create. The most common SSH clients allow the submission of a single command as part of their invocation. Example 8-3 and Example 8-2 show this using ssh on a AIX server and plink on a Windows server

Example 8-2 Transient connection to an SVC cluster from an AIX server

[root@heket]/>ssh -i privateKey -l admin rc-cluster-13 svcinfo lscluster -delim :id:name:location:partnership:bandwidth:cluster_IP_address:cluster_service_IP_address:id_alias00000200694052D0:rc-cluster-13:local:::9.71.42.19:9.71.42.75:00000200694052D0[root@heket]/>

Example 8-3 Transient connection to an SVC cluster from a Windows server

C:\Documents and Settings\Administrator>plink -i "C:\Documents and Settings\Administrator\My Documents\ITSO\private.ppk" -l admin 9.43.86.117 svcinfo lscluster -delim :id:name:location:partnership:bandwidth:cluster_IP_address:cluster_service_IP_address:id_alias0000020060406FCA:ITSOCL1:local:::9.43.86.117:9.43.86.118:0000020060406FCA

C:\Documents and Settings\Administrator>

These transient connections go through all 5 stages of running a command and return to the command line. You can redirect the two output streams (STDOUT and STDERR) using the operating system’s standard redirection operators to capture the responses.

These lengthy invocations can be shorted in client specific ways. The AIX SSH client allows for the use of user configuration files. The configuration file in Example 8-4 would allow you to create a transient connection as shown in Example 8-5.



Example 8-4 Sample SSH configuration file (saved as “sampleCfg”)

Host rc13HostName rc-cluster-13IdentityFile /privateKeyUser admin

Host someOtherClusterIdentityFile /someOtherKeyUser admin

Example 8-5 Transient connection to an SVC cluster using ssh using a configuration file

[root@heket]/>ssh -F sampleCfg rc13 svcinfo lscluster -delim :id:name:location:partnership:bandwidth:cluster_IP_address:cluster_service_IP_address:id_alias00000200694052D0:rc-cluster-13:local:::9.71.42.19:9.71.42.75:00000200694052D0[root@heket]/>

Shortening the plink invocation requires the creation of a PuTTY session.

In order to do this, open up the PuTTY application and enter admin@<cluster IP address> in the Host Name textbox as in Figure 8-2

Figure 8-2 Adding username and IP address to a PuTTY session

The next step is to configure the private key for this session. Select the Connection →SSH →Auth category as in Figure 8-3 on page 213



Figure 8-3 Setting the private key for a PuTTY SSH session

Select the appropriate private key using the ‘Browse...’ button and locating it in your filesystem.

The final step is to save the session for future use. A descriptive session name is advised as shown in Figure 8-4 on page 213

Figure 8-4 Saving a PuTTY session for use with plink



Once a session has been saved, it can be used for making transient connections via the command line as in Example 8-6

Example 8-6 Transient connection to an SVC cluster using plink with a PuTTY session

C:\Documents and Settings\Administrator>plink -load ITSOCL1 svcinfo lscluster -delim :id:name:location:partnership:bandwidth:cluster_IP_address:cluster_service_IP_address:id_alias0000020060406FCA:ITSOCL1:local:::9.43.86.117:9.43.86.118:0000020060406FCA

C:\Documents and Settings\Administrator>

8.2.2 Persistent connections

A persistent connection is a connection that exists beyond the submission and execution of a single command. As outlined above, the CIMOM provides a persistent connection, but it doesn’t provide direct access to the command line. In order to provide a persistent connection to the command line, you need to use multiple processes. There are as many ways to do this as there are programming languages. Example 8-12 includes a method of creating a persistent connection, using Korn Shell. Most methods involve creating a process which connects to the cluster and then writing to its STDIN stream and reading from its STDOUT and STDERR streams. The difficulty comes with detecting when the output from a command ends. This is handled in Example 8-12.

Persistent connections can be used in a number of ways.

� On a per-script basis. A script will open up a connection that will exist for the life of the script, allowing multiple commands to be submitted, but will end when the script ends

� As a stand-alone script. The connection will be opened and other scripts will communicate with this one script, to submit commands to the cluster. This allows the connection to be shared by multiple scripts. This, in turn, allows a greater number of independent scripts to access the cluster without using up all of the connection slots.

8.3 SVC command line scripting

When connected to the cluster command line, it is possible to do small amounts of automation, for instance

� Repeatedly submitting a single command to a set of SVC objects� Searching the configuration for objects conforming to certain criteria

The SVC command line is actually a highly restricted Bash shell. There is no access to the Unix commands like cd or ls. The only commands which are available are those which are built-ins, such as echo or read. In addition, redirection of inputs and outputs are not supported, but commands can be piped together.

Example 8-7 shows a sample script that will list all of the VDisks which are not online. This complements the filtervalue parameter of the lsvdisk command. The filtervalue parameter only allows matches when a property matches a value. This command line script allows matches done according to other criteria.

Example 8-7 SVC Command line script for listing all VDisks that are not online

001. svcinfo lsvdisk -nohdr | while read id name IOGid IOGname status rest



002. do003. if [ "$status" != "online" ]004. then005. echo "VDisk '$name' $$id$ is $status"006. fi007. done

Line 1 is of particular note. This line submits the svcinfo lsvdisk command and pipes the output to the read command, combined with a while command. This creates a loop which executes once per line of output from the svcinfo command.

The read command is followed by a list of variables. A line is read from the svcinfo command and the first word in that line is assigned to the first variable, the second word is assigned to the second variable and so on, with any left over words assigned to the final variable (with intervening spaces included).

We use the -nohdr parameter because we’re not interested in this information.

Lines 3 to 6 simply check the status variable. If it is not equal to ‘online’, then the information is printed to STDOUT.

8.3.1 Submitting command line scripts

Command line scripts can be submitted from an interactive prompt, if so required. However, the can also be submitted as ‘batch’ files. Example 8-9 and Example 8-10 show how this can be done with ssh or plink. Both commands submit a simple batch file shown in Example 8-8. This command lists the MDisks which are quorum MDisks; in a large configuration this can be a useful script to have.

Example 8-8 Example command line batch file ‘batchFile.sh’

svcinfo lsmdisk -nohdr | while read id name restdo svcinfo lsmdisk $id | while read key value do if [ "$key" == "quorum_index" ] then if [ "$value" != "" ] then echo "MDisk $id ($name) is quorum disk $value" fi fi donedone

Example 8-9 Submission of a batch file to SVC via ssh

[root@heket]/>ssh -F sampleCfg rc13 -T < batchFile.shMDisk 0 (mdisk0) is quorum disk 0MDisk 1 (mdisk1) is quorum disk 1MDisk 2 (mdisk2) is quorum disk 2



Example 8-10 Submission of a batch file to SVC via plink

C:\Documents and Settings\Administrator\My Documents\SVC>plink -load ITSOCL1 -m batchFile.shMDisk 0 (mdisk0) is quorum disk 0MDisk 1 (mdisk1) is quorum disk 1MDisk 2 (mdisk2) is quorum disk 2

8.3.2 An example SVC Command Line script

Example 8-11 shows a non-trivial script that can be executed directly from the SVC command line via batch submission.

The script generates a CSV table showing how many extents each VDisk gets from each MDisk

Example 8-11 Example script for generating a CSV representation of MDisk/VDisk extent usage

001. echo vDisk,mDisk,Controller,mDisk Group,Extents;002. 003. vdiskIds=(`svcinfo lsvdisk -nohdr | while read id rest; do echo -n "$id ";

done`)004. vdiskNames=(`svcinfo lsvdisk -nohdr | while read id name rest; do echo -n

"$name "; done`)005. vdiskNameMap=()006. for (( i = 0 ; i < ${#vdiskNames[@]} ; i++ ))007. do008. vdiskNameMap[${vdiskIds[$i]}]=${vdiskNames[$i]}009. done010. 011. svcinfo lsmdisk -nohdr | while read mdiskId mDiskName status mode mdgId

mdgName capacity LUN controllerName UniqueID;012. do013. svcinfo lsmdiskextent -nohdr $mdiskId | while read vdiskId extents;014. do015. echo ${vdiskNameMap[$vdiskId]},$mDiskName,$controllerName,$mdgName,

$extents;016. done017. done

Line 1 simply generates the table header row

Lines 3 and 4 work around a limitation of the bash shell. If you assign a value to a variable while inside a loop that is running from a pipe, that value is not available to you when outside the loop. A way to get around this is command substitution. Line 3 creates an array of VDisk ids by looping through all of the vdisks and printing the id. This output is captured using backticks. The parentheses turn this list of ids into an array. Line 4 does the same with VDisk names

Lines 5 to 9 create an associative array that maps a VDisk id to a VDisk name. The loop this time does not involve a pipe, so the changes to the vdiskNameMap array are available after line 9.

Line 11 begins a loop over all MDisks.



Line 13 loops over all of the MDisk Extent records for the current MDisk. A line is printed for each VDisk that uses extents from the current MDisk.

8.4 Server side scripting

Server side scripting involves scripting where the majority of the programming logic is executed on a server.

Part of server side scripting is the generation and management of connections to the SVC cluster. Section 8.4.1, “An example script for creating persistent SVC connections” shows how to create and manage a persistent connection to a cluster and how to manage requests coming from multiple scripts.

In addition to this example, there is an SVC Perl module available at http://alphaworks.ibm.com/tech/svctools (SVCTools) that handles the connection aspect of any script. As connection management is often the most complex part of any script, we recommend that you investigate this module. Currently, this module uses transient connections to submit commands to a cluster so it may not be the best approach if you plan to have multiple scripts attempting to submit commands independently.

8.4.1 An example script for creating persistent SVC connections

Example 8-12 shows an example of a script that opens up a persistent connection to a cluster and handles commands from other, external, scripts. In this section, we describe the various features of the script to give an idea of some of the issues that need to be resolved.

Example 8-12 Sample script for creating a persistent connection to an SVC cluster

001. #!/bin/ksh002. 003. #004. # SVC Command Queue005. #006. # This script implements a queue that any number of clients can write to.007. # The clients provide SVC commands and the name of a file to where the output 008. # should be written009. # Commands should be written to the queue thus:010. #011. # <command><delimiter><outFile>012. #013. # command: SVC command to be submitted to the cluster014. # delimiter: User definable character (defined in this script)015. # outFile: File to which the output of the SVC command should be written016. #017. # e.g.: echo svcinfo lscluster -delim :+/tmp/output018. #019. # This would send the 'svcinfo lscluster -delim :' command to the cluster and020. # place the output in the file '/tmp/output'021. #022. # The '+' character delimits the command and the outpue file023. 024. ###########################################################025. # Functions start


http://alphaworks.ibm.com/tech/svctools


026. ###########################################################027. #028. # Opens an SSH pipe and purges any extraneous data from the buffer029. # SSHPID gets set to the PID of the SSH process030. #031. # This function assumes that the connection will work032. #033. open_SSH_pipe()034. {035. echo "Opening SSH pipe to $CL..."036. ssh -p 22 -q admin@$CL 2>&1 |&037. SSHPID=$!038. 039. echo "Started SSH process: $SSHPID"040. 041. #042. # Purge any data from the connection043. #044. echo "Purging connection buffer..."045. svc_transaction "svcinfo lscluster" "/dev/null"046. echo "Connection buffer purged"047. }048. 049. #050. # Opens the command FIFO and connects it to file descriptor 4051. #052. open_command_FIFO()053. {054. # if the FIFO doesn't exists, create it055. if [ ! -a $FIFO ]056. then057. echo "Cannot find $FIFO so creating it"058. mkfifo $FIFO059. fi060. exec 4<$FIFO061. }062. 063. #064. # svc_transaction(command,outputfile)065. #066. # Submits the supplied command to the SVC cluster067. # Send the output to the outputfile068. #069. svc_transaction()070. {071. #072. # First, submit the command to the cluster073. # followed by a command to print the return code074. # and then a command to print the End Of Output string075. #076. print -p "$1"077. print -p "echo \$?"078. print -p "echo $COMMAND_ENDER"079. 080. #



081. # Now, read back from the SVC cluster082. #083. read -p prevLine084. while true085. do086. read -p currLine087. if [ "${currLine}" == "$COMMAND_ENDER" ]088. then089. #090. # Append the SVC Return Code to the output file091. #092. echo $RC_PREPEND$prevLine$RC_APPEND >> $2093. echo $COMMAND_ENDER >> $2094. return095. else096. echo "${prevLine}" >> $2097. prevLine=$currLine098. fi099. done100. }101. 102. ###########################################################103. # Functions end104. ###########################################################105. 106. 107. #108. # CL: cluster name - to be provided on the command line109. # FIFO: FIFO name110. # DELIM: Delimiter separating commands from their output files111. # COMMAND_ENDER: String that signifies that the SVC command output has ended112. # RC_PREPEND:113. # : Strings that wrap round the return code from the SVC cluster114. # RC_APPEND :115. #116. if [ "$1" == "" ]117. then118. echo "Please provide a cluster name"119. return 0120. fi121. CL=$1122. FIFO=/tmp/FIFO.$CL123. DELIM=+124. COMMAND_ENDER="+++COMPLETE+++"125. RC_PREPEND="+++RC:"126. RC_APPEND="+++"127. 128. 129. open_SSH_pipe130. open_command_FIFO131. 132. echo "Queue is ready..."133. while true134. do135. while read CMD <&4



136. do137. OUTFILE=`echo $CMD | cut -d $DELIM -f 2`138. echo "Outfile: $OUTFILE"139. SVCCMD=`echo $CMD | cut -d $DELIM -f 1`140. echo "SVC Command: $SVCCMD"141. #142. # Probably a good idea to check that the SSH process is still running and143. # handle it if it is not.144. #145. ps -p $SSHPID | grep -q ssh146. if [ $? = 0 ]147. then148. svc_transaction "$SVCCMD" $OUTFILE149. else150. #151. # Handle the fact that SSH is no longer running152. #153. echo "SSH pipe seems to have died"154. return 0155. fi156. done157. done158. 159. #160. # Closes FIFO, although we will never get here.161. #162. exec 4<&-

open_SSH_pipeThis function runs from lines 33 to 47 and has the job of opening up the connection to the SVC cluster. The only input is a global variable called $CL which holds the name of the cluster.

The function is quite simple, only 3 functional lines (36, 37 and 45). The first line opens up the connection, redirects STDERR to STDOUT and makes this connection a coprocess. A coprocess is a Korn shell term which means a process that is running in the background to the main process. STDIN and STDOUT of a coprocess are available to the main process for reading and writing.

Line 37 saves the process id into $SSHPID for future reference.

Line 45 runs a subroutine called svc_transaction to clear the IO buffer between the main process and the coprocess. This subroutine is described in , “svc_transaction”.

open_command_FIFOThis function runs from lines 52 to 61 and has the job of opening up a First In First Out (FIFO) file. The FIFO file acts as a queue of commands that this script must handle. External scripts add commands to this queue and this script takes them off the queue in the order that they came in and submits them to the cluster.

A FIFO file is a Unix construct. It is essentially a named pipe.

Lines 55 to 59 look to see if the FIFO file has been created (and create it if it hasn’t been).

Line 60 opens up the FIFO file with file descriptor 4 for reading. A file descriptor is simply a handle to this file; Unix shells can read and write from these file descriptors.



svc_transactionThis function runs from lines 69 to 100 and has the job of submitting a command to the SVC cluster and writing the output to a file. It takes two parameters; the first is the full command to submit to the cluster and the second is a local file, to which the output of the command should be written.

Lines 76 to 78 send information to the coprocess, i.e. the persistent connection. This is done with the print -p command.First, the command is sent. Then, a request to echo the return code of the command. Finally, a request to echo a known string. This is used to detect the end of output from the submitted command.

Lines 83 to 99 handle the gathering of the cluster’s response. The read -p reads from the coprocess and puts the information into the supplied variable. The loop is fairly simple:

1. Read one line from the coprocess and save as prevLine2. Start Of Loop3. Read next line from coprocess as currLine4. If currLine matches the end of command string

a. Write prevLine to output file with a wrapper indicating that it is the return codeb. Write the end of command string to the output filec. End subroutine

5. If not:a. Write prevLine to output fileb. Save currLine as prevLine

6. Goto step 2

Once this subroutine is complete, the IO buffer to the coprocess will be clear

Main ScriptThe main script runs from line 116 to 162 and sets up the FIFO and persistent connection and then sits waiting for commands to be added to the queue, before submitting them to the cluster.

Lines 116 to 126 set up various variables required for execution. A more mature implementation would allow these to be set via a configuration file, but this method suffices for a demonstration. The script requires that a cluster name be provided as a command line parameter.

Lines 129 and 130 open up the connection to the SVC cluster and open up a connection to the command queue (the FIFO file)

Lines 133 to 157 are the main loop. This loop monitors the command queue and handles incoming commands. The following process is followed

1. Read a line from the FIFO. This is a blocking call and will wait here until a line is available2. Split this line on a predefined delimiter. The first field is the command destined for the

cluster. The second field is the file to which all output should be directed3. Check that the connection to the cluster is still running. In this example, the script ends if

the connection ends. A more sophisticated response would be to re-attempt connection.4. If the connection is still running, the command is passed to the svc_transaction

subroutine

The final line should never be reached due to the constant loop, but its action is to close the file descriptor to the FIFO queue.



Command submissionIf multiple scripts are trying to write to the command queue that was created above, steps must be taken to ensure that the writes are atomic. That means that if two scripts attempt to write to the command queue at overlapping times, the commands written do not interleave.

Example 8-13 shows a simple C program which performs an atomic append to a file. This is a perfect utility for writing to the FIFO file, when multiple scripts are accessing at the same time. It will block on the write function on line 59 until it is free to write. All other scripts attempting to write will be blocked at the same time.

Example 8-13 Code listing for a utility that performs atomic appends to a file

001. #include <stdlib.h>002. #include <stdio.h>003. #include <fcntl.h>004. #include <string.h>005. #include <unistd.h>006. 007. int main(int argc, char *argv[])008. {009. /*010. * First parameter is the file011. * Second parameter is the text012. */013. 014. char *filename; // Name of file to append to015. int fileD; // File descriptor (for file access)016. char *text; // Text to append to the file in question017. int textLen; // Length of the text that is appended to the file018. 019. /*020. * Ensure that we have the required parameters021. */022. if (argc != 3)023. {024. printf("Usage: append <file> <text>\n");025. return -1;026. }027. 028. filename = argv[1];029. 030. /*031. * The text appended to the file will have a "newline" appended to it032. * We also need to include space in the buffer for the null-terminator033. */ 034. textLen = strlen(argv[2]) + 2;035. text = (char *) malloc((textLen)*sizeof(char));036. if(text == NULL)037. {038. printf("Could not allocate memory\n");039. return -2;040. }041. strcpy(text,argv[2]);042. strcat(text,"\n");043. 044. /*



045. * Open the file that we are appending to046. * NB: We do not attempt to create this file if it does not exist047. */ 048. fileD = open(filename, O_WRONLY | O_APPEND, 0);049. 050. if(fileD < 0)051. {052. perror("Cannot open append file for writing");053. return -2;054. }055. 056. /*057. * Write to append file and then close the file058. */ 059. write(fileD,text,textLen);060. 061. close(fileD);062. 063. return 0; 064. }

8.4.2 Handling the SVC response

From this point on, we assume that you have an implementation that allows you to submit commands to the cluster and retrieve the response in an effective manner. The next step is to be able to handle the responses usefully.

Handling errorsThe SVC Command Line Interface Guide provides a list of the possible errors that an SVC command can result in. It’s important to be able to recognize and handle these commands. Every SVC Cluster error has a unique code of the following format: CMMVCxxxxE for an error, or CMMVCxxxxW for a warning. These codes map to a single outcome, although different commands may result in identical errors. For example, the CMMVC5786E code maps on to the The action failed because the cluster is not in a stable state error, which may be returned for any command that seeks to change the cluster configuration by adding or removing an object.

It’s important that any automation solution at the very least reports these errors accurately, as they will be needed for troubleshooting. At the next level, an automation solution should be able to handle errors automatically. A large proportion of the errors are to do with invalid requests such as trying to use objects that do not exist, or trying to change the state of an object to an invalid state. These can be avoided by ensuring that your scripts check object states before changing them. They can also be used as sanity checks: if you hit one of these errors, something is obviously wrong and the user should be alerted.

svcinfo commandsThese are critical to automation as they tell you the current state of the cluster. It is important to ascertain object states before changing them if you are to implement an enterprise level automation solution.

If you use the SVCTools Perl module, then you will be able to access cluster object properties as a result of executing the functions that this module provides. In this section we discuss how to access these properties via the Korn Shell.



We strongly recommend that commands submitted to the SVC cluster by an automation solution are submitted using the -delim parameter; in general, we use the colon character (:) as the delimiter. This is just a convention and you can use any character that you would not otherwise expect in the output.

Once the output has been captured, each line represents an object. This line can then be split into words, separated by the chose delimiter. Each of these words represents a property

At this point, you have access to the information you need; write your scripts to perform the processes according to your business logic.

svctask commandsWhen these commands succeed, the feedback is minimal. For svctask commands that create objects, they will return with the id of the created object. The format of the response will be <object type>, id [<new object’s id>], successfully created. Thus, the newly created object’s id can be captured from this line. Example 8-14 shows how this might be done in a Perl program

Example 8-14 Method for capturing a newly created SVC object’s id in Perl

## $line is a SCALAR containing the response line from SVC#if ($line =~ /(.*), id \[(\d*)\], successfully created/){

$objectType = $1;$newId = $2;

}

This id is important as it allows you to execute further commands against the newly created object.

Other commands complete successfully with no feedback other than a return code of zero. For instance, commands that change an object will return no feedback if they are successful. It may be prudent to request their details before proceeding, to ensure that any data structures that you have created are in sync with the cluster objects’ states.

8.5 SVC CIMOM programming

Section 8.4, “Server side scripting” on page 217 discusses automation by connecting directly to the cluster command line. As outlined in section 8.2, “Creating connections” on page 211, this requires you to manage connections in order to prevent failures when the connection limit is hit. In addition to this, it is difficult to manage connections without sacrificing the user authentication process.

CIMOM Programming provides the opportunity to handle multiple connections from multiple sources while maintaining security. CIM Clients connect to the CIMOM with a username and password and then invoke methods to execute commands. CIM Agent’s Developer Reference, SC26-7904 contains details of the CIM Classes, Associations and Methods that you need, in order to interact with the cluster.

Creating a CIM Client is straightforward, once you have found a suitable framework. Our investigations have uncovered a few, but the Java™ WBEM Service project seems to be one of the more widely referenced ones. This can be found at



http://wbemservices.sourceforge.net/. There are implementations in other languages, including C++ and Python, but our experience with Java directed us to this one.

Example 8-15 shows a simple Java program that connects to an SVC CIMOM. It does nothing more than connect, but this is the first step in automating with the CIMOM.

Example 8-15 Java program for connecting to an SVC CIMOM

import java.util.*;

import javax.wbem.cim.*;import javax.wbem.client.*;

public class ITSOClient {

public static void main(String[] args){

String username = args[0];String password = args[1];String masterConsoleIP = args[2];String masterConsoleSecurePort = args[3];UserPrincipal user = new UserPrincipal(username);PasswordCredential pwd = new PasswordCredential(password);CIMNameSpace ns = new CIMNameSpace("https://”+

masterConsoleIP+”:”+masterConsoleSecurePort+”/root/ibm");

CIMClient client = null;try{

System.out.println("Connecting to CIMOM");client = new CIMClient(ns,user,pwd);

} catch (CIMException e){

// Handle the CIM Exceptione.printStackTrace();

}}

Once connected to the CIMOM, the next step is to identify the cluster that you wish to work with. Before we discuss that, we will discuss some CIM Agent concepts.

8.5.1 CIM concepts

A full definition of the CIM specification is beyond the scope of this book, but a brief description of a few concepts here will allow you to experiment with this technology. This section assumes some familiarity with Object Oriented Programming (OOP) concepts.

Namespace The scope within which all of the CIM classes are defined. For SVC, this contains all of the SVC CIM classes that your scripting will manipulate

Class The definition of an object within the SVC hierarchy. It is directly equivalent to an OOP class.


http://wbemservices.sourceforge.net/


Instance An instance of an object that is a member of a CIM class. It is directly equivalent to an OOP object and contains all the properties of an object in the SVC configuration

Method A way to implement a function on a class

Property An attribute used to characterize the instance of a class

Qualifier A value that defines the behavior of a property, classes and methods

Reference A pointer to a CIM instance. This is a typed string that is identified as being a reference to the instance in question

Object Path A formatted string that is used to access namespaces, classes and instances

Association A class with two references which defines a relationship between the two referenced classes

The CIM classes are part of a broader, cross-vendor specification and so do not always map exactly onto SVC object types. CIM Agent’s Developer Reference, SC26-7904 shows all of the relevant classes.

8.5.2 How SVC concepts map to CIM concepts

In order to administer Copy Services via the CIMOM, it’s important to understand how to translate between SVC and CIM concepts. Table 8-1 shows how these relate to one another.

Table 8-1 Relating SVC concepts to CIM concepts

SVC Concept CIM concept

CIM Name CIM Concept

Cluster IBMTSSVC_Cluster Class

Cluster ID ElementName Property

VDisk IBMTSSVC_StorageVolume Class

VDisk ID DeviceID Property

FlashCopy Mapping IBMTSSVC_LocalStorageSynchronized Association

FlashCopy Mapping Status SyncState Property

mkfcmap AttachReplica Method

preparefcmap ModifySynchronization Method

startfcmap ModifySynchronization Method

Remote Copy relationship IBMTSSVC_SyncCopyStorageSynchronizedSet

Association

Remote Copy relationship state NativeState Property

mkrcrelationship AttachReplica Method

startrcrelationship ModifySynchronization Method



8.5.3 Example script to create and start a FlashCopy mapping

Example 8-16 is the main method from a Java class that is designed to create and start a FlashCopy mapping. The other methods that are called are described in later examples. This serves as a demonstration of how CIMOM Methods can be used to control the cluster. In this section, method refers to a Java method. Method (initial capital) refers to a CIM Method.

Example 8-16 Java main method for creating and starting a FlashCopy Mapping

/* * FC Mapping states */private static UnsignedInt16 INITIALIZED = new UnsignedInt16(2);private static UnsignedInt16 PREPARING = new UnsignedInt16(3);private static UnsignedInt16 PREPARED = new UnsignedInt16(4);

public static void main(String[] args) throws CIMException{

/* * First step is to connect to the CIMOM */UserPrincipal user = new UserPrincipal("superuser");PasswordCredential pwd = new PasswordCredential("itso13sj");CIMNameSpace ns = new CIMNameSpace("https://9.43.86.115:5989/root/ibm");

CIMClient client = null;

client = new CIMClient(ns,user,pwd);

/* * Next, select the cluster that we're interested in */CIMInstance chosenCluster = getCluster("ITSOCL1",client);

/* * At this point, the relevant cluster has been selected * and 'chosenCluster' is a CIMInstance of this cluster * * Get the Config Service of this cluster */CIMObjectPath cService = getConfigService(chosenCluster, client);

/* * Now, get all of the VDisks in this cluster */Map<Integer,CIMObjectPath> vdisksById = getVDisks(chosenCluster,client);

/* * Select the FlashCopy Source * * In this case, VDisk 10 is our source * VDisk 11 is our target */CIMObjectPath fcSrc = vdisksById.get(new Integer(10));CIMObjectPath fcTgt = vdisksById.get(new Integer(11));



/* * Now, create FC Mapping */CIMValue rc = makeFlashCopyMapping("CIMOMTestMap", fcSrc, fcTgt, cService,

client,false);

/* * Now that this has been created, we need to get an * Object Path to the newly created Association */List<CIMObjectPath> fcMaps = getFCMappings(fcSrc, client);CIMObjectPath fcMapping = fcMaps.get(0);

/* * Now, we prepare the FC Mapping */CIMArgument[] outArgs = new CIMArgument[2];rc = prepareFCMapping(cService, fcMapping, client, outArgs);System.out.println("Got value:"+

Integer.toHexString(Integer.parseInt(rc.toString())));

/* * Loop until it is prepared */CIMValue fcMapState = new CIMValue(PREPARING);while(fcMapState.equals(new CIMValue(PREPARING))){

CIMInstance fcMapInfo = client.getInstance(fcMapping);fcMapState = fcMapInfo.getProperty("SyncState").getValue();

}

/* * Now start the FC Mapping */rc = startFCMapping(cService, fcMapping, client, outArgs);System.out.println("Got value:"+

Integer.toHexString(Integer.parseInt(rc.toString())));}

The assumption in this example is that your Java program is designed to control the same cluster every time. It is a relatively simple process to make it more flexible, but that is left to you.

getCluster methodExample 8-17 shows the getCluster method. This method returns the CIM Instance which corresponds to the cluster with the supplied name.

It does this by enumerating all of the instances of the class IBMTSSVC_Cluster and then checking the name of each one. When one is found which matches the supplied name, an object path to that instance is returned.

Example 8-17 getCluster method

static private CIMInstance getCluster(String clusterName, CIMClient client) throws CIMException



{CIMInstance chosenCluster = null;Enumeration<CIMInstance> clusters =

client.enumerateInstances(new CIMObjectPath("/root/ibm:IBMTSSVC_Cluster"));

while(clusters.hasMoreElements()){

CIMInstance possibleCluster = clusters.nextElement();String possibleName =

possibleCluster.getProperty("ElementName").getValue().toString();

if(possibleName.equals("\""+clusterName+"\"")){

chosenCluster = possibleCluster;}

}

return chosenCluster;}

getConfigService methodExample 8-18 shows the getConfigService method. The CIM_StorageConfigurationService class has no direct equivalent in an SVC, but an Instance of this Class is required for invoking Methods against.

In this method, we request all of the Instances that are associated with the supplied cluster. The Association that connects a cluster to its configuration service is CIM_HostedService. A cluster will only have configuration service associated with it, so we can select the first Object Path in the enumeration.

Example 8-18 getConfigService method

static private CIMObjectPath getConfigService(CIMInstance cluster, CIMClient client) throws CIMException{

Enumeration<CIMObjectPath> configServices = null;configServices = client.associatorNames(

cluster.getObjectPath(),"CIM_HostedService", "CIM_StorageConfigurationService", null, null);

return configServices.nextElement();}

getVDisks methodExample 8-19 shows the getVDisks method. This returns a Map, relating VDisk ids (as Integers) to IBMTSSVC_StorageVolume Object Paths.

The method requests all of the IBMTSSVC_StorageVolume Instances that are associated with the provided cluster Instance.



Example 8-19 getVDisks method

static private Map<Integer,CIMObjectPath> getVDisks(CIMInstance cluster, CIMClient client) throws CIMException{

Enumeration<CIMObjectPath> vdisks = client.associatorNames(cluster.getObjectPath(), null, "IBMTSSVC_StorageVolume", null, null);

Map<Integer,CIMObjectPath> vdisksById = new HashMap<Integer, CIMObjectPath>();

while(vdisks.hasMoreElements()){

CIMObjectPath vdiskOP = vdisks.nextElement();CIMValue vdiskId = vdiskOP.getKey("DeviceID").getValue();String idAsString = vdiskId.toString();String idNoQuotes = idAsString.substring(1, idAsString.length()-1);vdisksById.put(Integer.parseInt(idNoQuotes), vdiskOP);

}

return vdisksById;}

makeFlashCopyMapping methodExample 8-20 shows the makeFlashCopyMapping method. It does this by invoking the AttachReplica against the cluster configuration service.

CIM Methods take typed parameters. In this method, you can see the use of the argRef, argString and argUint16 methods. These are defined later in this section, but they act as shortcuts to generating the required arguments for the CIM Method.

Note the use of CopyType. The AttachReplica method is used for FlashCopy, Metro Mirror and Global Mirror and it’s the CopyType argument which indicates which type is required.

Example 8-20 makeFlashCopyMapping

static private CIMValue makeFlashCopyMapping(String name, CIMObjectPath source, CIMObjectPath target, CIMObjectPath configService, CIMClient client,boolean autodelete) throws CIMException

{CIMArgument src = argRef("SourceElement", source, "IBMTSSVC_StorageVolume");CIMArgument tgt = argRef("TargetElement", target, "IBMTSSVC_StorageVolume");CIMArgument fcName = argString("ElementName",name);CIMArgument type = argUint16("CopyType",autodelete?5:4);

CIMArgument[] inArgs = {src,tgt,fcName,type};CIMArgument[] outArgs = new CIMArgument[1];

CIMValue rc = client.invokeMethod(configService,



"AttachReplica", inArgs, outArgs);

return rc;}

getFCMappingsExample 8-21 shows the getFCMappings method. This method returns a List of all the FCMappings associated with the provided vdisk.

It requests a list of all of the Associations which reference the provided IBMTSSVC_StorageVolume. Currently, all of the Java WBEM Services methods of this type return Enumerations. This method converts this to a List for ease of use.

Example 8-21 getFCMappings method

static private List<CIMObjectPath> getFCMappings(CIMObjectPath vdisk, CIMClient client) throws CIMException{

Enumeration<CIMObjectPath> assocs = client.referenceNames(vdisk,"IBMTSSVC_LocalStorageSynchronized",null);

return Collections.list(assocs);}

prepareFCMapping methodExample 8-22 shows the prepareFCMapping method. This method prepares a FlashCopy Mapping.

Much like the AttachReplica Method, the ModifySynchronization Method is used to control FlashCopy, Metro Mirror and Global Mirror; it’s the Operation parameter which indicates what you actually want to do.

Example 8-22 prepareFCMapping

private static CIMValue prepareFCMapping(CIMObjectPath configService, CIMObjectPath fcMapping, CIMClient client,CIMArgument[] outArgs) throws CIMException

{CIMArgument operation = argUint16("Operation", 6);CIMArgument synch = argRef("Synchronization",

fcMapping,"IBMTSSVC_FlashCopyStorageSynchronized");

CIMArgument[] inArgs = new CIMArgument[]{operation,synch};outArgs = new CIMArgument[2];

return client.invokeMethod(configService, "ModifySynchronization", inArgs, outArgs);



}

startFCMapping methodExample 8-23 shows the startFCMapping method. This method starts a FlashCopy Mapping.

As in Example 8-22, this invokes the ModifySynchronization Method. Other than using a different Operation parameter, this method is identical.

Example 8-23 startFCMapping method

private static CIMValue startFCMapping(CIMObjectPath configService, CIMObjectPath fcMapping, CIMClient client,CIMArgument[] outArgs) throws CIMException

{CIMArgument operation = argUint16("Operation", 4);CIMArgument synch = argRef("Synchronization",

fcMapping,"IBMTSSVC_FlashCopyStorageSynchronized");

CIMArgument[] inArgs = new CIMArgument[]{operation,synch};outArgs = new CIMArgument[2];

return client.invokeMethod(configService, "ModifySynchronization", inArgs, outArgs);

}

Argument generatorsThis class uses 3 argument generators:

� argUint16 - Example 8-24

� argString - Example 8-25

� argRef - Example 8-26

argUint16 methodThis method returns an unsigned 16-bit integer typed argument

Example 8-24 argUint16 method

static private CIMArgument argUint16(String name, int arg){

return new CIMArgument(name,new CIMValue(

new UnsignedInt16(arg),new CIMDataType(CIMDataType.UINT16))

);}

argString methodThis method returns a string typed argument



Example 8-25 argString method

static private CIMArgument argString(String name, String str ){

return new CIMArgument(name,new CIMValue(

str,new CIMDataType(CIMDataType.STRING))

);}

argRef methodThis method returns a Reference typed argument. It is a Reference to the Instance that the provided Object Path indicates.

Example 8-26 argRef method

static private CIMArgument argRef(String name, CIMObjectPath path, String className )

{return new CIMArgument(

name,new CIMValue(

path,new CIMDataType(className))

);}

8.6 Logging

The object of automation is to remove the human factor from performing systems operations. Many steps in a process follow directly from their previous step with little or no decision making.

When performed by a human operator, these processes have a natural ‘logging’. A user will recall which steps were taken and the outcome of each step. When the human operator is removed from a process, it is important to log each step to aid in auditing and troubleshooting.

The SVC audit log will register whenever an svctask command is successfully executed. The SVC error log will also register when configuration events complete. These are useful, but it is important to generate your own logging as part of your automation solution.

You should log the following actions with a time and date stamp:

� Submission of any command to the SVC, including the full SVC CLI command or CIM Method invocation

� Completion of any command sent to the SVC (return code and any message)� ID of any generated SVC objects



� If the automation solution ‘gives up’, then full explanation of why.

In order to tally server, SVC and automation log, it’s important that all systems have a correctly set clock. The SVC cluster does not support the NTP protocol, so the only way to synchronize it with an NTP server is to use a proxy.

On a Unix server, with access to a cluster, you can use the commands in Example 8-27 on page 234 to synchronize the SVC clock with your local clock. This is a script which requires user interaction, because the local server’s timezone may map on to multiple possible cluster time zones. This script can be altered to be fully automatic by hard-coding the timezone code.

Example 8-27 Simple script for setting the SVC cluster clock to the local time

#!/usr/bin/ksh

sshInvocation="ssh -F sampleCfg rc13"possTZs=`$sshInvocation svcinfo lstimezones -delim : | grep :\`date +%Z\``

echo "Please select a timezone"for TZ in $possTZsdo echo $TZdoneecho "Selection (numerical code): \c"read choiceecho You chose: $choice

$sshInvocation svctask settimezone -timezone $choice$sshInvocation svctask setclustertime -time `date +%m%d%H%M%Y`

8.7 Auditing

When commands are successfully executed on the SVC cluster an entry is made in the SVC audit log. Example 8-28 shows a sample audit log entry. The audit log steadily fills up with entries until it is dumped manually with the svctask dumpauditlog command, or it gets too full and dumps automatically. The entries are dumped to an audit log dumpfile.

Example 8-28 Sample audit log entry

Auditlog Entry 1086Sequence Num: 1086Timestamp: Thu Apr 5 08:49:13 2007

: Epoch + 1175780953Cluster User: adminSSH Label: AdministratorICAT User: adminResult Code: 0Result Obj ID: Action Cmd: svctask chcluster -icatip 10.2.10.96:9080

The Auditlog Entry number indicates the position of the entry in the audit log dumpfile. It is unique within that file. The Sequence Num is a unique number which continually increments. If



the entry given in Example 8-28 is the last in a dumpfile, then the first entry in the next dumpfile will have an Auditlog Entry number of 1 and a Sequence Num of 1087.

The Timestamp is the time that the command was run. Epoch refers to a standard method of measuring time in a Unix environment. It is the number of seconds since midnight UTC of January 1st, 1970, not counting leap seconds.

The Cluster user is the username used to access the cluster CLI. This will either be service or admin. This will exist, even if the GUI or CIMOM was used, because, ultimately, the CIMOM accesses the cluster via the CLI.

The SSH Label refers to the id given to the public key that authenticated the connection. It’s a useful way to track who submitted a command. If private keys are truly kept private, then this can be an effective audit policy.

The ICAT User is populated when the connection to the cluster is made via the CIMOM. The username that is used to connect to the CIMOM is the one that is logged.

The Result Code is either 0 or 1, depending on whether the command is successful.

If the command is a mkxxx command, then Result Obj ID will hold the ID of the object that was created.

The final line, Action Cmd, lists the actual command that was submitted.

All of this information is sufficient to audit exactly who submitted which commands, and when. The audit log is only as useful as policies put in place to control access. If private keys and usernames are closely controlled, then the audit log give a precise history of who contacted the cluster.


Draft Document for Review February 10, 2008 5:17 pm 7574bibl.fm

Related publications

The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this book.

IBM Redbooks

For information about ordering these publications, see “How to get Redbooks” on page 238. Note that some of the documents referenced here may be available in softcopy only.

� IBM System Storage SAN Volume Controller, SG24-6423-05

� Get More Out of Your SAN with IBM Tivoli Storage Manager, SG24-6687

� IBM Tivoli Storage Area Network Manager: A Practical Introduction, SG24-6848

� IBM System Storage: Implementing an IBM SAN, SG24-6116

� Introduction to Storage Area Networks, SG24-5470

� SAN Volume Controller: Best Practices and Performance Guidelines, SG24-7521

� Using the SVC for Business Continuity, SG24-7371

� IBM System Storage Business Continuity: Part 1 Planning Guide, SG24-6547

� IBM System Storage Business Continuity: Part 2 Solutions Guide, SG24-6548

Other resources

These publications are also relevant as further information sources:

� IBM System Storage Open Software Family SAN Volume Controller: Planning Guide, GA22-1052

� IBM System Storage Master Console: Installation and User’s Guide, GC30-4090

� IBM System Storage Open Software Family SAN Volume Controller: Installation Guide, SC26-7541

� IBM System Storage Open Software Family SAN Volume Controller: Service Guide, SC26-7542

� IBM System Storage Open Software Family SAN Volume Controller: Configuration Guide, SC26-7543

� IBM System Storage Open Software Family SAN Volume Controller: Command-Line Interface User's Guide, SC26-7544

� IBM System Storage Open Software Family SAN Volume Controller: CIM Agent Developers Reference, SC26-7545

� IBM TotalStorage Multipath Subsystem Device Driver User's Guide, SC30-4096

� IBM System Storage Open Software Family SAN Volume Controller: Host Attachment Guide, SC26-7563


7574bibl.fm Draft Document for Review February 10, 2008 5:17 pm

Referenced Web sites

These Web sites are also relevant as further information sources:

� IBM TotalStorage home page:

http://www.storage.ibm.com

� SAN Volume Controller supported platform:

http://www-1.ibm.com/servers/storage/support/software/sanvc/index.html

� Download site for Windows SSH freeware:

http://www.chiark.greenend.org.uk/~sgtatham/putty

� IBM site to download SSH for AIX:

http://oss.software.ibm.com/developerworks/projects/openssh

� Open source site for SSH for Windows and Mac:

http://www.openssh.com/windows.html

� Cygwin Linux-like environment for Windows:

http://www.cygwin.com

� IBM Tivoli Storage Area Network Manager site:

http://www-306.ibm.com/software/sysmgmt/products/support/IBMTivoliStorageAreaNetworkManager.html

� Microsoft® Knowledge Base Article 131658:

http://support.microsoft.com/support/kb/articles/Q131/6/58.asp

� Microsoft Knowledge Base Article 149927:


� Sysinternals home page:

http://www.sysinternals.com

� Subsystem Device Driver download site:

http://www-1.ibm.com/servers/storage/support/software/sdd/index.html

� IBM TotalStorage Virtualization Home Page:

http://www-1.ibm.com/servers/storage/software/virtualization/index.html

How to get Redbooks

You can search for, view, or download Redbooks, Redpapers, Technotes, draft publications and Additional materials, as well as order hardcopy Redbooks, at this Web site:

ibm.com/redbooks

Help from IBM

IBM Support and downloads

ibm.com/support

IBM Global Services




http://www.ibm.com/support/

http://www.ibm.com/support/

http://www.storage.ibm.com



http://www.chiark.greenend.org.uk/~sgtatham/putty

http://oss.software.ibm.com/developerworks/projects/openssh

http://www.openssh.com/windows.html

http://www.cygwin.com

http://www-306.ibm.com/software/sysmgmt/products/support/IBMTivoliStorageAreaNetworkManager.html



http://www.sysinternals.com

http://www-1.ibm.com/servers/storage/support/software/sdd/index.html

http://www-1.ibm.com/servers/storage/software/virtualization/index.html

Draft Document for Review February 10, 2008 5:17 pm 7574bibl.fm

ibm.com/services

Related publications 239

http://www.ibm.com/services/

http://www.ibm.com/services/

7574bibl.fm Draft Document for Review February 10, 2008 5:17 pm


Draft Document for Review February 10, 2008 5:17 pm 7574IX.fm

Index

AAIX 175–176, 178, 211

importvg 179LVM 178recreatevg 178–179

application testing 100, 142, 149, 152asynchronous 15, 23–24, 158attributes 101authenticate 210authentication 209, 224automation 35, 84, 207–209auxiliary 28–30, 55, 67–68, 93auxiliary VDisk 28, 55, 93availability 42

Bbackground copy 3, 20, 26–28, 48, 70, 92, 94–95, 120, 126–127background copy rate 96backup 12, 22, 87, 93, 118, 142, 145balance 7bandwidth 7, 20, 26–28, 48, 69–70, 96, 211–212, 214bitmaps 17, 92, 102, 104boot 202

Ccache 2, 4, 7–9, 101–102, 106, 134, 159, 168, 175, 188–189caching 7–8, 10, 106capacity 17, 21, 27, 102, 158–159, 163, 175, 191, 200, 216channels 26chpartnership 22, 70, 90chrcconsistgrp 22, 79, 90chrcrelationship 22, 74–76CIMOM 207–208, 210CLI 68, 72, 84, 104, 117, 157, 233, 235

commands 87, 89cluster 3–4, 16, 20–21, 24–26, 47–49, 103, 105, 175, 179, 191, 208, 210–211

creation 25, 50IP address 212

cluster partnership 25, 29, 47, 50–51clusters 24–26, 46–48, 200, 229command line interface 45–46concepts 225–226configuration 3, 6, 18, 21, 27, 31–32, 62, 104–105, 174–176, 209, 211–212connected 31–33, 75, 83, 210, 214, 225connected state 34–36connectivity 27, 40, 208consistency 7, 12–13, 30–31, 33, 46, 53, 56, 92, 94–95, 124, 128, 159, 177–178, 185

© Copyright IBM Corp. 2007. All rights reserved.

consistency freeze 34consistency group 15–16, 30–31, 33, 56–57, 59, 94–95, 97, 160, 178, 193–194consistency groups 16, 31, 33, 38, 61, 63–64, 97–98, 105consistent 12–13, 15, 25–26, 28, 34, 57–59, 92–93, 95, 118, 127, 142, 175ConsistentDisconnected 37, 89ConsistentStopped 35, 40, 42–43, 73, 88–89ConsistentSynchronized 35, 43, 73, 89constrained link 30copy bandwidth 27–28copy process 26, 35–36, 47–48, 56, 92, 94, 105, 120, 127, 142copy rate 3, 49, 96, 120copy service 17, 102Copy Services 1, 5–7, 39, 61, 174, 188, 205, 207–208, 226

Windows 2000 188Windows 2000 limitations 188

copying state 3, 36, 108corruption 33, 39create a FlashCopy 149, 163

Ddata consistency 30data flow 10data mining 93, 100delete

a FlashCopy 149a VDisk 149

dependent writes 30, 39, 95detected 26, 39disconnected 31–33, 74–75, 88disk 2, 4, 8, 12, 20, 24, 28, 30, 59, 77, 82, 175, 179, 188, 215–216distance 3, 24–25, 54, 68documentation 17dual site 84dynamic volumes 188

Ee-mail xvempty state 38error 25–26, 33, 77, 93, 108, 111, 129, 138, 158, 210, 223, 233error log 37errors 26, 35–36, 87, 158, 174, 210, 223event 27, 32, 34, 108, 135events 106–108, 208, 233excludes 27extent 3, 103, 216extents 216–217

241

7574IX.fm Draft Document for Review February 10, 2008 5:17 pm

Ffailed node 42failover 82–84, 207file system 175, 181, 195FlashCopy 2, 6, 11–12, 91–93, 117–119, 177–179, 207, 226–227

bitmap 17–18, 101–102, 104, 120commands 22, 94, 106create 91–92, 118, 130Delete 108indirection layer 101mapping 15, 17, 20, 92, 94, 118–120, 174Modify 130prepare 106–107, 124, 137source 16, 18, 20, 92, 94, 97, 117, 120, 179, 188Start 124, 137, 178, 194target 18, 92, 94, 122, 142, 148, 178–179, 184

FlashCopy mapping 2, 15, 17–18, 97, 118–120, 227FlashCopy mappings 15, 18, 103, 118, 123, 161flexibility 22, 93, 103fmtdisk 30focal point 26focal point node 26foreground I/O latency 27format 40, 223–224

GGlobal xiv, 2–3, 6, 12, 15, 23–25, 46, 48–49, 93, 102, 105, 158, 174, 230–231Global Mirror 15–17, 23–24, 28, 46, 48–49, 102Global Mirror relationship 22Globally Unique Identifier see GUIDGM 158grain 2, 17, 20, 41–43, 100–102, 120, 126grain size 2, 17, 100–101, 120grains 2–3, 17, 41, 43, 96, 100, 102, 157, 169granularity 101GUI 21, 45–47, 117–118, 142, 208, 210, 235GUID 188

Hhelp xv, 82, 209host

configuration 184, 195creating 55definitions 184

II/O group 3, 24, 28, 38, 96, 102, 104, 126ICAT 21, 234–235identical data 30idling 28, 35–36, 78, 89, 203IdlingDisconnected 36, 89importvg 179, 185inconsistent 14–15, 31, 33–34, 93, 95inconsistent stopped state 36InconsistentCopying 36, 73, 88InconsistentDisconnected 37, 89

InconsistentStopped 35, 73, 88–89indirection 101indirection layer 101install 46integrity 30intercluster 20–21, 24–26, 46–48, 157intercluster link 26–28interval 8intracluster 24, 27, 61–62iogrp 104, 162, 177, 193IP address 212

Llatency 3, 16, 18–19, 27, 39, 96, 101LBA 27, 42, 101LDM 188, 195LDM database 188limiting factor 8Linux 200, 203–204local cluster 26–27, 75log 21–22, 35, 37, 85, 95, 174–175, 177, 210, 233–234logged 21, 25, 70, 235Logical Block Address 101Logical Disk Manager 188Logical Volume Manager see LVMlogins 26, 209logs 21, 95, 208, 210LU 204LUN 11–12, 216LUNs 11, 83–84LVM 178LVM data structures 179

Mmanagement xiv, 3, 7, 41, 95, 217mapping 2–3, 15, 17–18, 87, 92, 94, 96, 118, 120–121, 174–175, 192, 227maps 41–42, 111, 216, 223master 28–30, 67, 71–72, 159–160, 202master VDisk 28, 77MDisk 3–4, 215–217

showing 216memory 2, 8, 17, 102, 104, 222Metro 2–3, 6, 12, 15, 23–24, 28, 46, 48, 54, 102, 105, 117, 157–158, 200–202, 230–231Metro Mirror 3, 17, 20, 23–24, 30, 46, 48, 75, 102, 157–158, 160, 200, 202–203Metro Mirror consistency group 85Metro Mirror relationship 24, 83, 158, 203–204mirrored 28, 39, 188mkpartnership 69–70, 90mkrcconsistgrp 75–76, 90, 202mkrcrelationship 29–30, 71–72, 90, 202, 226mount 24, 177, 179, 181MPIO 175

Nnew mapping 107, 118, 130, 132


Draft Document for Review February 10, 2008 5:17 pm 7574IX.fm

NOCOPY 120, 139node 3, 26–27, 42, 175, 177, 183, 208–209

failure 42nodes 3, 25–27, 96, 103, 204non-redundant 26

Ooffline 40, 42, 84–85, 104, 106–107, 121online xv, 40–42, 72–73, 76, 104, 107, 121, 125, 158, 174, 176, 191, 214–215ordering 12, 20, 33, 39, 111overwritten 14, 33, 127

Ppartnership 25, 29, 47–49, 211–212, 214path offline 40paths 43, 175peak 21, 28per cluster 16, 103, 105performance 3, 7, 15, 19Physical Volume Identifier see PVIDplanning 5–6, 9, 177, 202plink 211–213point-in-time copy 34, 93, 158policy 235PPRC

configuration limits 38prepared state 178, 194primary 20, 24–25, 27, 56, 58, 64, 158–160, 174, 178–179priority 37, 97, 120, 126, 145private key 21, 209–210, 212production VDisk 28, 149, 151productivity xvpublic key 21, 209–210, 235PuTTY 212–214PuTTY application 212PuTTY session 212–213PVID 178PVIDs 178–179, 185

Qquickly 8, 158quorum disk 215–216

Rrecreatevg 178–179

AIX 179recreatevg command 179Redbooks Web site 238

Contact us xvredundant 9, 26registry 188relationship 12–14, 24, 26, 28, 46–47, 52, 92, 96, 102, 130, 139, 146, 174, 179, 202, 226remote cluster 20, 26, 50, 52, 55removed 38, 75, 92, 233restarting 120

rmrcconsistgrp 79–80, 90rmrcrelationship 75, 90

Ssample script 214SAN xiii–xiv, 1, 7–8, 10, 25, 46, 48, 68, 174SAN Volume Controller xiv, 1, 72, 90scripting 177, 202, 207, 214, 217scripts 1, 96, 214–215, 217SDD 87, 89, 175, 191, 200secondary 20, 24–25, 27, 48–49, 56, 93, 117, 157–158, 174, 178–179secondary site 27security 195, 208, 224sequential 19, 121serialization 20shared 16–17, 94, 102, 214shells 220site 20, 27, 33, 46, 82–83, 157SNIA xivSNMP 35Software 1software upgrade 43solution 6–7, 9, 142, 158, 207, 210–211sorting 54source 2–3, 20, 84, 92, 94–95, 118, 120–121, 174–175, 178, 227, 230space 2–4, 7–8, 16, 41, 101–102, 104, 120, 158, 196, 222split 96, 101–102, 224SSH 21, 208–209, 211SSH client 208, 211SSH keys 21startrcrelationship 22, 29–30, 72–73, 90, 226state 3, 12–13, 24, 26, 28, 52, 57–58, 96, 101–102, 123–125, 176, 178–179, 223, 226

consistent 30, 34, 57ConsistentDisconnected 37ConsistentStopped 35ConsistentSynchronized 35empty 30, 38, 76, 79, 129, 165idling 36–37, 78IdlingDisconnected 36inconsistent 35–36InconsistentCopying 36InconsistentDisconnected 37InconsistentStopped 35synchronized 24, 34, 57–58, 107

state fragments 32State overview 31state transitions 109states 25, 31, 34, 73, 75, 77, 106, 109, 134, 223–224, 227statistics 19stoprcconsistgrp 22, 78, 88–89, 203stoprcrelationship 22, 30, 35, 74, 78, 90storage virtualization xivstriped 121, 158–159, 176–177, 188superuser 210, 227SVC xiii, 1–3, 7–9, 24–26, 46–48, 91, 93, 96, 117, 120,

Index 243

7574IX.fm Draft Document for Review February 10, 2008 5:17 pm

145, 174–175, 179, 207–209SVC cluster 26, 30, 35, 50, 52, 55, 103, 194, 200, 210–212SVC configuration 226svcinfo 22, 40, 69–70, 72, 104, 116, 158–160, 175–176, 178, 211–212, 214svcinfo lsmdiskextent 216svctask 22, 69–71, 104, 161–163, 177–179, 224, 233–234svctask mkfcmap 162, 178, 193–194switchrcconsistgrp 22, 81, 90switchrcrelationship 22, 90synchronization 15, 29–30, 35, 93, 179, 202Synchronized 34–35synchronized 15, 30–31, 34, 55–56, 66, 93, 158synchronizing 29, 77

Ttarget 2–3, 20, 34, 82–84, 92–94, 118, 120, 122, 174, 177–178, 227, 230tasks 1, 61, 82, 94, 179, 194time 2, 8–9, 11, 31, 33–34, 56, 61, 75, 92–94, 118, 120, 122, 174, 178, 181, 216, 222, 228transitions 35–37, 109

Uupgrade 43

VVDisk 2–3, 7, 10–11, 24–25, 28, 52, 54–55, 92–94, 118, 120–121, 173–175, 215–217

assigning 107creating 28, 99, 121information 7, 101, 178working with 149

VGDA 178VGID 178virtualization xivvolume group 178–179, 185Volume Group Descriptor Area see VGDAVolume Group Identifier see VGID

WWindows

dynamic volumes 188Windows 2000

Copy Services 188limitations 188

Windows 2003 188, 191–192Write ordering 12, 33write ordering 12, 39write through mode 106writes 8–9, 11, 25, 27–28, 48–49, 95, 107, 115, 175, 188, 222write-through mode 12WWPN 176, 191, 200


To determine the spine w

idth of a book, you divide the paper PP

I into the number of pages in the book. A

n example is a 250 page book using P

lainfield opaque 50# smooth w

hich has a PP

I of 526. Divided

250 by 526 which equals a spine w

idth of .4752". In this case, you would use the .5” spine. N

ow select the S

pine width for the book and hide the others: S

pecial>C

on

ditio

nal

Text>Sh

ow/H

ide>S

pin

eSize(-->H

ide:)>S

et . Move the changed C

onditional text settings to all files in your book by opening the book file with the spine.fm

still open and File>Im

po

rt>Fo

rmats the

Conditional Text S

ettings (ON

LY!) to the book files.

Draft D

ocument for R

eview F

ebruary 10, 2008 5:17 pm7574sp

ine.fm

245

(0.1”spine)0.1”<

->0.169”

53<->

89 pages

(0.2”spine)0.17”<

->0.473”

90<->

249 pages

(1.5” spine)1.5”<

-> 1.998”

789 <->

1051 pages

(1.0” spine)0.875”<

->1.498”

460 <->

788 pages

(0.5” spine)0.475”<

->0.873”

250 <->

459 pages





(2.0” spine)2.0” <

-> 2.498”

1052 <->

1314 pages

(2.5” spine) 2.5”<

->nnn.n”

1315<->

nnnn pages

To determine the spine w

idth of a book, you divide the paper PP

I into the number of pages in the book. A

n example is a 250 page book using P

lainfield opaque 50# smooth w

hich has a PP

I of 526. Divided

250 by 526 which equals a spine w

idth of .4752". In this case, you would use the .5” spine. N

ow select the S

pine width for the book and hide the others: S

pecial>C

on

ditio

nal

Text>Sh

ow/H

ide>S

pin

eSize(-->H

ide:)>S

et . Move the changed C

onditional text settings to all files in your book by opening the book file with the spine.fm

still open and File>Im

po

rt>Fo

rmats the

Conditional Text S

ettings (ON

LY!) to the book files.

Draft D

ocument for R

eview F

ebruary 10, 2008 5:17 pm7574sp

ine.fm

246



®

SG24-7574-00 ISBN

Draft Document for Review February 10, 2008 5:17 pm

INTERNATIONAL TECHNICALSUPPORTORGANIZATION

BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE

IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.

For more information:ibm.com/redbooks

®


Learn about the new Advanced Copy Services functionality

Server integration examples

Scripting examples

This book describes the new features that have been added with the release of the IBM® System Storage™ SAN Volume Controller 4.2.1 code. Readers are expected to have an advanced knowledge of the SVC and SAN environment, and we recommend these books as background reading:

� IBM System Storage SAN Volume Controller, SG24-6423

� Introduction to Storage Area Networks, SG24-5470� SAN Volume Controller: Best Practices and

Performance Guidelines, SG24-7521� Using the SVC for Business Continuity, SG24-7371

Back cover


