perforce server clustering
TRANSCRIPT
Home › Blog › Technical Deep Dive: Perforce Server Clustering
Technical Deep Dive: Perforce ServerClusteringJan 20, 2015by Wendy Heffner, Michael Shields, and Samuel Penn of Perforce Software
The Perforce Server 2014.2 release marks the launch of a clustered server solution that assureshighly available, fault tolerant, and consistent revision data and scales up via workspace partitioning. Thecluster, designed to be deployed in a tightly coupled server center, is a collection of servers acting inconcert to appear as a single central server. One or more cluster routers front the servers; each routersupplies a concrete access point that maps the request to the workspace server hosting the user's clientdata.
Horizontal ScalabilitySuppose that you are a large company with a workforce that is growing past the capacity of a singleserver. The clustered server allows you to partition the workload by placing groups of clients on one ofseveral workspace servers. The users of a clustered server do not need to know the exact location of theirclients’ data, because a cluster router provides a static endpoint that dynamically maps the users’ requeststo the workspace server handling their data.
Server Cluster PerformanceConsistent execution time is a goal of a highly scalable system. Following on from lockless reads andrepository federation, server clustering is the next step in ensuring consistent execution time of Perforceoperations.
During development of the Perforce server cluster, simulations of varying loads were used to verify thescalability of the feature. Ideally, as the number of workspace servers increases, the number of Perforceoperations that can be performed with consistent execution time should increase linearly, too. Forexample, doubling the number of workspace servers should ideally result in double the number of Perforceoperations completing with consistent execution time.
In one suite of the simulations, two, three, and four workspace servers were used. The graphic belowshows the completion time of tasks performed by 30,000 simulated concurrent developers with twoworkspace servers; 45,000 with three workspace servers; and 60,000 with four workspace servers—withdevelopers at a production site averaging a delay of 10 minutes between operations.
Doubling the number of workspace servers from two to four accommodated doubling the number ofsimulated concurrent developers from 30,000 60,000 with only a 14% increase in the completion time ofthe tasks performed.
Newsletter
Get articles by our experts,and Perforce news monthlySign up now or view online.
Perforce GitFusionSee how it improvesthe Git experience
Learn More
Blog Categories
By Author
Demos Trials Pricing
Blog Company Community Contact
PRODUCTS SUPPORT RESOURCES PARTNERS CUSTOMERS DOWNLOADS
Search...
The average delay between operations in the simulation was pared to five seconds, or by a factor of 120,so that each concurrent developer role in the simulation might approximate 120 concurrent developers ata production site. This allowed for the simulation of a large number of concurrent developers using areasonable amount of hardware. Extended out, 256 concurrent developer roles (an average of 128 oneach of two workspace servers) simulated approximately 30,000 concurrent developers at a productionsite and 512 concurrent developer roles (an average of 128 on each of four workspace servers) simulated60,000 concurrent developers at a production site.
High AvailabilityThe versioning server is a key resource that often must be available at all times to both local employeesand remote replicas, but...
failures happen,
equipment requires maintenance, and
software upgrades must occur.
What problems does server clustering solve? Automatic failover can handle these issues by letting astandby take over the master role in the cluster. The former master is then offline and available forupgrade, repair, and/or maintenance. This approach is viable because in a Perforce server cluster thereis no loss of data and visible changes to the online repository are consistent between the master andstandby.
During normal operation the standby server can be used to produce online checkpoints of the master'sdata. Using the standby in this manner will not have an impact on the performance of the master becausethe process to assure consistency between servers is still running. Once the checkpoint has completed, thestandby will resume applying the journal records for the intervening transactions to its database.
Achieving Incremental GrowthA Perforce server cluster grows with the needs of a company. Most Perforce deployments start with asingle standard server, and perhaps a replica for manual failover. The company can transition to havingautomatic failover by upgrading to a Perforce server cluster. A minimal version of a simple HAconfiguration consists of:
a shared filesystem
1 depot master
1 depot standby
2 cluster routers—one each colocated with either the master or the standby and
a load balancer splitting the traffic between the two routers.
If there is a software or hardware failure on the underlying hardware for the master then automaticfailover will cause the standby to take over, and the remaining cluster router(s) will automatically updatetheir configurations to direct their requests to the new master.
As the company grows it can add workspace servers (similar to edge servers in distributed Perforce). Most operations can be partitioned by client and handled by the workspace server holding that particularclient. These commands, other than submit, can be handled exclusively by the workspaceserver hostingthat client. In this configuration, the depot master (similar to a commit server in distributed Perforce)handles changes to the revision data, which are only a small subset of the general workload of Perforceservers.
In addition to these Perforce servers we also employ a thirdparty distributed state managementapplication called Zookeeper. This software is produced by the Apache Foundation and serves as thehealth monitoring facility for the servers. When the status of any of the Perforce cluster memberschanges, Zookeeper notifies the interested cluster members so that they can respond. The Zookeeperservers are themselves fault tolerant and should be deployed in a group of 3 or 5 for optimalperformance.
Server Cluster ManagementThe Perforce Cluster Manager package, available on Ubuntu, Red Hat and CentOS, provides a commandline process (p4cmgr) that allows you to build and manage a cluster using a small set of commands,obviating the need to have detailed knowledge of how to configure Perforce servers. Not only does thissimplify installation, but it also means that your cluster is configured in a way that is easily understood andsupported by us.
Building a Perforce server clusterTo build a server cluster, install P4CMGR on Ubuntu by using:
aptget install perforceclustermanagerp4cmgr init s depot1 mycluster depotmasterhostname
p4cmgr add s zookeeper2 zookeeper workspace1hostnamep4cmgr add s zookeeper3 zookeeper routerhostname
POSTED IN: FEATURES & RELEASES, SERVER
p4cmgr add s depot2 depot depotstandbyhostnamep4cmgr add s workspace1 workspace workspace1hostnamep4cmgr add s router1 router routerhostname
This will create a simple cluster with a single master, a standby server, one workspace server and arouter which clients connect to.
On Red Hat or CentOS use:
yum install perforceclustermanager
The depot master is a commit server to which all versioningrelated tasks are submitted. Clients do notconnect directly to the depot master (the recommended network configuration is that everything otherthan the router is hidden behind a firewall); instead requests are forwarded to the depot master from theworkspace server.
The archive files on the depot master are written to a filesystem location which must be shared by all thenodes in the cluster.
A zookeeper is one server of a distributed Zookeeper service that is best deployed in groups of three. Bydefault, during the init step the first zookeeper server is deployed in the same host as the p4cmgr process.Two more zookeeper servers must be deployed during the build out of the cluster.
The depot standby is a replica of the master, and in a case where the master stops working, failover tothe standby will happen automatically, allowing the continuation of service. The standby is also used fortaking checkpoints without interrupting service for the users.
Each workspace server handles requests from clients and stores all the client data. Client data is notforwarded back to the master, which greatly reduces the workload on the single master (this is why asingle master can serve multiple workspace servers).
Workspace servers have access to the shared archive files, and will send requests back to the commitserver.
The cluster router listens on port 1666 and clients connect to this. The first time users connect, they willbe randomly assigned to a workspace server, and from that point on all their interactions will be routedthrough that workspace server.
Client workspace and have information will be stored on that workspace server and never sent to thedepot master, allowing the depot master to focus on handling commit operations.
Prerequisites and installationDocumentation on how to set up your system to use the Perforce package repository can be found athttp://package.perforce.com
For full documentation, read the Cluster Management guide for admins.
Before installing the Perforce Cluster Manager, it is necessary to configure your cluster nodes in asupported fashion:
An NFS share between all the nodes running a P4D service (master, standby and workstations).
Each machine visible to the others as a consistent name in DNS.
SSH access allowed between all machines.
SUDO access allowed for the user running p4cmgr commands.
Note: Perforce server cluster is not supported on Windows
Ready to try it? Download the Perforce Server here.
Related Articles:
P4 Tech Women: From Theatre to Security
Configuring Active Directory and LDAP Authentication Without Triggers
Client ChangeView: a window to the past
About the author:
See all posts by
« DevOps in Practice at Intuit New Release of Perforce Swarm; Perforce Microsoft
Office Integration Now Open Source; P4V Patch Now
Available »
Wendy Heffner is the Technical Lead of the Cluster Project in the server group.(Informally, she's the selfproclaimed "goddess of networking")
About the author: Michael Shields is a Server Performance Engineer at Perforce. He has developed andsupported software since 1977, specializing in server optimization for softwareproducts including Perforce, Sybase and Ingres. His hobbies include studying theeffects of solar radiation and eroded limestone on the human psyche.
About the author:Samuel Penn is a software engineer at Perforce Software.
Samuel Penn
OUR PRODUCTS
Overview
P4D Versioning Engine
Swarm
Git Fusion
Commons
Insights
SOLUTIONS
Continuous Delivery
Version Control
Git Management
Social Coding
TrunkBased Development
DevOps Repository
Document Collaboration
RESOURCES
Dev Trends
Documentation
Release Notes
Tutorial Videos
Blog
COMMUNITY
Forums
P4IdeaX
Workshop
COMPANY
About Us
Careers
Events
Newsletter
Press Room
Contact Us
SUPPORT AND SERVICES
Support Plans
Consulting
Knowledge Base
Customer Login
COMMUNICATION CENTER
Visit our Communication Center to sign up for News, Tips and Updates
Copyright © 2015 Perforce. All rights reserved. Site Map Privacy Policy Terms of Use Contact Site Credits