installation of mysql 5.1 cluster software on the solaris 10

Installation of MySQLTM 5.1 Cluster Software on the SolarisTM 10 OS for x64 Platforms

Rainer Walter

Sun Microsystems

September 2008

Sun Microsystems, Inc.

Copyright © 2008 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights reserved.

U.S. Government Rights - Commercial software. Government users are subject to the Sun Microsystems, Inc. standard license agreement and applicable provisions of the FAR and its supplements. Use is subject to license terms. This distribution may include materials developed by third parties.

Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and in other countries, exclusively licensed through X/Open Company, Ltd. X/Open is a registered trademark of X/Open Company, Ltd.

All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the United States and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc.

AMD, Opteron, the AMD logo, the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices.

Sun, Sun Microsystems, the Sun logo, MySQL, Solaris, Sun BluePrints, Sun Fire, and SunSolve are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and other countries.

This product is covered and controlled by U.S. Export Control laws and may be subject to the export or import laws in other countries. Nuclear, missile, chemical biological weapons or nuclear maritime end uses or end users, whether direct or indirect, are strictly prohibited. Export or reexport to countries subject to U.S. embargo or to entities identified on U.S. export exclusion lists, including, but not limited to, the denied persons and specially designated nationals lists is strictly prohibited.

DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Sun Microsystems, Inc.

Table of Contents

Summary...............................................................................................................................................................1

Introduction..........................................................................................................................................................2

The MySQL Software....................................................................................................................................2

MySQL Cluster Software...............................................................................................................................2

Node Types..................................................................................................................................................2

High Availability Concept..............................................................................................................................3

Partitioning.................................................................................................................................................3

Availability of the MySQL Cluster Software.....................................................................................................3

Scope and Limitations............................................................................................................................................4

Audience...............................................................................................................................................................4

Installation............................................................................................................................................................5

Prerequisites................................................................................................................................................5

Compilation of the Source............................................................................................................................5

Installation of the MySQL Server Software.....................................................................................................6

Configuration of a MySQL Cluster............................................................................................................................7

Overview.....................................................................................................................................................7

MySQL Cluster Configuration Files.................................................................................................................8

Starting and Stopping Components of a MySQL Cluster.................................................................................14

Cluster Messages and Error Log Files............................................................................................................16

User Accounts and File Ownership...............................................................................................................16

Storage Management in a MySQL Cluster.....................................................................................................16

Glossary...............................................................................................................................................................19

For More Information...........................................................................................................................................19

Acknowledgements..............................................................................................................................................21

Revision History...................................................................................................................................................21

1 Summary Sun Microsystems, Inc.

Summary

This document describes the installation of MySQLTM cluster software on three Sun FireTM X4100 M2 servers running

the SolarisTM 10 Operating System for x64 platforms. For two scenarios where data is loaded and manipulated, the

adjustment of the MySQL configuration necessary to execute these tasks is described. (Note: The installation was

performed with the Release Candidate Development Release of the MySQL 5.1 Community Edition.)

Introduction

The MySQL Software

MySQL software is the most popular relational database management system (RDBMS) software in the open source

space and part of the LAMP stack, which is a collection of software for dynamic web pages. Among the features that

differentiate MySQL software from other well-known RDBMS software are:

● Support for a broad variety of operating system platforms

● Choice of storage engines with different locking, indexing, and logging capabilities, enabling an optimal

tailoring of the database with regards to hardware resources and requirements of the application on top

MySQL Cluster Software

MySQL cluster software enables users to implement a MySQL database on commodity hardware using an approach for

fault tolerance, which ensures that data remains available even in the event of a failure of a database server or

storage component. The manner of clustering MySQL software described in this paper is accomplished on the

application level with no resources being shared among the cluster members at all, in contrast to cluster concepts

that operate on the operating system level, such as SunTM Cluster software, where the cluster members share one or

more storage resources. Implementing high availability for MySQL software using Sun Cluster software is described

elsewhere1.

While a detailed technical description of the design principles of MySQL cluster software can be found on the MySQL

web site2, an exhaustive discussion about the benefits and caveats of such a high availability solution is beyond the

scope of this paper. The following section provides a brief terminology of a MySQL cluster environment. It should be

noted that, while the availability and integrity of data in the MySQL database can be shielded against hardware

failures, existing client sessions, client connections, and active transactions are not protected by this MySQL

clustering concept and might suffer from such failures.

1 Sun Cluster Data Service for MySQL Guide for Solaris OS, August 2005, Part No: 819-1088-11 and http://blogs.sun.com/luojiach/entry/installation_and_configuration_sun_cluster1

2 M. Ronström, L. Thalmann: MySQL Cluster Architecture Overview, a MySQL technical white paper, 2004;New Features in MySQL Cluster 5.1, a MySQL technical white paper, August 2006.

Installation of MySQL 5.1 Cluster Software on the Solaris 10 OS for x64 Platforms

http://blogs.sun.com/luojiach/entry/installation_and_configuration_sun_cluster1

2 Introduction Sun Microsystems, Inc.

Node Types

There are three different functions in a MySQL cluster, which are associated with respective daemons:

● Data (storage) node with daemon ndbd

● SQL node with daemon mysqld

● Management node with daemon ndb_mgmd

A server can host more than one function of a MySQL cluster.

High Availability Concept

High availability is achieved on the application level by providing redundant services and storage resources on

independent servers in a shared-nothing architecture. Network Database (NDB) is the storage engine required for this

architecture. The data portion of the database that one data (storage) node holds is called the cluster partition.

Cluster partitions are managed as replicas. A primary replica is the data portion, which is assigned to a data node in

the cluster; a backup replica is a copy of a partition that is assigned to another node.

Data manipulation statements (update, insert, delete) and data definition statements (for example,

create table, create tablespace, and so on) are performed through a two-phase commit protocol,

ensuring that the database, being spread across multiple data nodes, remains consistent and that the content of

backup replicas is identical. The distribution of data in one replica across different storage nodes is accomplished by

horizontal partitioning.

Partitioning

Data in the database is evenly split into as many partitions as data nodes exist. Copies of partitions (replicas) can be

kept on data nodes other than the node that holds the original partition for redundancy. While the number of

partitions is always equal to the number of storage nodes, the number of replicas can be one or two, where a setup

with two replicas provides redundancy and is, therefore, recommended.

The number of replicas controls how many node groups are formed in a MySQL cluster. A node group is a subset of

data nodes, which holds replicas of the same partition; if all data nodes of a MySQL cluster that form one node group

are missing, one partition is not available and so the database cannot be brought online.

Availability of the MySQL Cluster Software

Until MySQL version 5.1 software, MySQL cluster software came in two flavors:

● As part of the "ordinary" MySQL software distribution3.

● As MySQL Cluster Carrier Grade Edition, a special version for network service providers and telecom

equipment manufacturers4, which offers some extra features not available in other distributions. The

versioning of this edition is different from the "ordinary" software distribution.

3 For MySQL software version 5.1: http://dev.mysql.com/downloads/mysql/5.1.html 4 More details on http://www.mysql.com/why-mysql/communications/

MySQL Cluster Carrier Grade Edition download: http://dev.mysql.com/downloads/cluster/index.html


http://dev.mysql.com/downloads/cluster/index.html

http://www.mysql.com/why-mysql/communications/

http://dev.mysql.com/downloads/mysql/5.1.html

3 Introduction Sun Microsystems, Inc.

The installation of the MySQL software was performed with the Release Candidate Development Release version 5.1.24

of the MySQL Community Edition software distribution. While cluster support is included in the package of MySQL

version 5.1.24 for the Solaris OS for SPARC® platforms, the official MySQL download site did not offer an analogous

package with cluster support for the Solaris OS for x64 platforms as of the writing this paper. Instead, the source of

MySQL version 5.1.24 was downloaded and was compiled prior to installation.

Planned future distributions of MySQL 5.1 binaries would no longer contain the MySQL cluster functionality; instead,

the official successor version of MySQL Cluster 5.0 software would be MySQL Cluster Carrier Grade Edition 6.2.

Scope and Limitations

This paper describes the installation of the MySQL software and the steps that are necessary to set up a shared-

nothing MySQL cluster in a minimal setup of three cluster nodes, which does not provide universal fault tolerance and

is not, therefore, recommended for production purposes.

The settings that need to be specified for loading and manipulating data in two different scenarios are discussed. An

investigation about performance optimization and about security aspects of a MySQL cluster environment is beyond

the scope of this paper.

The default paths to executables and configuration files might differ between MySQL installations performed from

binaries that are in Solaris package format and binaries built from source. For instance, the binaries of MySQL

daemons, such as ndbd, mysqld, ndb_mgmd, are located in the /bin folder of the MySQL installation directory

built from a Solaris package for SPARC platforms. However, installation on the Solaris OS for x64 platforms, which is

based on compiled binaries from source, keeps these daemon executables in the /libexec subfolder of the

installation directory, separate from other executables in /bin.

Audience

Readers of this paper should be familiar with concepts of relational databases and have basic knowledge of the SQL

programming language.


4 Installation Sun Microsystems, Inc.

Installation

The installation of MySQL 5.1.24 software that is described herein was performed on three Sun Fire X4100 M2 servers

with the following specifications:

● 2x dual-core AMD OpteronTM CPUs at 2.4 GHz

● 4 Gbytes of RAM

● Storage: Two 68-Gbyte internal hard disks

● Operating System: Solaris 10 5/08 OS for x64 platforms (s10x_u5wos09 X86)

The installation of the MySQL software from the source comprises three steps:

● Run the configure script in the MySQL source directory to produce makefile scripts.

● Use gmake to produce the appropriate binaries.

● Run gmake install to set up the directory structure and to copy binaries and configuration files.

Prerequisites

The MySQL source can be obtained in compressed tar format from the MySQL web site3.

For an installation of MySQL software from binaries that have been compiled from source, it is recommended that a

user account with the user name mysql exists and a group names mysql exists, which the mysql user is member

of. There is no need to set up a home directory for this user, nor to define a password.

For the compilation of the MySQL 5.1.24 source on the Solaris 10 OS for x64 platforms, the executables gcc (version

3.4.3) and gmake (version 3.80) in /usr/sfw/bin are used. Additionally, the ar executable in

/usr/ccs/bin is needed.

The PATH variable must, therefore, be adjusted:

PATH=$PATH:/usr/sfw/bin:/usr/ccs/bin

If 64-bit binaries are compiled, a symbolic link is needed in the directory /usr/sfw/lib/amd64:

ln -s libgcc_s.so.1 libgcc_s.so



Compilation of the Source

The MySQL source can be used to produce both 32-bit and 64-bit binaries. The procedure for both differs in the

environment variables that are supplied when executing the configure script, while two options need to be

provided in both cases when calling the configure script:

● The option --prefix specifies the installation directory of the MySQL binaries, which has been set to

/opt/mysql. If omitted, the binaries are installed in /usr/local/mysql.

● The option --with-ndbcluster instructs the configuration process to include the binaries of the NDB

engine, which is needed for storage nodes and the management node (or nodes) of a MySQL cluster, and

which are not included by default.

Building a 32-bit version of MySQL software: After extracting the tar archive of the MySQL source, the script

configure is called with the following parameters from the directory of the extracted source files (mysql-5.1.24-rc):

CC=gcc CXX=gcc ./configure --prefix=/opt/mysql --with-ndbcluster

Building a 64-bit version of MySQL software: After extracting the tar archive of the MySQL source, the script

configure is called with the following parameters from the directory of the extracted source files (mysql-5.1.24-rc):

CC='gcc -m64' CXX='g++ -m64' LDFLAGS="-R/usr/lib/64 -R/lib/64 -R/usr/sfw/lib/64 -L/usr/sfw/lib/64 -L/usr/lib/64 -L/lib/64 -lrt" ./configure --prefix=/opt/mysql --with-ndbcluster

The binaries are created by the following command, which is executed in the directory where the source files are

located:

gmake

Installation of the MySQL Server Software

The installation of the freshly compiled binaries is launched from the folder of the MySQL source files by issuing the

following command:

gmake install

After the installation of the MySQL software, a script must be run, which initializes the grant tables of the MySQL

server. This script is called mysql_install_db and it is located in the /bin folder of the MySQL installation

directory (/opt/mysql/bin, in this case).

This script creates a directory structure for two databases, mysql and test. Since these database directories must

be readable and writable by the MySQL server and, therefore, must be owned by the user mysql, the command

mysql_install_db, when executed in the context of the root user, should be run with the option --user=mysql, which ensures that the necessary privileges and ownership flags are set.



By default, the database directories are set up in the MySQL installation directory under the subfolder data. If these

database files and directories need to located elsewhere, two alternatives can be followed:

● Run the mysql_install_db command with the option --datadir=<path for parent directory of MySQL database files>

● Add an entry for datadir in the configuration file of the MySQL server, my.cnf, before running

mysql_install_db

The initial size of the folder containing the data files of the mysql database is 1.5 Mbytes. The size of the test database is initially 1 Kbyte.

Configuration of a MySQL Cluster

Overview

A MySQL cluster environment comprises three different functions. Each function is characterized by specific sections in

the my.cnf file.

● A MySQL cluster management node provides the configuration data of the cluster to storage nodes. It stops

single storage nodes or shuts down the entire cluster. The MySQL cluster management node is based on the

ndb_mgmd process, which reads the cluster configuration from the config.ini file. Cluster

management is performed using the management console

<MySQL_Install_Directory>/bin/ndb_mgm.

● A MySQL storage (data) node provides the storage capacity for data in a MySQL cluster database. It uses the

NDB storage engine, which is based on the ndbd process. The ndbd process of a storage node obtains its

configuration upon startup from the management node.

● A MySQL SQL node provides access to the data in a database of a MySQL cluster. It is based on a running

mysqld process. This daemon accepts connections from client programs, such as MySQL monitor,

<MySQL_Installation_Directory>/bin/mysql, which provides an interface for executing

SQL code.

The environment used for experiments described here is depicted in Figure 1.


7 Configuration of a MySQL Cluster Sun Microsystems, Inc.

MySQL Cluster Configuration Files

MySQL Server Configuration File: my.cnf

The MySQL server configuration file must exist on each member of a MySQL cluster and defines the roles in the cluster

that each member has. Multiple my.cnf files can exist on one system, which are processed in a hierarchical

manner:

● A my.cnf file with global settings must be located in /etc or in /etc/mysql5.

● Server-specific options can be supplied in a my.cnf that resides in the installation path of the MySQL

software or in its data directory.

● User-specific options are interpreted from a my.cnf file in the user's home directory.

In the setup described in this guide, one single my.cnf file in /etc/mysql was used.

my.cnf for the Management Node

The following entries were made in a my.cnf file that defines the management node of a MySQL cluster. While the

[mysqld] section exists in the template, the sections [ndb_mgm] and [ndb_mgmd] need to be added, which

both define the role as cluster management node of this server in the cluster.

[mysqld]

user = mysqlndbcluster

5 MySQL 5.1 Reference Manual, revision: 10113, 2008-03-04. See also http://dev.mysql.com/doc/refman/5.1/en/ for the most current, online version.


Figure 1. MySQL Cluster Topology

http://dev.mysql.com/doc/refman/5.1/en/


ndb-connectstring = 10.9.166.30datadir = /export/mysql/MySQL_Data

[ndb_mgm]

connect-string = 10.9.166.30

[ndb_mgmd]

config-file = /etc/mysql/config.ini

The entries in [mysqld] are:

● user = mysql – This setting lets the mysqld daemon run in the context of the user mysql.

● ndbcluster – This server can act as an SQL node in a MySQL cluster. DDL statements can use the NDB

storage engine. Although the binaries of the installation are compiled with the ndbcluster option, the

NDB storage engine is not automatically available when the MySQL server is started. The NDB storage engine

is activated by this entry in the my.cnf, or by specifying the --ndbcluster argument in the start

command of the MySQL server. To verify that the NDB storage engine is present and usable, the command

show engines can be executed using MySQL monitor, as shown in Figure 2.

● ndb-connectstring – This SQL node tries to contact the management node running on a system with

the IP address specified. A connection to such a management node is established if (1) this SQL node is listed

and specified in the management server's config.ini or (2) the management server has "blank" entries

for unspecified SQL nodes. No user authentication is required.

● datadir – This is the path to the files and directories of a MySQL server that the script

mysql_install_db creates.

The entry connect-string in the section [ndb_mgm] specifies the IP address of the server hosting the NDB

management daemon ndb_mgmd that the cluster management console is supposed to talk to.

The section [ndb_mgmd] indicates that the host acts as cluster management node. The value for config-file

points to the location of the MySQL cluster configuration file config.ini.

my.cnf for the Storage Nodes

The section [mysql_cluster] signals that the host acts as storage node in the cluster. The

ndb-connectstring value points to the management node from which a cluster storage node shall obtain its

cluster-specific configuration.


Figure 2. Output of a show engines Query


[mysqld]

user = mysqlndbclusterndb-connectstring = 10.9.166.30datadir = /export/mysql/MySQL_Data

[mysql_cluster]

ndb-connectstring = 10.9.166.30

MySQL Cluster Configuration File: config.ini

The topology and the configuration of a MySQL cluster is defined and maintained in one single file on the

management node: config.ini. The location of this MySQL cluster configuration file is specified in the my.cnf of the cluster management node.

The cluster storage nodes receive their individual configuration from their management node upon startup. Any

modification, therefore, requires a restart of the affected nodes. In special cases, for example, when settings of REDO

logs shall be changed, the affected data node must be restarted with the option --initial, which erases the

contents of the ndb_<NodeID>_fs folder, so that all data of databases using the NDB engine on this node are

lost unless an active replica node is available. Therefore, care should be taken when issuing the command:

<MySQL_Installation_Directory>/libexec/ndbd --initial

A template for a two-data-node cluster config.ini file (ndb-config-2-node.ini in

<MySQL_Install_Directory>/share/mysql), which comes with the installation of the MySQL

software, was used to produce the operative version for the tests described herein. The config.ini used for the

test environment covers five aspects:

● General properties of the cluster

● Management node

● Storage nodes

● SQL nodes

● TCP/IP settings

General properties of the cluster are listed in the [ndbd default] section. This section specifies:

● How many replicas shall be produced and maintained: NoOfReplicas. Possible values are 1 (no

redundancy) and 2.

● How much physical memory shall be assigned to MySQL: DataMemory, IndexMemory. The values for

these parameters depend on the total amount of RAM available on the host.

● The location of cluster logs and other files of the cluster: DataDir. Note that my.cnf uses a parameter

with the same name for specifying the location for files and directories that a mysqld daemon relies on. In

the setup used for tests described in this paper, different locations were supplied for the DataDir of

mysqld (in my.cnf) and ndbd (in config.ini).



Settings for storage nodes can be defined globally, that is, for all storage nodes, in the [ndbd default] section,

or individually for a single storage node in separate [ndbd] sections. For each storage node, an extra [ndbd] section must exist, which at least contains a HostName entry.

A SQL node is specified with its host name in a separate [mysqld] section. If a [mysqld] section does not

contain a HostName entry, any MySQL server can connect to the cluster if its my.cnf file contains a ndb-connectstring entry with the valid IP address of the cluster management node. It is recommended to define

one [mysqld] object in config.ini that does not have a specified host entry, which is helpful, for example,

for ndb_restore operations.

This is the initial version of config.ini:

[ndbd default]NoOfReplicas= 2MaxNoOfConcurrentOperations= 10000DataMemory= 2048MIndexMemory= 512MTimeBetweenWatchDogCheck= 30000DataDir= /export/mysql/mysql-clusterMaxNoOfOrderedIndexes= 512

[ndb_mgmd default]DataDir= /export/mysql/mysql-cluster

[ndb_mgmd]Id=1HostName= 10.9.166.30

[ndbd]Id= 2HostName= 10.9.166.28

[ndbd]Id= 3HostName= 10.9.166.29

[mysqld]#Id= 4HostName= 10.9.166.28

[mysqld]#Id= 5HostName= 10.9.166.29

[mysqld]HostName= 10.9.166.30

[mysqld]# One empty entry recommended

[tcp default]PortNumber= 63132

Depending on the database load characteristics, default settings must be changed by modifying existing entries or

adding new entries in the appropriate sections to override default values. In the following, some workloads have been

executed on the MySQL cluster, and the adjustments needed in the config.ini file for a smooth operation are

described. All tests were run with tables using disk-based storage. No attempts were made to optimize performance

during these tests.



Scenario 1: Table With 3 Columns, Simple Records

The table used for the tests with insert, update, and index creation is described in Figure 3.

Test #1: Inserting 10 Million Records

An SQL procedure with a loop was used to create 10 million records.

Result: No modifications in config.ini are necessary.

Test #2: Update 10 Million Records

An SQL statement updates the Date field of all records.

This attempt produces errors of the following kind:

● Got temporary error 233 'Out of operation records in transaction coordinator' from NDBCLUSTER

● Got temporary error 1217 'Out of operation records in local data manager' from NDBCLUSTER

Result: Two parameters for the configuration of data nodes have default values that are not sufficient to run an

update statement for millions of records:

● MaxNoOfConcurrentOperations (Default: 32768). This is the number of records that is updated

simultaneously in a transaction.

● MaxNoOfLocalOperations (Default: 1.1 * MaxNoOfConcurrentOperations)

By default, MaxNoOfLocalOperations is a function of MaxNoOfConcurrentOperations, so only

the latter needs to be changed in config.ini.

When the value for MaxNoOfConcurrentOperations was increased to 10,000,000 (the number of records

that shall be updated in one transaction), the storage nodes could not be started due to a lack of memory (error 2327).

This situation might depend on the amount of RAM assigned to the MySQL software and the capacity in RAM allocated

by table data. The update statement must, therefore, have a where clause that the number of records to be updated

keeps within the limit of a tolerable value for MaxNoOfConcurrentOperations.

In the test environment, updating 500,000 rows in a single transaction was possible using the following setting:

● MaxNoOfConcurrentOperations = 1000000


Figure 3. Schema of the Test Table


Test #3: Inserting 2x 10 Million Records Into a Table From Two Client Connections in Parallel

Result: One parameter in section [ndbd default] needs to be increased to avoid error 1205 (Lock wait timeout exceeded):

● TransactionDeadlockDetectionTimeout (default: 1200 ms).

The value for TransactionDeadlockDetectionTimeout was increased to 10000, and the parallel

execution of the insert statement went smoothly.

Test #4: Inserting 3x 10 Million Records Into a Table From Three Client Connections in Parallel

Result: No changes need to be made to config.ini to have the three insert statements complete successfully.

Test #5: Table With 10 Million Rows: Creating an Index

An additional index was created on the Firstname field using an ALTER TABLE statement.

Result: No modification in config.ini is necessary to run the statement.

Scenario 2: Table With BLOB Field

The table used for tests with BLOB data in the MySQL software has the schema shown in Figure 4.

Test #1: Insert 5000 Records With a Binary File

Five thousand individual binary files6 with file sizes ranging from 512 Kbytes to 3012 Kbytes were inserted using the

LOAD_FILE option in the INSERT INTO clause using a script with individual INSERT INTO statements.

Result: A data node was forced to shut down because the default value for SendBufferMemory was too low. The

largest file that could be inserted was 1624 Kbytes in size.

6 Binary files created with the command dd if=/dev/urandom of=<Path_of_File> bs=512 count=<N>


Figure 4. Schema of the Table With BLOB Field


The following parameter was added in the [tcp default] section of the cluster configuration file:

● SendBufferMenory = 10M

Error 1220 was thrown while loading data into the table: Got temporary error 1220 'REDO log files overloaded, consult online manual (increase FragmentLogFileSize)' from NDBCLUSTER.

The following parameter was added to the [ndbd default] section of config.ini:

● FragmentLogFileSize (default: 16 Mbytes)

The value for FragmentLogFileSize was set to 64 Mbytes.

Error 1221 was thrown while loading data into the table: Temporary error: 1221 'REDO buffers overloaded, consult online manual (increase RedoBuffer)'.

The following parameter was added to the [ndbd default] section of config.ini:

● RedoBuffer (default: 8 Mbytes)

The value for RedoBuffer was increased to 64 Mbytes.

Starting and Stopping Components of a MySQL Cluster

It is mandatory to start the cluster management daemon (ndb_mgmd) first because the storage nodes get their

cluster configuration from the config.ini file on the management node. This daemon is launched by executing

the following command:

<MySQL_Installation_Directory>/libexec/ndb_mgmd

The management console is brought up with the following command:

<MySQL_Installation_Directory>/bin/ndb_mgm

The initial state of the cluster environment described herein can be displayed by the command show, as shown in

Figure 5.



The cluster consists of two data nodes [ndbd(NDB)] and three SQL nodes [mysqld(API)].

All data nodes and two SQL nodes are bound to specified IP addresses (that is, connections from other MySQL servers,

even when having suitable entries in their my.cnf, will be rejected).

One SQL node connection is defined in the config.ini file without an IP address, so that any host with a

mysqld process running and with an appropriate ndb-connectstring in its my.cnf can establish a

connection with the management daemon of this MySQL cluster and act as an SQL node.

When the cluster management daemon is running, the storage node daemon (ndbd) can be started. If the ndbd process cannot contact the management node specified in the ndb-connectstring of the

[mysql-cluster] section of its my.cnf, an error message occurs: Unable to connect with connect string: nodeid=0, <IP address>:1186. Retrying every 5 seconds.

The execution of <MySQL_Installation_Directory>/libexec/ndbd launches two processes called

ndbd: The first ndbd process (angel process) watches its child ndbd process and restarts it when it detects that the

child process is no longer active.

After starting the ndbd daemon on the storage nodes, the show command in the cluster management console

displays the status shown in Figure 6.


Figure 5. MySQL Cluster Management Console


The MySQL cluster management console is the preferred tool for stopping single data nodes or for shutting down the

whole cluster. A single data node can be stopped by issuing the command <NodeID> stop; the whole cluster is

stopped with the command shutdown, as shown in Figure 7.

The MySQL cluster can be run independently from any MySQL server process. Such a process is needed only to access

databases that use the NDB storage engine and retrieve or modify records and database objects.

It is recommended to start an SQL node using the following script:

<MySQL_Installation_Directory>/bin/mysqld_safe &

It is also recommended that the mysqld process runs in the context of a user other than root, for example, in the

context of the user mysql (see the section about file ownership on page 16). If the my.cnf file does not contain an

entry like user=mysql, the command shown above can be executed with the option --user=mysql.


Figure 6. MySQL Cluster Running

Figure 7. Shutting Down MySQL Cluster


Alternatively, the shell script mysql.server in

<MySQL_Installation_Directory>/share/mysql can be copied to /etc/init.d so that the

MySQL server process is automatically started when the server is restarted.

On the server running the Solaris 10 OS for x64 platforms, the following adjustments were made in order to make the

MySQL server start script executable:

● Replaced the first line by the expression #!/bin/bash

● Specified the MySQL installation directory in the variable basedir

● Specified the path to the database files created by mysql_install_db in the variable datadir

● Specified the path of the my.cnf file in the variable conf

The mysqld process on an SQL node can be stopped using the following command:

<MySQL_Installation_Directory>/bin/mysqladmin -u root shutdown

Since the storage node process requires that the cluster management node daemon is up and running, it does not

appear reasonable to provide an analogous start script for the ndbd daemon.

Cluster Messages and Error Log Files

The error log files of the MySQL cluster are located in the directory specified in the DataDir parameters for data

and management nodes of the config.ini file.

The management node maintains the ndb_1_cluster.log, while each data node has an

ndb_<NodeID>_error.log and trace files, if errors have occurred. The value for NodeID refers to the

numbering provided in the management console view (see Figure 6). The cluster log and the error logs are rolled over

when their size reaches a threshold value.

User Accounts and File Ownership

The context of the user, under which a mysqld or ndbd daemon is started, is critical.

SQL node: After the installation of the MySQL software, it is recommended to run the script mysql_install_db as user mysql and not as user root. This user becomes the owner of the database directories and files, which are

created by the script. If the MySQL server is started with a different user ID, the MySQL server cannot access databases

and tables and produces errors upon startup.

Storage node: All content in the Datadir of the ndbd daemon are owned by the user that launched the ndbd daemon. There is no recommendation regarding a preferred user for running ndbd as long as the same user is always

starting this daemon. In the tests with the MySQL cluster described herein, the ndbd process was launched as user

root.



Storage Management in a MySQL Cluster

Prior to MySQL software version 5.1, all database data using the NDB storage engine was kept in the physical memory

of the storage nodes assigned to the ndbd daemon. With version 5.1, it is possible to store table content in columns

that are not indexed in disk-based tablespaces, while the indexed columns of a table reside in memory. The keyword

STORAGE DISK in a CREATE TABLE statement in conjunction with the ENGINE=NDB clause indicates

that columns without index shall allocate pages on disk rather than in memory.

The amount of RAM that is at disposal for a MySQL cluster is specified by the parameters DataMemory and

IndexMemory in the [ndbd] section of the config.ini. The utilization of physical memory by data and

indexes can be determined by executing the command all dump 1000 on the MySQL cluster management

console7 (see Figure 8).

The output of an all dump 1000 command is written to the MySQL cluster log, and shows the utilization of the

assigned memory for data and indexes as a percentage and as the number of pages, respectively, and might look like

the following:

2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Data usage is 42(27885 32K pages of total 65536)2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Index usage is 47(30959 8K pages of total 65568)2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Resource 0 min: 0 max: 639 curr: 2692008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Resource 1 min: 0 max: 0 curr: 2592008-05-21 03:39:25 [MgmSrvr] INFO -- Node 2: Resource 2 min: 0 max: 0 curr: 102008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Data usage is 42(27883 32K pages of total 65536)2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Index usage is 47(30958 8K pages of total 65568)2008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Resource 0 min: 0 max: 639 curr: 2692008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Resource 1 min: 0 max: 0 curr: 2592008-05-21 03:39:25 [MgmSrvr] INFO -- Node 3: Resource 2 min: 0 max: 0 curr: 10

Information about the utilization of disk space allocated in data files of tablespaces and undo files of log file groups

can be retrieved from the table INFORMATION_SCHEMA.Files (see Figure 9).

7 In planned future versions, the command ALL REPORT MEMORYUSAGE would do the same.


Figure 8. MySQL Cluster Management: All Dump 1000


Tablespaces of the NDB engine do not have an autoextend feature. When creating a data file of a tablespace using the

CREATE TABLESPACE or the ALTER TABLESPACE statement, either a file name or the full path of

the data file must be supplied. If only a file name is supplied, the file is created in the default path

<DataDir>/ndb_<NodeID_fs>.

The same applies to undo files that are created for a log file group. If undo files shall be physically separated from data

files to minimize concurrent I/O during insert and update activities, the full path of the undo/data file must be

supplied in the CREATE LOGFILE GROUP and CREATE TABLESPACE statements, respectively.

In a MySQL 5.1 cluster database with tables using disk-based storage, three types of logging can occur:

1. REDO logging. The data nodes of a MySQL cluster use, in the default configuration, 16 sets of 4x16 Mbyte files for

circular logging, that is, old log entries can be overwritten if three local checkpoints have been completed. The default

size of 1 Gbyte in total for REDO logs can be adjusted by supplying custom [ndbd] parameters. With MySQL REDO

logs, there is no danger of a continuous growth of disk space consumption.

2. UNDO logging. If data nodes of a MySQL cluster use disk data objects, that is, log file groups and tablespaces in

conjunction with a STORAGE DISK clause, disk space is allocated in undo files. Because these undo files cannot be

automatically extended, there is no immediate risk of filling up the file system over time.

3. Binary logging. The setting for log-bin in the my.cnf file ensures that all statements that manipulate data

and database objects are logged. These log files are written to the Datadir target of the mysqld daemon on SQL

nodes of a MySQL cluster. This binary log is needed for recovery after restoring the database from a backup and for

replication. In contrast to the logging mechanisms mentioned before, if not maintained, these binary logs grow,

accumulate, and might consume a significant amount of disk space on SQL nodes.


Figure 9. Data Files and Undo Files: Utilization of Disk-Based Storage

19 Glossary Sun Microsystems, Inc.

Glossary

The following table contains common abbreviations and acronyms that are used in this paper.

Abbreviation Explanation

BLOB Binary Large Object

DDL Data Definition Language

LAMP Linux, Apache, MySQL, Perl|Python|PHP

NDB Network Database

RDBMS Relational Database Management System

SQL Structured Query Language

For More Information

Here are additional resources:

● Documentation:

● MySQL documentation site: http://dev.mysql.com/doc/

● MySQL Cluster 5.1 - New Features white paper:

http://www.mysql.com/why-mysql/white-papers/mysql_wp_cluster51.php

● MySQL Cluster Architecture Overview white paper:http://www.mysql.com/why-mysql/white-papers/cluster-technical.php

● MySQL training courses: http://www.sun.com/software/products/mysql/training.jsp

● MySQL download sites:

● http://www.sun.com/software/products/mysql/getit.jsp

● http://dev.mysql.com/downloads/mysql/5.1.html

● MySQL forums: http://forums.mysql.com/

● Other web sites and resources:

● MySQL site on sun.com: http://www.sun.com/software/products/mysql/index.jsp● mysql.com site: http://www.mysql.com/● Features & Benefits - MySQL Cluster:

http://www.sun.com/software/products/mysql/cluster.jsp

● MySQL and MySQL Cluster FAQs: http://www.sun.com/software/products/mysql/faqs.jsp● Sun BluePrintsTM article: Optimize MySQL Server on Sun x64 Servers and Storage

http://wikis.sun.com/display/BluePrints/Optimize+MySQL+Server+on+Sun+x64+Servers+and+Storage




http://www.sun.com/software/products/mysql/faqs.jsp

http://www.sun.com/software/products/mysql/cluster.jsp

http://www.mysql.com/

http://www.sun.com/software/products/mysql/index.jsp

http://forums.mysql.com/

http://dev.mysql.com/downloads/mysql/5.1.html

http://www.sun.com/software/products/mysql/getit.jsp

http://www.sun.com/software/products/mysql/training.jsp

http://www.mysql.com/why-mysql/white-papers/cluster-technical.php

http://www.mysql.com/why-mysql/white-papers/mysql_wp_cluster51.php

http://dev.mysql.com/doc/

20 For More Information Sun Microsystems, Inc.

● Resources on BigAdmin:

● Setting Up MySQL Cluster Software Using Solaris Zones Partitioning Technology: http://www.sun.com/bigadmin/features/articles/mysql_cluster_zones.jsp

● MySQL Resources for System Administrators: http://www.sun.com/bigadmin/topics/mysql/

● Creating a Script for Scheduled MySQL Backups (community submission): http://www.sun.com/bigadmin/content/submitted/sched_mysql_backup.jsp

● Database Resource Collection:http://www.sun.com/bigadmin/collections/database.html

● Resources on Sun Developer Network (SDN):

● Using MySQL and Memcached on the GlassFish Application Server: http://developers.sun.com/appserver/reference/techart/mysql_gf/

● Sun and MySQL: How It Stacks Up for Developers: http://java.sun.com/developer/technicalArticles/mysql_acq/index.html

● Support:

● Register your Sun gear: https://inventory.sun.com/inventory/

● Services: http://www.sun.com/service/

● SunSolveSM Online: http://sunsolve.sun.com

● Events of interest to users of Sun products:

● Sun and MySQL Around the World--Meetup Mashup: http://www.sun.com/events/mysqltour/index.jsp

● Find a MySQL Meetup Group near you: http://mysql.meetup.com/

● Worldwide developer events and Sun Tech Days: http://developers.sun.com/events/

● Current events: http://www.sun.com/events/index.jsp


http://www.sun.com/events/index.jsp

http://developers.sun.com/events/

http://mysql.meetup.com/

http://www.sun.com/events/mysqltour/index.jsp

http://sunsolve.sun.com/

http://www.sun.com/service/

https://inventory.sun.com/inventory/

http://java.sun.com/developer/technicalArticles/mysql_acq/index.html

http://developers.sun.com/appserver/reference/techart/mysql_gf/

http://www.sun.com/bigadmin/collections/database.html

http://www.sun.com/bigadmin/content/submitted/sched_mysql_backup.jsp

http://www.sun.com/bigadmin/topics/mysql/

http://www.sun.com/bigadmin/features/articles/mysql_cluster_zones.jsp

21 Acknowledgements Sun Microsystems, Inc.

Acknowledgements

We want to thank the following colleagues for their kind help, assistance, and support during the work on this paper:

Ryan Arneson, Sun Microsystems in Broomfield, CO, for preparing the Sun Fire X4100 M2 servers, and for helping

immediately when a hardware piece caused trouble.

Matthias Schmidt, Sun Microsystems in Walldorf, for his help in preparing the shell scripts for data generation.

Jürgen Zimmermann, Sun Microsystems in Walldorf, for his support during the attempts to compile the MySQL source

code.

Ritu Kamboj, Sun Microsystems in Menlo Park, for helping with the parameters for the 64-bit compilation.

Hartmut Holzgraefe, Sun Microsystems, for reviewing the paper and for his valuable comments.

Revision History

Revision Notes Date

0.1 Draft for Revision 29.05.2008

1.0 First Version 16.06.2008

Licensing Information

Unless otherwise specified, the use of this software is authorized pursuant to the terms of the license found at

http://www.sun.com/bigadmin/common/berkeley_license.html.


http://www.sun.com/bigadmin/common/berkeley_license.html

installation of mysql 5.1 cluster software on the solaris 10

Documents