redis labs enterprise cluster documentationlabs... · redis labs enterprise cluster documentation...

Redis Labs Enterprise Cluster

Documentation

Version 4.2.1-30 – September 21, 2015

Copyright© 2015 Redis Labs Ltd. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form

by any means, electronic, mechanical, photocopying, recording or otherwise without the prior written

permission of Redis Labs Ltd.

Redis Labs Ltd. reserves the right to change or improve its products and accompanying technical

literature without specific notice of changes or improvements.

Redis Labs Enterprise Cluster Documentation Page 1

Table of Contents

Overview ....................................................................................................................................................... 4

The technology behind RLEC ................................................................................................................ 5

Installing and Upgrading ............................................................................................................................... 8

Hardware and software requirements ....................................................................................................... 8

Accessing and installing the setup package............................................................................................ 10

Uninstalling .............................................................................................................................................. 11

Upgrading ................................................................................................................................................ 11

Initial setup – creating a new cluster ........................................................................................................... 13

How to set the cluster name (FQDN) ...................................................................................................... 14

Cluster Administration ................................................................................................................................. 16

Joining a new node to a cluster ............................................................................................................... 16

Replacing a faulty node ........................................................................................................................... 17

Removing a node .................................................................................................................................... 17

Taking a node offline............................................................................................................................ 17

Permanently removing a node ............................................................................................................. 18

Replacing a faulty node ....................................................................................................................... 18

Replacing a healthy node .................................................................................................................... 18

Rebalancing nodes .................................................................................................................................. 19

Viewing and defining cluster settings ...................................................................................................... 19

Managing general settings ................................................................................................................... 19

Managing users ................................................................................................................................... 20

Managing cluster alerts ........................................................................................................................ 21

Viewing cluster metrics ............................................................................................................................ 21

Viewing node metrics .............................................................................................................................. 21

Viewing the log ........................................................................................................................................ 22

Best practices .......................................................................................................................................... 22

Configuration of AWS instances .......................................................................................................... 22

Using a single DNS server ................................................................................................................... 23

Persistent and ephemeral storage ....................................................................................................... 23

Disk size requirements for extreme write scenarios ............................................................................ 23

Synchronizing cluster server clocks to Network Time Protocol (NTP) ................................................ 24

Updating web server SSL certificates .................................................................................................. 24

Performance optimization .................................................................................................................... 24


Machine ports configuration ................................................................................................................. 25

CentOS / RHEL 7 firewall configuration............................................................................................... 26

Client prerequisites for mDNS ............................................................................................................. 26

Multi-IP & IPv6 support ........................................................................................................................ 27

Database configuration ............................................................................................................................... 29

Creating a new database......................................................................................................................... 29

Database memory limit ............................................................................................................................ 30

Database replication ................................................................................................................................ 31

Database persistence .............................................................................................................................. 31

Database clustering ................................................................................................................................. 31

Database eviction policy .......................................................................................................................... 34

Replica of ................................................................................................................................................. 35

Database backup ..................................................................................................................................... 38

Periodic backup ................................................................................................................................... 38

Ad-hoc backup - Export ....................................................................................................................... 38

Backup of a sharded database ............................................................................................................ 38

Database alerts ....................................................................................................................................... 38

Importing data to a database ................................................................................................................... 39

Exporting data from a database .............................................................................................................. 39

Updating database configuration ............................................................................................................. 40

Deleting a database ................................................................................................................................. 40

Viewing database metrics........................................................................................................................ 40

Database metrics ................................................................................................................................. 40

Shard metrics ....................................................................................................................................... 41

Viewing Redis Slow Log .......................................................................................................................... 41

Rack-zone awareness ................................................................................................................................. 44

rladmin Command Line Interface (CLI) ....................................................................................................... 45

System architecture .................................................................................................................................... 46

Node ........................................................................................................................................................ 47

RLEC compatibility ...................................................................................................................................... 49

Compatibility with open source Redis ..................................................................................................... 49

Compatibility with open source Redis Cluster ......................................................................................... 50

FAQs ........................................................................................................................................................... 51

I use open source Redis. Why should I switch to RLEC? ....................................................................... 51

What happens when my database fills up? ............................................................................................. 51


Can I keep my data safe and always available? ..................................................................................... 51

What am I allowed to do with the free version?....................................................................................... 51

What do you mean by the term Shard? ................................................................................................... 51

How many Redis databases can I create and manage? ......................................................................... 52

What client can I use to connect to the databases in RLEC? ................................................................. 52

Why should I care about Memcached reliability since it is just a cache? ................................................ 52

What are the RLEC payment schedule and terms? ................................................................................ 52

Troubleshooting .......................................................................................................................................... 53

Network configuration .............................................................................................................................. 53

Installation in environments with no Internet connection ......................................................................... 53

Replica of repeatedly fails ....................................................................................................................... 53

Database metrics are blank during resharding........................................................................................ 54

Creating a support package .................................................................................................................... 54

Subscription agreement .............................................................................................................................. 55

Release notes ............................................................................................................................................. 56

4.2.1-30 – September 21, 2015 ........................................................................................................... 56

4.0.0-30 – May 20, 2015 ...................................................................................................................... 58

0.99.5-24 - February 15, 2015 ............................................................................................................. 60

0.99.5-11 - January 5, 2015 ................................................................................................................. 61


Overview

What is Redis Labs Enterprise Cluster (RLEC)?

Redis Labs Enterprise Cluster (RLEC) enables you to install an enterprise-grade cluster that acts as a

container for managing and running multiple Redis databases in a highly available and scalable

manner, with predictable and stable top performance.

RLEC is based on the proven technology behind the Redis Cloud offering, already used by

thousands of customers.

You can install RLEC in your environment of choice, whether in an on-premises data-center or your

preferred cloud platform, giving you full control of your data and configuration.

What does RLEC enable?

RLEC’s architecture supports multiple databases created by multiple users across the same cluster

infrastructure while keeping the databases completely isolated from one another.

In addition, RLEC gives you the flexibility to run your Redis database in multiple configurations to

accommodate your specific performance and availability needs.

Each such Redis database can separately conform to one of the following deployment templates:

Standalone (a single master Redis server)

Highly-available (a single master with a single slave)

Cluster (multiple master shards)

Highly-available cluster (multiple master and slave shards)

RLEC avoids the "noisy neighbor" phenomenon (common in virtualized or dockerized environments)

by constantly monitoring every shard in the cluster. When a shard exhibits abnormal performance,

RLEC will automatically migrate it to an isolated and less-burdened node in the cluster.

Features Seamless scalability

With RLEC, a dataset can grow beyond the largest node in the cluster and be processed by any

number of cores. By creating a Redis cluster and sharding your dataset across multiple nodes (using

a sharding policy of choice), RLEC overcomes the memory limitations of a single node and the

performance limitations of a single core. Dynamic scaling is easy; you can increase your dataset’s

maximum size by simply changing the memory limit setting, or increase the number of shards with the

click of a button. Scaling, upgrades and downgrades do not incur any downtime.

True high-availability

In addition to replicating a database within the same data-center, one can also replicate a dataset

across data-centers and across regions, to achieve high-availability, disaster recovery and

performance benefits. If a node fails, the data is seamlessly served from a replacement node in the

cluster – without human intervention. RLEC technology is capable of automatically handling node

failures, OSS Redis failures and proxy failures. Auto-failover is carried out in a matter of a few

seconds.

https://redislabs.com/redis-cloud


Built-in data persistence, backups and replication RLEC enables the use of Redis AOF (Append Only File) every second, or snapshots of your dataset

every 1, 6 or 12 hours to persistent storage. In addition, you can back up your dataset periodically or

ad-hoc to an FTP server or to AWS S3. Other cloud storage options, such as Azure Geo-Redundant

Storage, SoftLayer Object Storage and Google Cloud Storage will be added in a future release.

Predictable high-performance Datasets are processed by multiple cores to guarantee the best performance. In addition, RLEC uses

advanced mechanisms to guarantee high-performance, even during intensive disk-access scenarios.

It employs a real-time migration mechanism to isolate high-load databases from other databases

running on the same node.

Multiple dedicated databases You can run multiple databases over a single RLEC deployment, each running in a dedicated process

and in a non-blocking manner.

Unlimited database connections Use as many database connections as you want.

Full-featured management UI RLEC provides a full-featured and easy-to-use management user interface (UI) for setting up,

configuring and monitoring the cluster and each database. In addition, RLEC provides configurable

alert notifications for important events at the single database level as well as the cluster level.

Cluster automation RLEC’s fully-managed solution takes care of all your database scaling, data-persistence tuning,

shards migration and auto-failover needs.

24/7 support Paying customers enjoy our premium 24/7 support via Redis Labs’ online or phone helpdesk.

The technology behind RLEC

RLEC’s unique and patented technology was developed to meet these main objectives:

Decouple data-path from cluster management

Ensure consistent top performance

Simplify and automate operations

Decouple data-path from cluster management

RLEC’s architecture completely decouples the data path from the cluster management in order to optimize the operations of each.


Data-path

The data path is based on multiple zero-latency, multi-threaded proxies that reside on each of the

cluster's nodes to mask the system’s underlying complexity. This model supports multiple proxies per

Redis database and permits the use of any regular or cluster-aware Redis client. This allows the

user’s code base to work as-is. Each proxy forwards clients’ requests to the relevant Redis servers

(shards). When new shards are added to a database, the proxy handles the new routing rules for the

application's requests transparently – immediately scaling the database performance and memory

capacity without any changes to the application itself.

The proxies also support Memcached’s text and binary protocols and perform in-flight translation

between Memcached and Redis protocol semantics. This allows Memcached users to enjoy many of

the features that are not available with open source Memcached, such as built-in replication, auto-

failover, data-persistence and backups and scaling (out/in) without losing data.

Cluster management

The cluster manager is a sophisticated governing function that provides capabilities such as re-sharding, rebalancing, auto-failover, rack-awareness, database provisioning, resource management, data-persistence configuration, backup and recovery. The cluster manager employs multiple watchdog mechanisms at the cluster’s node level and at the process level, which guarantee an instant response to events such as node, rack and data center failures, and can handle multiple failover events at the same time. Because the cluster manager is entirely decoupled from the data path components, changes to its software components do not affect the data path components.

Ensure consistent top performance

Multiple techniques are implemented by RLEC to provide consistent top performance for Redis:

A shared-nothing architecture maximizes the performance of each database. Automatic

migration of shards between nodes is performed when needed.

The zero-latency proxy utilizes several mechanisms for improving performance, including

just-in-time pipelining, socket connections, connection pooling and multiplexing.

Improvements to AOF data persistence and optimized rewrite algorithms.

File system-level improvements enable optimal access to storage and support burst write

operations without degrading (or blocking) database performance.

Simplified and automated operations

RLEC boosts productivity of developers and DevOps by simplifying and automating certain complex

and time-consuming ops-related tasks such as provisioning new databases, updating database

configuration, re-sharding existing databases and rebalancing shards across cluster nodes.

RLEC’s homogeneous, symmetric cluster architecture enables the underlying infrastructure to be fully

heterogeneous, where each node in the cluster can be physical or virtual, with variations in RAM

capacity, storage devices, and number of compute cores. This freedom of configuration allows the

operator to roll out clusters that make the most efficient use of the resources, with minimal system

limitations. For example, a cluster may be based on a few permanent bare-metal servers with

additional virtual servers for scaling out (or in) when load increases or decreases. Built-in configurable

monitoring functions help the operator keep an eye on a cluster's resources’ utilization levels.

RLEC also ensures uninterrupted operation of Redis databases by protecting against out-of-memory

events and tuning memory fragmentation automatically.


All of the above is complemented by an easy-to-use web-based user interface, CLI tools and APIs.

No special experience or knowledge is required to create, manage and monitor Redis clusters, nodes

and databases in RLEC.

You can read more on RLEC’s architecture in the System architecture section.


Installing and Upgrading

This section explains how to install, upgrade and uninstall RLEC.

Hardware and software requirements

Hardware requirements

The table below explains the minimum and the recommended hardware requirements:

Item Description Minimum Requirements

Recommended

# of nodes per cluster

At least 3 nodes are needed in order to support a reliable, highly available deployment that handles process failure, node failure and network split events in a consistent manner.

3 nodes >=3 nodes; must be odd number of nodes to maintain quorum

# of cores per node

RLEC is based on a multi-tenant architecture and can run multiple Redis processes (or shards) on the same core without significant performance degradation. Once the CPU load of a node has reached a certain threshold, RLEC will try to migrate “noisy” shards to a different node in the cluster and will send an alert to the operator. If your application is designed to impose a lot of load on your Redis database, you should make sure you have at least one available core for each shard of your database.

Additional recommendations:

1. If some of the cluster nodes are utilizing more than 80% of the CPU → rebalance the cluster

2. If all of the cluster nodes are utilizing over 80% of the CPU → scale-out the cluster

3. If all of the cluster nodes are utilizing less than 20% of the CPU → scale-in the cluster (until reaching the minimum number of nodes)

4 cores >=8 cores

RAM Defining your RAM size should be part of the capacity planning for your

15GB >=30GB


Redis usage. Since Redis uses a relatively large amount of buffers (i.e. for slave communication, for the client communication and for pub/sub commands) the operator should take extra care to leave at least 20% of the RAM “unused”.

Additional recommendations:

1. If some of the cluster nodes are utilizing over 80% of the RAM → rebalance the cluster

2. If all the cluster nodes are utilizing over 80% of the RAM → scale-out the cluster

3. If all the cluster nodes are utilizing less than 20% of the RAM → scale-in the cluster (until reaching the recommended minimum number of nodes)

Storage For better I/O performance, RLEC enables two storage systems to be connected to each node in the cluster (as described below). It is also highly recommended to use SSD-based storage to avoid performance issues when persisting data to disk.

Ephemeral Storage

Used for storing replication files (RDB format) and cluster log files.

Refer to Persistent and ephemeral storage for more information.

2x node’s RAM size

>=4x node’s RAM size

Persistent Storage

Used for storing snapshot (RDB format) and AOF files over a persistent storage media, which (unlike ephemeral storage) doesn’t get deleted in cases of node failure.

Refer to Persistent and ephemeral storage for more information.

Example of persistent storage devices:

AWS Elastic Block Storage (EBS)

Azure Data Disk

3x node’s RAM size

>=6x node’s RAM size

For extreme ‘write’ scenarios please refer to the Disk size requirements for extreme write scenarios

section to determine the right Persistent Storage size.

Network We recommend running RLEC over a low-latency high-throughput network, wherein each NIC can support a few hundred thousand packets per second.

1G >=10G


That said, RLEC can still perform very well over a single 1Gbps interface network that is used for processing application requests, inter-cluster communication and storage access.

Software requirements

Server

Ubuntu 14.04, 64 bit, Server or Desktop. The Server version is recommended for production

deployments.

CentOS / RHEL 6.5, 64 bit. Requires at least “Minimal Install” configuration.

CentOS / RHEL 7, 64 bit. Requires at least “Minimal Install” configuration. Refer to CentOS /

RHEL 7 firewall configuration for additional considerations.

Client

The following web browsers are supported:

On Windows 7 and Windows 8:

Chrome version 30 and up

Firefox version 35 and up

Internet Explorer version 11 and up

Opera 27 and up

On Ubuntu 14.04:

Chrome version 35 and up

Firefox version 35 and up

Opera 27 and up

Accessing and installing the setup package

Navigate to https://www.redislabs.com/redis-enterprise-downloads and select one of the following

options:

Installation package – Click on the download button for the OS you would like to use in

order to download the .tar file installation package.

AMI – Click on the AWS AMI button for the option you would like to use in order to launch an

instance on Amazon Web Services (AWS).

Note: If you are using the AMI option, or are installing a .tar file package, on an AWS instance, review

the guidelines in Configuration of AWS instances.

If you downloaded the .tar file installation package, install the package on a machine that will serve as

one of the nodes in the cluster by performing the following steps:

https://www.redislabs.com/redis-enterprise-downloads


1. In the operating system command-line-interface (CLI), also referred to as Terminal, use the

cd command to change the location to the directory where you saved the .tar file.

2. Extract the package by using the following command in the CLI:

tar vxf <tarfile name>

3. To initiate the installation in the CLI, use the following command:

./install.sh

4. The installation process begins, and prompts you for input several times.

If you do not run the installation in sudo mode, the installation process prompts you for the

administrator password in order to use sudo mode.

5. After installation is completed successfully, the CLI displays the URL of the page from which

you can initiate cluster setup (refer to Initial setup – creating a new cluster). The URL is in

the following format:

https://<IP address of machine on which you installed the package>:8443

Make sure to save this URL.

If you would like to automate the installation script, there are two options you can use that would

result in a “silent” installation that speeds the process:

1. Run the install script with “-y” as a parameter (i.e. ./install.sh -y), which will silently use

“Y” as a default response to all questions.

2. Run the install script with “-c” and an answers file path as parameters (i.e. ./install.sh –c

<answers file path and name>), which will use the answers provided in your answers file.

Here is sample content for the answers file: systune=yes

ntp=no

firewall=no

Uninstalling

To uninstall RLEC from a server, use the following command in the operating system CLI:

In Ubuntu:

sudo apt-get purge redislabs

In CentOS / RHEL:

sudo yum remove redislabs

After you approve, the uninstall process will run and clean the server of the RLEC installation.

Upgrading

Upgrading RLEC consists of upgrading the software on each of the nodes, in any order you wish.

Upgrading nodes

To upgrade the nodes’ software you need to install the RLEC installation package on all the machines

on which RLEC is installed. Refer to Accessing and installing the setup package for instructions.


During the node upgrade process the services running RLEC are restarted. This may result in a short

interruption to the node itself and the databases running on it.

It is important to upgrade the nodes one by one, and not try to upgrade more than one node at the

same time, in order to ensure cluster and databases’ availability.

If you have the RLEC management UI open in the browser while you are upgrading the nodes you

should make sure to refresh the browser before trying to work again with the UI.

Upgrading databases

Some RLEC upgrades add support for new Redis versions. In these cases, we recommend you

upgrade the databases to the new Redis version, although this is not mandatory because RLEC

upgrades are backwards compatible. RLEC also supports a mix of Redis database versions.

To check whether your Redis database versions match the latest Redis version supported by

RLEC:

In the rladmin CLI, run the status command (refer to rladmin Command Line Interface

(CLI)). If the Redis version is not the latest supported, it will be indicated in the command

output next to the database’s status.

In the Management UI, go to the Cluster > Configuration page. The page lists the latest

Redis version supported.

If the Redis database versions are older than the version supported by RLEC, we recommend

you upgrade your Redis databases.

To upgrade your database:

1. Make sure all the nodes in the RLEC cluster have been upgraded, as described in Upgrading

nodes. You can upgrade the databases only after all nodes in the cluster have been

upgraded.

2. In the rladmin CLI run the upgrade db command for each database. For more information

refer to rladmin Command Line Interface (CLI).

During the database upgrade process the database will be restarted. As a result:

For databases that have replication enabled (refer to Database replication) a failover occurs

prior to restarting the database to ensure that there is no downtime.

For databases that do not have replication enabled, some downtime occurs while the

database is restarting.

For databases that have neither replication nor persistence enabled (refer to Database

persistence), the database loses all its data after it is restarted.


Initial setup – creating a new cluster

A cluster typically consists of several nodes. For production deployment, we recommend an uneven

number of nodes with a minimum of three.

Note: In a cluster that includes only one node some features and capabilities are not enabled, such

as database replication that enables failover to ensure high availability.

To setup a new cluster, you must first install the installation package as described in the previous

section, and then set up the cluster as described below. After the cluster is created you can add

multiple nodes to the cluster (refer to Joining a new node to a cluster).

To create a cluster:

1. In a browser, navigate to https://<name or IP address of machine on which you installed

the package>:8443. For example, if you installed RLEC on a machine with IP address

1.2.3.4, then navigate to https://1.2.3.4:8443.

Note: The RLEC management UI uses SSL encryption. For more information, refer to

Updating web server SSL certificates.

Note: If the machine has both an internal IP address and an external IP address, you should

use the external IP address to access the setup UI.

2. In the window that appears, click Setup.

3. In the Node Configuration page that appears:

a. Enter a path for Persistent storage, or leave the default path. Refer to Persistent and

ephemeral storage.

b. You can enter a path for Ephemeral storage, or leave the default path. Refer to

Persistent and ephemeral storage.

c. In Cluster configuration, select Create new cluster.

d. In Cluster name (FQDN), enter a unique name for the cluster. For guidelines, refer to

How to set the cluster name (FQDN).

e. Choose whether to Enable rack-zone awareness. If you choose to enable it then you

must set the Rack-zone ID for the node. Refer to Rack-zone awareness.

f. Click Next.

4. In the Cluster authentication page, enter a cluster key if you purchased one. Read the

product Terms and Conditions and click Next.

5. In the Cluster registration page you have the option of entering your contact information.

This is needed if you wish to receive upgrade notifications.

6. In the Set admin credentials page, enter the credentials of the cluster administrator. You will

need these credentials in order to add nodes to the cluster, and to regularly login to the

cluster management UI.

After a short wait your cluster is created, and the cluster management UI appears.


7. You can now access any of the management capabilities. You will probably want to create a

database (refer to Creating a new database), or add a new node (refer to Joining a new

node to a cluster).

How to set the cluster name (FQDN)

FQDN (Fully Qualified Domain Name) is the unique cluster identifier name that enables clients to

connect to the different components that are part of the Redis Labs Enterprise Cluster (RLEC). The

FQDN is a crucial component of the high-availability mechanism in RLEC because it is used by the

internal DNS to enable the automatic and transparent failover of nodes, databases shards, and

endpoints, by automatically updating their IP addresses. For more information on these components

and their roles, refer to System architecture.

Once you set the cluster's FQDN it cannot be updated.

The FQDN must always comply with the IETF’s RFC 952 standard and section 2.1 of the RFC 1123

standard.

You have three options for naming the cluster FQDN:

1. DNS

Use this option if you already have your own domain, would like to make the cluster part of your

domain, and are able to update the DNS.

If you choose this option then it is up to you to make sure that the cluster and at least one node

(preferably all nodes) in the cluster are correctly configured in the DNS with the appropriate NS

entries.

For example:

Your domain is: "mydomain.com"

You would like to name the Redis Labs Enterprise Cluster: "redislabscluster"

You have two nodes in the cluster:

"node1" with IP 1.1.1.1

"node2" with IP 2.2.2.2

Then

In the FQDN field you should enter the value: "redislabscluster.mydomain.com", and

you should add the following records in the DNS for mydomain.com:

redislabscluster.mydomain.com NS node1.redislabscluster.mydomain.com

node2.redislabscluster.mydomain.com

node1.redislabscluster.mydomain.com A 1.1.1.1

node2.redislabscluster.mydomain.com A 2.2.2.2

2. Zero-configuration using mDNS

mDNS (Multicast DNS) is a standard protocol that provides DNS-like name resolution and service

discovery capabilities to machines on local networks with minimal to no configuration. Not all clients

support mDNS, so you need to make sure first that the clients that will be connecting to the cluster

http://tools.ietf.org/html/rfc952

http://tools.ietf.org/html/rfc1123


actually have mDNS support, and that the network infrastructure permits mDNS / multicasting

between them and the cluster nodes.

In order to configure the cluster to support mDNS you need to give it a ".local" name in the FQDN.

For example:

You would like to name the Redis Labs Enterprise Cluster: "redislabscluster "

Then

In the FQDN field you should enter the value "redislabscluster.local"

3. No DNS

Use this option only if you do not have a DNS or you have no way to configure your DNS.

Please note that when using this option, failover is not transparent, meaning that when a failure

occurs and the cluster promotes a slave to be a master, the IP used to connect to the new master will

change. When using the DNS or mDNS option, failover can be done transparently and the DNS is

updated automatically to point to the IP of the new master.

When you use this option, the FQDN does not need to have any special format because clients will

use IP addresses instead of hostnames to access the databases so you are free to choose whatever

name you wish.


Cluster Administration

After creating a cluster (as described in Initial setup – creating a new cluster), you can join new

nodes to the cluster, or remove nodes from the cluster.

In the management UI you can remove existing nodes, view and change various cluster settings, as

well as view a wide variety of metrics for the different cluster components.

Joining a new node to a cluster

You can add as many nodes as you wish to a cluster.

To add a node:

8. Install the installation package on the machine that will serve as the new node. Refer to

Accessing and installing the setup package.

9. In a browser, navigate to https://<name or IP address of machine on which you installed

the package>:8443. For example, if you installed the RLEC installation package on a

machine with IP address 5.6.7.8, then navigate to https://5.6.7.8:8443.

Note: The RLEC management UI uses SSL encryption. For more information, refer to

Updating web server SSL certificates.

10. In the window that appears, click Setup.

11. In the Node configuration page:

a. Enter a path for Persistent storage, or leave the default path. Refer to Persistent and

ephemeral storage.

b. You can enter a path for Ephemeral storage, or leave the default path. Refer to

Persistent and ephemeral storage.

c. In Cluster configuration, select Join cluster.

d. Enter the cluster name (refer to Viewing the cluster name), or an IP address of a node

in the cluster.

Note: if you enter the IP address of a node, you must use the internal IP address.

e. Enter the credentials of the cluster administrator, which you defined when you created

the cluster.

f. Click Next.

g. If the cluster is configured to support rack-zone awareness, you are redirected to a page

in which you must set the node’s Rack-zone ID. Refer to Rack-zone awareness.

After a short wait, the node is added to the cluster, and the cluster management UI appears,

displaying the Nodes page. This page lists all the cluster nodes, including the new node you just

added.

Note: The clocks on all nodes must always be synchronized. If the clock in the node you are trying to

join to the cluster is not synchronized with the nodes already in the cluster the action will fail and an

error message will appear indicating that you must synchronize the clocks first. For guidelines, refer

to Synchronizing cluster server clocks to Network Time Protocol (NTP).


Replacing a faulty node

If a node is faulty, its status appears as Failed in the Status column of the Nodes page, and in the

Cluster > Configuration page.

To replace a faulty node:

1. Add a new node, as described in Joining a new node to a cluster.

Make sure the new node has as much available memory as the faulty node.

2. A message appears, informing you that the cluster has a faulty node and that the new node

will replace the faulty node.

3. If the new node has insufficient memory, you will be requested to add a different node, one

with sufficient memory.

Note that if there is a faulty node in the cluster and you are adding a new node, RLEC will enforce

using the new node to replace the faulty one.

Removing a node

There are various reasons why you may want to remove a node:

You would like to temporarily take the node offline for maintenance.

You no longer need the extra capacity, meaning you wish to permanently remove the node.

You would like to replace a faulty node with a healthy node.

You would like to replace a healthy node with a different node.

The following section explains how each of these actions can be achieved, as well as their impact

and considerations.

Make sure to read through these explanations thoroughly before taking any action.

Taking a node offline

Taking a node offline means that you intend to temporarily remove the node from the cluster and

restore it in the near future. As a general rule, when a node is taken offline, the cluster does not try to

migrate all resources off the node, as it would when a node is permanently removed, because the

assumption is that the resources will soon come back online. There are some exceptions to this rule,

as detailed below.

When a node is taken offline the following occurs:

For databases with replication enabled, if a master shard or endpoint is currently on the

node being taken offline, the cluster first failovers the master shard / endpoint and promotes

the slave residing on another node to be the new master. Only then is the node taken offline

to ensure there is no downtime.

Note, that while the node is offline, the cluster does not create alternative slaves because it

assumes that the node will shortly be brought back online. As a result high-availability is not

ensured during this time.

For databases without replication, the cluster first creates new shards and endpoints on a

different node, promotes the newly created shards and endpoints to be the active ones, and


only then takes the node offline. This is practically a migration process to ensure databases’

availability.

Note, that in this case, before trying to take the node offline, you must first ensure that the

cluster has enough capacity for all the resources that will be migrated from the node,

otherwise you will not be able to take the node offline.

Permanently removing a node

Permanently removing a node means you are decreasing the cluster capacity. Before trying to

remove a node you must ensure that the cluster has enough capacity for all resources without that

node, otherwise you will not be able to remove the node.

If there is not enough capacity in the cluster to enable removing the node, you can either delete

databases or add another node instead of the one you wish to remove.

During the removal process, the cluster migrates all resources from the node being removed to other

nodes in the cluster. In order to ensure database connectivity, and database high-availability (when

replication is enabled), the cluster first creates replacement shards or endpoints on one of the other

nodes in the cluster, initiates failover as needed, and only then removes the node.

If a cluster has only two nodes (which is not recommended for production deployments) and some

databases have replication enabled, you will not be able to remove a node.

Replacing a faulty node

If the cluster has a faulty node that you would like to replace, all you need to do is add a new node to

the cluster. The cluster recognizes the existence of a faulty node and automatically replaces it with

the new node.

For guidelines, refer to Replacing a faulty node.

Replacing a healthy node

If you would like to replace a healthy node with a different node, you must first add the new node to

the cluster, migrate all the resources from the node you would like to remove, and only then remove

the node.

For further guidance, refer to Joining a new node to a cluster.

You can migrate resources using the rladmin Command Line Interface (CLI). For guidelines, refer to

rladmin Command Line Interface (CLI).

To take a node offline:

1. Click Take Offline at the top of the Node page of the node to be taken offline.

2. Approve the action.

3. RLEC examines the node and the cluster and takes the needed actions to take the node

offline.

4. At any point you can click the Abort button to stop the process. When aborted, the current

internal action will be completed, and then the process stops.

5. Once the process finishes, the node’s status shows as Offline.


6. You can now either bring the node back online by clicking Bring Online at the top of the

page, or permanently remove the node by clicking on Remove at the top of the page.

To remove a node:

1. Click Remove at the top of the Node page for the node to be removed.

2. Approve the action.

3. RLEC examines the node and the cluster and takes the necessary actions to remove the

node.

4. At any point you can click the Abort button to stop the process. When aborted, the current

internal action is completed, and then the process stops.

5. Once the process finishes, the node will no longer be displayed in the UI.

You can choose to receive an email alerts related to this process, as described in Managing cluster

alerts.

Rebalancing nodes

You can further optimize a cluster’s performance by rebalancing its nodes, a process in which shards

are migrated between nodes in order to ensure that the cluster is well balanced across all nodes. The

rebalancing process is performed by internal optimization algorithms.

Note that the rebalance process can take a relatively long time depending on the number of shards

that need to be migrated.

To initiate the rebalance process:

1. Click Rebalance at the top right of the Nodes page.

2. The Rebalance button changes to Abort and the progress status is displayed.

3. RLEC examines all shards on all nodes to determine whether there is a more optimal shard

placement across nodes and starts migrating nodes accordingly.

4. At any point you can click the Abort button to stop the rebalance process. When aborted, the

current shard migration is completed and then the rebalance process stops.

You can choose to receive email alerts related to this process, as described in Managing cluster

alerts.

Viewing and defining cluster settings

This section explains how to manage RLEC settings.

Managing general settings

You can view and set various cluster settings in the Settings > General page.


Entering a Cluster Key

After purchasing a cluster key, you can enter the key in the Cluster key field, either during initial

cluster creation or at any time afterwards. The key defines various cluster settings, such as the

maximum number of shards you can have in the cluster.

The page also displays general key details.

Viewing the maximum number of allowed shards

The maximum number of allowed shards, which is determined by the Cluster Key, appears in the

Max number of shards field.

Viewing the cluster name

The cluster name appears in the Cluster name field.

Setting your time zone

You can set your time zone in the Timezone field. This is recommended to ensure metrics and log

entries are shown in your preferred time zone.

Managing users

You can view and update the cluster users in the Settings >Team page.

User roles

A user has one of the following roles:

Admin – has full access to view and update anything in the system.

Member – has full access to databases management and read-only access to other parts of

the system.

Viewer – has read-only access to the system.

Adding a user

To add a user:

1. Click the + (plus) sign at the bottom of the table.

2. Enter the name, email, and password of the new user.

3. Select the user role.

4. Mark whether the user will receive email alerts.

5. Click Save.

Updating a user

To update a user:

1. Click the Edit icon at the far right of the user row.

2. You can edit any user details or delete the user.


Managing cluster alerts

On the Settings > Alerts page you can designate which cluster-level events will trigger alert

notifications.

Some alerts can only be turned on or off, such as Node failed or Node rebalance succeeded /

failed. Some alerts require setting a threshold, such as Node memory has reached a certain

percentage of its capacity.

Configured alerts appear in the relevant Cluster or Node Status fields, in the Log page, and can

also be sent via email.

To receive email alerts, you must both mark the checkbox at the bottom of the Alerts page, and mark

the relevant users in the Team page (as described in Managing users).

Viewing cluster metrics

On the Cluster > Metrics page you can view detailed real-time graphs of various metrics for the

entire cluster, as well as specific nodes.

You can choose which resource and metric are shown in each of the two detailed graphs at the top of

the page, as follows:

Below the top two graphs there are two groups of smaller graphs:

The first group displays all available resources: the cluster itself, and each of the cluster

nodes.

The second group displays all available metrics, such as CPU utilization, operations per

second, etc.

In each of the resources graphs, you can click the left arrow to display that resource in the

top left graph, and you can click the right arrow to display that resource in the top right graph.

In each of the metrics graphs, you can choose which metric to show for the two chosen

resources.

Thus, you can compare the same metric for two different resources.

In addition, the scale selector at the top of the page allows you to set the scale of the graphs’ X-axis

(time).

Viewing node metrics

On the Node > Metrics page you can view detailed graphs of various node metrics in real-time.

You can choose which metrics are shown in each of the two graphs at the top of the page, as follows:

Below the top two graphs there is a group of smaller graphs that displays all available

metrics, such as CPU utilization, operations per second, etc.

In each of the bottom graphs, you can click the left arrow to display that metric in the top left

graph; you can click the right arrow to display that metric in the top right graph.


(time).


Viewing the log

The Log page displays the following system and user events: alerts, notifications, and configurations.

For each log entry a timestamp is displayed, as well as a description of the event.

Best practices

This section discusses various best practices for RLEC configuration.

Configuration of AWS instances

When installing RLEC on an AWS instance it is important to take the following points into

consideration:

Choose an instance type that has enough free memory and disk space to meet RLEC

software requirements, as described in Hardware and software requirements.

If you want to have a persistent storage location that is always persistent, it should be located

outside of the instance, so make sure to add a new EBS volume connected to the instance

when you set up the instance in the AWS management console, and later, when setting up

RLEC on the instance, make sure the persistence storage (refer to Persistent and

ephemeral storage) is configured to this volume.

Note that after installing the RLEC package on the instance (refer to Accessing and

installing the setup package) and before running through the setup process (refer to Initial

setup – creating a new cluster), you must give the group ‘redislabs’ permissions to the EBS

volume by running the following command from the OS command-line-interface (CLI): chown

redislabs:redislabs /<ebs folder name>.

When configuring the Security Group:

Define a custom TCP rule for port 8443 to allow web browser access to the RLEC

management UI.

Depending on which cluster naming option you use (refer to How to set the cluster

name (FQDN)):

o If you are using the DNS resolving option, define a DNS UDP rule for port 53 to

allow access to the databases’ endpoints using DNS resolving mechanism.

o Alternatively, if you are using the mDNS option, define a custom UDP rule for

port 5353 to allow mDNS access.

If you would like to create a cluster that has multiple nodes all running as instances on

AWS, you need to define a security group that has an All TCP rule for all ports, 0 –

65535, and add it to all the instances that are part of the cluster. This will ensure that all

the nodes are able to communicate with each other. If you would not like to open all TCP

ports and instead define specific ports and ports ranges, refer to Machine ports

configuration for an comprehensive list of ports being used.

If you would like to have SSH access to the instance, you need to define an SSH TCP

rule for port 22.

After successfully launching the instance, setup the cluster as described in Initial setup – creating a

new cluster.


Using a single DNS server

RLEC deployment includes a DNS server for managing various internal cluster functionalities, such

as automatic failover or automatic migration. Therefore the machine on which an RLEC node is

provisioned should not run any other DNS server except for the one included with the RLEC

installation. The existence of another DNS server on the same machine might cause unexpected and

erroneous behavior.

Persistent and ephemeral storage

For each node in the cluster you can configure both a persistent storage and an ephemeral storage

path.

Persistent storage is mandatory; it is used by the cluster to store information that needs to

persist even if a shard or a node fails. For example, if you choose any type of persistence for

a database, then the persistence information is stored in this location.

Ephemeral storage is optional. If defined, it is used by the cluster to store information that

does not need to persist. This aids in optimization and helps to reduce the load on the

persistent storage.

For disk size requirements refer to the Hardware and software requirements section for general

guidelines regarding the ideal disk size each type of storage, and to the Disk size requirements for

extreme write scenarios section for special considerations when dealing with a high rate of write

commands.

Disk size requirements for extreme write scenarios

For disk size requirements in standard usage scenarios, refer to the Hardware and software

requirements section.

In extreme write scenarios, when AOF is enabled, the AOF rewrite process might require much larger

persistent disk space.

To estimate the required persistent disk space in such cases, use the formula described below.

The required persistent disk space for AOF rewrite purposes in extreme write scenarios,

assuming identical shard sizes:

X (1 + 3Y +Y2)

where:

X = each shard size

Y = number of shards

Following are examples of database configurations and the persistence disk space they would require

in this scenario:

Example 1 Example 2 Example 3 Example 4

Database size (GB) 10 10 40 40

Number of shards 4 16 5 15

Shard size (GB) 2.5 0.625 8 2.67


Required disk space (GB) 73 191 328 723

Synchronizing cluster server clocks to Network Time Protocol (NTP)

It is important to ensure that the clocks of all cluster servers are synchronized.

When you initially install RLEC the install script asks you for permissions to configure a scheduled

Cron job that makes sure the server clock is always synchronized.

If you did not approve configuring this job during the installation process, then you should regularly

use the Network Time Protocol (NTP) to ensure all server clocks are synchronized.

You should use the command for your specific operating system to synchronize the server clock. For

example, in Ubuntu, the following command can be used to synchronize a server’s clock to an NTP

server:

/etc/network/if-up.d/ntpdate

We recommend configuring a scheduled Cron job that executes this command daily on all servers, if

you did not approve the install script to automatically do so.

Updating web server SSL certificates

The RLEC management UI uses a self-signed SSL certificate for HTTP encryption. As a result, when

you open the management UI in the browser you receive a notification stating that the connection is

untrusted. Depending on the browser you might be able to allow the connection for this specific

session, or add an exception so that this site is trusted in future sessions as well.

You can also replace the default certificates with your own certificate using the following steps.

Note: If you choose to update the certificates you must follow these steps on all machines that are

part of the cluster, and on all machines that you add to the cluster in the future.

1. On a cluster machine, navigate to the /etc/opt/redislabs folder.

2. Replace the following files with your own files and rename them with the same exact names

as the original files:

cm_server.key

cm_server.crt

3. If you are using a certificate issued by an intermediate certificate authority (CA), you can also

add to the same folder the chain file with the following name: chain_certs.pem.

4. Run the following command from the OS CLI to restart the service:

service cm_server restart

Repeat these steps on all other machines in the cluster.

Performance optimization

RLEC employs various algorithms to optimize performance. As part of this process RLEC examines

usage characteristics and load and adjusts its run-time configuration accordingly. Depending on your


specific usage characteristics and load, it might take RLEC some time to adjust itself to optimal

performance.

To ensure optimal performance, you must run your workload several times and for a long duration

until performance stabilizes.

In addition, RLEC can be optimized for two different environments:

Cloud environment

Local-network environment

Depending on which configuration you choose, RLEC will use different thresholds to make operation

related decisions.

The default configuration is for cloud environments. If you are running RLEC on a non-cloud

environment then we recommend changing the configuration.

How to change the environment configuration

In the rladmin command line interface (CLI), use the following command:

rladmin tune watchdog profile [default | local-network]

Use default for a cloud environment and local-network for a non-cloud environment.

If after following all of the instructions above, you find that RLEC still does not meet your performance

expectations, contact us at [email protected] to help you optimize RLEC to your specific needs.

Machine ports configuration

Machines that are used as RLEC nodes should ideally have all ports open between them for internal

cluster communication purposes. If for some reason you cannot leave all ports open between the

machines, see below the list of ports and port ranges that are used by RLEC.

In addition, you should make sure that the ICMP protocol is enabled for communications between the

nodes.

By default, the cluster assigns ports in the range of 10,000 – 19,999 to database endpoints. If you

assign a specific port for a database when creating it, even outside of this range, the cluster will only

verify that the assigned port is not already in use.

List of ports and port ranges used by RLEC

Port Description

3333, 3334, 3335, 3336, 36379, 36380

Internal cluster usage

8443 Used to access the management UI

8080, 9443 Used to expose a REST API for the management UI

10000-19999 Used to expose databases externally

20000-29999 Used for internal communications with database shards

mailto:[email protected]


CentOS / RHEL 7 firewall configuration

CentOS / RHEL7 distributions have, by default, a restrictive firewall mechanism based

on firewalld (which in turn configures the standard iptables system). The default configuration

assigns the network interfaces to the public zone and blocks all ports, except 22 (SSH).

RLEC installation on CentOS / RHEL 7 automatically creates two firewalld system services:

A service named redislabs, which includes all ports and protocols needed for

communications between cluster nodes.

A service named redislabs-clients, which includes the ports and protocols needed for

communications external to the cluster.

These services are defined but not allowed through the firewall by default. As part of the

installation process, the installer prompts you to confirm auto-configuration of a default (public) zone

to allow the redislabs service.

This behavior makes the installation process simple and straightforward, but may be insecure if the

machine's network environment is not secured (for example by means of an external firewall, EC2

security groups, etc.). You can use firewalld configuration tools such as firewall-cmd (command line)

or firewall-config (UI) to create more specific firewall policies that allow these two services through

the firewall, as necessary.

Note: If databases are created with non-standard RLEC ports (refer to Machine ports configuration),

you need to explicitly configure firewalld to make sure those ports are not blocked.

Client prerequisites for mDNS

If you choose to use the mDNS protocol when How to set the cluster name (FQDN), then you need

to make sure the required configurations and perquisites are met in order to be able to resolve

databases’ endpoints. These configurations and perquisites are needed on the client machines, as

well as the machines you are using as RLEC nodes in case you are using the Replica of feature.

First, you need to ensure that the machines acting as clients and the machines acting as the nodes

are on the same physical network, or have the networking infrastructure configured to allow

multicasting between them.

Second, you need to install the prerequisite packages, which are different depending on the operating

system you are using:

In Ubuntu:

apt-get install libnss-mdns

In RHEL / CentOS 6.5:

rpm -ivh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

yum install nss-mdns

service avahi-daemon start

In RHEL / CentOS 7:

rpm -ivh http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm

yum install nss-mdns

http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm


service avahi-daemon start

If you are using mDNS along with IPv6 addresses (see more details in Multi-IP & IPv6 support), you

should also make the following update to the “/etc/nsswitch.conf” file:

Update the hosts line to: hosts: files mdns4_minimal [NOTFOUND=return] mdns

Multi-IP & IPv6 support

RLEC supports machines that have multiple IP addresses, some of which can be IPv6 type

addresses.

RLEC related traffic can be logically and physically divide into internal traffic and external traffic:

“Internal traffic” refers to internal cluster communications, such as communications between

the nodes for cluster management purposes.

“External traffic” refers to communications between the clients and the databases, as well as

connections to the management UI in the browser.

When only one IP address exists on a machine serving as an RLEC node, it is used for both internal

and external traffic.

When more than one IP address exists on a machine serving as an RLEC node, the following occurs:

One of the IPv4 addresses is used for internal traffic

Other IPs may be used only for external traffic

At the end of the node bootstrap process, one of the machine’s IPv4 addresses is selected for

internal traffic use.

If you would like to use a different IPv4 address for internal traffic you can change it using the rladmin

Command Line Interface (CLI) node set | add | remove commands.

Note: The internal address can only be updated when the node is in the offline state.

When manually configuring an internal address for a node, make sure the address is valid and bound

to an active interface on the node. Failure to do so will prevent the node from coming back online and

rejoining the cluster.

When configuring external addresses, it is possible to list external addresses that are not bound to an

active interface, but are otherwise mapped or configured to route traffic to the node (e.g., AWS Elastic

IPs, a load balancer VIP, etc.).

rladmin node address commands syntax:

node <id> addr set <addr>

node <id> external_addr set <addr 1> <addr N>

node <id> external_addr [ add | remove ] <addr>

Where

addr - the internal address (can be used only when the node is offline)

external_addr - the external address(es)

Note: While Joining a new node to a cluster during the node bootstrap process, when asked to supply the Cluster Name (FQDN) or an IP of an existing node in the cluster, if you use the node’s IP, then you should supply the node’s internal IP address.


Database configuration

This section explains various aspects of database configuration.

Creating a new database

You can create as many databases as you wish in the cluster, so long as you do not exceed the

available memory or the number of shards you purchased in your subscription.

To create a new database:

1. On the Databases page, click the + (plus) sign underneath the table.

2. Select the type of database to create: a Redis database or a Memcached database.

3. Enter a name for the database.

The database name should adhere to the following rules:

Length: up to 63 characters.

Characters: only letters, digits, and hyphen (‘-‘) are allowed.

Must start with a letter and end with a letter or digit.

Case insensitive: note that uppercase and lowercase letter are treated exactly the same.

4. Set the memory limit of the database. The application displays the total amount of memory

available in the cluster. Refer to Database memory limit for relevant considerations.

Note: The name and the memory limit are the only mandatory parameters.

5. Specify whether to enable replication. For a full explanation, refer to Database replication.

Note that enabling replication affects the total database size, as explained in Database

memory limit.

a. If the cluster is configured to support rack-zone awareness, once you enable

replication you can also choose whether to enable rack-zone awareness for the

database. Refer to Rack-zone awareness.

6. Specify whether to enable data persistence, and if so, what type to employ. For a full

explanation, refer to Database persistence.

7. Next, you can specify security settings:

If you are creating a Redis database, enter a Redis password.

If you are creating a Memcached database, enter a username and password for SASL

Authentication.

8. If you would like to define the port number that will be part of the endpoint used to connect to

the database, you can insert it in the endpoint port number field. If you do not define it the

system will allocate a randomly selected free port. Note that the port number cannot be

changed once defined during database creation. For more information, refer to Machine

ports configuration.


9. Select whether to enable database clustering. If you enable it, select the number of

database shards. For a Redis database, select also the hashing policy. For more information,

refer to Database clustering.

10. Set the data eviction policy (refer to Database eviction policy). This policy is applied when

the total size of the database reaches its memory limit.

11. If you would like the database to be a replica of one or more databases, you can define the

source databases with which this database synchronizes with on an ongoing basis. For more

information, refer to Replica of.

12. Specify whether to perform a periodic back up of the database. For more information, refer

to Database backup. If you specify periodic backup, set the interval and specify the backup

location parameters, as appropriate.

13. Configure database alerts. For more information, refer to Database alerts.

14. Click Activate.

Database memory limit

When you set a database’s memory limit you define the maximum size the database can reach in

the cluster, across all database replicas and shards, including:

Slave shards (if database replication is enabled)

Database shards (if database clustering is enabled)

If the total size of the database in the cluster reaches the memory limit then the data eviction policy

that was defined for the database is applied.

The following examples show how different database configurations affect the total database size.

Example 1

You create a database and:

Set the memory limit to 4 GB

Enable database replication in order to ensure high-availability

The cluster will create 2 shards: a master and a slave. Each of the shards will have a maximum size

of 2 GB. In this case, the maximum dataset size that you will be able to store in the database is 2 GB.

Example 2



Enable database clustering and configure the database to have 3 shards

Do not enable replication

The cluster will create 3 shards. Each of these shards can have a different size depending on the

amount of data stored in it, as long as the total size across all shards does not exceed 6 GB. In this

case, the maximum dataset size you will be able to store in the database is 6 GB.


Example 3



Enable database clustering and configure the database to have 3 shards

Enable database replication in order to ensure high-availability

The cluster will create 6 shards in total - 3 master shards and 3 slave shards. Each of these shards

can have a different size depending on the amount of data stored in it, as long as the total size across

all master shards does not exceed 3 GB. In this case, the maximum dataset size you will be able to

store in the database is 3 GB.

Note: If you edit an existing database that already has data in it, some updates might fail, as they

could cause the total database size to exceed the memory limit. For example, enabling replication

doubles the existing database size, which may then exceed the memory limit. In these cases you

must update the memory limit before you can make the desired change.

Database replication

Database replication provides a mechanism to ensure high availability. When replication is enabled,

your dataset is replicated to a slave node, which is constantly synchronized with the master node. If

the master node fails, an automatic failover occurs and the slave node is promoted to be the new

master node. When the old master node recovers, it becomes the slave node of the new master

node. This auto-failover mechanism guarantees data is served with minimal to no interruption.

When rack-zone awareness is used, there is additional and more advanced logic used to determine

which nodes gets designated as the master or slave, as explained in Rack-zone awareness.

Note that enabling replication has implications for the total database size, as explained in Database

memory limit.

Database persistence

Similarly to open source Redis, RLEC provides several persistence options:

RDB persistence, which performs point-in-time snapshots of your dataset every 1, 6 or 12

hours.

AOF (Append Only File) persistence, which logs every write operation so that it can be

replayed again at server startup, if needed, to reconstruct the original dataset. Commands

are logged using the same format as the Redis protocol itself, in an append-only fashion.

Redis is able to rewrite the log when it gets too big, as a background task.

Database clustering

Redis is (mostly) a single-threaded process. This is a design decision that allows it be extremely

performant, while keeping its implementation simple. However, the downside of that architectural

choice is that Redis cannot be easily scaled. A single Redis process is ultimately bound by the CPU

core it is running on, as well as the amount of memory the server has.

To overcome these limitations, Redis Labs Enterprise Cluster (RLEC) supports database clustering.

A database cluster is a set of Redis processes, in which each process manages a subset of the


database's keyspace. This allows you to overcome scaling challenges via horizontal scaling by using

multiple cores and multiple servers' RAM resources.

In a Redis database cluster, the keyspace is partitioned into hash slots. At any given time a slot

resides on, and is managed by, a single node. Each node that belongs to a Redis database cluster

can manage multiple slots. This division of the key space, a.k.a. sharding, is achieved by hashing the

keys' names, or parts of these (key hash tags), to obtain the slot in which a key should reside.

Despite running multiple Redis processes, database clustering is nearly transparent to the application

using it. The database cluster is accessible via a single endpoint that automatically routes all

operations to the relevant shards, without requiring a cluster-aware Redis client. This allows

applications to benefit from using the database clustering without performing any code changes, even

if they were not designed beforehand to use it.

Abbreviations

Tag / Hash Tag A part of the key that is used in the hash calculation.

Slot / Hash Slot The result of the hash calculation.

Shard Redis process that is part of the Redis clustered database.

When to use sharding

Sharding the keyspace is an efficient way of scaling Redis that should be employed when:

The dataset is large enough that it would benefit from using the RAM resources of more than

one server. Redis Labs recommends sharding a dataset once it reaches a size of 50-100 GB.

The operations performed against the database are CPU intensive, causing performance

degradation. By having multiple CPU cores manage the database's shards, the load of

operations is distributed among them.

Number of shards

When enabling database clustering, you can set the number of database shards. The minimum

number of shards per database is 2 and the maximum depends on the subscription you purchased.

Note: Once database clustering has been enabled and the number of shards has been set, you

cannot disable database clustering or reduce the number of shards. You can only increase the

number of shards by a multiple of the original number of shards. For example, if the original number

of shards was 3, then you can increase to 6, 9,12, etc.

Supported sharding policies

Standard hashing policy

When using the standard hashing policy, a clustered database behaves just like a standard, open-

source, Redis cluster:

Keys with a hash tag: a key's hash tag is any substring between '{' and '}' in the key's name.

That means that when a key's name includes the pattern '{...}', the hash tag is used as input

for the hashing function. For example, the following key names have the same hash tag and

would therefore be mapped to the same slot: foo{bar}, {bar}baz, foo{bar}baz.

Keys without a hash tag: when a key does not contain the '{...}' pattern, the entire key’s

name is used for hashing.


You can use the '{...}' pattern to direct related keys to the same hash slot so that multi-key operations

can be executed on these keys. On the other hand, not using a hash tag in the key's name results in

a (statistically) even distribution of keys across the keyspace's shards. If your application does not

perform multi-key operations then you do not need to construct key names with hash tags.

Custom hashing policy

A clustered database can be configured to use a custom hashing policy. A custom hashing policy is

required when different keys need to be kept together on the same shard to allow multi-key

operations. The custom hashing policy is provided via a set of Perl Compatible Regular Expressions

(PCRE) rules that describe the dataset's key name patterns.

To configure a custom hashing policy, enter the regular expression (RegEx) rules that identify the

substring in the key's name - hash tag - on which hashing will be done. The hashing tag is denoted in

the RegEx by the use of the `tag` named sub-pattern. Different keys that have the same hash tag will

be stored and managed in the same slot.

Once you enable the custom hashing policy, the following default RegEx rules are implemented. You

should update these rules to fit your specific logic:

RegEx Rule Description

.*\{(?<tag>.*)\}.* Hashing is done on the substring between the curly braces.

(?<tag>.*) The entire key's name is used for hashing.

You can modify existing rules, add new ones, delete rules, or change their order to suit your

application's requirements.

Custom hashing policy notes and limitations

1. You can define up to 32 RegEx rules, each up to 256 characters.

2. RegEx rules are evaluated in their order and the first rule matched is used. Therefore, strive

to place common key name patterns at the beginning of the rule list.

3. Key names that do not match any of the RegEx rules will trigger an error.

4. The '.*(?<tag>)' RegEx rule forces keys into a single slot because the hash key will always be

empty. Therefore, when used, this should be the last, catch-all rule.

5. The following flag is enabled in the regular expression parser:

PCRE_ANCHORED: the pattern is constrained to match only at the start of the string being

searched.

Changing the sharding policy

The sharding policy of a clustered database can be changed. However, most sharding policy changes

trigger the deletion (i.e., FLUSHDB) of the data before they can be applied.

Examples of such changes include:


Changing the hashing policy from standard to custom, or vice versa.

Changing the order of custom hashing policy rules.

Adding new rules in the custom hashing policy.

Deleting rules from the custom hashing policy.

Note: The recommended workaround for updates that are not enabled, or require flushing the

database is to back-up the database and import the data to a newly configured database.

Multi-Key operations

Operations on multiple keys in a clustered database are supported with the following limitations:

Multi-key commands: Redis offers several commands that accept multiple keys as

arguments. In a clustered database, most multi-key commands are not allowed across tags.

The following multi-key commands are allowed across tags: DEL, MSET, MGET, KEYS,

HMGET, HMSET, FLUSHALL.

Please note that when using these commands in a sharded setup, the command is

distributed across multiple shards and the responses from all shards are combined into a

single response.

All other multi-key commands are not allowed across tags and if used across tags will return

an error.

Examples of such commands are: BITOP, BLPOP, BRPOP, BRPOPLPUSH, MSETNX,

RPOPLPUSH, SDIFF, SDIFFSTORE, SINTER, SINTERSTORE, SMOVE, SORT, SUNION,

ZINTERSTORE, ZUNIONSTORE.

Transactions: All operations within a WATCH / MULTI / EXEC block should be performed on

keys that have the same tag.

Lua scripts: All keys used by a Lua script must have the same tag and must be provided as

arguments to the EVAL / EVALSHA commands (as per the Redis specification). Using keys

in a Lua script that were not provided as arguments might violate the sharding concept but

will not result in the proper violation error being returned.

Renaming keys: The use of the RENAME / RENAMENX commands is allowed only when

the key's original and new values are mapped to the same tag.

Database eviction policy

The eviction policy designates a data eviction method to use when the database size reaches its limit.

You can select any of the following:

noeviction: returns an error if the memory limit has been reached when trying to insert more

data

allkeys-lru: evicts the least recently used keys out of all keys

allkeys-random: randomly evicts keys out of all keys

volatile-lru: evicts the least recently used keys out of all keys with an "expire" field set

volatile-ttl: evicts the shortest time-to-live and least recently used keys out of all keys with an

"expire" field set

volatile-random: randomly evicts keys with an "expire" field set


Replica of

The Replica of feature enables you to designate a database to be a replica (destination) of one or

more databases (sources). Once you have done so and the initial data load is completed, all write

commands are synchronized from the source(s) to the destination. This allows you to keep a

database (destination) that is an exact replica of other databases.

This configuration can be very useful, for example, if you would like to distribute the read load of your

application across multiple databases. In addition, this feature can be used to do a one-time

synchronization of a database, either within RLEC or external to RLEC, to a database in RLEC.

The “Replica of” is defined in the context of the destination database by specifying the source

databases.

A destination database can have a maximum of five source databases.

If only one source is defined, then the command execution order in the source is kept in the

destination. However, when multiple sources are defined, commands that are replicated from the

source databases are executed in the order in which they reach the destination database. As a result,

commands that were executed in a certain order when compared across source databases might be

executed in a different order on the destination database.

Note: The Replica of feature should not be confused with the In-memory Database replication

feature, which is used to create a master / slave configuration that ensures database high availability.

Replica of sources

RLEC has a security mechanism in which an internal admin password is assigned to each database.

This password helps protect the database from being used as a replica source and is required in

order to define another database as a replica target of this database.

The source databases can be located in the same Redis Labs Enterprise Cluster (RLEC) as the

destination database, in a different RLEC, or they can be Redis databases that are not part of an

RLEC.

When a source database is from within RLEC, the source URL has the following format:

<database name>: redis://admin:<internal database password>@<database endpoint

with port> where the internal database password is the password is automatically assigned

and populated by RLEC. In the RLEC management UI, when you enter the textbox to define

a replica source from within the target database page, the list of existing databases is shown

with the appropriate URL, including the internal admin password, already set up.

When a source database is from a different RLEC, the source URL has the same exact

format as indicated above (except for the <database name>: prefix), but in this case the URL

does not show up as an option in the UI. In order to configure the target database as a replica

of a database from a different RLEC you need to extract the source database URL, including

the internal admin password, from the source database. This can be done in the UI from the

source database page by clicking on the “Get Replica of source URL” link next to the

Endpoint field. In addition, you can regenerate the internal admin password from the same

UI. If you regenerate the internal admin password, any existing replica destinations already

configured will stop working until you update them.

Compression: when a source database is located on a different RLEC, there is also an

option to enable compression of the data being replicated. Refer to the “Replica of” data

compression section for additional details.


When a source database is external to RLEC, the source URL has the following format:

redis://:<redis password>@<hostname>:<port> where the password is the Redis

password assigned by the user, represented with URL encoding escape characters. If no

password was defined for the database then the following format should be used:

redis://hostname:port

When multiple sources are defined there is no meaning to the order in which they are defined or

presented.

If you make changes to the definition of the sources (such as edit, add or delete a source), then the

synchronization process is restarted from scratch for all the source databases.

Note: If you used the mDNS protocol when naming the cluster name (FQDN), then you must ensure

the client mDNS perquisites are installed in order for the Replica of feature to work. Refer to the

Client prerequisites for mDNS for additional details.

Replication process

When a database is defined as a replica of other databases, all its existing data is deleted and

replaced by data that is loaded from the source databases.

Once the initial data load is completed, an ongoing synchronization process takes place to keep the

destination synchronized with its sources. During the ongoing synchronization process there is a

certain delay between the time when a command was executed on the source and when it is

executed on the destination. This delay is referred to as the Lag.

When a synchronization error occurs, then depending on the error type, the process might stop,

or it might continue running on the assumption that the error will be automatically resolved (more

details below).

In addition, the user can manually stop the synchronization process.

When the process is in the stopped state – whether stopped by the user or by the system – the user

can restart the process. Restarting the process causes the synchronization process to be

restarted from the beginning.

Replica of status

The replication process can have the following statuses:

Syncing - indicates that the synchronization process has started from scratch. Progress is

indicated in percentages (%).

Synced - indicates that the initial synchronization process was completed and the destination

is synchronizing changes on an ongoing basis. The Lag delay in synchronization with the

source is displayed as a duration of time.

Sync stopped - indicates that the synchronization process is not currently running and the

user needs to restart it in order for it to resume. This status occurs if the user stops the

process, or if certain errors arose that prevent synchronization from continuing without

manual intervention (more details below).

The statuses above are displayed for each source database. In addition, a timestamp is shown for

each source indicating when the last command from the source was executed on the destination.

The system also displays the destination database status as an aggregate of the statuses of all the

sources.


Note: Please refer to the troubleshooting section Replica of repeatedly fails if you encounter issues

with the Replica of process.

Synchronization errors

Certain errors that occur during the synchronization process require user intervention for their

resolution. When such errors occur the synchronization process is automatically stopped.

For other errors the synchronization process continues running on the assumption that the error will

be automatically resolved.

Examples of errors that require user intervention for their resolution, and cause the synchronization

process to stop, include:

Error authenticating with the source database

Cross-slot violation error while executing a command on a sharded destination database

Out-of-memory error that occurs on a source or on the destination database

Example of an error that does not stop the synchronization process:

Connection error with the source database. Occasionally a connection error may result from

temporary network issues that resolve themselves. Depending on the connection error and its

duration, the process might be able to start syncing from the last point it reached (partial

sync), or require a complete resynchronization across all sources (full sync).

“Replica of” data compression

When the Replica of is defined across different RLEC clusters, it may be beneficial to compress the

data flowing through the network (depending on where the clusters physically reside and the available

network).

Compressing the data reduces the traffic and can help:

Resolve throughput issues

Reduce network traffic costs

Compressing the data has drawbacks as well, which is why it should not always be turned on by

default. For example:

It uses CPU resources to compress the data before sending it to the network and

decompress it on the other side

It takes time to compress and decompress the data which can increase latency

If traffic is too fast and the compression takes too much time it can cause the replication

process to fail and be restarted

In the RLEC management UI, when designating a Replica of source from a different RLEC, there is

also an option to enable compression. When enabled, gzip compression with level -6 is utilized.

Database clustering (sharding) implications

If a source database is sharded, the entire database is treated as a single source for the destination

database.


If the destination database is sharded, then when the commands replicated from the source are

executed on the destination database and the destination database’s hashing function is executed to

determine to which shard/s the command refers.

Database backup

You can back up your dataset to an FTP server, Amazon S3, or OpenStack Object Storage (“Swift”),

periodically, or ad-hoc. You cannot back up to the local machine. Other cloud storage options, such

as Azure Geo-Redundant Storage, SoftLayer Object Storage and Google Cloud Storage will be

added in a future release.

The backup process creates compressed (.gz) RDB files.

Note that it is your responsibility to manage the backup location; RLEC does not delete old backups

or monitor the amount of space left in the folder.

FTP location

To backup to FTP you must ensure permissions to read and write to the FTP location.

If needed, you can specify the username and password for the FTP location as part of the path with

the following structure: ftp://user:password@host:port/path/

Amazon S3 or OpenStack Swift location

To backup to Amazon S3 or OpenStack Swift, you simply select the location type and enter the details in relevant fields. You should make sure you have all these details from your chosen platform first.

Periodic backup

You can configure the database to be periodically backed up to a chosen location. If you do so,

specify the interval of how often to back up the database: every 4, 12, or 24 hours.

Ad-hoc backup - Export

You can, at any time, backup any database as described in Exporting data from a database.

Backup of a sharded database

If you backup a database configured for database clustering, the system creates a backup file for

each shard, and places all these backup files in the specified backup location.

Database alerts

In the Alerts section of the database page, you can designate which database events will trigger alert

notifications.

Some alerts can only be turned on or off, such as Periodic back-up failed. Some alerts require

setting a threshold, such as Dataset size has reached a certain percentage of its limit.

Configured alerts appear in the database Status field, in the Log page, and can also be sent by

email.

To enable receiving email alerts you must both mark the checkbox at the bottom of the section, and

mark the relevant users in the Team page (as described in Managing users).


Importing data to a database

You can import data into a Redis database from an HTTP location, FTP server, Amazon S3, or

OpenStack Object Storage (“Swift”).

You can either import from a single file, or from multiple files in case you are importing from a backup

of a sharded database, refer to the Backup of a sharded database section in Database backup.

To import data into a database:

1. Click the relevant database row in the Databases page. The selected database page

appears.

2. Select the Configuration tab.

3. Click Import at the bottom of the page.

4. A warning appears, informing you that importing a dataset into the database will erase all

existing database content. Click Continue.

5. In the popup that appears, select the location type and populate the relevant fields. The RDB

file path / object name should have an RDB format or a compressed RDB format file. To

import from multiple files, enter each file path in a new line.

6. You can also choose whether you would like to receive email notifications related to the

import process.

7. Click Import.

FTP location

To import from FTP you must ensure permissions to read from the FTP location.




To backup to Amazon S3 or OpenStack Swift, you simply select the location type and enter the details in relevant fields. You should make sure you have all these details from your chosen platform first.

Exporting data from a database

Using the Export option, you can back up a database at any time to an FTP server, Amazon S3, or

OpenStack Object Storage (“Swift”). You cannot back up to the local machine. Other cloud storage

options, such as Azure Geo-Redundant Storage, SoftLayer Object Storage and Google Cloud

Storage will be added in a future release.

To perform database export:


appears.


3. Click Export at the bottom of the page.

4. In the popup that appears, choose the location type and populate the relevant fields.


5. You can also choose whether you would like to receive email notifications related to the

export process

6. Click Export.

FTP location

To export to FTP you must ensure permissions to read and write to the FTP location.




To export to Amazon S3 or OpenStack Swift, you simply select the location type and enter the details in relevant fields. You should make sure you have all these details from your chosen platform first.

Updating database configuration

You can change the configuration of a database at any time.

To edit the configuration of a database:


appears.


3. Click Edit at the bottom of the page.

4. The database settings appear. You can edit any of the configurable parameters. For

explanation, refer to Creating a new database.

5. Click Update.

Deleting a database

To delete a database:


appears.


3. Click Delete at the bottom of the page.

4. Confirm the deletion.

Viewing database metrics

On the Database > Metrics page you can view detailed real-time graphs of various database metrics,

as well as shard-related metrics.

Database metrics

You can choose which metrics are shown in each of the two detailed graphs at the top of the page, as

follows:


Below the top two graphs there is a group of smaller graphs which display all available

metrics, such as CPU utilization, operations per second, etc.

In each of the bottom graphs, you can click the left arrow to display that metric in the top left

graph, and the right arrow to display that metric in the top right graph.


(time).

Shard metrics

You can choose which resource and metric are shown in each of the two detailed graphs at the top of

the page, as follows:

Below the top two graphs there are two groups of smaller graphs:

The first group displays all available resources: the database itself, and each of the

database shards. These can be master shards, slave shards (if Database replication is

enabled), or shards that take part of a clustered database (refer to Database clustering).

The shard name refers to the shard role (whether master or slave).

Each small shard graph also lists the node it is located on, and the hash slot to which it is

mapped, if it is part of a clustered database.

Note that corresponding master / slave shards in a clustered database have the same

hash slot.

The second group displays all available metrics, such as operations per second, used

memory, etc.

In each of the resources graphs, you can click the left arrow to display that resource in the

top left graph, and the right arrow to display that resource in the top right graph.

In each of the metrics graphs, you can click the left arrow to display that metric in the top left

graph, and the right arrow to display that metric in the top right graph.


(time).

Viewing Redis Slow Log

On the Database > Slow Log page, you can view Slow Log details for Redis databases.

Redis Slow Log is one of the best tools for debugging and tracing your Redis database, especially if

you experience high latency and high CPU usage with Redis operations. Because Redis is based on

a single threaded architecture, Redis Slow Log can be much more useful than slow log mechanisms

of multi-threaded database systems such as MySQL Slow Query Log.

Unlike tools that introduce lock overhead (which makes the debugging process very complex) Redis

Slow Log is highly effective at showing the actual processing time of each command.

Redis Labs Enterprise Cluster (RLEC) includes enhancements to the standard Redis Slow Log

capabilities that enable you to analyze the execution time complexity of each command. This

enhancement can help you better analyze Redis operations, allowing you to compare the differences

between execution times of the same command, observe spikes in CPU usage, and more.

This is especially useful with complex commands such as ZUNIONSTORE, ZINTERSTORE, and

ZRANGEBYSCORE.

http://redis.io/commands/slowlog

https://groups.google.com/forum/?fromgroups=#!searchin/redis-db/Strange$20growing$20cpu$20usage$20in$202.6.7/redis-db/zY9FMEMBnJA/Eenw58zspe4J

http://redis.io/commands/zunionstore

http://redis.io/commands/zinterstore

http://redis.io/commands/zrangebyscore

http://redis.io/commands/zrangebyscore


The enhanced RLEC Slow Log adds the Complexity Info field to the output data.

View the Complexity Info data by its respective Command in the table below:

Command Value of interest Complexity

LINSERT N - list len O(N)

LREM N - list len O(N)

LTRIM N - number of removed elements O(N)

PUBLISH N - number of channel subscribers

M - number of subscribed patterns

O(N+M)

PSUBSCRIBE N - number of patterns client is subscribed to

argc - number of arguments passed to the command

O(argc*N)

PUNSUBSCRIBE N - number of patterns client is subscribed to

M - total number of subscribed patterns


O(argc*(N+M))

SDIFF N - total number of elements in all sets O(N)

SDIFFSTORE N - total number of elements in all sets O(N)

SINTER N - number of elements in smallest set


O(argc*N)

SINTERSTORE N - number of elements in smallest set


O(argc*N)

SMEMBERS N - number of elements in a set O(N)

SORT N - number of elements in the list/set/zset

M - number of elements in result

O(N+M*log(M))O(N)

when no sorting

SUNION N - total number of elements in all sets O(N)

SUNIONSTORE N - total number of elements in all sets O(N)

UNSUBSCRIBE N - total number of clients subscribed to all channels O(N)

ZADD N - number of elements in the zset O(log(N))


Command Value of interest Complexity

ZCOUNT N - number of elements in the zset

M - number of elements between min and max

O(log(N)+M)

ZINCRBY N - number of elements in the zset O(log(N))

ZINTERSTORE N - number of elements in the smallest zset

K - number of zsets

M - number of elements in the results set

O(N*K)+O(M*log(M))

ZRANGE N - number of elements in the zset

M - number of results

O(log(N)+M)

ZRANGEBYSCORE N - number of elements in the zset


O(log(N)+M)

ZRANK N - number of elements in the zset O(log(N))

ZREM N - number of elements in the zset


O(argc*log(N))

ZREMRANGEBYRANK N - number of elements in the zset

M - number of elements removed

O(log(N)+M)

ZREMRANGEBYSCORE N - number of elements in the zset

M - number of elements removed

O(log(N)+M)

ZREVRANGE N - number of elements in the zset


O(log(N)+M)

ZREVRANK N - number of elements in the zset O(log(N))

ZUNIONSTORE N - sum of element counts of all zsets

M - element count of result

O(N)+O(M*log(M))


Rack-zone awareness

The rack-zone awareness feature enables more sophisticated placement of database shards and

endpoints in high-availability scenarios, compared to non-rack-zone aware configuration.

When you enable rack-zone awareness in the cluster, you need to assign to each node a rack-zone

ID. This ID is used to map the node to a physical rack or a logical zone (which itself is mapped to a

physical rack). The cluster can then ensure that master shards and corresponding slave shards, and

similarly endpoints, are placed on nodes in different racks. Thus, if a rack fails and master / slave

failover takes place, the slaves are much more likely to be available because they reside on different

racks.

There is no limitation on the number of racks-zones per cluster; each node can belong to a different

rack or multiple nodes can belong to the same rack.

Rack-zone awareness affects various cluster, node and database related actions, such as: node

rebalancing, node removal, nodes being taken offline, node replacement, database failover, shard

and endpoint migration, etc.

To benefit from rack-zone awareness, you need to configure it at the cluster, node and database

levels.

Cluster/node configuration

In order for the cluster to support rack-zone awareness, you need to enable it when initially creating

the cluster. Afterwards, every time you add a new node to the cluster you need to define a “rack-zone

ID” for the node.

The rack-zone ID must adhere to the following rules:

Length: up to 63 characters.

Characters: only letters, digits and hyphen (‘-‘) are allowed.

Must start with a letter and end with a letter or a digit.

Case insensitive: note that uppercase and lowercase letter are treated exactly the same.

Database configuration

Once the cluster has been configured to support rack-zone awareness, and it includes nodes with at

least two different rack-zone IDs, then you can create a rack-zone aware database.

Rack-zone awareness is only relevant for databases that have slave shards, meaning that the

database has replication enabled. After you enable replication for the database you can choose to

also enable rack-zone awareness.

If you do not enable rack-zone awareness for the database, the cluster does not ensure that the

nodes on which the masters and corresponding slaves are placed, reside in different racks. However,

the cluster still ensures master and slaves are placed on different nodes, in order to ensure high-

availability, as it would in a non-rack-zone aware cluster.


rladmin Command Line Interface (CLI)

RLEC includes a Command Line Interface (CLI), called rladmin that can be used for various

advanced administrative tasks. Some of these tasks can also be performed through the management

UI and some are unique to the CLI.

The following are examples of tasks that can be performed through the CLI:

View the detailed status of the cluster, including information about each node, each

database, each shard, etc.

Execute failover between a master and its slave shard or a master and its slave’s endpoint.

Migrate a specific shard or endpoint between nodes.

Tune specific internal parameters of a database.

Upgrade the version of a database to the most current version supported by the cluster.

Accessing the rladmin CLI

To open the rladmin CLI:

1. Open the operating system CLI.

2. Change user to root using the following command: sudo su -

3. Enter the following command in the CLI: rladmin

Note that if the CLI does not recognize the rladmin command then you might need to run the

following command to load the needed configuration first: bash –l

4. A message confirms that you are now running rladmin.

rladmin features

In the rladmin CLI you can:

Enter “?“ to view the full list of available commands.

Type “help“ followed by the name of a command to view a detailed explanation of the

command and its usage.

Use the Tab key for automatic command completion.


System architecture

Redis Labs Enterprise Cluster (RLEC) is a container for running multiple Redis databases in a highly

available and scalable manner, while ensuring top performance.

The cluster is made up of one or more nodes that work together. At any point in time, one of the

nodes is defined as the cluster master node which is in charge of making cross-node cluster level

decisions such as cluster health related decisions, migration of shards across nodes, provisioning

and de-provisioning requests, fetching cluster statistics, etc.

Note: The cluster ‘master’ role refers to the node on which the cluster manager entity is now

managing the entire cluster, and is completely different from the ‘master’ role associated with a Redis

deployment.

Note: In RLEC, Memcached databases are implemented as Redis instances, as described below,

and are exposed as Memcached using protocol translation at the proxy level.

The following figure illustrates the major components of the RLEC architecture.

The following sections explain the major components and concepts of the RLEC architecture.


Node

A node is a physical or virtual machine on which the RLEC installation package was installed and the

setup process was run in order to make the machine part of the cluster.

Each node is a container for running multiple open source Redis instances, referred to as “shards”.

The recommended configuration for a production cluster is an uneven number of nodes, with a

minimum of three. Other configurations should not be used for production purposes. Note that in

some configurations, certain functionalities might be blocked. For example, if a cluster has only one

node you cannot enable database replication, which helps to achieve high availability.

A node is made up of several components, as detailed below, and works together with the other

cluster nodes.

Redis Instance – Shard

As indicated above, each node serves as a container for hosting multiple database instances,

referred to as “shards”.

RLEC supports various database configurations:

Standard Redis database – A single Redis shard with no replication or data sharding.

Highly available Redis database – Every database master shard has a replicated slave

shard, so that if the master shard fails, the cluster can automatically failover to the slave

shard with minimal to no impact. Master and slave shards are always placed in separate

nodes to ensure high availability.

Clustered Redis database – The data stored in the database is split across a number of

shards. The number of shards can be defined by the user. Various performance optimization

algorithms define where shards are placed within the cluster. During the lifetime of the cluster

these algorithms might migrate a shard between nodes.

Clustered & highly available Redis database – Each master shard in the clustered

database has a slave shard, enabling failover if the master shard fails.

Proxy

Each node includes one or more zero-latency multithreaded proxies (written in low level C) that mask

the underlying system complexity. The proxies are in charge of forwarding Redis operations to the

database shards.

The proxy simplifies the cluster operation, from the application or Redis client point of view enabling

the use of a standard Redis client. The zero-latency proxy is built over a cut-through architecture and

employs various optimization methods. For example, to help ensure high-throughput and low-latency

performance, the proxy might use instruction pipelining even if not instructed to do so by the client.

Database Endpoint

Each database is served by a database endpoint that is managed by the proxy. The endpoint is in

charge of forwarding Redis operations to specific database shards. If database replication is enabled,

then each database has two endpoints: a master endpoint and a slave endpoint.

If the master shard fails and the slave shard is promoted to master, the master endpoint is updated to

point to the new master shard.


If the master endpoint fails, the slave endpoint is promoted to be the new master endpoint, and is

updated to point to the master shard.

Similarly, if both the master shard and the master endpoint fail, then both the slave shard and the

slave endpoint are promoted to be the new master shard and master endpoint.

As can be understood from the above, a shard and its endpoint do not have to reside within the same

node in the cluster.

In the case of a clustered database with multiple database shards, only one master endpoint acts as

the master endpoint for all master shards, forwarding Redis operations to all shards as needed.

Cluster Manager

The cluster manager is in charge of all node management related tasks, and the cluster manager in

the master node is also in charge of all the cluster related tasks.

The cluster manager (written in Python) is designed in a way that is totally decoupled from the Redis

operation. This enables RLEC to react in a much faster and accurate manner to failure events, so

that, for example, a node failure event will trigger mass failover operations of all the master endpoints

and master shards that are hosted on the failed node. In addition, this architecture guarantees that

each Redis shard is only dealing with processing Redis commands in a shared-nothing architecture,

thus maintaining the inherent high-throughput and low-latency of each Redis process. Lastly, this

architecture guarantees that any change in the cluster manager itself does not affect the Redis

operation.

Some of the main functionalities of the cluster manager include:

Deciding where shards will be created

Deciding when shards will be migrated and to where

Monitoring database size

Monitoring databases and endpoints across all nodes

Running the database re-sharding process

Running the cluster rebalancing process

Running the database provisioning and de-provisioning processes

Gathering operational statistics

Enforcing licensing limitations

Management UI

Each node runs a web server that is used to provide the user with access to the management user

interface (UI). The management UI enables viewing and managing the entire cluster, so it does not

matter which node is used to access the UI.

REST API

Each node exposes a REST API to the cluster. Using the REST API you can automate various tasks

that are managed through the management UI.

For example, using the REST API you can instruct the cluster to create a new database, update an

existing database, or fetch various statistics.


RLEC compatibility

This section explains RLEC’s compatibility with the open source technology.

Compatibility with open source Redis

Redis Labs Enterprise Cluster (RLEC) is fully compatible with open source Redis. Redis Labs

contributes extensively to the open source Redis project and employs it in RLEC. As a rule, Redis

Labs adheres to the open source’s specifications and makes every effort to update RLEC with the

latest version of Redis.

Any standard Redis client can be used with RLEC.

That said, some of Redis’ functionality is not applicable in the context of RLEC, as follows:

Shared databases are not supported in RLEC given their potential negative impact on

performance. We recommend using dedicated databases instead. Therefore, the following

commands are blocked and will produce an error when executed:

MOVE

SELECT

Data persistency and backups are managed from RLEC’s management UI, so the following

commands are blocked:

BGREWRITEAOF

BGSAVE

SAVE

LASTSAVE

Because replication is managed automatically by RLEC and because it could present a

security risk, the following commands are blocked:

MIGRATE

REPLCONF

SLAVEOF

SYNC/PSYNC

Commands that are not relevant for a hosted Redis instance are blocked:

DEBUG OBJECT/SEGFAULT

OBJECT

SHUTDOWN

Lastly, only a subset of Redis’ configuration settings (via CONFIG GET/SET) is applicable to

RLEC. Attempts to get or set a configuration parameter that is not included in the following

list will result in an error:


hash-max-ziplist-entries

hash-max-ziplist-value

list-max-ziplist-entries

list-max-ziplist-value

lua-time-limit (value must be between 0 and 60000)

notify-keyspace-events

set-max-intset-entries

slowlog-log-slower-than (value must be larger than 1000)

slowlog-max-len (value must be between 128 and 1024)

zset-max-ziplist-entries

zset-max-ziplist-value

Compatibility with open source Redis Cluster

RLEC works with any standard Redis client, both standalone Redis clients and Redis Cluster clients.

For details regarding the RLEC architecture, refer to the System architecture section.


FAQs

This section includes various frequently asked questions.

I use open source Redis. Why should I switch to RLEC?

RLEC helps you overcome the limitations of open source Redis. It provides scalable Redis

databases, even for datasets larger than the largest server. Moreover, it scales without the hassle of

setting up or dealing with nodes and clusters, while maintaining the same endpoints.

Achieving true high-availability is very easy with RLEC. At the click of a button, you can set up in-

memory replication, data persistence, and remote backups.

RLEC also shields your Redis dataset from virtual environment noises, resulting in stable and

predictable top performance.

And last but not least – it is fully automated, so you do not need to worry about scaling or recovery

from failures.

What happens when my database fills up?

As detailed in the open source Redis FAQ, under “What happens if Redis runs out of memory?”:

…[you] can use the “maxmemory” option in the config file to put a limit to the memory

Redis can use. If this limit is reached Redis will start to reply with an error to write

commands (but will continue to accept read-only commands), or you can configure it

to evict keys when the max memory limit is reached in the case you are using Redis

for caching.

You can set the maxmemory value of each RLEC database in the management UI using the

Memory limit property, as well as configure an eviction policy by setting it to any of the standard

Redis behaviors, without interrupting database operations.

Can I keep my data safe and always available?

RLEC offers a comprehensive suite of high-availability provisions, including in-memory replication,

persistent storage and backups.

What am I allowed to do with the free version?

You can use the full capabilities of RLEC, but you cannot deploy RLEC to production. In addition, the

free version allows a maximum of four shards. The free version does not provide the same support

options as the paid version. Finally, no SLA is provided with the free version.

What do you mean by the term Shard?

A shard is any type of provisioned Redis instance, such as a master copy, slave copy, database

shard that is part of a clustered database, etc.

http://redis.io/topics/faq


How many Redis databases can I create and manage?

The number of databases is unlimited. The limiting factor is the available memory in the cluster, and

the number of shards in the subscription (refer to What do you mean by the term Shard?).

Note the impact of the specific database configuration on the number of shards it consumes. For

example:

Enabling database replication, without enabling database clustering, creates two shards: a

master shard and a slave shard.

Enabling database clustering creates as many database shards as you configure.

Enabling both database replication and database clustering creates double the number of

database shards you configure.

What client can I use to connect to the databases in RLEC?

RLEC works with all existing standard Redis clients; it does not require you to use any special clients.

Why should I care about Memcached reliability since it is just a

cache?

Developers use Memcached because they care about their application performance. However, with

the unreliable nature of Memcached, you may lose some or all of your Memcached data (part of

which may have taken your app hours or days to calculate), due to a node failure, scaling event,

software upgrade, or process restart.

If memory loss occurs, your main database will be overloaded with queries, which may result in

dramatic performance degradation or even a complete outage of your application.

Furthermore, many web applications and popular platforms such as Magento, Drupal, and Joomla

store session-cookies in Memcached without backing them up to persistent storage. Session losses

due to Memcached failures could have severe implications on your business. For example, if you are

running an e-commerce site, users might be forced to immediately logout and their shopping carts

would be emptied.

Memcached is not just a cache, but rather a cornerstone of your application’s performance. Its

reliability should be your concern. RLEC provides Memcached in-memory replication capability, a

storage engine and an instant auto-failover mechanism, which guarantee highly available

Memcached.

What are the RLEC payment schedule and terms?

Contact us at [email protected] to learn about RLEC pricing.



Troubleshooting

This section includes various troubleshooting tips and tricks.

Network configuration

Nodes in the same cluster must be on the same VLAN. If you cannot host the nodes on the same

VLAN then you must configure all ports between them to be open. For more information, refer to

Machine ports configuration.

Installation in environments with no Internet connection

By default, the installation process requires an Internet connection to enable installing dependency

packages, and for synchronizing the operating system clock against an NTP server (refer to

Synchronizing cluster server clocks to Network Time Protocol (NTP)).

If you install RLEC on a machine with no Internet connection you need to perform these two tasks

manually, as follows:

First, you need to install the required dependency packages. When RLEC is installed on a machine

that is not connected to the Internet, the installation process fails and displays an error message

informing you it failed to automatically install dependencies. Review the installation steps in the

console to see which missing dependencies the process attempted to install. Install all these

dependency packages and then run the installation again.

At the end of the installation the process asks you whether you would like to setup NTP time

synchronization. If you choose Yes while you are not connected to the Internet then the action fails

and shows the appropriate error message, but the installation still completes successfully. Although

the installation completes successfully, you should still configure all nodes for NTP time

synchronization as described in Synchronizing cluster server clocks to Network Time Protocol

(NTP).

Replica of repeatedly fails

There might be instances in which the Replica of process repeatedly fails and restarts itself. This can

result from the Redis “client-output-buffer-limit” setting on the source database being configured to a

relatively small value, which causes the connection to be dropped.

To identify this issue you can review the Redis log of the source database and find an entry in the log

describing this issue.

In order to resolve this issue you should reconfigure the buffer on the source database to a bigger

value using the following instructions:

If the source database is a Redis that is part of RLEC, you should use the following command

in rladmin Command Line Interface (CLI) to increase the slave buffer size of the source

database: tune db <db:id | name> slave_buffer <value>

If the source database is a Redis that is not part of RLEC, you should use the config set

command through Redis CLI to increase the client output buffer size of the source database:

http://redis.io/commands/config-set


config set client-output-buffer-limit "slave <hard limit> <soft limit> <soft

seconds>"

You can refer to the following blog post to learn more about this issue and configuration: Top Redis

Headaches for Devops – Replication Buffer.

Database metrics are blank during resharding

Because data is being moved around across shards during the resharding process, it is impossible to

report the values of some of the metrics in an accurate manner. Therefore, several database metrics

are not collected, and as a result the graph is blank when you view the database or shard metrics.

Examples of metrics that are not collected during the resharding process are:

Used memory

Memory usage

Total keys

Write misses/sec

Read misses/sec

Expired objects/sec

Evicted objects/sec

Incoming traffic

Outgoing traffic

Creating a support package

If you encounter any issues that you are not able to resolve yourself and need to contact Redis Labs

support for assistance, you can create a support package that gathers all essential information to help

us debug your issues.

To create a support package:

1. Click Support at the top right of the management UI.

2. Click Create Support Package at the bottom of the page, and approve the action.

The package is created and downloaded to your browser.

Note that creating a support package might take several minutes and might overload the system, so

do not attempt to initiate this process unless it is necessary.

https://redislabs.com/blog/top-redis-headaches-for-devops-replication-buffer

https://redislabs.com/blog/top-redis-headaches-for-devops-replication-buffer


Subscription agreement

You can view the subscription agreement on the following link:

https://redislabs.com/company/terms#software

https://redislabs.com/company/terms#software


Release notes

4.2.1-30 – September 21, 2015

Overview

If you are upgrading from a previous version, make sure to review the upgrade instructions before

beginning the upgrade process.

New features

rsyslog logging support – RLEC now writes by default to syslog, which enables monitoring

through rsyslog.

Profile support for tuning cloud and non-cloud environments - enables the administrator to

configure RLEC to run with different performance profiles that are optimized for either cloud

or non-cloud environments. Refer to the Performance optimization section for more details.

SLA for AOF rewrite - enables the administrator to configure database parameters, using the

rladmin tune command, related to when AOF rewrite is triggered based on the time it would

take to load the database from the AOF file, and on the maximum AOF rewrite file size. In

addition, updates to the AOF rewrite mechanism minimize chances of the disk getting full.

New warning and alerts related to AOF rewrite mechanism – a warning is shown during the

setup process in case the disk size is lower than twice the size of the RAM. New cluster level

alerts added to alert when node available disk space is lower than the needed disk space for

AOF rewrite purposes, and when node performance is degraded due to reaching disk I/O

limits.

Replica of support for multiple sources – the Replica of feature is enhanced to support

creating a database that is a replica of multiple source databases. Refer to the Replica of

section for more details

Cross cluster Replica of – the Replica of feature now supports defining a database that is a

replica of databases that belong to a different RLEC cluster. Refer to the Replica of section

for more details.

Multi-IP support – on a node that has multiple IPs, enables the administrator to specify which

IP address is used for internal traffic and which IP addresses are used for external traffic.

Refer to Multi-IP & IPv6 support for more details.

IPv6 support for external traffic – on a node that has multiple IPs, external IP addresses can

be of IPv6 type. Refer to Multi-IP & IPv6 support section or more details.

Support for OpenStack Object Store (“Swift”) location for import / export / backup. Refer to

Database backup and Importing data to a database sections for more details.

Import of a sharded database – support for importing data of a sharded database by

indicating multiple files paths. Refer to the Importing data to a database section for more

details.

https://redislabs.com/redis-enterprise-documentation/installing-and-upgrading/upgrading


Enable running the install script in silent mode using “-y” parameter for default answers (“Y”)

or “-c” for file path parameters for custom answers. Refer to Accessing and installing the

setup package section for more details.

New rladmin command-line-interface “info” command allows for fetching current value of

tunable parameters.

Changes

rladmin command-line-interface can only be run under user root or redislabs. Refer to the

rladmin Command Line Interface (CLI) section.

Import / export / backup to/from Amazon S3 requires supplying the credentials per usage

instance; it does not use central cloud credentials that used to be supplied in the Settings ->

General page.

Fields related to storing Amazon S3 Cloud credentials have been removed from Settings ->

General page.

Performance optimization in the database resharding mechanism.

Persistent and ephemeral storage cluster level alerts are enabled by default and set to 70%.

Various enhancements to rladmin command-line-interface (CLI) to support new commands.

Redis version updated to 2.8.21 that addresses CVE-2015-4335/DSA-3279 – Redis Lua

Sandbox Escape.

Port 3336 added to the list of ports being used by RLEC.

Node “Network utilization” alert measured in percentages (%) has been updated to “Network

throughput” alert measured in MBps. If the alert was defined then it will be disabled when

upgrading to this version and the user needs to reconfigure it with a new value.

Performance improvements to the database Import to make the process much faster.

Enhancements to the support package to include additional details.

Removed support for SSL 2.0/3.0 protocols due to security vulnerabilities.

Disabled “dofile” functionality in Redis to solve a possible security vulnerability.

rladmin CLI “tune watchdog profile" command syntax updated to "tune cluster

watchdog_profile”.

Fixed issues

Support for relative path for backup and import functionalities.

Fix issue with TLS configuration of email server settings.

Email alerts not sent for database export and import processes.

Fix erroneous entries in the Log page.

Sometimes a wrong value is reported in the UI for node used ephemeral storage space.

Sometime the wrong Replica of Lag value is reported in the UI.

RLEC-6875 - email server settings not working when using port 587.

RLEC-5498 - Improve rladmin response time when a node is down.

https://groups.google.com/forum/#!msg/redis-db/4Y6OqK8gEyk/Dg-5cejl-eUJ

https://groups.google.com/forum/#!msg/redis-db/4Y6OqK8gEyk/Dg-5cejl-eUJ


Validation of Replica of source definition did not fail in the UI and would only fail in runtime if it

was using the Cluster Name (FQDN) and the FQDN was not properly configured in the DNS.

Various improvements to error messages reported by rladmin.

Known issues

Issue: In the Replica of process, if the target database does not have replication enabled and

it is restarted or fails for any reason, the data on target database might not be in sync with the

source database, although the status of the Replica of process indicates it is.

Workaround: You need to manually stop and restart the synchronization process in order to

ensure the databases are in sync.

Issue: In the Replica of process, if the source database is resharded while the replica of

process is active, the synchronization process will fail.

Workaround: You need to manually stop and restart the synchronization process after

resharding of the source database is done.

Issue: In the replica of process, if there is very high traffic on the database the replica of

process might be restarted frequently as result of the “slave buffer” being exceeded. In this

case you see the status of the replica of process as “Syncing” frequently.

Workaround: You need to manually reconfigure the “slave buffer” through rladmin and set

the buffer size to a new size. In order to find the appropriate buffer size please contact

support at: [email protected].

Issue: In a cluster that is configured to support rack-zone awareness, if the user forces

migration of a master or slave shard, through rladmin, to a node on the same rack-zone as its

corresponding master or slave shard, and later runs the rebalance process, the rebalance

process will not migrate the shards to ensure rack-zone awareness compliance.

Workaround: In the scenario described above, you need to manually migrate the shard,

through rladmin, to a node on a valid rack-zone in order to ensure rack-zone awareness

compliance.

4.0.0-30 – May 20, 2015

Overview


running through the upgrade process.

In addition, when running the install.sh script to upgrade the software, you might be prompted to

approve changes to a configuration file named ccs-redis.conf. It is crucial that you choose Yes

when asked whether to update this file.

New features

Support for Red Hat Enterprise Linux (RHEL) and CentOS 6.5 and 7 operating systems.

Support for additional AWS AMIs for RHEL, CantOS, Ubuntu and Amazon Linux, on multiple

AWS regions.




Support for additional browsers and operating systems for the management UI.

Replica of feature, which enables creating a Redis database that keeps synchronizing data

from another Redis database.

Rack-zone awareness feature which enables mapping nodes to racks/zones to ensure a

more sophisticated high-availability mechanism.

Database related alerts and email alerts.

Auto-configuration of synchronization of cluster server clocks with NTP as part of installation

script.

Database Export functionality.

Email alerts on database Export and Import.

Changes

Database Backup Now functionality replaced with Export functionality.

Database performance improvements to increase throughput and reduce latency.

Improvement to AOF rewrite mechanism to deal with extreme-write scenarios and limited disk

space configurations.

Enhancements to rladmin CLI to support additional commands.

Fixed issues

Cluster stability improvements related to removing nodes and taking nodes offline.

rladmin CLI bug fixes.

Known issues

Issue: RLEC-6819 - Uninstall on Red Hat Enterprise Linux does not stop all services and if

you try to install the software again on the same machine the new installation might use prior

installation data.

Workaround: Before installing the software again restart the machine or verify that all

services are down.

Issue: In the replica of process, if the source database is resharded while the replica of

process is active, the synchronization process will fail.

Workaround: You need to manually stop and restart the synchronization process after

resharding of the source database is done.

Issue: In the replica of process, if there is very high traffic on the database the replica of

process might be restarted frequently as result of the “slave buffer” being exceeded. In this

case you see the status of the replica of process as “Syncing” frequently.

Workaround: You need to manually reconfigure the “slave buffer” through rladmin and set

the buffer size to a new size. In order to find the appropriate buffer size please contact

support at: [email protected].



Issue: In a cluster that is configured to support rack-zone awareness, if the user forces

migration of a master or slave shard, through rladmin, to a node on the same rack-zone as its

corresponding master or slave shard, and later runs the rebalance process, the rebalance

process will not migrate the shards to ensure rack-zone awareness compliance.

Workaround: In the scenario described above, you need to manually migrate the shard,

through rladmin, to a node on a valid rack-zone in order to ensure rack-zone awareness

compliance.

Issue: In case you deploy a cluster and use the DNS option for the cluster name (see details

in How to set the Cluster Name (FQDN)), do not configure the DNS entries for the cluster

nodes, and try to configure a database that is a replica of another database within the cluster,

then the UI allows you to configure the source database but the replica of process fails in

runtime.

Workaround: The configuration indicated in this issue is not a valid cluster configuration. If

you choose to use the DNS option for the cluster name then you must configure DNS entries

for the nodes, otherwise the cluster does not operate correctly. You should either update the

DNS accordingly, or recreate the cluster and use the mDNS option for the cluster name as

described in How to set the Cluster Name (FQDN).

Issue: When taking a node offline or removing a node, if the node being taken offline or

removed is currently serving as the web server for the web browser being used to view the

management UI, then the management UI appears down while the node is down.

Workaround: If you are using the cluster name in order to connect to the management UI in

the browser, and the cluster name is registered in your external DNS or you are using the

mDNS option, then the DNS entries will be updated to point to another node in the cluster

after a few seconds and the UI will open properly. If you are not using the cluster name but

rather the node IP in order to connect to the management UI in the web browser, then you

should simply use the IP of another node in the cluster to access the management UI.

0.99.5-24 - February 15, 2015

Overview


running through the upgrade process.

If you are upgrading from version 0.99.5-11:

1. You must restart the services after the upgrade using the following command with user root (sudo

su). From the operating system’s CLI, run the following command:

cnm_ctl restart

2. After the upgrade, rladmin status command will report that the databases are from an old version.

It is recommended that you upgrade the databases as soon as possible, as described in the

upgrade instructions.

https://redislabs.com/redis-enterprise-documentation/initial-setup-creating-a-new-cluster/how-to-set-the-cluster-name-fqdn

https://redislabs.com/redis-enterprise-documentation/initial-setup-creating-a-new-cluster/how-to-set-the-cluster-name-fqdn




New features

None.

Changes

Enhancements to memtier_benchmark tool that is included in the installation package. You

can find more details in the GitHub project.

Fixed issues

Improvements and fixes related to node failover, remove node and take node offline

functionality.

Known issues










0.99.5-11 - January 5, 2015

New features

Initial release, everything is new!

Changes

Initial release, no changes!

Fixed issues

None.

Known issues








https://github.com/RedisLabs/memtier_benchmark

redis labs enterprise cluster documentationlabs... · redis labs enterprise cluster documentation...

Documents