0.89 -Online Configuration

brandboat

1

❏ Add a utility to reload configurations in Region Server and HMaster

➢ Create a utility that can talk to the Region Servers and make them reload configurations on

disk.This however, does not make other classes reload, which have their own copies of conf

property values.

❏ Allow individual classes to get notified when Configuration is reloaded

➢ The purpose of this feature is to let individual classes who are interested in observing online

configuration changes, to be notified when the Conf is reloaded from disk.

Online configuration upgrade feature

2

Add a utility to reload configurations in

Region Server [HBASE-8544]

❏ How to run (see the usage):

❏ Flow :

1. get HRegionServer information2.conf.reloadConfiguration()

3.getConfigurationManager.notifyAllObserver

s(conf)

${HBASE_HOME}/bin/hbase org.jruby.Main ${HBASE_HOME}/bin/reload_rs_config.rb usage

3

Conti. (Flow)

4

HBaseAdmin HRegionServer Configuration

updateConfiguration(address)updateConfiguration()

// Update configuration for region server at this address.

public void updateConfiguration(HServerAddress address)

throws IOException {

HRegionInterface server =

connection.getHRegionConnection(address);

server.updateConfiguration();

}

// Reload the configuration from disk.

@Override

public void updateConfiguration() {

LOG.info("Reloading the configuration from disk.");

// Reload the configuration from disk.

conf.reloadConfiguration();

// Notify all the observers that the configuration has

// changed.

configurationManager.notifyAllObservers(conf);

}

conf.reloadConfiguration()

/**

* Reload configuration from previously added resources.

*

* This method will clear all the configuration read from the added

* resources, and final parameters. This will make the resources to

* be read again before accessing the values. Values that are added

* via set methods will overlay values read from the resources.

*/

public synchronized void reloadConfiguration() {

properties = null; // trigger reload

finalParameters.clear(); // clear site-limits

}

5

Conti. properties = null ?

private synchronized Properties getProps() {

if (properties == null) {

properties = new Properties();

loadResources(properties, resources, quietmode);

if (overlay!= null)

properties.putAll(overlay);

}

return properties;

}

6

public String get(String name)

public String getRaw(String name)

public void set(String name, String value)

…… has getProps()

/**

* Return the number of keys in the configuration.

*

* @return number of keys in the configuration.

*/

public int size() {

return getProps().size();

}

Added a utility to make the HMaster reload

its configuration from disk [HBASE-8587]

❏ How to run :

❏ Flow :

1. get HMaster information2.conf.reloadConfiguration()

3.getConfigurationManager.notifyAllObserver

s(conf)

${HBASE_HOME}/bin/hbase org.jruby.Main ${HBASE_HOME}/bin/reload_master_config.rb

7

Allow individual classes to get notified when

Configuration is reloaded [HBASE-8576]

<ConfigurationManager.java> <ConfigurationObserver.java>

❖ If a class has configuration properties which you would like to be able to change on-

the-fly, do the following :

1. Implement the ConfigurationObserver interface.

2. Register the appropriate instance of the class with the ConfigurationManager instance,

using the ConfigurationManager#registerObserver(ConfigurationObserver)

method.

3. Deregister the instance using the ConfigurationManager#deregisterObserver(ConfigurationObserver) method

when it is going out of scope.

8

ConfigurationManager.java// The set of Configuration Observers. These classes would like to get

// notified when the configuration is reloaded from disk

private Set<ConfigurationObserver> configurationObservers =

Collections.newSetFromMap(new WeakHashMap<ConfigurationObserver,Boolean>());

// Register an observer class

public void registerObserver(ConfigurationObserver observer)

// Deregister an observer class

public void deregisterObserver(ConfigurationObserver observer)

// The conf object has been repopulated from disk, and we have to notify

// all the observers that are expressed interest to do that.

public void notifyAllObservers(Configuration conf)

// Return the number of observers.

public int getNumObservers()

9

ConfigurationObserver.java/**

* Every class that wants to observe changes in Configuration properties,

* must implement interface (and also, register itself with the

* <code>ConfigurationManager</code> object.

*/

public interface ConfigurationObserver {

/**

* This method would be called by the <code>ConfigurationManager</code>

* object when the <code>Configuration</code> object is reloaded from disk.

*/

void notifyOnChange(Configuration conf);

}

10

Conti. notifyAllObservers()

// ConfigurationManager.java

public void notifyAllObservers(Configuration conf) {

synchronized (configurationObservers) {

for (ConfigurationObserver observer : configurationObservers) {

try {

observer.notifyOnChange(conf);

} catch (NullPointerException e) {

............

11

Ex. CacheConfig// HRegionServer.java

public static final ConfigurationManager configurationManager =

new ConfigurationManager();

configurationManager.registerObserver(cacheConfig);

-------------------------------------------------------------------------------------

-------------------------------------------------------------------------------------

// CacheConfig.java

public class CacheConfig implements ConfigurationObserver { . . . . . . .

@Override

public void notifyOnChange(Configuration conf) {

new CacheConfigBuilder(conf).update(this);

}

12

configurationManager.notifyAllObservers(conf);

OthersOnline Configuration

13

Number of compaction threads &

CompactionConfiguration class [HBASE-8805]

Compaction Ratio

OffPeak Compaction Ratio

Throttle Point

OffPeak Start Hour

OffPeak End Hour

Major Compaction Period

Major Compaction Jitter

Min Files to Compact

Max Files to Compact

Min Compact Size

Max Compact Size

Should Exclude Bulk

Should Delete Expired

❏ Making the number of large/small compaction threads and the

properties in the CompactionConfiguration class online-

configurable.

❏ Properties :

14

Make L2 cache online configurable

❏ Allows the L2 cache to be disabled or L2 cache policies

to be modified without restarting the RS. (This is a

preliminary diff ,not fully done testing.)

❏ Properties (CacheConfig.java)

L2_CACHE_BLOCKS_ON_FLUSH_KEY

L2_EVICT_ON_PROMOTION_KEY (Not yet implemented!)

L2_EVICT_ON_CLOSE_KEY

L2_BUCKET_CACHE_SIZE_KEY

15

Update memstore flush threads in online

fashion

❏ The number of flush thread can be updated in

an online fashion.

❏ hbase.regionserver.flusher.count

16

Online configuration change for loaded

coprocessors

❏ Trying out if online config works when new

coprocessors are added, so far so good.

17

Allow number of SplitLogWorker threads to

be online-configurable

❏ the number of worker threads to use < the number of

workers in use, extraneous workers are stopped.

❏ the number of worker threads to use > the number of

workers in use, the appropriate number of additional

SplitLogWorker threads are added.

❏ hbase.hregionserver.hlog.split.workers.num

18

Make server side profiling online

configurable

❏ Make the parameter to enable/disable server side

profiling configurable online.

19

Hbase 1.0 Online Config Change[HBASE-12147]

❏ HBASE-8805 HBASE-8544 in 89-fb.

❏ Improves operational efficiency in managing clusters

that are serving production traffic.

❏ The idea is to have a central configuration which can

manage notifying the configuration observers.

❏ The observers in turn should update their local state

from the latest config. Minor caveats where

configuration variables are corelated should be taken

care of with additional care. 20