-
Local setup for HBase, Phoenix and Squirrel
Setting up HBaseExecuting HBase ShellDifferent modes of HBase
Standalone ModeDistributed Mode
Execution StepsSample QueriesExecution Steps Sample Queries
Setting up SquirrelInstallation StepsTroubleshooting
References
This page aims to serve as a brief guide for doing a local setup for HBase (in standalone mode), Phoenix - for querying over HBase and Squirrel client as additional GUI over Phoenix for SQL querying over HBase
Setting up HBase
HBase can be quickly downloaded and installed using brew formula - Run the command on MAC terminal (and this should complete the installation in /usr/local directory)
brew install hbase
After brew installation is complete - validate the installation by running command hbase version to get output as below
Executing HBase Shell
Once installed, hbase can be executed in shell mode after some configurations. HBase can be run in Stand alone mode (without hadoop on local file system) or in distributed mode (on HDFS). In stand alone mode, HBase also manages its own zookeeper instance.
For running in standalone mode - navigate/open the hbase-site.xml file and make changes as below (hbase-site xml file is located at directory - )/usr/local/Cellar/hbase/1.2.6.1/libexec/conf/
Remove any existing properties from the section of hbase-site.xml and add the below property file
hbase.rootdir file:///Users//hbase
The above property setting essentially creates a directory on local file system where hbase stores all its tables and metadata information
this setup is primarily targeted for MacOS
-
1.
2. 3. 4. 5.
a.
b.
Also ensure - for the standalone mode HBASE_MANAGES_ZK is set - "export HBASE_MANAGES_ZK=true" in file /libexec/conf/hbase-env.sh
Different modes of HBase
Standalone Mode
This is for local setup and development, executing queries and getting results without much configuration setup
Distributed Mode
Tables are created in HDFS - this requires Hadoop Installation, and can be configured over clustered setup with multiple region servers and multiple/single zookeeper instances - For production purposes, this is the route/approach that should be adopted
Execution Steps
Once the hbase-site.xml is updated appropriately, perform below steps to run hbase in shell mode
Navigate to hbase installation and move to bin folder - here all executable shell files are available for starting/stopping hbase etc - Path - /usr/local/Cellar/hbase/1.2.6.1//binExecute start-hbase.sh script from the above directoryOnce the script completes - run jps to ensure HMaster java process is started and runningExecute command in the terminalhbase shellIf the installation was proper the the hbase-site.xml configuration was properly loaded, then the shell should load without any errors
Validate shell starts up and looks as below:
Execute a test command to ensure shell was properly started - enter the command "list" to list all the create tables / system tables etc as below:
Sample Queries
-
1. a. b.
2.
3.
4. 5. 6. 7.
8.
To create a new table "testTable" with column family "cf1" - execute command - "create 'testtable','cf1'" on shell and validate the output as below.
The newly created table should configuration can be obtained using - "describe testtable" command
Setting up Phoenix
Once HBase standalone mode is up and running, its now time to setup Phoenix
Phoenix is nothing but a SQL wrapper over HBase and allows you to read and write to HBase tables through simple SQL like queries.
The SQL queries are underneath converted into HBase scans by Phoenix which results in retrieval or persistence of data from/to HBase tables.
Download Phoenix from apache download mirror - http://www.apache.org/dyn/closer.lua/phoenix/apache-phoenix-4.14.0-HBase-1.4/bin/apache-phoenix-4.14.0-HBase-1.4-bin.tar.gz
Execution Steps
Once phoenix is downloaded, unzip the tar/zip file to a directory at some location.
Copy / Update the required JARS from phoenix installation to HBase lib Navigate to Phoenix install folder and copy "phoenix-4.14.0-HBase-1.2-server.jar" Paste this JAR to HBase lib directory - /usr/local/Cellar/hbase/1.2.6.1/libexec/lib
Add an environment variable HBASE_CONF_DIR - this should point to path of conf folder in hbase installation - like /usr/local/Cellar/hbase/1.2.6.1/libexec/conf/ (this is used by sqlline python file for running phoenix command line)
Also update the hbase-site.xml by adding 2 more property for mapping hbase schema to phoenix
phoenix.schema.isNamespaceMappingEnabled true phoenix.schema.mapSystemTablesToNamespace true
Stop and Start HBase once againNavigate to Phoenix install directory and move to bin folder - here all python scripts to start SQL query terminal are availableExecute - ./sqlline.py localhostOnce the command completes - Phoenix terminal should be up and running (if there are any errors - go through the above steps and check if mentioned steps were followed and changes were made)The phoenix terminal should look like this
It should be kept in mind that while downloading Phoenix, the supported version of HBase should be kept in mind - So first retrieve HBase version and depending on that download Phoenix version - For HBase 1.2 - Phoenix 4.14 is the supported version
http://www.apache.org/dyn/closer.lua/phoenix/apache-phoenix-4.14.0-HBase-1.4/bin/apache-phoenix-4.14.0-HBase-1.4-bin.tar.gzhttp://www.apache.org/dyn/closer.lua/phoenix/apache-phoenix-4.14.0-HBase-1.4/bin/apache-phoenix-4.14.0-HBase-1.4-bin.tar.gz
-
8.
9. Execute a test query on phoenix to ensure the setup was successful - run command " " to list all the tables available - (this !tableslists down all system tables as well as all tables that were created using phoenix)
Sample Queries
Creating a simple table in Phoenix is through normal SQL Queries -
ex - to create a student table with fields = id (Integer Primary key), birthDate (Date), name ( varchar), age(Integer) is as below
create table student(id INTEGER NOT NULL PRIMARY KEY, birthdate DATE, name varchar(50), age INTEGER);
Execute above statement and listing down tables should now list the newly created student table as well
Setting up Squirrel
After the installation of HBase and Phoenix - Squirrel which is a GUI client for writing SQL queries can be setup for local execution of queries and development/testing of scripts.
Its a generic SQL client which supports various Databases that have JDBC drivers like - MySQL Postgres etc.
-
1. 2. 3.
a. b.
c. 4. 5.
6.
Squirrel itself is a Java application.
Installation Steps
Download Squirrel client from - http://squirrel-sql.sourceforge.net/Follow the Squirrel Install guide - http://squirrel-sql.sourceforge.net/paper/GettingStartedusingtheSQuirreLSQLClient.pdfCopy the Phoenix client jar from phoenix install location to Squirrel Resources directory
Copy " " from Phoenix install directoryphoenix-4.14.0-HBase-1.2-client.jarPaste this to Squirrel resources directory - Navigate to squirrel icon in Applications folder - Right click on Squirrel Icon and click Show Package Contents
Navigate to /Contents/Resources/java/lib and paste the copied JAROpen Squirrel SQL Client and setup the Phoenix driver and alias Open Phoenix alias (created in step 4 above) and click "Connect" in order to connect to existing instance of HBase
Once connected - it should display all the tables from HBase
http://squirrel-sql.sourceforge.net/http://squirrel-sql.sourceforge.net/paper/GettingStartedusingtheSQuirreLSQLClient.pdf
-
6.
1.
a. b. c.
2.
a.
b. c.
1. 2. 3.
Troubleshooting
On MacOS - a lot of times - Squirrel does not work out of the box - i.e. the launcher does not work or show up the application as its unable to resolve the installation path of Squirrel - To fix the issue
Navigate to Squirrel icon in Applications folder - Right click on Squirrel Icon and click Show Package ContentsOpen - /MacOS/squirrel-sql.shSearch for SQUIRREL_SQL_HOME and update the value to "Applications/SQuirreLSQL.app/Contents/Resources/Java"
While configuring driver for Phoenix, a lot of times its unable to find out the apt hbase-site xml configuration due to which it reports errors for schema namespace mapping properties - To fix the issue - update the HBase client jar to include the hbase-site.xml file in the jar and then refer this jar while creating phoenix driver in Squirrel
If while connecting to HBase instance from Squirrel - it complains about namespace mapping schema properties, then chances are there that the desired hbase-site.xml properties are not being loaded / correctly obtained by Squirrel client - hence in order to do that - package the HBase client jar with hbase-site xmlExecute the following command (After copying the required hbase-site xml to the Phoenix HBase client jar directory) - jar uf phoenix-4.7.0.2.5.3.0-37-client.jar hbase-site.xml Copy the new jar and paste it to the Squirrel Java lib directory and restart Squirrel Now reconnect with the already set up alias -It should now successfully connect and show all the HBase tables
References
Phoenix Grammar - https://phoenix.apache.org/language/index.htmlSquirrel Install - http://squirrel-sql.sourceforge.net/Apache HBase documentation - https://hbase.apache.org/book.html
Before executing / connecting to Squirrel SQL ensure that HBase is up and running
https://phoenix.apache.org/language/index.htmlhttp://squirrel-sql.sourceforge.net/https://hbase.apache.org/book.html
Local setup for HBase, Phoenix and SquirrelSetting up HBaseExecuting HBase ShellDifferent modes of HBaseStandalone ModeDistributed Mode
Execution StepsSample QueriesExecution Steps Sample Queries
Setting up SquirrelInstallation StepsTroubleshooting
References