how to automate downloading, unzipping and loading of ip2locaion geolocation data into linux mysql
TRANSCRIPT
http://www.ip2location.com
IntroductionWe will show how to download the
IP2Location Geolocation csv data from the web server and then unzip it before loading the data into a MySQL server on a Linux platform
We will be using DB24 database in this guide.
http://www.ip2location.com
Configure login detailsFirstly, setup some login details for the
IP2Location web server and MySQL server information as below.
# Configuration
LOGIN="IP2LOCATION_WEBSITE_LOGIN"
PASS="IP2LOCATION_WEBSITE_PASSWORD"
CODE="DB24CSV"
DBHOST="YOUR_DATABASE_HOST"
DBUSER="YOUR_DATABASE_USERNAME"
DBPASS="YOUR_DATABASE_PASSWORD"
DBNAME="YOUR_DATABASE_NAME"
http://www.ip2location.com
Checking for pre-requisitesLinux packages
like wget, unzip, mysql, wc, find and grepare needed for this script to run.
for a in wget unzip mysql wc find grep; do
if [ -z "$(which $a)" ]; then
error "Command \"$a\" not found."
exit 0
fi
done
http://www.ip2location.com
Creating the temporary folder
If the temporary folder does not exist, it will be created.
if [ ! -d /tmp/ip2location ]; then
echo -n "Create temporary directory.......................... "
mkdir /tmp/ip2location
if [ ! -d /tmp/ip2location ]; then
error "Failed to create /tmp/ip2location"
exit 0
fi
success "[OK]"
fi
http://www.ip2location.com
Downloading the zipped data file from the web server
wget -O database.zip -q http://www.ip2location.com/download?login=$LOGIN\&password=$PASS\&productcode=$CODE 2>&1if [ ! -f database.zip ]; then
error "Download failed."exit 0
fiif [ ! -z "$(grep 'NO PERMISSION' database.zip)" ]; then
error "Permission denied."exit 0
fiif [ ! -z "$(grep '5 times' database.zip)" ]; then
error "Download quota exceed."exit 0
fiif [ $(wc -c < database.zip) -lt 102400 ]; then
error "Download failed."exit 0
fi
http://www.ip2location.com
Decompressing the zipped file
If decompression is not successful then an error message will be shown and the script will terminate itself.
unzip -q -o database.zip
if [ -z $(find . -name 'IP-COUNTRY*.CSV') ]; then
echo "ERROR:"
exit 0
fi
NAME="$(find . -name 'IP-COUNTRY*.CSV')"
http://www.ip2location.com
Creating a temporary table in MySQL
Drops the temporary table if it already exists and then creates the table.
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'DROP TABLE IF EXISTS `ip2location_database_tmp`;' 2>&1)"
if [ ! -z "$(echo $RESULT | grep 'connect')" ]; then
error "Failed to connect MySQL host."
exit 0
fi
if [ ! -z "$(echo $RESULT | grep 'Access denied')" ]; then
error "MySQL authentication failed."
exit 0
fi
http://www.ip2location.com
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'CREATE TABLE `ip2location_database_tmp` (`ip_from` INT(10) UNSIGNED ZEROFILL NOT NULL,`ip_to` INT(10) UNSIGNED ZEROFILL NOT NULL,`country_code` CHAR(2) NOT NULL,`country_name` VARCHAR(64) NOT NULL,`region_name` VARCHAR(128) NOT NULL,`city_name` VARCHAR(128) NOT NULL,`latitude` DOUBLE NULL DEFAULT NULL,`longitude` DOUBLE NULL DEFAULT NULL,`zip_code` VARCHAR(12) NULL DEFAULT NULL,`time_zone` VARCHAR(8) NULL DEFAULT NULL,`isp` VARCHAR(255) NOT NULL,`domain` VARCHAR(128) NOT NULL,`net_speed` VARCHAR(8) NOT NULL,`idd_code` VARCHAR(5) NOT NULL,`area_code` VARCHAR(30) NOT NULL,`weather_station_code` VARCHAR(10) NOT NULL,`weather_station_name` VARCHAR(128) NOT NULL,`mcc` VARCHAR(128) NULL DEFAULT NULL,`mnc` VARCHAR(128) NULL DEFAULT NULL,`mobile_brand` VARCHAR(128) NULL DEFAULT NULL,`elevation` INT(10) NOT NULL,`usage_type` VARCHAR(11) NOT NULL,INDEX `idx_ip_from` (`ip_from`),INDEX `idx_ip_to` (`ip_to`),INDEX `idx_isp` (`isp`)) ENGINE=MyISAM;' 2>&1)“
if [ ! -z "$(echo $RESULT)" ]; then
error "Unable to create temporary table."
exit 0
fi
http://www.ip2location.com
Loading the CSV data into the MySQL temporary table
Load the CSV data into the temporary table.
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'LOAD DATA LOCAL INFILE '\'''$NAME''\'' INTO TABLE `ip2location_database_tmp` FIELDS TERMINATED BY '\'','\'' ENCLOSED BY '\''\"'\'' LINES TERMINATED BY '\''\r\n'\'';' 2>&1)“
if [ ! -z "$(echo $RESULT)" ]; then
error "Failed."
exit 0
fi
http://www.ip2location.com
Dropping the existing data table
Drop the existing data table.
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'DROP TABLE IF EXISTS `ip2location_database`;' 2>&1)“
if [ ! -z "$(echo $RESULT)" ]; then
error "Failed to drop \"ip2location_database\" table."
exit 0
fi
http://www.ip2location.com
Rename the temporary table to become the live data tableRename the temporary table to become the
live data table.
RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'RENAME TABLE `ip2location_database_tmp` TO `ip2location_database`;' 2>&1)"
if [ ! -z "$(echo $RESULT)" ]; then
error "Failed to rename table."
exit 0
fi
http://www.ip2location.com
Remove temporary download folder and the downloaded data file
Remove the temporary download folder and all files in that folder.
rm -rf /tmp/ip2location
http://www.ip2location.com
Refer to http://www.ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysql for further information
Thank you!
http://www.ip2location.com