How to automate downloading, unzipping & loading of IP2Location DB24 data into Linux MySQL

The example guides you on how to automate the updating of the IP2Location DB24 database. We will show how to download the IP2Location DB24 csv data from the web server and then unzip it before loading the data into MySQL server on a Linux platform. The whole shell script code is available at http://ip2location.com/tutorials/automate-downloading-unzipping-loading-db24-into-linux-mysql.

Step 1: Configure login details

Firstly, we need to setup some login details for the IP2Location web server as well as the MySQL server information. Under the Configuration section of the script, you will need to key in all the relevant login information in the fields below: # Configuration LOGIN="IP2LOCATION_WEBSITE_LOGIN" PASS="IP2LOCATION_WEBSITE_PASSWORD" CODE="DB24CSV" DBHOST="YOUR_DATABASE_HOST" DBUSER="YOUR_DATABASE_USERNAME" DBPASS="YOUR_DATABASE_PASSWORD" DBNAME="YOUR_DATABASE_NAME" For most cases, you shouldnt have to modify anything else in the script but we will still explain what is going on in case you want to modify the code.

Step 2: Checking for pre-requisites

Some Linux packages like wget, unzip, mysql, wc, find and grep are needed for this script to run. If any of them are not installed, an error message will be shown and the script will terminate itself. for a in wget unzip mysql wc find grep; do if [ -z "$(which $a)" ]; then error "Command \"$a\" not found." exit 0 fi done

Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved.

Step 3: Creating the temporary folder for storing the downloaded zipped data file
If the temporary folder does not exist, it will be created. An error message will be shown if the folder cannot be created and the script will terminate itself. if [ ! -d /tmp/ip2location ]; then echo -n "Create temporary directory........... " mkdir /tmp/ip2location if [ ! -d /tmp/ip2location ]; then error "Failed to create /tmp/ip2location" exit 0 fi success "[OK]" fi

Step 4: Downloading the zipped data file from the web server
If download is not successful then an error message will be shown and the script will terminate itself. wget -O database.zip -q http://www.ip2location.com/download?login=$LOGIN\&password=$PASS\&prod uctcode=$CODE 2>&1 if [ ! -f database.zip ]; then error "Download failed." exit 0 fi if [ ! -z "$(grep 'NO PERMISSION' database.zip)" ]; then error "Permission denied." exit 0 fi if [ ! -z "$(grep '5 times' database.zip)" ]; then error "Download quota exceed." exit 0 fi if [ $(wc -c < database.zip) -lt 102400 ]; then error "Download failed." exit 0 fi

Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved.

Step 5: Decompressing the zipped file

If decompression is not successful then an error message will be shown and the script will terminate itself. unzip -q -o database.zip if [ -z $(find . -name 'IP-COUNTRY*.CSV') ]; then echo "ERROR:" exit 0 fi NAME="$(find . -name 'IP-COUNTRY*.CSV')"

Step 6: Creating a temporary table in MySQL to load the data

Drops the temporary table if it already exists and then creates the table. If an error is encountered, an error message will be shown and the script will terminate itself. RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'DROP TABLE IF EXISTS `ip2location_database_tmp`;' 2>&1)" if [ ! -z "$(echo $RESULT | grep 'connect')" ]; then error "Failed to connect MySQL host." exit 0 fi if [ ! -z "$(echo $RESULT | grep 'Access denied')" ]; then error "MySQL authentication failed." exit 0 fi RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'CREATE TABLE `ip2location_database_tmp` (`ip_from` INT(10) UNSIGNED ZEROFILL NOT NULL,`ip_to` INT(10) UNSIGNED ZEROFILL NOT NULL,`country_code` CHAR(2) NOT NULL,`country_name` VARCHAR(64) NOT NULL,`region_name` VARCHAR(128) NOT NULL,`city_name` VARCHAR(128) NOT NULL,`latitude` DOUBLE NULL DEFAULT NULL,`longitude` DOUBLE NULL DEFAULT NULL,`zip_code` VARCHAR(12) NULL DEFAULT NULL,`time_zone` VARCHAR(8) NULL DEFAULT NULL,`isp` VARCHAR(255) NOT NULL,`domain` VARCHAR(128) NOT NULL,`net_speed` VARCHAR(8) NOT NULL,`idd_code` VARCHAR(5) NOT NULL,`area_code` VARCHAR(30) NOT NULL,`weather_station_code` VARCHAR(10) NOT NULL,`weather_station_name` VARCHAR(128) NOT NULL,`mcc` VARCHAR(128) NULL DEFAULT NULL,`mnc` VARCHAR(128) NULL DEFAULT NULL,`mobile_brand` VARCHAR(128) NULL DEFAULT NULL,`elevation` INT(10) NOT NULL,`usage_type` VARCHAR(11) NOT NULL,INDEX `idx_ip_from` (`ip_from`),INDEX `idx_ip_to` (`ip_to`),INDEX `idx_isp` (`isp`)) ENGINE=MyISAM;' 2>&1)" if [ ! -z "$(echo $RESULT)" ]; then error "Unable to create temporary table."
Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved.

exit 0 fi

Step 7: Loading the CSV data into the MySQL temporary table
Loads the CSV data into the temporary table. If an error is encountered, an error message will be shown and the script will terminate itself. RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'LOAD DATA LOCAL INFILE '\'''$NAME''\'' INTO TABLE `ip2location_database_tmp` FIELDS TERMINATED BY '\'','\'' ENCLOSED BY '\''\"'\'' LINES TERMINATED BY '\''\r\n'\'';' 2>&1)" if [ ! -z "$(echo $RESULT)" ]; then error "Failed." exit 0 fi

Step 8: Dropping the existing data table

Drops the existing data table. If an error is encountered, an error message will be shown and the script will terminate itself. RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'DROP TABLE IF EXISTS `ip2location_database`;' 2>&1)" if [ ! -z "$(echo $RESULT)" ]; then error "Failed to drop \"ip2location_database\" table." exit 0 fi

Step 9: Rename the temporary table to become the live data table
Renames the temporary table to become the live data table. If an error is encountered, an error message will be shown and the script will terminate itself. RESULT="$(mysql -h $DBHOST -u $DBUSER -p$DBPASS $DBNAME -e 'RENAME TABLE `ip2location_database_tmp` TO `ip2location_database`;' 2>&1)" if [ ! -z "$(echo $RESULT)" ]; then error "Failed to rename table." exit 0 fi

Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved.

Step 10: Remove temporary download folder and the downloaded data file
Remove the temporary download folder and all files in that folder. rm -rf /tmp/ip2location

Copyright 2001-2014 Hexasoft Development Sdn. Bhd. All rights reserved.