Theme I want to… Sqoop Example

General See what version of Sqoop is Sqoop version

installed.
General List the databases I have sqoop list-databases --connect "jdbc:sqlserver://
permission to see. [IPAddress]:
[PortNumber];username=[UserName];password=[Pass
word]"
General List the tables I have sqoop list-tables --connect "jdbc:sqlserver://
permission to see for a specific [IPAddress]:[PortNumber];database]
database. [Database];username=[UserName];password=[Passwor
d]"
Import Import a specific table into sqoop import --connect "jdbc:sqlserver://[IPAddress]:
HDFS. [PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName]
Import Import a specified table from a sqoop import --connect "jdbc:sqlserver://[IPAddress]\\
named instance into HDFS. [InstanceName];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName]
Import Import a specific table from a sqoop import --connect "jdbc:sqlserver://[IPAddress]:
specified schema into HDFS. [PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] -- --schema [SchemaName]
Import Import all table from a sqoop import-all-tables --connect "jdbc:sqlserver://
database. [IPAddress]:[PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]"
Import Import all table from a databasesqoop import-all-tables --connect "jdbc:sqlserver://
with specified exceptions. [IPAddress]:[PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –exclude-tables [TableName1],[TableName2]
Import Import a specific table into a sqoop import --connect "jdbc:sqlserver://[IPAddress]:
defined HDFS location. [PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] –target-dir [Location]
Import Import specified columns from sqoop import --connect "jdbc:sqlserver://[IPAddress]:
a table into HDFS. [PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] –columns “[Column1],
[Column2], [Column3]”
Import Import into HDFS using a T- sqoop import \
SQL query. --query 'SELECT a.*, b.* FROM a JOIN b on (a.id =
b.id) WHERE $CONDITIONS' \
--split-by a.id
Import Spread the load of an import sqoop import --connect "jdbc:sqlserver://[IPAddress]:
across the cluster nodes [PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] –m [NumberOfNodes]
Import Import a specific table into a sqoop import --connect "jdbc:sqlserver://[IPAddress]:
parquet format file in HDFS. [PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] –as-parquetfile
General Define an Options File for code #
reuse # Options File for Sqoop Import
#

# Command
import

# Connection String
jdbc:sqlserver://[IPAddress]:[PortNumber]
--database [DatabaseName]

# Credentials
--username [UserName]
--password [Password]

# End

Import Use an Options File to import sqoop --options-file [LoactionOfFile]/[NameOfFile]

data into HDFS --table Customers

Import Import a table into Hive sqoop import --connect "jdbc:sqlserver://[IPAddress]:

Import Overwrite an existing Hive
table during an Import
Import Import a table into Hive and
handle NULLs
Import Import a table into Hive and
remove Hive delimiters from
the data
[PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] –hive-import
sqoop import --connect "jdbc:sqlserver://[IPAddress]:
[PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] –hive-import –hive-overwrite
sqoop import --connect "jdbc:sqlserver://[IPAddress]:
[PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] –hive-import –null-string ‘\\N’
–null-non-string ‘\\N’
sqoop import --connect "jdbc:sqlserver://[IPAddress]:
[PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] –hive-import –hive-drop-
import-delims
Import Import a table into a Hive sqoop import --connect "jdbc:sqlserver://[IPAddress]:
partition [PortNumber];database]
[Database];username=[UserName];password=[Passwor
d]" –table [TableName] –hive-import –hive-partition-
key [PartitionColumn] –hive-partition-value
“[ParitionValue]”
Export Export data from HDFS to sqoop export --connect ‘jdbc:sqlserver://
SQL Server [IPAddress];database=[DatabaseName];instancename=[
InstanceName];username=[UserName];password=[Pass
word]’ --table [SQLServerTableName] --export-dir
[HDFSTableLocation]

Export Export specific columns from sqoop export --connect ‘jdbc:sqlserver://

HDFS to SQL Server [IPAddress];database=[DatabaseName];instancename=[
InstanceName];username=[UserName];password=[Pass
word]’ --table [SQLServerTableName] –columns
‘[Column1],[Column2],[Column3]’--export-dir
[HDFSTableLocation]

Export Export data from HDFS to sqoop export --connect ‘jdbc:sqlserver://

SQL Server and handle NULLs [IPAddress];database=[DatabaseName];instancename=[
InstanceName];username=[UserName];password=[Pass
word]’ --table [SQLServerTableName] --export-dir
[HDFSTableLocation] –input-null-string –input-non-
null-string

Export Update SQL Server data from sqoop export --connect ‘jdbc:sqlserver://
HDFS using an Export [IPAddress];database=[DatabaseName];instancename=[
command InstanceName];username=[UserName];password=[Pass
word]’ --table [SQLServerTableName] --export-dir
[HDFSTableLocation] –update-key ‘[KeyColumn]’

Export Update SQL Server data and sqoop export --connect ‘jdbc:sqlserver://
Insert new rows from HDFS [IPAddress];database=[DatabaseName];instancename=[
using an Export command InstanceName];username=[UserName];password=[Pass
word]’ --table [SQLServerTableName] --export-dir
[HDFSTableLocation] –update-key ‘[KeyColumn]’
--allowinsert
Export Export data from HDFS to sqoop export --connect ‘jdbc:sqlserver://
SQL Server in one batch [IPAddress];database=[DatabaseName];instancename=[
process InstanceName];username=[UserName];password=[Pass
word]’ --table [SQLServerTableName] --export-dir
[HDFSTableLocation] --batch