We use cookies to give you the best experience on our website. If you continue to browse, then you agree to our privacy policy and cookie policy.
Unfortunately, activation email could not send to your email. Please try again.
Syncfusion Feedback

Pre-requisites for executing jobs in Sqoop

Sqoop requires JDBC connector jar files to be present in lib folder (SDK\Sqoop\lib) for execution of jobs.

Please follow below steps to move JDBC connector jar files to Sqoop lib folder.

 

Step 1 – Download jar files

 

Refer below table for download links on required JDBC connector jars

 

 

 

Step 2 – Move the downloaded JDBC connector jars to Sqoop\lib folder.

  1. For local cluster installed with big data studio, move the jar files to below location.

Directory:

<InstalledDirectory>\Syncfusion\BigData\<Version>\BigDataSDK\SDK\Sqoop\lib

 

Screenshot for reference:

C:\Users\rajendran.sp\Documents\My Received Files\local.PNG

  1. For Hadoop cluster formed with the help of Syncfusion big data cluster manager, move the jar files to each node where the Sqoop is installed.

 

Note: By default Sqoop will be installed in active name node machine.

 

Reference directory for remote cluster:

<InstalledDirectory>\Syncfusion\HadoopNode\<Version>\BigDataSDK\SDK\Sqoop\lib

 

Screenshot for reference:

C:\Users\rajendran.sp\Documents\My Received Files\remote.png

 

Note: if you run into errors with the message “ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class:”, then please ensure that the proper connector jar files present in the Sqoop lib folder.

Article ID: Published Date: Last Revised Date: Platform: Control:
6548 03/15/2016 04/05/2017 Big Data Platform General
Did you find this information helpful?
Comments
Tajudeen Akinosho May 10, 2016
I am having this problem please help sqoop import --connect "jdbc:mysql://127.0.0.1:3308/mytweets" --username root --password ademi1970 --table clean_tweets --target-dir /user -m 1 Warning: HBASE_HOME and HBASE_VERSION not set. Warning: HCAT_HOME not set Warning: HCATALOG_HOME does not exist HCatalog imports will fail. Please set HCATALOG_HOME to the root of your HCatalog installation. Warning: ACCUMULO_HOME not set. Warning: ZOOKEEPER_HOME not set. Warning: HBASE_HOME does not exist HBase imports will fail. Please set HBASE_HOME to the root of your HBase installation. Warning: ACCUMULO_HOME does not exist Accumulo imports will fail. Please set ACCUMULO_HOME to the root of your Accumulo installation. Warning: ZOOKEEPER_HOME does not exist Accumulo imports will fail. Please set ZOOKEEPER_HOME to the root of your Zookeeper installation. 16/05/10 20:52:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5 16/05/10 20:52:16 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/05/10 20:52:17 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 16/05/10 20:52:17 INFO tool.CodeGenTool: Beginning code generation 16/05/10 20:52:35 ERROR manager.SqlManager: Error executing statement: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:988) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:341) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2251) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2284) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2083) at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:806) at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:404) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:410) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:328) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:877) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:736) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:759) at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:269) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240) at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236) Caused by: java.net.ConnectException: Connection refused: connect at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:300) ... 33 more 16/05/10 20:52:35 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1584) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236) C:\Syncfusion\BigData\2.11.0.92\BigDataSDK\SDK\Sqoop\bin>
Reply
Not Known May 11, 2016
Hi Tajudeen, Thank you for your update. We have analyzed the provided log information, and suspect that the issue might have occurred because of the following reasons. • IP 127.0.0.0 did not bind with MYSQL server o Could you please try with 0.0.0.0/localhost instead of 127.0.0.0 in Sqoop query? • Admin privilege did not set properly o Execute the following command in MYSQL command line client before execute Sqoop Job GRANT ALL PRIVILEGES ON *.* TO 'root'@'172.16.102.12' identified by 'root' with grant option; flush privileges; Here 172.16.102.12 is IP address for Client machine Please ensure to connect MYSQL server properly with any other client like JDBC connection from your side and revert if you face any issues or in case of assistance. Thanks, Rajasekar
Add Comment
You must log in to leave a comment

Warning Icon You are using an outdated version of Internet Explorer that may not display all features of this and other websites. Upgrade to Internet Explorer 8 or newer for a better experience.Close Icon