We use cookies to give you the best experience on our website. If you continue to browse, then you agree to our privacy policy and cookie policy.
Unfortunately, activation email could not send to your email. Please try again.

Question Requires Assisting on SQOOP to HIVE.(Server SQL)

Thread ID:

Created:

Updated:

Platform:

Replies:

119466 Jun 25,2015 04:39 AM Jul 16,2015 01:21 AM Big Data Platform 5
loading
Tags: General
alex
Asked On June 25, 2015 04:39 AM

FIrst of all, I may ask some stupid question, because I'm still new in BIG data platform.

 Well I had import one "Test" database from Server SQL into HDFS by using SQOOP, The image 0.png shows the example the job being done, Image 1.png is the SQL serer 2012 database, and Image 2.png is the databse store in HDFS.

And The question is 
1)does these database can be used in Hive ? if can how does it being used?
2)does the 'records' in the SQL server will also imported to the HDFS or they just import the 'Table' design only?
3)can i load the 'records' into the table that is created at HIVE? or how I select those data from HIVE
4)from the image 2.png, why is it the data have converted into part-m-00000, how can i read in HIVE AND PIG

Please and Thank you :)





Attachment: Image_1c4419d5.rar

Praveena P [Syncfusion]
Replied On June 28, 2015 03:26 PM

Hi Alex,

Thank you for using Syncfusion Products.

S.No

Query

Response

1

Does the 'records' in the SQL server will also imported to the HDFS or they just import the 'Table' design only?

The records in the table from SQL server are imported to the HDFS as a set of files containing a copy of a table and not the table design.

2

Can I load the 'records' into the table that is created at HIVE? or how I select those data from HIVE

Yes, certainly the records of table  transferred from SQL server to  HDFS using Sqoop can be loaded in to the table created in Hive and can query the records from hive.

This can be achieved by hive script .

Hive script for loading  imported data in to a table created in hive.

load data inpath ‘/user/SYSTEM/Members’ into table HiveTable;

HiveTable –  name of the table created in hive.

‘/user/SYSTEM/Members’ – HDFS path of the imported data.

The data in Hive can be queried similar to SQL query

Example : Select  * from  HiveTable;

Please refer to hive module in studio for details.

3

From the image 2.png, why is it the data have converted into part-m-00000, how can i read in HIVE AND PIG

The import process of Sqoop is an Mapping job and the output files of the imported data are by default named based on below

Part – (m) based on  mapper job – mapper_job_ number.

In pig ,the files in HDFs can be read by load commands in Pig.

Members = LOAD ‘/user/SYSTEM/Members’ using PigStorage(',') as

(ID,NAME,AGE,EMAIL);

And perform operations on data using pig Script.

In Hive, Please refer the solution of query 2 as both queries are similar.

Please refer the getting started samples of pig and hive for your reference.

4

Does these database can be used in Hive ? if can how does it being used?

I hope that this query is similar to the query (2) , which is addressed earlier. You need to import the contents of the database to HDFS using Sqoop and then load it in hive using ‘Load’ command.

Please let me know if you have any concerns.

Regards,

Praveena


alex
Replied On June 29, 2015 03:39 AM

Hi Praveena P [Syncfusion], 

Thanks for the assisting on the my question. I have better understanding of it already

Rilfanayasmin Jamal Nasar [Syncfusion]
Replied On June 30, 2015 12:02 AM

Hi Alex,
Thank you for your update.
Please let us know if you require any further assistance on this.
Regards,
Yasmin

Andy Menon
Replied On July 15, 2015 12:33 AM

In addition, if you need a 101 understanding of HDFS, HIVE and SQOOP check out Wintellects free Big Data Studio Tutorial. Shows how to use SQOOP to move data between SQL Server and HDFS.

https://www.wintellectnow.com/Videos/Watch?videoId=introduction-to-hadoop-using-syncfusion-big-data-platform/

This URL is iffy.  You may have to do a google search to see if you can get an unbroken link.

But this is 2 hours well spent!

Andy

Praveena P [Syncfusion]
Replied On July 16, 2015 01:21 AM

Hi Andy,

Thanks for your update.

Please find the following unbroken video link which leads to the proper site for Wintellects free Big Data Studio Tutorial.

https://www.wintellectnow.com/Videos/Watch?videoId=introduction-to-hadoop-using-syncfusion-big-data-platform

Please let me know if you need any further assistance on this.

Regards,
Praveena.

CONFIRMATION

This post will be permanently deleted. Are you sure you want to continue?

Sorry, An error occured while processing your request. Please try again later.

You are using an outdated version of Internet Explorer that may not display all features of this and other websites. Upgrade to Internet Explorer 8 or newer for a better experience.

;