We use cookies to give you the best experience on our website. If you continue to browse, then you agree to our privacy policy and cookie policy.
Unfortunately, activation email could not send to your email. Please try again.

Relocate HDFS datanode directory for Syncfusion Big Data Platform

Thread ID:

Created:

Updated:

Platform:

Replies:

123757 Apr 18,2016 08:44 PM Apr 25,2016 04:49 AM Big Data Platform 3
loading
Tags: General
Kelly
Asked On April 18, 2016 08:44 PM

Hi,

I've installed Syncfusion Big Data Platform to root drive [D:\] which has limited available space.  Can you please advise how can I relocate/change the HDFS datanode directory to another directory [E:\]?

Current datanode directory:  D:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode
Destination datanode directory:  E:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode


Thank you.

Kelly.

Mahendran Maragatha Patchai [Syncfusion]
Replied On April 19, 2016 09:52 AM

Hi Kelly,                                                                                
Thank you for using Syncfusion Big Data Platform.
You can able to Relocate the HDFS data node directory to another directory without any data loss. Please find the steps to “Relocate HDFS Datanode directory for Syncfusion Big Data Platform” below.

1.      Stop the Hadoop services, if running.

               

2.      Copy the Data node directory from "D:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode" to the Destination Directory “E:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode



                      

3.      Go to “D:\Syncfusion\BigData\2.11.0.92\BigDataSDK\SDK\Hadoop\etc\hadoop” Directory and locate the hdfs-site.xml.               
           

4.      Find out the “dfs.datanode.data.dir” property which is located in hdfs-site.xml and change the value of that property to your NEW_LOCATION (E:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode)
                       

5.      Restart the Hadoop Services. 
                    


Please let us know if you have any other queries.                                        
Regards,
Mahendran Maragatha Patchai                                                                                                        
 


Kelly
Replied On April 24, 2016 11:02 PM

Hi Mahendran,

I've successfully relocate the HDFS datanode directory.  However, I'm getting error while trying sample scripts for scala, python, and iPython.  These sample scripts were able to execute without error before relocating the datanode directory.  Appreciate your advise on the resolution.

Here's the changes done in hdfs-site.xml file.  All services started successfully after this change.
From old value
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/Syncfusion/BigData/2.11.0.92/BigDataSDK/Metadata/data/dfs/datanode</value>
</property>


To new value:
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///I:/MODTEST/Data/Syncfusion/BigData/2.11.0.92/BigDataSDK/Metadata/data/dfs/datanode</value>
</property>



Here's sample script:
// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

sqlContext.sql("create External table if not exists recommend_Ratings1(criticid string,movieid string,rating double) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/Data/Ratings/'")

// Queries are expressed in HiveQL
sqlContext.sql("FROM recommend_Ratings1 SELECT criticid,movieid,rating").collect().foreach(println)


Thank you for your help!

Regards,
Kelly

Attachment: errormessage_41400dd4.zip

Mahendran Maragatha Patchai [Syncfusion]
Replied On April 25, 2016 04:49 AM

 Hi Kelly,

A support incident has been created under your account, for follow-up of this requirement. Please log on to our support website to check for further updates

https://www.syncfusion.com/account/login?ReturnUrl=%2fsupport%2fdirecttrac%2fincidents

Please let me know if you have any questions.

Regards,
Mahendran Maragatha Patchai
 


CONFIRMATION

This post will be permanently deleted. Are you sure you want to continue?

Sorry, An error occured while processing your request. Please try again later.

You are using an outdated version of Internet Explorer that may not display all features of this and other websites. Upgrade to Internet Explorer 8 or newer for a better experience.

;