We use cookies to give you the best experience on our website. If you continue to browse, then you agree to our privacy policy and cookie policy. Image for the cookie policy date

Relocate HDFS datanode directory for Syncfusion Big Data Platform

Hi,

I've installed Syncfusion Big Data Platform to root drive [D:\] which has limited available space.  Can you please advise how can I relocate/change the HDFS datanode directory to another directory [E:\]?

Current datanode directory:  D:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode
Destination datanode directory:  E:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode


Thank you.

Kelly.

3 Replies

MM Mahendran Maragatha Patchai Syncfusion Team April 19, 2016 01:52 PM UTC

Hi Kelly,                                                                                
Thank you for using Syncfusion Big Data Platform.
You can able to Relocate the HDFS data node directory to another directory without any data loss. Please find the steps to “Relocate HDFS Datanode directory for Syncfusion Big Data Platform” below.

1.      Stop the Hadoop services, if running.

               

2.      Copy the Data node directory from "D:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode" to the Destination Directory “E:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode



                      

3.      Go to “D:\Syncfusion\BigData\2.11.0.92\BigDataSDK\SDK\Hadoop\etc\hadoop” Directory and locate the hdfs-site.xml.               
           

4.      Find out the “dfs.datanode.data.dir” property which is located in hdfs-site.xml and change the value of that property to your NEW_LOCATION (E:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode)
                       

5.      Restart the Hadoop Services. 
                    


Please let us know if you have any other queries.                                        
Regards,
Mahendran Maragatha Patchai                                                                                                        
 



KE Kelly April 25, 2016 03:02 AM UTC

Hi Mahendran,

I've successfully relocate the HDFS datanode directory.  However, I'm getting error while trying sample scripts for scala, python, and iPython.  These sample scripts were able to execute without error before relocating the datanode directory.  Appreciate your advise on the resolution.

Here's the changes done in hdfs-site.xml file.  All services started successfully after this change.
From old value
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/Syncfusion/BigData/2.11.0.92/BigDataSDK/Metadata/data/dfs/datanode</value>
</property>


To new value:
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///I:/MODTEST/Data/Syncfusion/BigData/2.11.0.92/BigDataSDK/Metadata/data/dfs/datanode</value>
</property>



Here's sample script:
// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

sqlContext.sql("create External table if not exists recommend_Ratings1(criticid string,movieid string,rating double) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/Data/Ratings/'")

// Queries are expressed in HiveQL
sqlContext.sql("FROM recommend_Ratings1 SELECT criticid,movieid,rating").collect().foreach(println)


Thank you for your help!

Regards,
Kelly

Attachment: errormessage_41400dd4.zip


MM Mahendran Maragatha Patchai Syncfusion Team April 25, 2016 08:49 AM UTC

 Hi Kelly,

A support incident has been created under your account, for follow-up of this requirement. Please log on to our support website to check for further updates

https://www.syncfusion.com/account/login?ReturnUrl=%2fsupport%2fdirecttrac%2fincidents

Please let me know if you have any questions.

Regards,
Mahendran Maragatha Patchai
 


Loader.
Live Chat Icon For mobile
Up arrow icon