Relocate HDFS datanode directory for Syncfusion Big Data Platform

[Kelly]

Hi,

I've installed Syncfusion Big Data Platform to root drive [D:\] which has limited available space.  Can you please advise how can I relocate/change the HDFS datanode directory to another directory [E:\]?

Current datanode directory:  D:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode

Destination datanode directory:  E:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode

Thank you.

Kelly.

[Mahendran Maragatha Patchai]

Hi Kelly,                                                                                
Thank you for using Syncfusion Big Data Platform.
You can able to Relocate the HDFS data node directory to another directory without any data loss. Please find the steps to “Relocate HDFS Datanode directory for Syncfusion Big Data Platform” below.

1.      Stop the Hadoop services, if running.

               

2.      Copy the Data node directory from "D:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode" to the Destination Directory “E:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode

                      

3.      Go to “D:\Syncfusion\BigData\2.11.0.92\BigDataSDK\SDK\Hadoop\etc\hadoop” Directory and locate the hdfs-site.xml.               
           

4.      Find out the “dfs.datanode.data.dir” property which is located in hdfs-site.xml and change the value of that property to your NEW_LOCATION (E:\Syncfusion\BigData\2.11.0.92\BigDataSDK\Metadata\data\dfs\datanode)
                       

5.      Restart the Hadoop Services. 
                    

Please let us know if you have any other queries.                                        
Regards,
Mahendran Maragatha Patchai                                                                                                        
 

[Kelly]

Hi Mahendran,

I've successfully relocate the HDFS datanode directory.  However, I'm getting error while trying sample scripts for scala, python, and iPython.  These sample scripts were able to execute without error before relocating the datanode directory.  Appreciate your advise on the resolution.

Here's the changes done in hdfs-site.xml file.  All services started successfully after this change.

From old value

<property>

<name>dfs.datanode.data.dir</name>

<value>file:/Syncfusion/BigData/2.11.0.92/BigDataSDK/Metadata/data/dfs/datanode</value>

</property>

To new value:

<property>

<name>dfs.datanode.data.dir</name>

<value>file:///I:/MODTEST/Data/Syncfusion/BigData/2.11.0.92/BigDataSDK/Metadata/data/dfs/datanode</value>

</property>

Here's sample script:

// sc is an existing SparkContext.

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

sqlContext.sql("create External table if not exists recommend_Ratings1(criticid string,movieid string,rating double) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/Data/Ratings/'")

// Queries are expressed in HiveQL

sqlContext.sql("FROM recommend_Ratings1 SELECT criticid,movieid,rating").collect().foreach(println)

Thank you for your help!

Regards,

Kelly

Attachment: errormessage_41400dd4.zip

[Mahendran Maragatha Patchai]

 Hi Kelly,

A support incident has been created under your account, for follow-up of this requirement. Please log on to our support website to check for further updates

https://www.syncfusion.com/account/login?ReturnUrl=%2fsupport%2fdirecttrac%2fincidents

Please let me know if you have any questions.

Regards,
Mahendran Maragatha Patchai