Local Cluster Development

[Oula Alshiekh]

After Local cluster Installation
if we access the cluster through cluster manager we will be able to access cluster  monitoring  functions as described here https://help.syncfusion.com/bigdata/cluster-manager/cluster-monitoring-and-alert

but what if we want to handle hdfs and hive operations ?
i think if iam right we should connect to this cluster through syncfusion big data studio  Add Cluster window

but iam getting the error
Primary Name node Wasn't accessible
even though all services are working

is this way right ,and what is the problem please

[Baskaran]

Hi Oula,

 

Could you please share us the following details to assist you further?

 

1.       1. Are studio and single node cluster running on different machines or in same machine?

2.       2. How did you try to add the cluster in studio, using IP or hostname?

3.       3. Please share the following logs from single node cluster machine.

           Agent logs:  <drive>:\Syncfusion\BigData\<version>\Agent\logs\

          Configuration file: <drive>:\Syncfusion\BigData\<version>\Agent\Configuration.xml

4.       4. Please share the following logs from studio machine.

          <drive>:\Syncfusion\BigData\<version>\BigDataSDK\Logs\Studio

 

Regards,

Baskaran V

[Oula Alshiekh]

thanks for your help
1 Syncfusion big data studio is working on my local pc while cluster is working on a vm that is hosted on my pc  through vmware
2- i added  the cluster by ip     (of course iam able to ping this ip )
3 - the strange thing is that:
when i tried to connect to the local cluster which is hosted on a vm on my pc  by adding host name every thing was ok and i was able to handle hive instructions
but after a restart of my pc when i entered cluster manager on hosted vm  i find it as dead and  all services are  not working and iwas  not able to connect it through big data studio by entering host name now

i wish if there was a task bar telling you that start all services or any service operation is happening now (what time should it take does that depends on ram capacity)
here are the logs
vm  which hosts local cluster and contains cluster manager is called  WIN-GD3ERV5R8PT

i couldn't find this path in  single node cluster machine instead i find this log under the path
C:\Syncfusion\BigData\3.1.0.3\BigDataAgent\Manager Service\logs\InstallerAgent

and configuration.xml under the path
C:\Syncfusion\BigData\3.1.0.3\BigDataAgent\Manager Service

Note: my local cluster is working on a vm which is 2 Gb Ram is there any hard ware requirements for local cluster ?

Attachment: Local_cluster_Log_b1382cf5.rar

[Oula Alshiekh]

this is the message i get when connecting to local cluster

Attachment: Local_Cluster_Photo_2facbf69.rar

[Oula Alshiekh]

when i hit the button collect logs from local cluster these are the logs i got

Attachment: OULA_bb6214d7.zip

[Baskaran]

Hi Oula, 

  

Thank you for the update.  

  

Please find the response as follows. 

  

Query 

Response 

Regarding restarting PC that hosts local cluster on a VM and 

Message to start all services or any service operation is happening now 

Studio services will be automatically started when system is restarted whereas multi node cluster or local cluster are having different workflow. You can start local cluster service as shown in below screenshot using cluster manager and then try to add this cluster in Studio. 

  

 

  

My local cluster is working on a vm which is 2 Gb Ram is there any hard ware requirements for local cluster? 

Yes. As we recommended earlier threads Studio, local cluster and cluster nodes all require specific hardware requirement to run the services properly. Once again I am listing out the requirement as follows. 

  

	RAM	Local Disk	Number of machines or VMs needed
Studio	8 GB or higher	100 GB or higher	1
Local cluster	8 GB or higher	100 GB or higher	1
Multi node cluster	Name node - 8 GB or higher Data Node - 4 GB or higher	100 GB or higher	3 or higher

  

As we are following you in multiple threads, we understand that you couldn't evaluate the Syncfusion Big Data Platform especially cluster manager due to hardware shortage. We recommend you to avail the above mentioned hardware requirements to configure production environment properly.  

  

Or if you couldn't avail required hardware as of now, please let us know your current available machines, hardware details and the specific thing you are trying to work with (e.g. polybase, hive, etc.) so that we can check and let you know some workaround and tweaks to meet your requirement with available hardware resources. 

  

Regards, 

Baskaran V 

[Tim Maher]

Hi, How do I find out what Hostname/IP my local pc is?

[Aravindraja Thinakaran]

Hi Tim, 

Thanks for contacting Syncfusion support. 

You can get the IP address and Hostname of your local PC as follows. 

Get the IP address of PC	Open Command prompt, run the ipconfig command and you can get the IP address as like below
Get the host name of PC	Open Command prompt, run the hostname command and you can get the hostname of your local pc as like follows.

Thanks, 

Aravindraja T