Spark can't be initialized

[amir]

On syncfusion bigdata studio ver 3.1.0.59

after enabling spark on spark tab the scal shows 

Starting shell. Please Wait......

and IPython shows

initializing spark, please wait but it stays on this message.

on log file i didn't find anything related to spark except this line

Syncfusion.BigData.InstallerAgent.Base.HadoopCommandHandler Removed Firewall rule - Syncfusion Big Data Agent IphythonSpark

what's the problem and what should i do?

[Nandhini K]

Hi Amir, 

 

We regret for the inconvenience caused.  

 

In Big Data Studio, Spark Memory Alert is shown if required RAM memory for Spark is not available. As you said, you did force start the Spark tab. We suspect it as a memory related problem. 

 

Please ensure the below mentioned memory requirements to run Spark. 

 

System Memory	·         Spark needs a minimum required memory in system to start shell and to do executions with it. We recommend a machine with minimum 8 GB RAM or higher.  ·         In Big Data Studio, Spark is running with the default memory configuration. Please ensure that the memory required for Spark is available in your system. Refer the following link to know more about the spark required memory calculation.  https://www.syncfusion.com/kb/7111/memory-recommendation-for-spark-execution
Resource Manager Memory	·         Ensure that you have enough memory in the Resource Manager by checking the resource manager URL.  http://<name_node_host>:8088/cluster   e.g. http://localhost:8088/cluster   ·         Enough Memory for Spark = (Memory Total – Memory Used) > Spark required memory
If System Memory is available, still shell fails?	·         You can increase allocated memory by changing the yarn.nodemanager.resource.memory-mb property value in the yarn-site.xml file available in the below location.    “C:\Syncfusion\BigData\<version>\BigDataSDK\SDK\Hadoop\etc\hadoop”   You have to specify the memory value in megabytes here.

 

If the same issue persists even after trying out the above mentioned troubleshooting, please provide the following logs to investigate it further. 

Log location: 

“C:\Syncfusion\BigData\<version>\BigDataSDK\Logs\Studio” 

“C:\Syncfusion\BigData\<version>\BigDataSDK\Utilities\BigDataManagementStudio\Logs\Console\Spark” 

 

 

Regards, 

Nandhini K