We use cookies to give you the best experience on our website. If you continue to browse, then you agree to our privacy policy and cookie policy.
Unfortunately, activation email could not send to your email. Please try again.
Syncfusion Feedback

Spark can't be initialized

Thread ID:

Created:

Updated:

Platform:

Replies:

129797 Apr 4,2017 08:40 AM UTC Apr 5,2017 07:09 AM UTC Big Data Platform 1
loading
Tags: General
amir
Asked On April 4, 2017 08:40 AM UTC

On syncfusion bigdata studio ver 3.1.0.59

after enabling spark on spark tab the scal shows 

Starting shell. Please Wait......

and IPython shows

initializing spark, please wait but it stays on this message.

on log file i didn't find anything related to spark except this line

Syncfusion.BigData.InstallerAgent.Base.HadoopCommandHandler Removed Firewall rule - Syncfusion Big Data Agent IphythonSpark


what's the problem and what should i do?


Nandhini K [Syncfusion]
Replied On April 5, 2017 07:09 AM UTC

Hi Amir, 
 
We regret for the inconvenience caused.  
 
In Big Data Studio, Spark Memory Alert is shown if required RAM memory for Spark is not available. As you said, you did force start the Spark tab. We suspect it as a memory related problem. 
 
Please ensure the below mentioned memory requirements to run Spark. 
 
System Memory   
·         Spark needs a minimum required memory in system to start shell and to do executions with it. We recommend a machine with minimum 8 GB RAM or higher. 
·         In Big Data Studio, Spark is running with the default memory configuration. Please ensure that the memory required for Spark is available in your system. Refer the following link to know more about the spark required memory calculation. 
 
Resource Manager Memory 
·         Ensure that you have enough memory in the Resource Manager by checking the resource manager URL. 
·         Enough Memory for Spark = (Memory Total – Memory Used) > Spark required memory  
If System Memory is available, still shell fails? 
·         You can increase allocated memory by changing the yarn.nodemanager.resource.memory-mb property value in the yarn-site.xml file available in the below location.   
“C:\Syncfusion\BigData\<version>\BigDataSDK\SDK\Hadoop\etc\hadoop”  
You have to specify the memory value in megabytes here. 
 
If the same issue persists even after trying out the above mentioned troubleshooting, please provide the following logs to investigate it further. 
Log location: 
“C:\Syncfusion\BigData\<version>\BigDataSDK\Logs\Studio” 
“C:\Syncfusion\BigData\<version>\BigDataSDK\Utilities\BigDataManagementStudio\Logs\Console\Spark” 
 
 
Regards, 
Nandhini K 


CONFIRMATION

This post will be permanently deleted. Are you sure you want to continue?

Sorry, An error occured while processing your request. Please try again later.

Warning Icon You are using an outdated version of Internet Explorer that may not display all features of this and other websites. Upgrade to Internet Explorer 8 or newer for a better experience.Close Icon

;