-
Notifications
You must be signed in to change notification settings - Fork 312
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG]: Trying to follow the "Getting Started" guide step by step #49
Comments
I ran the code again but without the backtick
Now, I'm getting this:
|
were you able to run |
I have the same issue. And please note that the jar to be used is 0.1.0 not 1.0.0-alpha (microsoft-spark-2.4.x-0.1.0.jar) |
It appears that you are using Spark 2.4.2, which is not supported yet. Can you please try either 2.4.0, 2.4.1 or 2.3.*? More intfo #43. |
@imback82 Yes it works! I have used 2.3.3. Thanks! |
After the result (computed) I have this error. 2019-04-25 14:56:48 INFO AbstractConnector:318 - Stopped Spark@38ef1355{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} |
This is a known in Spark: https://issues.apache.org/jira/browse/SPARK-12216 |
Great! Thanks! |
One temporary work-around to avoid seeing the spark temporary files error is to add the following two lines in the log4j.properties file of your spark installation:
Source: Stack Overflow |
Just to help others - in your spark directory there is a conf directory - add those two lines to the "log4j.properties" file. If there is no "log4j.properties" there should be a "log4j.properties.template" - copy the .template and remove the ".template" then add those lines at the top and the error will be hidden |
I got the same issue and found out it is because I am using Spark 2.4.2. |
Wow, thanks for the nice blog @FahaoTang! (btw, you may want to fix the link in your comment. |
Thanks for the reminder. I just fixed the link. :) |
@imback82 I installed spark v2.4.1 ... yes it is running. But still getting this:
|
Interesting ... I added the extension to
|
Did you follow my post? it should work. |
@jalchr Do you have |
I was running in debug mode, which produces .dll file.
Now, I'm getting an .exe file and works perfectly. Thanks all |
I followed the steps, and managed to run it with this command: C:\bin\spark-2.4.1-bin-hadoop2.7\bin\spark-submit --class org.apache.spark.deploy.DotnetRunner --master local microsoft-spark-2.4.x-0.1.0.jar HelloSpark.dll But encountered this exception: Missing Python executable 'python', defaulting to 'C:\bin\spark-2.4.1-bin-hadoop2.7\bin..' for SPARK_HOME environment variable. Please install Python or specify the correct Python executable in PYSPARK_DRIVER_PYTHON or PYSPARK_PYTHON environment variable to detect SPARK_HOME safely. Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. Looks like it could not find the runtimes folder, but it is there The files in the output folder:
|
Please try:
as the README says here https://github.com/dotnet/spark#get-started |
Now I managed to run it, but has some exception after it finishes. First, I would suggest to give a working sample of the command (IN-ONE-LINE), like:
Second, at the end of the program, it throws some Spark exceptions: 19/05/02 10:05:24 WARN SparkEnv: Exception while deleting Spark temp dir: C:\Users\user\AppData\Local\Temp\spark-e362e9c1-4189-4c58-9988-a415148242f5\userFiles-7b6adef9-4da3-49a8-bc4b-4ecdae22ad36 |
Please refer to #49 (comment) |
@imback82 my bad, I searched before I made the comment, I suggest the search function in GitHub backed by Elasticsearch has some keyword parsing issue. Anyway, all good. Thanks a lot for the promptly reply. I came a long way for Spark,, more than 1 year ago I evaluated Mobius but found out it has issue and can't meet our requirement, so we end up using native Scala solution, which is not ideal for .NET developers. Finally we have this .NET solution, let's hope and work together to make it beneficial to big data projects for .NET community. |
We look forward to working together, @unruledboy! |
Describe the bug
I was following the Getting Started guide step by step. When I execute the following:
I get this:
The text was updated successfully, but these errors were encountered: