Differences

This shows you the differences between two versions of the page.

--- spark:windows_dev_environment [2017/11/27 13:32] – created david
+++ spark:windows_dev_environment [2021/01/12 21:08] (current) – external edit 127.0.0.1
@@ Line 3: / Line 3: @@
 Inspiration stolen/taken from: [[https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-tips-and-tricks-running-spark-windows.html|Running Spark Applications on Windows - Jacek Laskowski]]
+====== Download ======
 Download:
   * Spark: [[https://spark.apache.org/downloads.html]]
@@ Line 13: / Line 14: @@
   * Copy ''winutils.exe'' to: ''C:\APPLICATIONS\hadoop\bin''
+Set paths:
 <code>
 set HADOOP_HOME=c:\APPLICATIONS\hadoop
@@ Line 19: / Line 21: @@
 echo %HADOOP_HOME%
-winutils.exe chmod -R 777 C:\tmp\hive
 </code>
 **Note:** Path settings should be set in control panel for persistence.
@@ Line 34: / Line 33: @@
     - Paste in: ''C:\APPLICATIONS\hadoop\bin''
+Configure & Test:
+<code>
+winutils.exe chmod -R 777 C:\tmp\hive
+winutils.exe ls -F C:\tmp\hive
+</code>
 ====== Install/Configure Spark ======
@@ Line 42: / Line 46: @@
 ===== Run it =====
 <code>
-C:\APPLICATIONS\Spark\spark-2.2.0-bin-hadoop2.7\bin>spark-shell2.cmd
+cd C:\APPLICATIONS\Spark\spark-2.2.0-bin-hadoop2.7\bin
+spark-shell2.cmd
+</code>
+Custom run... (local master, 2 cores)
+<code>
+spark-shell2.cmd --master local[2]
+</code>
+Output:
+<code>
 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
 Setting default log level to "WARN".
@@ Line 65: / Line 79: @@
 Type :help for more information.
 </code>
+Test command:
+<code>
+spark.range(1).withColumn("status", lit("All seems fine. Congratulations!")).show(false)
+</code>
+exit:
+<code>
+sys.exit
+</code>
+----
+  * Web Interface: [[http://localhost:4040/jobs/]]
+    * Configuration: [[http://localhost:4040/environment/]]