9. Introduction to Spark

WHAT IS SPARK?

"A fast and general engine for large-scale data processing"

It's Scalable

Process Flow for Spark
		Executor -Cache Tasks
Driver Program -Spark Context	Cluster Manager (Spark,YARN)	Executor -Cache Tasks
		Executor -Cache Tasks

It's Fast

• "Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk."

• DAG Engine (directed acyclic graph) optimizes workflows

It's Hot

• Amazon

• Ebay: log analysis and aggregation

• NASA JPL: Deep Space Network

• Groupon

• TripAdviser

• Yahoo

• Many others: https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark

It's Not That Hard

• Code in Python, Java, or Scala

• Build around one main concept: the Resilient Distributed Dataset (RDD)

Components Of Spark

SPARK CORE
Spark Streaming	Spark SQL	MLLLib	GraphX

This Course Uses Scala

• Why Scala?

  • Spark itself is written in Scala

  • Scala's functional programming model is a good fit for distributed processing

  • Gives you fast performance (Scala compiles to Java bytecode)

  • Less code & boilerplate stuff than Java

  • Python is slow in comparison

• But...

  • You probably don't know Scala

  • So we'll have to learn the basics first.

  • It's not as hard you think!

Fear Not

• Scala code in Spark looks a LOTlike Python code.

• Python code to square numbers in a data set:

    nums = sc.parallelize([1, 2, 3, 4])
    squared = nums.map[lambda x: x * x].collect()

• Scala code to square numbers in a data set:

    val nums = sc.parallelize(List(1, 2, 3, 4))
    val squared = nums.map(x -> x * x).collect()