34. Introducing Amazon Elastic MapReduce

RUNNING ON A CLUSTER

Distributed Spark

• This is the layout of a Spark Job

Layout of Spark Job
	Spark Driver
	Cluster Manager
Cluster Worker/ Executors	Cluster Worker/ Executors	Cluster Worker/ Executors

Other Spark-Submit Parameters

• --master

  • yarn - for running a YARN / Hadoop cluster

  • hostname:port - for connecting to a master on a Spark standalone cluster

  • mesos://masternode:port

  • A master in your SparkConf will override this!!!

• --num-executors

  • Must set explicitly with YARN, only 2 by default

• --executor-memory

  • Make sure you don't try to use more more memory than you have

• --total-executor-cores

Amazon Elastic MapReduce

• A quick way to create a cluster with Spark, Hadoop, and YARN pre-installed

• You pay by the hour-instance and for network and storage IO

• Let's run our one-million-ratings movie recommender on a cluster

Let's Use Elastic MapReduce

• Very quick and easy way to rent time on a cluster of your own

• Sets up a default spark configuration for you on top of Hadoop's YARN cluster manager

  • Buzzword alert! We're using Hadoop! Well, a part of it anyhow.

• Spark also has a built-in standalone cluster manager, and scripts to set up its own EC2-based cluster.

  • But the AWS console is even easier.

• Spark on EMR isn't really expensive, but it's not cheap either.

  • Unlike MapReduce with MRJob, you'll be using m3.xlarge instances.

  • I racked up about $30 running Spark jobs over a few hours preparing this course.

  • You also have to remember to shutdown your clusters when you're done, or else...

  • So you might just want to watch, and not follow along.

• Make sure things run locally on a subset of your data first.

Getting Set Up On EMR

• Make an Amazon Web Services account

• Create an EC2 key pair and download the .pem file

• On Windows, you'll need a terminal like PUTTY

  • For PUTTY, need to convert the .pem to a .ppk private key file

• I'll walk you through this now.