Apache 2.0 Spark with Scala
  • Introduction
  • Introduction
    • Introduction
  • Section 1: Getting Started
    • 1. Warning about Java 9 and Spark2.3!
    • 2. Introduction, and Getting Set Up
    • 3. [Activity] Create a Histogram of Real Movie Ratings with Spark!
  • Section 2: Scala Crash Course
    • 4. [Activity] Scala Basics, Part 1
    • 5. [Exercise] Scala Basics, Part 2
    • 6. [Exercise] Flow Control in Scala
    • 7. [Exercise] Functions in Scala
    • 8. [Exercise] Data Structures in Scala
  • Section 3: Spark Basics and Simple Examples
    • 9. Introduction to Spark
    • 10. Introducing RDD's
    • 11. Ratings Histogram Walkthrough
    • 12. Spark Internals
    • 13. Key /Value RDD's, and the Average Friends by Age example
    • 14. [Activity] Running the Average Friends by Age Example
    • 15. Filtering RDD's, and the Minimum Temperature by Location Example
    • 16. [Activity] Running the Minimum Temperature Example, and Modifying it for Maximum
    • 17. [Activity] Counting Word Occurences using Flatmap()
    • 18. [Activity] Improving the Word Count Script with Regular Expressions
    • 19. [Activity] Sorting the Word Count Results
    • 20. [Exercise] Find the Total Amount Spent by Customer
    • 21. [Exercise] Check your Results, and Sort Them by Total Amount Spent
    • 22. Check Your Results and Implementation Against Mine
  • Section 4: Advanced Examples of Spark Programs
    • 23. [Activity] Find the Most Popular Movie
    • 24. [Activity] Use Broadcast Variables to Display Movie Names
    • 25. [Activity] Find the Most Popular Superhero in a Social Graph
    • 26. Superhero Degrees of Seperation: Introducing Breadth-First Search
    • 27. Superhero Degrees of Seperation: Accumulators, and Implementing BFS in Spark
    • 28. Superhero Degrees of Seperation: Review the code, and run it!
    • 29. Item-Based Collaborative Filtering in Spark, cache(), and persist()
    • 30. [Activity] Running the Similiar Movies Script using Spark's Cluster Manager
    • 31. [Exercise] Improve the Quality of Similiar Movies
  • Section 5: Running Spark on a Cluster
    • 32. [Activity] Using spark-submit to run Spark driver scripts
    • 33. [Activity] Packaging driver scripts with SBT
    • 34. Introducing Amazon Elastic MapReduce
    • 35. Creating Similar Movies from One Million Ratings on EMR
    • 36. Partitioning
    • 37. Best Practices for Running on a Cluster
    • 38. Troubleshooting, and Managing Dependencies
  • Section 6: SparkSQL, DataFrames, and DataSets
    • 39. Introduction to SparkSQL
    • 40. [Activity] Using SparkSQL
    • 41. [Activity] Using DataFrames and DataSets
    • 42. [Activity] Using DataSets instead of RDD's
  • Section 7: Machine Learning with MLLib
    • 43. Introducing MLLib
    • 44. [Activity] Using MLLib to Produce Movie Recommendations
    • 45. [Activity] Using DataFrames with MLLib
    • 46. [Activity] Using DataFrames with MLLib
  • Section 8: Intro to Spark Streaming
    • 47. Spark Streaming Overview
    • 48. [Activity] Set up a Twitter Developer Account, and Stream Tweets
    • 49. Structured Streaming
  • Section 9: Intro to GraphX
    • 50. GraphX, Pregel, and Breadth-First-Search with Pregel.
    • 51. [Activity] Superhero Degrees of Seperation using GraphX
  • Section 10: You Made It! Where to Go from Here.
    • 52. Learning More, and Career Tips
    • 53. Bonus Lecture: Discounts on my other "Big Data" / Data Science Courses.
Powered by GitBook
On this page
  • Activity
  • Exercise
  1. Section 2: Scala Crash Course

7. [Exercise] Functions in Scala

Activity

  • For this activity, go ahead and import from the resource the file LearningScala3.scala

  • LearningScala3.scala mainly talks about the building blocks of Functional Programming for Scala Code

  • The follow defines how to define a function in Scala

  • Scala do not need to explicitly declare a return statement to return the value

  • Scala will return the last value found in the function scope

      // Functions
    
      // Format is def <function name>(parameter name: type...) : return type = { expression }
      // Don't forget the = before the expression!
      def squareIt(x: Int) : Int = {
          x * x
      }                                               
      //> squareIt: (x: Int)Int
    
      def cubeIt(x: Int): Int = {x * x * x}           
      //> cubeIt: (x: Int)Int
    
      println(squareIt(2))                            
      //> 4
    
      println(cubeIt(2))                              
      //> 8
  • Scala can take in other functions as parameter for its current Function parameter

  • Need to declare => as the other function to transform to this current method return value

      // Functions can take other functions as parameters
    
      def transformInt(x: Int, f: Int => Int) : Int = {
          f(x)
      }                                               
      //> transformInt: (x: Int, f: Int => Int)Int
    
      val result = transformInt(2, cubeIt)            
      //> result  : Int = 8
    
      println (result)                                
      //> 8
  • You can declare Lambda functions in Scala without even explicitly giving them a name

      // "Lambda functions", "anonymous functions", "function literals"
      // You can declare functions inline without even giving them a name
      // This happens a lot in Spark.
      transformInt(3, x => x * x * x)                 //> res0: Int = 27
    
      transformInt(10, x => x / 2)                    //> res1: Int = 5
    
      transformInt(2, x => {val y = x * 2; y * y})    //> res2: Int = 16

Exercise

    // EXERCISE
    // Strings have a built-in .toUpperCase method. For example, "foo".toUpperCase gives you back FOO.
    // Write a function that converts a string to upper-case, and use that function of a few test strings.
    // Then, do the same thing using a function literal instead of a separate, named function.

    def transform(str: String) : String = {str.toUpperCase}
    //> transform: (str: String)String

    transform("testing1")                           
    //> res3: String = TESTING1
    transform("testing2")                           
    //> res4: String = TESTING2
    transform("testing3")                           
    //> res5: String = TESTING3

    def transformString(str: String, functionTesting: String => String) : String = {
        functionTesting(str)
    }                                               
    //> transformString: (str: String, functionTesting: String => String)String

    transformString("testing4", transform)   
    //> res6: String = TESTING4

    transformString("testingliteral", x => x.toUpperCase)
    //> res7: String = TESTINGLITERAL
Previous6. [Exercise] Flow Control in ScalaNext8. [Exercise] Data Structures in Scala

Last updated 7 years ago