# 41. \[Activity] Using DataFrames and DataSets

## Activity

* You don't have to use SQL with SparkSQL necessarily
* You can also call functions directly without actually relying on the sql syntax
* That can be little bit more efficient
* Import DataFrames.scala from sourcefolder into SparkScalaCourse in Spark-Eclipse IDE
* Open DataFrames.scala and look at the code

## Looking At The Code

* You are still importing the SQL package from Saprk and alot of the codes looks the same as SparkSQL.scala

```
    println("Let's select the name column:")
    people.select("name").show()
```

* Instead of calling spark.sql select name, I am going to operate on the dataset directly by calling .select("name").show()
* This will quickly show the top 20 results for that dataset

```
    println("Filter out anyone over 21:")
    people.filter(people("age") < 21).show()
```

* We can also call people on our dataset, by calling the filter functio to filter out people who are over the age of 21
* This will show the top 20 results for that dataset

```
    println("Group by age:")
    people.groupBy("age").count().show()
```

* This will group the people by their age and count the total number for that age
* This reduces the need for SQL like query

```
    println("Make everyone 10 years older:")
    people.select(people("name"), people("age") + 10).show()
```

* We are selecting the name of people, and selecting the age column adding 10 as we go and show these results as we go.
* So now let's go ahead and run these results
* You should now see the top 20 results for each of the respective functions you specified