RDD(Resilient Distributed Datasets)

Spark RDD : fault-tolerant collection of elements that can be operated on in parallel.

two types:

1. parallelized collections : take an existing Scala collection and run functions on it in parallel
2. Hadoop datasets : run functions on each record of a file in Hadoop distributed file system or any other storage system supported by Hadoop

scala> val data = Array(1,2,3,4,5)
data: Array[Int] = Array(1, 2, 3, 4, 5)

scala> val distData = sc.parallelize(data)
distData: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at :14

One important parameter for parallel collections is the number of slices to cut the dataset into.

RDDs support two types of operations: transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset.