大数据Spark学习：Scala基础第一课
计划：
阶段1：
精通Spark内核
阶段2：
精通千万级的项目
阶段3：
机器学习
JAVA本身不是伟大的语言，伟大的是JVM，构件分布式平台什么的，依赖的是JVM，不一定要JAVA语言
可认为Scala是JAVA的升级语言，JAVA是支持面向对象的语言，而非纯面向对象的语言。Scala是一切皆对象， 是纯面向对象语言。面向对象结合函数式编程。
不可变变量声明val result = 10+2 不可再次赋值，分布式数据的时候，传输数据、校验数据等不想改变这个数据
可变变量var name = "Spark" 这个是可变的，可以再次赋值
建议首先用val，没办法的时候才用var
不指定类型可根据只自动识别，也可指定类型
var name:String = "Spark"
一旦指定类型了，赋值只能赋值指定类型的子类型
也可一次性赋值多个
val age1,age2,age3 = 0
一切皆对象的理解，最基本数据也是对象：
scala> 0.to(5)
res4: scala.collection.immutable.Range.Inclusive = Range(0, 1, 2, 3, 4, 5)
scala> 1+1
res5: Int = 2
scala> 1.+(1)
warning: there were 1 deprecation warning(s); re-run with -deprecation for detai
ls
res6: Double = 2.0
其中的+其实是方法
scala> import scala.math._
import scala.math._
scala> min(20,4)
res8: Int = 4
=========apply=======
scala> val array = Array(1,2,3,4)
array: Array[Int] = Array(1, 2, 3, 4)
scala> array
res11: Array[Int] = Array(1, 2, 3, 4)
scala> val arr = Array.apply(1,2,3,4)
arr: Array[Int] = Array(1, 2, 3, 4)
scala> arr
res12: Array[Int] = Array(1, 2, 3, 4)
=========表达式if=======
scala> val age = 19
age: Int = 19
scala> if(age>=18)"adult" else "child"
res14: String = adult
=========表达式块=======
scala> val result = if(age>=18) {
| "adult"
| }else{
| "child"
| }
result: String = adult
scala> var buffered = 0
buffered: Int = 0
scala> val result = if(age>=18){
| "adult"
| }
result: Any = adult
scala> val result = if(age>=18){
| "adult"
| buffered = 10
| buffered
| }
<console>:13: warning: a pure expression does nothing in statement position; you
may be omitting necessary parentheses
"adult"
^
result: AnyVal = 10
Scala的最后的代码快的值就是返回值
=========打印，换行、占位符等=======
scala> println("Spark")
Spark
scala> print("Spark")
Spark
scala> print("\nSpark")
Spark
scala> printf(" %s is the future of big data computation framework", "Spark")
Spark is the future of big data computation framework
scala> printf(" %s is the future of big data computation framework \n ", "Spark
")
Spark is the future of big data computation framework
=========readLine=======
scala> readLine
res21: String = ss
scala> readLine(" Please input you password: ")
Please input you password: res22: String = hahah
scala> readInt
res23: Int = 9
=========循环while=======
scala> var element = 100
element: Int = 100
scala> while(element>10){
| println(element)
| element -= 10
| }
100
90
80
70
60
50
40
30
20
=========循环for=======
scala> 0 to element
res25: scala.collection.immutable.Range.Inclusive = Range(0, 1, 2, 3, 4, 5, 6, 7
, 8, 9, 10)
scala> for(i<-0 to element)println(i)
0
1
2
3
4
5
6
7
8
9
10
scala> for(i<-0 to element if i%2==0)println(i)
0
2
4
6
8
10
scala> import scala.util.control.Breaks._
import scala.util.control.Breaks._
scala> for(i<-0 to element if i%2==0){
| println(i)
| if(i==4) break
| }
0
2
4
scala.util.control.BreakControl
scala> def f1:Any={
| for(i<-1 to 10){
| if(i==10) return i
| println(i)
| }
| }
f1: Any
scala> f1
1
2
3
4
5
6
7
8
9
res30: Any = 10
=========函数默认参数和带名参数(可不按照参数顺序来传)=======
scala> def f2(param1:String,param2:Int = 30) = param1+param2
f2: (param1: String, param2: Int)String
scala> f2("Spark")
res33: String = Spark30
scala> f2(param2=100,param1 ="Scala")
res34: String = Scala100
=========函数变长参数=======
scala> def sum(numbers:Int*)={var result=0;for(element<-numbers)result+=element;
result}
sum: (numbers: Int*)Int
scala> sum(1,2,3,4,5,6,7,8,9,10)
res0: Int = 55
scala> sum(1,2,3,4,5,6,7)
res1: Int = 28
如果要操作1到100这样，range不是整形
scala> sum(1 to 100)
<console>:9: error: type mismatch;
found : scala.collection.immutable.Range.Inclusive
required: Int
sum(1 to 100)
^
:_*是把range里面的每个元素提取出来（超重点语法）
scala> sum(1 to 100:_*)
res3: Int = 5050
=========过程(无结果，只有过程)=======
scala> def f3(con: String) = {println("Good "+con)}
f3: (con: String)Unit
scala> f3("yyy")
Good yyy
scala> def f3(con: String):Unit = {println("Good "+con)}
f3: (con: String)Unit
scala> f3("xxx")
Good xxx
=========lazy(只有使用到的时候才会加载，比较耗时时用)=======
scala> import scala.io.Source._
import scala.io.Source._
//文件不存在，因为有lazy，所以不报错
scala> lazy val content = fromFile("F:/xxx")
content: scala.io.BufferedSource = <lazy>
//文件不存在，因为没lazy，所以报错
scala> val content = fromFile("F:/xxx")
java.io.FileNotFoundException: F:\xxx (系统找不到指定的文件。)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
//文件存在，没lazy，不会报错
scala> val content = fromFile("F:/FH11000001201601121730091044660854A")
content: scala.io.BufferedSource = non-empty iterator
=========异常=======
scala> import java.io.FileNotFoundException
import java.io.FileNotFoundException
try{
val content = fromFile("F:/xxx")
}catch{
case _:FileNotFoundException=>println("hahahh,not found")
}finally{
println("88")
}
=========不可变数组=======
val表示array指向的地址不能修改，而不是里面的元素
scala> val array = new Array[Int](5)
array: Array[Int] = Array(0, 0, 0, 0, 0)
scala> array(3)
res14: Int = 0
scala> array(2) = 8
scala> array
res16: Array[Int] = Array(0, 0, 8, 0, 0)
scala> val arr1 = Array("Scala","Spark")
arr1: Array[String] = Array(Scala, Spark)
scala> arr1(2) = "hadoop"
java.lang.ArrayIndexOutOfBoundsException: 2
at .<init>(<console>:13)
at .<clinit>(<console>)
=========可变数组=======
scala> val arrBuffer = ArrayBuffer[Int]()
arrBuffer: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer()
scala> arrBuffer +=10
res18: arrBuffer.type = ArrayBuffer(10)
scala> arrBuffer +=(11,1,2,37,8,)
<console>:1: error: illegal start of simple expression
arrBuffer +=(11,1,2,37,8,)
^
scala> arrBuffer +=(11,1,2,37,8)
res19: arrBuffer.type = ArrayBuffer(10, 11, 1, 2, 37, 8)
scala> arrBuffer ++=Array(1,2,3,4)
res20: arrBuffer.type = ArrayBuffer(10, 11, 1, 2, 37, 8, 1, 2, 3, 4)
scala> arrBuffer.trimEnd(3)
scala> arrBuffer
res22: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(10, 11, 1, 2, 37
8, 1)
scala> arrBuffer.trimEnd(3)
scala> arrBuffer
res24: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(10, 11, 1, 2)
scala> arrBuffer.insert(2,100)
scala> arrBuffer
res28: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(10, 11, 100, 1, 2
)
scala> arrBuffer.insert(2,33,44)
scala> arrBuffer
res30: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(10, 11, 33, 44, 1
00, 1, 2)
scala> arrBuffer.remove(3)
res32: Int = 44
scala> arrBuffer
res33: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(10, 11, 33, 100,
1, 2)
scala> arrBuffer.remove(3,2)
scala> arrBuffer
res35: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(10, 11, 33, 2)
=========数组操作：可变数组和不可变数组转换以及排序等等=======
scala> arrBuffer.toArray
res36: Array[Int] = Array(10, 11, 33, 2)
scala> val array2 = arrBuffer.toArray
array2: Array[Int] = Array(10, 11, 33, 2)
scala> array2.toBuffer
res37: scala.collection.mutable.Buffer[Int] = ArrayBuffer(10, 11, 33, 2)
scala> for(elem<-array2)println(elem)
10
11
33
2
//每2个位置打印一次
scala> for(i<-0 until (array2.length,2))println(array2(i))
10
33
//从尾部开始
scala> for(i<-(0 until array2.length).reverse)println(array2(i))
2
33
11
10
//排序
scala> import scala.util.Sorting
import scala.util.Sorting
scala> Sorting.quickSort(array2)
scala> array2
res44: Array[Int] = Array(2, 10, 11, 33)
//每个元素搜集起来
scala> for(i<-array2) yield i*i
res45: Array[Int] = Array(4, 100, 121, 1089)
//取里面小于3的数
scala> for(i<-array2 if i/3==0) yield i
res46: Array[Int] = Array(2)
//更牛的写法
scala> array2.filter(_%3==0).map(i=>i*i)
res47: Array[Int] = Array(1089)
=========Map=======
//不可变Map
scala> val person = Map("Spark"->6,"Hadoop"->11)
person: scala.collection.immutable.Map[String,Int] = Map(Spark -> 6, Hadoop -> 1
1)
scala> person("Hadoop")
res51: Int = 11
//可变Map
scala> val persons = scala.collection.mutable.Map("Spark"->6,"Hadoop"->11)
persons: scala.collection.mutable.Map[String,Int] = Map(Hadoop -> 11, Spark -> 6
)
scala> persons += ("Flink"->5)
res52: persons.type = Map(Hadoop -> 11, Spark -> 6, Flink -> 5)
scala> val sparkValue = if(persons.contains("Spark"))persons("Spark") else 1000
sparkValue: Int = 6
scala> val sparkValue = persons.getOrElse("Spark",1000)
sparkValue: Int = 6
scala> for((key,value)<-persons) println(key+":"+value)
Hadoop:11
Spark:6
Flink:5
//排过序的
scala> val persons = scala.collection.immutable.SortedMap("Spark"->6,"Hadoop"->1
1)
persons: scala.collection.immutable.SortedMap[String,Int] = Map(Hadoop -> 11, Sp
ark -> 6)
=========Tuple(函数最后返回若干不同类型的时候，就用Tuple)=======
scala> val tuple = ("Spark",1,3.0)
tuple: (String, Int, Double) = (Spark,1,3.0)
scala> tuple._1
res55: String = Spark
之后稍微过了一下Spark的一些源码，主要体现上面这些代码
作业一：
移除一个数组中第一个负数后的所有负数（不包括第一个）
我的解答：
def main (args :Array[String]):Unit={
//val array = Array(1,3,7,-12,4,8,9,-11,-10,2,8,9)
val array = Array(1,3,7,4,8,9,2,8,9)
val firstIndex = getFirstIndex(array )
if(firstIndex>= 0){
val arrayBuffer = array.filter(_ >=0). map(i =>i). toBuffer
arrayBuffer.insert (firstIndex , array (firstIndex ))
for(element <-arrayBuffer )print (element +" ")
}else{
val array2 = array. filter(_>= 0).map (i =>i )
for(element <-array2)print(element+ " ")
}
}
def getFirstIndex( array:Array[Int]):Int={
for(i<-0 until array.length ){if(array(i)<0 )return i}
-1
}
老师的答案：
方式一：
发现第一个负数之后的每一个负数会立即进行移除，性能较差，多次移动数组
方式二：
首先记录所有不需要移除的元素的索引，最后一次性移除所有需要移除的元素,性能相对较高
问题：firstNegative是啥东西？Scala自带的第一个负数识别？还是啥意思？
本文出自 “一支花傲寒” 博客，谢绝转载！