topbykey: Returns the top k (largest) elements for each key from this RDD as defined by the specified implicit Ordering[T]. If the number of elements for a certain key is less than k, all of them will be returned.
val hiveCtx = new HiveContext(sc)
val url = "jdbc:mysql://192.168.36.94/homed_spider"
val tableName = "t_spider_program_final"
import java.util.Properties
val connProperties = new java.util.Properties
connProperties.setProperty("user", "root")
connProperties.setProperty("password", "123456")
val jdbcDF = hiveContext.read.jdbc(url, tableName, connProperties)
def tmiestampTostr(ts :Long) = {
val SimpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss")
SimpleDateFormat.format(new Date(ts*1000L))
}
2.时间转时间戳
def getTimestamp(t: String):Long = {
val date1 = """(\d\d\d\d)(\d\d)(\d\d)""".r
val date2 = """(\d\d\d\d)-(\d\d)-(\d\d).*""".r
val fm1 = new SimpleDateFormat("yyyyMMdd")
val fm2 = new SimpleDateFormat("y-M-d")
t match {
case date1(y, m, d) => fm1.parse(t).getTime / 1000
case date2(y, m, d) => fm2.parse(t).getTime / 1000
case _ => {
//Logger.info(s"can't parse time from this string:$t")
//curTime.set(1970, 0, 0)
0L
}
}
}
scala 函数柯里化
scala> def sum(x:Int)(y:Int)=x+y
sum: (x: Int)(y: Int)Int
scala> val second = sum(1)_
second: Int => Int =
scala> second(2)
res1: Int = 3
柯里化函数执行时，分解为两个函数执行，步骤与下面的方法调用过程类似
scala> def first(x:Int) = (y:Int)=>x+y
first: (x: Int)Int => Int
scala> val second = first(1)
second: Int => Int =
scala> second(2)
res4: Int = 3
scala> val r1 = sc.parallelize(Array(1,2,3,4))
r1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at :27
scala> val r2 = sc.parallelize(Array(5,6,7,8))
r2: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at :27
scala> val r = sc.parallelize(Array((1,2,r1),(2,3,r2)))
r: org.apache.spark.rdd.RDD[(Int, Int, org.apache.spark.rdd.RDD[Int])] = ParallelCollectionRDD[2] at parallelize at :31
scala> r.first
res0: (Int, Int, org.apache.spark.rdd.RDD[Int]) = (1,2,ParallelCollectionRDD[0] at )
scala> r
res1: org.apache.spark.rdd.RDD[(Int, Int, org.apache.spark.rdd.RDD[Int])] = ParallelCollectionRDD[2] at parallelize at :31
[error] type mismatch;
[error] found : Array[?B]
[error] required: scala.collection.GenTraversableOnce[?]
[error] Note that implicit conversions are not applicable because they are ambiguous:
[error] both method booleanArrayOps in object Predef of type (xs: Array[Boolean])scala.collection.mutable.ArrayOps[Boolean]
[error] and method byteArrayOps in object Predef of type (xs: Array[Byte])scala.collection.mutable.ArrayOps[Byte]
[error] are possible conversion functions from Array[?B] to scala.collection.GenTraversableOnce[?]
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 6 s, completed 2017-10-27 11:40:59
org.apache.spark.SparkException: RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(x => rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063.
SPARK-5063 relates to better error messages when trying to nest RDD operations, which is not supported.
It’s a usability issue, not a functional one. The root cause is the nesting of RDD operations and the solution is to break that up.
Here we are trying a join of dRDD and mRDD. If the size of mRDD is large, a rdd.join would be the recommended way otherwise, if mRDD is small, i.e. fits in memory of each executor, we could collect it, broadcast it and do a ‘map-side’ join.
WARN HiveConf: HiveConf of name hive.stats.map.parallelism does not exist
hive.stats.map.parallelism
Default Value: 1
Added In: Hive 0.13 with HIVE-5369
Removed In: Hive 0.14 with HIVE-7156
The Hive/Tez optimizer estimates the data size flowing through each of the operators. For the GROUPBY operator, to accurately compute the data size map-side parallelism needs to be known. By default, this value is set to 1 since the optimizer is not aware of the number of mappers during compile-time. This Hive configuration property can be used to specify the number of mappers for data size computation of the GROUPBY operator. (This configuration property was removed in release 0.14.0.)
1.Only for parquet tables. set spark.sql.hive.convertMetastoreParquet=true;
2.That is to say the partition exists in hive table, but the related path was removed manually, and spark 1.6.1 would throw exception
3.This exception is caused by “the HiveContext cache the metadata of the table” ;When you delete the hdfs, you must refresh table by using sqlContext.refreshTable(tableName);
但是从结果来看，在spark UI中，依然产生了很多SQL句柄，怀疑是val sqlContext = new HiveContext(sc)每次申明一个sqlCcontext都看做是一个sql句柄。因此，只在程序入口申请一个sqlContext然后传递给下面的函数，观察spark UI,sql句柄果然少了下来。
对于创建的sqlContext，其生命周期的问题需要以后关注一下
The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-
针对这个异常，网上很多方法都是修改文件的权限，但是个人总觉得不是很对，然后发现了一个新的说法
Don’t do chmod (777)… The correct is (733):
Hive 0.14.0 and later: HDFS root scratch directory for Hive jobs, which gets created with write all (733) permission. For each connecting user, an HDFS scratch directory hive.exec.scratchdir/iscreatedwith{hive.scratch.dir.permission}.
只需要在调用该对象合适(比如下列的setStyles)的方法后让该方法返回该对象（通过this 因为一旦一个函数称为一个对象方法的话那么在这个方法内部this（结合下面的setStyles）指向这个对象）
function create(type){
var element=document.createElement(type);
//this=element;
JAX-WS
SOAP Version 1.2 Part 0: Primer (Second Edition)
SOAP Version 1.2 Part 1: Messaging Framework (Second Edition)
SOAP Version 1.2 Part 2: Adjuncts (Second Edition)
Which style of WSDL
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml
