import org.apache.spark.SparkConf
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
/**
* @author clebeg
*/
object FirstTry {
def main(args: Array[String]): Unit = {
val conf = new SparkConf
conf.setMaster("spark://yourip:7077")
conf.set("spark.app.name", "first-tryspark")
val sc = new SparkContext(conf)
val rawblocks = sc.textFile("hdfs://yourip:9000/user/hadoop/linkage")
println(rawblocks.first)
}
}
5 部分错误汇总
大部分问题上面已经提到，这里不多说，下面提几个主要的问题：
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 分析问题：点开运行ID对应的运行日志发现下面的错误：
15/10/10 08:49:01 INFO executor.CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
15/10/10 08:49:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/10/10 08:49:02 INFO spark.SecurityManager: Changing view acls to: hadoop,Administrator
15/10/10 08:49:02 INFO spark.SecurityManager: Changing modify acls to: hadoop,Administrator
15/10/10 08:49:02 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop, Administrator); users with modify permissions: Set(hadoop, Administrator)
15/10/10 08:49:02 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/10/10 08:49:02 INFO Remoting: Starting remoting
15/10/10 08:49:02 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:58708]
15/10/10 08:49:02 INFO util.Utils: Successfully started service 'driverPropsFetcher' on port 58708.
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1643)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:65)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:146)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:245)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [120 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:107)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:97)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:159)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:65)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
... 4 more
15/10/10 08:51:02 INFO util.Utils: Shutdown hook called
可是为什么再一次失败之后就变成直接跳过那个要求
enter full pathname of java.exe的界面
这个java.exe是你的Oracle 11g安装目录中例如：【F:\app\chen\product\11.2.0\dbhome_1\jdk\jre\bin】下的java.exe 。不是你的电脑安装的java jdk下的java.exe！
注意第一次，使用SQL D
一、rank()/dense_rank() over(partition by ...order by ...)
现在客户有这样一个需求，查询每个部门工资最高的雇员的信息，相信有一定oracle应用知识的同学都能写出下面的SQL语句：
select e.ename, e.job, e.sal, e.deptno
from scott.emp e,
(se
ReadTimeoutHandler的实现思路：
开启一个定时任务，如果在指定时间内没有接收到消息，则抛出ReadTimeoutException
这个异常的捕获，在开发中，交给跟在ReadTimeoutHandler后面的ChannelHandler，例如
private final ChannelHandler timeoutHandler =
new ReadTim
Chrome Web Store 安装地址： https://chrome.google.com/webstore/detail/replace-google-cdn/kpampjmfiopfpkkepbllemkibefkiice
由于众所周知的原因，只需替换一个域名就可以继续使用Google提供的前端公共库了。 同样，通过script标记引用这些资源，让网站访问速度瞬间提速吧
2.1 默认方法(default method)
java8引入了一个default medthod;
用来扩展已有的接口,在对已有接口的使用不产生任何影响的情况下,添加扩展
使用default关键字
Spring 4.2支持加载在默认方法里声明的bean
2.2
将要被声明成bean的类
public class DemoService {
