TaskCommitDenied 警告

1、在开启了spark.speculation参数后宝如下警告：

参数：spark.speculation       true

18/04/26 19:29:14 WARN scheduler.TaskSetManager: Lost task 1930.1 in stage 4.0 (TID 3793, 192.168.200.175): TaskCommitDenied (Driver denied 
task commit) for job: 4, partition: 1930, attemptNumber: 1
18/04/26 19:29:14 INFO scheduler.TaskSetManager: Task 1930.1 in stage 4.0 (TID 3793) failed, but another instance of the task has already su
cceeded, so not re-queuing the task to be re-executed.
18/04/26 19:29:14 INFO scheduler.OutputCommitCoordinator: Task was denied committing, stage: 4, partition: 1930, attempt: 1
18/04/26 19:29:14 WARN scheduler.TaskSetManager: Lost task 1933.1 in stage 4.0 (TID 3794, 192.168.200.175): TaskCommitDenied (Driver denied 
task commit) for job: 4, partition: 1933, attemptNumber: 1
18/04/26 19:29:14 INFO scheduler.TaskSetManager: Task 1933.1 in stage 4.0 (TID 3794) failed, but another instance of the task has already su
cceeded, so not re-queuing the task to be re-executed.
18/04/26 19:29:14 INFO scheduler.OutputCommitCoordinator: Task was denied committing, stage: 4, partition: 1933, attempt: 1
18/04/26 19:29:14 INFO scheduler.TaskSetManager: Finished task 1999.0 in stage 4.0 (TID 3791) in 1358 ms on 192.168.200.165 (1984/2000)
18/04/26 19:29:14 INFO scheduler.TaskSetManager: Finished task 1998.0 in stage 4.0 (TID 3790) in 1358 ms on 192.168.200.165 (1985/2000)
18/04/26 19:29:14 WARN scheduler.TaskSetManager: Lost task 1945.1 in stage 4.0 (TID 3796, 192.168.200.175): TaskCommitDenied (Driver denied 
task commit) for job: 4, partition: 1945, attemptNumber: 1
18/04/26 19:29:14 INFO scheduler.TaskSetManager: Task 1945.1 in stage 4.0 (TID 3796) failed, but another instance of the task has already su
cceeded, so not re-queuing the task to be re-executed.
18/04/26 19:29:14 INFO scheduler.OutputCommitCoordinator: Task was denied committing, stage: 4, partition: 1945, attempt: 1
18/04/26 19:29:14 WARN scheduler.TaskSetManager: Lost task 1947.1 in stage 4.0 (TID 3798, 192.168.200.165): TaskCommitDenied (Driver denied 
task commit) for job: 4, partition: 1947, attemptNumber: 1
18/04/26 19:29:14 INFO scheduler.TaskSetManager: Task 1947.1 in stage 4.0 (TID 3798) failed, but another instance of the task has already su
cceeded, so not re-queuing the task to be re-executed.
18/04/26 19:29:14 INFO scheduler.OutputCommitCoordinator: Task was denied committing, stage: 4, partition: 1947, attempt: 1
18/04/26 19:29:14 WARN scheduler.TaskSetManager: Lost task 1946.1 in stage 4.0 (TID 3797, 192.168.200.166): TaskCommitDenied (Driver denied 
task commit) for job: 4, partition: 1946, attemptNumber: 1
18/04/26 19:29:14 INFO scheduler.TaskSetManager: Task 1946.1 in stage 4.0 (TID 3797) failed, but another instance of the task has already su
cceeded, so not re-queuing the task to be re-executed.

2、解析：

开启speculation参数后，运行较慢的task会在其他executor上同时再启动一个相同的task,如果其中一个task执行完毕，相同的另一个task就会被禁止提交。因此产生了这个WARN。

这个WARN是因为task提交commit被driver拒绝引发，这个错误不会被统计在stage的failure中，这样做的目的是防止你看到一些具有欺骗性的提示。