kaldi做aishell的nnet3训练耗时44个小时

kaldi做aishell，其中倒数第3步是local/nnet3/run_tdnn.sh，一共花了44个小时，

主机配置32G内存，E3 1231 V3，2个gtx950 2G的显卡，机械硬盘。

其中只有steps/nnet3/train_dnn.py这个步骤需要用到GPU，也是最耗时的步骤，花了38个小时。对显存容量要求不高。

注意一定要把local/nnet3/run_tdnn.sh中第96行的--use-gpu true修改为--use-gpu wait，因为在nvidia显卡EXCLUSIVE_PROCESS模式下，最后需要12块显卡才能支持，因为最后并发数量是12。当修改为--use-gpu wait，那么并发作业拿不到显卡，程序会等待，而不是直接报错。

local/nnet3/run_tdnn.sh完整的运行日志如下

local/nnet3/run_ivector_common.sh: preparing directory for low-resolution speed-perturbed data (for alignment)
utils/data/perturb_data_dir_speed_3way.sh: making sure the utt2dur and the reco2dur files are present
... in data/train, because obtaining it after speed-perturbing
... would be very slow, and you might need them.
utils/data/get_utt2dur.sh: data/train/utt2dur already exists with the expected length.  We won't recompute it.
utils/data/get_reco2dur.sh: data/train/reco2dur already exists with the expected length.  We won't recompute it.
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train, in data/train_sp_speed0.9
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_speed0.9
utils/data/perturb_data_dir_speed.sh: generated speed-perturbed version of data in data/train, in data/train_sp_speed1.1
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_speed1.1
utils/data/combine_data.sh data/train_sp data/train data/train_sp_speed0.9 data/train_sp_speed1.1
utils/data/combine_data.sh: combined utt2uniq
utils/data/combine_data.sh [info]: not combining segments as it does not exist
utils/data/combine_data.sh: combined utt2spk
utils/data/combine_data.sh [info]: not combining utt2lang as it does not exist
utils/data/combine_data.sh: combined utt2dur
utils/data/combine_data.sh: combined reco2dur
utils/data/combine_data.sh [info]: **not combining feats.scp as it does not exist everywhere**
utils/data/combine_data.sh: combined text
utils/data/combine_data.sh [info]: **not combining cmvn.scp as it does not exist everywhere**
utils/data/combine_data.sh [info]: not combining vad.scp as it does not exist
utils/data/combine_data.sh [info]: not combining reco2file_and_channel as it does not exist
utils/data/combine_data.sh: combined wav.scp
utils/data/combine_data.sh [info]: not combining spk2gender as it does not exist
fix_data_dir.sh: kept all 360294 utterances.
fix_data_dir.sh: old files are kept in data/train_sp/.backup
utils/data/perturb_data_dir_speed_3way.sh: generated 3-way speed-perturbed version of data in data/train, in data/train_sp
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp
local/nnet3/run_ivector_common.sh: making MFCC features for low-resolution speed-perturbed data
steps/make_mfcc_pitch.sh --cmd run.pl --mem 24G --nj 70 data/train_sp exp/make_mfcc/train_sp mfcc_perturbed
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp
steps/make_mfcc_pitch.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
Succeeded creating MFCC & Pitch features for train_sp
steps/compute_cmvn_stats.sh data/train_sp exp/make_mfcc/train_sp mfcc_perturbed
Succeeded creating CMVN stats for train_sp
fix_data_dir.sh: kept all 360294 utterances.
fix_data_dir.sh: old files are kept in data/train_sp/.backup
local/nnet3/run_ivector_common.sh: aligning with the perturbed low-resolution data
steps/align_fmllr.sh --nj 30 --cmd run.pl --mem 24G data/train_sp data/lang exp/tri5a exp/tri5a_sp_ali
steps/align_fmllr.sh: feature type is lda
steps/align_fmllr.sh: compiling training graphs
steps/align_fmllr.sh: aligning data in data/train_sp using exp/tri5a/final.alimdl and speaker-independent features.
steps/align_fmllr.sh: computing fMLLR transforms
steps/align_fmllr.sh: doing final alignment.
steps/align_fmllr.sh: done aligning data.
steps/diagnostic/analyze_alignments.sh --cmd run.pl --mem 24G data/lang exp/tri5a_sp_ali
steps/diagnostic/analyze_alignments.sh: see stats in exp/tri5a_sp_ali/log/analyze_alignments.log
413 warnings in exp/tri5a_sp_ali/log/align_pass1.*.log
382 warnings in exp/tri5a_sp_ali/log/align_pass2.*.log
2 warnings in exp/tri5a_sp_ali/log/fmllr.*.log
local/nnet3/run_ivector_common.sh: creating high-resolution MFCC features
utils/copy_data_dir.sh: copied data from data/train_sp to data/train_sp_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_hires
utils/copy_data_dir.sh: copied data from data/dev to data/dev_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_hires
utils/copy_data_dir.sh: copied data from data/test to data/test_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/test_hires
utils/data/perturb_data_dir_volume.sh: data/train_sp_hires/feats.scp exists; moving it to data/train_sp_hires/.backup/ as it wouldn't be valid any more.
utils/data/perturb_data_dir_volume.sh: added volume perturbation to the data in data/train_sp_hires
steps/make_mfcc_pitch.sh --nj 10 --mfcc-config conf/mfcc_hires.conf --cmd run.pl --mem 24G data/train_sp_hires exp/make_hires/train_sp mfcc_perturbed_hires
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_hires
steps/make_mfcc_pitch.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
Succeeded creating MFCC & Pitch features for train_sp_hires
steps/compute_cmvn_stats.sh data/train_sp_hires exp/make_hires/train_sp mfcc_perturbed_hires
Succeeded creating CMVN stats for train_sp_hires
fix_data_dir.sh: kept all 360294 utterances.
fix_data_dir.sh: old files are kept in data/train_sp_hires/.backup
utils/copy_data_dir.sh: copied data from data/train_sp_hires to data/train_sp_hires_nopitch
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_hires_nopitch
utils/data/limit_feature_dim.sh: warning: removing data/train_sp_hires_nopitch/cmvn.cp, you will have to regenerate it from the features.
utils/validate_data_dir.sh: Successfully validated data-directory data/train_sp_hires_nopitch
steps/compute_cmvn_stats.sh data/train_sp_hires_nopitch exp/make_hires/train_sp mfcc_perturbed_hires
Succeeded creating CMVN stats for train_sp_hires_nopitch
steps/make_mfcc_pitch.sh --nj 10 --mfcc-config conf/mfcc_hires.conf --cmd run.pl --mem 24G data/dev_hires exp/make_hires/dev mfcc_perturbed_hires
steps/make_mfcc_pitch.sh: moving data/dev_hires/feats.scp to data/dev_hires/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_hires
steps/make_mfcc_pitch.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
Succeeded creating MFCC & Pitch features for dev_hires
steps/compute_cmvn_stats.sh data/dev_hires exp/make_hires/dev mfcc_perturbed_hires
Succeeded creating CMVN stats for dev_hires
fix_data_dir.sh: kept all 14326 utterances.
fix_data_dir.sh: old files are kept in data/dev_hires/.backup
utils/copy_data_dir.sh: copied data from data/dev_hires to data/dev_hires_nopitch
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_hires_nopitch
utils/data/limit_feature_dim.sh: warning: removing data/dev_hires_nopitch/cmvn.cp, you will have to regenerate it from the features.
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_hires_nopitch
steps/compute_cmvn_stats.sh data/dev_hires_nopitch exp/make_hires/dev mfcc_perturbed_hires
Succeeded creating CMVN stats for dev_hires_nopitch
steps/make_mfcc_pitch.sh --nj 10 --mfcc-config conf/mfcc_hires.conf --cmd run.pl --mem 24G data/test_hires exp/make_hires/test mfcc_perturbed_hires
steps/make_mfcc_pitch.sh: moving data/test_hires/feats.scp to data/test_hires/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/test_hires
steps/make_mfcc_pitch.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
Succeeded creating MFCC & Pitch features for test_hires
steps/compute_cmvn_stats.sh data/test_hires exp/make_hires/test mfcc_perturbed_hires
Succeeded creating CMVN stats for test_hires
fix_data_dir.sh: kept all 7176 utterances.
fix_data_dir.sh: old files are kept in data/test_hires/.backup
utils/copy_data_dir.sh: copied data from data/test_hires to data/test_hires_nopitch
utils/validate_data_dir.sh: Successfully validated data-directory data/test_hires_nopitch
utils/data/limit_feature_dim.sh: warning: removing data/test_hires_nopitch/cmvn.cp, you will have to regenerate it from the features.
utils/validate_data_dir.sh: Successfully validated data-directory data/test_hires_nopitch
steps/compute_cmvn_stats.sh data/test_hires_nopitch exp/make_hires/test mfcc_perturbed_hires
Succeeded creating CMVN stats for test_hires_nopitch
local/nnet3/run_ivector_common.sh: computing a subset of data to train the diagonal UBM.
utils/data/subset_data_dir.sh: reducing #utt from 360294 to 90073
local/nnet3/run_ivector_common.sh: computing a PCA transform from the hires data.
steps/online/nnet2/get_pca_transform.sh --cmd run.pl --mem 24G --splice-opts --left-context=3 --right-context=3 --max-utts 10000 --subsample 2 exp/nnet3/diag_ubm/train_sp_hires_nopitch_subset exp/nnet3/pca_transform
Done estimating PCA transform in exp/nnet3/pca_transform
local/nnet3/run_ivector_common.sh: training the diagonal UBM.
steps/online/nnet2/train_diag_ubm.sh --cmd run.pl --mem 24G --nj 30 --num-frames 700000 --num-threads 8 exp/nnet3/diag_ubm/train_sp_hires_nopitch_subset 512 exp/nnet3/pca_transform exp/nnet3/diag_ubm
steps/online/nnet2/train_diag_ubm.sh: Directory exp/nnet3/diag_ubm already exists. Backing up diagonal UBM in exp/nnet3/diag_ubm/backup.sth
steps/online/nnet2/train_diag_ubm.sh: initializing model from E-M in memory, 
steps/online/nnet2/train_diag_ubm.sh: starting from 256 Gaussians, reaching 512;
steps/online/nnet2/train_diag_ubm.sh: for 20 iterations, using at most 700000 frames of data
Getting Gaussian-selection info
steps/online/nnet2/train_diag_ubm.sh: will train for 4 iterations, in parallel over
steps/online/nnet2/train_diag_ubm.sh: 30 machines, parallelized with 'run.pl --mem 24G'
steps/online/nnet2/train_diag_ubm.sh: Training pass 0
steps/online/nnet2/train_diag_ubm.sh: Training pass 1
steps/online/nnet2/train_diag_ubm.sh: Training pass 2
steps/online/nnet2/train_diag_ubm.sh: Training pass 3
local/nnet3/run_ivector_common.sh: training the iVector extractor
steps/online/nnet2/train_ivector_extractor.sh --cmd run.pl --mem 24G --nj 10 data/train_sp_hires_nopitch exp/nnet3/diag_ubm exp/nnet3/extractor
steps/online/nnet2/train_ivector_extractor.sh: Directory exp/nnet3/extractor already exists. Backing up iVector extractor in exp/nnet3/extractor/backup.Raq
steps/online/nnet2/train_ivector_extractor.sh: doing Gaussian selection and posterior computation
Accumulating stats (pass 0)
Summing accs (pass 0)
Updating model (pass 0)
Accumulating stats (pass 1)
Summing accs (pass 1)
Updating model (pass 1)
Accumulating stats (pass 2)
Summing accs (pass 2)
Updating model (pass 2)
Accumulating stats (pass 3)
Summing accs (pass 3)
Updating model (pass 3)
Accumulating stats (pass 4)
Summing accs (pass 4)
Updating model (pass 4)
Accumulating stats (pass 5)
Summing accs (pass 5)
Updating model (pass 5)
Accumulating stats (pass 6)
Summing accs (pass 6)
Updating model (pass 6)
Accumulating stats (pass 7)
Summing accs (pass 7)
Updating model (pass 7)
Accumulating stats (pass 8)
Summing accs (pass 8)
Updating model (pass 8)
Accumulating stats (pass 9)
Summing accs (pass 9)
Updating model (pass 9)
utils/data/modify_speaker_info.sh: copied data from data/train_sp_hires_nopitch to exp/nnet3/ivectors_train_sp/train_sp_sp_hires_nopitch_max2, number of speakers changed from 1020 to 180399
utils/validate_data_dir.sh: Successfully validated data-directory exp/nnet3/ivectors_train_sp/train_sp_sp_hires_nopitch_max2
steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 24G --nj 30 exp/nnet3/ivectors_train_sp/train_sp_sp_hires_nopitch_max2 exp/nnet3/extractor exp/nnet3/ivectors_train_sp
steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors
steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs
steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3/ivectors_train_sp using the extractor in exp/nnet3/extractor.
steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 24G --nj 8 data/dev_hires_nopitch exp/nnet3/extractor exp/nnet3/ivectors_dev
steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors
steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs
steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3/ivectors_dev using the extractor in exp/nnet3/extractor.
steps/online/nnet2/extract_ivectors_online.sh --cmd run.pl --mem 24G --nj 8 data/test_hires_nopitch exp/nnet3/extractor exp/nnet3/ivectors_test
steps/online/nnet2/extract_ivectors_online.sh: extracting iVectors
steps/online/nnet2/extract_ivectors_online.sh: combining iVectors across jobs
steps/online/nnet2/extract_ivectors_online.sh: done extracting (online) iVectors to exp/nnet3/ivectors_test using the extractor in exp/nnet3/extractor.
local/nnet3/run_tdnn.sh: creating neural net configs
tree-info exp/tri5a_sp_ali/tree 
steps/nnet3/xconfig_to_configs.py --xconfig-file exp/nnet3/tdnn_sp/configs/network.xconfig --config-dir exp/nnet3/tdnn_sp/configs/
nnet3-init exp/nnet3/tdnn_sp/configs//init.config exp/nnet3/tdnn_sp/configs//init.raw 
LOG (nnet3-init[5.5]:main():nnet3-init.cc:80) Initialized raw neural net and wrote it to exp/nnet3/tdnn_sp/configs//init.raw
nnet3-info exp/nnet3/tdnn_sp/configs//init.raw 
nnet3-init exp/nnet3/tdnn_sp/configs//ref.config exp/nnet3/tdnn_sp/configs//ref.raw 
LOG (nnet3-init[5.5]:main():nnet3-init.cc:80) Initialized raw neural net and wrote it to exp/nnet3/tdnn_sp/configs//ref.raw
nnet3-info exp/nnet3/tdnn_sp/configs//ref.raw 
nnet3-init exp/nnet3/tdnn_sp/configs//ref.config exp/nnet3/tdnn_sp/configs//ref.raw 
LOG (nnet3-init[5.5]:main():nnet3-init.cc:80) Initialized raw neural net and wrote it to exp/nnet3/tdnn_sp/configs//ref.raw
2019-04-05 22:08:04,965 [steps/nnet3/train_dnn.py:36 -  - INFO ] Starting DNN trainer (train_dnn.py)
steps/nnet3/train_dnn.py --stage=-10 --cmd=run.pl --mem 24G --feat.online-ivector-dir exp/nnet3/ivectors_train_sp --feat.cmvn-opts=--norm-means=false --norm-vars=false --trainer.num-epochs 4 --trainer.optimization.num-jobs-initial 2 --trainer.optimization.num-jobs-final 12 --trainer.optimization.initial-effective-lrate 0.0015 --trainer.optimization.final-effective-lrate 0.00015 --egs.dir  --cleanup.remove-egs true --cleanup.preserve-model-interval 500 --use-gpu wait --feat-dir=data/train_sp_hires --ali-dir exp/tri5a_sp_ali --lang data/lang --reporting.email= --dir=exp/nnet3/tdnn_sp
['steps/nnet3/train_dnn.py', '--stage=-10', '--cmd=run.pl --mem 24G', '--feat.online-ivector-dir', 'exp/nnet3/ivectors_train_sp', '--feat.cmvn-opts=--norm-means=false --norm-vars=false', '--trainer.num-epochs', '4', '--trainer.optimization.num-jobs-initial', '2', '--trainer.optimization.num-jobs-final', '12', '--trainer.optimization.initial-effective-lrate', '0.0015', '--trainer.optimization.final-effective-lrate', '0.00015', '--egs.dir', '', '--cleanup.remove-egs', 'true', '--cleanup.preserve-model-interval', '500', '--use-gpu', 'wait', '--feat-dir=data/train_sp_hires', '--ali-dir', 'exp/tri5a_sp_ali', '--lang', 'data/lang', '--reporting.email=', '--dir=exp/nnet3/tdnn_sp']
2019-04-05 22:08:04,970 [steps/nnet3/train_dnn.py:177 - train - INFO ] Arguments for the experiment
{'ali_dir': 'exp/tri5a_sp_ali',
 'backstitch_training_interval': 1,
 'backstitch_training_scale': 0.0,
 'cleanup': True,
 'cmvn_opts': '--norm-means=false --norm-vars=false',
 'combine_sum_to_one_penalty': 0.0,
 'command': 'run.pl --mem 24G',
 'compute_per_dim_accuracy': False,
 'dir': 'exp/nnet3/tdnn_sp',
 'do_final_combination': True,
 'dropout_schedule': None,
 'egs_command': None,
 'egs_dir': None,
 'egs_opts': None,
 'egs_stage': 0,
 'email': None,
 'exit_stage': None,
 'feat_dir': 'data/train_sp_hires',
 'final_effective_lrate': 0.00015,
 'frames_per_eg': 8,
 'initial_effective_lrate': 0.0015,
 'input_model': None,
 'lang': 'data/lang',
 'max_lda_jobs': 10,
 'max_models_combine': 20,
 'max_objective_evaluations': 30,
 'max_param_change': 2.0,
 'minibatch_size': '512',
 'momentum': 0.0,
 'num_epochs': 4.0,
 'num_jobs_compute_prior': 10,
 'num_jobs_final': 12,
 'num_jobs_initial': 2,
 'online_ivector_dir': 'exp/nnet3/ivectors_train_sp',
 'preserve_model_interval': 500,
 'presoftmax_prior_scale_power': -0.25,
 'prior_subset_size': 20000,
 'proportional_shrink': 0.0,
 'rand_prune': 4.0,
 'remove_egs': True,
 'reporting_interval': 0.1,
 'samples_per_iter': 400000,
 'shuffle_buffer_size': 5000,
 'srand': 0,
 'stage': -10,
 'train_opts': [],
 'use_gpu': 'wait'}
2019-04-05 22:08:05,016 [steps/nnet3/train_dnn.py:227 - train - INFO ] Initializing a basic network for estimating preconditioning matrix
2019-04-05 22:08:05,063 [steps/nnet3/train_dnn.py:237 - train - INFO ] Generating egs
steps/nnet3/get_egs.sh --cmd run.pl --mem 24G --cmvn-opts --norm-means=false --norm-vars=false --online-ivector-dir exp/nnet3/ivectors_train_sp --left-context 16 --right-context 12 --left-context-initial -1 --right-context-final -1 --stage 0 --samples-per-iter 400000 --frames-per-eg 8 --srand 0 data/train_sp_hires exp/tri5a_sp_ali exp/nnet3/tdnn_sp/egs
File data/train_sp_hires/utt2uniq exists, so augmenting valid_uttlist to
include all perturbed versions of the same 'real' utterances.
steps/nnet3/get_egs.sh: creating egs.  To ensure they are not deleted later you can do:  touch exp/nnet3/tdnn_sp/egs/.nodelete
steps/nnet3/get_egs.sh: feature type is raw
feat-to-dim scp:exp/nnet3/ivectors_train_sp/ivector_online.scp - 
steps/nnet3/get_egs.sh: working out number of frames of training data
steps/nnet3/get_egs.sh: working out feature dim
steps/nnet3/get_egs.sh: creating 52 archives, each with 394272 egs, with
steps/nnet3/get_egs.sh:   8 labels per example, and (left,right) context = (16,12)
steps/nnet3/get_egs.sh: copying data alignments
copy-int-vector ark:- ark,scp:exp/nnet3/tdnn_sp/egs/ali.ark,exp/nnet3/tdnn_sp/egs/ali.scp 
LOG (copy-int-vector[5.5]:main():copy-int-vector.cc:83) Copied 360291 vectors of int32.
steps/nnet3/get_egs.sh: Getting validation and training subset examples.
steps/nnet3/get_egs.sh: ... extracting validation and training-subset alignments.
... Getting subsets of validation examples for diagnostics and combination.
steps/nnet3/get_egs.sh: Generating training examples on disk
steps/nnet3/get_egs.sh: recombining and shuffling order of archives on disk
steps/nnet3/get_egs.sh: removing temporary archives
steps/nnet3/get_egs.sh: removing temporary alignments
steps/nnet3/get_egs.sh: Finished preparing training examples
2019-04-05 22:49:03,244 [steps/nnet3/train_dnn.py:275 - train - INFO ] Computing the preconditioning matrix for input features
2019-04-05 22:53:51,530 [steps/nnet3/train_dnn.py:286 - train - INFO ] Computing initial vector for FixedScaleComponent before softmax, using priors^-0.25 and rescaling to average 1
2019-04-05 22:54:12,361 [steps/nnet3/train_dnn.py:293 - train - INFO ] Preparing the initial acoustic model.
2019-04-05 22:54:19,160 [steps/nnet3/train_dnn.py:318 - train - INFO ] Training will run for 4.0 epochs = 237 iterations
2019-04-05 22:54:19,160 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 0/236    Epoch: 0.00/4.0 (0.0% complete)    lr: 0.003000    
2019-04-05 22:57:07,706 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 1/236    Epoch: 0.00/4.0 (0.1% complete)    lr: 0.002992    
2019-04-05 22:59:52,272 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 2/236    Epoch: 0.01/4.0 (0.2% complete)    lr: 0.002983    
2019-04-05 23:02:37,357 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 3/236    Epoch: 0.01/4.0 (0.4% complete)    lr: 0.002975    
2019-04-05 23:05:25,685 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 4/236    Epoch: 0.02/4.0 (0.5% complete)    lr: 0.002967    
2019-04-05 23:08:09,379 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 5/236    Epoch: 0.02/4.0 (0.6% complete)    lr: 0.002959    
2019-04-05 23:10:53,866 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 6/236    Epoch: 0.03/4.0 (0.7% complete)    lr: 0.002951    
2019-04-05 23:13:38,331 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 7/236    Epoch: 0.03/4.0 (0.8% complete)    lr: 0.002942    
2019-04-05 23:16:23,374 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 8/236    Epoch: 0.04/4.0 (1.0% complete)    lr: 0.002934    
2019-04-05 23:19:10,803 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 9/236    Epoch: 0.04/4.0 (1.1% complete)    lr: 0.002926    
2019-04-05 23:21:56,003 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 10/236    Epoch: 0.05/4.0 (1.2% complete)    lr: 0.002918    
2019-04-05 23:24:42,471 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 11/236    Epoch: 0.05/4.0 (1.3% complete)    lr: 0.002910    
2019-04-05 23:27:29,937 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 12/236    Epoch: 0.06/4.0 (1.4% complete)    lr: 0.004353    
2019-04-05 23:32:39,287 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 13/236    Epoch: 0.06/4.0 (1.6% complete)    lr: 0.004335    
2019-04-05 23:37:48,546 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 14/236    Epoch: 0.07/4.0 (1.8% complete)    lr: 0.004317    
2019-04-05 23:42:57,747 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 15/236    Epoch: 0.08/4.0 (2.0% complete)    lr: 0.004299    
2019-04-05 23:48:06,678 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 16/236    Epoch: 0.09/4.0 (2.2% complete)    lr: 0.004281    
2019-04-05 23:53:21,294 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 17/236    Epoch: 0.09/4.0 (2.3% complete)    lr: 0.004264    
2019-04-05 23:58:35,909 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 18/236    Epoch: 0.10/4.0 (2.5% complete)    lr: 0.004246    
2019-04-06 00:03:50,992 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 19/236    Epoch: 0.11/4.0 (2.7% complete)    lr: 0.004228    
2019-04-06 00:09:06,003 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 20/236    Epoch: 0.12/4.0 (2.9% complete)    lr: 0.004211    
2019-04-06 00:14:23,242 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 21/236    Epoch: 0.12/4.0 (3.1% complete)    lr: 0.004193    
2019-04-06 00:19:38,874 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 22/236    Epoch: 0.13/4.0 (3.2% complete)    lr: 0.004176    
2019-04-06 00:24:54,063 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 23/236    Epoch: 0.14/4.0 (3.4% complete)    lr: 0.004159    
2019-04-06 00:30:03,795 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 24/236    Epoch: 0.14/4.0 (3.6% complete)    lr: 0.004141    
2019-04-06 00:35:14,407 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 25/236    Epoch: 0.15/4.0 (3.8% complete)    lr: 0.004124    
2019-04-06 00:40:24,596 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 26/236    Epoch: 0.16/4.0 (4.0% complete)    lr: 0.004107    
2019-04-06 00:45:35,149 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 27/236    Epoch: 0.17/4.0 (4.1% complete)    lr: 0.004090    
2019-04-06 00:50:50,207 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 28/236    Epoch: 0.17/4.0 (4.3% complete)    lr: 0.004073    
2019-04-06 00:56:05,661 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 29/236    Epoch: 0.18/4.0 (4.5% complete)    lr: 0.004056    
2019-04-06 01:01:20,715 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 30/236    Epoch: 0.19/4.0 (4.7% complete)    lr: 0.004040    
2019-04-06 01:06:36,168 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 31/236    Epoch: 0.19/4.0 (4.9% complete)    lr: 0.004023    
2019-04-06 01:11:46,296 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 32/236    Epoch: 0.20/4.0 (5.0% complete)    lr: 0.004006    
2019-04-06 01:16:56,093 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 33/236    Epoch: 0.21/4.0 (5.2% complete)    lr: 0.003990    
2019-04-06 01:22:11,559 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 34/236    Epoch: 0.22/4.0 (5.4% complete)    lr: 0.003973    
2019-04-06 01:27:21,740 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 35/236    Epoch: 0.22/4.0 (5.6% complete)    lr: 0.003957    
2019-04-06 01:32:32,328 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 36/236    Epoch: 0.23/4.0 (5.8% complete)    lr: 0.005254    
2019-04-06 01:37:57,318 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 37/236    Epoch: 0.24/4.0 (6.0% complete)    lr: 0.005225    
2019-04-06 01:43:26,665 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 38/236    Epoch: 0.25/4.0 (6.2% complete)    lr: 0.005196    
2019-04-06 01:48:55,859 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 39/236    Epoch: 0.26/4.0 (6.5% complete)    lr: 0.005167    
2019-04-06 01:54:24,345 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 40/236    Epoch: 0.27/4.0 (6.7% complete)    lr: 0.005139    
2019-04-06 02:00:00,250 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 41/236    Epoch: 0.28/4.0 (7.0% complete)    lr: 0.005110    
2019-04-06 02:05:28,891 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 42/236    Epoch: 0.29/4.0 (7.2% complete)    lr: 0.005082    
2019-04-06 02:10:57,812 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 43/236    Epoch: 0.30/4.0 (7.5% complete)    lr: 0.005054    
2019-04-06 02:16:26,757 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 44/236    Epoch: 0.31/4.0 (7.7% complete)    lr: 0.005026    
2019-04-06 02:21:56,025 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 45/236    Epoch: 0.32/4.0 (7.9% complete)    lr: 0.004998    
2019-04-06 02:27:24,615 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 46/236    Epoch: 0.33/4.0 (8.2% complete)    lr: 0.004971    
2019-04-06 02:32:52,893 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 47/236    Epoch: 0.34/4.0 (8.4% complete)    lr: 0.004943    
2019-04-06 02:38:21,770 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 48/236    Epoch: 0.35/4.0 (8.7% complete)    lr: 0.004916    
2019-04-06 02:43:50,027 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 49/236    Epoch: 0.36/4.0 (8.9% complete)    lr: 0.004889    
2019-04-06 02:49:19,210 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 50/236    Epoch: 0.37/4.0 (9.1% complete)    lr: 0.004862    
2019-04-06 02:54:48,616 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 51/236    Epoch: 0.38/4.0 (9.4% complete)    lr: 0.004835    
2019-04-06 03:00:17,559 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 52/236    Epoch: 0.38/4.0 (9.6% complete)    lr: 0.004808    
2019-04-06 03:05:46,016 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 53/236    Epoch: 0.39/4.0 (9.9% complete)    lr: 0.004782    
2019-04-06 03:11:14,678 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 54/236    Epoch: 0.40/4.0 (10.1% complete)    lr: 0.004755    
2019-04-06 03:16:43,340 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 55/236    Epoch: 0.41/4.0 (10.3% complete)    lr: 0.004729    
2019-04-06 03:22:12,294 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 56/236    Epoch: 0.42/4.0 (10.6% complete)    lr: 0.004703    
2019-04-06 03:27:40,942 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 57/236    Epoch: 0.43/4.0 (10.8% complete)    lr: 0.004677    
2019-04-06 03:33:09,954 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 58/236    Epoch: 0.44/4.0 (11.1% complete)    lr: 0.004651    
2019-04-06 03:38:38,395 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 59/236    Epoch: 0.45/4.0 (11.3% complete)    lr: 0.004626    
2019-04-06 03:44:07,010 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 60/236    Epoch: 0.46/4.0 (11.5% complete)    lr: 0.005750    
2019-04-06 03:52:04,680 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 61/236    Epoch: 0.47/4.0 (11.8% complete)    lr: 0.005710    
2019-04-06 03:59:55,592 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 62/236    Epoch: 0.49/4.0 (12.1% complete)    lr: 0.005671    
2019-04-06 04:07:45,841 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 63/236    Epoch: 0.50/4.0 (12.4% complete)    lr: 0.005632    
2019-04-06 04:15:36,356 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 64/236    Epoch: 0.51/4.0 (12.7% complete)    lr: 0.005593    
2019-04-06 04:23:21,298 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 65/236    Epoch: 0.52/4.0 (13.0% complete)    lr: 0.005555    
2019-04-06 04:31:06,630 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 66/236    Epoch: 0.53/4.0 (13.3% complete)    lr: 0.005516    
2019-04-06 04:38:56,944 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 67/236    Epoch: 0.55/4.0 (13.6% complete)    lr: 0.005478    
2019-04-06 04:46:47,735 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 68/236    Epoch: 0.56/4.0 (13.9% complete)    lr: 0.005440    
2019-04-06 04:54:37,933 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 69/236    Epoch: 0.57/4.0 (14.2% complete)    lr: 0.005403    
2019-04-06 05:02:22,998 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 70/236    Epoch: 0.58/4.0 (14.5% complete)    lr: 0.005366    
2019-04-06 05:10:13,947 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 71/236    Epoch: 0.59/4.0 (14.8% complete)    lr: 0.005329    
2019-04-06 05:18:04,563 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 72/236    Epoch: 0.61/4.0 (15.1% complete)    lr: 0.005292    
2019-04-06 05:25:54,800 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 73/236    Epoch: 0.62/4.0 (15.4% complete)    lr: 0.005256    
2019-04-06 05:33:45,425 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 74/236    Epoch: 0.63/4.0 (15.7% complete)    lr: 0.005219    
2019-04-06 05:41:36,070 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 75/236    Epoch: 0.64/4.0 (16.0% complete)    lr: 0.005183    
2019-04-06 05:49:20,220 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 76/236    Epoch: 0.65/4.0 (16.3% complete)    lr: 0.005148    
2019-04-06 05:57:09,897 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 77/236    Epoch: 0.67/4.0 (16.6% complete)    lr: 0.005112    
2019-04-06 06:04:54,356 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 78/236    Epoch: 0.68/4.0 (16.9% complete)    lr: 0.005077    
2019-04-06 06:12:39,038 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 79/236    Epoch: 0.69/4.0 (17.2% complete)    lr: 0.005042    
2019-04-06 06:20:23,535 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 80/236    Epoch: 0.70/4.0 (17.5% complete)    lr: 0.005007    
2019-04-06 06:28:19,924 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 81/236    Epoch: 0.71/4.0 (17.8% complete)    lr: 0.004973    
2019-04-06 06:36:04,929 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 82/236    Epoch: 0.73/4.0 (18.1% complete)    lr: 0.004938    
2019-04-06 06:43:49,329 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 83/236    Epoch: 0.74/4.0 (18.4% complete)    lr: 0.005885    
2019-04-06 06:51:58,149 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 84/236    Epoch: 0.75/4.0 (18.8% complete)    lr: 0.005836    
2019-04-06 07:00:11,816 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 85/236    Epoch: 0.77/4.0 (19.2% complete)    lr: 0.005788    
2019-04-06 07:08:24,420 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 86/236    Epoch: 0.78/4.0 (19.5% complete)    lr: 0.005740    
2019-04-06 07:16:20,554 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 87/236    Epoch: 0.80/4.0 (19.9% complete)    lr: 0.005693    
2019-04-06 07:24:17,048 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 88/236    Epoch: 0.81/4.0 (20.3% complete)    lr: 0.005646    
2019-04-06 07:32:02,517 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 89/236    Epoch: 0.82/4.0 (20.6% complete)    lr: 0.005599    
2019-04-06 07:39:58,485 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 90/236    Epoch: 0.84/4.0 (21.0% complete)    lr: 0.005553    
2019-04-06 07:47:55,532 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 91/236    Epoch: 0.85/4.0 (21.3% complete)    lr: 0.005507    
2019-04-06 07:55:53,403 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 92/236    Epoch: 0.87/4.0 (21.7% complete)    lr: 0.005461    
2019-04-06 08:03:51,507 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 93/236    Epoch: 0.88/4.0 (22.1% complete)    lr: 0.005416    
2019-04-06 08:11:36,471 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 94/236    Epoch: 0.90/4.0 (22.4% complete)    lr: 0.005371    
2019-04-06 08:19:31,173 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 95/236    Epoch: 0.91/4.0 (22.8% complete)    lr: 0.005327    
2019-04-06 08:27:27,606 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 96/236    Epoch: 0.93/4.0 (23.1% complete)    lr: 0.005283    
2019-04-06 08:35:25,060 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 97/236    Epoch: 0.94/4.0 (23.5% complete)    lr: 0.005239    
2019-04-06 08:43:23,162 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 98/236    Epoch: 0.95/4.0 (23.9% complete)    lr: 0.005196    
2019-04-06 08:51:21,322 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 99/236    Epoch: 0.97/4.0 (24.2% complete)    lr: 0.005153    
2019-04-06 08:59:21,091 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 100/236    Epoch: 0.98/4.0 (24.6% complete)    lr: 0.005110    
2019-04-06 09:07:24,573 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 101/236    Epoch: 1.00/4.0 (24.9% complete)    lr: 0.005068    
2019-04-06 09:15:20,315 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 102/236    Epoch: 1.01/4.0 (25.3% complete)    lr: 0.005026    
2019-04-06 09:23:15,277 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 103/236    Epoch: 1.03/4.0 (25.7% complete)    lr: 0.004985    
2019-04-06 09:31:06,464 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 104/236    Epoch: 1.04/4.0 (26.0% complete)    lr: 0.004943    
2019-04-06 09:39:05,912 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 105/236    Epoch: 1.06/4.0 (26.4% complete)    lr: 0.004903    
2019-04-06 09:47:15,058 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 106/236    Epoch: 1.07/4.0 (26.7% complete)    lr: 0.004862    
2019-04-06 09:55:14,148 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 107/236    Epoch: 1.08/4.0 (27.1% complete)    lr: 0.005625    
2019-04-06 10:05:32,506 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 108/236    Epoch: 1.10/4.0 (27.5% complete)    lr: 0.005571    
2019-04-06 10:15:51,075 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 109/236    Epoch: 1.12/4.0 (27.9% complete)    lr: 0.005517    
2019-04-06 10:26:10,157 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 110/236    Epoch: 1.13/4.0 (28.4% complete)    lr: 0.005464    
2019-04-06 10:36:29,603 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 111/236    Epoch: 1.15/4.0 (28.8% complete)    lr: 0.005412    
2019-04-06 10:46:48,151 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 112/236    Epoch: 1.17/4.0 (29.2% complete)    lr: 0.005359    
2019-04-06 10:57:06,747 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 113/236    Epoch: 1.19/4.0 (29.6% complete)    lr: 0.005308    
2019-04-06 11:07:25,161 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 114/236    Epoch: 1.20/4.0 (30.0% complete)    lr: 0.005257    
2019-04-06 11:17:44,240 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 115/236    Epoch: 1.22/4.0 (30.5% complete)    lr: 0.005206    
2019-04-06 11:28:03,277 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 116/236    Epoch: 1.24/4.0 (30.9% complete)    lr: 0.005156    
2019-04-06 11:38:22,045 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 117/236    Epoch: 1.25/4.0 (31.3% complete)    lr: 0.005106    
2019-04-06 11:48:40,206 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 118/236    Epoch: 1.27/4.0 (31.7% complete)    lr: 0.005057    
2019-04-06 11:58:58,844 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 119/236    Epoch: 1.29/4.0 (32.2% complete)    lr: 0.005008    
2019-04-06 12:09:17,135 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 120/236    Epoch: 1.30/4.0 (32.6% complete)    lr: 0.004960    
2019-04-06 12:19:52,856 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 121/236    Epoch: 1.32/4.0 (33.0% complete)    lr: 0.004912    
2019-04-06 12:30:11,419 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 122/236    Epoch: 1.34/4.0 (33.4% complete)    lr: 0.004865    
2019-04-06 12:40:30,226 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 123/236    Epoch: 1.35/4.0 (33.8% complete)    lr: 0.004818    
2019-04-06 12:50:49,282 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 124/236    Epoch: 1.37/4.0 (34.3% complete)    lr: 0.004771    
2019-04-06 13:01:07,761 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 125/236    Epoch: 1.39/4.0 (34.7% complete)    lr: 0.004725    
2019-04-06 13:11:26,513 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 126/236    Epoch: 1.40/4.0 (35.1% complete)    lr: 0.004680    
2019-04-06 13:21:44,917 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 127/236    Epoch: 1.42/4.0 (35.5% complete)    lr: 0.004635    
2019-04-06 13:32:03,987 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 128/236    Epoch: 1.44/4.0 (35.9% complete)    lr: 0.004590    
2019-04-06 13:42:22,224 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 129/236    Epoch: 1.45/4.0 (36.4% complete)    lr: 0.004546    
2019-04-06 13:52:40,895 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 130/236    Epoch: 1.47/4.0 (36.8% complete)    lr: 0.004502    
2019-04-06 14:02:59,529 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 131/236    Epoch: 1.49/4.0 (37.2% complete)    lr: 0.005095    
2019-04-06 14:13:33,325 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 132/236    Epoch: 1.51/4.0 (37.7% complete)    lr: 0.005039    
2019-04-06 14:24:11,763 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 133/236    Epoch: 1.53/4.0 (38.2% complete)    lr: 0.004984    
2019-04-06 14:34:52,974 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 134/236    Epoch: 1.55/4.0 (38.6% complete)    lr: 0.004929    
2019-04-06 14:45:40,241 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 135/236    Epoch: 1.56/4.0 (39.1% complete)    lr: 0.004875    
2019-04-06 14:56:29,425 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 136/236    Epoch: 1.58/4.0 (39.6% complete)    lr: 0.004821    
2019-04-06 15:07:21,115 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 137/236    Epoch: 1.60/4.0 (40.1% complete)    lr: 0.004768    
2019-04-06 15:18:06,794 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 138/236    Epoch: 1.62/4.0 (40.6% complete)    lr: 0.004716    
2019-04-06 15:28:53,828 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 139/236    Epoch: 1.64/4.0 (41.0% complete)    lr: 0.004664    
2019-04-06 15:39:33,290 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 140/236    Epoch: 1.66/4.0 (41.5% complete)    lr: 0.004612    
2019-04-06 15:50:46,043 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 141/236    Epoch: 1.68/4.0 (42.0% complete)    lr: 0.004562    
2019-04-06 16:01:23,675 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 142/236    Epoch: 1.70/4.0 (42.5% complete)    lr: 0.004511    
2019-04-06 16:12:03,797 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 143/236    Epoch: 1.72/4.0 (43.0% complete)    lr: 0.004462    
2019-04-06 16:22:48,814 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 144/236    Epoch: 1.74/4.0 (43.4% complete)    lr: 0.004413    
2019-04-06 16:33:35,533 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 145/236    Epoch: 1.76/4.0 (43.9% complete)    lr: 0.004364    
2019-04-06 16:44:44,469 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 146/236    Epoch: 1.78/4.0 (44.4% complete)    lr: 0.004316    
2019-04-06 16:55:44,284 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 147/236    Epoch: 1.80/4.0 (44.9% complete)    lr: 0.004268    
2019-04-06 17:06:25,686 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 148/236    Epoch: 1.81/4.0 (45.4% complete)    lr: 0.004221    
2019-04-06 17:17:21,944 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 149/236    Epoch: 1.83/4.0 (45.9% complete)    lr: 0.004175    
2019-04-06 17:28:01,171 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 150/236    Epoch: 1.85/4.0 (46.3% complete)    lr: 0.004129    
2019-04-06 17:38:42,093 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 151/236    Epoch: 1.87/4.0 (46.8% complete)    lr: 0.004083    
2019-04-06 17:49:38,944 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 152/236    Epoch: 1.89/4.0 (47.3% complete)    lr: 0.004039    
2019-04-06 18:00:24,808 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 153/236    Epoch: 1.91/4.0 (47.8% complete)    lr: 0.003994    
2019-04-06 18:11:20,365 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 154/236    Epoch: 1.93/4.0 (48.3% complete)    lr: 0.003950    
2019-04-06 18:22:13,062 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 155/236    Epoch: 1.95/4.0 (48.7% complete)    lr: 0.004395    
2019-04-06 18:35:07,483 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 156/236    Epoch: 1.97/4.0 (49.3% complete)    lr: 0.004341    
2019-04-06 18:48:01,121 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 157/236    Epoch: 1.99/4.0 (49.8% complete)    lr: 0.004287    
2019-04-06 19:00:54,746 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 158/236    Epoch: 2.01/4.0 (50.4% complete)    lr: 0.004234    
2019-04-06 19:13:48,476 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 159/236    Epoch: 2.04/4.0 (50.9% complete)    lr: 0.004181    
2019-04-06 19:26:41,752 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 160/236    Epoch: 2.06/4.0 (51.4% complete)    lr: 0.004130    
2019-04-06 19:39:57,499 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 161/236    Epoch: 2.08/4.0 (52.0% complete)    lr: 0.004079    
2019-04-06 19:52:50,774 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 162/236    Epoch: 2.10/4.0 (52.5% complete)    lr: 0.004028    
2019-04-06 20:05:44,245 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 163/236    Epoch: 2.12/4.0 (53.1% complete)    lr: 0.003978    
2019-04-06 20:18:38,084 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 164/236    Epoch: 2.14/4.0 (53.6% complete)    lr: 0.003929    
2019-04-06 20:31:31,682 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 165/236    Epoch: 2.17/4.0 (54.1% complete)    lr: 0.003880    
2019-04-06 20:44:25,145 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 166/236    Epoch: 2.19/4.0 (54.7% complete)    lr: 0.003832    
2019-04-06 20:57:19,088 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 167/236    Epoch: 2.21/4.0 (55.2% complete)    lr: 0.003785    
2019-04-06 21:10:13,384 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 168/236    Epoch: 2.23/4.0 (55.8% complete)    lr: 0.003738    
2019-04-06 21:23:07,873 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 169/236    Epoch: 2.25/4.0 (56.3% complete)    lr: 0.003692    
2019-04-06 21:36:01,472 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 170/236    Epoch: 2.27/4.0 (56.9% complete)    lr: 0.003646    
2019-04-06 21:48:55,328 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 171/236    Epoch: 2.30/4.0 (57.4% complete)    lr: 0.003601    
2019-04-06 22:01:48,904 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 172/236    Epoch: 2.32/4.0 (57.9% complete)    lr: 0.003556    
2019-04-06 22:14:42,666 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 173/236    Epoch: 2.34/4.0 (58.5% complete)    lr: 0.003512    
2019-04-06 22:27:36,168 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 174/236    Epoch: 2.36/4.0 (59.0% complete)    lr: 0.003469    
2019-04-06 22:40:30,298 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 175/236    Epoch: 2.38/4.0 (59.6% complete)    lr: 0.003426    
2019-04-06 22:53:23,611 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 176/236    Epoch: 2.40/4.0 (60.1% complete)    lr: 0.003384    
2019-04-06 23:06:16,792 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 177/236    Epoch: 2.43/4.0 (60.6% complete)    lr: 0.003342    
2019-04-06 23:19:10,594 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 178/236    Epoch: 2.45/4.0 (61.2% complete)    lr: 0.003667    
2019-04-06 23:32:11,121 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 179/236    Epoch: 2.47/4.0 (61.8% complete)    lr: 0.003617    
2019-04-06 23:45:10,901 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 180/236    Epoch: 2.50/4.0 (62.4% complete)    lr: 0.003567    
2019-04-06 23:58:28,143 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 181/236    Epoch: 2.52/4.0 (63.0% complete)    lr: 0.003518    
2019-04-07 00:11:23,083 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 182/236    Epoch: 2.54/4.0 (63.6% complete)    lr: 0.003470    
2019-04-07 00:24:18,207 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 183/236    Epoch: 2.57/4.0 (64.2% complete)    lr: 0.003422    
2019-04-07 00:37:13,343 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 184/236    Epoch: 2.59/4.0 (64.8% complete)    lr: 0.003375    
2019-04-07 00:50:08,378 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 185/236    Epoch: 2.62/4.0 (65.4% complete)    lr: 0.003328    
2019-04-07 01:03:03,127 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 186/236    Epoch: 2.64/4.0 (66.0% complete)    lr: 0.003283    
2019-04-07 01:15:58,134 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 187/236    Epoch: 2.66/4.0 (66.6% complete)    lr: 0.003238    
2019-04-07 01:28:53,279 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 188/236    Epoch: 2.69/4.0 (67.2% complete)    lr: 0.003193    
2019-04-07 01:41:48,413 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 189/236    Epoch: 2.71/4.0 (67.8% complete)    lr: 0.003149    
2019-04-07 01:54:43,938 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 190/236    Epoch: 2.74/4.0 (68.4% complete)    lr: 0.003106    
2019-04-07 02:07:39,153 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 191/236    Epoch: 2.76/4.0 (69.0% complete)    lr: 0.003063    
2019-04-07 02:20:34,106 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 192/236    Epoch: 2.78/4.0 (69.6% complete)    lr: 0.003021    
2019-04-07 02:33:29,112 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 193/236    Epoch: 2.81/4.0 (70.2% complete)    lr: 0.002980    
2019-04-07 02:46:23,764 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 194/236    Epoch: 2.83/4.0 (70.8% complete)    lr: 0.002939    
2019-04-07 02:59:19,275 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 195/236    Epoch: 2.86/4.0 (71.4% complete)    lr: 0.002898    
2019-04-07 03:12:14,709 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 196/236    Epoch: 2.88/4.0 (72.0% complete)    lr: 0.002859    
2019-04-07 03:25:09,664 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 197/236    Epoch: 2.90/4.0 (72.6% complete)    lr: 0.002819    
2019-04-07 03:38:04,778 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 198/236    Epoch: 2.93/4.0 (73.2% complete)    lr: 0.002780    
2019-04-07 03:50:59,600 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 199/236    Epoch: 2.95/4.0 (73.8% complete)    lr: 0.002742    
2019-04-07 04:03:54,327 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 200/236    Epoch: 2.98/4.0 (74.4% complete)    lr: 0.002705    
2019-04-07 04:17:11,148 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 201/236    Epoch: 3.00/4.0 (75.0% complete)    lr: 0.002667    
2019-04-07 04:30:05,815 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 202/236    Epoch: 3.02/4.0 (75.6% complete)    lr: 0.002894    
2019-04-07 04:45:33,919 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 203/236    Epoch: 3.05/4.0 (76.3% complete)    lr: 0.002850    
2019-04-07 05:01:02,689 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 204/236    Epoch: 3.08/4.0 (76.9% complete)    lr: 0.002807    
2019-04-07 05:16:30,969 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 205/236    Epoch: 3.10/4.0 (77.6% complete)    lr: 0.002765    
2019-04-07 05:31:59,749 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 206/236    Epoch: 3.13/4.0 (78.2% complete)    lr: 0.002723    
2019-04-07 05:47:28,094 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 207/236    Epoch: 3.16/4.0 (78.9% complete)    lr: 0.002682    
2019-04-07 06:02:56,898 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 208/236    Epoch: 3.18/4.0 (79.6% complete)    lr: 0.002641    
2019-04-07 06:18:25,132 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 209/236    Epoch: 3.21/4.0 (80.2% complete)    lr: 0.002601    
2019-04-07 06:33:53,694 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 210/236    Epoch: 3.24/4.0 (80.9% complete)    lr: 0.002562    
2019-04-07 06:49:22,680 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 211/236    Epoch: 3.26/4.0 (81.6% complete)    lr: 0.002523    
2019-04-07 07:04:50,841 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 212/236    Epoch: 3.29/4.0 (82.2% complete)    lr: 0.002485    
2019-04-07 07:20:18,889 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 213/236    Epoch: 3.31/4.0 (82.9% complete)    lr: 0.002448    
2019-04-07 07:35:47,688 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 214/236    Epoch: 3.34/4.0 (83.5% complete)    lr: 0.002411    
2019-04-07 07:51:15,928 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 215/236    Epoch: 3.37/4.0 (84.2% complete)    lr: 0.002374    
2019-04-07 08:06:44,152 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 216/236    Epoch: 3.39/4.0 (84.9% complete)    lr: 0.002338    
2019-04-07 08:22:12,565 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 217/236    Epoch: 3.42/4.0 (85.5% complete)    lr: 0.002303    
2019-04-07 08:37:41,115 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 218/236    Epoch: 3.45/4.0 (86.2% complete)    lr: 0.002268    
2019-04-07 08:53:09,536 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 219/236    Epoch: 3.47/4.0 (86.8% complete)    lr: 0.002234    
2019-04-07 09:08:38,521 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 220/236    Epoch: 3.50/4.0 (87.5% complete)    lr: 0.002200    
2019-04-07 09:24:29,832 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 221/236    Epoch: 3.53/4.0 (88.2% complete)    lr: 0.002167    
2019-04-07 09:39:58,290 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 222/236    Epoch: 3.55/4.0 (88.8% complete)    lr: 0.002134    
2019-04-07 09:55:27,495 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 223/236    Epoch: 3.58/4.0 (89.5% complete)    lr: 0.002102    
2019-04-07 10:10:56,395 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 224/236    Epoch: 3.61/4.0 (90.1% complete)    lr: 0.002070    
2019-04-07 10:26:24,934 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 225/236    Epoch: 3.63/4.0 (90.8% complete)    lr: 0.002039    
2019-04-07 10:41:53,553 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 226/236    Epoch: 3.66/4.0 (91.5% complete)    lr: 0.002191    
2019-04-07 10:57:23,848 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 227/236    Epoch: 3.69/4.0 (92.2% complete)    lr: 0.002155    
2019-04-07 11:13:00,463 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 228/236    Epoch: 3.72/4.0 (92.9% complete)    lr: 0.002119    
2019-04-07 11:28:52,988 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 229/236    Epoch: 3.75/4.0 (93.6% complete)    lr: 0.002084    
2019-04-07 11:44:27,803 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 230/236    Epoch: 3.77/4.0 (94.4% complete)    lr: 0.002050    
2019-04-07 12:00:03,155 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 231/236    Epoch: 3.80/4.0 (95.1% complete)    lr: 0.002016    
2019-04-07 12:15:32,951 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 232/236    Epoch: 3.83/4.0 (95.8% complete)    lr: 0.001983    
2019-04-07 12:31:04,365 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 233/236    Epoch: 3.86/4.0 (96.5% complete)    lr: 0.001950    
2019-04-07 12:47:06,198 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 234/236    Epoch: 3.89/4.0 (97.2% complete)    lr: 0.001918    
2019-04-07 13:03:29,878 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 235/236    Epoch: 3.92/4.0 (98.0% complete)    lr: 0.001887    
2019-04-07 13:19:59,148 [steps/nnet3/train_dnn.py:352 - train - INFO ] Iter: 236/236    Epoch: 3.95/4.0 (98.7% complete)    lr: 0.001800    
2019-04-07 13:35:54,253 [steps/nnet3/train_dnn.py:398 - train - INFO ] Doing final combination to produce final.mdl
2019-04-07 13:35:54,263 [steps/libs/nnet3/train/frame_level_objf/common.py:491 - combine_models - INFO ] Combining set([224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 218, 219, 220, 221, 222, 223]) models.
2019-04-07 13:36:50,887 [steps/nnet3/train_dnn.py:407 - train - INFO ] Getting average posterior for purposes of adjusting the priors.
2019-04-07 13:39:47,914 [steps/nnet3/train_dnn.py:418 - train - INFO ] Re-adjusting priors based on computed posteriors
2019-04-07 13:39:48,078 [steps/nnet3/train_dnn.py:428 - train - INFO ] Cleaning up the experiment directory exp/nnet3/tdnn_sp
steps/nnet2/remove_egs.sh: Finished deleting examples in exp/nnet3/tdnn_sp/egs
exp/nnet3/tdnn_sp: num-iters=237 nj=2..12 num-params=12.2M dim=43+100->3032 combine=-0.49->-0.49 (over 4) loglike:train/valid[157,236,combined]=(-0.54,-0.47,-0.47/-0.70,-0.69,-0.69) accuracy:train/valid[157,236,combined]=(0.820,0.837,0.838/0.777,0.784,0.784)
steps/nnet3/decode.sh --nj 40 --cmd run.pl --mem 24G --online-ivector-dir exp/nnet3/ivectors_dev exp/tri5a/graph data/dev_hires exp/nnet3/tdnn_sp/decode_dev
steps/nnet3/decode.sh: feature type is raw
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 24G --iter final exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/diagnostic/analyze_lats.sh: see stats in exp/nnet3/tdnn_sp/decode_dev/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(1,3,20) and mean=8.8
steps/diagnostic/analyze_lats.sh: see stats in exp/nnet3/tdnn_sp/decode_dev/log/analyze_lattice_depth_stats.log
score best paths
+ steps/score_kaldi.sh --cmd 'run.pl --mem 24G' data/dev_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/score_kaldi.sh --cmd run.pl --mem 24G data/dev_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/score_kaldi.sh: scoring with word insertion penalty=0.0,0.5,1.0
+ steps/scoring/score_kaldi_cer.sh --stage 2 --cmd 'run.pl --mem 24G' data/dev_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/scoring/score_kaldi_cer.sh --stage 2 --cmd run.pl --mem 24G data/dev_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_dev
steps/scoring/score_kaldi_cer.sh: scoring with word insertion penalty=0.0,0.5,1.0
+ echo 'local/score.sh: Done'
local/score.sh: Done
score confidence and timing with sclite
Decoding done.
steps/nnet3/decode.sh --nj 20 --cmd run.pl --mem 24G --online-ivector-dir exp/nnet3/ivectors_test exp/tri5a/graph data/test_hires exp/nnet3/tdnn_sp/decode_test
steps/nnet3/decode.sh: feature type is raw
steps/diagnostic/analyze_lats.sh --cmd run.pl --mem 24G --iter final exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/diagnostic/analyze_lats.sh: see stats in exp/nnet3/tdnn_sp/decode_test/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(1,4,26) and mean=11.4
steps/diagnostic/analyze_lats.sh: see stats in exp/nnet3/tdnn_sp/decode_test/log/analyze_lattice_depth_stats.log
score best paths
+ steps/score_kaldi.sh --cmd 'run.pl --mem 24G' data/test_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/score_kaldi.sh --cmd run.pl --mem 24G data/test_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/score_kaldi.sh: scoring with word insertion penalty=0.0,0.5,1.0
+ steps/scoring/score_kaldi_cer.sh --stage 2 --cmd 'run.pl --mem 24G' data/test_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/scoring/score_kaldi_cer.sh --stage 2 --cmd run.pl --mem 24G data/test_hires exp/tri5a/graph exp/nnet3/tdnn_sp/decode_test
steps/scoring/score_kaldi_cer.sh: scoring with word insertion penalty=0.0,0.5,1.0
+ echo 'local/score.sh: Done'
local/score.sh: Done
score confidence and timing with sclite
Decoding done.
nnet3-info exp/nnet3/tdnn_sp/configs//ref.raw