评估与改进机器学习模型

Ml strategy

Single Numble Evaluation Metric

Precision : n% actually are …

Recall : n% was correctly recognized

F1 score : $$\frac{2}{\frac{1}{P}+\frac{1}{R}}$$

optimizing and satisficing metric

cost = accuracy - 0.5 * running time

N matrics : 1 optimizing , (N-1) reach threshold (satisficing)

if doing well on your metric + dev/test set does not crrespond to doing welll on your application , change your metric and/or dev/test set.

Improving model performance

Two fundamental

1. fit the training set well
2. the training set performance generalizes pretty well to the dev/test set

Reduce bias and variance

human lever <–> training error <–> dev error (avoidable bias / variance)

Avoidable bias

• Train bigger model
• Train longer/better optimization algorithms
• NN architecture/hyperparameters search

variance

• more data
• regularization
• NN architecture/hyperparameters search

error analysis

ways

• dev examples to evalueate ideas ( 5/100 -> |10%->9.5% )
• evaluate multiple ideas in parallel

Incorrectly labled examples

diffenrent distributions

human leber <–> training error <–> training-dev error <–> dev/test error

(avoidable bias -> variance -> data mismatch)

address data mismatch

• understand diffenrence between training and dev/test sets
• collect more data similar to dev/test sets

Transfer learning

• the same input
• a lot more data for A than B
• low lever features of A

change the $$w^{[l]};b^{[l]}$$ to pre-training (initial the weights)and pine-tuning (a large number of datas)

Multi-task learning

change the $yand\hat{y}$ dimension

end-to-end

more data learn well

audio --> transcript