COMP5328 - Advanced Machine Learning
Assignment 2
Due: 11 November 2021, 23:59PM
This assignment is to be completed in groups of 2 to 3 students. It is worth
25% of your total mark.
Introduction
The objective of this assignment is to build a transition matrix estimator and two
classification algorithms that are robust to label noise.
Three input datasets are given. For each dataset, the training and validation
data contains class-conditional random label noise, whereas the test data is clean.
You need to build at least two different classifiers trained and validated on
the noisy data, which can have a good classification accuracy on the clean test
data. You are required to compare the robustness of the two algorithms to label
noise.
For the first two datasets, the transition matrices are provided. You can directly
use the given transition matrices for designing classifiers that are robust to label
noise.
For the last dataset, the transition matrix is not provided. You are required to
build a transition matrix estimator to estimate the transition matrix. Then,
employ your estimated transition matrix for classification. Your estimated tran-
sition matrix must be included in your final report. Note that to validate the
effectiveness of your transition matrix estimator, you could use your estimator on
the first two datasets and compare your estimation to the given transition matri-
ces. The code contained in tutorial 9 could be a good starting point.
Data prepossessing is allowed, but please remember to clarify and justify it in
the report carefully.
1
1 A Guide to Using the Datasets
Three image datasets with .npz format are provided. You can download them via
canvas.
1.1 Attributes Contained in a Dataset
The following code is used to load a dataset and check the shape of its attributes.
import numpy as np
Remember to r ep l a c e the $FILE PATH
datase t = np . load ($FILE PATH)
Xtr va l = datase t [ ’ Xtr ’ ]
S t r v a l = datase t [ ’ Str ’ ]
Xts = datase t [ ’ Xts ’ ]
Yts = datase t [ ’ Yts ’ ]
print ( Xtr va l . shape )
print ( S t r v a l . shape )
print ( Xts . shape )
print ( Yts . shape )
1.1.1 Training and validation data
The variable Xtr val contains the features of the training and validation data.
The shape is (n, image shape) where n represents the total number of the in-
stances.
The variable Str val contains the noisy labels of the n instances. The shape
is (n, ). For all datasets, the class set of the noisy labels is {0, 1, 2}.
Note that do not use all the n examples to train your models. You are re-
quired to independently and randomly sample 80% of the n examples to train a
model and use the rest 20% examples to validate the model.
1.1.2 Test data
The variable Xts contains features of the test data. The shape is (m, image shape),
where m represents the total number of the test instances.
2
The variable Yts contains the clean labels of the m instances. The class set
of the clean labels is also {0, 1, 2}.
1.2 Dateset Description
1.2.1 FashionMINIST0.3.npz
Number of the training and validation examples n = 18000.
Number of the test examples m = 3000.
The shape of each example image shape = (28× 28).
The transition matrix T =0.7 0.3 00 0.7 0.3
0.3 0 0.7.
1.2.2 FashionMINIST0.6.npz
Number of the training and validation examples n = 18000.
Number of the test examples m = 3000.
The shape of each example image shape = (28× 28).
The transition matrix T =0.4 0.3 0.30.3 0.4 0.3
0.3 0.3 0.4 .
1.2.3 CIFAR.npz
Number of the training and validation examples n = 15000.
Number of the test examples m = 3000.
The shape of each example image shape = (32× 32× 3).
The transition matrix T is unknown.
3
2 Performance Evaluation
The performance of each classifier will be evaluated with the top-1 accuracy metric,
that is,
top-1 accuracy =
number of correctly classified examples
total number of test examples 100%.
To have a rigorous performance evaluation, you need to train each classifier
at least 10 times with the different training and validation sets gener-
ated by random sampling. Then report both the mean and the standard
derivation of the test accuracy.
3 Tasks
You need to implement at least two label noise robustness classifiers with at
least one not taught in this course and test their performance on the three datasets.
You need to implement an estimator to estimate the transition matrix. The
code must be written in Python 3. You are allowed to use external libraries for
optimization and linear algebraic calculation. If you have any ambiguity about
whether you can use a particular library or a function, please post your question
on canvas or Ed.
3.1 Image Classification with Known Flip Rates
For the first two datasets, the transition matrices are provided. You can directly
use the given transition matrices for designing classifiers that are robust to label
noise. As mentioned in the section 2, for each classifier, you should report the
mean and the standard derivation of the test accuracy.
3.2 Image Classification with Unknown Flip Rates
For the last dataset, Since the transition matrix is not provided, you need to imple-
ment an estimator to estimate the transition matrix. Then use the estimated
transition matrix to build a noise robust classifier. Note that you can use the
provided transition matrices of the first two datasets to validate the effectiveness
of your transition matrix estimator. You need to include your estimated transition
matrix in the final report. You also need to report the mean and the standard
derivation of the test accuracy for each of your designed noise robustness classi-
fiers. Both estimation accuracy of the transition matrix and the test accuracy on
the last dataset contribute to the final mark.
4
3.3 Report
The report should be organized similar to research papers, and should contain the
following sections:
In abstract, you should briefly introduce the topic of this assignment, your
methods, and describe the organization of your report.
In introduction, you should first introduce the problem of learning with
label noise, and then its significance and applications. You should give an
overview of the methods you want to use.
In related work, you are expected to review the main idea of related label
noise methods (including their advantages and disadvantages).
In methods, you should describe the details of your classification models,
including the formulation of the cost functions, the theoretical foundations
or views (if any) of the cost functions, and the optimization methods. You
should describe the details of the transition matrix estimation methods, the-
oretical foundations (if any), and optimization algorithms.
In experiments, you should introduce your experimental setup (e.g., datasets,
algorithms, evaluation metric, etc.). Then, you should show the experimen-
tal results, compare, and analyze your results. If possible, give your personal
reflection or thoughts on these results.
In conclusion, you should summarize your methods, results, and your in-
sights for future work.
In references, you should list all references cited in your report and format-
ted all references in a consistent way.
In appendix, you should provide instructions on how to run your code.
The layout of the report:
Font: Times New Roman; Title: font size 14; Body: font size 12
Length: Ideally 10 to 15 pages - maximum 20 pages
Note: Submissions must be typeset in LaTex using the provided template.
5
4 Submissions
Detailed instructions are as follows:
- The submission contains two parts: report and source code.
(a) report (a pdf file): the report should include each member’s details
(student id and name).
(b) code (a compressed folder)
i. algorithm (a sub-folder): your code could be multiple files.
ii. data (an empty sub-folder): although two datasets should be inside
the data folder, please do not include them in the zip file. We will
copy those datasets to the data folder when we test the code. - The report (file type: pdf) and the codes (file type: zip) must be named
as student ID numbers of all group members separated by underscores. For
example, “xxxxxxxx xxxxxxxx xxxxxxxx .pdf”. - Only one student needs to submit your report (file type: pdf) to Assignment
- (report) and upload your codes (file type: zip) to Assignment 1 (codes).
- Your submission should include the report and the code. A plagiarism
checker will be used. - You need to clearly provide instructions on how to run your code in the
appendix of the report. - Indicate the contribution of each group member.
- A penalty of minus 5% marks per each day after due (email late submissions
to TA and confirm late submission dates with TA). The maximum delay is - days, after that assignments will not be accepted.
- Remember, the submission deadline is 11 November 2021, 23:59PM.
6 - Marking scheme
Category Criterion Marks Comments
Report [80]
Abstract [3]
problem, methods, and organization
Introduction [6]
the problem you intend to solve
the importance of the problem
Previous work [8]
previous relevant methods used in literature
their advantages and disadvantages
Label noise methods with known flip
rates [23]
pre-processing (if any)
label noise methods’ formulation
cross-validation method for model selection
or avoiding overfitting (if any)
experiments
discussions
Noise rate estimation method [12]
noise rate estimation method’s formulation
experiments
discussions
Label noise methods with unknown flip
rates [10]
pre-processing (if any)
label noise methods’ formulation (if different
from above)
cross-validation method for model selection
or avoiding overfitting (if any)
experiments
discussions
Conclusions and future work [3]
meaningful conclusions based on the results
meaningful future work suggested
7
Presentation [8]
academic style, grammatical sentences, no
spelling mistakes
good structure and layout, consistent format-
ting
appropriate citation and referencing
use graphs and tables to summarize data
Other [7]
at the discretion of the assessor: illustrate
outstanding comprehensive theoretical analy-
sis, demonstrate the insightful and compre-
hensive assessment of the significance of their
results, provide descriptions and explanations
that have depth but clarity, and are concisely
worded
Code [20]
reasonable code running time
well organized, commented and documented
Note: Marks for each category is indicated in square brackets. The minimum mark for the assignment will be 0