CAP 4611 Final Exam

Final Exam  
Instructor: Amrit Singh Bedi  
Instructions  
This exam is worth a total of 100 points. Please answer all questions clearly  
and concisely. Show all your work and justify your answers.  
• For Question 1 and 2, please submit the PDF version of your solution  
via webcourses. You can either write it in latex or do it on paper and  
submit the scanned version. But if you do it on paper and scan it,  
you are responsible for ensuring it is readable and properly scanned.  
There will be zero marks if it is not clearly written or scanned.  
• The total time to complete the exam is 24 hours and it is due at 4:00  
pm EST, Friday (April 25th, 2025). This is a take-home exam. Please  
do not use AI like ChatGPT to complete the exam. There are zero  
marks if found (believe me, we would know if you use it).  
Question 1 50 marks  
Context: In supervised learning, understanding the bias-variance tradeoff  
is crucial for developing models that generalize well to unseen data.  
Problem 1 10 marks  
Define the terms bias, variance, and irreducible error in the context of su pervised learning. Explain how each contributes to the total expected error  
of a model.  
1  
Problem 2 20 marks  
Derive the bias-variance decomposition of the expected squared error for a  
regression problem. That is, show that:  
ED,ε[(y − f  
ˆ(x))2  
] =  Bias[f  
ˆ(x)]  
2  

• Var[f  
  ˆ(x)] + σ  
•  
  where f  
  ˆ(x) is the prediction of the model trained on dataset D, y = f(x)+ε,  
  and σ  
•  
  is the variance of the noise ε.  
  Hint: You can start by taking y = f(x) + ε, where E[ε] = 0, and  
  Var[ε] = σ  
•  
  . Let f  
  ˆ(x) be a learned function from the training set D. Then  
  proceed towards the derivation.  
  Problem 3 10 marks  
  Consider two models trained on the same dataset:  
  • Model A: A simple linear regression model.  
  • Model B: A 10th-degree polynomial regression model.  
  Discuss, in terms of bias and variance, the expected performance of each  
  model on training data and 代写CAP 4611 Final Exam unseen test data. Which model is more likely  
  to overfit, and why?  
  Problem 4 10 marks  
  Explain how increasing the size of the training dataset affects the bias and  
  variance of a model. Provide reasoning for your explanation. (10 marks)  
  Question 2: Using Transformer Attention 50  
  marks  
  Context. Consider a simplified Transformer with a vocabulary of six to kens:  
  • I (ID 0): embedding  1.0, 0.0  
    
  • like (ID 1): embedding  0.0, 1.0  
    
  • to (ID 2): embedding  1.0, 1.0  
    
•  
  • eat (ID 3): embedding  0.5, 0.5  
    
  • apples (ID 4): embedding  0.6, 0.4  
    
  • bananas (ID 5): embedding  0.4, 0.6  
    
  All three projection matrices are the 2 × 2 identity:  
  WQ = WK = WV = I2.  
  When predicting the next token, the model uses masked self-attention: the  
  query comes from the last position, while keys and values come from all  
  previous tokens. (Note: show step by step calculation for all questions  
  below)  
  (a) (10 marks) For the input sequence [I, like, to] (IDs [0, 1, 2]),  
  compute the query, key and value vectors for each token.  
  (b) (15 marks) Let Q be the query of the last token and K, V the keys  
  and values of all three tokens.  
  • Compute the row vector of raw attention scores qK⊤, where q is  
  the query of the last token and K is the 3×2 matrix of keys. .  
  • Scale by √  
  dk (with dk = 2) and apply softmax to obtain attention  
  weights.  
  • Compute the context vector as the weighted sum of the values.  
  (c) (15 marks) Given the context vector c ∈ R  
•  
  from part (b), com pute the unnormalized score for each vocabulary embedding via c ·  
  embed(w), i.e. dot-product.  
  • Apply softmax over these six scores to get a probability distribu tion.  
  • Which token has the highest probability? [Note: Because the six  
  embeddings are synthetic and not trained on real text, the token  
  that receives the highest probability may look ungrammatical in  
  normal English; this is an artifact of the toy setup.]  
  (d) (10 marks) Explain why the model selects the token you found in  
  (c). In your answer, discuss:  
  • How the attention weights led to that choice.  
  • Explain why keys/values may include the current token but never