STATS 4014Advanced Data ScienceAssignment 3Jono TukeSemester 1 2019CHECKLIST: Have you shown all of your working, including probability notation where necessary? : Have you given all numbers to 3 decimal places unless otherwise stated? : Have you included all R output and plots to support your answers where necessary? : Have you included all of your R code? : Have you made sure that all plots and tables each have a caption? : If before the deadline, have you submitted your assignment via the online submission on MyUni? : Is your submission a single pdf file - correctly orientated, easy to read? If not, penalties apply. : Penalties for more than one document - 10% of final mark for each extra document. Note that youmay resubmit and your final version is marked, but the final document should be a single file.? : Penalties for late submission - within 24 hours 40% of final mark. After 24 hours, assignment is notmarked and you get zero. : Assignments emailed instead of submitted by the online submission on MyUni will not be markedand will receive zero. : Have you checked that the assignment submitted is the correct one, as we cannot accept othersubmissions after the due date?Due date: Friday 3rd May 2019 (Week 7), 5pm.Q1. Bayesian connection to lasso and ridge regressiona. Suppose thatYi = β0 + β1xi1 + . . . + βpxip + i,where ～ iid N(0, σ2).Write the likelihood for the data.b. Let βj , j = 1, . . . , p have priors that are iid with�i.e., they are i.i.d. with a double-exponential distribution with mean 0, and common scale parameter b.Write out the posterior for β given the likelihood in Part a. Show that the lasso estimate is the modeof the posterior.c. Let βj , j = 1, . . . , p have priors that are i.i.d. normal distribution with a mean zero and variance c.Write the posterior of βj , j = 1, . . . , p. Hence show that the ridge regression is both the mean and themode of the posterior.1Q2. Using data.tableIn the following, you are advised to use data.table. Trying to use standard data manipulation may crashyour computer or take too long. The data in DNA_combined.csv is real data on DNA methylation in modernand ancient DNA samples.Each row in the dataset is a segment of DNA for which we have the following information: chr: the chromosome the segment is from, pos: the starting position of the segment on the chromosome, N: the length of the segment in number of bases, X: the number of the bases that are methylated, type: whether the DNA is modern or ancient, and ID: the ID of the individual that the DNA is from.Also we have a spreadsheet of metadata given in Data_Info.xlsx. Each row is an individual and we havethe following information: Filename: the filename of the compressed file that had the data. I used this to get the samples for you, SampleID: the ID of each individual, Sex: the gender of the individual, Tissue: the area of the body that the DNA was extracted from, Type: whether the DNA is modern or ancient, and Age_kyr: the age of the individual in 1000’s year.Our goal is to find the proportion of samples for each tissue / type combination that has a higher proportionof methylation compared to the mean for each tisse / type combination.Perform the following steps:a. Read in both datasets.b. Rename the SampleID column to ID in the metadata data.table.c. Find which samples IDs are repeated in the metadata.d. Remove from the metadata any samples that are not Hairpin.e. What is the total number of samples? What is the total number of modern samples and the totalnumber of ancient samples?f. Calculate the proportion of methylation for each sample.g. What is the total number of samples for each combination of tissue and type.h. Calculate the mean proportion of methylation for each combination of tissue and type.i. What proportion of samples have a methylation proportion greater than the mean proportion ofmethylation for each tissue / type combination?Q3. WebscrapingIn this question, we are going to webscrape data from the internet movie database. As before there are marksfor webscraping and cleaning 代做STATS 4014作业、代写Data Science作业、R程序语言作业代写、代做R编程设计作业 代做R语言编程|the dataset, but if you prefer not to do this, the cleaned dataset is provided.a. Webscraping the data. The main package for webscraping is rvest:https://rvest.tidyverse.org/Also the chrome extension selectorgadget is really useful to identify the parts of the webpage that containsthe information:https://selectorgadget.com/https://rvest.tidyverse.org/articles/selectorgadget.htmlI have written a template function to start you off based on the following tutorial:2https://www.analyticsvidhya.com/blog/2017/03/beginners-guide-on-web-scraping-in-r-using-rvest-with-hands-on-knowledge/The function is## Get libs ----pacman::p_load(rvest, tidyverse, glue)# get top 100## Take a year and get information from imdb on the top 100 movies for that year.## @param year year to get the data from## @return data frame with the information## @author Jono Tuke## Wednesday 27 Mar 2019get_top_100 # Create url for the given year split to make easier readingurl count=100&release_date={year},{year},&title_type=feature)# Read in the webpagehtml if(try-error %in% class(html)){cat(Cannot load webpage, url, \n)return(NA)}# Get title of moviestitles %html_nodes(.lister-item-header a) %>%html_text()# Ratingsratings html %>%html_nodes(.ratings-imdb-rating strong) %>%html_text()## Put togetherinfo year = year,title = titles,rating = ratings)return(info)}At present, it gets only year, title and ratings for the top 100 movies for a given year.Write a function that will get the following## # A tibble: 6 x 11## year title description runtimes genre rating vote director actors## 3## 1 1980 The ~ \n A f~ 146 min \nD~ 8.4 768,~ Stanley~ Jack ~## 2 1980 Star~ \n Aft~ 124 min \nA~ 8.8 1,03~ Irvin K~ Mark ~## 3 1980 The ~ \n Jak~ 133 min \nA~ 7.9 165,~ John La~ John ~## 4 1980 Flyi~ \n A m~ 88 min \nC~ 7.8 188,~ Jim Abr~ Rober~## 5 1980 Flas~ \n A f~ 111 min \nA~ 6.5 45,2~ Mike Ho~ Sam J~## 6 1980 The ~ \n In ~ 104 min \nA~ 5.7 57,7~ Randal ~ Brook~## # ... with 2 more variables: metascore , gross Then webscrape the data for 1980 to 2018 inclusively.b. Cleaning the data. I will leave the decisions on the cleaning to you, but so that you know - I kept thetop 20 most prolific directors and top 20 most prolific actors - the rest became Other. Also I created aboolean column for each genre.c. Which movies are the most highly rated and the most lowly rated?d. Which director has the highest mean rating?e. Fit a lasso regression to predict rating with the following predictors: year, runtimes, vote, metascore, gross, and Animation1.What is the best model? What is the first coefficient to be shrunk to zero as λ increases, and what isthe last coefficient?=======1Just because I am obsessed with animation.4Mark schemePart Marks Difficulty Area Type CommentsQ11a 4 0.00 Lasso/ridge proof 4 for derivation1b 7 0.29 Lasso/ridge proof 5 for derivation; 2 for justification1c 7 0.29 Lasso/ridge proof 5 for derivation; 2 for justificationTotal 18Q22a 1 0.00 data.table analysis 1 for coding2b 1 1.00 data.table analysis 1 for coding2c 2 0.50 data.table analysis 2 for coding2d 1 0.00 data.table analysis 1 for coding2e 2 0.00 data.table analysis 2 for coding2f 1 0.00 data.table analysis 1 for coding2g 4 0.50 data.table analysis 4 for coding2h 2 0.00 data.table analysis 2 for coding2i 5 0.60 data.table analysis 2 for coding; 3 for over presentation ofcode in this QTotal 19Q33a 7 0.29 Lasso/ridge coding 5 for coding; 2 for quality of code3b 10 0.20 Lasso/ridge analysis 5 for code; 5 for explanation of code3c 2 0.00 Lasso/ridge analysis 2 for code3d 2 0.00 Lasso/ridge analysis 2 for code3e 8 0.38 Lasso/ridge interpretation 4 for coding; 4 for interpretation ofresultsTotal 29Assignment total 665转自：http://www.7daixie.com/2019050430752661.html
