COMP5349: Cloud Computing Sem. 1/2019Assignment 1: Data Analysis with MapReduce and SparkIndividual Work: 20% 22.03.20191 IntroductionThis assignment tests your ability to implement simple data analytic workload using basicfeatures of MapReduce and Spark framework. In particular, you are encouraged to practicethe skills of designing algorithms by organizing data as key value pairs, the concept thatis central to both frameworks. The data set you will work on is adapted from TrendingYoutube Video Statistics data from Kaggle. There are two workloads you should designand implement against the given data set. You are required to implement one workloadwith MapReduce and the other workload with Spark.2 Input Data Set DescriptionThe dataset contains several months’ records of daily top trending YouTube video in thefollowing ten countries: Canada,France, Germany, India,Japan, Mexico, Russia, South Korea,United Kingdom and United States of America. There are up to 200 trending videoslisted per day.In the original data set, each country’s data is stored in a separate CSV file, with eachrow representing a trending video record. If a video is listed as trending in multiple days,each trending appearance has its own record. The record includes video id, title, trendingdate, publish time, number of views, and so on. The record also includes a category idfield. The categories are slightly different in each country. A JSON file defineing themapping between category ID and category name is provided for each country.The following preprocessing have been done to ensure that you can focus on the mainworkload design.� Merge the 10 individual CSV files into a single CSV file;� Add a column category to store the actual category name based on the mapping� Add a column country to store the trending country, each country is represented bytwo capital letter code.1� Remove rows with invalid video id values� Remove textual columns that are not relevant to the workloads and may cause encodingand parsing issueThe results is a CSV file AllVideos short.csv with mostly numeric and date columns.3 Analysis Workload Description3.1 Category and Trending CorrelationSome videos are trending in multiple countries. We are interested to know if there is anycorrelation between video category and trending popularity among countries. For instance,we all know that “music has an universal appeal”, in the context of Youtube videos, wemay expect to see a common set of trending music videos among many countries. Onthe contrary, we may expect to see each country with a distinctive set of trending politicalvideos.In this workload, you are asked to find out the average country number for videos ineach category. For instance, if in the data set there are five videos belonging to categorySports, their trending data are as follows:video id category trending date views country1 Sports 18.17.02 700 US1 Sports 18.18.02 1500 US2 Sports 18.11.03 3000 US2 Sports 18.11.03 2000 CA2 Sports 18.11.03 5000 IN2 Sports 18.12.03 7000 IN3 Sports 18.17.04 2000 JP4 Sports 18.16.04 3000 KR4 Sports 18.17.04 9000 KR5 Sports 18.16.04 4000 RUWe can see that video 1 appears in 1 country; video 2 appears in 3 countries; video 3, 4and 5 each appears in 1 country respectively. The average country number for videos incategory Sports would be (1+3+1+1+1)5 = 1:4 The final result of this work load would looklike the following:Music: 1.31News & Politics: 1.05...23.2 Controversial Trending Videos IdentificationListing a video as trending would help it attract more views. However, not all trendingvideos are liked by viewers. It is not unusual for a trending video to have more dislikesthan likes; For some video, listing it as trending would increase its dislikes numbermore than the increase of its likes number. This workload aims to identify such videos.Below are a few records of a particular video demonstrating the change of various numbersover time:video id trending date views likes dislikes countryQwZT7T-TXT0 2018-01-03 13305605 835378 629120 USQwZT7T-TXT0 2018-01-04 23389090 1082422 1065772 USQwZT7T-TXT0 2018-01-05 28407744 1204072 1278887 USQwZT7T-TXT0 ... ... ... ... USQwZT7T-TXT0 2018-01-09 37539570 1402578 1674420 USQwZT7T-TXT0 2018-01-03 13305605 835382 629123 GBQwZT7T-TXT0 2018-01-04 23389090 1082426 1065772 GBQwZT7T-TXT0 2018-01-05 728407744 1204074 1278889 GBQwZT7T-TXT0 ... ... ... ... GBQwZT7T-TXT0 2018-01-18 45349447 1572111 1944971 GBThe video has multiple trending appearances in US and GB. In both countries, its views,likes and dislikes all increase over time with each trending appearance. As highlightedin the table above, the dislikes number grows much faster than the likes numbers. Inboth countries, the video ended with higher number of dislikes than likes albeit startingwith higher likes number.In this workload, you are asked to find out the top 10 videos with fastest growthof dislikes number between its first and second trending appearances. Here we measurethe growth of dislikes number by the gap of dislikes increase and likes increasebetween the first two trending appearances in the same country.For instance, the dislikes growth of video QwZT7T-TXT0 in US is computed as follows:(1065772 ?? 629120) ?? (1082422 ?? 835378) = 189608Where the first component is the increase of dislikes and the second component isthe increase of likes between the first and second trending appearances .The result of this workload should show a few details of the top 10 videos, including thevideo id, category, dislike growth value and country code. Below is a few sample results:BEePFpC9qG8, 366556, Film & Animation, DERmZ3DPJQo2k, 334594, Music, KR1Aoc-cd9eYs, 192222, Entertainment, GBQwZT7T-TXT0, 189608, Entertainment, USQwZT7T-TXT0, 189605, Entertainment, GB3If a video has changed its category name over time, you can use the category of thefirst appearance. It is possible to include the same video multiple times in top 10 list if ithas large dislikes growth in multiple countries. Video QwZT7T-TXT0 is such an example.4 Coding and Execution RequirementBelow are requirements on coding and Execution:� You can implement the workloads in either Java or Python. No other language isallowed.� You must use MapReduce and Spark framework in your implementation. Implementationusing plain language features will not achieve any point. Implementation“pretends” to use the framework will not achieve any point. A typical example of“pretending” to use MapReduce framework is to design the workload as one job,with an identity mapper and a bloated reducer; The identity mapper just pass theinput as is to the reducer, which has all the implemented everything in one function.� Your code must take two parameters: an input path and an output path. The outputshould be written to a file.� Your code must execute on AWS EMR with emr-5.21.0 software release.� For Java implementation, a script (e.g. ant build file) must be provided to allow easycreation of the executable jar file. The job submission command must be provided ina read.me file.� For Python implementation, a shell script must be provided with job submission command.45 DeliverableThere are two deliverables: source code and brief report (up to 2 pages). Both are dueon Wednesday 10th of April 23:59 (Week 7).JAVA submission should be organized in the following folder structure and packed as azip file:/workload1/srcbuildfileread.me/workload2/srcbuildfileread.me/report-report.pdfPython submission should be organized in the following folder structure and packagedas a zip file:/workload1xxx.py (multiple files)run.sh/workload2xxx.py (multiple files)run.sh/report-report.pdfThe submitted zip file should be called ---.zip.There will be a demo in week 7 during tutorial time. The tutor will upload all submissionson AWS and run your code on EMR cluster during the demo. You are expected toanswer design and implementation related questions duing the demo.Submit a hard copy of your report together with signed cover sheet during the demo.The report must contain a computation graph for each workload, with very brief descriptions.A sample report will be uploaded as reference.本团队核心人员组成主要包括BAT一线工程师，精通德英语！我们主要业务范围是代做编程大作业、课程设计等等。我们的方向领域：window编程 数值算法 AI人工智能 金融统计 计量分析 大数据 网络编程 WEB编程 通讯编程 游戏编程多媒体linux 外挂编程 程序API图像处理 嵌入式/单片机 数据库编程 控制台 进程与线程 网络安全 汇编语言 硬件编程 软件设计 工程标准规等。其中代写编程、代写程序、代写留学生程序作业语言或工具包括但不限于以下范围:C/C++/C#代写Java代写IT代写Python代写辅导编程作业Matlab代写Haskell代写Processing代写Linux环境搭建Rust代写Data Structure Assginment 数据结构代写MIPS代写Machine Learning 作业 代写Oracle/SQL/PostgreSQL/Pig 数据库代写/代做/辅导Web开发、网站开发、网站作业ASP.NET网站开发Finance Insurace Statistics统计、回归、迭代Prolog代写Computer Computational method代做因为专业，所以值得信赖。如有需要，请加QQ：99515681 或邮箱：[email protected] 微信：codehelp
