基于Python的豆瓣Top250排行榜影片数据爬取和分析毕业论文+开题报告+答辩PPT+视频讲解+项目源码及运行结果

博主介绍：✌在职Java研发工程师、专注于程序设计、源码分享、技术交流、专注于Java技术领域和毕业设计✌
项目名称
基于Python的豆瓣Top250排行榜影片数据爬取

系统说明
随着互联网的快速发展,在“互联网+”的时态下,大数据的挖掘和分析已成为业界和学术界研究的热点。大数据挖掘可以挖掘先前未知且潜在有用的信息样型或规则，进而转化为有价值的信息或知识，帮助决策者迅速做出适当决策。巧妇难为无米之炊，进行大数据挖掘之前，首先应该获取数据，目前使用Python爬虫技术是使用最广泛的方法之一，可以成功获取互联网上的大数据。为了帮助用户进行影片选择，本文主要基于Python的Scrapy框架，设计并实现对豆瓣电影网上海量影视数据的采集，清洗，保存到本地。并用Pandas，Numpy库对影评进行处理，使用WordCloud对处理的影评进行词云展示，让用户对电影有一个认知。用Matplotlib、Pygal展示口碑+人气电影。

环境需要
1.运行环境：最好是java jdk 1.8，我们在这个平台上运行的。其他版本理论上也可以。
2.IDE环境：IDEA，Eclipse,Myeclipse都可以。推荐IDEA;
3.tomcat环境：Tomcat 7.x,8.x,9.x版本均可
4.硬件环境：windows 7/8/10 1G内存以上；或者 Mac OS；
5.数据库：MySql 5.7版本；
6.是否Maven项目：否；

技术栈

后端：Spring+SpringMVC+Mybatis
前端：JSP+CSS+JavaScript+jQuery

使用说明

使用Navicat或者其它工具，在mysql中创建对应名称的数据库，并导入项目的sql文件；
使用IDEA/Eclipse/MyEclipse导入项目，Eclipse/MyEclipse导入时，若为maven项目请选择maven;
若为maven项目，导入成功后请执行maven clean;maven install命令，然后运行；
将项目中springmvc-servlet.xml配置文件中的数据库配置改为自己的配置;
运行项目，在浏览器中输入http://localhost:8080/ 登录

运行截图

用户管理控制层:
package com.houserss.controller;

import javax.servlet.http.HttpSession;

import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;

import com.houserss.common.Const;
import com.houserss.common.Const.Role;
import com.houserss.common.ServerResponse;
import com.houserss.pojo.User;
import com.houserss.service.IUserService;
import com.houserss.service.impl.UserServiceImpl;
import com.houserss.util.MD5Util;
import com.houserss.util.TimeUtils;
import com.houserss.vo.DeleteHouseVo;
import com.houserss.vo.PageInfoVo;

/**

Created by admin
*/

@Controller
@RequestMapping("/user/")
public class UserController {

@Autowired
private IUserService iUserService;

/**
 * 用户登录
 * @param username
 * @param password
 * @param session
 * @return
 */
@RequestMapping(value = "login.do",method = RequestMethod.POST)
@ResponseBody
public ServerResponse login(User user,String uvcode, HttpSession session){
    String code = (String)session.getAttribute("validationCode");
    if(StringUtils.isNotBlank(code)) {
        if(!code.equalsIgnoreCase(uvcode)) {
            return ServerResponse.createByErrorMessage("验证码不正确");
        }
    }
    ServerResponse response = iUserService.login(user.getUsername(),user.getPassword());
    if(response.isSuccess()){
        session.setAttribute(Const.CURRENT_USER,response.getData());
    }
    return response;
}

}
管理员管理控制层:

package com.sxl.controller.admin;

import java.util.List;
import java.util.Map;

import javax.servlet.http.HttpServletRequest;

import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;

import com.sxl.controller.MyController;

@Controller("adminController")
@RequestMapping(value = "/admin")
public class AdminController extends MyController {



@RequestMapping(value = "/index")
public String frame(Model model, HttpServletRequest request)throws Exception {
    return "/admin/index";
}

@RequestMapping(value = "/main")
public String main(Model model, HttpServletRequest request)throws Exception {
    return "/admin/main";
}

@RequestMapping(value = "/tj1")
public String tj1(Model model, HttpServletRequest request)throws Exception {
    String sql="select DATE_FORMAT(insertDate,'%Y-%m-%d') dates,sum(allPrice) price from t_order order by DATE_FORMAT(insertDate,'%Y-%m-%d')  desc";
    List list = db.queryForList(sql);
    model.addAttribute("list", list);
    System.out.println(list);
    return "/admin/tj/tj1";
}


@RequestMapping(value = "/password")
public String password(Model model, HttpServletRequest request)throws Exception {
    return "/admin/password";
}


@RequestMapping(value = "/changePassword")
public ResponseEntity loginSave(Model model,HttpServletRequest request,String oldPassword,String newPassword) throws Exception {
    Map admin = getAdmin(request);
    if(oldPassword.equals(admin.get("password").toString())){
        String sql="update t_admin set password=? where id=?";
        db.update(sql, new Object[]{newPassword,admin.get("id")});
        return renderData(true,"1",null);
    }else{
        return renderData(false,"1",null);
    }
}

}
修改密码业务逻辑:

package com.sxl.controller.admin;

import java.util.Map;

import javax.servlet.http.HttpServletRequest;

import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;

import com.sxl.controller.MyController;

@Controller("userController")
@RequestMapping(value = "/user")
public class UserController extends MyController {



@RequestMapping(value = "/index")
public String frame(Model model, HttpServletRequest request)throws Exception {
    return "/user/index";
}

@RequestMapping(value = "/main")
public String main(Model model, HttpServletRequest request)throws Exception {
    return "/user/main";
}


@RequestMapping(value = "/password")
public String password(Model model, HttpServletRequest request)throws Exception {
    return "/user/password";
}


@RequestMapping(value = "/changePassword")
public ResponseEntity loginSave(Model model,HttpServletRequest request,String oldPassword,String newPassword) throws Exception {
    Map user = getUser(request);
    if(oldPassword.equals(user.get("password").toString())){
        String sql="update t_user set password=? where id=?";
        db.update(sql, new Object[]{newPassword,user.get("id")});
        return renderData(true,"1",null);
    }else{
        return renderData(false,"1",null);
    }
}
@RequestMapping(value = "/mine")
public String mine(Model model, HttpServletRequest request)throws Exception {

Map user =getUser(request);Map map = db.queryForMap("select * from t_user where id=?",new Object[]{user.get("id")});model.addAttribute("map", map); return "/user/mine";

}



@RequestMapping(value = "/mineSave")
public ResponseEntity mineSave(Model model,HttpServletRequest request,Long id
    ,String username,String password,String name,String gh,String mobile) throws Exception{
    int result = 0;
        String sql="update t_user set name=?,gh=?,mobile=? where id=?";
        result = db.update(sql, new Object[]{name,gh,mobile,id});
    if(result==1){
        return renderData(true,"操作成功",null);
    }else{
        return renderData(false,"操作失败",null);
    }
}
}

通用管理模块:
package com.sxl.controller;

import java.nio.charset.Charset;
import java.util.Locale;
import java.util.ResourceBundle;

import javax.servlet.http.HttpServletRequest;

import org.apache.commons.lang.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;

import com.sxl.util.JacksonJsonUtil;
import com.sxl.util.StringUtil;
import com.sxl.util.SystemProperties;

public class BaseController {

public static final Long EXPIRES_IN = 1000 * 3600 * 24 * 1L;// 1天

@Autowired
private SystemProperties systemProperties;

/**
 * 获得配置文件内容
 */
public String getConfig(String key) {
    return systemProperties.getProperties(key);
}

/**
 * 返回服务器地址 like http://192.168.1.1:8441/UUBean/
 */
public String getHostUrl(HttpServletRequest request) {
    String hostName = request.getServerName();
    Integer hostPort = request.getServerPort();
    String path = request.getContextPath();

    if (hostPort == 80) {
        return "http://" + hostName + path + "/";
    } else {
        return "http://" + hostName + ":" + hostPort + path + "/";
    }
}

/***
 * 获取当前的website路径 String
 */
public static String getWebSite(HttpServletRequest request) {
    String returnUrl = request.getScheme() + "://"
            + request.getServerName();

    if (request.getServerPort() != 80) {
        returnUrl += ":" + request.getServerPort();
    }

    returnUrl += request.getContextPath();

    return returnUrl;
}



/**
 * 初始化HTTP头.
 * 
 * @return HttpHeaders
 */
public HttpHeaders initHttpHeaders() {
    HttpHeaders headers = new HttpHeaders();
    MediaType mediaType = new MediaType("text", "html",
            Charset.forName("utf-8"));
    headers.setContentType(mediaType);
    return headers;
}

/**
 * 返回 信息数据
 * 
 * @param status
 * @param msg
 * @return
 */
public ResponseEntity renderMsg(Boolean status, String msg) {
    if (StringUtils.isEmpty(msg)) {
        msg = "";
    }
    String str = "{\"status\":\"" + status + "\",\"msg\":\"" + msg + "\"}";
    ResponseEntity responseEntity = new ResponseEntity(str,
            initHttpHeaders(), HttpStatus.OK);
    return responseEntity;
}

/**
 * 返回obj数据
 * 
 * @param status
 * @param msg
 * @param obj
 * @return
 */
public ResponseEntity renderData(Boolean status, String msg,
        Object obj) {
    if (StringUtils.isEmpty(msg)) {
        msg = "";
    }
    StringBuffer sb = new StringBuffer();
    sb.append("{");
    sb.append("\"status\":\"" + status + "\",\"msg\":\"" + msg + "\",");
    sb.append("\"data\":" + JacksonJsonUtil.toJson(obj) + "");
    sb.append("}");

    ResponseEntity responseEntity = new ResponseEntity(
            sb.toString(), initHttpHeaders(), HttpStatus.OK);
    return responseEntity;
}


/***
 * 获取IP（如果是多级代理，则得到的是一串IP值）
 */
public static String getIpAddr(HttpServletRequest request) {
    String ip = request.getHeader("x-forwarded-for");
    if (ip == null || ip.length() == 0 || "unknown".equalsIgnoreCase(ip)) {
        ip = request.getHeader("Proxy-Client-IP");
    }

    if (ip == null || ip.length() == 0 || "unknown".equalsIgnoreCase(ip)) {
        ip = request.getHeader("WL-Proxy-Client-IP");
    }

    if (ip == null || ip.length() == 0 || "unknown".equalsIgnoreCase(ip)) {
        ip = request.getRemoteAddr();
    }

    if (ip != null && ip.length() > 0) {
        String[] ips = ip.split(",");
        for (int i = 0; i < ips.length; i++) {
            if (!"unknown".equalsIgnoreCase(ips[i])) {
                ip = ips[i];
                break;
            }
        }
    }

    return ip;
}

/**
 * 国际化获得语言内容
 * 
 * @param key
 *            语言key
 * @param args
 * @param argsSplit
 * @param defaultMessage
 * @param locale
 * @return
 */
public static String getLanguage(String key, String args, String argsSplit,
        String defaultMessage, String locale) {
    String language = "zh";
    String contry = "cn";
    String returnValue = defaultMessage;

    if (!StringUtil.isEmpty(locale)) {
        try {
            String[] localeArray = locale.split("_");
            language = localeArray[0];
            contry = localeArray[1];
        } catch (Exception e) {
        }
    }

    try {
        ResourceBundle resource = ResourceBundle.getBundle("lang.resource",
                new Locale(language, contry));
        returnValue = resource.getString(key);
        if (!StringUtil.isEmpty(args)) {
            String[] argsArray = args.split(argsSplit);
            for (int i = 0; i < argsArray.length; i++) {
                returnValue = returnValue.replace("{" + i + "}",
                        argsArray[i]);
            }
        }
    } catch (Exception e) {
    }

    return returnValue;
}

}
源码获取：
大家点赞、收藏、关注、评论啦、查看QQ号获取联系方式
打卡文章更新 254/365天
精彩专栏推荐订阅：在下方专栏

基于Python的豆瓣Top250排行榜影片数据爬取和分析毕业论文+开题报告+答辩PPT+视频讲解+项目源码及运行结果

你可能感兴趣的:(java)