基于社交网络信息爬虫的设计与实现

发布时间：2018-05-30 04:37

本文选题：社交网络 + 网络爬虫　；参考：《复旦大学》2014年硕士论文

【摘要】：社交网络作为Web2.0时代的标志,提供了以互联网用户为中心的交流模式和平台。通过社交网络平台,用户可以进行发表消息,分享内容,添加好友以及关注自己感兴趣的人物等操作。社交网络平台中的用户一般是以亿为单位,用户与用户之间通过相互关注和好友关系连成了一张巨大的社会网络,消息能够迅速的在这种网络上进行传播。大多数的社交网络平台都提供开放的API供用户和开发者获取平台相关的数据,但是对AP1的调用次数通常有限制,这使得需要获取大量数据的需求显得极为不方便。因此,对社交网络信息爬虫的相关研究有非常重要的意义。本文以社交网络平台为研究对象,研究了网络爬虫的相关技术。目前社交网络平台大多采用AJAX技术,以便提供丰富的功能以及良好的用户体验,对社交网络平台进行爬取的过程也是AJAX页面解析的一个过程。社交网络平台产生的数据量巨大,数据通常是一些稀疏的非结构化的格式,因此使用传统的关系型数据库进行存储显得不方便。本文使用非关系型数据库MongoDB进行数据的存储；之后在MongoDB存储的基础上进行信息的检索,以便获取用户感兴趣的信息。本文所作的主要工作如下：1、分析了当前社交网络信息在进行爬取时存在的问题,从而引出爬虫设计需要实现的目标；2、构造了一个适合爬取社交网络信息的网络爬虫,使用广度优先的策略对社交网络数据进行爬取；3、使用BeautifulSoup作为做AJAX的解析器,进行网页数据的爬取,从而解决社交网络平台自身提供的开放API对数据的获取有次数限制的缺陷；4、爬取的信息包括用户信息、用户关系信息、网络内容信息以及内容评论信息；5、使用非关系型数据库MongoDB进行数据存储,从而解决社交网络数据不断膨胀的问题。
[Abstract]:As a symbol of Web2.0 era, social network provides a communication mode and platform centered on Internet users. Social networking platforms allow users to post messages, share content, add friends and focus on people they are interested in. The users in the social network platform are usually in billions of units. The users and users through mutual concern and friends into a huge social network, information can be quickly spread on this network. Most social network platforms provide open API for users and developers to access platform-related data, but the number of calls to AP1 is usually limited, which makes the need to obtain a large amount of data extremely inconvenient. Therefore, the research on social network information crawler is of great significance. In this paper, we take the social network platform as the research object, and study the related technology of the web crawler. At present, most of the social network platforms use AJAX technology to provide rich functions and good user experience. The process of crawling the social network platform is also a process of AJAX page parsing. Because of the huge amount of data generated by the social network platform, the data is usually in some sparse unstructured format, so it is not convenient to use the traditional relational database to store the data. In this paper, the non-relational database MongoDB is used to store the data, and then the information is retrieved on the basis of the MongoDB storage in order to obtain the information of interest to the user. The main work of this paper is as follows: 1. This paper analyzes the problems existing in the crawling of the current social network information, which leads to the goal of crawler design and constructs a web crawler suitable for crawling social network information. Using the strategy of breadth first to crawl the social network data, using BeautifulSoup as the parser of the AJAX, crawling the web page data, In order to solve the social network platform provided by the open API has limited access to the number of data defects, crawling information including user information, user relationship information, Web content information and content review information 5, using non-relational database MongoDB for data storage, so as to solve the social network data expansion problem.
【学位授予单位】：复旦大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP393.09

【相似文献】