招投标信息监测系统设计及实现

发布时间：2018-03-02 08:17

本文关键词： 招标信息监测中标信息监测 Mysql 蜘蛛爬虫 ACE中间件　出处：《吉林大学》2014年硕士论文　论文类型：学位论文

【摘要】：随着国内经济的深入发展，随着《招投标法》的贯彻与落实，政府不断加强透明与反腐倡廉，积极促进了招投标市场的巨大发展，随着市场经济的深入发展，企业也越来越多通过公开招标的方式选择合作伙伴。招投标企业数量越来越多，招标信息发布平台也越来越多---各省、市、县区都建有公开发布的招标信息。市场正面临着供给与需求的不对称，标讯发布方希望有实力参与的企业能够进行应标，潜在应标企业也希望能够跨越地域区隔，第一时间获悉自己具有竞争有事的招标项目，并及时参与公开竞标。面对浩如烟海的过剩信息，从中筛选出企业关注的招标信息并不容易，这造成了供需信息的不对称—标讯发布方不能找到足够多的优质竞标方，，想要参与竞标的优秀企业及时找到自己有优势的招标项目并不容易。市场需要这样一个信息化程度高的智能招投标信息监测系统平台来解决信息严重不对称的问题。本系统能够对网络上的超过2000家招标、中标信息发布平台进行实时监测，通过系统自动、智能获取标讯对应的结构化内容，并与客户关注的内容进行匹配，让用户通过该系统平台，即时找到优质竞标项目。系统自动分析和识别标讯内容，如：标讯发布地区--大区/省/直辖市/地级市、标讯公告名称、标讯文件下载地址、标讯快照、采购方式、招标机构、招标内容、项目预算、发标时间、开标时间、来源等。监测系统通过结构化数据结构化数据库实时存储，供客户随时通过各种终端在线使用（浏览、检索、导出）。系统除了对招标信息进行监测外，还提供对招标信息的行业、地域等进行统计分析和统计分析以及对中标信息提供监测和统计分析服务。系统采用了通用蜘蛛框架算法，采用多线程模式，设计简洁稳定，可以支持单服务器多进程部署，或者多服务器分布式部署。抓取全国数千家标讯站点的数据，半小时内完成，系统性能稳定，支撑国内标讯抓取已经足够用，在下一阶段的升级版本，我们希望将竞争情报以及国外标讯纳入监测范围，会对蜘蛛框架及算法进行升级，考虑使用云架构，基于Hadoop系统进行蜘蛛集群部署。 BMS Spider招投标监测蜘蛛子系统是采用事件驱动，流水线作业的多线程蜘蛛系统，使用了ACE的stream、Task等模式及ACE Socket Wrapper Fa ade实现对数千家标讯发布站点进行定点抓取、过滤、识别、索引与存储等功能。数据存储进入标讯数据库，供前台User Platform用户标讯使用平台调用。系统启动后会从标讯站点字典中读取种子URL列表，并压入待抓取标讯URL队列中，蜘蛛读取待抓取URL现成从此队列抽取URL后进行DNS解析并通过网页下载线程对该标讯URL进行抓取，成功抓取的标讯内容页是HTML代码，经过编码、HTML内容解析后，过滤掉导航信息、广告、版权等无效信息后，将种子URL页面内标讯链接进行识别，识别符合规则的标讯链接进入标讯URL列表库，并通过标讯URL链接发射线程根据待抓取URL队列长度，实时补充进入待抓取URL队列列表；标讯URL抓取成功后，通过3.11图示的流程进行Dom节点标注，通过过滤算法过滤掉无效信息后，进行结构化标讯数据分析与提取，提取后的结构化标讯，存储进入进入Mysql标讯内容数据库中，供客户端调用。
[Abstract]:With the further development of the domestic economy, with the implement of "Bidding Law", the government continues to strengthen transparency and anti-corruption, and actively promote the great development of the bidding market, with the development of market economy, more and more enterprises through public bidding to choose partners.
The bidding number of enterprises more and more, more and more tender information publishing platform, the provinces, city, counties have built public bidding information. The market is facing the asymmetry of supply and demand, news release party hope to have the strength in the enterprise can be marked, the potential should be the standard of enterprise also hope to across geographical area the first time that he has historic, competitive advantage of the tender project, and timely public bidding. In the face of the multitude of excess information, selected from the enterprises concerned the bidding information is not easy, which caused the supply and demand information asymmetry as news publishers cannot find enough quality bidders, to find their own advantage the bidding project outstanding enterprises want to participate in the bidding is not easy. The market needs such a high degree of information intelligent bidding platform bidding information monitoring system to solve information A serious problem of asymmetry.
This system can on the network more than 2000 tenders, bid information release platform for real-time monitoring, the system automatically, the corresponding intelligent access to structured content notices, and match the customer's attention, let the user through the system platform, real-time quality bid was found.
System of automatic analysis and recognition of news content, such as news release area -- region / Province / municipality / City, news bulletin, news file download address, notices snapshot, procurement, bidding agency tender, project budget, issuing time, opening time, source monitoring system through structured data structure. The real-time database storage for customers at any time through a variety of terminal use (online browsing, retrieval, export).
In addition to monitoring the bidding information, the system also provides statistical analysis and statistical analysis for the industry and region of the bidding information, and provides monitoring and statistical analysis services for the winning bid information.
The system uses the universal spider framework algorithm, multi thread mode design is simple and stable, can support the deployment of single or multi process server, distributed multi server deployment. Grasping the thousands of news site data, completed within half an hour, the system performance is stable, support domestic grab enough notices in the upgraded version of the next stage. We hope that the competitive intelligence, and foreign news will be included in the scope of monitoring, to upgrade the spider framework and algorithm, consider the use of Cloud Architecture, Hadoop system based on spider cluster deployment.
BMS Spider bidding monitoring subsystem is the event driven spider spider, multi thread pipelining system, using ACE stream, Task ACE Socket Wrapper Fa ade model and Realization of thousands of home news release sites are designated capture, filtering, identification, indexing and storage. Data is stored into the database for tenders. The User Platform user notices using platform invoke.
The system will start to read the list from the news site URL seed dictionary, and pressed to grab notices in the URL queue, the spider crawl URL from the queue to be read off from URL after DNS analysis and through the website of the news thread crawl URL, successfully grab notices content page is HTML code, after encoding. HTML content analysis, filtering out the navigation information, advertising, copyright information is invalid, the seeds of the URL page link to tenders in accordance with the rules of recognition, recognition standard news link into the URL list and through tenders, tenders URL link process according to URL ray capture real-time queue length, added to be captured URL queue list; URL news crawl after the success of the Dom node marked through 3.11 graphic process, through filtering algorithm to filter out invalid information, analyze and extract structured data structure after extraction of the news, Enter the storage of tenders, tenders Mysql content database, for the clients.

【学位授予单位】：吉林大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP274

【相似文献】