基于规则的智能农业大数据聚合平台的设计与实现

发布时间：2018-04-18 13:24

本文选题：农业信息化 + 农业专家系统　；参考：《云南师范大学》2017年硕士论文

【摘要】：数据是人类发展最终的智慧沉淀,是知识的集散地,是人们在自然活动中对自然规律的探索以及研究的结果集合。大数据是指大量的数据进行集中的结果集合。现在一般指的是用IT以及相关的信息化手段把传统的人们对自然探索得出的知识与智慧的沉淀进行集合,然后利用计算机的高速计算能力最终作用于我们的生活。在如今信息爆炸的今天,大数据的价值已经被越来越多的人关注。越来越多的行业将在大数据这一板块进行嫁接。对于农业来说,它的发展,到了今天已经产生了极其多的经验与智慧。如果把这些经验与智慧化为大数据,那么对整个农业未来的发展将是极其有利的。于是,农业大数据聚合平台的研究将是极其有价值的。但是,由于互联网上的数据量过于庞大,种类极其多,很多数据都是无效数据。因此,本文对智能农业大数据聚合的相关理论与技术进行研究,提出了一种快速获取有效农业大数据的架构与机制,设计并实现了基于规则的智能农业大数据聚合平台。本文的主要研究内容及研究成果如下:(1)设计了高性能网络蜘蛛。考虑到互联网数据量极其巨大,为了能够高效获取有效的农业数据,本文基于多任务并发机制,设计了一个高性能的网络蜘蛛。在此过程中由于要频繁进行大数据的存储,会造成内存碎片,降低操作系统性能,所以基于内存的智能垃圾回收机制设计了智能农业大数据聚合平台的内存池与数据通用存储对象。整个高性能网络蜘蛛的核心组件包括:内存池、通用存储对象、配置管理器、URI标准化模块、数据获取器、基于规则的智能推理过滤器、URL识别器、爬行队列、编码转换器、特殊字符过滤器等11个子模块。(2)构建了基于规则的智能农业大数据聚合平台。由于互联网数据种类繁多、庞杂,本文设计了基于规则的农业专家系统来进行数据有效性判断,精准的把互联网中农业相关的数据进行聚合,把农业无关的无效数据进行排除。整个专家系统基于农业特征知识库构建,基于知识库中的农业特征与相关的逻辑关系,在高性能网络蜘蛛进行数据获取的基础上通过它来进行智能推理过滤。最终,实现了一个在海量的、庞杂的互联网的数据中能有效聚合农业相关数据的农业大数据聚合平台。
[Abstract]:Data is the final wisdom precipitation of human development, is the distribution of knowledge, is the people in natural activities in the exploration of natural laws and the results of the collection of research.Big data refers to a large number of data for the set of results.At present, it is generally referred to the collection of traditional knowledge and wisdom from natural exploration by IT and related information methods, and then the use of the high speed computing power of the computer to act on our lives.In today's information explosion, big data's value has been paid more and more attention.More and more industries will be in big data this plate graft.For agriculture, its development, to today has produced a great deal of experience and wisdom.If these experiences and wisdom become big data, then the future development of the whole agriculture will be extremely beneficial.Therefore, the agricultural big data aggregation platform research will be extremely valuable.However, due to the huge amount of data on the Internet, a lot of data are invalid.Therefore, this paper studies the relevant theory and technology of big data aggregation of intelligent agriculture, proposes a framework and mechanism for obtaining effective agricultural big data quickly, and designs and implements a rule-based intelligent agricultural big data aggregation platform.The main contents and results of this paper are as follows: 1) A high performance web spider is designed.Considering the huge amount of Internet data, in order to obtain effective agricultural data efficiently, a high performance web spider is designed based on multi-task concurrency mechanism.In this process, due to the frequent storage of big data, memory fragmentation will be caused, and the performance of the operating system will be reduced.Therefore, the memory storage pool and data universal storage object of big data aggregation platform are designed based on the memory intelligent garbage collection mechanism.The core components of the whole high-performance web spider include: memory pool, universal storage object, configuration manager Uri standardization module, data acquirer, intelligent reasoning filter based on rules, URL recognizer, crawling queue, encoding converter, etc.Special character filter and other 11 submodules. 2) A rule-based intelligent agricultural big data aggregation platform is constructed.Due to the wide variety of Internet data, this paper designed a rule-based agricultural expert system to judge the validity of the data, accurately aggregate the agri-related data in the Internet, and eliminate the agnostic invalid data.The whole expert system is constructed based on the agricultural feature knowledge base, and based on the agricultural features and related logic relations in the knowledge base, the intelligent reasoning filtering is carried out through the high performance web spider on the basis of its data acquisition.Finally, an agri- aggregation platform which can effectively aggregate agri-related data in a large amount of Internet data is implemented.
【学位授予单位】：云南师范大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13;S126

【参考文献】