基于实时存储的海量大气颗粒物在线分析系统的研究

发布时间：2018-09-15 06:26

【摘要】：近年来,国内雾霾天气频发,范围广,时间长,严重影响民众的身体健康,同时也对交通、电力和农业造成的较大威胁,雾霾的治理已经引起了政府和社会的高度关注。但由于各大城市空气污染情况各异,且受到地理位置、气象条件、工业成分、城市格局等因素影响,因此治理环境污染必须对城市污染来源进行定性定量的科学研究,从而制定有明显针对性的防治措施。大气颗粒物的监测与分析是了解空气质量的重要手段,而传统的大气颗粒物分析主要依靠颗粒物总体分析技术、人工识别颗粒物类别和来源解析,这些技术手段有明显的缺陷:⑴传统的颗粒物总体分析法无法反映颗粒物内部特征,而现行基于单颗粒质谱仪的分析技术则更加科学;⑵单颗粒质谱仪每天可采集数GB数据,日积月累,数据量庞大,且颗粒物数据呈现半结构化特点,传统的关系型数据库不适用于本场景;⑶传统的人工分析手段耗时长、人工成本高、准确率低,在面临大数据量时无能为力,亟待一种颗粒物自动分析技术。本文针对传统大气颗粒物分析技术的缺陷,设计了一种基于实时存储技术的海量大气颗粒物在线分析系统,该系统由两个子系统组成,分别是基于Google levelDB存储引擎的海量数据存储子系统RyDB和基于数据挖掘的在线分析子系统。底层数据存储系统RyDB是一种KV型NoSQL数据库,采用levelDB存储引擎,支持主从复制和集群部署,用于存储实时采集或者离线收集的大气颗粒物数据;上层的在线分析系统采用自适应谐振理论(ART)网络聚类和逻辑回归分类等数据挖掘技术,实现对颗粒物数据的分类统计、来源解析等功能。经过实验测试,数据存储系统RyDB性能优异,在测试环境中每秒读写能达10万次,具有高吞吐、低时延的特点,能满足实时存储的需求;颗粒物在线分析系统的实验表明,系统时效性较强,32万组颗粒物能够在两小时内分析完毕,颗粒物分类的精确度为80%以上,满足系统需求,实现颗粒物数据的自动化分析。
[Abstract]:In recent years, domestic haze weather frequently, the scope is wide, the time is long, seriously affects the people's health, but also causes the bigger threat to the transportation, the electric power and the agriculture, the haze management has already caused the government and the society to pay close attention. However, due to the fact that the situation of air pollution in major cities is different and affected by the factors such as geographical location, meteorological conditions, industrial composition and urban pattern, it is necessary to carry out qualitative and quantitative scientific research on the sources of urban pollution in order to control the environmental pollution. Therefore, the prevention and cure measures have obvious pertinence. The monitoring and analysis of atmospheric particulate matter is an important means to understand the air quality, while the traditional analysis of atmospheric particulate matter mainly depends on the overall analysis technology of particulate matter, and manually identifies the category and source of particulate matter. These techniques have obvious defects. The traditional particle mass analysis method can not reflect the internal characteristics of particles, but the current analysis technology based on single particle mass spectrometer is more scientific and can collect GB data per day. The traditional relational database is not suitable for the traditional manual analysis method of this scene, which takes a long time, high labor cost and low accuracy. In the face of large amount of data, there is no way to analyze particulate matter automatically. Aiming at the defects of traditional atmospheric particulate matter analysis technology, this paper designs an on-line analysis system of massive atmospheric particulate matter based on real-time storage technology. The system consists of two subsystems. It is a mass data storage subsystem RyDB based on Google levelDB storage engine and an online analysis subsystem based on data mining. The underlying data storage system (RyDB) is a KV type NoSQL database, which uses levelDB storage engine to support master-slave replication and cluster deployment, and is used to store atmospheric particulate data collected in real time or offline. Data mining techniques such as adaptive resonance theory (ART) network clustering and logical regression classification are used in the upper layer online analysis system to realize the classification statistics and source analysis of particulate matter data. The experimental results show that the data storage system RyDB has excellent performance, can read and write 100000 times per second in the test environment, has the characteristics of high throughput and low delay, and can meet the demand of real-time storage. 320000 groups of particles can be analyzed in two hours, and the accuracy of classification of particulate matter is more than 80%, which meets the requirement of the system and realizes the automatic analysis of particulate matter data.
【学位授予单位】：中国科学院大学(中国科学院沈阳计算技术研究所)
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【参考文献】