基于存储测试的大容量数据处理技术研究

发布时间：2018-03-27 05:25

本文选题：存储测试　切入点：LabVIEW　出处：《中北大学》2017年硕士论文

【摘要】：在存储测试领域中,为了获得更高的数据测量精度,数据采集与存储系统的数据采集时间越来越长、数据采样率越来越大,因而采集存储的数据容量也越来越大,有的数据容量甚至达到几百个G,采集存储的数据文件里的通道数目也越来越多,有的通道数目甚至达到几千个,采集存储的数据文件的数目也越来越多,有的数据文件的数目甚至达到几千个。传统的数据处理技术已经不能满足对大容量数据的处理要求,所以对大容量数据处理技术的研究变的至关重要,它是研发能针对大容量数据进行处理的数据处理软件的前提条件。本文首先对多个存储测试系统进行分析,得出存储测试领域中数据处理软件的常规功能,然后对这些常规功能涉及到的大容量数据处理难题进行技术方法研究,也是本文的研究重点:用内存映射文件技术代替分块读取数据技术读取大容量数据,以提高大容量数据的读取效率;对数据可视化的等距抽点算法进行改进,提出了块内数据分析算法,以精确捕获瞬态数据和优化数据显示;用文件索引定位方法代替遍历文件名称方法查找目标数据文件,以提高目标数据文件的查找效率;用基于Windows API函数的多线程技术代替基于MSComm控件的中断服务方式进行大容量数据的采样读数,既解决了大容量数据采样读数过程中数据丢失的问题,又提高了系统效率;用粗糙惩罚算法代替五点三次平滑算法对大容量数据进行平滑,改善了大容量数据的平滑效果。最后,采用LabVIEW编程语言作为软件的开发平台,从软件工程的角度出发,将大容量数据处理技术应用到车辆测试软件的研发中。采用“白盒测试”和“黑盒测试”软件测试方法设计相关的软件测试用例,对软件的内部逻辑结构和实现功能进行测试。测试结果表明车辆测试软件内部逻辑结构合理并且实现功能可靠,能同时满足对大容量数据和小容量数据的处理要求,解决了传统数据处理软件对大容量数据处理效果不理想甚至不能处理的难题,证明了本文所研究的大容量数据处理技术在存储测试领域中具有很高的实用价值。
[Abstract]:In the field of storage testing, in order to obtain higher data measurement accuracy, data acquisition and storage system data acquisition time is getting longer and longer, the data sampling rate is increasing, so the data capacity of data acquisition and storage is also increasing. Some of the data capacity even reached several hundred Gs, the number of channels in the data files collected and stored is increasing, and the number of channels in some cases has even reached several thousand, and the number of data files collected and stored is also increasing. The number of some data files even reaches several thousand. The traditional data processing technology can not meet the requirements of large capacity data processing, so the research on large capacity data processing technology becomes very important. It is the precondition of developing data processing software which can deal with large capacity data. Firstly, this paper analyzes several storage and test systems, and obtains the conventional functions of data processing software in the field of storage and testing. Then the technical methods of these conventional functions involved in large capacity data processing are studied, which is also the key point of this paper: using memory mapping file technology instead of block reading data technology to read large capacity data. In order to improve the reading efficiency of large capacity data, the isometric pumping algorithm of data visualization is improved, and the block data analysis algorithm is proposed to accurately capture transient data and optimize data display. In order to improve the searching efficiency of the target data file, the file index location method is used instead of traversing the file name method to find the target data file. Using multithreading technology based on Windows API function to replace interrupt service mode based on MSComm control to sample large capacity data not only solves the problem of data loss in the process of large capacity data sampling and reading, but also improves the system efficiency. The rough penalty algorithm is used instead of the 5.3 times smoothing algorithm to smooth the large capacity data, which improves the smoothing effect of the large capacity data. Finally, the LabVIEW programming language is used as the development platform of the software, starting from the point of view of software engineering. The large capacity data processing technology is applied to the research and development of vehicle test software. The software test cases are designed by using "white box test" and "black box test" software test method. The test results show that the internal logic structure of the vehicle test software is reasonable and the realization function is reliable, and it can meet the processing requirements of both large capacity and small capacity data. It solves the difficult problem that the traditional data processing software can not deal with the large capacity data. It proves that the large capacity data processing technology studied in this paper has a high practical value in the field of storage and testing.
【学位授予单位】：中北大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333;TP274

【参考文献】