基于飞腾平台硬件数据压缩的研究与设计实现

发布时间：2018-03-19 12:13

  本文选题：大数据　切入点：zlib函数库　出处：《国防科学技术大学》2013年硕士论文　论文类型：学位论文

【摘要】：数据压缩是有效处理大数据的关键技术,随着互联网技术的快速发展和普及,需要通过计算机存储和处理的数据正在以指数级的速度增长,研究如何对存储的大规模数据进行高效率的压缩以减少存储空间,以及如何对网络传输的数据进行压缩以减少网络传输流量,是现在大规模数据处理的重点。在国产飞腾处理器平台上基于zlib库的数据压缩解压通常采用软件实现,飞腾处理器在进行大数据的压缩解压时存在以下问题:1)受限于处理器性能,采用软件压缩解压时速度慢,耗费大量时间;2)在进行大量数据压缩解压时CPU使用率居高不下,影响了整个系统的性能。针对这些问题,本文通过研究zlib编程函数库,结合飞腾处理器的特性,完成了在飞腾平台中基于硬件的数据压缩解压缩实现。首先,本文研究了数据压缩的的相关原理,并介绍了数据压缩实现的两种方式:软件数据压缩和硬件数据压缩;接着介绍了硬件数据压缩的编码方式以及使用硬件实现数据压缩的三种方法:采用带固件协同处理器的方式、采用专用集成电路的方式和将二者相结合起来的协同处理器加专用集成电路的方式;接着介绍了软件数据压缩的常用算法,并分析了这些算法的优缺点,并从算法的压缩效率、压缩比、算法适应性以及硬件实现难易程度对这些算法进行了深入的研究分析。其次,介绍了飞腾平台的硬件架构及其处理器的特点:FT(飞腾)1000处理器采用sparc精简指令集、4核32线程,在多线程的任务处理方面性能突出;硬件CN61XX架构和特点:CN61XX包含一个MIPS处理器和专用压缩解压协处理器,适合对本地大批量数据进行压缩和解压处理,结合这些硬件特点和原始zlib函数库和CN61XX驱动基础,最后在飞腾平台上设计了FTHC(FT Hardware Compression)系统软件,该系统软件主要包括应用函数库模块、内核驱动模块和底层硬件模块。本文设计实现的FTHC系统结构简单合理、适应性强,可运行于windows、linux、sparc体系结构的系统中。再次,对实现数据压缩解压关键算法和内核驱动优化所使用的基于滑动窗口的字符匹配策略、数据分片技术、CN61XX识别技术、DMA缓冲区分配管理技术以及地址映射技术进行了深入研究,并结合飞腾平台和CN61XX特点,设计实现了FT-zlib编程函数库并提出了CN61XX高效DMA传输机制和基于一致性内存的命令环机制。FT-zlib编程函数库相对以前的zlib函数库进行了优化,采用了数据分片技术、增加了对CN61XX设备的识别探测机制并且对算法接口进行了设计,提高了函数库的可移植性;高效数据传输机制是为了提高CN61XX和主机之间数据传输的效率而提出,通常情况下采用聚集DMA操作方式,其缺点是一次操作大量数据不利于单个控制,而高效数据传输机制弥补了聚集DMA操作时对于单个DMA方向性的控制还提高了DMA传输的效率;基于一致性内存的命令环机制的提出有效的解决了设备驱动和CN61XX设备对同一块内存访问的需要,对于内存空间很容易出现写完之后再读的而产生的数据不一致,从而导致程序出错,命令环中使用的数据一致性管理策略很好的解决了主机和CN61XX互斥访问的问题,同时提高了整个系统的实时性。最后,采用Google提供的标准Snappy套件对整个系统进行了性能测试,以及使用Hadoop工具对系统进行了功能测试,测试结果表明在飞腾平台中基于硬件的数据压缩相对于以前的软件压缩效率提高了30到50倍,而软件的解压效率也提高了10倍,从而有效的缓解了飞腾平台中对大数据的压缩解压处理所带来的性能下降问题。
[Abstract]:Data compression is the key technology in the processing of large data, along with the rapid development and popularization of Internet technology, through the computer storage and processing data with exponential growth, how to study the compression of large-scale data on the storage of high efficiency to reduce the storage space, and how to compress the data transmission network to reduce network the traffic flow, is now focusing on large-scale data processing. In the domestic intelligent processor platform based on zlib data compression and decompression is usually realized by software, intelligent processor there is a problem in the big data compression decompression: 1) limited to processor performance, the software compression and decompression speed, spend a lot of time; 2) in the use of the CPU high rate of large amounts of data compression and decompression, affects the performance of the whole system. To solve these problems, this article through Study on zlib programming library, combined with the characteristics of intelligent processor, completed in the FT platform hardware based data compression and decompression implementation. Firstly, this paper studies the data compression principle, and introduces two methods to realize data compression, data compression software and hardware data compression; then introduces three methods to realize the data compression hardware data compression encoding and the use of hardware: with firmware coprocessor, using ASIC and co processor and special integrated circuit two ways to combine both of them; then introduces the software algorithms of data compression, and analyzes the advantages and disadvantages of these algorithms, and from the efficiency of compression, the compression ratio, the adaptability of algorithm and hardware implementation of the degree of difficulty of in-depth research and analysis of these algorithms. Secondly, introduces the fly The characteristics of the hardware platform and the architecture of Teng processor: FT (1000 ft) RISC processor using SPARC, 4 core 32 thread in a multi-threaded task processing, outstanding performance; CN61XX hardware architecture and features: CN61XX contains a MIPS processor and dedicated coprocessor for compression and decompression, compression and decompression treatment on local a large amount of data, combined with the hardware features and the original zlib function library and CN61XX driver foundation, finally in Feiteng platform design of FTHC (FT Hardware Compression) system software, the system software mainly includes the use function module, kernel module and hardware module of FTHC system. This paper introduces the design and implementation of simple and reasonable structure, adaptability strong, can run on windows, Linux, SPARC system. Thirdly, to realize the data compression and decompression algorithms using kernel driver optimization based on slip Moving window matching strategy, data slicing technique, CN61XX technique, DMA buffer allocation management technology and address mapping technology is studied, and combined with the Feiteng platform and CN61XX characteristics, design and implementation of the FT-zlib programming library and puts forward CN61XX efficient DMA transmission mechanism and based on the uniform memory command ring mechanism.FT-zlib the relative programming library zlib library before were optimized using data slices, increase the recognition of the CN61XX device detection mechanism and algorithm interface were designed, improved library portability; efficient data transfer mechanism is put forward in order to improve the efficiency of data transmission between the CN61XX and the host. Usually the aggregation of DMA mode of operation, its disadvantage is that a single control is not conducive to the operation of large amounts of data, and efficient data transmission mechanism for gathering DMA The operation for the control of a single DMA direction also improves the efficiency of DMA transmission; memory consistency mechanism based on command ring is proposed to solve the needs of device driver and CN61XX device access to the same memory, the memory space is very easy to appear after read and produce inconsistent data, resulting in the program is in error, the data consistency management strategy using the command in the ring is a good solution to the host and CN61XX exclusive access problems, and improve the real-time performance of the whole system. Finally, through the use of Google standard Snappy kit to test the property of the whole system, and the use of Hadoop tool system function test, test the results show that the intelligent platform based on hardware data compression compared to the previous software compression efficiency is improved by 30 to 50 times, while the software decompression efficiency is increased by 10 times, It effectively alleviates the performance degradation caused by the compression and decompression of large data in the flying platform.

【学位授予单位】：国防科学技术大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333
，

本文编号：1634217