面向高频证券大数据的流式处理框架及关键技术研究
发布时间：2018-12-25 21:02
【摘要】：信息化技术在各行各业的普及,促使大规模数据产生于不同领域,给大数据处理带来了全新的技术挑战。高频证券交易数据是典型的“流式大数据”,具有数据规模大、结构复杂、流动速度快等特点。如何利用有限的系统资源,构建稳定、可靠、高效的数据处理框架,在高频推送的流式数据周期内及时完成数据响应,是证券数据价值挖掘场景亟待解决的问题。本文在分析研究大数据流式处理模型的基础上,结合多种大数据处理技术构建了一个面向高频证券大数据流式处理框架,对其中涉及关键技术进行研究和改进,并应用于证券数据实时分析场景,实现了高效的数据流处理、管理与查询。全文以构建契合高频证券大数据特征的流式数据处理框架为主线,并深入研究该框架中涉及的关键技术,论文主要完成工作如下:(1)分析设计面向高频证券大数据的流式处理框架。以Storm流式处理框架和Redis内存数据库为技术原型,将二者进行结合并经过改进,设计了面向高频证券大数据的流式处理框架以及流式数据分层处理模型。(2)针对该框架中Storm组件的缺陷和不足,分别从物理、逻辑和应用层面对Storm进行优化改进,以增强其面对高频流式大数据的实时处理能力。(3)设计实现符合证券大数据高效存取的基于Redis的共享内存中心。通过对Redis内存数据库的改进,既保留数据存储的灵活性需求和可扩展性优势,又考虑数据I/O的高效性,弥补了流式处理框架中Storm组件不能保存状态数据的缺陷,为上层应用的深度挖掘提供高效I/O保障。(4)本文设计的框架在高频证券实时分析场景中的应用。完成了面向高频证券大数据的流式处理框架的应用,为后续证券交易策略开发和实现提供框架支撑。
[Abstract]:The popularization of information technology in various industries makes large-scale data come into being in different fields, which brings new technical challenges to big data. High-frequency securities trading data is a typical "flow big data" with the characteristics of large scale, complex structure and fast flow rate. How to make use of the limited system resources to construct a stable, reliable and efficient data processing framework, and to complete the data response timely in the high-frequency push flow data cycle, is an urgent problem to be solved in the securities data value mining scene. On the basis of analyzing and studying the large data flow processing model, this paper constructs a high frequency securities large data flow processing framework combined with various big data processing techniques, and researches and improves the key technologies involved in it. It is applied to real-time analysis of securities data to realize efficient data stream processing, management and query. The main line of this paper is to construct a flow data processing framework that fits the characteristics of high frequency securities big data, and to study the key technologies involved in the framework. The main work of this paper is as follows: (1) the flow processing framework for high frequency securities big data is analyzed and designed. Using Storm streaming processing framework and Redis memory database as the technical prototype, the two technologies are combined and improved. A streaming processing framework for high-frequency securities big data and a hierarchical model for streaming data processing are designed. (2) aiming at the defects and shortcomings of Storm components in this framework, the physical, logical and application layers are optimized and improved from the physical, logical and application layers to the Storm, respectively. In order to enhance its real-time processing ability to face high-frequency flow big data. (3) Design and implement the shared memory center based on Redis which conforms to the efficient access of securities big data. Through the improvement of Redis memory database, it not only preserves the flexibility and expansibility of data storage, but also considers the high efficiency of data I / O, which makes up for the defect that Storm component can not save state data in streaming processing framework. It provides an efficient I / O guarantee for the depth mining of the upper application. (4) the application of the framework designed in this paper in the real-time analysis scenario of high-frequency securities. The application of flow processing framework for high frequency securities big data is completed, which provides framework support for the development and implementation of subsequent securities trading strategies.
【学位授予单位】：西北大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：F830.91;TP311.13
本文编号：2391643
[Abstract]:The popularization of information technology in various industries makes large-scale data come into being in different fields, which brings new technical challenges to big data. High-frequency securities trading data is a typical "flow big data" with the characteristics of large scale, complex structure and fast flow rate. How to make use of the limited system resources to construct a stable, reliable and efficient data processing framework, and to complete the data response timely in the high-frequency push flow data cycle, is an urgent problem to be solved in the securities data value mining scene. On the basis of analyzing and studying the large data flow processing model, this paper constructs a high frequency securities large data flow processing framework combined with various big data processing techniques, and researches and improves the key technologies involved in it. It is applied to real-time analysis of securities data to realize efficient data stream processing, management and query. The main line of this paper is to construct a flow data processing framework that fits the characteristics of high frequency securities big data, and to study the key technologies involved in the framework. The main work of this paper is as follows: (1) the flow processing framework for high frequency securities big data is analyzed and designed. Using Storm streaming processing framework and Redis memory database as the technical prototype, the two technologies are combined and improved. A streaming processing framework for high-frequency securities big data and a hierarchical model for streaming data processing are designed. (2) aiming at the defects and shortcomings of Storm components in this framework, the physical, logical and application layers are optimized and improved from the physical, logical and application layers to the Storm, respectively. In order to enhance its real-time processing ability to face high-frequency flow big data. (3) Design and implement the shared memory center based on Redis which conforms to the efficient access of securities big data. Through the improvement of Redis memory database, it not only preserves the flexibility and expansibility of data storage, but also considers the high efficiency of data I / O, which makes up for the defect that Storm component can not save state data in streaming processing framework. It provides an efficient I / O guarantee for the depth mining of the upper application. (4) the application of the framework designed in this paper in the real-time analysis scenario of high-frequency securities. The application of flow processing framework for high frequency securities big data is completed, which provides framework support for the development and implementation of subsequent securities trading strategies.
【学位授予单位】：西北大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：F830.91;TP311.13
【参考文献】
相关期刊论文 前7条
1 张琛;;证券投资基本分析及量化交易指标的应用研究[J];经营管理者;2015年09期
2 徐宗本;冯芷艳;郭迅华;曾大军;陈国青;;大数据驱动的管理与决策前沿课题[J];管理世界;2014年11期
3 程学旗;靳小龙;王元卓;郭嘉丰;张铁赢;李国杰;;大数据系统和分析技术综述[J];软件学报;2014年09期
4 孙大为;张广艳;郑纬民;;大数据流式计算:关键技术及系统实例[J];软件学报;2014年04期
5 Yaxiong Zhao;Jie Wu;Cong Liu;;Dache: A Data Aware Caching for Big-Data Applications Using the MapReduce Framework[J];Tsinghua Science and Technology;2014年01期
6 申德荣;于戈;王习特;聂铁铮;寇月;;支持大数据管理的NoSQL系统研究综述[J];软件学报;2013年08期
7 邓建高;潘江波;杨出云;;基于数据仓库的证券交易决策支持系统设计与开发[J];计算机工程与设计;2008年03期
相关博士学位论文 前1条
1 欧阳琳;分布式数据流处理系统动态负载管理研究[D];武汉理工大学;2009年
相关硕士学位论文 前2条
1 唐黎;面向金融大数据的高效数据处理机制的研究与设计[D];北京邮电大学;2015年
2 顾昕;分布式流式计算框架关键技术的研究与实现[D];北京邮电大学;2012年
，本文编号：2391643
本文链接：https://www.wllwen.com/jingjilunwen/jinrongzhengquanlunwen/2391643.html
