面向证券交易数据的消息中间件的存储系统的设计与实现

发布时间：2018-02-24 14:03

本文关键词： 消息中间件消息持久化分布式存储 Memcached 文件存储　出处：《浙江大学》2012年硕士论文　论文类型：学位论文

【摘要】：金融领域中存在着众多复杂的、分布式的、异构的应用系统。这些应用系统之间存在着大量的业务数据交互需求。国内金融行业某交易中心为了使证券交易数据能够异步地、可靠地、安全地传递给各家交易机构的应用系统,采用消息中间件MessageHub作为可靠的消息传递手段。持久化存储系统是该消息中间件的重要模块,它的实现技术对消息中间件的功能和性能影响很大。 MessageHub现有的持久化系统在扩展性和健壮性等方面存在一些缺点,不能很好地满足该应用场景的需要。为了解决这些问题,本文对消息中间件及其消息持久化技术进行了分析与研究,借鉴分布式存储的相关技术,结合MessageHub对数据存储的需求和特点,设计了一个可靠性高、易扩展、高可用、高性能的,面向证券交易数据传输的消息中间件的持久化存储系统,并通过精心的设计,使其满足该场景对消息一致性的需要。论文详细介绍了存储系统的架构和各个模块的具体设计。存储系统的整体设计借鉴了Memcached这个分布式缓存系统,由客户端组件和服务端组件两部分组成。为了提高存储的可靠性,存储系统对数据进行双备份,并结合一致性哈希算法和虚拟节点技术,实现数据在存储节点的均衡分布。系统对已发送消息的队列store采用主动复制方式,而对待发送消息的队列queue采用被动复制方式,以满足它们对一致性不同的需要。系统通过监听器机制支持存储节点的动态变化。由于MessageHub传输的消息量大,存储系统采用基于文件存储的持久化方法,分别对store和queue设计了各自的存储方法和持久化流程。基于以上研究和创新设计,本文实现了存储系统的原型系统,验证了该系统的可行性,并在模拟环境下进行了相关的测试。测试表明,该系统可靠高效,具有较好的性能和扩展性。
[Abstract]:There are many complex, distributed and heterogeneous applications in the field of finance. There are a lot of business data interaction requirements among these applications. The message middleware (MessageHub) is used as the reliable means of message delivery in the application system which is passed to each trading organization reliably and safely. The persistent storage system is the important module of the message middleware. Its implementation technology has great influence on the function and performance of message middleware. The existing persistence system of MessageHub has some shortcomings in scalability and robustness, which can not meet the needs of the application scenario. In this paper, the message middleware and message persistence technology are analyzed and studied. By referring to the related technology of distributed storage and combining the demand and characteristics of MessageHub for data storage, a high reliability, easy to extend, high availability and high performance is designed. The persistence storage system of message middleware for data transmission of securities exchange is designed carefully to meet the need of message consistency in this scenario. This paper introduces the architecture of the storage system and the design of each module in detail. The overall design of the storage system is based on the Memcached, a distributed cache system. In order to improve the reliability of storage, the storage system makes double backup of data, and combines the consistent hash algorithm and virtual node technology. To realize the balanced distribution of data in the storage node, the system adopts the active replication mode for the queue store of the sent message, while the passive replication method is used for the queue queue that sends the message. The system supports the dynamic change of storage nodes through the listener mechanism. Because of the large amount of messages transmitted by MessageHub, the storage system adopts the method of persistence based on file storage. The storage methods and persistence processes of store and queue are designed respectively. Based on the above research and innovative design, the prototype system of storage system is implemented in this paper, and the feasibility of the system is verified. The test results show that the system is reliable, efficient, and has good performance and expansibility.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【参考文献】