支持分库分表和读写分离的ORM框架的设计与实现

发布时间：2018-08-13 11:18

【摘要】：随着互联网的飞速发展,越来越多的企业在百度搜索引擎上投放广告,实现流量变现。所以广告业务系统的数据呈现爆炸性的增长,单一的数据库服务器已经不足以存储所有的数据,所以按照一系列规则对数据库做了分库分表的处理,并提供主从库来分离数据库的读写操作,以提高读操作的性能。而在广告业务系统中需要一个统一的模块去和所有的数据库进行交互。虽然目前开源的对象映射(ORM)框架已经很多,但是这些框架属于通用型的,不能和业务及数据库最好地配合。而系统目前使用的ORM框架性能较低,对缓存服务器内存的使用比较浪费。为了更好地和数据库、缓存等交互,并节省缓存服务器的内存,提出支持数据库分库分表以及读写分离的ORM框架的需求。该框架使用JAVA语言开发,并和MYSQL服务器和REDIS缓存服务器进行交互。其设计上分为六个模块:配置管理、数据源管理、映射管理、缓存管理、数据存取和数据报告。其中配置管理完成了数据库和缓存配置文件的定义和加载。数据源管理完成了如何将数据库和缓存的配置解析并注册为相应的数据源对象,并如何集中管理和查找数据源。映射管理完成了如何通过JAVA注解将JAVA的对象映射到数据库表。缓存管理完成了缓存的结构以及如何通过一致性哈希管理缓存服务器。数据存取完成了数据库和缓存的增删改查功能的实现,并完成了数据库读写分离的实现和如何通过分布式缓存解决数据库的主从延迟。作为框架的核心,其性能会影响系统的处理能力,所以提升数据库和缓存操作的性能成为关键。最后完成了数据报告模块,通过输出框架运行时的各项数据指标,协助分析框架的性能和缓存使用情况。框架还通过JAVASSIST字节码框架和多线程大大提高了框架的性能。目前完成了框架的所有需求,并通过模拟线上请求进行了功能测试和性能测试。目前已经上线并稳定运行一段时间,在并发量较大的情况下也能大大地提升性能,而对缓存服务器内存的使用也大大减少。
[Abstract]:With the rapid development of the Internet, more and more enterprises put advertisements on Baidu search engine to realize traffic realization. So the data of the advertising business system showed explosive growth, a single database server is no longer enough to store all the data, so according to a series of rules, the database is divided into databases and tables. And provides master-slave library to separate database read and write operations, to improve the performance of reading operations. In the advertising business system, a unified module is needed to interact with all databases. Although there are many open source object mapping (ORM) frameworks, these frameworks are generic and can not best match business and database. At present, the ORM framework used in the system has low performance and is a waste of cache server memory. In order to better interact with database, cache and save memory of cache server, the paper puts forward the requirement of supporting database sorting and reading and writing separate ORM framework. The framework is developed using the JAVA language and interacts with MYSQL servers and REDIS cache servers. The design is divided into six modules: configuration management, data source management, mapping management, cache management, data access and data reporting. Configuration management completes the definition and loading of database and cache configuration files. Data source management completes how to parse and register the configuration of database and cache as the corresponding data source object, and how to centrally manage and find the data source. Mapping management completes how to map JAVA objects to database tables through JAVA annotations. Cache management completes the structure of cache and how to manage cache server through consistency hash. Data access completes the functions of database and cache modification and deletion, and realizes the separation of database reading and writing and how to solve the master-slave delay of database by distributed cache. As the core of the framework, its performance will affect the processing ability of the system, so improving the performance of database and cache operation becomes the key. Finally, the data reporting module is completed, and the performance and cache usage of the framework are analyzed by outputting the data indexes of the framework. The framework also greatly improves the performance of the framework through JAVASSIST bytecode framework and multi-thread. At present, all the requirements of the framework have been completed, and functional and performance tests have been carried out by simulating online requests. At present, it has been running steadily for a period of time, and can greatly improve the performance under the condition of large amount of concurrency, and reduce the memory usage of the cache server greatly.
【学位授予单位】：哈尔滨工业大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】