基于kettle的分布式科技资源采集和共享平台设计与实现

发布时间：2018-03-03 17:36

本文选题：资源共享　切入点：数据采集　出处：《山东大学》2017年硕士论文　论文类型：学位论文

【摘要】：对于科技活动来说,人力、财力、物力和相应的信息、组织、管理等软件和硬件要素的总称,或者是某几项要素的集合,都被定义为科技资源。它们为科学和技术的发展提供了物质支持,是国家的重要战略保障[1]。中国科技资源的配置需要接受世界的挑战,科技资源的共享无疑成为最为高效和实用的配置方式之一。随着我国互联网基础设施的不断完善和提升,互联网开始向传统领域加速渗透,如何将互联网思维应用到科技资源的共享上,成为我国政府面临的一个问题[2]。近年来,在科技资源共享方面各地政府都做出了创新,但仍然存在科技资源分散、重复、低效,现有科技平台服务模式和方式落后、用户体验不佳等问题。因此,建设统一的科技资源共享平台是我国科技发展的急切要求,只有这样才能提高科技资源开放共享水平,以市场机制充分释放科技资源潜力,服务大众创业、万众创新。本文对科技资源共享的研究现状进行了分析,指出了当前我国科技资源共享信息化所存在的关键问题,并给出了可实现的解决方案。本文设计实现的科技资源共享平台使用了前后端开发完全分离的开发模式,通过高速服务框架实现多应用间的调用和通信。其中后端服务接口采用java语言实现,前端使用浪潮website框架,系统整体则是基于当前主流的Springframework框架开发。该系统主要分为两大部分解决科技资源共享的两大问题,一是数据中心模块,该模块通过开源ETL工具KETTLE实现对多数据库的分布式数据采集,从而解决科技资源整合困难的问题;二是共享门户模块,该模块主要实现科技资源的展示和搜索,将所采集的资源数据导入Solr搜索引擎建立索引,从而实现共享门户的个性化搜索和资源展现。测试结果表明该系统能够完成用户的需求。科技资源共享平台是本人在浪潮软件公司实习过程中参与开发的项目,目前该项目一期已完成并应用于重庆市科技厅,旨在为重庆市创新创业解决找资源难、选资源难、用资源难等问题。现阶段该平台在重庆市已经融入了仪器设备、专家人才、文献资料、科研机构、研发成果等各个方面的共享开放资源。在所有资源中,大型仪器设备是开放共享的重点,包含了重庆市高等学校、研究院所、企业单位等106家机构,其中超过20万元的高级仪器设备3031台。文献资料方面,整合了高校、企业等8家单位共140多个中外文资料库,并加入了万方数据,、中国知网和国研数据中心等近40TB的资源。
[Abstract]:A sum of software and hardware elements, such as human, financial, material, and corresponding information, organization, management, or a collection of certain elements, for scientific and technological activities. Are defined as scientific and technological resources. They provide material support for the development of science and technology and are an important strategic guarantee for the country. [1] the allocation of China's scientific and technological resources needs to meet the challenges of the world. The sharing of scientific and technological resources has undoubtedly become one of the most efficient and practical allocation methods. With the continuous improvement and upgrading of China's Internet infrastructure, the Internet has begun to accelerate its penetration into the traditional fields. How to apply Internet thinking to the sharing of scientific and technological resources has become a problem facing our government [2]. In recent years, local governments have made innovations in the sharing of scientific and technological resources, but there are still scattered, duplicated and inefficient scientific and technological resources. The existing service mode and mode of sci-tech platform are backward, and the user experience is not good. Therefore, building a unified sci-tech resource sharing platform is an urgent requirement for the development of science and technology in our country, only in this way can we improve the level of scientific and technological resources open and sharing. This paper analyzes the present situation of scientific and technological resources sharing, points out the key problems existing in the information sharing of science and technology resources in our country, using the market mechanism to fully release the potential of scientific and technological resources, to serve the masses' entrepreneurship and to innovate. The scientific and technological resource sharing platform designed and implemented in this paper uses the completely separate development mode of front and rear development. Among them, the back-end service interface is implemented by java language, and the front end uses wave website framework. The system is based on the current mainstream Springframework framework development. The system is divided into two parts to solve the two major problems of scientific and technological resources sharing, one is the data center module, This module realizes the distributed data acquisition of multi-database through the open source ETL tool KETTLE, so as to solve the difficult problem of the integration of scientific and technological resources, and the second is the sharing portal module, which mainly realizes the display and search of scientific and technological resources. The collected resource data is imported into the Solr search engine to build an index. The test results show that the system can fulfill the needs of users. The platform of sharing scientific and technological resources is a project that I participated in in the process of practicing in Tide Software Company. At present, the first phase of the project has been completed and applied to Chongqing Science and Technology Department, which aims to solve the problems of finding resources, selecting resources and using resources for Chongqing's innovative entrepreneurship. At this stage, the platform has been integrated into instruments and equipment, experts and talents in Chongqing. Among all the resources, large-scale instruments and equipment are the focus of open sharing, including 106 institutions of higher learning, research institutes and enterprises in Chongqing. Among them, more than 200,000 yuan of advanced instruments and equipment have been set up. In terms of literature and materials, more than 140 Chinese and foreign language databases have been integrated into 8 units, such as universities and enterprises, and nearly 40 terabytes of resources have been added to Wanfang data Center, China knowledge Network and National Institute data Center.
【学位授予单位】：山东大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.52

【相似文献】