分布式在线社交网络数据存储及优化技术研究

发布时间：2018-08-29 12:57

【摘要】：近年来,在线社交网络(Online Social Network,OSN)取得了巨大成功,在全球范围内拥有数十亿用户。通过OSN,用户可以结交新的朋友或与自己的朋友共享信息。目前比较流行的商用OSN包括Facebook,Twitter,人人网,新浪微博,腾讯微信等,它们都采用了集中式数据存储架构,所有的用户数据集中存储到服务提供商所运维的服务器上。服务提供商可以使用和分析这些数据,甚至直接将之出售给第三方,破坏了用户隐私。因此,这种集中式在线社交网络(Centralized Online Social Network,COSN)中存在严重的用户数据隐私泄露问题,已经引起了广泛的关注。在此背景下,分布式在线社交网络(Decentralized Online Social Network,DOSN)被提出来解决上述用户数据隐私泄露问题。尽管DOSN还不如COSN流行和成熟,但是业界对它的研究非常活跃,正处于快速发展时期。在DOSN中,为了保护隐私,用户数据绕过服务器直接在朋友圈内存储和转发。DOSN虽然能防止服务提供商泄露用户隐私数据,但是存在数据可用性低下的问题:当某一用户处于离线状态时,其它用户不能访问存储在该离线用户处的数据。为了在数据隐私保护约束条件下提高数据可用性,必须设计适用于DOSN场景的数据存储方案及相应的优化策略,这是目前DOSN研究中的最大挑战之一。DOSN具有如下特点:1)用户动态性高;2)接入终端存储资源有限;3)大部分用户的朋友圈不大;4)社交数据以小数据为主,且很少修改。本文通过深入研究已有的DOSN数据存储技术和存储优化相关工作发现,已有工作主要关注用户动态性,而忽略了其它特点对数据存储优化目标的影响。针对现有工作的不足,本文系统地研究了以数据隐私保护约束下提高数据可用性为主要目标的DOSN数据存储和存储优化问题,主要包括如下几个方面:1.存储容量敏感的DOSN数据可用性建模与分析已有的DOSN数据存储方案通常假设朋友总是能为用户提供足够的存储容量保存用户发布的数据,然而,这个假设在DOSN中是不合适的。为了保证不泄露用户隐私,未受保护的用户隐私数据只能存储在朋友圈内。而如下原因将导致朋友圈内的总存储容量有限:1)在线朋友数量有限;2)DOSN用户通常使用移动智能设备,它们的存储容量通常有限。直观地,有限的朋友圈总存储容量降低了数据可用性。但是仅仅知道这一粗略结论是不够的,我们还希望获悉存储容量对数据可用性的影响程度,以确定是否有必要进行数据存储优化。因此,在重新设计DOSN数据存储方案之前,有必要定量分析朋友圈可贡献的总存储容量与所能达到的数据可用性之间的关系,这是本文要解决的首要问题。针对此问题,本文提出了一个存储容量敏感的数据可用性模型,深入分析了朋友圈可贡献的总存储容量与所能达到的数据可用性之间的定量关系。此外,朋友圈内的在线朋友高度动态变化,影响了朋友圈可贡献的总存储容量,进而导致数据可用性也高度动态变化。针对这一问题,本文通过预测朋友圈实时总存储容量预测实时数据可用性,进一步研究了朋友圈总存储容量与所能达到数据可用性之间的动态变化关系。最后,本文还进行了大量实验,验证了存储容量敏感的数据可用性模型的有效性。基于存储容量敏感的数据可用性模型,给定预期数据可用性可以确定朋友圈所需的最小总存储容量,进而可以确定每个朋友所需贡献的平均最小存储容量,为应用程序存储容量的分配提供依据;反之,给定朋友圈总存储容量,可以确定朋友圈所能达到的最大数据可用性,从而可以确定预期数据可用性是否能够得到满足,并指导下一步的数据存储方案设计。2.云辅助的dosn数据存储方案cadros如上文所述,dosn中,为了保证用户隐私不被泄露,数据在未受保护的情况下只能冗余存储在朋友圈内。但是dosn是一个高度动态的网络,用户可以随时添加和删除朋友,且朋友可以随时上线和下线,所以朋友圈内在线朋友集合和所贡献的总存储容量是有限且动态变化的。如果仅依赖朋友圈冗余存储用户数据,将不能获得较高的数据可用性。以数据隐私保护约束下提高数据可用性为主要目标,设计适用于dosn的数据存储方案是本文要解决的第二个关键问题。针对该问题,本文基于存储容量敏感的数据可用性模型,提出了一种云辅助的dosn数据存储方案cadros,引入云服务器提高数据可用性。当朋友圈不能满足数据存储需求时,将多余数据用纠删码技术分片编码后存储到云服务器中,且保证云服务器中存储的数据片段数量不超过恢复原始数据所需要的数据片段数量,以防止云服务提供商获得原始数据,从而保护了用户数据隐私。本文定量研究了cadros的数据存储能力,讨论了cadros的数据可用性,从理论上证明了cadros方案的可行性和有效性;同时还建立了朋友圈内朋友动态行为的概率模型,通过预测朋友圈将来的数据存储能力和存储需求,建立了cadros实时数据可用性预测模型,可以为下一步设计数据存储策略提供依据。3.社交数据在dosn中的存储优化技术上一研究中的实时数据可用性预测结果只是表明cadros在给定朋友圈总存储容量的前提下有能力达到相应的数据可用性。朋友圈最终是否能获得预期的数据可用性还依赖于数据存储策略。即使朋友圈能提供足够的存储容量,如果没有一个好的数据存储策略,也不能获得理想的数据可用性。在cadros数据存储方案中,如何基于实时数据可用性预测结果,针对dosn用户行为特征,设计一个适用的数据存储策略是本文要解决的第三个关键问题。针对这一问题,本文进一步优化Cadros数据存储方案,研究社交数据在DOSN中的存储优化技术,首先提出了一个开销敏感的数据划分方法和存储策略,分别确定存储到朋友圈内和云服务器中的数据,可充分利用朋友圈可用存储容量,使系统开销最小;然后,提出一个可用性驱动的DOSN数据副本放置方法,合理地将数据放置到朋友圈内,可实现预期的数据可用性,且能均衡系统负载,降低系统为实现数据可用性所造成的维护开销。4.社交数据在云服务器中的存储优化技术如上文所述,Cadros数据存储方案不仅将用户数据冗余存储在朋友圈内,还在朋友圈不能满足数据存储需求时将一部分数据存储到云服务器中。云服务器具有长期高可用的特点,因此云服务器上数据的可用性可近似认为达到100%,不存在数据可用性问题。然而当用户访问云服务器上社交数据时,却存在访问性能不高的问题。社交数据以小数据为主,并很少修改。云服务器通常采用传统的分布式文件系统来存储和管理用户数据,在处理海量社交小数据时性能较低。如何提高云服务器中社交小数据的访问性能是本文要解决的第四个关键问题。为了解决这个问题,本文首先研究了分布式文件系统处理海量社交小数据的性能瓶颈;然后提出了一种轻量级的文件系统iFlatLFS对社交小数据进行优化存储和管理。iFlatLFS大大简化了元数据结构和数据访问流程。新的元数据总量仅占原元数据总量的一小部分,可以被全部缓存到服务器内存中,消除了小数据寻址开销,提高了性能。最后本文在CentOS 5.5操作系统中实现了一个iFlat LFS原型,并集成到了开源分布式文件系统TFS中。本章最后进行了大量实验,结果证明,iFlatLFS能够优化存储海量社交小数据,大幅提高数据访问性能。综上所述,针对DOSN数据存储及存储优化问题研究面临的挑战和现有工作的分析与总结,本文首先定量分析了朋友圈贡献的总存储容量与所能达到的数据可用性之间的关系;在此基础之上,针对朋友圈总存储容量有限所导致的数据可用性低下问题,提出了一种云辅助的DOSN数据存储方案Cadros,解决了数据隐私保护问题,提高了数据可用性,并从理论上证明了Cadros方案的可行性和有效性,建立了实时数据可用性预测模型;然后研究了社交数据在朋友圈内的存储优化问题,基于预测结果提出了一种开销敏感的数据划分方法和存储策略以及可用性驱动的数据放置方法,可达到预期的数据可用性,且能均衡系统负载,降低数据可用性维护开销;最后研究了社交数据在云服务器中的存储优化问题,设计了一种高效的轻量级文件系统iFlatLFS,可改善云服务器上社交数据的访问性能。
[Abstract]:In recent years, the online social network (OSN) has achieved great success, with billions of users worldwide. Through OSN, users can make new friends or share information with their own friends. With centralized data storage architecture, all user data is centrally stored on servers operated and maintained by service providers. Service providers can use and analyze this data, and even sell it directly to third parties, thus destroying user privacy. In this context, distributed online social network (DOSN) has been proposed to solve the problem of user data privacy leakage. Although DOSN is not as popular and mature as COSN, the research on it is very active. In DOSN, to protect privacy, user data is stored and forwarded directly in a friend's circle bypassing the server. Although DOSN can prevent service providers from leaking user's privacy data, there is a problem of low data availability: when a user is offline, other users cannot access the data stored in the offline circle. In order to improve data availability under data privacy protection constraints, data storage schemes and corresponding optimization strategies must be designed for DOSN scenarios, which is one of the biggest challenges in DOSN research. 4) Social data is mainly small data, and rarely modified. Through the in-depth study of existing DOSN data storage technology and storage optimization related work found that the existing work mainly focused on user dynamics, while ignoring other characteristics of the impact of data storage optimization goals. This paper systematically studies the DOSN data storage and storage optimization problem with the main objective of improving data availability under data privacy protection constraints. It mainly includes the following aspects: 1. Storage capacity-sensitive DOSN data availability modeling and analysis. Existing DOSN data storage schemes usually assume that friends always provide sufficient storage for users. Storage capacity holds data published by users, however, this assumption is inappropriate in DOSN. In order not to disclose user privacy, unprotected user privacy data can only be stored in the circle of friends. Energy devices usually have limited storage capacity. Intuitively, limited total Friends storage capacity reduces data availability. But it's not enough to know this rough conclusion. We also want to know how much storage capacity affects data availability to determine whether data storage optimization is necessary. Before the DOSN data storage scheme, it is necessary to quantitatively analyze the relationship between the total storage capacity contributed by the friend circle and the data availability that can be achieved, which is the primary problem to be solved in this paper. In addition, the dynamic changes of the online friends'height in the circle of friends affect the total storage capacity that the circle of friends can contribute to, and consequently lead to a high degree of dynamic changes in data availability. To solve this problem, this paper predicts the real-time data availability by predicting the total real-time storage capacity of the circle of friends. Finally, a large number of experiments are carried out to verify the validity of the storage capacity-sensitive data availability model. Based on the storage capacity-sensitive data availability model, given the expected data availability can be determined. The minimum total storage capacity required by the circle of friends can then determine the average minimum storage capacity that each friend needs to contribute and provide a basis for the allocation of application storage capacity; conversely, given the total storage capacity of the circle of friends, the maximum data availability that the circle of friends can achieve can be determined, thus determining the expected data availability is 2. Cloud-assisted dosn data storage scheme cadros, as mentioned above, in dosn, data can only be stored redundantly in the friends'circle without protection in order to ensure the privacy of users is not leaked. But dosn is a highly dynamic network, users can at any time. Adding and deleting friends, and friends can be online and offline at any time, so the collection of online friends and the total storage capacity contributed by friends are limited and dynamic changes. To achieve this goal, designing a data storage scheme suitable for dosn is the second key problem to be solved in this paper. To solve this problem, a cloud-assisted dosn data storage scheme, cadros, is proposed based on the storage capacity-sensitive data availability model. Cloud servers are introduced to improve data availability. When the Friendship Circle can not meet the data storage needs. In order to prevent cloud service providers from obtaining original data and protect user data privacy, this paper quantitatively studies cadros The data storage capability is discussed, and the data availability of cadros is discussed, which proves the feasibility and validity of the cadros scheme theoretically. At the same time, the probabilistic model of the dynamic behavior of friends in the circle of friends is established. By predicting the future data storage capacity and storage requirements of the circle of friends, a real-time data availability prediction model of cadros is established. The next step is to design the data storage strategy to provide the basis. 3. The real-time data availability prediction results of the research on the storage optimization technology of social data in dosn only show that cadros has the ability to achieve the corresponding data availability under the premise of the total storage capacity of a given circle of friends. It also depends on the data storage strategy. Even if the friend circle can provide enough storage capacity, the ideal data availability can not be achieved without a good data storage strategy. In the cadros data storage scheme, how to design a suitable data storage based on the prediction results of real-time data availability for dosn user behavior characteristics To solve this problem, this paper further optimizes the Cadros data storage scheme and studies the storage optimization technology of social data in DOSN. Firstly, an overhead-sensitive data partitioning method and storage strategy are proposed to determine the data stored in friends and cloud servers, respectively. Make full use of the available storage capacity of the friend circle to minimize the system overhead; then, propose a usability-driven DOSN data replica placement method, reasonably put the data into the friend circle, can achieve the expected data availability, and can balance the system load, reduce the system maintenance overhead to achieve data availability. 4. Social number According to the storage optimization technology in cloud server as mentioned above, Cadros data storage scheme not only stores user data redundancy in the friend circle, but also stores some data in the cloud server when the friend circle can not meet the data storage requirements. The cloud server has the characteristics of long-term high availability, so the data on the cloud server is available. Usability is approximated to 100%. There is no data availability problem. However, when users access social data on cloud servers, there is a problem of poor access performance. Social data is mainly small data and rarely modified. How to improve the access performance of small social data in cloud servers is the fourth key problem to be solved in this paper. To solve this problem, this paper first studies the performance bottleneck of distributed file systems for handling large amounts of small social data, and then proposes a lightweight file system iFlatLFS pair. IFlatLFS greatly simplifies the metadata structure and data access process. The total amount of new metadata accounts for only a small part of the total amount of original metadata and can be cached into the server memory, eliminating the small data addressing overhead and improving performance. Finally, this paper implements it in the CentOS 5.5 operating system. A prototype of iFlat LFS is implemented and integrated into the open source distributed file system TFS. At the end of this chapter, a large number of experiments are carried out. The results show that iFlat LFS can optimize the storage of large amounts of social small data and greatly improve the data access performance. In this paper, firstly, we quantitatively analyze the relationship between the total storage capacity contributed by the friend circle and the data availability that can be achieved. On this basis, we propose a cloud-assisted DOSN data storage scheme Cadros, which solves the problem of low data availability caused by the limited total storage capacity of the friend circle. The protection problem improves the data availability, and theoretically proves the feasibility and validity of the Cadros scheme, establishes a real-time data availability prediction model; then studies the storage optimization problem of social data in the circle of friends, and proposes an overhead-sensitive data partitioning method and storage strategy based on the prediction results, as well as availability. Sex-driven data placement method can achieve the expected data availability, and can balance the system load and reduce the maintenance overhead of data availability. Finally, the storage optimization of social data in cloud servers is studied, and an efficient lightweight file system iFlatLFS is designed to improve the access performance of social data on cloud servers.
【学位授予单位】：国防科学技术大学
【学位级别】：博士
【学位授予年份】：2014
【分类号】：TP393.09;TP333

【相似文献】