检索调研环境自适应抓取算法的研究与实现

发布时间：2019-07-25 12:58

【摘要】：检索调研环境是按照搜索引擎线上检索端模块搭建的供策略研发工程师进行策略调研的线下实验环境。验证策略生效最直接的方式是检索结果评估。策略调研效果评估的一个重要方式是搭建两套调研环境,原始环境和策略升级后环境,用一批query抓取这两个环境,然后对抓取结果送评,通过人工对比打分,由策略研发工程师对送评结果分析,决定是否上线策略。这应该是一个快速迭代的过程;但每个环节都有可能拉长迭代周期。调研环境的稳定性、资源不足、抓取效率等原因影响了策略调研阶段的时间,调研效率提升问题日渐突出。对调研过程中调研环境抓取改进进行研究,并实现了具有自适应性的在线学习抓取算法,极大地提升了抓取效率,减少抓取给调研效率上带来的负面影响。
[Abstract]:The retrieval research environment is an offline experimental environment built according to the retrieval module on the search engine for strategic research and development engineers to carry out strategic research. The most direct way to verify the effectiveness of the strategy is to evaluate the retrieval results. One of the important ways to evaluate the effect of strategic research is to build two sets of research environments, the original environment and the upgraded environment of the strategy, and then use a batch of query to grasp the two environments, and then send the results of the grasp to the evaluation. Through manual comparison, the strategic R & D engineer analyzes the evaluation results and determines whether or not to launch the strategy. This should be a fast iterative process; but it is possible for each link to extend the iterative cycle. The stability of the research environment, the shortage of resources and the efficiency of grasping affect the time of the strategic research stage, and the problem of improving the efficiency of the research is becoming more and more prominent. The improvement of research environment grasping in the process of investigation is studied, and an adaptive online learning grasping algorithm is implemented, which greatly improves the grasping efficiency and reduces the negative impact of grabbing on the research efficiency.
【作者单位】：计算机科学联合研究院(首师大&中科院计算所);百度在线网络技术(北京)有限公司;中科院计算所;
【分类号】：TP391.3

【参考文献】