基于Hadoop平台的天光残留成分的自动识别与检测

发布时间：2018-01-17 18:20

本文关键词：基于Hadoop平台的天光残留成分的自动识别与检测　出处：《山东大学》2017年硕士论文　论文类型：学位论文

【摘要】：天光作为一种主要的噪声,叠加在目标天体光谱之中,降低了光谱的信噪比。LAMOST作为我国最大的光纤光谱望远镜,其拥有一套完整的观测运行系统以及数据处理的流程,其中减天光作为光谱数据处理中重要的步骤之一,目的在于减去目标光谱中叠加的天光噪声,减天光处理过程的有效性将直接影响目标光谱的信噪比。若经过减天光处理之后光谱中仍含有大量强度高的天光残差将不利于对目标光谱进行后续的分析。目前,自动识别减天光异常恒星光谱的研究较少,只能通过人工检测的方法去寻找减天光异常的光谱,这将大大降低了检测的效率。此外,LAMOST项目在每个观测夜可观测数以万记的光谱数据,因此为了提高对海量光谱数据的处理能力,需要一个可靠和高效的处理平台。而Hadoop作为一个分布式的数据处理平台,可以实现对海量光谱中出现减天光异常光谱进行可靠、高效的识别与检测。综上,本课题主要完成以下工作:(1)首先简要叙述LAAMOST光谱的处理流程,并分析影响减天光结果的因素,找出减天光异常光谱的特征,然后提出一种简单有效的方法能够自动识别LAMOST经过Pipeline处理后仍然存在减天光异常的恒星光谱并检测其位置。(2)基于Hadoop平台对光谱数据进行预处理,然后利用中值滤波算法实现分布式的连续谱归一化处理,其目的在于扣除光谱中的连续谱信息,仅仅保留光谱数据中需要的谱线和噪声信息。实验结果表明,该算法可以有效的保留谱线信息,并且应用Hadoop平台大大提高了对海量光谱的处理效率。(3)利用Hadoop的高效性来检测天光线附近是否有一定强度的类似发射线或吸收线的残留来判定该天光线位置是否出现减天光异常,最后得出光谱中所有的减天光异常的天光位置。通过对LAMOST光谱数据的实验表明,这种方法可以有效识别出减天光异常的光谱和发现不同残留强度的天光线异常位置,并且该方法简单易懂,识别效率高,可以应用于大量的减天光异常光谱的识别与检测问题。
[Abstract]:As a main noise, skylight is superimposed on the target celestial spectrum, which reduces the signal-to-noise ratio of the spectrum. LAMOST is the largest optical fiber spectral telescope in China. It has a complete observation operation system and data processing process, in which reducing sky light as one of the important steps in spectral data processing, the purpose is to subtract the superposition of sky noise in the target spectrum. The effectiveness of the process of reducing sky light will directly affect the SNR of target spectrum. If there are still a large number of high-intensity sky light residuals in the spectrum after subtractive sky light processing, it will be unfavorable to the subsequent analysis of target spectrum. . There are few researches on automatic recognition of the spectrum of abnormal stars, which can only be found by manual detection, which will greatly reduce the efficiency of detection. The LAMOST project can observe thousands of spectral data per observation night, so in order to improve the processing capacity of the massive spectral data. A reliable and efficient processing platform is needed, and Hadoop, as a distributed data processing platform, can reliably deal with the abnormal spectrum of reducing sky light in the massive spectrum. In summary, the main work of this paper is as follows: 1) first of all, the processing process of LAAMOST spectrum is briefly described, and the factors that affect the results of reducing sky light are analyzed. Find out the characteristics of the abnormal spectrum of subtractive sky light. Then, a simple and effective method is proposed to automatically identify the spectra of stars whose LAMOST has been processed by Pipeline and detect their positions. The spectral data is preprocessed based on Hadoop platform. Then the median filtering algorithm is used to realize the distributed continuous spectral normalization processing, which aims to deduct the continuous spectral information from the spectrum, and only retain the spectral line and noise information needed in the spectral data. The experimental results show that. The algorithm can effectively preserve the spectral line information. And the application of Hadoop platform greatly improves the processing efficiency of mass spectrum. The high efficiency of Hadoop is used to detect whether there is a similar emission line or the residue of absorption line near the sky light line to determine whether the position of the day light is abnormal or not. Finally, all the abnormal positions of reducing sky light in the spectrum are obtained, and the experimental results of LAMOST spectrum data show that. This method can effectively identify the spectrum of the sky light anomaly and find out the abnormal position of the sky light with different residual intensity, and the method is simple and easy to understand, and the recognition efficiency is high. It can be applied to the recognition and detection of a large number of abnormal spectra of reducing sky light.
【学位授予单位】：山东大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：P111.2

【参考文献】