Efficient Optical Measurement of Welding Studs With Normal Maps and Convolutional Neural Network

Studs are small screws welded on a car's body. Measuring the key point positions of the studs within production cycle time, also known as online measurement, is crucial for the quality management of the cars. However, the existing stud measuring systems can only measure one stud in a camera position, which is not efficient enough for online measurement. Accurately measuring multiple studs in one camera position is still a challenge. In this article, we propose a stud online measuring system (SOMS), which can measure multiple studs with high accuracy based on photometric stereo and deep learning techniques. To achieve accurate measurements, the SOMS uses normal maps, which are obtained by photometric stereo. Conventional photometric-stereo-based inspection methods reconstruct the surface from the normal maps; however, the reconstruction is inaccurate when the object has a large depth variation range. Instead of reconstruction, the SMOS directly uses a convolutional neural network (CNN) to localize the key points of studs from the normal maps. To adapt CNN to normal map input, we proposed a fusion block which extracts the most significant features among three dimensions of the normal maps. A test based on a collection of different types of studs was conducted to evaluate the advantage of the SOMS. Our system outperformed benchmarks in a series of experiments. Because the SOMS was developed without any prior knowledge on the geometry of the stud, it also has the potential for other optical challenging geometric features' online measurements.