Measuring Code Similarity in Large-Scaled Code Corpora

Source code similarity measurement is a fundamental technique in software engineering research. Techniques to measure code similarity have been invented and applied to various research areas such as code clone detection, finding bug fixes, and software plagiarism detection. We perform an evaluation of 30 similarity analysers for source code. The results show that specialised tools including clone and plagiarism detectors, with proper parameter tuning, outperform general techniques such as string matching. Although these specialised tools can handle code similarity in local code bases, they fail to locate similar code artefacts from large-scaled corpora. This is increasingly important considering the rising amount of online code artefacts. We propose a scalable search system specifically designed for source code. It lays a foundation to discovering online code reuse, large-scale code clone detection, finding usage examples, detecting software plagiarism, and finding software licensing conflicts. Our proposed code search framework is a hybrid of information retrieval and code clone detection techniques. This framework will be able to locate similar code artefacts instantly. The search is not only based on textual similarity, but also syntactic and structural similarity. It is resilient to incomplete code fragments that are normally found on the Internet.