Performance of Traditional and Dense Vector Information Retrieval Models in Code Search

The application of traditional information retrieval, based explicitly on vector space and probabilistic models, has been the basis of code search development. On the other hand, information-based and language-based models also need to be explored in information retrieval for code search systems. Simultaneously, further investigation into utilizing dense vector-based code search remains necessary. An investigation is required to assess the feasibility of constructing an application that uses information retrieval techniques for code search. This experiment uses a dataset from the CodeSearchNet Challenge as the basis for developing this code search. This study focuses on Java code snippets from the dataset. In the pre-processing step, this experiment uniquely extracts all code snippet identifiers. These extraction identifiers will become tokens to form a document in code search. The traditional information retrieval model application explores the TFIDF, BM25, Divergence from Randomness, Divergence from Independence, information-based models, and Language model methods. Based on their performance evaluation, approaches with probabilistic and information-based models provide slightly better performance than vector space models. However, the MRR-based performance rate of all traditional IR models is around 0.67. Meanwhile, the application of dense vectors in information retrieval uses the SentenceBERT model, which utilizes the all-MiniLM-L6-v2 pre-trained model. The performance evaluation for dense vector information retrieval uses K-NN ranking with K={5,10,15,20,25,50}. Retrieving the 10-nearest neighbors for vector distance between the query and source code delivers the best performance. Overall, the performance results of dense vectors provide better performance by about 8% than traditional information retrieval.