Improving Estimation of Conditional Probability for Determining Importance of a Bigram

We conducted experiments for improving the estimation of importance of a bigram. We will have the benefit in the information retrieval for using the indexed list of bigrams in Japanese. Importance is determined by verifying if the bigram is included in a title. In this study, this importance is expressed by the probability that the bigram appear in title provided that it also appears in abstract. Dealing with less frequently occurring data is always a challenge to estimate the probability, due to the danger of generating false results. Using direct estimation, we estimated the conditional probability by adding a regularization parameter A. We compared our method against the ones used in Apriori and Maximum Likelihood Estimation to see the improvement. We also observed the changes in the result by changing the A. Our method was found to be effective, and the required parameter was also robust.