Feasibility of using YouTube Conversations for Pair Programming Intent Classification

Pair programming conversational agents demand vast amounts of data for training. Recently a benchmark dataset of developer-developer and developer-agent pair programming conversations, from lab studies (8,324 utterances) was released for training natural language unit of a pair programming conversational agent. Unfortunately, this dataset is limited to a single domain and language. To investigate if it was feasible to utilize already available pair programming conversations from online video hosting platforms (i.e. YouTube), we collected five Youtube videos (roughly 350 minutes) with 4,822 utterances. We used transformer-based language model BERT to compare the lab studies with online videos. We found that a transfer learning approach, first training BERT on online videos and then fine-tuning with specific developer-agent data, resulted in the best performance.