A Comparison of Large Language Models and Genetic Programming for Program Synthesis

Large language models have recently become known for their ability to generate computer programs, especially through tools such as GitHub Copilot, a domain where genetic programming has been very successful so far. Although they require different inputs (free-text vs. input/output examples) their goal is the same – program synthesis. Therefore, in this work we compare how well GitHub Copilot and genetic programming perform on common program synthesis benchmark problems. We study the structure and diversity of the generated programs by using well-known software metrics. We find that GitHub Copilot and genetic programming solve a similar number of benchmark problems (85.2% vs. 77.8%, respectively). We find that GitHub Copilot generated smaller and less complex programs as genetic programming, while genetic programming is able to find new and unique problem solving strategies. This increase in diversity of solutions comes at a cost. When analyzing the success rates for 100 runs per problem, GitHub Copilot outperforms genetic programming on over 50% of the problems.