Do ChatGPT 4o, 4, and 3.5 Generate “Similar” Ratings? Findings and Implications

In this article, I generate ratings for a set of innovation ideas using three versions of ChatGPT: 4o, 4, and 3.5. I find that, although the resulting ratings are highly correlated among those three versions, both 4o and 4 generate ratings whose means are statistically significantly greater than those generated with version 3.5. In addition, the variances of the ratings when using versions 4o and 4 are smaller than the variance for version 3.5. These results indicate that the more recent versions of ChatGPT have larger ratings estimates and less variation. These evolutionary changes suggest some implications of using ChatGPT for ratings of ideas, objects, people, and software and indicate that this property may limit applicability in some settings. I examine some other implications and also analyze what the findings potentially mean for the newly emerging concept of selectively advantageous instability.