A critical analysis of the research methodologies reported in the full papers of the proceedings of ICALT 2004

Five reviewers conducted a methodological review of the full papers in the proceedings of ICALT 2004. They analyzed the papers in terms of research design, common defects, section proportions, measures used, and region of the first author's affiliation. It was found that the majority of articles were methodological/theoretical articles or program descriptions that did not report research on human participants. Of the articles that involved human participants - posttest-only designs without controls were the most common, many articles lacked appropriate interpretations and needed controls, the majority of space was devoted to program description, and student surveys were the most common type of measurement. Recommendations for ICALT authors and reviewers are given.