Can we beat google translate?

This paper presents a machine translation evaluation study for Croatian-English language pair. In-domain and out-of-domain translations from Croatian into English have been obtained from Google Translate, our own statistical machine translation system LegTran, and from a professional translator. These translations have been evaluated by six different automatic metrics. The gains obtained from increasing the number of reference translations have been explored and measured. System level correlation between automatic evaluation metrics is given and the significance of the results is discussed. Bootstrapping, approximate randomization and the sign test have been used for confidence intervals and hypothesis testing.