LLM Translation Benchmark (TR⇔EN)

⚠️ Disclaimer: This is ongoing academic research — the methodology and results are subject to change, and new models will be added over time.

This benchmark evaluates open and closed LLMs on Turkish ↔ English document-level machine translation. Each model is scored in both directions (en→tr and tr→en) over 800 source documents drawn from four diverse domains, using XCOMET — a learned neural metric that correlates strongly with human MQM judgments.

Datasets (200 documents each, 800 total per direction):

• FLORES Wiki — encyclopedic Wikipedia content from the FLORES-200 evaluation set.
• NTREX News — news articles from the NTREX-128 multilingual news translation benchmark.
• AYM Legal — Turkish Constitutional Court decisions, formal legal register.
• TWiST Tech — technical and software documentation.
• XQuAD Educational — educational/QA passages spanning multiple subjects.

Methodology: Each model translates the English documents to Turkish and the Turkish documents to English. Then XCOMET scores are computed for each translation against the reference translation.

Prompt: All models receive the same instruction-style prompt (taken from the TranslateGemma technical report):

You are a professional {source_lang} ({src_lang_code}) to {target_lang} ({tgt_lang_code}) translator. Your goal is to accurately convey the meaning and nuances of the original {source_lang} text while adhering to {target_lang} grammar, vocabulary, and cultural sensitivities. Produce only the {target_lang} translation, without any additional explanations or commentary. Please translate the following {source_lang} text into {target_lang}:


{text}

XCOMET (Guerreiro et al., 2023, arXiv:2310.10482) is a state-of-the-art reference-based MT evaluation metric.

Prepared by Atahan Uz and Hüseyin Emir Akdağ.