NIST 2009 Open Machine Translation Evaluation (MT09)
Official Release of Results

Date of release: Tue Oct 27 15:48:58 2009
Version: mt09_public_v1

Introduction

This release page is limited to the Current Test for the Urdu-to-English track.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Participants

The following table lists the submissions received from the sites participating in the Urdu-to-English Current Test.

Site ID	Organization	Location	Current Test Set, Urdu-to-English
			Single System Track	System Combination Track
afrl	Air Force Research Laboratory	USA	Yes	-
amsterdam	University of Amsterdam	Netherlands	Late and/or debugged submission	-
cmu-ebmt	Carnegie Mellon EBMT	USA	Yes^(1)	-
cmu-statxfer	Carnegie Mellon StatXfer	USA	Yes	Yes
hongkong	City University of Hong Kong	China	Yes	-
isi-lw	University of Southern California / Language Weaver Inc.	USA	Yes	Yes
jhu	Johns Hopkins University	USA	Late and/or debugged submission	-
systran	SYSTRAN Software Inc.	USA	Yes	-
umd	University of Maryland	USA	Yes	-
upc-lsi	UPC-LSI (Universitat Politècnica de Catalunya, Llenguatges i Sistemes Informàtics)	Spain	Yes	-

⁽¹⁾A late and/or debugged system was also submitted, not reported here.

Current Test Results - Single System Track

Urdu-to-English (Table 1)

Site ID	System	BLEU-4 (mteval-v13a)
		Overall	Newswire	Web
Constrained Data Track
isi-lw	isi-lw_u2e_cn_primary	0.3120	0.3753	0.2440
umd	UMD_u2e_cn_primary	0.2395	0.2697	0.2056
cmu-statxfer	CMU-Stat-Xfer_u2e_cn_primary	0.2322	0.2614	0.2013
afrl	AFRL_u2e_cn_primary	0.2235	0.2526	0.1904
upc-lsi	UPC.LSI_u2e_cn_primary	0.1971	0.2325	0.1610
cmu-ebmt	cmu_cunei_u2e_cn_primary^(1)	0.1863	0.2122	0.1568
hongkong	CityU_u2e_cn_primary	0.0254	0.0291	0.0215
UnConstrained Data Track
systran	SYSTRAN_u2e_un_primary	0.2555	0.3034	0.2051

⁽¹⁾A late and/or debugged system was also submitted, not reported here.

Current Test Results - System Combination Track

Urdu-to-English (Table 2)

Site ID	System	BLEU-4 (mteval-v13a)
		Overall	Newswire	Web
Constrained Data Track
isi-lw	isi-lw_u2e_cn_combo1	0.3187	0.3816	0.2509
cmu-statxfer	CMU-Stat-Xfer_u2e_cn_combo1	0.2504	0.2910	0.2082

Current Test Results - Single System Track and System Combination Track - All metrics

Urdu-to-English (Table 3)

Site ID	System	BLEU-4 (mteval-v13a)			IBM BLEU (bleu-1.04)			NIST (mteval-v13a)			TER (tercom-0.7.25)			METEOR (meteor-0.7)
		Overall	Newswire	Web	Overall	Newswire	Web	Overall	Newswire	Web	Overall	Newswire	Web	Overall	Newswire	Web
Constrained Data Track
isi-lw	isi-lw_u2e_cn_combo1	0.3187	0.3816	0.2509	0.3187	0.3815	0.2509	8.948	9.465	7.540	0.5866	0.5495	0.6199	0.5616	0.6237	0.5027
isi-lw	isi-lw_u2e_cn_primary	0.3120	0.3753	0.2440	0.3119	0.3752	0.2440	8.836	9.379	7.413	0.5845	0.5495	0.6159	0.5539	0.6160	0.4951
cmu-statxfer	CMU-Stat-Xfer_u2e_cn_combo1	0.2504	0.2910	0.2082	0.2504	0.2907	0.2084	8.244	8.550	7.218	0.6273	0.6128	0.6404	0.5380	0.5838	0.4947
umd	UMD_u2e_cn_primary	0.2395	0.2697	0.2056	0.2394	0.2696	0.2056	7.871	8.089	7.051	0.6519	0.6436	0.6593	0.5355	0.5815	0.4919
cmu-statxfer	CMU-Stat-Xfer_u2e_cn_primary	0.2322	0.2614	0.2013	0.2323	0.2612	0.2017	7.743	7.919	7.029	0.6960	0.6926	0.6991	0.5445	0.5907	0.5006
afrl	AFRL_u2e_cn_primary	0.2235	0.2526	0.1904	0.2238	0.2526	0.1907	7.957	8.123	7.157	0.6537	0.6483	0.6584	0.5396	0.5815	0.4996
upc-lsi	UPC.LSI_u2e_cn_primary	0.1971	0.2325	0.1610	0.1972	0.2324	0.1612	7.510	7.945	6.361	0.6607	0.6526	0.6680	0.5015	0.5530	0.4524
cmu-ebmt	cmu_cunei_u2e_cn_primary^(1)	0.1863	0.2122	0.1568	0.1863	0.2121	0.1568	6.811	6.899	6.065	0.6977	0.7001	0.6956	0.5274	0.5801	0.4773
hongkong	CityU_u2e_cn_primary	0.0254	0.0291	0.0215	0.0256	0.0293	0.0218	2.647	2.788	2.430	1.1169	1.1565	1.0813	0.2929	0.3196	0.2676
UnConstrained Data Track
systran	SYSTRAN_u2e_un_primary	0.2555	0.3034	0.2051	0.2553	0.3032	0.2050	8.203	8.622	7.090	0.6310	0.6060	0.6534	0.5401	0.5913	0.4921

⁽¹⁾A late and/or debugged system was also submitted, not reported here.