NIST 2009 Open Machine Translation Evaluation (MT09)
Official Release of Results

Date of release: Tue Oct 27 15:48:58 2009
Version: mt09_public_v1

Introduction

This release page is limited to the Current Test for the Urdu-to-English track.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Participants

The following table lists the submissions received from the sites participating in the Urdu-to-English Current Test.

Site ID

Organization

Location

Current Test Set, Urdu-to-English

Single System Track

System Combination Track

afrl

Air Force Research Laboratory

USA

Yes

-

amsterdam

University of Amsterdam

Netherlands

Late and/or debugged submission

-

cmu-ebmt

Carnegie Mellon EBMT

USA

Yes(1)

-

cmu-statxfer

Carnegie Mellon StatXfer

USA

Yes

Yes

hongkong

City University of Hong Kong

China

Yes

-

isi-lw

University of Southern California / Language Weaver Inc.

USA

Yes

Yes

jhu

Johns Hopkins University

USA

Late and/or debugged submission

-

systran

SYSTRAN Software Inc.

USA

Yes

-

umd

University of Maryland

USA

Yes

-

upc-lsi

UPC-LSI (Universitat Politècnica de Catalunya, Llenguatges i Sistemes Informàtics)

Spain

Yes

-

(1)A late and/or debugged system was also submitted, not reported here.

Current Test Results - Single System Track

Urdu-to-English (Table 1)

Site ID

System

BLEU-4 (mteval-v13a)

Overall

Newswire

Web

Constrained Data Track

isi-lw

isi-lw_u2e_cn_primary

0.3120

0.3753

0.2440

 

 

 

umd

UMD_u2e_cn_primary

0.2395

0.2697

0.2056

 

 

 

cmu-statxfer

CMU-Stat-Xfer_u2e_cn_primary

0.2322

0.2614

0.2013

 

 

 

afrl

AFRL_u2e_cn_primary

0.2235

0.2526

0.1904

 

 

 

upc-lsi

UPC.LSI_u2e_cn_primary

0.1971

0.2325

0.1610

 

 

 

cmu-ebmt

cmu_cunei_u2e_cn_primary(1)

0.1863

0.2122

0.1568

 

 

 

hongkong

CityU_u2e_cn_primary

0.0254

0.0291

0.0215

 

 

 

UnConstrained Data Track

systran

SYSTRAN_u2e_un_primary

0.2555

0.3034

0.2051

 

 

 

(1)A late and/or debugged system was also submitted, not reported here.

Current Test Results - System Combination Track

Urdu-to-English (Table 2)

Site ID

System

BLEU-4 (mteval-v13a)

Overall

Newswire

Web

Constrained Data Track

isi-lw

isi-lw_u2e_cn_combo1

0.3187

0.3816

0.2509

 

 

 

cmu-statxfer

CMU-Stat-Xfer_u2e_cn_combo1

0.2504

0.2910

0.2082

 

 

 

Current Test Results - Single System Track and System Combination Track - All metrics

Urdu-to-English (Table 3)

Site ID

System

BLEU-4 (mteval-v13a)

IBM BLEU (bleu-1.04)

NIST (mteval-v13a)

TER (tercom-0.7.25)

METEOR (meteor-0.7)

Overall

Newswire

Web

Overall

Newswire

Web

Overall

Newswire

Web

Overall

Newswire

Web

Overall

Newswire

Web

Constrained Data Track

isi-lw

isi-lw_u2e_cn_combo1

0.3187

0.3816

0.2509

0.3187

0.3815

0.2509

8.948

9.465

7.540

0.5866

0.5495

0.6199

0.5616

0.6237

0.5027

isi-lw

isi-lw_u2e_cn_primary

0.3120

0.3753

0.2440

0.3119

0.3752

0.2440

8.836

9.379

7.413

0.5845

0.5495

0.6159

0.5539

0.6160

0.4951

cmu-statxfer

CMU-Stat-Xfer_u2e_cn_combo1

0.2504

0.2910

0.2082

0.2504

0.2907

0.2084

8.244

8.550

7.218

0.6273

0.6128

0.6404

0.5380

0.5838

0.4947

umd

UMD_u2e_cn_primary

0.2395

0.2697

0.2056

0.2394

0.2696

0.2056

7.871

8.089

7.051

0.6519

0.6436

0.6593

0.5355

0.5815

0.4919

cmu-statxfer

CMU-Stat-Xfer_u2e_cn_primary

0.2322

0.2614

0.2013

0.2323

0.2612

0.2017

7.743

7.919

7.029

0.6960

0.6926

0.6991

0.5445

0.5907

0.5006

afrl

AFRL_u2e_cn_primary

0.2235

0.2526

0.1904

0.2238

0.2526

0.1907

7.957

8.123

7.157

0.6537

0.6483

0.6584

0.5396

0.5815

0.4996

upc-lsi

UPC.LSI_u2e_cn_primary

0.1971

0.2325

0.1610

0.1972

0.2324

0.1612

7.510

7.945

6.361

0.6607

0.6526

0.6680

0.5015

0.5530

0.4524

cmu-ebmt

cmu_cunei_u2e_cn_primary(1)

0.1863

0.2122

0.1568

0.1863

0.2121

0.1568

6.811

6.899

6.065

0.6977

0.7001

0.6956

0.5274

0.5801

0.4773

hongkong

CityU_u2e_cn_primary

0.0254

0.0291

0.0215

0.0256

0.0293

0.0218

2.647

2.788

2.430

1.1169

1.1565

1.0813

0.2929

0.3196

0.2676

UnConstrained Data Track

systran

SYSTRAN_u2e_un_primary

0.2555

0.3034

0.2051

0.2553

0.3032

0.2050

8.203

8.622

7.090

0.6310

0.6060

0.6534

0.5401

0.5913

0.4921

(1)A late and/or debugged system was also submitted, not reported here.