WMT 2016

WMT is now a conference, but the shared task remains.

New languages are Turkish (hard!) and Romanian (easy). We also did not build Finnish systems last year, and threw out systems together with Edinburgh's in a joint submission.

Currently, some baseline systems [25]-[36] are being built. The web interface to experiments is accessible on CLSP machines under

 http://ndc06/gems/

If that does not work, you have to build a tunnel.

 ssh -fNL 11111:ndc06:80 [username]@login.clsp.jhu.edu

and then access the web site with

 http://localhost:11111/gems/

Cluster Advice

I spent a lot of time last year to run the training pipeline properly on the CLSP cluster.

The recommended usage is to call experiment.perl with the switch -cluster, so the scheduler submits jobs to the cluster.

You will see a number of specification in the config files like:

 [INTERPOLATED-LM]
 interpolate:qsub-settings = "-l 'arch=*64,mem_free=100G,ram_free=100G'"

These ensure that for specific jobs enough memory (and cores) are reserved.

The most challenging aspect is running the decoder on the massive models that are built for some of the language pairs. The recommended way to deal with that is to first copy the models to local disk and then start the decoder. This is done automatically with the cache-model specification in the config files.

To avoid that too many instances of the model files are copied on too many machines, the use of machines for decoding and tuning is restricted. For instance, run tuning only on the b0* machines:

 tune:qsub-settings = "-l 'hostname=b0*,arch=*64,mem_free=50G,ram_free=50G' -pe smp 16"

To Do

There are a bunch of things we could try:

1. morphological handling of Finnish and Turkish (baseline would be Morphessor) (Huda)
2. low resource things for Turkish
3. part of the official data are massive amounts of monolingual CommonCrawl - we may be able to use them all, or do subsampling (Amittai)
4. use of the neural LM / joint model that is already in Moses (Shuoyang, Gaurav)
5. re-ranking with an attention model (Shuoyang, Gaurav, Kevin)
6. some submission used additional data for Finnish, we may want to look into that as well - also for Romanian
7. syntax-based systems (Shuoyang)
8. anything else

Baselines

The first baseline uses all official data. The second baseline includes Brown clustering (which helped on average 0.5 BLEU last time around). This matches mostly the WMT 2015 system. A write-up of that system is here.

• JHU in 2015: final systems, most finished after submission deadline
• Best in 2015 according to the matrix.
• found a bug affecting reordering tables for factors other than surface, fixed in [26-2] for English-German

Results on official test