WMT 2015

The WMT 2015 evaluation campaign is a good excuse to build baseline systems that use all best known methods.

The somewhat original contribution is the use of word classes for all components, i.e.,

• language model (done before)
• operation sequence model (done before, see paper)
• sparse features (new!)
• reordering mode (new!)

This can be seen as a last desperate attempt to avoid the inevitable onslaught of neural networks by emulating one of their benefits: the pooling of evidence in more generalized representations.

It is also a useful exercise how to run such large scale experiments on the cluster

Lessons on cluster usage

Lesson 1: Store each language pair onto its own disk.

I build machine translation systems for 8 language pairs (not doing Finnish). I started out storing them all on /export/b10 which was fine at first, but then ran into serious trouble, when running many (10-20) processes that all access this disk heavily.

Especially decoding requires the reading of typically 50GB of model files (mostly LM) from disk. While at the same time, other processes write on disk (building translation tables), the disk becomes very very slow, which is even quite noticeable on the command line (starting vi taking 1 minute...).

Lesson 2 Not learned yet. Maybe spread out decoding over multiple processes, or not.

Use of a Grid Engine cluster allows for distributing decoder runs onto multiple machines. This seemed to work at first, but then ran into problems with starting up the Moses processes - at crunch time it took up to 5 hours to load the models.

There is not final verdict on this, since it is related to lesson 1. Maybe it is possible to have 5 processes using 20 cores each to run the decoder, but there will be load time / decoding time tradeoff: more processes, longer load time for each process, but faster decoding.

Lesson 3: Cluster configuration in EMS's config

Some steps require a lot of memory or multiple CPUs. This needs to be properly communicated to GridEngine.

Run experiment.perl with the -cluster switch and have the following settings in your config

• [GENERAL] qsub-settings = "-l 'arch=*64'"
• [LM] train:qsub-settings = "-l 'arch=*64,mem_free=30G,ram_free=30G'"
• [INTERPOLATED-LM] interpolate:qsub-settings = "-l 'arch=*64,mem_free=100G,ram_free=100G'"
• [TRAINING] run-giza:qsub-settings = "-l 'arch=*64,mem_free=10G,ram_free=10G' -pe smp 9"
• [TRAINING] run-giza-inverse:qsub-settings = "-l 'arch=*64,mem_free=10G,ram_free=10G' -pe smp 9"
• [TUNING] set jobs to appropriate number (maybe just 1)
• [TUNING] tune:qsub-settings = "-l 'arch=*64,mem_free=50G,ram_free=50G' -pe smp 20"
• [EVALUATION] set jobs to appropriate number (maybe just 1)
• [EVALUATION] decode:qsub-settings = "-l 'arch=*64,mem_free=50G,ram_free=50G' -pe smp 20"

Experiments

Start with hill-climbing to the desired setup. Note: mkcls for many classes gets very very slow. Worst case 2000 classes for 1 billion word French-English parallel corpus takes about a month (GIZA++ takes even longer, so what gives).