Code Guide

Github, branching, and merging

If you want to code with Moses, you should create your own repository in one of a number of ways.

The preference is that you fork the repository if you're doing long-term research. If you fixed a bug, please commit it yourself, or create a pull request.

• Clone the moses github repository to your hard disk and work with it:

        git clone https://github.com/moses-smt/mosesdecoder.git mymoses
        cd mymoses
           edit files ....
        git commit -am "Check in"

You don't need a github login or permission to do this. All changes are stored on your own hard disk

• Clone AND branch the repository:

        git clone https://github.com/moses-smt/mosesdecoder.git mymoses
        cd mymoses
        git checkout -b mybranch
           edit files ....
        git commit -am "Check in"

You still don't need a github login or permission to do this.

• Clone and branch AND push to github:

        git clone https://github.com/moses-smt/mosesdecoder.git mymoses
        cd mymoses
        git checkout -b mybranch
           edit files ....
        git commit -am "Check in"
        git push origin mybranch
           edit files ....
        git commit -am "Check in again"
        git push

You need a github account. And you have to ask one of the Moses administrators to add you as a committer to the Moses repository.

NB. To delete a LOCAL branch:

    git branch -D old-branch

To delete a branch on the github server:

    git push origin --delete old-branch

• Fork the repository. You need a github account. You don't need permission from the Moses administrators. Log into github.com on their webiste, and go to the Moses page:

         https://github.com/moses-smt/mosesdecoder

Press the Fork button. This creates a new repository only you have write access to. Clone that repository and do whatever you want. Eg.

         git clone https://github.com/hieuhoang/mosesdecoder.git
         cd hieuhoang
           edit files ....
         git commit -am "Check in again"
         git push

• Clone and check into master

        git clone https://github.com/moses-smt/mosesdecoder.git mymoses
        cd mymoses
           edit files ....
        git commit -am "Check in"
        git push

You need a github account and write permission to the Moses repository.

• Create pull request. Fork a repository and read the instructions here:

        https://help.github.com/articles/using-pull-requests

Working with multiple branches

Assuming you've done Fork the repository, you can merge the latest changes from the main Moses repository with this command:

   git pull https://github.com/moses-smt/mosesdecoder.git

In your own repository, you can create branches and switch between them using

   git checkout -b new-branch
   edit files...
   git commit -am "check in"

   git checkout master
   edit files...
   ...

To get the latest changes from the main Moses repository to your branch, on your fork:

   git checkout master
   git pull https://github.com/moses-smt/mosesdecoder.git
   git checkout new-branch
   git merge master

Regression test

If you've changed any of the C++ code and intend to check into the main Moses repository, please run the regression test to make sure you haven't broken anything:

  git submodule init
  git submodule update
  ./bjam with-irstlm=... --with-srilm=... -a --with-regtest >& reg.out &

Check the output for any failures:

   grep FAIL reg.out

Contact the administrators

Contact Hieu Hoang or Barry Haddow, or any of the other administrators you might know, if you need help or permission to the github repository.

The code

This section gives a overview of the code. All the source code is commented for Doxygen, so you can browse a current snapshot of the source documentation. Moses is implemented using object-oriented principles, and you can get a good idea of its class organization from this documentation

The source code is in the following directories

• moses/util contains some shared utilities, such as code for reading and parsing files, and for hash tables. This was originally part of KenLM.
• moses/lm contains KenLM, Moses default language model.
• moses/src contains the code for the decoder
• moses/src/LM contains the language model wrappers used by the decoder
• Other subdirectories of moses/src contain some more specialised parts of the decoder, such as alternative chart decoding algorithms.
• moses-cmd/src contains code relevant to the command line version of the phrase-based decoder
• moses-cmd/src contains code relevant to the command line version of the chart-based decoder
• mert contains the code for the Moses mert implementation, originally described here

In the following, we provide a short walk-through of the decoder.

Quick Start

• The main function: moses-cmd/src/Main.cpp
• Initialize the decoder
  • moses/src/Parameter.cpp specifies parameters
  • moses/src/StaticData.cpp contains globals, loads tables
• Process a sentence
  • Manager.cpp implements the decoding algorithm
  • TranslationOptionCollection.cpp contains translation options
  • Hypothesis.cpp represents partial translation
  • HypothesisStack.cpp contain viable hypotheses, implements pruning
• Output results: moses-cmd/src/Main.cpp
  • moses-cmd/src/IOStream::OutputBestHypo print best translation
  • n-best lists generated in Manager.cpp, output in IOStream.cpp

Detailed Guides

• Style Guide
• Background on factor, word, and phrase data structure
• Chart decoder
• Multithreading
• Adding feature functions