



\section{
\label{regression-testing.tex}
Regression Testing}


\subsection{
Goals}


The goal of regression testing is to ensure that any changes made to the decoder do not break what has been determined to be correct, previously.  The regression test suite is fast enough to run often, but still should provide adequate confidence that nothing substantial has changed about the internal workings of moses.  The regression test suite is designed to run on most UNIX-like systems. The regression test suite is run as part of the nightly build\footnote{\sf http://www.statmt.org/moses/cruise/}, so if you have problems with the regression tests you should first check if the nightly build succeeded.



\subsection{
Test suite}


The following regression tests are currently implemented (and {\bf many} more have been added since this list was written):

\begin{compactitem}
\item {\tt basic-surface-only} Tests basic translation, compares output strings and probabality scores.

\item {\tt basic-surface-binptable} Tests binary phrase table

\item {\tt consensus-decoding-surface} Basic test of consensus decoding

\item {\tt ptable-filtering} Tests the filtering of the phrase table by estimated phrase cost, ensures that the estimated phrase cost stays the same and that the same list of phrases is consistent.  Matches pharaoh.

\item {\tt multi-factor} Test that moses can do translation with two factors (Currently does a very basic test- it should be enhanced to at least include OOV words).

\item {\tt multi-factor-binptable} Tests factored setup with binary phrase table.

\item {\tt multi-factor-drop} Test of dropping words in a multi-factor model.

\item {\tt nbest-multi-factor} Tests n-best list generation for multi-factor models

\item {\tt n-best} Test n-best filtering, ensure consistency of top scores and score components. This will require ensuring that any moses binary is capable of generating n-best lists.

\item {\tt lattice-surface} Tests lattice decoding

\item {\tt lattice-distortion} Tests lattice decoding with distortion (?)

\item {\tt confusionNet-surface-only} Tests confusion network decoding

\item {\tt confusionNet-multi-factor} Tests confusion network decoding with multiple factors

\item {\tt lexicalized-reordering} Tests lexical reordering model

\item {\tt lexicalized-reordering-cn} Tests lexical reordering model in combination with confusion network

\item {\tt xml-markup} Tests XML Markup in input to specify translations

\end{compactitem}

\subsection{
Running the test suite}


Download the regression tests






\begin{SaveVerbatim}{myverb}
git clone https://github.com/moses-smt/moses-regression-tests.git
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

From the Moses root, run






\begin{SaveVerbatim}{myverb}
./bjam --with-irstlm=/path/to/irst --with-cmph=/path/to/cmph --with-regtest=/path/to/moses-regression-tests -j8
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

This will run the regression tests in parallel (-j8) so be sure to set a number of CPUs that your machine can handle.  



If all goes well, you will see a list of the tests run, their status (hopefully pass), and a path where the results are archived.



\subsection{
Running an individual test}


You can run a specific test by providing the name followed by ".passed"






\begin{SaveVerbatim}{myverb}
./bjam --with-irstlm=/path/to/irst --with-cmph=/path/to/cmph --with-regtest=/path/to/moses-regression-tests mert.basic.passed
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

The test name is the same as the directory name in /path/to/moses-regression-tests/tests .  



\subsection{
How it works}


The test suite invokes moses to decode a few sample phrases with well-known models.  The output from these invocations is then scraped for information (for example, the output translation of a sentence or its probability score) which is stored in a file called {\tt results.dat}.  These values are then compared to a ground truth, which was established either by hand, from a prior moses run, or from a pharaoh run.



This will provide a point-by-point analysis of each failure or success in the test as well as information.



{\bf Note}: Since the test suite relies on the output of moses, changes to the output format may result in broken tests.  If you make changes that affect presentation only, you will need to update the testing filters (which convert the raw moses output into the {\tt results.dat} format).



\subsection{
Writing regression tests}


Writing regression tests is easy, but since these tests must be able to be run anywhere, it is important to keep a few things in mind.  First, check out the regression-testing module from the Git repository.  Settle on what you would like to test in and choose a test name (henceforth, this name will be {\tt TEST-NAME}).  Create a directory for it under regression testing.



Place the following into the directory {\tt regression-testing/tests/TEST-NAME}:






\begin{compactitem}
\item {\tt to-translate}, which contains the text that will be translated by moses.

\item {\tt moses.ini}.  This moses.ini file should have {\em no absolute paths}.  All paths should be expressed in terms of the variables {\tt \$$\{$LM\_PATH$\}$} and {\tt \$$\{$MODELS\_PATH$\}$}.

\item The filter files, {\tt filter-stderr} and {\tt filter-stdout}.  These files should read from {\tt STDIN} and write results of the form {\tt KEY = value} to {\tt STDOUT}.  No other output should be generated.  Numeric values (such as times) that do not require exact matches can have the form {\tt KEY \textasciitilde{} value}.  These files are the trickiest part about writing a new regression test.  However, they allow great flexibility in verifying specific aspects of a decoding run.

\item {\tt truth/results.txt} This file should have the values (as produced by filter-stderr and filter-stdout) that are expected from the test run.

\end{compactitem}

If you need to add language models, phrase tables, generation tables or anything like this, you will need to increment the required data version number in {\tt MosesRegressionTesting.pm}.  Then, you will need to create a new {\tt .tgz} file that contains the data for all the tests (the data dependencies are not checked into the Git repository because they are extremely large).  This must then be made available for download.







\begin{center}\textit{\small Subsection last  modified on October 31, 2015, at 10:40 PM}\end{center}
