



\section{
\label{code-guide-ff.tex}
Adding Feature Functions}


{\em History:\\April 13th, 2012: Checked and revised for latest version (Barry Haddow)}



The log-linear model underlying statistical machine translation allows for the combination of several components that each weigh in on the quality of the translation. Each component is represented by one or more features, which are weighted, and multiplied together.



Formally, the probability of a translation {\bf e} of an input sentence {\bf f} is computed as


\begin{equation} p(\text{\bf e}|\text{\bf f}) = \prod_i h_i(\text{\bf e},\text{\bf f})^{\lambda_i} \end{equation}



where $\text{h}_\text{i}$ are the feature functions and $\text{$\lambda$}_\text{i}$ the corresponding weights.



Note that the decoder internally uses logs, so in fact what is computed is


\begin{equation} \text{log} \; p(\text{\bf e}|\text{\bf f}) = \sum_i \text{log}( h_i(\text{\bf e},\text{\bf f}) ) \; \lambda_i \end{equation}



The tuning (Section \ref{tuning.tex}) stage of the decoder is used to set the weights.



The following components are typically used:

\begin{compactitem}
\item phrase translation model (4 features, described here (Section \ref{training6.tex}))

\item language model (1 feature)

\item distance-based reordering model (1 feature)

\item word penalty (1 feature)

\item lexicalized reordering model (6 features, described here (Section \ref{training7.tex}))

\end{compactitem}

One way to attempt to improve the performance of the system is to add additional feature functions. This section explains what needs to be done to add a feature. Unless otherwise specified the Moses code files are in the directory {\tt moses/}. In the following we refer to the new feature as {\tt xxx}.



Side note: Adding a new component may imply that several scores are added. In the following, as in the Moses source code, we refer to both components and scores as {\em features}. So, a feature may have multiple features. Sorry about the confusion.



\subsection{
Video}


There is a 10 minute video demonstrating how to create your own feature function here\footnote{\sf http://www.youtube.com/watch?v=x-uo522bplw}.



\subsection{
Other resources}

\begin{SaveVerbatim}{myverb}
Kenton Murray has a nice blog on how to add a new feature function to Moses
http://kentonmurray.com/blogs/addingafeaturetomoses.html
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
Feature Function}


The feature computes one or more values, and we need to write a {\bf feature function} for it.



One important question about the new feature is, if it depends on just on the current phrase translation, or also on prior translation decision. We call the first case {\bf stateless}, the second {\bf stateful}. If the new feature is stateless, then it should inherit from the class {\tt StatelessFeatureFunction}, otherwise it should inherit from {\tt StatefulFeatureFunction}



The second case causes additional complications for the dynamic programming strategy of recombining hypotheses. If two hypotheses differ in their past translation decisions which matters for the new feature, then they cannot be recombined. 



For instance, the word penalty does only depend on the current phrase translation and is hence stateless. The distortion features also depend on the previous phrase translation and they are hence stateful. You can see the implementation of {\tt WordPenaltyProducer} and {\tt DistortionScoreProducer} in the directory {\tt moses/FF}.



However, new features are usually more complicated. For instance, it requires reading in a file and representing it with a data structure and more complex computations. See {\tt moses/LM/SRI.h} and {\tt moses/LM/SRI.cpp} for something more involved.



In the following, we assume such a more complex feature, which is implemented in its own source files {\tt XXX.h} and {\tt XXX.cpp}. The feature is implemented as a class which inherits from either {\tt StatefulFeatureFunction} or {\tt StatelessFeatureFunction}. So, you will write some code in {\tt XXX.h} that starts with






\begin{SaveVerbatim}{myverb}
namespace Moses
{
class XXX : public StatefulFeatureFunction {
...
}
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The class must contain the constructor:

\begin{SaveVerbatim}{myverb}
XXX::XXX(const std::string &line)
: StatefulFeatureFunction(line)
{ 
....
}
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The constructor must call the 

\begin{SaveVerbatim}{myverb}
StatelessFeatureFunction(...) or 
StatefulFeatureFunction(...) or 
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

or something that eventually calls one of this functions.



The constructor must also call the method

\begin{SaveVerbatim}{myverb}
ReadParameters()
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This is inherited from class FeatureFunction, it should NOT be overridden.



The {\em line} is the complete line from the ini file that instantiate this feature, eg.

\begin{SaveVerbatim}{myverb}
KENLM factor=0 order=5 num-features=1 lazyken=1 path=path/file
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The class must also contain the function:

\begin{SaveVerbatim}{myverb}
bool IsUseable(const FactorMask &mask) const;
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This function returns true if, given a target phrase only factors in mask, the feature can be evaluated. If the feature doesn't need to look at words in the target phrase, always return true. Return true if you don't use factors. 



A good example of IsUseable() is in 

\begin{SaveVerbatim}{myverb}
moses/LM/SingleFactor.cpp
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This is the only necessary method the class HAS to implement. All other methods are optional.



An important function to override is

\begin{SaveVerbatim}{myverb}
void Load(AllOptions::ptr const& opts)
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Override this function if the feature needs to load files. For example, language model classes load their LM files here. The first thing this function should do is save a pointer to the current set of options that is passed as the parameter:

\begin{SaveVerbatim}{myverb}
void Load(AllOptions::ptr const& opts) {
m_options = opts;
...
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Many feature function needs parameters to be passed in from the ini file. For example, 

\begin{SaveVerbatim}{myverb}
KENLM factor=0 order=5 num-features=1 lazyken=1 path=path/file
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

has the parameters {\em factor, order, num-features, lazyken, path}. To read in these parameters, override the method

\begin{SaveVerbatim}{myverb}
void FeatureFunction::SetParameter(const std::string& key, const std::string& value)
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

This method MUST call the same method in it's parent class if the parameter is unknown, eg.

\begin{SaveVerbatim}{myverb}
if (key == "input-factor") {
m_factorTypeSource = Scan<FactorType>(value);
} else {
StatelessFeatureFunction::SetParameter(key, value);
}
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The feature function needs to be registered in Factory.cpp, FeatureRegistry():

\begin{SaveVerbatim}{myverb}
#include "XXX.h"
...
MOSES_FNAME(XXX);
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
Stateless Feature Function}


The above is all that is required to a feature function. However, it doesn't do anything yet.



If the feature is stateless, it should override one of these methods from the class @FeatureFunction@@ :

\begin{SaveVerbatim}{myverb}
1. virtual void EvaluateInIsolation(const Phrase &source
, const TargetPhrase &targetPhrase
, ScoreComponentCollection &scoreBreakdown) const
2. virtual void EvaluateWithSourceContext(const InputType &input
, const InputPath &inputPath
, const TargetPhrase &targetPhrase
, const StackVec *stackVec
, ScoreComponentCollection &scoreBreakdown
, ScoreComponentCollection *estimatedFutureScore) const
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Or it can override one of these methods, specific to the {\tt StatelessFeatureFunction} class.

\begin{SaveVerbatim}{myverb}
3. virtual void EvaluateWhenApplied(const Hypothesis& hypo,
ScoreComponentCollection* accumulator) const
4. virtual void EvaluateWhenApplied(const ChartHypothesis &hypo,
ScoreComponentCollection* accumulator) const
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Usually, method (1) should be overridden. See {\tt WordPenaltyProducer.cpp} for a simple example using (1). 



{\em Note} - Only scores evaluted in (1) is included in future cost estimation in phrase-based model



Some stateless feature functions needs to know the entire input sentence to evaluate, for example. a bag of word feature. In this case, use method (2). 



Use method (3) or (4) if the feature function requires the segmentation of the source, or any other information available from the context. Note - these methods are identical to the those used by stateful features, except that they don't return state.



Each stateless feature function can override 1 or more of the above methods. So far (June, 2013) all stateless feature override only 1 method. 



The methods are called at different stages in the decoding process. 

\begin{compactitem}
\item (1) is called before the search process, when the translation rule is created. This could be when the phrase-table is loaded (in the case of memory-based phrase-table), or just before the search begin for binary phrase tables.

\item (2) is called just before the search begins.

\item (3) and (4) are called during search when hypotheses are created. 

\end{compactitem}

\subsection{
Stateful Feature Function}


Stateful feature functions should inherit from class {\tt StatefulFeatureFunction}. There are 2 class methods that can be overridden by the feature functions to score hypotheses:

\begin{SaveVerbatim}{myverb}
5.  virtual FFState* EvaluateWhenApplied(
const Hypothesis& cur_hypo,
const FFState* prev_state,
ScoreComponentCollection* accumulator) const = 0;

6. virtual FFState* EvaluateWhenApplied(
const ChartHypothesis& /* cur_hypo */,
int /* featureID - used to index the state in the previous hypotheses */,
ScoreComponentCollection* accumulator) const = 0;
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

As the names suggest, (5) is used to score a hypothesis from a phrase-based model. (6) is used to score 1 from the hierarchical/syntax model.



In addition, a stateful feature function can also override methods (1) and (2) from the base {\tt FeatureFunction} class. 



For example, language models are stateful. All language model implementation should override (5) and (6). However, they should also override (1) to score the translation rule in isolation. See classes {\tt LanguageModelImplementation} and {\tt LanguageModel} for the implementation of scoring language models.



Stateful feature function must also implement

\begin{SaveVerbatim}{myverb}
const FFState* EmptyHypothesisState() const
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsection{
Place-holder features}


Some features don't implement any Evaluate() functions. Their evaluation is more interwoven with the creation of the translation rule, the feature function is just used as a placeholder where the scores should be added.



Phrase-table (class PhraseDictionary), generation model (class GenerationDictionary), unknown word feature (class UnknownWordPenaltyProducer), and input scores for confusion networks and lattices (class InputFeature).



\subsection{
moses.ini}


All feature functions are specified in the {\tt [feature]} section. It should be in the format:

\begin{SaveVerbatim}{myverb}
* Feature-name key1=value1 key2=value2 ....
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

For example, 

\begin{SaveVerbatim}{myverb}
KENLM factor=0 order=3 num-features=1 lazyken=0 path=file.lm.gz
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

Keys must be unique. There must be a key 

\begin{SaveVerbatim}{myverb}
* num-features=??
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

which specifies the number of dense scores for this feature. 



The key

\begin{SaveVerbatim}{myverb}
* name=??
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

is optional. If it is specified, the feature name must be unique. If it is not specified, then a name is automatically created. All other key/value pairs are up to the feature function implementation.



\subsection{
Examples}


The struck-out examples are formatted in the old Moses v.1, and before. The clear examples are for current Moses in github.



NB. moses.ini files in the old format can still be read by the new decoder, if they just contain the common, vanilla features (ie. no sparse features, suffix arrays, or new features that have recently been added).



NB. 2 - Do NOT mix the old and new format in 1 ini file.



\subsubsection*{
Phrase-tables}


{\bf In-memory phrase-table (phrase-based):}

\begin{SaveVerbatim}{myverb}
PhraseDictionaryMemory num-features=5 path=phrase-table.gz input-factor=0 output-factor=0 table-limit=20
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}

Note - The old method is relaxed about whether you add '.gz' to the file name; it will try it with and without and see what exists. The new method is strict - you MUST specify '.gz' if the file ends with .gz, otherwise you must NOT specify '.gz'



{\bf Binary phrase-table (phrase-based):}

\begin{SaveVerbatim}{myverb}
PhraseDictionaryBinary num-features=5 path=phrase-table.gz input-factor=0 output-factor=0 
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}

Note - the binary phrase table consist of 5 files with the following suffixes:

\begin{SaveVerbatim}{myverb}
binphr.idx
binphr.srcvoc
binphr.tgtvoc
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

and (without word alignment):

\begin{SaveVerbatim}{myverb}
binphr.srctree 
binphr.tgtdata
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

or (WITH word alignment)

\begin{SaveVerbatim}{myverb}
binphr.srctree.wa 
binphr.tgtdata.wa
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The path value must point to the PREFIX of the files. For example, if the files are called:

\begin{SaveVerbatim}{myverb}
folder/pt.binphr.idx, folder/pt.binphr.srcvoc, folder/pt.binphr.tgtvoc ....
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

then

\begin{SaveVerbatim}{myverb}
path=folder/pt
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

{\bf In-memory phrase-table (hierarchical/syntax):}

\begin{SaveVerbatim}{myverb}
PhraseDictionaryMemory num-features=5 path=phrase-table.gz input-factor=0 output-factor=0 table-limit=20
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}

See ''In-memory phrase-table (phrase-based) for notes.



{\bf On-disk phrase-table (hierarchical/syntax):}

\begin{SaveVerbatim}{myverb}
PhraseDictionaryOnDisk num-features=5 path=phrase-table.gz input-factor=0 output-factor=0 table-limit=20
\end{SaveVerbatim}
\begin{shaded}
\scriptsize
\UseVerbatim{myverb}
\end{shaded}

Note - the on-disk phrase-table consists of 5 files:

\begin{SaveVerbatim}{myverb}
Misc.dat
Source.dat
TargetColl.dat
TargetInd.dat
Vocab.dat
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The path value must point to the FOLDER in which these files are found.



\subsubsection*{
Language models}


{\bf SRILM:}

\begin{SaveVerbatim}{myverb}
SRILM factor=0 order=5 path=lm.gz
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

{\bf IRSTLM:}

\begin{SaveVerbatim}{myverb}
IRSTLM factor=0 order=5 path=lm.gz
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

{\bf KenLM:}

\begin{SaveVerbatim}{myverb}
KENLM factor=0 order=5 path=lm.gz
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

{\bf Lazy KenLM:}

\begin{SaveVerbatim}{myverb}
KENLM factor=0 order=5 path=lm.gz lazy=1
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
Reordering models}

\begin{SaveVerbatim}{myverb}
LexicalReordering num-features=6 type=msd-bidirectional-fe input-factor=0 output-factor=0 path=reordering-table.msd-bidirectional-fe.0.5.0-0.gz
\end{SaveVerbatim}
\begin{shaded}
\tiny
\UseVerbatim{myverb}
\end{shaded}

\subsubsection*{
Misc features}


New mose must have Distortion, WordPenalty, and UnknownWordPenalty explicitly in the list of feature functions. They require no arguments, ie. 

\begin{SaveVerbatim}{myverb}
[feature]
UnknownWordPenalty
WordPenalty
Distortion
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

In the old moses, they were implicitly added by the decoder.



\subsubsection*{
Sparse features}


There are lots of ad-hoc features are currently implemented. You must look at the code and ask the developer to see how to run them












\begin{center}\textit{\small Subsection last  modified on January 05, 2016, at 08:14 AM}\end{center}
