#include <TranslationOptionCollection.h>
Public Member Functions | |
TranslationOptionList * | GetTranslationOptionList (size_t startPos, size_t endPos) |
list of trans opt for a particular span | |
TranslationOptionList const * | GetTranslationOptionList (size_t startPos, size_t endPos) const |
virtual | ~TranslationOptionCollection () |
const InputType & | GetSource () const |
input sentence/confusion network | |
const std::vector< const Phrase * > & | GetUnknownSources () const |
List of unknowns (OOVs). | |
virtual void | CreateTranslationOptions () |
Create all possible translations from the phrase tables. | |
virtual bool | CreateTranslationOptionsForRange (const DecodeGraph &decodeStepList, size_t startPosition, size_t endPosition, bool adhereTableLimit, size_t graphInd)=0 |
Create translation options that exactly cover a specific input span. | |
virtual bool | HasXmlOptionsOverlappingRange (size_t startPosition, size_t endPosition) const |
Check if this range has XML options. | |
virtual bool | ViolatesXmlOptionsConstraint (size_t startPosition, size_t endPosition, TranslationOption *transOpt) const |
Check if a subsumed XML option constraint is satisfied. | |
virtual void | CreateXmlOptionsForRange (size_t startPosition, size_t endPosition) |
Create xml-based translation options for the specific input span. | |
virtual const SquareMatrix & | GetEstimatedScores () const |
returns future cost matrix for sentence | |
TranslationOptionList const * | GetTranslationOptionList (const Range &coverage) const |
list of trans opt for a particular span | |
const InputPathList & | GetInputPaths () const |
ttasksptr | GetTranslationTask () const |
TO_STRING () | |
Protected Member Functions | |
TranslationOptionCollection (ttasksptr const &ttask, InputType const &src) | |
void | CalcEstimatedScore () |
void | ProcessUnknownWord () |
Force a creation of a translation option where there are none for a particular source position. | |
virtual void | ProcessOneUnknownWord (const InputPath &inputPath, size_t sourcePos, size_t length=1, const ScorePair *inputScores=NULL) |
special handling of ONE unknown words. | |
void | Prune () |
pruning: only keep the top n (m_maxNoTransOptPerCoverage) elements */ | |
void | Sort () |
sort all trans opt in each list for cube pruning */ | |
void | Add (TranslationOption *translationOption) |
virtual void | ProcessUnknownWord (size_t sourcePos)=0 |
implemented by inherited class, called by this class | |
void | EvaluateWithSourceContext () |
void | EvaluateTranslationOptionListWithSourceContext (TranslationOptionList &) |
void | CacheLexReordering () |
void | GetTargetPhraseCollectionBatch () |
bool | CreateTranslationOptionsForRange (const DecodeGraph &decodeGraph, size_t startPos, size_t endPos, bool adhereTableLimit, size_t graphInd, InputPath &inputPath) |
void | SetInputScore (const InputPath &inputPath, PartialTranslOptColl &oldPtoc) |
Protected Attributes | |
ttaskwptr | m_ttask |
std::vector< std::vector < TranslationOptionList > > | m_collection |
InputType const & | m_source |
SquareMatrix | m_estimatedScores |
const size_t | m_maxNoTransOptPerCoverage |
const float | m_translationOptionThreshold |
size_t | m_max_phrase_length |
size_t | max_partial_trans_opt |
std::vector< const Phrase * > | m_unksrcs |
InputPathList | m_inputPathQueue |
Friends | |
std::ostream & | operator<< (std::ostream &out, const TranslationOptionCollection &coll) |
The precomputation of a collection of instances of such TranslationOption depends on the input condition, but they all are presented to decoding algorithm in the same form, using this class.
This is a abstract class, and cannot be instantiated directly. Instantiate 1 of the inherited classes instead, for a particular input type
Definition at line 63 of file TranslationOptionCollection.h.
Moses::TranslationOptionCollection::TranslationOptionCollection | ( | ttasksptr const & | ttask, | |
InputType const & | src | |||
) | [protected] |
constructor; since translation options are indexed by coverage span, the corresponding data structure is initialized here This fn should be called by inherited classe
Definition at line 54 of file TranslationOptionCollection.cpp.
References Moses::InputType::GetSize(), m_collection, and m_max_phrase_length.
Moses::TranslationOptionCollection::~TranslationOptionCollection | ( | ) | [virtual] |
destructor, clears out data structures
Definition at line 80 of file TranslationOptionCollection.cpp.
References m_inputPathQueue, and Moses::RemoveAllInColl().
void Moses::TranslationOptionCollection::Add | ( | TranslationOption * | translationOption | ) | [protected] |
Add translation option to the list
translationOption | translation option to be added |
Definition at line 579 of file TranslationOptionCollection.cpp.
References Moses::Range::GetEndPos(), Moses::InputType::GetSize(), Moses::TranslationOption::GetSourceWordsRange(), Moses::Range::GetStartPos(), m_collection, m_source, and UTIL_THROW_IF2.
Referenced by Moses::TranslationOptionCollectionLattice::CreateTranslationOptions(), Moses::TranslationOptionCollectionConfusionNet::CreateTranslationOptionsForRangeLEGACY(), Moses::TranslationOptionCollectionText::CreateXmlOptionsForRange(), and ProcessOneUnknownWord().
void Moses::TranslationOptionCollection::CacheLexReordering | ( | ) | [protected] |
Definition at line 615 of file TranslationOptionCollection.cpp.
References Moses::InputType::GetSize(), m_collection, m_source, Moses::LexicalReordering::SetCache(), and stop.
Referenced by Moses::TranslationOptionCollectionLattice::CreateTranslationOptions(), and CreateTranslationOptions().
void Moses::TranslationOptionCollection::CalcEstimatedScore | ( | ) | [protected] |
compute future score matrix in a dynamic programming fashion. This matrix used in search. Call this function once translation option collection has been filled with translation options
Definition at line 251 of file TranslationOptionCollection.cpp.
References Moses::TranslationOptionList::begin(), count, Moses::TranslationOptionList::end(), Moses::TranslationOption::GetFutureScore(), Moses::SquareMatrix::GetScore(), Moses::InputType::GetSize(), IFVERBOSE, Moses::SquareMatrix::InitTriangle(), m_collection, m_estimatedScores, m_source, Moses::SquareMatrix::SetScore(), Moses::TranslationOptionList::size(), and TRACE_ERR.
Referenced by Moses::TranslationOptionCollectionLattice::CreateTranslationOptions(), and CreateTranslationOptions().
void Moses::TranslationOptionCollection::CreateTranslationOptions | ( | ) | [virtual] |
Create all possible translations from the phrase tables.
Create all possible translations from the phrase tables for a particular input sentence. This implies applying all translation and generation steps. Also computes future cost matrix.
Reimplemented in Moses::TranslationOptionCollectionConfusionNet, Moses::TranslationOptionCollectionLattice, and Moses::TranslationOptionCollectionText.
Definition at line 334 of file TranslationOptionCollection.cpp.
References CacheLexReordering(), CalcEstimatedScore(), CreateTranslationOptionsForRange(), EvaluateWithSourceContext(), Moses::DecodeGraph::GetBackoff(), Moses::StaticData::GetDecodeGraphs(), Moses::InputType::GetSize(), Moses::StaticData::Instance(), m_collection, m_max_phrase_length, m_source, ProcessUnknownWord(), Prune(), Sort(), and VERBOSE.
Referenced by Moses::TranslationOptionCollectionConfusionNet::CreateTranslationOptions(), Moses::Manager::Decode(), and Moses::TrainingTask::Run().
virtual bool Moses::TranslationOptionCollection::CreateTranslationOptionsForRange | ( | const DecodeGraph & | decodeStepList, | |
size_t | startPosition, | |||
size_t | endPosition, | |||
bool | adhereTableLimit, | |||
size_t | graphInd | |||
) | [pure virtual] |
Create translation options that exactly cover a specific input span.
Implemented in Moses::TranslationOptionCollectionConfusionNet, Moses::TranslationOptionCollectionLattice, and Moses::TranslationOptionCollectionText.
bool Moses::TranslationOptionCollection::CreateTranslationOptionsForRange | ( | const DecodeGraph & | decodeGraph, | |
size_t | startPos, | |||
size_t | endPos, | |||
bool | adhereTableLimit, | |||
size_t | graphInd, | |||
InputPath & | inputPath | |||
) | [protected] |
Definition at line 385 of file TranslationOptionCollection.cpp.
References Moses::DecodeGraph::begin(), Moses::PartialTranslOptColl::DetachAll(), Moses::DecodeGraph::end(), Moses::PartialTranslOptColl::GetList(), Moses::DecodeStep::GetPhraseDictionaryFeature(), Moses::PartialTranslOptColl::GetPrunedCount(), Moses::InputPath::GetTargetPhrases(), UTIL_THROW_IF2, Moses::XmlConstraint, Moses::XmlExclusive, and Moses::XmlPassThrough.
Referenced by CreateTranslationOptions(), Moses::TranslationOptionCollectionText::CreateTranslationOptionsForRange(), Moses::TranslationOptionCollectionConfusionNet::CreateTranslationOptionsForRangeNew(), and ProcessUnknownWord().
void Moses::TranslationOptionCollection::CreateXmlOptionsForRange | ( | size_t | startPosition, | |
size_t | endPosition | |||
) | [virtual] |
Create xml-based translation options for the specific input span.
Populates the current Collection with XML options exactly covering the range specified. Default implementation does nothing. called by CreateTranslationOptionsForRange()
sPos | first position in input sentence | |
lastPos | last position in input sentence |
Reimplemented in Moses::TranslationOptionCollectionText.
Definition at line 571 of file TranslationOptionCollection.cpp.
Referenced by Moses::TranslationOptionCollectionConfusionNet::CreateTranslationOptionsForRangeLEGACY().
void Moses::TranslationOptionCollection::EvaluateTranslationOptionListWithSourceContext | ( | TranslationOptionList & | translationOptionList | ) | [protected] |
Definition at line 503 of file TranslationOptionCollection.cpp.
References Moses::FeatureFunction::EvaluateTranslationOptionListWithSourceContext(), Moses::FeatureFunction::GetFeatureFunctions(), Moses::StaticData::Instance(), Moses::StaticData::IsFeatureFunctionIgnored(), and m_source.
Referenced by EvaluateWithSourceContext().
void Moses::TranslationOptionCollection::EvaluateWithSourceContext | ( | ) | [protected] |
Definition at line 490 of file TranslationOptionCollection.cpp.
References Moses::TranslationOptionList::begin(), Moses::TranslationOptionList::end(), EvaluateTranslationOptionListWithSourceContext(), Moses::InputType::GetSize(), m_collection, and m_source.
Referenced by CreateTranslationOptions().
virtual const SquareMatrix& Moses::TranslationOptionCollection::GetEstimatedScores | ( | ) | const [inline, virtual] |
returns future cost matrix for sentence
Definition at line 166 of file TranslationOptionCollection.h.
References m_estimatedScores.
Referenced by Moses::SearchCubePruning::CreateForwardTodos(), and Moses::SearchNormal::ExpandAllHypotheses().
const InputPathList& Moses::TranslationOptionCollection::GetInputPaths | ( | ) | const [inline] |
const InputType& Moses::TranslationOptionCollection::GetSource | ( | ) | const [inline] |
input sentence/confusion network
Definition at line 132 of file TranslationOptionCollection.h.
References m_source.
Referenced by Moses::DecodeStepTranslation::ProcessLEGACY().
void Moses::TranslationOptionCollection::GetTargetPhraseCollectionBatch | ( | ) | [protected] |
Definition at line 651 of file TranslationOptionCollection.cpp.
References Moses::DecodeGraph::begin(), Moses::DecodeGraph::end(), Moses::StaticData::GetDecodeGraphs(), Moses::PhraseDictionary::GetTargetPhraseCollectionBatch(), Moses::StaticData::Instance(), m_inputPathQueue, and m_ttask.
Referenced by Moses::TranslationOptionCollectionText::CreateTranslationOptions(), and Moses::TranslationOptionCollectionLattice::CreateTranslationOptions().
TranslationOptionList const* Moses::TranslationOptionCollection::GetTranslationOptionList | ( | const Range & | coverage | ) | const [inline] |
list of trans opt for a particular span
Definition at line 172 of file TranslationOptionCollection.h.
References Moses::Range::GetEndPos(), Moses::Range::GetStartPos(), and GetTranslationOptionList().
TranslationOptionList const * Moses::TranslationOptionCollection::GetTranslationOptionList | ( | size_t | startPos, | |
size_t | endPos | |||
) | const |
Definition at line 641 of file TranslationOptionCollection.cpp.
References m_collection, NULL, and UTIL_THROW_IF2.
TranslationOptionList * Moses::TranslationOptionCollection::GetTranslationOptionList | ( | size_t | startPos, | |
size_t | endPos | |||
) |
list of trans opt for a particular span
Definition at line 631 of file TranslationOptionCollection.cpp.
References m_collection, NULL, and UTIL_THROW_IF2.
Referenced by Moses::SearchCubePruning::CreateForwardTodos(), Moses::VW::EvaluateWhenApplied(), Moses::SearchNormal::ExpandAllHypotheses(), GetTranslationOptionList(), Moses::operator<<(), Moses::SearchNormal::ProcessOneHypothesis(), and ProcessUnknownWord().
ttasksptr Moses::TranslationOptionCollection::GetTranslationTask | ( | ) | const [inline] |
Definition at line 180 of file TranslationOptionCollection.h.
References m_ttask.
Referenced by ProcessOneUnknownWord().
const std::vector<const Phrase*>& Moses::TranslationOptionCollection::GetUnknownSources | ( | ) | const [inline] |
List of unknowns (OOVs).
Definition at line 137 of file TranslationOptionCollection.h.
References m_unksrcs.
Referenced by Moses::Manager::OutputUnknowns().
bool Moses::TranslationOptionCollection::HasXmlOptionsOverlappingRange | ( | size_t | startPosition, | |
size_t | endPosition | |||
) | const [virtual] |
Check if this range has XML options.
Check if this range overlaps with any XML options. This doesn't need to be an exact match, only an overlap. by default, we don't support XML options. subclasses need to override this function. called by CreateTranslationOptionsForRange()
sPos | first position in input sentence | |
lastPos | last position in input sentence |
Reimplemented in Moses::TranslationOptionCollectionText.
Definition at line 546 of file TranslationOptionCollection.cpp.
Referenced by Moses::TranslationOptionCollectionConfusionNet::CreateTranslationOptionsForRangeLEGACY().
void Moses::TranslationOptionCollection::ProcessOneUnknownWord | ( | const InputPath & | inputPath, | |
size_t | sourcePos, | |||
size_t | length = 1 , |
|||
const ScorePair * | inputScores = NULL | |||
) | [protected, virtual] |
special handling of ONE unknown words.
special handling of ONE unknown words. Either add temporarily add word to translation table, or drop the translation. This function should be called by the ProcessOneUnknownWord() in the inherited class At the moment, this unknown word handler is a bit of a hack, if copies over each factor from source to target word, or uses the 'UNK' factor. Ideally, this function should be in a class which can be expanded upon, for example, to create a morphologically aware handler.
sourceWord | the unknown word | |
sourcePos | ||
length | length covered by this word (may be > 1 for lattice input) | |
inputScores | a set of scores associated with unknown word (input scores from latties/CNs) |
Definition at line 174 of file TranslationOptionCollection.cpp.
References Add(), Moses::FactorCollection::AddFactor(), Moses::Phrase::AddWord(), Moses::ScoreComponentCollection::Assign(), EPSILON, Moses::TargetPhrase::EvaluateInIsolation(), StringPiece::find_first_of(), Moses::FloorScore(), Moses::PhraseDictionary::GetColl(), Moses::InputPath::GetPhrase(), Moses::TargetPhrase::GetScoreBreakdown(), Moses::Factor::GetString(), GetTranslationTask(), Moses::Phrase::GetWord(), Moses::FactorCollection::Instance(), Moses::UnknownWordPenaltyProducer::Instance(), m_unksrcs, NULL, range, Moses::TargetPhrase::SetAlignmentInfo(), Moses::TranslationOption::SetInputPath(), Moses::Word::SetIsOOV(), Moses::TransformScore(), and UNKNOWN_FACTOR.
Referenced by Moses::TranslationOptionCollectionLattice::CreateTranslationOptions(), Moses::TranslationOptionCollectionText::ProcessUnknownWord(), and Moses::TranslationOptionCollectionConfusionNet::ProcessUnknownWord().
virtual void Moses::TranslationOptionCollection::ProcessUnknownWord | ( | size_t | sourcePos | ) | [protected, pure virtual] |
implemented by inherited class, called by this class
Implemented in Moses::TranslationOptionCollectionConfusionNet, Moses::TranslationOptionCollectionLattice, and Moses::TranslationOptionCollectionText.
void Moses::TranslationOptionCollection::ProcessUnknownWord | ( | ) | [protected] |
Force a creation of a translation option where there are none for a particular source position.
Force a creation of a translation option where there are none for a particular source position. ie. where a source word has not been translated, create a translation option by 1. not observing the table limits on phrase/generation tables 2. using the handler ProcessUnknownWord() Call this function once translation option collection has been filled with translation options
This function calls for unknown words is complicated by the fact it must handle different input types. The call stack is Base::ProcessUnknownWord() Inherited::ProcessUnknownWord(position) Base::ProcessOneUnknownWord()
Definition at line 130 of file TranslationOptionCollection.cpp.
References CreateTranslationOptionsForRange(), Moses::StaticData::GetDecodeGraphs(), Moses::InputType::GetSize(), GetTranslationOptionList(), Moses::StaticData::Instance(), m_source, m_ttask, and Moses::TranslationOptionList::size().
Referenced by CreateTranslationOptions().
void Moses::TranslationOptionCollection::Prune | ( | ) | [protected] |
pruning: only keep the top n (m_maxNoTransOptPerCoverage) elements */
Definition at line 87 of file TranslationOptionCollection.cpp.
References Moses::InputType::GetSize(), m_collection, m_maxNoTransOptPerCoverage, m_source, m_translationOptionThreshold, Moses::TranslationOptionList::PruneByThreshold(), Moses::TranslationOptionList::SelectNBest(), Moses::TranslationOptionList::size(), and VERBOSE.
Referenced by Moses::TranslationOptionCollectionLattice::CreateTranslationOptions(), and CreateTranslationOptions().
void Moses::TranslationOptionCollection::SetInputScore | ( | const InputPath & | inputPath, | |
PartialTranslOptColl & | oldPtoc | |||
) | [protected] |
Definition at line 471 of file TranslationOptionCollection.cpp.
References Moses::InputPath::GetInputScore(), Moses::PartialTranslOptColl::GetList(), Moses::TranslationOption::GetScoreBreakdown(), Moses::InputFeature::InstancePtr(), NULL, and Moses::ScoreComponentCollection::PlusEquals().
void Moses::TranslationOptionCollection::Sort | ( | ) | [protected] |
sort all trans opt in each list for cube pruning */
Definition at line 520 of file TranslationOptionCollection.cpp.
References Moses::TranslationOptionList::begin(), Moses::TranslationOptionList::end(), Moses::InputType::GetSize(), m_collection, m_source, and sort().
Referenced by Moses::TranslationOptionCollectionLattice::CreateTranslationOptions(), and CreateTranslationOptions().
Moses::TranslationOptionCollection::TO_STRING | ( | ) |
bool Moses::TranslationOptionCollection::ViolatesXmlOptionsConstraint | ( | size_t | startPosition, | |
size_t | endPosition, | |||
TranslationOption * | transOpt | |||
) | const [virtual] |
Check if a subsumed XML option constraint is satisfied.
Check if an option conflicts with any constraint XML options. Okay, if XML option is substring in source and target. by default, we don't support XML options. subclasses need to override this function. called by CreateTranslationOptionsForRange()
sPos | first position in input sentence | |
lastPos | last position in input sentence |
Reimplemented in Moses::TranslationOptionCollectionText.
Definition at line 559 of file TranslationOptionCollection.cpp.
std::ostream& operator<< | ( | std::ostream & | out, | |
const TranslationOptionCollection & | coll | |||
) | [friend] |
Definition at line 598 of file TranslationOptionCollection.cpp.
std::vector< std::vector< TranslationOptionList > > Moses::TranslationOptionCollection::m_collection [protected] |
Definition at line 69 of file TranslationOptionCollection.h.
Referenced by Add(), CacheLexReordering(), CalcEstimatedScore(), CreateTranslationOptions(), EvaluateWithSourceContext(), GetTranslationOptionList(), Prune(), Sort(), and TranslationOptionCollection().
Definition at line 71 of file TranslationOptionCollection.h.
Referenced by CalcEstimatedScore(), and GetEstimatedScores().
Definition at line 77 of file TranslationOptionCollection.h.
Referenced by Moses::TranslationOptionCollectionLattice::CreateTranslationOptions(), Moses::TranslationOptionCollectionLattice::Extend(), GetInputPaths(), GetTargetPhraseCollectionBatch(), Moses::TranslationOptionCollectionConfusionNet::TranslationOptionCollectionConfusionNet(), Moses::TranslationOptionCollectionText::TranslationOptionCollectionText(), and ~TranslationOptionCollection().
size_t Moses::TranslationOptionCollection::m_max_phrase_length [protected] |
Definition at line 74 of file TranslationOptionCollection.h.
Referenced by CreateTranslationOptions(), and TranslationOptionCollection().
const size_t Moses::TranslationOptionCollection::m_maxNoTransOptPerCoverage [protected] |
InputType const& Moses::TranslationOptionCollection::m_source [protected] |
Definition at line 70 of file TranslationOptionCollection.h.
Referenced by Add(), CacheLexReordering(), CalcEstimatedScore(), Moses::TranslationOptionCollectionLattice::CreateTranslationOptions(), CreateTranslationOptions(), Moses::TranslationOptionCollectionConfusionNet::CreateTranslationOptionsForRangeLEGACY(), Moses::TranslationOptionCollectionText::CreateXmlOptionsForRange(), EvaluateTranslationOptionListWithSourceContext(), EvaluateWithSourceContext(), GetSource(), Moses::TranslationOptionCollectionText::HasXmlOptionsOverlappingRange(), Moses::operator<<(), Moses::TranslationOptionCollectionConfusionNet::ProcessUnknownWord(), ProcessUnknownWord(), Prune(), Sort(), and Moses::TranslationOptionCollectionText::ViolatesXmlOptionsConstraint().
const float Moses::TranslationOptionCollection::m_translationOptionThreshold [protected] |
ttaskwptr Moses::TranslationOptionCollection::m_ttask [protected] |
Definition at line 68 of file TranslationOptionCollection.h.
Referenced by Moses::TranslationOptionCollectionConfusionNet::CreateTranslationOptionsForRangeLEGACY(), GetTargetPhraseCollectionBatch(), GetTranslationTask(), ProcessUnknownWord(), and Moses::TranslationOptionCollectionConfusionNet::TranslationOptionCollectionConfusionNet().
std::vector<const Phrase*> Moses::TranslationOptionCollection::m_unksrcs [protected] |
Definition at line 76 of file TranslationOptionCollection.h.
Referenced by GetUnknownSources(), and ProcessOneUnknownWord().
size_t Moses::TranslationOptionCollection::max_partial_trans_opt [protected] |
Definition at line 75 of file TranslationOptionCollection.h.