



\section{
\label{web-translation.tex}
Translating Web pages with Moses}


{\em (Code and documentation written by Herve Saint-Amand.)}



We describe a small set of publicly available Perl scripts that provide the mechanisms to translate a Web page by retrieving it, extracting all sentences it contains, stripping them of any font style markup, translating them using the Moses system, re-inserting them in the document while preserving the layout of the page, and presenting the result to the user, providing a seamless translation system comparable to those offered by Google, BabelFish and others.



\subsection{
Introduction}


\subsubsection*{
Purpose of this program}


Moses is a cutting-edge machine translation program that reflects the latest developments in the area of statistical machine
translation research. It can be trained to translate between any two languages, and yields high quality results. However, the Moses
program taken alone can only translate plain text, i.e., text stripped of any formatting or style information (as in {\tt .txt}
files). Also, it only deals with a single sentence at a time.



A program that can translate Web pages is a very useful tool. However, Web pages contain a lot of formatting information,
indicating the color, font and style of each piece of text, along with its position in the global layout of the page. Most Web
pages also contain more than one sentence or independent segment of text. For these reasons a Web page cannot be fed directly to
Moses in the hope of obtaining a translated copy.



The scripts described in this document implement a Web page translation system that, at its core, relies on Moses for the actual
translation task. The scripts' job, given a Web page to translate, is to locate and extract all strings of text in the page, split
paragraphs into individual sentences, remove and remember any style information associated with the text, send the normalized,
plain-text string to Moses for translation, re-apply the style onto the translated text and re-insert the sentence at its place in
the original page structure, (hopefully) resulting in a translation of the original.



\subsubsection*{
A word of warning}


These scripts are a proof-of-concept type of demonstration, and should not be taken for more than that. They most probably still
contain bugs, and possibly even security holes. They are not appropriate for production environments.



\subsubsection*{
Intended audience and system requirements}


This document is meant for testers and system administrators who wish to install and use the scripts, and/or to understand how they work.



Before starting, the reader should ideally possess basic knowledge of:

\begin{compactitem}
\item UNIX-type command-line environments

\item TCP/IP networking (know what a hostname and a port are)

\item how to publish a Web page using a CGI script on an Apache server

\item how to configure and operate the Moses decoder

\end{compactitem}

and have the following resources available:






\begin{compactitem}
\item an Apache (or similar) Web server

\item the possibility of running CPU- and memory-intensive programs, either on the Web server itself (not recommended), or on one or several other machines that can be reached from the Web server

\item Moses installed on those machines

\end{compactitem}

\subsubsection*{
Overview of the architecture}




\begin{center}
\includegraphics[scale=1.11940298507463]{images/architecture.png}
\end{center}




The following is a quick overview of how the whole system works. An attempt at illustrating the architecture is in the figure above. File names refer to files available from an Git repository, as explained in the download section.






\begin{compactenum}\item The Moses system is installed and configured on one or several computers that we designate as {\bf Moses servers}.

\item On each Moses server, a daemon process, implemented by {\tt daemon.pl}, accepts network connections on a given port and copies everything it gets from those connections straight to Moses, sending back to the client what Moses printed back. This basically {\em plugs} Moses directly onto the network.

\item Another computer, which we designate as the {\bf web server}, runs Apache (or similar) Web server software.

\item Through that server, the CGI scripts discussed in this document ({\tt index.cgi}, {\tt translate.cgi} and supporting files) are served to the client, providing the user interface to the system. It is a simple matter to configure {\tt translate.cgi} so that it knows where the Moses servers are located.

\item A client requests {\tt index.cgi} via the Web server. A form containing a textbox is served back, where the user can enter a URL to translate.

\item That form is submitted to {\tt translate.cgi}, which does the bulk of the job. It fetches the page from the Web, extracts translatable plain text strings from it, sends those to the Moses servers for translation, inserts the translations back into the document, and serves the document back to the client. It adjusts links so that if any one is clicked in the translated document, a translated version will be fetched rather than the document itself.

\end{compactenum}
The script containing all the interesting code, {\tt translate.cgi}, is heavily commented, and programmers might be interested in
reading it.



\subsubsection*{
Mailing list}


Should you encounter problems you can't solve during the installation and operation of this program, you can write to the moses
support mailing list at {\tt moses-support@mit.edu}. Should you not encounter problems, the author (whose email is found in the
source file headers) would be astonished to hear about it.



\subsection{
Detailed setup instructions}


\subsubsection*{
Obtaining a copy of the scripts}


The scripts are stored in the {\tt contrib/web} directory in the Moses distribution.



\subsubsection*{
Setting up the Web server}


The extracted source code is ready to be run, there is no installation procedure that compiles or copies files. The program is
entirely contained within the directory that was downloaded from Github. It now needs to be placed on a Web server, in a properly
configured location such that the CGI scripts (the two files bearing the {\tt .cgi} extension) are executed when requested from a
browser.



For instance, if you are on a shared Web server (e.g., a server provided by your university) and your user directory contains a
directory named {\tt public\_html}, placing the {\tt moses-web} directory inside {\tt public\_html} should make it available via
the Web, at an address similar to {\tt {\sf http://www.dept.uni/\textasciitilde{}you/moses-web/}}.



\subsubsection*{
Troubleshooting}

\begin{compactitem}
\item {\bf 404 Not Found} Perhaps the source code folder is not in the right location? Double-check the directory names. See if the home folder (parent of {\tt moses-web} itself) is reachable. Ask your administrator.

\item {\bf 403 Forbidden}, or you see the Perl source code of the script in your browser$\}$ The server is not configured to execute CGI scripts in this directory. Move {\tt moses-web} to the {\tt cgi-bin} subdirectory of your Web home, if it exists. Create a {\tt .htacess} file in which you enable the {\tt ExecCGI} option (see the Apache documentation).

\item {\bf Internal server error} Perhaps the scripts do not have the right permissions to be executed. Go in {\tt moses-web} and type the command {\tt chmod 755 *cgi}.

\end{compactitem}

The scripts are properly installed once you can point your browser at the correct URL and you see the textbox in which you should
enter the URL, and the 'Translate' button. Pressing the button won't work yet, however, as the Moses servers need to be installed
and configured first.



\subsubsection*{
Setting up the Moses servers}


You now need to install Moses and the {\tt daemon.pl} script on at least one machine.



\subsubsection*{
Choosing machines for the Moses servers}


Running Moses is a slow and expensive process, at least when compared to the world of Web servers where everything needs to be
lightweight, fast and responsive. The machine selected for running the translator should have a recent, fast processor, and as many
GBs of memory as possible (see the Moses documentation for more details).



Technically, the translations could be computed on the same machine that runs the Web server. However, the loads that Moses places
on a system would risk seriously impacting the performance of the Web server. For that reason, we advise not running Moses on the
same computer as the Web server, especially not if the server is a shared server, where several users host their files (such as Web
servers typically provided by universities). In case of doubt we recommend you ask your local administrator.



For the sake of responsiveness, you may choose to run Moses on several machines at once. The burden of translation will then be
split equally among all the hosts, thus more or less diving the total translation time by the number of hosts used. If you have
several powerful computers at your disposal, simply repeat the installation instructions that follow on each of the machines
independently.



The main translation script, which runs on the Web server, will want to connect to the Moses servers via TCP/IP sockets. For this
reason, the Moses servers must be reachable from the Web server, either directly or via SSH tunnels of other proxy
mechanisms. Ultimately the translation script on the Web server must have a hostname/port address it can connect to for each Moses
server.



\subsubsection*{
Installing the scripts}


\paragraph*{
Install Moses}


For each Moses server, you will need to install and configure Moses for the language pair that you wish to use. If your Moses
servers are all identical in terms of hardware, OS and available libraries, installing and training Moses on one machine and then
copying the files over to the other ones should work, but your mileage may vary.



\paragraph*{
Install daemon.pl}


Once Moses is working, check out, on each Moses server, another copy of the {\tt moses-web} source directory by following again the
instructions in the download section. Open {\tt bin/daemon.pl}, and edit the {\tt \$MOSES} and {\tt \$MOSES\_INI} paths to
point to the location of your {\tt moses} binary and your {\tt moses.ini} configuration file.



\paragraph*{
Choose a port number}


Now you must choose a port number for the daemon process to listen on. Pick any number between 1,024 and 49,151, ideally not a
standard port for common programs and protocols to prevent interference with other programs (i.e., pick a port not mentioned in
your {\tt /etc/services} file).



\paragraph*{
Start the daemon}


To activate a Moses server, simply type, in a shell running on that server:






\begin{SaveVerbatim}{myverb}
./daemon.pl <hostname> <port>
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

where {\tt $<$hostname$>$} is the name of the host you're typing this on (found by issuing the {\tt hostname} command), and {\tt $<$port$>$} is the port you selected. It may be misleading that despite its name, this program does not fork a background process, it
is the background process itself. To truly launch the process in the background so that it continues running after the shell is
closed, this command might be more useful:






\begin{SaveVerbatim}{myverb}
nohup ./daemon.pl <hostname> <port> & 
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

The {\tt bin/start-daemon-cluster.pl} script distributed with this program provides an automation mechanism that worked well in the
original setup on the University of Saarland network. It was used to start and stop the Moses servers all at once, also setting up
SSH tunneling on startup. Because it is very simple and trimmed to the requirements of that particular installation, we do not
explain its use further here, but the reader might find inspiration in reading it.



\paragraph*{
Test the Moses servers}


The daemon should now be listening on the port you chose. When it receives a connection, it will read the input from that
connection one line at a time, passing each line in turn to Moses for translation, and printing back the translation followed by a
newline.



If you have the NetCat tool installed, you can test whether it worked by going to a shell on the Web server and typing {\tt echo "Hello world" $|$ nc $<$hostname$>$ $<$port$>$}, replacing {\tt Hello world} by a phrase in your source language if it is not English, and {\tt $<$hostname$>$} and {\tt $<$port$>$} by the values pointing to the Moses server you just set up. A translation should be printed back.



\paragraph*{
Configure the tokenizer}


The {\tt translate.cgi} script uses external tokenizer and detokenizer scripts. These scripts adapt their regular expressions depending on the
language parsed, and so tokenizing is improved if the correct language is selected. This is done by opening {\tt translate.cgi}
with your favourite text editor, and setting {\tt \$INPUT\_LANG} and {\tt \$OUTPUT\_LANG} to the appropriate language
codes. Currently the existing language codes are the file extensions found in the {\tt bin/nonbreaking\_prefixes} directory. If
yours are not there, simply use {\tt en} -{}- end-of-sentence detection may then be suboptimal, and translation quality may be
impacted, but the system will otherwise still function.



\paragraph*{
Configure the Web server to connect to the Moses servers}


The last remaining step is to tell the frontend Web server where to find the backend Moses servers. Still in {\tt translate.cgi},
set the {\tt @MOSES\_ADDRESSES} array to the list of {\tt hostname:port} strings identifying the Moses servers.



Here is a sample valid configuration for three Moses servers named {\tt server01}, {\tt server02} and {\tt server03}, each with the
daemon listening on port 7070:






\begin{SaveVerbatim}{myverb}
my @MOSES_ADDRESSES = ("server01:7070", "server02:7070", "server03:7070");
\end{SaveVerbatim}
\begin{shaded}
\small
\UseVerbatim{myverb}
\end{shaded}

\paragraph*{
Stopping the daemons once done}


The daemon processes continuously keep a copy of Moses running, so they consume memory even when idle. For this reason, we
recommend that you stop them once they are not needed anymore, for instance by issuing this command on each Moses server: {\tt killall daemon.pl}







\begin{center}\textit{\small Subsection last  modified on July 28, 2013, at 07:01 AM}\end{center}
