Project

General

Profile

Pair Detection » History » Version 4

Vassilis Papavassiliou, 2016-05-31 05:21 PM

1 1 Prokopis Prokopidis
# Pair Detection
2 1 Prokopis Prokopidis
3 1 Prokopis Prokopidis
It detects pairs of parallel documents based on website graph, specific patterns in URLs, occurrences of common images, similarity of sequences of digits and similarity of structure. For each detected pair, a cesAlign is generated. The basename of a cesAling file consists of the basenames of the paired documents and the identifier of the method which provided this pair (e.g. eng-12_ell-18_x.xml, where x stands for a, u, p, i, d, h, m and l). This file holds references to the paired documents.
4 1 Prokopis Prokopidis
5 1 Prokopis Prokopidis
```
6 3 Vassilis Papavassiliou
java -cp /opt/ilsp-fc/ilsp-fc-2.2.3-jar-with-dependencies.jar gr.ilsp.fc.bitext.PairDetector \
7 3 Vassilis Papavassiliou
-pdm "aupidh" -lang "en;it" -oxslt  -i (crawlpath up to the auto-generated xml dir) \
8 3 Vassilis Papavassiliou
-bs (fullpath and basename on which all files for easier content navigation will be generated)
9 1 Prokopis Prokopidis
 &>"/var/www/html/tests/eng-ita/log-pairdetect_www_esteri_it_eng-ita"
10 1 Prokopis Prokopidis
```
11 1 Prokopis Prokopidis
12 3 Vassilis Papavassiliou
## Options
13 3 Vassilis Papavassiliou
14 4 Vassilis Papavassiliou
```
15 3 Vassilis Papavassiliou
-pdm   : methods to be used for pair detection. Put a string which contains a for checking links, u for checking urls for patterns, p for combining common images and digits, i for using common images, d for examining digit sequences, h for examining structures.
16 1 Prokopis Prokopidis
17 2 Vassilis Papavassiliou
-ifp   : image_fullpath. Keep image fullpath for pair detection for representing an image instead of its name only.
18 1 Prokopis Prokopidis
19 1 Prokopis Prokopidis
-u_r   : url_replacements. Besides the default patterns, the user could add more patterns separated by ;
20 1 Prokopis Prokopidis
21 1 Prokopis Prokopidis
-del   : delete redundant files. Deletes cesDoc files that have not been paired				
22 4 Vassilis Papavassiliou
23 4 Vassilis Papavassiliou
```