Near Deduplication » History » Version 5
Version 4 (Vassilis Papavassiliou, 2016-05-31 04:56 PM) → Version 5/6 (Vassilis Papavassiliou, 2016-05-31 05:10 PM)
# Near Deduplication
Examines the cesDoc files in a directory and removes (near)duplicates.
It processes the exported cesDoc files (so the argument of option -i should be the crawlpath up to the "xml" directory), directory) and detects near duplicates,
and discards the shortest (in terms of tokens) file in a pair of near duplicates.
It also creates a text file (based on the argument of option -bs) with a list of the fullpaths of the remaining cesDoc files.
If asked (-oxslt option is used) an HTML file with links pointing to the xls transformations is generated too.
~~~
java -Dlog4j.configuration=file:/opt/ilsp-fc/log4j.xml -jar /opt/ilsp-fc/ilsp-fc-2.2.3-jar-with-dependencies.jar \
-dedup -i (crawlpath up to the "xml" dir) -oxslt -bs (fullpath of file with paths of generated cesDoc) \
&>"/var/www/html/tests/eng-ita/log-dedup_www_esteri_it_eng-ita"
~~~
## Options
```
-dedup : for (near) deduplication.
-i : crawlpath up to the "xml" dir generated by the export module
-bs : Basename to be used in generating all files for easier content navigation
-oxslt : Export crawl results with the help of an xslt file for better examination of results.
-ex : exclude files. List of CesDocFiles (separated by ;) to be excluded from deduplication.
```
## Other options
-m : Method type for deduplication: 1 for deduplication based on lists of words and quantized frequencies, 2 for deduplication based on common paragraphs, and 0 for applying both methods (default).
-mtl : minimum length of a token. Tokens with less than MIN_TOK_LEN (default is 3) are excluded from content and are not included into list of words.
-mpl : minimum length of a paragraph in tokens. Paragraphs with less than MIN_PAR_LEN (default is 3) tokens are excluded from content.
-ithr : intersection of paragraphs. Documents for which the ratio the common paragraphs
with the shortest of them is more than this threshold are considered duplicates
-int : inputType. Type of input files, default is xml, also supports txt.
-ex : exclude files. List of CesDocFiles (separated by ;) to be excluded from deduplication.
-bs : Basename to be used in generating all files for easier content navigation
-oxslt : Export crawl results with the help of an xslt file for better examination of results.
```
Examines the cesDoc files in a directory and removes (near)duplicates.
It processes the exported cesDoc files (so the argument of option -i should be the crawlpath up to the "xml" directory), directory) and detects near duplicates,
and discards the shortest (in terms of tokens) file in a pair of near duplicates.
It also creates a text file (based on the argument of option -bs) with a list of the fullpaths of the remaining cesDoc files.
If asked (-oxslt option is used) an HTML file with links pointing to the xls transformations is generated too.
~~~
java -Dlog4j.configuration=file:/opt/ilsp-fc/log4j.xml -jar /opt/ilsp-fc/ilsp-fc-2.2.3-jar-with-dependencies.jar \
-dedup -i (crawlpath up to the "xml" dir) -oxslt -bs (fullpath of file with paths of generated cesDoc) \
&>"/var/www/html/tests/eng-ita/log-dedup_www_esteri_it_eng-ita"
~~~
## Options
```
-dedup : for (near) deduplication.
-i : crawlpath up to the "xml" dir generated by the export module
-bs : Basename to be used in generating all files for easier content navigation
-oxslt : Export crawl results with the help of an xslt file for better examination of results.
-ex : exclude files. List of CesDocFiles (separated by ;) to be excluded from deduplication.
```
## Other options
-m : Method type for deduplication: 1 for deduplication based on lists of words and quantized frequencies, 2 for deduplication based on common paragraphs, and 0 for applying both methods (default).
-mtl : minimum length of a token. Tokens with less than MIN_TOK_LEN (default is 3) are excluded from content and are not included into list of words.
-mpl : minimum length of a paragraph in tokens. Paragraphs with less than MIN_PAR_LEN (default is 3) tokens are excluded from content.
-ithr : intersection of paragraphs. Documents for which the ratio the common paragraphs
with the shortest of them is more than this threshold are considered duplicates
-int : inputType. Type of input files, default is xml, also supports txt.
-ex : exclude files. List of CesDocFiles (separated by ;) to be excluded from deduplication.
-bs : Basename to be used in generating all files for easier content navigation
-oxslt : Export crawl results with the help of an xslt file for better examination of results.
```