ILSP Focused Crawler

# Near Deduplication

Examines the cesDoc files in a directory and removes (near)duplicates.

It processes the exported cesDoc files (so the argument of option -i should be the crawlpath up to the "xml" directory) and detects near duplicates.

It also creates a text file (based on the argument of option -bs) with a list of the fullpaths of the remaining cesDoc files.

If asked (-oxslt option is used) an HTML file with links pointing to the xls transformations is generated too.

~~~

java -Dlog4j.configuration=file:/opt/ilsp-fc/log4j.xml -jar /opt/ilsp-fc/ilsp-fc-2.2.3-jar-with-dependencies.jar \

-dedup -lang "en;it" -oxslt -xslt \

-o (crawlpath up to the auto-generated xml dir)  \

-of (fullpath of file with paths of generated cesDoc) \

-ofh (fullpath of file with paths of generated transformed cesDoc) \

 &>"/var/www/html/tests/eng-ita/log-dedup_www_esteri_it_eng-ita"

~~~

-dedup		:	for (near) deduplication.

Some more supported parameters are explained in gr.ilsp.fc.dedup.DeduplicatorOptions

-m 		: Method type for deduplication: 1 for deduplication based on lists of words and quantized frequencies, 2 for deduplication based on common paragraphs, and 0 for applying both methods (default).

-mtl 		: minimum length of a token. Tokens with less than MIN_TOK_LEN (default is 3) are excluded from content and are not included into list of words.

-mpl 		: minimum length of a paragraph in tokens. Paragraphs with less than MIN_PAR_LEN (default is 3) tokens are excluded from content.

-ithr 	: intersection of paragraphs. Documents for which the ratio the common paragraphs

        		with the shortest of them is more than this threshold are considered duplicates

-int 		: inputType. Type of input files, default is xml, also supports txt.

-ex 		: exclude files. List of CesDocFiles (separated by ;) to be excluded from deduplication.