ILSP Focused Crawler

<?xml version="1.0" encoding="UTF-8"?>


yourmail@mail.com
www.youraddress.com



4
Minimum number of terms that must exist in clean
content of each web page in order to be stored.This number
is multiplied with the median value of the terms'weights and
the result is the threshold for the absolute relevance score.


4
Minimum unique terms that must exist in clean content


0.2
The absolute relevance score is divided by the length
(in terms of tokens) of the clean content of a document and the
calculated relative relevance score is compared with this value


4
Maximum depth to crawl before abandoning a specific path. Depth
is increased every time a link is extracted from a non-relevant web page.




Max number of urls to fetch per run
512


10000
Socket timeout in milliseconds(per URL)


10000
Connection timeout in milliseconds(per URL)


2
Max number of attempts to fetch a Web page before giving up


0
Min bytes-per-seconds for fetching a web page





<!--
-->
Accepted mime types


1500
delay in milliseconds between requests


531072
Max content size (bytes) for downloading a web page


512
Max fetch set size per run (Sets are made by URLs from the same host)


512
Max URLs from a specific host per run


32
Max number of fetching threads for each host



1000
Max web pages to fetch per host


5
Max number of redirects


600000
Max time to wait for Fetcher to get all URLs in a run