ILSP Focused Crawler

# Introduction

ILSP Focused Crawler (ILSP-FC) is a research prototype for acquiring domain-specific monolingual and bilingual corpora. ILSP-FC integrates modules for text normalization, language identification, document clean-up, metadata extraction, text classification, identification of bitexts (documents that are translations of each other), alignment of segments, and filtering of segment pairs. A detailed description of each module is available at http://aclweb.org/anthology/W/W13/W13-2506.pdf .

# Workflows

The current version of ILSP-FC offers the user the option to [[GettingStarted|run all relevant processes]] in a pipeline or to select a specific subset (e.g. alignment and merging only).

In a configuration for acquiring monolingual data, ILSP-FC applies the following processes (one after the other):

- crawls the web until an expiration criterion is met (i.e. harvests webpages and stores the ones that are in the targeted language, and relevant to a targeted topic if required)
- exports the stored data (i.e. stores downloaded web pages/documents and for each page generates a CesDoc files with its content and metadata).
- discards (near) duplicate documents

In a configuration for acquiring parallel data, it applies the following processes (one after the other):

- [[Crawl|crawls]] a website with content in the targeted languages (i.e. harvests the website and stores pages that are in the targeted languages, and relevant to a targeted topic if required)
- [[Export|exports]] the stored data (i.e. stores downloaded web page/document and generates a CesDoc file with its content and metadata).
- discards (near) [[NearDeduplication|duplicate]] documents
- identifies [[PairDetection|pairs]] of (candidate) parallel documents and generates a cesAlign file for each detected pair.
- [[SegmentAlignment|aligns]] the segments in each detected document pair and generates a TMX for each document pair
- [[TMXmerging|merges]] TMX files corresponding to each document pair in order to create the final output, i.e. a TMX that includes all (or a selection of) segment pairs

# Input

In case of general monolingual crawls, the ILSP-FC travels across the web and stores web pages/documents with content in the targeted language. The required input from the user is:

- a list of seed URLs (i.e. a text file with one URL per text line).

In case of focused monolingual crawls (i.e. when the crawler visits/processes/stores web pages that are in the targeted language and related to a targeted domain), the input should include:

- a list of seed URLs pointing to relevant web pages. An example seed URL list for _Environment_ in English can be found at [[ENV_EN_seeds.txt]].
- a list of term triplets (_\<relevance,term,subtopic\>_) that describe a domain (i.e. this list is required in case the user aims to acquire domain-specific documents) and, optionally, subcategories of this domain. An example domain definition can be found at [[ENV_EN_topic.txt]] for the _Environment_ domain in English.

In case of general bilingual crawling, the input from the user includes:
- a seed URL list which should contain URL(s) from only one web site with content in both of the targeted languages(e.g. [[ENV_EN_ES_seed.txt]]). The crawler will follow only links pointing to pages inside this web site. Examples of seed URLs can be found at [[seed_examples.txt]].

In case of focused bilingual crawls, the input should also include:

- a list of term triplets (_\<relevance,term,subtopic\>_) that describe a domain (i.e. this list is required in case the user aims to acquire domain-specific documents) and, optionally, subcategories of this domain in both the targeted languages (i.e. the union of the domain definition in each language). An example domain definition of _Environment_ for the English-Spanish pair can be found at [[ENV_EN_ES_topic.txt]].
Note that in case a thematic website is targeted, it is very likely that examination of domainnesses could be avoid (i.e. construction and use of a list of terms that define the targeted topic might be redundant).

# Output

Each module of the tool provides its own output which feeds the next module in the pipeline:

[[Crawl|Crawl]]: Creates the "run" directories (i.e. directories containing all resources fetched/extracted/used/required for each cycle of this crawl). (See setting _dest_)
cesDoc (lists),
cesAlign (lists),
TMX (lists),
merged TMX (TMX, HTML, MD)

# Documentation

- [[HowToGet|How to get ILSP-FC]]: Learn about the different ways to get ILSP-FC
- [[DeveloperSetup|Developer Setup]]: Learn how to build ILSP-FC
- [[GettingStarted|Getting Started]]: Learn how to run ILSP-FC
- [[Languages-supported|Languages Supported]] Learn about supported languages
- [[Resources|Resources]] acquired with ILSP-FC