Project

General

Profile

Overview

ILSP Focused Crawler (ILSP-FC) is a research prototype for acquiring domain-specific monolingual and bilingual corpora. The required input from the user consists of a list of seed URLs pointing to relevant web pages and a list of terms that describe a topic. ILSP-FC integrates modules for text normalization, language identification, document clean-up, text classification, bilingual document alignment (i.e. identification of pairs of documents that are translations of each other) and sentence alignment. If the user does not provide a list of terms, the software can be used as a general crawler.

ILSP-FC is being developed by researchers of the ILSP/Athena RIC and currently being used in the European Language Resource Coordination effort, which launched in April 2015 and aims to identify and gather language and translation data relevant to national public services, administrations and governmental institutions across all 30 European countries participating in the Connecting Europe Facility (CEF) Programme.

An initial version of the crawler was produced during PANACEA, an EU FP7 project for the acquisition and production of Language Resources. It was then extended during the QTLaunchPad project, a European Commission-funded collaborative research initiative dedicated to overcoming quality barriers in machine and human translation and in language technologies; and the FP7-PEOPLE Abu-MaTran project, which seeks to enhance industry-academia cooperation in the adoption of machine translation technologies.

More information on how to get and use ILSP-FC can be found in the Documentation.

ILSP-FC is a Java project released under the GNU GPL, v. 3.0 license. It depends on open-source libraries for web mining and building data-processing workflows. If you would like to try ILSP-FC as a bilingual crawler before installing it, you can use this web service on small websites.

If you use ILSP-FC in scientific work, please cite: Papavassiliou, V., Prokopidis, P. & Gregor Thurmair. (2013). A modular open-source focused crawler for mining monolingual and bilingual corpora from the web. In Proceedings of the Sixth Workshop on Building and Using Comparable Corpora, pages 43-51. Sofia, Bulgaria : Association for Computational Linguistics (BibTeX)

The pair detection module of ILSP-FC was used for aligning documents in the WMT16 Bilingual Document Alignment Shared Task. The system reached a recall of 91% in the soft scoring setting prepared by the organizers. More details are presented in the system paper: Papavassiliou, V., Prokopidis, P., and Piperidis, S. (2016). The ILSP/ARC submission to the WMT 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation. Berlin, Germany: Association for Computational Linguistics (BibTeX)

Please send any questions to {vpapa, prokopis} at-ilsp-dot-gr.

Issue tracking

open closed Total
Bug 0 2 2
Feature 0 0 0
Support 0 1 1

View all issues