Finding viable seed URLs for web corpora: A scouting approach and comparative study of available sources

Adrien Barbaresi (1)

1 ICAR - Interactions, Corpus, Apprentissages, Représentations

The conventional tools of the "web as corpus" framework rely heavily on URLs obtained from search engines. Recently, the corresponding querying process became much slower or impossible to perform on a low budget. I try to find acceptable substitutes, i.e. viable link sources for web corpus construction. To this end, I perform a study of possible alternatives, including social networks as well as the Open Directory Project and Wikipedia. Four different languages (Dutch, French, Indonesian and Swedish) taken as examples show that complementary approaches are needed. My scouting approach using open-source software leads to a URL directory enriched with metadata which may be used to start a web crawl. This is more than a drop-in replacement for existing tools since said metadata enables researchers to filter and select URLs that fit particular needs, as they are classified according to their language, their length and a few other indicators such as host- and markup-based data.

Mots clés: web corpus construction; URL seeds; search engine queries; social networks; Open Directory Project; Wikipedia; Web document filtering; language identification; URL directory

Domaines: Linguistique; Informatique et langage [cs.CL]

Adrien Barbaresi. Finding viable seed URLs for web corpora: A scouting approach and comparative study of available sources. 9th Web as Corpus Workshop (WaC-9), 14th Conference of the European Chapter of the Association for Computational Linguistics, Apr 2014, Gothenburg, Sweden. pp.1-8. ⟨halshs-00986144⟩