Challenges in web corpus construction for low-resource languages in a post-BootCaT world

Adrien Barbaresi (1)

1 ICAR - Interactions, Corpus, Apprentissages, Représentations

The state of the art tools of the "web as corpus" framework rely heavily on URLs obtained from search engines. Recently, this querying process has become very slow or impossible to perform on a low budget. In order to find reliable data sources for Indonesian, I perform a case study of different kinds of URL sources and crawling strategies. First, I classify URLs extracted from the Open Directory Project and Wikipedia for Indonesian, Malay, Danish, and Swedish in order to enable comparisons. Then I perform web crawls focusing on Indonesian and using the mentioned sources as the start URLs. My scouting approach using open-source software results in a URL database with metadata which can be used to replace or at least to complement the BootCaT approach.

Mots clés: web crawling; web corpus construction; under-resourced languages; Indonesian language; LRL

Domaines: Linguistique; Informatique et langage [cs.CL]; Web

Adrien Barbaresi. Challenges in web corpus construction for low-resource languages in a post-BootCaT world. 6th Language & Technology Conference, Less Resourced Languages special track, Dec 2013, Poznan, Poland. pp.69-73. ⟨halshs-00919410v2⟩