Tools and Resources for the Czech and Estonian Wordnets

Version 1, Final

January 15, 1999

Contributors:

Karel Pala, Pavel Ševeček, Masaryk University
Haldur  Õim, Kadri Vider, Leho Paldre, Heili Orav, Tartu University

Deliverable 2D006, WP2.1
EuroWordNet, LE4-8283

Identification number
	LE4-8283-2D-006
	
Type
	Document	
Title
	Tools and Resources for the Czech and Estonian Wordnets	
Status
	Final	
Deliverable
	2D006	
Work Package
	WP2	
Task
	T2.1	
Period covered
	July 1998 - September 1998	
Date
	January 15, 1999	
Version
	1	
Number of pages
	15	
Authors
	•	Karel Pala, Pavel Ševeček, Masaryk University Brno
•	Kadri Vider, Haldur Oim, Tartu University
	
WP/Task responsible	BRN	
Project contact point	 Piek Vossen
 Computer Linguistiek
 University of Amsterdam
 Spuistraat 134
 1012 VB Amsterdam
 The Netherlands
 tel.  +31 20 525 4669
 fax. +31 20 525 4429
 e-mail: Piek.Vossen@hum.uva.nl	
EC project officer	Ray Hudson	
Status
	Public	
Actual distribution
	Project Consortium, the EuroWordNet User Group, the world via http://www.let.uva.nl/~ewn
	
Suplementary notes
	n.a.	
Key words	Linguistic Resources, Multilingual Wordnets, Language Engineering
Czech, Estonian	
Abstract	This deliverable gives an overview of the tools and resources used by BRN and TAR to build the input to the EWN database (synsets, language-internal relations, relations to the ILI) for Czech and Estonian, respectively. It can be seen as a catalogue of the resources and tools that are available and/or have been used by either of the above mentioned sites. Information concerning required hardware and software platforms, availability, and distributor of each tool and resource is included.	
Status of the abstract
	Draft	
Received on	
	
Recipient’s catalogue number		

Executive Summary

This deliverable gives an overview of the tools and resources used by BRN and TAR to build the input to the EWN database (synsets, language-internal relations, relations to the ILI) for Czech and Estonian, respectively. It can be seen as a collection of tools and resources that are available and/or have been used by either of the above mentioned sites. Information concerning required hardware and software platforms, availability, and distributor of each tool and resource is included.
The first part of the document describes the Czech resources and tools. The main lexical resources used by BRN are machine Explanatory Dictionary of Czech (Výkladový slovník češtiny, about 55 000 headwords and 65 000 senses), Dictionary of Literary Czech (Slovník spisovné češtiny), a monolingual explanatory dictionary which also exists in the machine readable form (45 000 headwords and approximately 58 000 senses), Lingea Lexicon, v.2, a machine readable bilingual Czech-English-Czech dictionary containing at the present moment in the Czech-English direction about 54 000 entries and 58 000 senses, in the English-Czech direction about 78 000 entries and 102 400 senses, and having as its integral parts the machine stem dictionaries together with the lemmatizer for both mentioned languages. The last lexical resource is Dictionary of Czech Synonyms, by K.Pala and J.Vsiansky, containing approximately 22 000 synsets (entries) and existing in the electronic version with automatic lemmatization as well.
As supplementary lexical resources we have compiled the List of Czech Collocations containing approx. 18 000 items which has been obtained from the text corpus ESO (about 61 mil. word forms, built and mantained at the Faculty of Informatics), sorted according several criteria (MI score, frequencies, syntactic criteria – word order) and used for processing of the present-day Czech collocations. Further resource  is a fully tagged (both structurally and grammatically) corpus DESAM  that has been built at the Faculty of Informatics in the course of the last two years. It contains mostly newspaper and magazine texts from 1992-96 and its size is 1 mil. Czech word forms. The largest resource is the corpus ESO built at the Faculty of Informatics in the course 1998 from newspaper and magazine texts (1996-98), size 61 mil. Czech word forms, partially tagged (lemmatized).

The tools applied by BRN are used to exploit the information provided in all the mentioned dictionaries (e.g. to create equivalence links automatically), view and modify the resulting databases obtained from the corpora, and convert them to different formats.

The second part of the document describes the resources and tools used by TAR. The most important lexical resource is Explanatory Dictionary of Estonian,  followed by Dictionary of Synonyms and Dictionary of Antonyms. There is also Estonian Thesaurus, English Dictionary for Speakers of Estonian, English-Estonian-English Dictionary and Collection of Estonian Corpora.
The tools used by TAR are KeeleWeb, Thes, Polaris an Set of programs for Estonian Morphology.  

Table of Contents

1 Resources and tools  used by BRN 	
1.1 Lexical resources	
1.1.1 Explanatory Dictionary of Czech…………………………………………………………………………………  6
1.1.2 Dictionary of Literary Czech……………………………………………………………………………………… 6
1.1.3 Lingea Lexicon v.2…………………………………………………………………………………………………. .7    
1.1.4 Dictionary of Czech Synonyms 	8
    1.1.5 Corpora resources: Collocations, ESO and DESAM……………………………………………………………9
1.2 Tools	9
1.2.1 Dict	9
1.2.2 Wn-make	10
1.2.3 Czech and English morphological analyzer and lemmatizer 	11
1.2.4 Polaris	11
2 Resources and tools developed and used by TAR	11
2.1 Resources	11
2.1.1 Explanatory Dictionary of Estonian	11
2.1.2 Dictionary of Synonyms	11
2.1.3 Dictionary of Antonyms……………………………………………………………………………………………13
+2.1.4 Estonian Thesaurus	13
    2.1.5 English Dictionary for Speakers of Estonian  ………………………………………………………………….13
     2.1.6 English-Estonian-English Dictionary…………………………………………………………………………   13
     2.1.7 Collection of  Corpora………………………………………………………………………………………….    14  
2.2 Tools	14
2.2.1 KeeleWeb	14
2.2.2 THES	
2.2.3 Polaris	
    2.2.4 Set of Programs for Estonian Morphology ……………………………………………………………………  16

1	
1	Resources and tools developed and used by BRN

1.1	Lexical resources
1.1.1	Dictionary of Literary Czech
Description: it is a standard monolingual explanatory dictionary (published by Academia Publishing House in 1986, the second edition in 1994, CD in 1998) and at the present moment the only one of  the Czech traditional dictionaries that can be used  as a resource for building Czech WordNet – thanks to the fact that it exists in a machine readable form. It is a midsize dictionary containing about 45 000 entries and approximately 58 000 senses. The older and larger Dictionary of Literary Czech Language (published by Academia Publishing House in 1960) which contains almost 200 000 entries is not suitable for the purpose: it does not exist in the machine readable form and also more than  20 % of its stock is obsolete or completely archaic. 
The morphological level: each inflected entry in DLC contains some basic information about the inflectional paradigms, typical and also exceptional endings.  Other  grammatical categories like gender, number, case and animateness for nouns and aspect for verbs are given only for selected (irregular) entries. The verb entries also contain a basic information about the most frequent prefixes – that is why some entries contain quite complicated nesting.  
The syntactic level: a basic information about parts of speech and valency frames can be found in DLC, however, quite often this information has to be derived from the examples. 
The semantic level: typically the word meanings are described in two ways: either using synonyms, or we would say, synsets, or via standard definitions based on hyperonymy/hyponymy relations with the appropriate semantic markers added where necessary. The examples of the word meanings and typical collocations can be found in DLC as well as the basic idioms and phraseology. The main disadvantage of DLC is that the word meaning definitions (glosses) are often mutually inconsistent – the phenomena of the same kind are quite frequently described in a different ways. The second problem is that the word senses are not carefully distinguished, some of the the core senses are missing.  
Owner: Leda Publishing House,  Institute of Czech Language
Programming language: probably C++
Platform: MS-DOS, Windows
Availability: can be ordered as book and CD
Distributor: Academia, book, Leda Publishing House, CD

1.1.2	Explanatory Dictionary of Czech
Description:
Explanatory Dictionary of Czech (EDC, by P.Sevecek and K.Pala, Brno 1997-8) is being built continually and its present size is about 55 000 entries and 65 000 senses. The main reason for its building is the fact that the primary resource, Dictionary of Literary Czech (see above), in some respects displays some systematic inconsistencies in the word sense definitions and distinguishing the word senses. Another reason is formal, the machine readable version of the Dictionary of Literary Czech is an old one and rather a primitive one, thus for working on Czech WordNet it was necessary to convert it into a completely new format and transform it into a text database allowing a systematic automatic processing in electronic form. The notation for the text database has been developed by P.Ševeček and it is based on the same principles as SGML, however, in our opinion, its implementation is more effective and requires less memory. The present form of EDC is in fact an integrated lexical database that contains as its part the morphological stem dictionary and morphological analyzer (recognizer, generator) allowing to perform  automatic lemmatization, word form recognition and generation as well as basic word derivation (quite rich in Czech).
Morphological level: in comparison with Dictionary of Literary Czech the entries in EDC give complete morphological information that is available for an arbitrary Czech word form.
More precisely, the built-in morhological analyzer and generator (based on the stem dictionary containing about 165 000 items) is able to produce the required morphological information (all the possible grammatical categories) for any input  Czech word form depending on the selected parameter.
Syntactic level: in the present version of EDC the valency frames are associated with approximately 15 000 Czech verbs. The frames are written in a special notation (Horák, Proceedings of Text, Speech, Dialog’98 Workshop, Brno 1998) which makes it possible to search for the selected language internal relations, particularly for the roles, in a semiautomatical way. The results obtained in this way will be confronted with the output of the partial syntactic parser DES (Žáčková, Horák, Proceedings of Text, Speech, Dialog ’98 Workshop, Beno 1998 ) analyzing among others the verb groups in corpus texts. In this way the ILR for Czech WordNet can be obtained in a relatively short time and avoiding most of the manual work.
Semantic level: the word senses in EDC are basically described in two ways: by synsets and standard definitions (glosses) using hypero/ hyponymy relations. However, in comparison with Dictionary of Literary Czech much effort has been devoted to removing the inconsistencies from the definitions, many new glosses have been added and encoded more formally in the text database. At the same time the word senses are being examined more thoroughly and new ones are distinguished. This work is still going on and it is the main reason that prevents us from the inserting the hypero/hyponymy relations to the whole present Czech Wordnet now: first, all the checks have to be performed and senses examined, in our view it is quite obvious that it does not  make sense to do this work twice – now and then second time when all the checks were made. In fact, we are doing two things in a paralel way: building EDC entries and at the same time preparing them for the use within the Czech WordNet.
Owner: Lingea,  Faculty of Informatics
Programming language:  C, data structure – text database 
Platform: Linux (Unix), MS-DOS, Windows95
Availability: not free, for internal use
Distributor: Lingea 

1.1.3 Lingea Lexicon
Description: the third and the most important resource used by BRN is Lingea Lexicon v.2 which is a machine readable bilingual Czech-English dictionary containing at the present moment about 54 000 entries and 58 000 senses in Czech-English direction, and about 78 000 entries and 102 400 senses in English-Czech direction. It forms a part of the larger lexical database comprising Czech, English and German. As its integral parts it also comprises machine stem dictionaries together with lemmatizers for each of the three mentioned languages. Lingea Lexicon includes synonyms (synsets) for for all three languages and the entries in it are organized into 60 semantic fields. They correspond in a certain degree to some of the semantic clusters as they are used within WordNet 1.5. The present version of Lingea Lexicon also has as its integral part WordNet 1.5 thus making the preparation of the Czech synsets easier.
Morphological level: LL is organized in the same way as the above mentioned EDC, however, it contains not only the stem dictionary for Czech (165 000 items), but also for English (150 000 items) and German (140 000 items) and the morphological analyzer (lemmatizer) for all three languages. Thus complete morphological information can be obtained on demand for any of the included languages. 
Semantic level: the Czech entries have assigned their English and German translation equivalents. About 60 semantic fields similar to WordNet 1.5 semantic cluster exist in Lingea Lexicon. Thanks to this organization it is possible to associate Czech entries through Lingea Lexicon English equivalents with WordNet 1.5 counterparts and in this way establish the equivalence links to WordNet 1.5 items and to ILIs.
Owner: Lingea 
Programming language: C, data structure – text database
Platform: Linux (Unix), MS-DOS, Windows95
Availability: can be ordered on CD as Lingea Lexicon v.2, Cz-E, Cz-G
Distributor: Lingea
1.1.4	Dictionary of Czech Synonyms
Description: Dictionary of Czech Synonyms, by K.Pala and J.Vsiansky, NLN, Prague, 1994-5, size: approx. 22 000 synsets (entries) exists both in the printed and electronic version. It is also equipped with automatic lemmatization, word form recognition and basic word derivation when implemented within Lingea Lexicon framework. DCS can be also used  independently.
Morphological level: the printed version of  DCS offers only basic grammatical information where it is necessary but its electronic version integrated into Lingea Lexicon or Czech localization of MS Word v.7 can display the morphological information as LL and EDC. 
Syntactic level: the syntactic information in DCS  concerns parts of speech  and also for verbs their valency frames are given when they help to distinguish the different senses.
Semantic level: all the entries in DCS are organized in a way that is very close to the synsets in WordNet 1.5 though DCS was compiled before our work on WordNet database was started. Thus all the entries in DCS are suitable for integrating into Czech WordNet after some minor (mostly automatic) modifications and will become its part (including adjectives).

Owner: Lingea 
Programming language: C, data structure – text database
Platform: Linux (Unix), MS-DOS, Windows95
Availability: can be ordered as a book or on CD as a part of LL
Distributor: Publishing House Lidové Noviny, book, Lingea, CD

1.1.5	Corpus Resources: Collocations, ESO and DESAM
List of Czech Collocations
A list of Czech collocations containing approx. 20 000 lines has been obtained from the text corpus ESO (approx. 61 mil. word forms, built and mantained at the Faculty of Informatics), sorted according several criteria (MI score, frequencies, syntactic criteria) and prepared for the processing of the present-day Czech collocations.A number of items from this list has already been included in Czech Subset 1 because DLC contains only a few collocations.
Tagged Corpus DESAM 
Fully tagged corpus DESAM (both structurally and grammaticaly) that has been compiled at the Faculty of Informatics during the last two years (1997-98). It contains mostly newspaper and magazine texts from 1992-96 and its size is 1 mil. Czech word forms. It is exploited mostly for obtaining data about the verb valency frames which, in our view, should be included in Czech WordNet in a more elaborated way than it is done in WordNet 1.5,
where the relations between the verb valencies and their senses are rather neglected. This
also causes some difficulties with distinguishing verb senses, especially when the fine grained senses are considered. 
Corpus ESO
Corpus ESO has been built at the Faculty of Informatics in the course 1998 from newspaper and magazine texts (1996-98), contains about 61 mil. Czech word forms and is partially tagged (lemmatized). It is used to get a new data about Czech in a more complete way than it is possible with DESAM. 

Owner: Faculty of Informatics, Dept.of Information Technologies 
Programming language: C, data structure – corpus files in SGML-like format, corpus manager CQP (from IMS Stuttgart)
Platform: Linux, Unix (Solaris)
Availability: the access can be negotiated on the non-commercial basis only
Distributor: Faculty of Informatics, Masaryk University, Brno
1.2	Tools
1.2.1 Dict
Description: Specialized sorting program combining together several Unix utilities and functions (grep, wc and others) in one larger script. Dict can perform various kinds of sorting, ordering, selecting  and processing of the dictionary files having the format of the text database. In one of its modes it can be used to parse dictionary entries in Dictionary of Literary Czech and to select genus proximum definitions from it. In this way the candidates for the set of Base Concepts have been obtained.

Owner: Lingea 
Programming language: C
Platform:  Linux , Unix 
Availability: not free, for internal use
Distributor: Lingea

1.2.2	Wn-make
Description: A simple translating program able to process a bilingual dictionary (Czech-English-Czech): it takes the Czech entries from Lingea Lexicon together with their English equivalents and then tries to associate them with the WordNet 1.5 synsets and establish the equivalence links. Its second function is to translate the files in Lingea Lexicon text database format into WordNet 1.5 Export/Import format and vice versa.
Owner: Lingea
Programming language: C
Platform: Linux,Unix
Availability: not free, for internal use only
Distributor: Lingea
1.2.3	Czech and English morphological analyzer and lemmatizer 
Description: Czech morphological analyzer (generator) and lemmatizer by P.Sevecek and K.Osolsobe, containing about 165 000 Czech stems and able to perform full morphological analysis of an arbitrary Czech text. Depending on the selected mode it gives as the output a part of speech for any input word form,  its lemma and all the possible grammatical categories that can be associated with the analysed word form in Czech (3-4 categories for nouns, adjectives, numerals and pronouns and up to 6 categories for verbs) including also the complete information about inflection paradigm. It also captures the main word derivation processes in Czech, i.e. forming deverbatives, deadjectives, adverbs and possessives. In this way about at least 200 000 new lemmata can be dynamically derived. English lemmatizer based on the same principles as the Czech one is used as a built-in component within Lingea Lexicon. The mentioned Czech lemmatizer has been used for tagging corpus texts from corpus ESO and also for building the tagged  corpus DESAM.
Owner: Lingea
Programming language: C
Platform: Linux, Unix, Windows95, NT
Availability: for internal use only, non-commercial applications can be negotiated
Distributor: Lingea
1.2.4	Polaris
Description: Polaris 1.3.6 EuroWordNet database tool obtained from P.Vossen of Amsterdam University and EuroWordNet 2 Consortium has been acquired for the further work, particularly for the processing of the sense links, editing and comparison the individual WordNets developed within EuroWordNet 1 and 2 project.
Owner: Learnout  & Hauspie 
Programming language: C++
Platform: Windows95, NT
Availability: for internal use within EuroWordNet 2 Consortium
Distributor: Learnout  & Hauspie, AMS

2	
Resources and tools developed and used by TAR

2.1	Resources

2.1.1	 Explanatory Dictionary of Estonian 
Description: Eesti kirjakeele seletussõnaraamat, EKSS (Explanatory Dictionary of Estonian). Unfortunately, at the moment EKSS contains published materials only from A to R (though the work on the missing part is proceeding rapidly). And second, since work on EKSS-s electronic version has been carried out during a long period, its original markup is inconsistent and often defective. Thus we first had to convert it to a coherently coded form, so that it would be possible to automatically extract sense descriptions (glosses), examples etc.
At present EKSS contains:
-	50.890 entries (the number of entries is in fact much larger, since many compound word expressions, phraseologisms etc are merged in these main entries),
-	118.442 senses (including 103.525 senses for which sense descriptions are provided).
Owner: Institute of Estonian Language / University of Tartu
Platform: UNIX
Availability: not free, for internal use only
Distributor: owner

2.1.2	Dictionary of Synonyms
Description: Sünonüümisõnastik (Dictionary of Synonyms) by Asta Oim.
It contains about 10.000 synonym sets which include also multiword expressions and phraseologisms.

Owner: Asta Õim
Platform: Internet
Availability: http://www.ibs.ee.estonian/syn/
Distributor: owner

2.1.3	Dictionary of Antonyms

Description: Antonüümisõnastik (Dictionary of Antonyms) by Asta Oim.
Contains 2.000 antonymous pairs where each member of a pair is actually a set of synonyms compiled largely on the basis of Dictionary of Synonyms.

Owner: Asta Õim
Platform: Internet
Availability: http://www.ee/Anton/
Distributor: owner

2.1.4	Estonian Thesaurus
Description: Estonian Thesaurus for Microsoft Office ´97 compiled by Filosoft. It contains mainly information on synonyms and antonyms.
Number of synonym sets: 12.800 (including 3.400 multiword expressions);
number of synonym sets with antonyms: 3.800;
total number of words and expressions: 51.600.
The lexicon contains only lemmas, inflected words are included only when these are synonymous with words of some other part of speech.

Owner: Filosoft
Platform: Internet
Availability: http://www.filosoft.ee/thes_et
Distributor: owner

2.1.5	English Dictionary for Speakers of Estonian
Description: Password – English Dictionary for Speakers of Estonian, Kernerman Semi-Bilingual Dictionaries (translated and published by TEA Publishers, 1995; the electronic version by Festart 1995).
It contains 25.000 headwords with sense descriptions in English, 36.000 translations into Estonian. It also includes index as Estonian-English Translation Dictionary.

Owner: Festart Ltd. / Keelfirma TEA
Platform: Windows
Availability: 
Distributor: owner

2.1.6	English-Estonian-English Dictionary
Description: more than 17000 English words with translations in Estonian.

Owner: Institute of Baltic Studies
Platform: Internet
Availability: http://www.ibs.ee/dict/
Distributor: owner

2.1.7	Collection of Estonian Corpora
Description:	Eesti Kirjakeele Korpus, EKK (Corpus of Written Estonian). The internet version users have also available tools for analyzing corpus texts: morphological analyzer, Estonian HTML speller, Concordance for Estonian etc.
Size of the Corpus: 1 million words, tagged according to TEI. Texts are selected from issues printed on 1980s and according to the classification of the Brown Corpus.

In addition to EKK there are (untagged) corpora compiled with the aim of enabling the study of the usage of Estonian in newspapers and in fiction at different periods:
-1960s (330.000 words),
-1950s (308.000 words),
-1930s (369.000 words),
-1890s (348.000 words).

Thirdly, there is BNS (=Baltic News Service) News Corpus (plain text), about 3 million words a year since 1996.
Owner:  University of Tartu
Platform: Internet
Availability: http://www.cl.ut.ee/index_en.html
Distributor: free download from Internet

2.2	Tools

2.2.1	KeeleWeb
Description: For working with Estonian lexical resources in Internet there is a new site called KeeleWeb (LinguaWeb), where it is possible to make united query over more than 5 different monolingual dictionaries.

Owner: KeeleWeb
Programming language: 
Platform: Internet
Availability: http://ee.www.ee/
Distributor: owners

2.2.2	Thes
Description: We started to create Estonian wordnet in 1997. Before co-operating with EWN-2 we reached to create about 900 synsets (mainly EWN-1 BCs translations into Estonian and about 150 synsets of 30 most frequent Estonian verb). The format of that former database was close to WordNet 1.5, but not a copy of that, so we could not use WN 1.5 interface to handle the Estonian wordnet.
For inserting lexical data to our former wordnet we used MS Word 7.0 macro (made by Filosoft). It offers windows with blank fields for filling them with lexical data, and it is able to check whether the inserted word does already exist in the database and in what synsets and contexts. After our historical decision to use Polaris for developing Estonian wordnet we only change totally our former database format (with help some little UNIX scripts) to make it understandable for Polaris.

Owner: Filosoft
Programming language: Visual Basic
Platform: Word 7.0
Availability: for internal use only
Distributor: owner

2.2.3	Polaris
Description: Polaris 1.3.6 EuroWordNet database tool obtained from P.Vossen of Amsterdam University and EuroWordNet 2 Consortium has been acquired for the further work, particularly for the processing of the sense links, editing and comparison the individual WordNets developed within EuroWordNet 1 and 2 project.
Owner: Learnout  & Hauspie 
Programming language: C++
Platform: Windows95, NT
Availability: for internal use within EuroWordNet 2 Consortium
Distributor: Learnout  & Hauspie, AMS

2.2.4	Set of programs for Estonian morphology
Description:	We have programs for morphological analysis, morphological disambiguation and morphological generation of Estonian.
The first two programs were used, for instance, to analyze and disambiguate the texts of sense descriptions (glosses) of Estonian Explanatory Dictionary (see 2.1.1) in the process of creating the first version of Estonian base concepts.

Owner: Filosoft
Programming language: C, C++
Platform: UNIX, Windows
Availability: http://www.ee/eks
Distributor: owner