We are happy to announce that 66 new monolingual lexicons and 1 speech resource are now available in our catalogue. Moreover, 4 speech resources are now available at reduced fees.

 

1) New Language Resources:

 

Bitext Lexical Datasets

The series of Bitext Lexical Datasets for the generic vocabulary includes Lemmas, POS tagging, Frequency, Named Entities and Offensive features. Depending on the dataset and language, other syntactic and morphological features are also provided. The following 15 languages are available:

As a complement to the datasets mentioned above, 11 datasets of Language Variants can also be obtained:

 

1. Arabic (MSA) dataset and Arabic Language Variants dataset consisting of Arabic Gulf, Arabic Najdi, Arabic Egypt and Arabic MSA variants,
2. Chinese (Simplified) dataset, Chinese (Traditional) dataset, and Chinese Language Variants dataset (Simplified + Traditional),
3. Dutch dataset and Dutch Language Variants dataset consisting of Netherlands and Belgium variants,
4. English dataset and English Language Variants dataset consisting of United States, United Kingdom and India variants,
5. Finnish dataset and Finnish Language Variants dataset consisting of Standard and Colloquial Finnish variants,
6. French dataset and French Language Variants dataset consisting of France, Canada and Switzerland variants,
7. German dataset and German Language Variants dataset consisting of Germany and Switzerland variants,
8. Indonesian dataset,
9. Italian dataset and Italian Language Variants dataset consisting of Italy and Switzerland variants,
10. Malay dataset,
11. Norwegian (Bokmal) dataset and Norwegian Language Variants dataset consisting of Bokmal and Nynorsk variants,
12. Portuguese dataset and Portuguese Language Variants dataset consisting of Portugal and Brazil variants,
13. Spanish dataset and Spanish Language Variants dataset consisting of Spain, North America, Central America, Andes and Southern Cone variants,

 

Bitext Synthetic Data

The Bitext Synthetic Data consist of pre-built training data for intent detection and are provided for 20 verticals for English and Spanish languages. They cover the most common intents for each vertical and include a large number of example utterances for each intent, with optional entity/slot annotations for each utterance. Data is distributed as models or open text files.

For each language, the following verticals are available:

1. Automotive: 52 intents (English, Spanish)
   
2. Retail banking: 26 intents (English, Spanish)
3. Education: 37 intents (English, Spanish)
4. Event and ticketing: 25 intents (English, Spanish)
5. Field Service: 27 intents (English, Spanish)
6. Healthcare: 40 intents (English, Spanish)
7. Hospitality: 24 intents (English, Spanish)
8. Insurance: 38 intents (English, Spanish)
9. Legal : 29 intents (English, Spanish)
10. Manufacturing: 34 intents (English, Spanish)
11. Media Streaming: 24 intents (English, Spanish)
12. Mortgage and loans: 39 intents (English, Spanish)
13. Moving and storage: 29 intents (English, Spanish)
14. Real estate and construction: 28 intents (English, Spanish)
15. Restaurant/ bar chains: 30 intents (English, Spanish)
16. Retail Ecomm: 34 intents (English, Spanish)
17. Telecommunication: 26 intents (English, Spanish)
18. Travel: 33 intents (English, Spanish)
19. Utilities: 21 intents (English, Spanish)
20. Wealth management: 24 intents (English, Spanish)

 

Persian Kids’ Speech Corpus

The Persian Kids’ Speech Corpus consists of speech signals recorded by 286 children (141 girls, 145 boys), from 6 to 9 years old, through an Andreas Mic Anti-Noise microphone and a Premium Speechmike headphone. This recorded data was manually checked and labeled. Finally, a corpus containing 162,395 samples with a duration of 33 hours and 44 minutes was created. The samples are distributed as follows:

1. 29,057 Words (478 minutes),
2. 17,429 SubWords (260 minutes),
3. 43,838 Syllables (485 minutes),
4. 70,078 Phonemes (765 minutes),
5.  1,993 Extra Vocabulary (36 minutes).

The prepared speech corpus comprehensively contains all the 29 Persian phonemes, 118 syllables, 56 sub-words, and 711 words and is particularly applicable to speech recognition and linguistics studies.

 

2) Reduced fees for the following speech resources:

• Chinese Mandarin (South) database
• Chinese Mandarin (North) database
• Japanese Kids Speech database (Lower Grade)
• Japanese Kids Speech database (Upper Grade) 

For more information on the catalogue or if you would like to enquire about having your resources distributed by ELRA, please contact us.
_________________________________________

Visit the ELRA Catalogue of Language Resources
Visit the Universal Catalogue 
Archives of ELRA Language Resources Catalogue Updates

Membership Year 2024 publication preview 

Fall 2023 data scholarship recipients

New publications:
AIDA Scenario 1 Practice Topic Source Data
AIDA Scenario 1 and 2 Reference Knowledge Base

Membership Year 2024 publication preview 
The 2024 membership year is approaching and plans for next year’s publications are in progress. Among the expected releases are: 

• KASET: 147 hours of Sorani Kurdish and Kurmanji Kurdish conversational telephone speech and web broadcasts, 65 hours transcribed
• AIDA topic source data and annotations: multimodal source data and annotations in multiple languages (Russian, Ukrainian, English, Spanish) for information and entity extraction
• RATS Low Speech Density Data: 87 hours of Levantine Arabic, English, Persian, Pushto, and Urdu audio files selected from RATS speech activity detection and keyword spotting data sets, also including communications systems sounds and silence
• Call My Net 1: 364 hours of conversational telephone speech recordings in Tagalog, Cebuano, Cantonese, and Mandarin from speakers in the Philippines and China using various handsets under diverse noise conditions

• Ravnursson Faroese Speech and Transcripts: 109 hours of read speech from 433 native speakers with transcripts
• Diaspora Tibetan Speech: elicited, read, and spontaneous speech from 73 native Tibetan speakers in Katmandu’s diaspora Tibetan community, some recordings transcribed

• IARPA MATERIAL language packs: conversational telephone speech, transcripts, English translations, annotations, and queries in multiple languages (e.g., Bulgarian, Somali, Georgian)
• LORELEI: representative and incident language packs containing monolingual text, bi-text, translations, annotations, supplemental resources, and related tools in various languages (e.g., Farsi, Hungarian, Hindi, Amharic) 

Check your inbox in the coming weeks for more information about membership renewal.

Fall 2023 data scholarship recipients
Congratulations to the recipients of LDC's Fall 2023 data scholarships:

Nessma Diab: Ain-Shams University (Egypt): Pre-PhD student, Linguistics. Nessma is awarded copies of CALLHOME Egyptian Arabic Speech LDC97S45 and CALLHOME Egyptian Arabic Transcripts LDC97T10 for her work in machine translation.
Soheir Elssakkout: Ain-Shams University (Egypt): PhD candidate. Soheir is awarded copies of Turkish Broadcast News and Transcripts LDC2012S06 and Middle East Technical University Turkish Microphone Speech v 1.0 LDC2006S33 for her work in speech recognition.
Matheus Franco: Witten/Herdecke University (Germany): Post-doctoral scholar, Faculty of Management, Economics and Society. Matheus is awarded a copy of Avocado Research Email Collection LDC2015T03 for his work in emotional foundations of dynamic capabilities.

Kamal Jarrar: Birzeit University (Palestine): Master’s student, Applied Statistics and Data Science Program. Kamal is awarded copies of Arabic Gigaword Fifth Edition LDC2011T11 and BOLT Arabic Discussion Forums LDC2018T10 for his work in part-of-speech tagging for dialectal Arabic.
Minkyoung Kim: Yonsei University (Korea); PhD candidate, Graduate School of Information. Minkyoung is awarded a copy of The New York Times Annotated Corpus LDC2018T19 for her work in event extraction and semantic event annotation.
Humaira Mehmood: Fatima Jinnah Women University (Pakistan): Master’s student, Computer Sciences. Humaira is awarded a copy of ARL Urdu Speech Database, Training Data LDC2007S03 for her work in machine translation.
Diyam Mousa: Birzeit University (Palestine): PhD candidate, Computer Science Department. Diyam is awarded copies of Arabic Treebank: Part 3 v. 3.2 LDC2010T08 and BOLT Egyptian Arabic Treebank – Discussion Forum LDC2018T23 for her work in morphological tagging for dialectal Arabic.

For information about the program, visit the Data Scholarships page.

New publications:
AIDA Scenario 1 Practice Topic Source Data was developed by LDC and is comprised of 1511 files (text, image, and video) from English, Russian, and Ukrainian web sources. Each phase of the AIDA program centered on a specific scenario, or broad topic area, with related subtopics designated as either practice subtopics or evaluation subtopics. The Phase 1 scenario focused on political relations between Russia and Ukraine in the 2010s. This corpus constitutes the full set of topic-focused documents for Phase 1 practice subtopics. Data was collected from web sources by a combination of automatic and manual processes.

The DARPA AIDA (Active Interpretation of Disparate Alternatives) program aimed to develop a multi-hypothesis semantic engine to generate explicit alternative interpretations of events, situations, and trends from a variety of unstructured sources. LDC supported AIDA by collecting, creating and annotating multimodal linguistic resources in multiple languages.

The knowledge base for entity detection and linking annotation for all AIDA Scenario 1 and 2 corpora is available separately as AIDA Scenario 1 and 2 Reference Knowledge Base (LDC2023T10).

2023 members can access this corpus through their LDC accounts. Non-members may license this data for $1000.

AIDA Scenario 1 and 2 Reference Knowledge Base contains the English knowledge base (KB) used for all AIDA entity linking annotation in Scenario 1 (Russia-Ukraine Relations) and Scenario 2 (Crisis in Venezuela). The KB content was drawn from GeoNames, the CIA World Leaders List, and the CIA World Factbook and was supplemented with manually-created KB entries developed by LDC specifically for AIDA data.

This knowledge base supported the AIDIA entity detection and linking task for 13 entity types: GPE (Geo-Political Entity), LOC (Location), PER (Person), ORG (Organization), FAC (Facility), MHI (Medical/Health Issue), WEA (Weapon), SID (Side), COM (Commodity), CRM (Crime), LAW (Law), VEH (Vehicle), and BAL (Ballot).

2023 members can access this corpus through their LDC accounts. Non-members may license this data for $250.