AI training data

Training data with its rights on record.

Multilingual text, source code and biomedical corpora. Every document carries its licence class and the evidence behind it.

Explore the datasetsCurate your own data
One record as delivered

130+languages, many of them rare and low-resource

{
  "text": "Die Studie untersucht, wie …",
  "language": "de",
  "subject": "medicine",
  "licence_class": "public-domain",
  "licence_evidence": "source statement",
  "quality": "clean",
  "text_hash": "9f2c…e41a"
}

10programming languages, from COBOL to Rust

{
  "file": "payroll.cbl",
  "language": "cobol",
  "licence": "Apache-2.0",
  "licence_evidence": "licence file",
  "notice": "included in delivery",
  "text": "IDENTIFICATION DIVISION. …"
}

Text + recordsfrom pharmacology to tropical medicine

{
  "text": "The compound inhibits …",
  "language": "en",
  "subject": "pharmacology",
  "licence_class": "attribution",
  "licence": "CC-BY-4.0",
  "attribution": "included in delivery",
  "quality": "clean"
}
  • English
  • Deutsch
  • Français
  • Latina
  • Ἑλληνική
  • Italiano
  • Беларуская
  • Հայերեն
  • Español
  • हिन्दी
  • संस्कृतम्
  • বাংলা
  • Ελληνικά
  • मराठी
  • Afrikaans
  • Nederlands
  • Cymraeg
  • Kiswahili

Three ways to work with CurateLM.

Datasets130+

languages of text, plus source code and biomedical corpora. Licensed by language, subject and size.

See the datasets
Data curationYour data

Extracted, cleaned, de-duplicated and scrubbed of personal data. Processed in Germany or inside your environment.

How it works
BenchmarkA to E

A measured quality report on any text or code dataset. Numbers only, never your text.

What is measured

From document to record

Every document tells you what it is.

1

Language as recorded

Each document keeps the language it was recorded with. Independent detectors check that label wherever they can, and the catalogue says how each label was confirmed.

2

Subject

Every document is filed under one of more than 140 subjects, from law and government records to medicine and engineering, so you can license exactly the slice you need.

3

Licence with its evidence

The licence class sits next to the reason we hold it: the source's own statement, the work itself, or its publication date. If the reason is unclear, the document is held back.

4

Quality flags

Damaged text, navigation debris and heavy repetition are flagged, not silently deleted. You decide what to keep.

record.jsonlDocument
{
  "text": "Die Studie untersucht, wie …",
  "language": "de",
  "subject": "medicine",
  "licence_class": "public-domain",
  "licence_evidence": "source statement",
  "quality_flags": []
}

The corpus

Built for the languages and subjects web crawls miss.

130+languages, many of them rare and low-resource
140+subjects, from law and government to medicine
10programming languages, from COBOL to Rust
0documents delivered under non-commercial or share-alike terms

The corpora grow every day. The catalogue gives exact, dated figures, rounded down.

Licence policy

Rights on record for every document.

You see what every document may be used for, and why we say so.

Read the licence policy
  • Licence class recorded per document: public domain, CC0 or attribution.
  • Evidence recorded with it: the source's own statement, the work itself, or its publication date.
  • Attribution notices delivered with the data wherever a licence asks for them.
  • Non-commercial, no-derivatives, share-alike and copyleft material is never delivered.

Data curation

Your data stays
in Germany.

The pipeline that builds our corpora, run on your documents, scans, audio and video. On our own machines in Germany under a data processing agreement, or inside your environment. Deleted after delivery, with written confirmation.

How data curation works

Tell us what data you need or what data you have.

We reply within one business day with the catalogue, an evaluation sample or a pilot proposal.

Contact usExplore the datasets