Datasets

Three corpora licensed for training.

Collected under one licence policy and delivered as JSONL or Parquet. See the catalogue and an evaluation sample before you decide.

Multilingual text

The long tail of human language.

A text corpus for the languages and subjects that general web crawls cover poorly: rare and low-resource languages, and institutional and specialist writing in science, law, government, medicine, finance and engineering.

Volume
Growing every day. Dated figures are in the catalogue.
Languages
More than 130, many of them low-resource
Subjects
More than 140
Licence classes
Public domain, CC0, attribution (CC BY)
Formats
JSONL, Apache Parquet

Source code

From COBOL to Rust.

Banks, insurers and public bodies still run on COBOL, Fortran, JCL and PL/I, and models that modernise those systems need to learn them. This corpus pairs that legacy code with modern languages, under permissive licences only.

Volume
Growing. Dated figures are in the catalogue.
Legacy
COBOL, Fortran, JCL, PL/I
Modern
Rust, C, C++, Java, Go, MATLAB
Licences
MIT, Apache-2.0, BSD, ISC, Zlib, 0BSD, CC0, Unlicense. No GPL or AGPL.
Licence evidence
Recorded for every file
Credentials
Files with credential-like strings are delivered only with synthetic values.

Biomedical and life science

For drug discovery and clinical language.

Biomedical text from pharmacology and neurology to tropical medicine and biochemistry, with structured records available alongside it. Nothing here is patient-level data.

Volume
Growing. Dated figures are in the catalogue.
Subjects
Pharmacology, neurology, biochemistry, immunology, epidemiology, tropical medicine and more
Structured records
Regulatory drug labelling, de-identified adverse-event reports, clinical-trial registry records, measured binding affinities, gene and pathway annotations
Not offered
Patient-level data. Adverse-event reports are de-identified regulatory filings, not a patient cohort.

Coverage

Languages and subjects in the text corpus.

Volumes differ a lot by language. The catalogue shows what is available for the ones you need.

Languages include

Abkhazian · Acehnese · Afrikaans · Albanian · Amharic · Ancient Greek · Armenian · Avar · Aymara · Azerbaijani · Bambara · Bashkir · Basque · Belarusian · Bengali · Bislama · Breton · Bulgarian · Burmese · Catalan · Cebuano · Chamorro · Chechen · Chichewa · Chinese · Cornish · Croatian · Czech · Danish · Dutch · Dzongkha · English · Esperanto · Estonian · Fijian · Finnish · French · Fula · Galician · Ge'ez · Georgian · German · Gothic · Greek · Guaraní · Gujarati · Hausa · Hawaiian · Hindi · Hungarian · Icelandic · Igbo · Indonesian · Inuktitut · Irish · Italian · Japanese · Javanese · Kazakh · Khmer · Kinyarwanda · Korean · Kyrgyz · Lao · Latin · Latvian · Lezgian · Lingala · Lithuanian · Lower Sorbian · Luganda · Macedonian · Maithili · Malay · Malayalam · Maltese · Manx · Māori · Marathi · Marshallese · Mon · Mongolian · Nahuatl · Nepali · Norwegian · Occitan · Odia · Oromo · Ossetian · Polish · Portuguese · Punjabi · Quechua · Romanian · Russian · Samoan · Sanskrit · Serbian · Sesotho · Shona · Sinhala · Slovak · Slovenian · Somali · Spanish · Sundanese · Swahili · Swedish · Tagalog · Tajik · Tamil · Tatar · Telugu · Thai · Tibetan · Tigrinya · Tok Pisin · Tongan · Tsonga · Tswana · Turkish · Turkmen · Ukrainian · Uzbek · Vietnamese · Welsh · Wolof · Xhosa · Yoruba · Zulu

Subjects include

Science and research · Law, courts and legislation · Government and parliamentary records · Medicine · Traditional medicine, Ayurveda and Islamic medicine (Tibb Nabawi) · Indigenous knowledge · Banking, finance and tax · Engineering and aeronautics · Agriculture · Architecture · Astronomy and chemistry · Linguistics · Theology and humanities · Education · Legacy IT systems

A language is listed as recorded on the document. Languages still under quality review appear in the catalogue with that status.

Every record

A selection you can audit and reproduce.

  • Licence class, licence and the evidence behind it
  • Attribution notice wherever a licence requires one
  • Language as recorded, and subject
  • Token count and a content hash
  • Quality flags, reported and never silently removed
record.jsonl
{
  "text": "…",
  "language": "de",
  "subject": "medicine",
  "licence_class": "public-domain",
  "licence_evidence": "source statement",
  "tokens": 1842,
  "text_hash": "9f2c…e41a",
  "quality_flags": []
}

Licence policy

One rule for everything we deliver.

A document is delivered only if its licence allows commercial use, including training. Anything we cannot classify is held back, not delivered.

What we deliver

  • Public domain and CC0
  • CC BY, with the attribution delivered alongside the data
  • Code under MIT, Apache-2.0, BSD, ISC, Zlib, 0BSD, CC0 and Unlicense

What we never deliver

  • Non-commercial (NC) and no-derivatives (ND) licences
  • Share-alike (SA) licences
  • GPL, AGPL and other copyleft code licences
  • Anything with a missing or unclear licence

Delivery

Delivered in the size you need.

Long documents are cut into self-contained units at sentence boundaries, so the volume per subject matches what you need.

SmallUp to 2

pages per unit

Medium2–20

pages per unit

Large20–100

pages per unit

How licensing works

From first look to delivery.

  1. Catalogue

    Volumes by language, subject and licence, rounded down and dated.

  2. Evaluation sample

    Real units in the delivery format, so you can judge schema and quality.

  3. NDA and agreement

    We sign an NDA, then agree scope and licence terms.

  4. Delivery

    JSONL or Parquet over an encrypted transfer, with documentation and attribution notices.

Questions buyers ask.

Can we see data before we sign anything?

Yes. The catalogue lists volumes by language, subject and licence. An evaluation sample in the delivery format follows, and larger evaluation packs are shared under NDA.

Do you disclose where the data comes from?

No. Sourcing is confidential. Each document carries its licence class and the evidence for it, and attribution is delivered wherever a licence requires it.

Is personal data removed?

Personal-data detection can be run on any delivery, and the detected rates are reported with it. Automated detection does not find everything, so we report what was found and never claim that nothing remains.

Can we license a single language or subject?

Yes. Every document is labelled with its language and subject, so a delivery can be cut by language, subject, licence class or size.

How is a delivery sized?

Long documents are cut into self-contained units at sentence boundaries, in three sizes: up to about 2 pages, about 2 to 20 pages, and about 20 to 100 pages.

Do you provide a quality report with the data?

Yes. A benchmark report can be delivered with the dataset. It measures duplicates, detected personal data, licence mix and overlap with evaluation sets.

Ask for the catalogue.

Tell us which languages, subjects or programming languages you need.