Module tf.about.corpora
Corpora
Text-Fabric corpora are usually stored on GitHub/GitLab and Text-Fabric knows how to download a corpus from GitHub/GitLab if you specify the org/repo.
Most corpora are configured by metadata in a directory app in the repo.
You can load a corpus into a Python datastructure by
from tf.app import use
A = use("org/repo")
And you can get it in the Text-Fabric browser by saying this on a command prompt:
text-fabric org/repo
Here is a list of corpora that can be loaded this way. Since everybody can put a Text-Fabric corpus on GitHub/GitLab, the list may not be complete!
- annotation/banks
-
modern english Iain M. Banks, 1954 - 2013, 99 words from the SF novel Consider Phlebas, Dirk Roorda
to see the details
- annotation/mobydick
-
english Herman Melville, 1819 - 1891, Novel, 1851; converted from TEI in the Oxford Text Archive, Dirk Roorda
with tokens and sentences from Spacy woven in
- annotation/mondriaan
-
english Piet Mondriaan, 1872 - 1944, Test corpus of 14 letters; converted from TEI from the Huygens Institute, together with RKD and HuC; many people involved
with tokens and sentences from Spacy woven in
Cambridge Semitics Lab
- CambridgeSemiticsLab/nena_tf
-
Aramaic North Eastern Neo-Aramaic Corpus, 2000, Nena Cambridge, Cody Kingham
with a client-side, offline search interface in Javascript
CenterBLC Andrews University
- CenterBLC/LXX
- Greek Septuagint, 300 - 100 BCE, LXX Rahlf's edition 1935 plus additional features by CenterBLC; earliest extant Greek translation of Hebrew Bible books; Oliver Glanz, Adrian Negrea
- CenterBLC/NA
- Greek New Testament, 100 - 400, GNT Nestle-Aland edition 1904 with new features by CentrBLC, converted from biblicalhumanities/Nestle1904 contributed by Ulrik Sandborg Petersen, Jonathan Robie; Oliver Glanz
- CenterBLC/SBLGNT
- Greek New Testament, 100 - 400, converted from James Tauber's morphgnt/sblgnt with additional features by CenterBLC; Adrian Negrea, Clacir Virmes, Oliver Glanz, Krysten Thomas
CLARIAH
- CLARIAH/descartes-tf
-
French, Latin, Dutch Letters from and to Descartes, 1619 - 1650, René Descartes - Correspondance; Ch. Adam et G. Milhaud (eds. and illustrations, 1896-1911); Katsuzo Murakami, Meguru Sasaki, Takehumi Tokoro (ascii digitization, 1998); Erik-Jan Bos (ed, 2011); Dirk Roorda (converter TEI, 2011 and TF 2023)
with math display and illustrations
- CLARIAH/wp6-ferdinandhuyck
-
Dutch a novel by Jacob van Lennep, 1840, Jacob van Lennep - Ferdinand Huyck; From DBNL, TEI-Lite; Dirk Roorda (converter TEI to TF), see also
tf.convert.tei
with tokens and sentences from Spacy woven in
- CLARIAH/wp6-missieven
- Dutch General Missives, 1600 - 1800, General Missives, Dutch East-Indian Company, Jesse van der Does, Sophie Arnoult, Dirk Roorda
- CLARIAH/wp6-daghregisters
-
Dutch Dagh Registers Batavia, 1640 - 1641, Daily events at Batavia, Indonesia, historical source for the operation of the Dutch East-Indian Company, Lodewijk Petram, Dirk Roorda. work in progress, currently only volume 4
with many ocr errors and an attempt to detect them
Cody Kingham
- codykingham/tischendorf_tf
- Greek New Testament, 50 - 450, Tischendorf 8th Edition, Cody Kingham, Dirk Roorda
Digital Theologians of the University of Copenhagen
- DT-UCPH/sp
- Hebrew Samaritan Pentateuch , 516 BCE - 70 AD, MS Dublin Chester Beatty Library 751 + MS Garizim 1, Martijn Naaijer, Christian Canu Højgaard
Eep Talstra Centre for Bible and Computer
- ETCBC/bhsa
-
Hebrew Bible (Old Testament), 1000 BCE - 900 AD, Biblia Hebraica Stuttgartensia (Amstelodamensis), ETCBC + Dirk Roorda
the canonical TF dataset, where it all started
- ETCBC/dhammapada
- Pāli and Latin Ancient Buddhist verses, 300 BCE and 1855 AD, Transcription with Latin translations based on Viggo Fausbøll's book, Bee Scherer, Yvonne Mataar, Dirk Roorda
- ETCBC/dss
- Hebrew Dead Sea Scrolls, 300 BCE - 100 AD, Transcriptions with morphology based on Martin Abegg's data files, Martijn Naaijer, Jarod Jacobs, Dirk Roorda
- ETCBC/nestle1904
- Greek New Testament, 100 - 400, GNT Nestle-Aland edition 1904 from LOWFAT-XML syntax trees, converted from biblicalhumanities/greek-new-testament contributed by Jonathan Robie and Micheal Palmer; Oliver Glanz, Tony Jurg, Saulo de Oliveira Cantanhêde, Dirk Roorda
- ETCBC/peshitta
- Syriac Peshitta (Old Testament), 1000 BCE - 900 AD, Vetus Testamentum Syriace, Hannes Vlaardingerbroek, Dirk Roorda
- ETCBC/syrnt
- Syriac New Testament, 0 - 1000, Novum Testamentum Syriace, Hannes Vlaardingerbroek, Dirk Roorda
KNAW/Huygens
- hermans/works
-
Dutch Complete Works of W.F. Hermans. The conversion to Text-Fabric is work in progress. So far these works have been done:
- Paranoia
- Sadistisch Universum
- Nooit meer slapen
Bram Oostveen, Peter Kegel, Dirk Roorda
Not publicly accessible, the book is under copyright.
with a critical apparatus
- mondriaan/letters
-
Dutch Letters of Piet Mondriaan. , 1892-1923. Straight conversion from TEI to Text-Fabric, Peter Boot et al, Dirk Roorda
** Work in progress, not yet public.
with tokens and sentences from Spacy woven in
NINO Cuneiform
- Nino-cunei/ninmed
- Akkadian / cuneiform Medical Encyclopedia from Nineveh, ca. 800 BCE, Medical documents with lemma annotations, Cale Johnson, Dirk Roorda
- Nino-cunei/oldassyrian
- Akkadian / cuneiform Old Assyrian documents, 2000 - 1600 BCE, Documents from Ashur Cale Johnson, Alba de Ridder, Martijn Kokken, Dirk Roorda
- Nino-cunei/oldbabylonian
- Akkadian / cuneiform Old Babylonian letters, 1900 - 1600 BCE, Altbabylonische Briefe in Umschrift und Übersetzung, Cale Johnson, Dirk Roorda
- Nino-cunei/uruk
-
proto-cuneiform Uruk, 4000 - 3100 BCE, Archaic tablets from Uruk, Cale Johnson, Dirk Roorda
with lots of illustrations
Protestant Theological University
- Greek Literature
- Greek Literature, -400 - +400, Perseus Digital Library and Open Greek and Latin Project The result of a massive conversion effort by Ernst Boogert.
- pthu/athenaeus
- Greek Works of Athenaeus, 80 - 170, Deipnosophistae, Ernst Boogert
Quran
- q-ran/quran
- Arabic Quran, 600 - 900, Quranic Arabic Corpus, Cornelis van Lit, Dirk Roorda
University of Utrecht: Cornelis van Lit
- among/fusus
- Arabic Fusus Al Hikam, 1165- 2000, editions (Lakhnawi and Afifi) of Ibn Arabi's Fusus plus commentaries in the centuries thereafter, Cornelis van Lit, Dirk Roorda
Intentions
- oldroyal
- Akkadian-Sumerian cuneiform Bilingual royal inscriptions, 2000 - 1600, more info to come, Martijn Kokken, Dirk Roorda
Get corpora
Automatically
Text-Fabric downloads corpus data and apps from GitHub/GitLab on demand.
See tf.about.use
.
Data ends up in a logical place under your ~/text-fabric-data/
.
The TF data is fairly compact.
Size of data
There might be sizable additional data for some corpora, images for example. In that case, take care to have a good internet connection when you use a TF app for the first time.
Manually
TF data of corpora reside in a backend repo. You can manually clone such a data repository and point Text-Fabric to that data.
First, take care that your clone ends up in github/
orgName or gitlab/
orgName
(relative your home directory)
where orgName is the organization or person or group on GitHub/GitLab under which you have
found the repo.
Then, when you invoke the app, pass the specifier :clone
.
This instructs Text-Fabric to look in your local GitHub/GitLab clone, rather
than online or in your local text-fabric-data
, where downloaded data is stored.
use('org/repo:clone', checkout="clone")
text-fabric org/repo:clone --checkout=clone
In this way, you can work with data that is under your control.
Size of data
Cloning a data repository is more costly then letting Text-Fabric download the data. A data repository may contain several versions and representations of the data, including the their change histories. There might also be other material in the repo, such as source data, tutorials, programs.
For example, the ETCBC/bhsa
repo is several GB, but the TF data for a specific
version is only 25MB.
Extra data
Researchers are continually adding new insights in the form of new feature
data. TF apps make it easy to use that data alongside the main data source.
Read more about the data life cycle in tf.about.datasharing
.
Expand source code Browse git
"""
.. include:: ../docs/about/corpora.md
"""