The workflow is designed around a **collection of independent services accessible via APIs**, which integrate seamlessly into the native workflows of a **designated repository**.

In order to implement the workflow fully, it is necessary to have complete **control over the repository implementation** and be able to adapt its internal workflows and services. In this context, the [Archaeological Map of the Czech Republic](https://doi.org/10.17616/R31NJMYW), based on the [Fedora repository system](https://fedorarepository.org/), is utilized as the demonstrator for the implementation; however, the approach is adaptable to alternative repository implementations.

The workflow requires **textual data as input**, such as _ALTO XML_ or _plain text_. The preparation of this input data is handled in a separate OCR/HTR and segmentation workflows, which are able to process original PDF or image based files and store the resulting machine readable text.

To enable the further workflow steps, **develop the repository modules** for transferring files to processing service APIs and storing the results in the repository.

The raw input to the processing is any textual document (usually _PDF_), which is uploaded to the repository through the user interface. Subsequently, the document is sent asynchronously to segmentation and [OCR/HTR Workflow](). The output of this step is a file in _ALTO XML_ format with UTF-8 encoding. The file should be automatically processed and stored in the repository as a **new alternative distribution of the original document**.

A module for integration with additional processing tools must support the **asynchronous transmission** of the document's _ALTO XML_ distribution as **input** to a specified API endpoint and accept the **output** in _JSON_ format. This output should be then stored as another distribution of the document, providing a specific type of enrichment to the original data.

The **workflow calls** from the repository may involve user interaction or be executed as part of batch processing for selected datasets or specific documents.

Use automatic or manual invocation of the module implemented in Step 1 to **get** _ALTO XML_ data, transform it to _plain text_, and **transmit** it to the [UDPipe](https://lindat.mff.cuni.cz/services/udpipe/) REST API with the specified parameters.

Use the pre-trained models for the appropriate languages (e.g. Czech or English) and **tokenize** and **lemmatize** the input text with **default settings**.

The **output** is a _JSON_ file, which contains details about the model used, acknowledgements, and the results of tokenization, tagging, lemmatization, and dependency analysis, formatted in accordance with the selected output format (i.e., CoNLL-U). This output should be then stored as another distribution of the document, providing a specific type of enrichment to the original data.

Use automatic or manual invocation of the module implemented in Step 1 to **transmit**  _ALTO XML_ data to the [Keyword Extractor (KER)](https://lindat.mff.cuni.cz/services/ker/) REST API with the specified parameters.

The **parameters** are the language of input (Czech or English), minimum selection score (0.2 by default) and maximum number of generated keywords.

The **output** is a _JSON_ file, which contains a list of keywords, keyword scores for individual keywords and number of [MorphoDiTa](http://lindat.mff.cuni.cz/services/morphodita/) calls in the processing run. This output should be then stored as another distribution of the document, providing a specific type of enrichment to the original data. The repository data handling module should include **paradata**, such as the KER version and run parameters, either within the JSON file itself or in the metadata record of the distribution.

Use automatic or manual invocation of the module implemented in Step 1 to **transmit the output of the Step 2** (_CoNLL-U_ from the result element of the _JSON_ file produced by UDPipe) to the [NameTag](https://lindat.mff.cuni.cz/services/nametag/info.php) REST API with the specified parameters, while using [NameTag 3](https://ufal.mff.cuni.cz/nametag/3) models.

The **parameters** are the language of input (Czech or English), its type (_CoNLL-U_) and desired output serialization (_XML_ by default).

The **output** is a _JSON_ file, which contains details about the model used, acknowledgements, and the results of NER. This output should be then stored as another distribution of the document, providing a specific type of enrichment to the original data.

NameTag can be [further trained](https://ufal.mff.cuni.cz/nametag/1/users-manual#training_data) to **recognize custom tokens**.

In the described demonstrator, the goal is to enable the model to **identify accession numbers**, which are crucial for linking individual descriptive objects and parts of documentation. Additionally, the training can include specific **named entities aligned to specialised vocabularies**, such as those contained in the [AMCR vocabularies](https://teater.aiscr.cz/) (archaeological dating, types of sites and finds, materials, etc.).

The **training process** relies on the creation of a training dataset by providing a sufficient amount (300-700 examples per category) of manually annotated texts in [CoNLL format](https://github.com/ufal/nametag3/blob/main/examples/cs_gold_2.conll). Different versions of [NameTag 1](https://ufal.mff.cuni.cz/nametag/1) offer different opportunities for model training, with [NameTag 2](https://ufal.mff.cuni.cz/nametag/2) being the most complex, NameTag 1 relatively easy to train [following the manual](https://ufal.mff.cuni.cz/nametag/1/users-manual#custom_models), and [NameTag 3](https://ufal.mff.cuni.cz/nametag/3) potentially suitable for training but currently lacking appropriate documentation. Therefore, for custom models, it is recommended to primarily **use NameTag 1**, combined with version 3 for the recognition of generic named entities covered by the standard model.

For recognizing specific vocabulary terms and related entity linking, an [alternative workflow]() developed for processing of English written archaeological publications and grey literature can also be utilized.

The implementation creates opportunities to leverage additional services, such as the integration of automated translations. Given the expected volume of texts to be translated, translating on the fly would not be efficient. It is more practical to **prepare a basic translation from the minority language** — in this case, Czech — into a widely used language, such as English.

This is achieved by **integrating** the [LINDAT Translation](https://lindat.mff.cuni.cz/services/translation/) tool into the workflow. The tool provides a robust API with _plain text_ as **input and output**, enabling translation calls. These translations can be invoked either at the user’s request for specific metadata fields and data segments or as a common step of data and metadata post-processing.

The **response received** as part of the translation process is saved again as an alternative version of the translated file, either in _ALTO XML_ format or as a _JSON_ file containing the translations of specific metadata fields.

For smaller languages, such as Czech, this approach represents a critical step toward achieving genuine international usability for the data content of domain- and region-specific repositories.

After **processing and storing all data in steps 1–7 using the original repository** (the data should be kept as alternate distributions of the same dataset), the data is indexed and distributed using **discovery services**. This includes raw data files and their enrichment by individual tools. Discovery services should be tailored to leverage specific outputs for enhanced user services, such as:
* extraction of embedded digital objects identified by segmentation (**step 2** outputs);
* full-text search (**step 3** outputs);
* faceted search based on keywords (**step 4** outputs);
* faceted search based on named entities (**steps 5 and 6** outputs);
* enhancing international findability and accessibility (**step 7** outputs).

All these data should be **indexed together with other metadata** to create a comprehensive information package for user access and further re-use.

For the **demonstrator** described here, this workflow is to be implemented in the [AMCR Digital Archive](https://digiarchiv.aiscr.cz/). In addition to providing a graphical user interface that operates on indexed data, it is also going to offer **machine-readable data** through an [OAI-PMH API](https://api.aiscr.cz/).

Outputs from the API can be further **mapped to standard ontologies** (in this case, [AO-Cat](https://doi.org/10.5281/zenodo.7818374) based on CIDOC-CRM) and **aggregated** by other services, such as the [ARIADNE Portal](https://portal.ariadne-infrastructure.eu/) used in archaeology. Similarly, additional user tools and services for processing enriched data stored in the repository can be integrated through this pipeline or via a REST API, if available.

David Novák

Petr Pajdla

Pavel  Stranak

The workflow addresses the **machine processing and enhancement of archaeological textual data** (e.g., grey literature, reports, field forms) derived from image text recognition outputs or extracted from digital-born PDFs. The goal is to provide an integrated solution that facilitates the **processing of legacy data and new uploads within the selected repository system**. It aims to enhance searching and processing of documents while streamlining archival procedures by automating key steps in common archiving and annotation workflows.

By applying this workflow, multiple downstream applications become possible, including:
* automatic linking of related documents,
* quality assessment,
* data extraction,
* interlinking with other documents,
* and automated abstract generation.

These capabilities contribute to the long-term preservation and accessibility of archaeological documentary archives. Furthermore, the workflow supports **natural language processing (NLP)** applications, enabling the creation of corpora for analysis using large language models (LLMs) and their training.

The workflow emphasizes the sustainable **integration of existing tools accessible via APIs** as services. It leverages the outputs of these tools to meet specific user needs, particularly within data archiving and publishing workflows, ensuring adaptability and scalability in diverse use cases.

The [Archaeological Map of the Czech Republic (AMCR)](https://digiarchiv.aiscr.cz/) repository serves as a **demonstrator** in the workflow.


Edit actor

Other URLs relevant to the resource, e.g. ORCID, etc.

URL, ideally landing page, of the resource.

Person/group/body who helped create the resource.

Should be concise and raise interest in the entry. Allows <Link>Markdown</Link> content.

Other URLs relevant to the resource, e.g. code, DOI, etc.

Upload media

URL pointing to info about the access policy.

The activities you can do with the resource. See <Link></Link>.

Name of the conference where paper was given.

Describes the discipline covered by resource. See <Link></Link>.

Locations where the Resource is offered. See <Link></Link>.

URL to helpdesk for incidents & user requests.

Audience targeted by the resource. See <Link></Link>.

A particular published issue of a journal.

Concept or term related to MP entry. See <Link></Link>.

Language(s) in which a resource is available. See <Link></Link>.

Select license pertaining to resource. See <Link></Link>. If license is unknown, use the "Terms of use" property field to describe the usage terms.

Status of the Resource life-cycle. See <Link></Link>.

Mode of use for the resource. See <Link></Link>.

File format of the linked resource. See <Link></Link>.

Page #s of a resource, separate by commas.

Refine the resource's nature and function. See <Link></Link>.

Links to non-MP materials that are relevant.

Info about performance levels provider expected to deliver.

Standard used by the resource. See <Link></Link>.

Technology Readiness Level of the Resource. See <Link></Link>.

Resource user manual and documentation URL.

Version of the tool or service that is linked.

The volume of a journal or multivolume book.

Edit source

Name

If you are having trouble with the sign in process, please <ContactLink>contact the SSH Open Marketplace team</ContactLink>.

If you are having trouble with the sign up process, please <ContactLink>contact the SSH Open Marketplace team</ContactLink>.

Semi-automated extraction of information from textual documents in a domain-specific repository

Media

Related items(1)

AMCR Digital Archive

Workflow steps(8)

1 Selection of a target repository and meeting basic requirements

2 Integration to data processing workflows of the selected repository

3 Data enhancement for improved search

4 Keyword extraction

5 Named entity recognition

6 Named entity recognition - optional training for enhanced results

7 Automated translations

8 Integration to user workflows