If you are having trouble with the sign in process, please <ContactLink>contact the SSH Open Marketplace team</ContactLink>.

If you are having trouble with the sign up process, please <ContactLink>contact the SSH Open Marketplace team</ContactLink>.

Subject

Social Sciences & Humanities Open Marketplace

Successfully submitted!

## Region segmentation  

With region segmentation, we identify the components of the layout, like text blocks, images, tables, columns, headings, footers, etc. After segmentation, the regions can be further processed or analyzed.  
The segmentation is done with **bounding boxes**, which can either be rectangular like in [YOLO](https://github.com/ultralytics/ultralytics) models, or polygonal like [Kraken](https://github.com/mittagessen/kraken) segmentation models.  


## Line segmentation 

When we perform line segmentation, we want to precisely detect text in an image, which is crucial because most of the current state-of-the-art ATR machine learning models rely on the correct extraction of text lines before transcribing them.  

There are two approaches:
* Drawing a **baseline** (or topline), either manually or automatically, before the software calculates the corresponding polygon to the text;
* Drawing a **bounding box**, either manually or autmatically, around the line of text. This means having only a rectangular mask without necessarily having a baseline.  


## Region vs. Line segmentation
These two processes are not mutually exclusive, nor do they need to be performed simultaneously. Many segmentation models can do both at once, some are trained to do only one. In fact, if you want to simply perform automatic text recognition, identifying the regions of the document is completely optional while detecting text lines is mandatory for most transcription software.  

Every segmented zone can be annotated using a **controlled vocabulary** which ensures that your data is:
* **Consistent**: everyone working on a document uses the same terms to describe the zones;
* **Searchable**: searching and retrieving specific information is made easier with annotated documents;
* **Interoperable**: data sharing is facilitated because the annotations are standardized;
* **Reusable**: an ontology enables you to mutualize annotated data.  

## The SegmOnto Ontology
We recommend using the **SegmOnto** controlled vocabulary. It is designed to "describe the content of books or manuscript pages, in order to homogenize the data required by layout analysers" (Simon Gabay, Ariane Pinche, Kelly Christensen, Jean-Baptiste Camps, & Nicola Carboni, _SegmOnto, A Controlled Vocabulary to Describe the Layout of Pages_, version 0.9, Genève/Lyon/Paris, 2023, [https://segmonto.github.io/](https://segmonto.github.io/)). Using the vocabulary offered by the SegmOnto ontology means that you can standardize your labels when annotating the layout of your documents. 

Text region annotations are useful in the context of digitization and creating digital editions. It allows information extraction and document understanding.  

Raw text, as opposed to structued text, presents limitations for extracting and processing the information it contains. Without finely-structured content, we provide the machine with a degraded version of the document we want to process. We thereby limit its ability to reproduce the information extraction that we, as humans, would be capable of. Without spatial and logical information, the content becomes nothing more than a continuous text without hierarchy, especially when we are dealing with highly structured content such as glossed texts, tables, dictionaries, etc. Extracting the structure allows us to retain the semantic meaning embedded in the layout.  

Layout Analysis is mandatory to get to the text level, but also to the page level and document level. It allows us to get information about the document itself, namely, for downstream tasks like:
* facilitating the navigation inside a transcribed document, 
* reducing the scope of queries, 
* ignoring sections, 
* plugging different tools to process these regions depending on the content of the regions (i.e. applying information extraction tool in target zones), or 
* simply to represent the logical structure of a document.

Layout analysis is often included in expert ATR software like [eScriptorium](https://escriptorium.inria.fr/) or [Transkribus](https://www.transkribus.org/en). There are also tools designed only for layout analysis, such as [dhsegment](https://github.com/dhlab-epfl/dhSegment) and [LAREX](https://github.com/OCR4all/LAREX). Additionally, there are environments designed for any type of image annotation, including layout analysis, such as [Label Studio](https://labelstud.io/) or [Roboflow](https://roboflow.com/).  

Annotation analysis can be done manually by drawing bounding boxes and text lines, or by using machine learning models trained to do that task.

A segmentation model can learn how to segment text lines and regions, and how to label them according to an ontology of line types and region types, while manually annotating the training data.

There are usually four methods with which you can perform automatic layout analysis:
1. with convolutional neural networks,
2. with transformers,
3. with object detection models,
4. with heuristics or rule-based methods.

Annotations are stored using specific XML standards such as PAGE XML and ALTO XML. These two formats have their own schemas that intend to store the details of the layout of the segmented documents in combination with the text resulting from automatic recognition. They work at the image level. Annotation files are automatically generated by ATR software.

## Manual annotation

1. You have a corpus you need to segment before transcribing.
2. You import your corpus into an ATR software.
3. Your corpus is small and can be annotated manually.
4. You don’t need to annotate the text regions
    1. You annotate the text lines
    2. Decide on the ontology you will use to label text regions
    3. Label your text lines
    4. You transcribe the text lines
    5. And you can then export you data in ALTO XML or PAGE XML
5. OR you want to annotate the text regions
    1. You annotate the text regions and the text lines
    2. Decide on the ontology you will use to label text regions
    3. Label your text lines and your text regions
    4. You transcribe the text lines
    5. And you can then export your data in ALTO XML or PAGE XML  


## Automatic annotation

1. You have a corpus you need to segment before transcribing.
2. You import your corpus into an ATR software.
3. Your corpus is too voluminous, you can use a model to annotate it
4. You don’t need to annotate the text regions
    1. You can use a segmentation model to segment the text lines
    2. Decide on the ontology you will use to label text regions
    3. If the segmented text lines are not good enough, you can correct them and re-train a new model with them as training data
    4. Label your text lines and retrain a model with labeling examples so that the model learn to classify the text lines and the text regions
    5. Then, you can transcribe the segmented text lines
    6. And you can export them in ALTO XML or PAGE XML
5. OR you want to annotate the text regions:
    1. You can use a segmentation model to segment the text lines and the text regions
    2. If the segmented text lines and/or text regions are not good enough, you can correct the data and train a new model with it
    3. Label your text lines and retrain a model with labeling examples so that the model learn to classify the text lines and the text regions
    4. Then, you can transcribe the segmented text lines
    5.And you can export them in ALTO XML or PAGE XML

Alix Chagué

Hugo Scheithauer

Layout analysis is a computer vision tasks that allows us to identify the structure of a document and localize the lines of text. We generate a series of _x_ and _y_ coordinates corresponding to regions/lines on the image, which we associate to a corresponding labels.  

Text line segmentation is mandatory in most ATR systems, but without a good line segmentation, transcription performances will not be sufficient. For example, lines can be missed, read in the wrong order, or even broken down into several lines. 

Synonyms: "segmentation", "zoning", "document analysis", "optical layout analysis".

ATR (4) - Layout Analysis

Media

Workflow steps(7)

1 Segmentation

2 Annotation

3 Why do we need Layout Analysis?

4 Software

5 Technologies used by Layout Analysis

6 Saving the results

7 SUMMARY OF THE WORKFLOW