![\"\"](\"https:\/\/gini.net\/wp-content\/uploads\/Gini-OCR-Figure-1-1200x466.png\")
Figure 1. An example image processing and OCR pipeline in Gini. Left: an example input image. Middle: an ideal preprocessed image. Right: OCR results in JSON format.<\/p><\/div>\n
<\/p>\n
Image Processing<\/h2>\n
The performance of OCR strongly relies on image quality. Therefore, in order to enhance the quality of images, we first implemented a series of convolutional neural network-based models that perform the following data transformations:<\/p>\n
- \n
- Orientation detection and rotation.<\/li>\n
- Homography detection, cropping, and perspective warping.<\/li>\n
- Region of interest detection, e.g. tables, QR codes, or remittance slips (\u201c\u00dcberweisungstr\u00e4ger\u201d in German, a widely used bank transfer form in Germany and Austria).<\/li>\n
- Binarization and de-noising of certain regions of interest (ROIs).<\/li>\n<\/ul>\n
The images obtained after a series of such transformations have corrected orientation, restored perspective, and improved overall quality (Figure 1).<\/p>\n
Text Detection<\/h2>\n
The improved images are now ready for our OCR, and the first step is text detection. It is important to note that modern OCR no longer works on character level but rather utilizes the recent advances in NLP and computer vision deep learning techniques to predict at sequence level (Figure 2). Instead of characters, we extract the snippets of words or even phrases from document images.<\/p>\n
![\"\"](\"https:\/\/gini.net\/wp-content\/uploads\/Gini-OCR-Figure-2.png\")
Figure 2. An example of a common text detection model (EAST, Zhou et al., https:\/\/arxiv.org\/abs\/1704.03155).<\/p><\/div>\n
After the text detection, the full invoice image is split into a batch of snippets, each consisting of 1 or more words closely connected horizontally (Figure 3). It enables the text recognition step to handle them as a sequential word prediction task (a well-studied problem in the NLP domain).<\/p>\n
![\"\"](\"https:\/\/gini.net\/wp-content\/uploads\/Gini-OCR-Figure-3-1200x732.png\")
Figure 3. Left: preprocessed image. Right: detected image snippets.<\/p><\/div>\n
Text Recognition<\/h2>\n
Once we have batches of image snippets of single-line words or phrases from the text detection step, it is time to transform them into actual text (Figure 5).<\/p>\n
![\"\"](\"https:\/\/gini.net\/wp-content\/uploads\/Gini-OCR-Figure-4.png\")
Figure 4. The flow of 4-stage scene text recognition (Baek et al., https:\/\/arxiv.org\/abs\/1904.01906).<\/p><\/div>\n
A typical state-of-the-art text recognition model (Figure 4) consists of the following components:<\/p>\n
- \n
- Transformation: changing an irregular-shaped text image into a normalized, straight single-line snippet using a certain spatial transformation network.<\/li>\n
- Feature extraction: converting the input images into convoluted feature maps using a common convolutional neural network (e.g. VGG, ResNet, RCNN). The feature maps contain important visual features emphasized and irrelevant visual features suppressed.<\/li>\n
- Sequence Modeling: capturing contextual information via a recurrent neural network (e.g. BiLSTM).
\nPrediction: predicting text using sequences of feature vectors from the previous step.<\/li>\n<\/ul>\nAfter our image preprocessing, the invoice text should be fairly well-structured, not curved or distorted. This allows us to skip the above transformation step and reduce some computational costs. In other words, we are using only 3 out of 4 text recognition steps.<\/p>\n
![\"\"](\"https:\/\/gini.net\/wp-content\/uploads\/Gini-OCR-Figure-5-1200x538.png\")
Figure 5. Left: image snippets after the text detection. Right: text recognition results.<\/p><\/div>\n
Post Processing<\/h2>\n
With the text predicted via the model described above, bounding boxes calculated via the coordinates of the image snippets, we are only one step away from the information extraction step.<\/p>\n
To help our text extractors achieve higher accuracy, we need to perform some post-processing corrections.<\/p>\n
- \n
- Post-OCR text correction: since the text recognition process already involves a sequence prediction model, there is no need for a lot of correction \u2013 the model learns enough information about the language. We only perform some extra corrections with the help of dictionaries. Additionally, we are planning to look into using the pre-trained language models to improve the prediction quality further.<\/li>\n
- Reading order correction: despite the fact that we use coordinates in our downstream extraction tasks, the reading order still influences the extraction accuracy. To mitigate this, we implement some reading order correction methods which help us to reconstruct the intended order as close to the original as possible. In the future, we may also implement a pre-trained order detection model (e.g. https:\/\/arxiv.org\/abs\/2108.11591).<\/li>\n
- JSON\/XML output creation: finally, we put the results into a structured, deserializable output format for both internal downstream information extraction services and API used by our clients\/users. We support both JSON and XML formats.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/gini.net\/wp-content\/uploads\/Gini-OCR-Figure-6-1200x600.png\")
Figure 6. Left: text recognition results. Middle: text after reading order correction. Right: final OCR output in JSON format.<\/p><\/div>\n
In the end, the structured layout in XML format is passed to downstream information extraction tasks.<\/p>\n
Summary<\/h2>\n
Using a complete, in-house developed pipeline of deep learning-based image processing, OCR, and information extraction, we provide a fast and accurate solution that is specialized for mobile input. It helps banks and insurance to gather the best possible quality from mobile phone input in a decent amount of time to drive great user experiences and stickiness of the features like Photo Payment<\/a> or Pay Connect<\/a>.<\/p>\n
.\u00a0 \u00a0 \u00a0 \u00a0 .\u00a0 \u00a0 \u00a0 \u00a0 .<\/p>\n
<\/p>\n