Project Overview

While assessing intelligent document processing, it became evident that IDP systems work great in extracting required data from the documents; however, with handwritten text, the industry standard is much lower at 64% accuracy. Moreover, production dataset has more coherent data and the handwritten text is also subject to human perception of it. I figured out that latest multimodal LLMs do run on images & videos as well. A client of ours had handwritten medical forms over printed text that they wanted to digitise, which became a perfect opportunity to try out these models.

My Technical Approach

I trialled out Microsoft Phi4, TrOCR - YOLO (You Only Look Once), and Qwen 2.5 Vision models available on Hugging Face for local execution, out of which Qwen performed the best. The solution was built on Azure Kubernetes Service, where the individual pod was spec’d to use A100 GPU to achieve this. I was able to host a Python Flask server and a Streamlit app to showcase the capability of this model to decipher handwritten text with 80% accuracy over these forms. I implemented supervised fine-tuning over a set of 100 images to train the model using LORA (Low Rank Adaptation) and Zero 3 framework to offload compute to CPU as the A100's 80 GB capacity wasn't enough. The outcome was an SFT trained version with an accuracy of 91% over the next dataset provided by the client.

Ongoing Development

The solution is being worked on at the moment to enable this as a API based headless engine to extract data from digital documents, where internally the document would be split in images and with certain preprocessing using CV libraries eventually be consumed by multimodal LLM to decipher results. With this solution, it was possible to restrict client's data in Australia for any execution thereby falling into adept privacy laws and ISM policies laid out by Australia.

Requirement

Over the period of time, one of the most common requirement I have faced is to build visualisations for SES staff to showcase what the system is capable of and how is it performing for their business.

Tools Used

The following are the tools that I have used for various reporting requirements -

• Tableau - with data sourced from SAP, Salesforce and Oracle platforms.
• Kibana - sourced from various sources using Beats, Logstash and Elastic Agents.
• Splunk (on Prem) - sourced from leading cloud hyperscalars like Azure and AWS.
• SAP Analytics Cloud - sourced from SAP CRM and PSCD systems.
• PowerBI - various data sources within microsoft landscape.

Presales Work

For presales work at Reason Group, it was necessary to provide valid solution architectures and assess best possible technical outcomes for a customer. As a part of proving the ROI, I have accomplished many proof of concepts with the following products to create a capability matrix that can go along with RFT and RFQ responses -

• SAP Tax and Revenue Management System
• SAP S4HANA
• SAP Business Technology Platform
• D365 Sales, Service and Field Service
• Salesforce
• ServiceNow
• TechOne
• Halo ITSM
• Ivanti
• IBM QRadar SIEM, Splunk
• Elasticsearch and Kibana
• Open Fisca
• Cleaner Cars
• TextEngine.io
• Azure, AWS and Oracle Cloud based micro service offerings including AI/ML services.

Clients

Clients for the RFQs and RFTs were both federal and private agencies.

Overview

One of the most common requirement across projects is to Integrate and Orchestrate. Below are some use cases, platform and tools used by me -

Use Cases

1. Integrate with enterprise systems of federal agencies - using SAP PI/PO for exchanging SOAP/XML based messages with SAML security context.
2. REST APIs published by other organisations using JWT, Key based and certificate based authentication.
3. Adapter based integrations like Outlook, CTI, BAPIs etc.
4. Business Process Modeler based orchestration and integrations.
5. Asynchronous event based message processing.

Tools

Postman, insomnia, SOAP tester, VsCode with plugins, respective hyperscalar's integration workbenches are some of the tools I've worked with.

Project Scope

State Revenue Office has over 200 types of documents spanning over smartforms, paper forms and handwritten letters that they receive in volumes of over 100,000 monthly submissions. Majority of these documents have information that should be applied in the Tax and Revenue Management system to carry out revenue assessments. They wanted to introduce a capability to digitise this information in a standard way and represent it in their existing document repository system and also apply the digital dataset in TRMS.

My Role and Responsibilities

I was the solutions architect and delivery lead on this entire engagement and the following activities were undertaken to accomplish the task -

• Market research for leading IDP providers, which included their existing Kofax system (cloud version), Microsoft IDP, Oracle's document understanding and Abby; with due diligence Oracle document understanding was chosen the best fit.
• Generate business case, detailed costings, project roadmap, and detailed solution architecture of decommissioning Kofax and introducing OCI with Document Understanding solution.
• Implement the solution based on the solution design and retrofit technical learnings to bridge any gaps that were discovered.
• By using multi label classification of the images and custom model for key value extraction, it was possible to identify and extract data from the documents to be applied in TRM system.
• With the help of OIC 3, we built endpoints to replace XML files that Kofax Legacy system generated for legacy system to carry out post processing of the items.
• For physical scans we had a requirement to split the documents based on a particular datestamp that the team used to stamp every letter that they have received. This was achieved using computer vision based PyTesseract lib, before the documents were transactionalised per customer in the solution.

Outcome

The solution had over 90% accuracy over printed text forms which was the case with the department.

Project Context

For federal or state government client, Australia has service panels like Digital Transformation Agency setup which standardises the way how agencies can come to industry for any ICT work. This usually is in the form of Services or Hired Labour. Both of them come up as opportunities on ICT which has to be responded by with documentation specifically tailored for the opportunity. This project was about ingesting any of the two and generate a response to the bid that can be reviewed by human to be submitted later.

Data and Design

In terms of data points, Reason Group already has a pool of resources and their respective CV and Skillset. It already has a databank of the bids filed across for numerous agencies in Sharepoint. I designed the solution in Azure Kubernetes Service by hosting two pods, one for Ollama where Phi 3 was chosen for execution to generate the response. On the other pod, we resolved to anythingllm as product to assist with Retrieval Augmented Generation and use inbuilt vector database (LanceDB) to support the text embedding and tokenisation of the previous bids that Reason Group has applied.

Results

As an outcome, we were able to generate responses for hired labour task successfully, where the system used to scan through profiles available in the job pool, match the skills, generate the response to the selection criterion as close as possible based on the experience summary.