Open-Source LLM for Automated Document Parsing

The situation

A company relied on manual data entry to convert unstructured text records into structured formats for its internal management system. Staff received confirmation documents in free-text format from external partners. Each document contained quantities, identifiers, dates, classifications, and monetary values, but the formatting varied widely between sources. An analyst would read each document, identify the relevant fields, and enter them into the system by hand.

The process was slow, error-prone, and did not scale. As transaction volume grew, the manual bottleneck consumed increasing analyst hours that could have been spent on higher-value work. The company needed a way to automate the extraction without sending sensitive records through third-party APIs.

What we built

CorrDyn designed and delivered an end-to-end machine learning pipeline for automated document parsing, from training data creation through production model serving.

Benchmark dataset creation. We built a custom QA portal that paired each raw text input with its expected structured output. A team of CorrDyn analysts validated over 3,000 examples against detailed field-by-field criteria, supported by heuristics that flagged examples likely to contain errors. We defined four categories of training data error: hallucinated fields (model outputs information not present in the input), missing information (model omits data that is present), incorrect mappings (fields contain wrong values), and incorrect ordering (outputs do not match input sequence). Each example was reviewed for all four error types before inclusion in the training set.

Synthetic data development. To ensure the training set covered the full space of potential outputs, we developed a synthetic data pipeline that generated realistic document variations. We experimented with different ratios of synthetic to real examples, measuring model performance at each level. The final training set used a 4:1 ratio of synthetic to actual examples — enough to fill coverage gaps without diluting the signal from validated production data.

Open-source LLM benchmarking. We evaluated multiple open-source language models for structured data extraction accuracy, latency, and resource requirements. Models were fine-tuned using the validated benchmark dataset on CorrDyn’s H100 GPU infrastructure with HuggingFace and Axolotl. We iterated through multiple training runs, adjusting hyperparameters and evaluating each model version against a holdout set until accuracy met production requirements.

Production serving infrastructure. The final model was packaged as a Docker container with a FastAPI interface for serving predictions. The container image was stored in Google Artifact Registry alongside the model artifact in Cloud Storage. We evaluated both dedicated GPU hosting and serverless GPU options to balance cost against latency, ultimately deploying a solution that runs inference during business hours to minimize infrastructure spend.

Automated retraining pipeline. We built a monthly retraining workflow that incorporates new validated examples from production, retrains the model, evaluates it against the benchmark, and promotes it only if performance exceeds the current production version. This closed-loop system ensures the model adapts to evolving input patterns without risking degradation.

What changed

Analysts who previously spent hours per day on manual data entry now review automated outputs, intervening only when the parser flags low-confidence results. The system processes documents in seconds that previously took minutes of manual work per record. Because the model runs entirely within the client’s infrastructure, the security team approved it without the data-handling exceptions that would have been required for a commercial API integration. The retraining pipeline keeps the model current as new document formats and edge cases appear in production.