Modernizing America's Labor Market Data Platform

The Challenge

The National Labor Exchange (NLx) is a public-private partnership between the National Association of State Workforce Agencies (NASWA) and DirectEmployers, aggregating job posting data from across the United States. Their legacy data warehouse had become a bottleneck: it couldn’t securely share data with the growing community of labor market researchers who needed access, and it lacked the architecture to scale with increasing data volumes. The system had no API layer, meaning every data request required manual intervention, and there was no way to enforce the granular access controls that different research agreements required.

Our Approach

CorrDyn started with a needs assessment that included NLx leadership, NASWA stakeholders, and representative researchers. From those conversations, we defined requirements for the modernized platform. We designed an event-driven architecture that processed incoming XML job posting data from S3, transforming and loading it into a serverless Aurora PostgreSQL database that scaled automatically with demand. On top of the data layer, we built a containerized Django API with a credential management system supporting multiple access tiers. The API provided two modes: a synchronous endpoint for real-time queries powering dashboards and applications, and an asynchronous endpoint for bulk data downloads that researchers could use to pull large datasets without impacting system performance.

The Results

The modernized platform delivered $500,000 in cost savings compared to the legacy system’s operational overhead and the manual processes it required. For the first time, hundreds of labor market researchers across universities, government agencies, and policy organizations could access NLx data through self-service API credentials, with access scoped to their specific research agreements. The dual API architecture ensured that real-time applications and bulk research workloads could coexist without performance degradation, and the serverless infrastructure scaled seamlessly during peak periods without manual intervention.