Public Labor Market Dashboard for a National Workforce Agency

The situation

A national labor exchange program had received grant funding to make its job posting data available to researchers, workforce development professionals, and the public. The data was extensive: terabytes of current and historical records covering job volume, location, occupation, and employer information across the United States.

The existing infrastructure stored this data in an Aurora PostgreSQL database optimized for transactional workloads (inserting and updating millions of jobs daily). It was not designed for analytical queries. Researchers who wanted access had to download bulk export files into their own environments. There was no interactive way to explore the data, and every custom request required staff intervention.

The organization needed three things: a data warehouse that could aggregate terabytes for multiple users simultaneously, a public-facing visualization interface with latency that met web standards, and an architecture with total cost of ownership under $25K per year.

What we built

CorrDyn evaluated the data warehousing and visualization landscape against the organization’s cost constraints and concluded that open-source and open-core tools were the right fit. We built a proof of concept using MotherDuck for the analytical warehouse and Evidence, an open-source data analytics framework built on DuckDB and SvelteKit, for the public-facing interface.

The architecture works by aggregating data server-side in MotherDuck, then transferring results to a DuckDB instance running in the user’s browser. This client-side execution minimizes latency for the end user and keeps server costs low. We built a data pipeline between the organization’s existing transactional database and MotherDuck to refresh the public dashboard on a schedule that balanced freshness against cost.

We interviewed stakeholders to define what questions the dashboard needed to answer: largest occupations, fastest-growing job categories, geographic distribution, and trends over time. We contributed additional analytical questions based on our experience with labor market data and iterated through several dashboard versions with feedback from internal and external stakeholders before public release.

What changed

The organization released the public-facing dashboard to researchers and workforce professionals with strong reception. Questions about the U.S. labor market that previously required bulk data downloads and custom analysis could now be answered interactively in a browser. Total cost of ownership came in under the $25K annual target, even accounting for growing usage. The architecture requires minimal ongoing maintenance, which was a core requirement given the organization’s limited internal technical resources. CorrDyn has continued the engagement, building additional data products and enhancements on top of the same infrastructure.