Data Infrastructure for a Regional Food Bank

The situation

A regional food bank serving hundreds of thousands of households each year had invested in an IT team and operational data systems, but lacked the infrastructure to turn that data into organizational intelligence. Program visit data (household demographics, dietary needs, income levels, provisions distributed, and program participation) was stored in a case management system across multiple related tables. The data existed, but there was no reliable way to aggregate it for leadership reporting, program evaluation, or grant compliance.

The organization’s leadership wanted to understand whether their programs were reaching the right populations, whether resources were allocated efficiently, and how to demonstrate measurable impact to funders. These questions required data from multiple systems to be integrated, cleaned, and made accessible for analysis. The internal IT team had built initial data pipelines, but the organization recognized it needed external expertise to assess the full landscape of its data capabilities and build a roadmap for sustainable improvement.

What we built

CorrDyn conducted a data assessment that included interviews with stakeholders across leadership, program management, and IT. The assessment examined how data flowed through the organization, where gaps existed between available data and decision-making needs, and what infrastructure investments would produce the highest return on organizational effectiveness.

Based on the assessment findings, we built automated ETL pipelines that extract program visit data from the case management system into a data warehouse running on Azure. The pipeline handles nine source tables covering visit activities, provisions, household addresses, dietary considerations, income, ethnicity, self-identity, household members, and social program participation. Data lands in the warehouse on a daily schedule, where it is available for reporting and analysis.

When pipeline runtimes began hitting timeout limits as data volume grew (some tables taking over 3,000 seconds per pull), we optimized the pipeline to use incremental loading. Daily runs now pull only new and updated records based on timestamp and ID columns. A weekly full refresh serves as a safety net. The optimization eliminated the timeout failures that had been disrupting the daily data refresh and reduced routine runtimes to a fraction of what full table pulls required.

What changed

Leadership gained access to integrated program data for the first time. Questions about program reach, household demographics, and provisions distribution that previously required manual data pulls from the IT team can now be answered from the warehouse directly. The daily refresh cycle runs reliably without the timeout failures that had plagued the full-table approach. The data infrastructure positions the organization to build dashboards, automate grant reporting, and measure program impact with the rigor that funders increasingly require.