Analysis proves patience is needed to properly measure the impact of a grant

Programs conclude. Participants submit final reports. Immediate outcomes are captured. Then attention shifts to the next round.

But many grants are designed to create change that unfolds over years, not weeks.

Case study: our examination of performing arts funding impact

Over the past 3½ years, Our Community and SmartyGrants worked with two international performing arts organisations on a longitudinal evaluation.

Both organisations help artists to find new audiences in their own countries and across the world through festivals and special events.

The aim was to understand how participation in these festivals and events influences professional development over time, not just immediately after the event.

The initiative followed 75 artists through a baseline survey and four follow-ups over approximately 2½ years.

Participants were paid for their time. Surveys were structured around original goals, decision-making and relationship development, and they invited respondents to be reflective.

As expected, participation reduced over time. Even so, the longitudinal data provides rare insights into how professional impact develops in complex international contexts.

Our Community provided the evaluation design and the SmartyGrants platform pro bono, alongside staff time and direct financial investment.

Longer-term study uncovered medium-term impacts

The most consistent shift identified in the data was in confidence. Participants reported increased confidence in:

• engaging with decision-makers
• being selective about opportunities
• assessing professional risk
• articulating their work and strategy.

For one cohort, survey data showed a statistically significant increase in self-reported confidence two years after participation in festivals and events. That level of statistical strength is unusual in small longitudinal samples.

Participants also described a common pattern: an initial post-event slump followed by renewed clarity and momentum. That cycle is widely discussed anecdotally. This study provides structured evidence of the phenomenon, and also underlines the weakness of evaluations that take into account only immediate outcomes.

Importantly, participants did not attribute outcomes to a single intervention. Impact was cumulative, shaped by multiple experiences, funding decisions and relationships over time.

Why this matters for grantmakers

Short-term reporting should capture early indicators that may shape later outcomes, even when final results are not yet visible.

Changes in confidence and decision-making can influence:

• who participants follow up with
• what risks they take
• which opportunities they decline
• how they position themselves.

These shifts often occur well before longer-term outcomes become visible. However, they can also fluctuate over time. Early signals such as dips or gains in confidence may change or rebound as participants continue through a program.

For grantmakers, the task is not to treat these indicators as final outcomes, but to track them as early signals that may help explain later results and guide program improvement.

How SmartyGrants tracked this impact

SmartyGrants was used not only for program applications, but for the multi-year follow-up surveys. Each survey response was tied to a single secure record. This enabled:

• automatic carry-forward of each participant’s original goals
• structured longitudinal reflection
• consistent data integrity
• combined qualitative and quantitative analysis.

Participants effectively evaluated their own progress against baseline goals. That depth of reflective tracking is difficult to achieve using standalone survey tools.

“What made this evaluation unique was how the technology enabled a ‘journal-style’ follow-up process,” said report author Jen Riley, until recently SmartyGrants’ chief impact officer.

“Using SmartyGrants, participants were able to return to the same form over time, progressively adding updates as their projects evolved. This allowed them to reflect on their original intentions and assess their progress against their own goals, rather than responding to a static, one-off evaluation.”

Longer-term analysis an opportunity for other sectors

Longitudinal, reflective evaluation is practical and cost-effective. It does not require large samples or complex infrastructure. It requires thoughtful design and a platform capable of linking data over time.

The implications extend beyond the arts sector. Leadership programs, workforce development initiatives, community capacity building grants and innovation funds all face the same challenge: outcomes emerge slowly and not always how we expect.

Grantmakers who invest in not just monitoring early signals but measuring results over time will have stronger evidence and clearer narratives, and will ultimately make better funding decisions.

More information

SmartyGrants is committed to advancing evaluation practice, not just administering grants. If you are interested in testing longitudinal evaluation within your programs, we would welcome a discussion.

Contact our Managed Services team at service@smartygrants.com.au (Australia and New Zealand) or service@smartygrants.co.uk (UK/EU).