Presentation at AEA on Adaptive Evaluation Designs


In collaboration with New York University and Sesame Workshop, D3 Systems presented a panel session titled “Adaptive Design for Education Evaluations in Post-Conflict Environments” at the 2016 conference of the American Evaluation Association.

Development and humanitarian programs that are implemented in the most challenging environments are often those most in need of rigorous evidence to demonstrate their effectiveness. But how does one overcome ground realities while preserving an evaluation’s validity? This panel reviewed the evaluation designs for two education interventions in Afghanistan and how they were adapted to the unique challenges of this post-conflict environment.

The Assessment of Learning Outcomes and Social Effects of Community-Based Education is a randomized control trial designed to assess the impact of community-based schooling in rural Afghan villages. NYU’s Vedrana Misic presented on challenges faced during the first phase of ALSE research, including organizational staff turnover, navigating ethnolinguistic and cultural sensitivity, and instances of noncompliance and attrition among study participants. She also described how changes in the intervention rollout have impacted the planned second phase of research.

D3’s Connor Brassil reviewed some of the fieldwork challenges that occurred in the course of collecting data for ALSE, including adaptations to the sampling methodology and other challenges, such as mix-gender interviewing teams, managing security with large field teams and repeated visits to sampled villages, and coordinating with local implementing partners.

Kyle Block from D3 Systems presented on D3’s collaboration with Sesame Workshop to design a rigorous, quasi-experimental impact evaluation that assessed the improvement in learning outcomes among young Afghan children from listening to Sesame Street broadcasts on the radio. Mr. Block shared how the design of the evaluation responded to unexpected challenges the evaluation encountered but was still able to maintain strong methodological rigor.

Download the presentation and report. For more information, please visit the ALSE website.