For the past month or so, my team has been on the road (literally) collecting on-site data for our various evaluation projects (e.g., conducting on site observations of classroom or instructional and after-school activities, completing formal and informal interviews with project staff members and stakeholders, attendance at the Board, steering committee/advisory meetings, completing focus groups with participants and parents/stakeholders, and debriefing the team before leaving).
This post is on the utility of conducting formative evaluations, providing just-in-time feedback to the team to allow for programmatic iterations/changes to make the program better. This is called the feedback-loop in an evaluation process.
Simply stated, formative evaluation takes place during the implementation phases of a project (early years) with the aim of improving the planning, designing, and performance of the project being implemented. Evaluations of this type are sometimes also referred to as “playing small ball,” a baseball analogy that suggests that the primary goal of a formative evaluation is not to hit a Grand Slam but to simply make a base hit. Sometimes, formative evaluations take place throughout the grant period and overlap with Developmental and Summative evaluations (now, that’s a post for another day.)
At the end of each visit (that lasted anywhere from 5 to 10-12 hours spread across 1 to 2 days), we immersed ourselves to better understand and assess the nuts and bolts of the program as proposed vs. planned vs. as being implemented.
Keep in mind that we are not out of COVID yet and everyone is simply taking a deep breath and trying very hard to catch up.
Suffice it to say, in some instances, what we observed was radically different from what was being shared with us (virtually) and on paper.
We were a bit taken aback. One of the team members was also a key witness to a somewhat serious student participant incident while on site that was not handled well. There were little to no consequences to the action and business was conducted as usual.
After our formal observations, my team sat down with the respective project team members (in one case, they had to leave the premises at the end of the day and set up a zoom meeting instead for a debriefing session) to go over what they saw/observed, learned, seek clarity, and where warranted, provided feedback. Verbal informal feedback was followed by a memo-style written feedback documenting the observations with good practices, lessons learned, and recommendations for change within 7-10 work days.
Upon receipt, there was some push back from one of the project teams whilst most of the others were receptive to the recommendations my team made with a plan/promise to follow-through and make amends.
The simple point I am trying to make here is that as program evaluators, it is our job to conduct a thorough formative evaluation- on or off site (as feasible) with due diligence with our eyes and ears open and Say Something When We See Something-both for good and not-so-good things.
