Since ’tis officially the season, I am back with my top instances and situations that tend to spook us evaluators.
I was thinking about them while on the metro from the AAAS-organized #SSTEM symposium that concluded today, which by the way was very well attended (~750 folks) and the panel on evaluation on Friday AM was attended by 126 people (per the Whova app). The room’s capacity was ~100 so you can image how crowded the session turned out to be. It was really fun….not spooky at all.
If you are on twitter, then check out some cool pictures there- @KavitaMNA, #2022SSTEM.
Back to my list. Here are my top 10 and a sub-part. I am sure there are several more.
10. Serving as “regular/designated” meetings note takers and also ending up acting as project managers and data coordinators on the grant we are actually evaluating. (Not our job. But, it has happened.)
9. Clients interfering with evaluation activities. (Mainly because they don’t understand what evaluators do on- and-off site or sometimes they are just suspicious.)
8. Receiving data really really late (after several requests) and then the data look kind of suspicious (i.e. probably falsified).
7. Asking formative evaluation to be conducted by one evaluator and summative evaluation by another. (I know this seems rather strange, but this has happened too.)
6. Asking for an RCT or at least a QED because they read about them somewhere, it’s so cool and these score them high points with the reviewers. (“Isn’t there like a boiler plate narrative around them you could easily use?” they say.)
5. Conflating research and evaluation. (ooo, that’s one of my fav.)
4. Requesting to pick and complete someone else’s evaluation/work (mid-way or towards the end for a small budget or a stipend).
3. Requesting to complete a whole evaluation for $5K or less. (“It’s only basic data analyses that even I can do but not allowed to and a short report,” they say.)
2. Receiving a call on Friday at 4.45PM to write a (really robust) evaluation plan for a big proposal that’s due that day at 11.59pm ET (come on, you have 7+ hours, can’t you do it? “Surely, you have been in this situation before,” they say.)
1. Inferring causation from correlation (sure, why not?!)
1A. Changing the evaluation budget drastically after the award is made; of course, without informing the evaluator. (“We had to change it based on the agency’s request but the eval plan is totally unchanged so there is no need to worry,” they say. Well, that’s WHY we are really spooked!)
1B. Losing our invoices repeatedly. (Honestly, I haven’t been able to understand how this really happens.) And paying us very very late! (*sigh*)
1C. Asking us to work without a contract. (See my post from last week.) (“It’s only a few hours,” they say.)
Did I miss anything? Maybe the list should be much longer. (Edit: it is getting longer.)
