Earlier this month, I received an invitation from the Hispanic Serving Institutions STEM Resource Hub (HSISTEMHub) based at the New Mexico State University to write a piece on the differences between a Research plan vs. an Evaluation plan for National Science Foundation grants.
I thank the HSISTEMHub for the opportunity. As someone who often writes either (or sometimes both) a research and/or an evaluation plan for NSF grants, this topic is very near and dear to me.
And unfortunately, this area can also be one of the sticking points in an otherwise strong proposal that could lead to it being declined for funding.
In my experience, there are a few main reasons for this:
1. General confusion around terminologies, definitions, and differences between the two
2. Lack of understanding of what each area entails for an NSF grant (read them carefully)
3. Inability to write clearly how the two areas are different yet converge at some points to complete a grant, and
4. Not hiring a researcher and/or an evaluator early enough (pre-award phase) so they can plan, coordinate, and write their sections well.
I hope my piece below provides some guidance on this topic for PIs as well as researchers and/or evaluators as they plan and write their respective sections for the next solicitation.
Needless to say, always engage your researcher and evaluator early and often.
My piece appeared here: https://hsistemhub.org/portfolio-item/august-2020-newsletter/
*********************************************************************************************
In general, National Science Foundation solicitations including HSI, ITEST, IUSE, and S-STEM, require separate research and evaluation plans. It is important to be able to distinguish between the two and write both sections well for a competitive proposal.
Research vs Evaluation
Several fellow researchers and evaluators have done a tremendous job of explaining the differences between the two including: John LaVelle (AEA365 blog post ); Sandra Mathison’s book chapter; and Patricia Rogers’ Better Evaluation blog post.
Research and evaluation are conducted using similar data collection and analytic methods. The difference is in the intent or purpose and the audience to which results will be reported.
In essence, the research aspect of an NSF grant is intended to test a hypothesis and its results are generalizable in nature. Research measures participant-level metrics, mediating, and moderating factors in a study.
Research asks: What’s so? How does it work? questions.
Typically, results are reported at the end of the research project and are meant to inform the field of study.
The evaluation aspect of an NSF grant particularizes as it is designed to improve the program itself. Evaluation assesses the value or merit of the program.
Evaluation asks the So what? and How well does it work? questions.
Therefore, evaluation results should be provided to principal investigators, Co-PIs, other project leads, and major stakeholder groups throughout the life of the study, and not just at the end.
Research Plan
A research plan typically comprises:
- Hypothesis- The hypothesis can examine the intervention’s outcomes to measure participants’ achievements in STEM courses, training, careers/jobs, and the gain in knowledge that may be the result of the intervention(s). (Hint: If/Then)
- Research Question- The research question answers questions such as:
- What is….?
- How does….?
- In what ways do/es (a particular intervention) work?
- How does it show its intended effects on participants’/stakeholders’ cognition and/or affective behaviors?
- If and how participants’ background characteristics (e.g, demographics, academic performance measures, affective factors) play a role in decision making for choosing STEM majors, engaging in training opportunities, utilizing available resources, services, and careers/jobs when exposed to the intervention(s)?
- Study Plan- The study plan refers to the type of research design the researchers intend to use. It could be a single case pre-post design, a quasi-experimental design, or a randomized control trial. The study design will depend on the type of research questions that are being developed, the feasibility of undertaking a complex study, time, and budget among other factors.
- Analytical techniques- Analytical techniques comprise the types of analyses the researcher will undertake to derive findings from the study. Qualitative data collected via interviews, focus groups and other means would require coding the data to develop common themes to build narratives. Quantitative data can be analyzed using descriptive or predictive analyses which would depend on the data quality and quantity (e.g., sample size) and outcome measures that may require more complex analytical techniques.
Evaluation Plan
An evaluation plan typically comprises:
- Evaluation Questions that begin with:
- To what extent…?
- With what fidelity…?
- Has an intervention and/or a program model been planned and implemented as intended?
- What worked (well) and what are the lessons learned?
- With what quality are the research activities planned and completed?
- What is the scope for a broader impact of the intervention/program model?
- Evaluation Approach/Design– Among other options, approaches include formative and summative evaluations.
- Formative evaluation, usually completed in the first 2 years of the grant, typically informs the attributes of the program such as adherence, delivery, and quality as proposed and provides just-in-time feedback to help improve the program in its formative stages of planning and implementation.
- Summative evaluation, usually completed in the latter years of the grant, focuses on the type and number of outputs or products as well as if the intended outcomes were achieved and how well. It also assesses the model of the program—its merit and broader impacts insofar as its viability to sustain beyond its funding cycle and/or scale up.
- Data Collection & Analytical Techniques- Data collection sources as well as analytical techniques are similar to the research plan and will depend on evaluation questions, feasibility of undertaking the study, budget, and time among other factors.As I envision, NSF grant proposal work is a triangle with including the PI/Co-PI (grant writer) at the top vertex with a researcher and an evaluator at the other two vertices working in close concert to plan and develop a winning proposal.Always plan to engage a researcher and an evaluator early and often!
