...

Evaluating an Advisory Dean Program: A Program Evaluation Strategy

by user

on
Category: Documents
3

views

Report

Comments

Transcript

Evaluating an Advisory Dean Program: A Program Evaluation Strategy
Evaluating an Advisory Dean Program: A Program Evaluation Strategy
Aubrie Swan Sein, PhD, EdM, Lisa Mellman, MD, Boyd Richards, PhD,
Columbia University College of Physicians and Surgeons; Center for Education Research and Evaluation
BACKGROUND
Evaluative Inquiry
Learning Processes
Evaluative Inquiry, developed by
• Not just talk
• Surfacing multiple points of view
• Making the ‘un-discussable’
discussable
• Developing a shared meaning and
shared understanding
• Fostering a sense of community and
connection
Preskill and Torres (1999) engages the
discussion of, reflection on, and
questioning of participant knowledge,
beliefs, and values about a program.1
Dialogue
Introduction/Aim(s): Columbia
University College of Physicians and
Surgeons, utilized evaluative inquiry to
evaluate its Advisory Dean program,
started in 2003. In this program, six
Deans provide academic and career
advising to assigned communities of
students in group and one-on-one
sessions.2
Use of Evaluative Inquiry for
Program Evaluation: Steps in
evaluative inquiry implemented via a
series of meetings with program
directors, advisory deans, and
evaluators over a six-month period.
Participants reviewed and reflected
on the results of advisory dean
interviews, years of student surveys,
meeting agendas, and personal
experiences with the program.
Reflection
• “Unpacking” ideas and
understandings
• Linking what we know (cognitive)
with what we feel (affective,
experiential)
• Checking alignment between
actions and beliefs
• Considering the implications of
actions
1. Focusing the
Inquiry
• Identifying what is meaningful; do-able
• Articulating issues of concern
• Clarifying intended use and users
• Crafting evaluation questions
• Collecting relevant, credible information
2. Carrying out the
•
Engaging
with
data
and,
as
a
group,
Inquiry
assigning meaning
• Uncovering and identifying
central issues
• Developing curiosity
• Stimulating learning
• When we fail to ask questions, we
miss out on deeper levels of
learning
Asking
questions
Clarifying
values,
beliefs, and
assumptions
Results/ Discussion:
• Evaluative inquiry was a useful
strategy to evaluate and
understand the functioning of the
Advisory Dean program in order to
sustain the program over time.
• Participants readily engaged in the
steps of the process (see central
panel).
• It was important to involve the
voices of all stakeholders.
• The quality of reflections was
dependent on the quality of the
data available.
Three Phases of Evaluative Inquiry
• Understanding our own
projections of ‘reality’
• Surfacing motivations, opinions,
and attitudes
• Considering other points of
view
3. Applying
learning
• Communicating findings
• Developing and implementing action plan
• Monitor progress
Systems and
structures
Infrastructure Needed for
Evaluative Inquiry
Communication
Leadership
Culture
• Engenders trust
• Supports risk-taking
• Values learning and
improvement
• Supports and
models the
processes of
evaluative inquiry
• Uses information as
a means to share
learning
• Works to eliminate
barriers to
communication
• Supports
collaboration
• Rewards democratic
processes
Preskill & Torres, 1999
OUR EXPERIENCE
Focusing the Inquiry
1) Regular meetings with program
leader and evaluators to identify
goals
2) Review literature
3) Identify conceptual framework as lens
to frame inquiry and interpret results:
Bolman &Deal model of organization
functioning--structural, human
resource, political, and symbolic3
4) Decision to focus inquiry on keys for
successful maintenance
Carrying out the Inquiry
1) Review evaluation data: (AD
interviews, student surveys, minutes
from meetings)
2) Iterative discussions, framed by
Bolman & Deal) to determine how AD
program has evolved and been
sustained
Applying Learning
1) Clarify program strengths and
limitations
2) Communicate findings of inquiry to
Advisory Deans, medical education
administration, etc
3) Implement changes to better sustain
program, such as increased AD
professional development and
sharing at AD team meetings
4) Detailed description of process,
findings in Academic Medicine
article4
References:
1. Preskill H, Torres RT. Building capacity
for organizational learning through
evaluative inquiry. Evaluation. 1999;
5:42–60.
2. Macaulay W, Mellman LA, Quest DO,
Nichols GL, Haddad J, Puchner PJ. The
advisory dean program: A personalized
approach to academic and career
advising for medical students. Acad
Med. 2007; 82:718–722.
3. Bolman LG, Deal TE. Reframing
Organizations: Artistry, Choice, and
Leadership. San Francisco, Calif: JosseyBass; 1991.
4. Swan-Sein, A, Mellman, L, Balmer, D,
Richards, B. Sustaining an advisory dean
program through continuous improvement
and evaluation. Acad Med.
2012;87:523-528.
Fly UP