Evaluating an Advisory Dean Program: A Program Evaluation Strategy
Evaluating an Advisory Dean Program: A Program Evaluation Strategy Aubrie Swan Sein, PhD, EdM, Lisa Mellman, MD, Boyd Richards, PhD, Columbia University College of Physicians and Surgeons; Center for Education Research and Evaluation BACKGROUND Evaluative Inquiry Learning Processes Evaluative Inquiry, developed by • Not just talk • Surfacing multiple points of view • Making the ‘un-discussable’ discussable • Developing a shared meaning and shared understanding • Fostering a sense of community and connection Preskill and Torres (1999) engages the discussion of, reflection on, and questioning of participant knowledge, beliefs, and values about a program.1 Dialogue Introduction/Aim(s): Columbia University College of Physicians and Surgeons, utilized evaluative inquiry to evaluate its Advisory Dean program, started in 2003. In this program, six Deans provide academic and career advising to assigned communities of students in group and one-on-one sessions.2 Use of Evaluative Inquiry for Program Evaluation: Steps in evaluative inquiry implemented via a series of meetings with program directors, advisory deans, and evaluators over a six-month period. Participants reviewed and reflected on the results of advisory dean interviews, years of student surveys, meeting agendas, and personal experiences with the program. Reflection • “Unpacking” ideas and understandings • Linking what we know (cognitive) with what we feel (affective, experiential) • Checking alignment between actions and beliefs • Considering the implications of actions 1. Focusing the Inquiry • Identifying what is meaningful; do-able • Articulating issues of concern • Clarifying intended use and users • Crafting evaluation questions • Collecting relevant, credible information 2. Carrying out the • Engaging with data and, as a group, Inquiry assigning meaning • Uncovering and identifying central issues • Developing curiosity • Stimulating learning • When we fail to ask questions, we miss out on deeper levels of learning Asking questions Clarifying values, beliefs, and assumptions Results/ Discussion: • Evaluative inquiry was a useful strategy to evaluate and understand the functioning of the Advisory Dean program in order to sustain the program over time. • Participants readily engaged in the steps of the process (see central panel). • It was important to involve the voices of all stakeholders. • The quality of reflections was dependent on the quality of the data available. Three Phases of Evaluative Inquiry • Understanding our own projections of ‘reality’ • Surfacing motivations, opinions, and attitudes • Considering other points of view 3. Applying learning • Communicating findings • Developing and implementing action plan • Monitor progress Systems and structures Infrastructure Needed for Evaluative Inquiry Communication Leadership Culture • Engenders trust • Supports risk-taking • Values learning and improvement • Supports and models the processes of evaluative inquiry • Uses information as a means to share learning • Works to eliminate barriers to communication • Supports collaboration • Rewards democratic processes Preskill & Torres, 1999 OUR EXPERIENCE Focusing the Inquiry 1) Regular meetings with program leader and evaluators to identify goals 2) Review literature 3) Identify conceptual framework as lens to frame inquiry and interpret results: Bolman &Deal model of organization functioning--structural, human resource, political, and symbolic3 4) Decision to focus inquiry on keys for successful maintenance Carrying out the Inquiry 1) Review evaluation data: (AD interviews, student surveys, minutes from meetings) 2) Iterative discussions, framed by Bolman & Deal) to determine how AD program has evolved and been sustained Applying Learning 1) Clarify program strengths and limitations 2) Communicate findings of inquiry to Advisory Deans, medical education administration, etc 3) Implement changes to better sustain program, such as increased AD professional development and sharing at AD team meetings 4) Detailed description of process, findings in Academic Medicine article4 References: 1. Preskill H, Torres RT. Building capacity for organizational learning through evaluative inquiry. Evaluation. 1999; 5:42–60. 2. Macaulay W, Mellman LA, Quest DO, Nichols GL, Haddad J, Puchner PJ. The advisory dean program: A personalized approach to academic and career advising for medical students. Acad Med. 2007; 82:718–722. 3. Bolman LG, Deal TE. Reframing Organizations: Artistry, Choice, and Leadership. San Francisco, Calif: JosseyBass; 1991. 4. Swan-Sein, A, Mellman, L, Balmer, D, Richards, B. Sustaining an advisory dean program through continuous improvement and evaluation. Acad Med. 2012;87:523-528.