Effective Practices
Respond to an invitation to serve as an external reviewer
- Learn about the institution’s and expectations, e.g., proposed time frame for designing and completing the self-study report; specific issues and topics of interest to the department or administration; timetable, structure, and expectations of the site visit and reviewers’ report.
- Learn about how the institution selects and discuss its plans for having other external reviewers on the team, including the diversity of reviewers’ backgrounds, expertise (e.g., department chair experience), and how it would like the team to function.
- Ask how the information from the review will be used, e.g., to develop an action plan for improvement and/or as part of a cyclic review process.
- Review a list of department faculty and staff and inform the institution of any relationships you or others might see as conflicts of interest, e.g., mentor–mentee, collaborators, close friends, instructor–former student.
- Inform the institution if you or a close family member has had or will have a personal involvement or financial interest in the department or institution, e.g., your partner is interviewing for a position at the institution; a relative works for a company selling lab or demonstration equipment for physics programs; you or an immediate family member has a financial interest in or is an author of curricular resources the department is using or might use.
- Consider if you as an individual (in the case of a single or as a member of a team have the expertise and perspectives necessary, e.g., familiarity with departments and institutions of this type, including size, populations served, and mission.
- Consider whether the department’s planned external review team will have the needed size, diversity, and expertise (e.g., department chair experience), and advise the department to reconsider if it does not.
- Gain experience as a first-time reviewer by requesting to serve with others who have been reviewers in the recent past.
- Decide whether you will have the time required for reviewing the self-study report, participating in the site visit, collaborating with other team members, and completing the report by the deadline. See the resource on Program Review Timeline for more details.
- Assess whether you are comfortable serving as a reviewer, based on what you know about the department and institution, their , their expectations, real or perceived conflicts of interest, diversity of the review team, proposed compensation, and your knowledge of the relevant physics resources.
- Consider stepping aside if the team already has many people like you and/or if the department is not adequately considering the diversity of the team.
- Review your institution’s policies governing your service as an (with respect to e.g., time away from campus and accepting an honorarium) and discuss any restrictions with the institution of the department to be reviewed.
- Recommend other reviewers if you are unable to serve.
- Finalize the expectations and commitments with a Memorandum of Understanding or contract. See the resource on Sample MOU for Engaging Each External Reviewer for Physics Program Review for more details.
- Notify your supervisor of your off-campus commitment as an .
Prepare for the visit
- Schedule a videoconference with the entire team of reviewers well in advance of the site visit.
- Discuss the expertise of team members (e.g., department chair or administrative experience, prior experience serving as a reviewer, experience leading reviews of one’s own department, and expertise in the particular areas where the department is looking to improve), the diversity of the team (e.g., whether it includes members of and people with a variety of research specialties and institution types), and whether any changes or additions to the team should be recommended.
- Discuss the department’s plan or institutional template for the self study; other information team members would like included in the self study; representatives of key offices, other departments, and other individuals or groups the team would like to meet with during the site visit; and the timeline for the visit.
- Identify topics (e.g., recruiting, persistence and graduation rates, departmental climate, equity, facilities, lab space, start-up funding, undergraduate research), questions (about, e.g., students and their progress, the departmental curriculum and resources, alignment of curriculum with student career paths, departmental climate, faculty contributions, balance of tenure-track and term faculty), and possible concerns the reviewers would like the department to address in its internal review process and self-study report to supplement guidance given by institutional templates.
- Identify a team member who will serve as the primary contact with the department, if the institution has not assigned one.
- Learn about topics, questions, and concerns the institution and department would like to address and request information relevant to these be included in the self study, e.g., data on retention, lab budgets, surveys of student satisfaction, student job placement, faculty productivity and currency in the field, and departmental practices around equity, diversity, and inclusion.
- Suggest additional topics, questions, or concerns that could help the department and/or institution reflect on their program.
- Encourage a development process for the self study that includes a broad range of stakeholders, e.g., faculty, staff, current students, graduates, and other relevant stakeholders.
- Discuss the timeline for the entire process and request that the self-study report and other briefing materials (e.g., prior report) be sent electronically at least two weeks in advance of the visit. See the resource on Program Review Timeline for more details.
- Discuss any concerns regarding the expertise or diversity of the team and recommendations for how they could be addressed.
- Advise department members about resources available to help them conduct the . See the section on How to Undertake an Undergraduate Program Review for more details.
- Discuss how peer and aspirant departments are identified, relevant comparative information (e.g., budgets, facilities, courses offered, number of student credit hours, majors, faculty, staff, and graduates), and ways to obtain these data, e.g., other physics department chairs, your office of institutional research, and outside sources such as the AIP Statistical Research Center or the web page How Does Your Institution Compare?.
- Discuss the department’s plan for identifying and gathering data about peer and aspirant departments and suggest other departments it might consider including.
- Suggest the department develop a draft plan, to be included with the self-study report, that proposes actions and strategies to help the department sustain strengths and address challenges.
- See the resource on Sample Agenda for an External Reviewers’ Site Visit for more details.
- Ask for time at the start of the visit for team members to confer privately about insights and questions generated by reviewing the briefing materials (e.g., self-study report, site visit agenda, background material on the institution) and to discuss visit plans (e.g., topics to explore; questions to ask different groups; ways to elicit different perceptions of institutional and departmental strengths and challenges; division of labor including who leads sessions, takes notes, and writes each section of the report; and timing for preparing and delivering the oral and written reports). For potential topics the review team may also want to explore, see the resource on Topics a Department Might Explore in a Program Review.
- Request an opening conversation with the departmental contact person to gain perspectives on issues the department and institution face (e.g., budget; staffing; enrollments; concerns brought by faculty, staff, alumni, and administrators; equity, diversity, and inclusion) and to gain answers to reviewers’ questions.
- Ask for a conversation with an academic administrator (e.g., the dean or provost) early in the visit to discuss their perspective, perceptions, questions, and expectations of the review.
- Include as many meetings with individuals or groups (e.g., separate meetings with students, pre-tenure faculty, tenured faculty, non-tenure track faculty, adjunct faculty, library staff, members of , and representatives of departments requiring physics courses) as the schedule permits, to gain a broad view of the department and its institutional context.
- Build time for drafting and editing key elements of the report into the agenda or, if this is not possible, ensure that each reviewer reserves time directly following the visit to allow a timely return of the final report.
- Ensure there is a final exit meeting with the department followed by a separate meeting with the administration to report out key findings and recommendations.
- Study the materials before the visit, keeping in mind the departmental and institutional context, to allow for a potential intervention (e.g., asking for additional data or modifying the schedule) before the process begins, if needed.
- Prepare strategies for how reviewers might address specific questions the department or administration poses to the review team.
- Annotate materials with questions or requests for clarification to discuss with other members of the review team during the meeting prior to on-campus interviews and discussions.
- Develop and coordinate questions in advance based on the self-study report. Develop both general questions for all groups, and specific questions for different groups, e.g., administrators, faculty, staff, students, and members of other departments. Questions for faculty might include: What is your department’s relationship to the dean? What do you see as the most important issue facing the department? What is your biggest concern? What do you think your students would say is their biggest concern? What has your biggest success been in the past few years? Questions for students might include: What are the best aspects of learning physics in this department? What are the greatest challenges to your learning? How does the curriculum align with your career plans? What has been your experience with advising?
- Avoid deciding on recommendations before the visit.
Conduct the visit
- Enter the process with an open mind, seeking to understand institutional and departmental contexts (e.g., institution type, mission, resources, departmental goals, externally imposed demands) and determine actions based on what you learn, rather than bringing a predetermined idea for what the department should do based on your experiences.
- Listen more than you talk. Listen, e.g., for consistency of information from different individuals and groups, things that might have been overlooked as explanations of problems or options for addressing difficulties, shared (or not) support for the proposed initiatives, broad (or not) involvement of members of the department in developing the self-study report.
- Elicit information using pre-developed questions that invite participants to share examples, explanations, aspirations, and frustrations. Avoid leading questions that assume answers.
- Explain how things will be held in confidence.
- Maintain confidentiality, holding in confidence findings, recommendations, and any non-public information. In other words, do not share these with individuals outside the institution.
- Avoid revealing identities in reports of comments that may negatively affect individuals (by, e.g., embarrassing them, damaging their relationships with others in the department, affecting their performance reviews), particularly taking into account power dynamics within the department.
- Establish whether discussions (or parts of discussions) are on or off the record before beginning each conversation.
- When administrators discuss issues or concerns, ask whether the department is aware of these issues or concerns. If not, ask how they might be shared or addressed, to avoid surprises if you address them in the reviewers’ report.
- Begin meetings with students, faculty, and staff with introductions, an explanation of the purpose of the reviewers’ visit, and a discussion of the potential impact of the review on the department’s planning for the future.
- Use pre-determined, open-ended questions, following up as necessary based on responses.
- Ensure that all voices are heard during conversations by eliciting responses from quiet individuals and politely redirecting away from those who may dominate.
- Elicit the broader picture by redirecting conversations to allow a wider range of ideas to be discussed.
- Monitor the dynamics in the room to determine whether private conversations might be needed to draw out difficult issues.
- Look for discrepancies in comments from various individuals or groups that provide clues to know when to dig deeper in specific areas, e.g., when faculty and students disagree about the environment for student success or about reasons for low enrollment in certain courses.
- Serve in a collaborative role by making recommendations or suggesting potential solutions (including suggestions adopted elsewhere) to issues observed throughout the review.
- Explore issues and questions raised by the department in the self-study report and by the upper-level administrators, listening for their suggestions on how to address them.
- Probe how the department addresses issues that impact students, including differences in student backgrounds and aspirations, differences in student entry points into the major course sequence, career preparation, equity and inclusion, availability of critical courses, advising, etc.
- Ask how department members view their roles, understand implicit and explicit expectations, make contributions, mentor other faculty and build working relationships.
- Identify and explore any observed inconsistencies in answers from different groups or individuals to illuminate critical issues.
- Inquire about how department members contribute to creating an inclusive environment, through e.g., recruiting initiatives, educating the students they have, mentoring, collegiality, cooperative versus competitive interactions, and creating a supportive instructional and welcoming departmental environment.
- Explore instructional, structural, and resource issues (e.g., personnel, budgets, facilities, curriculum, pedagogy) and social and climate issues, e.g., inclusivity, student spaces, mentoring, collegiality, cooperative versus competitive interactions, ethics.
- See if significant differences exist among department members on critical issues and, if so, consider how they can be addressed in your recommendations, e.g., consider recommending facilitated discussions that might bring factions together to reach a common understanding and direction.
- Confirm writing assignments (e.g., introduction, findings, recommendations, particular topics within these sections) for team members.
- Confer at regular intervals throughout the visit, noting observations, possible findings, and suggestions that might go into the reports, to look for consensus and to sharpen and prioritize recommendations.
- Discuss findings resulting from separate meetings promptly, particularly if any significant issues arise.
- Propose revisions to the agenda if issues arise that need to be explored with another group.
- Develop collective advice through team consultations so that reports and final discussions focus on shared ideas and suggestions.
- Determine who will present the report, e.g., one member of the team or multiple people.
- Develop and frame the report as consisting of consensus recommendations from the team.
- Prepare succinct statements for which elaborations can be provided in the written report and in the meeting, if requested.
- Begin with findings of departmental strengths, before identifying opportunities for improvement.
- Use careful introductions to set the stage for written recommendations that might otherwise be surprising.
- Provide clear recommendations, highlighting those that are critical for departmental advancement.
- If there are separate meetings for oral reports to department members and upper-level academic administrators, convey the same elements to both, in order to enhance subsequent working relationships between these two groups and to allow reviewers to continue to be seen as ongoing resources for implementing positive changes.
- Request suggestions for elaborations on any topic in the oral report, or on topics not addressed in the oral report, to be included in the written report.
Develop the written report
- Reaffirm assignments for writing subsections (if not completed on site) and share drafts promptly (within one week) to avoid losing connections to specific observations.
- Limit the length of the report (e.g., 5–6 pages for smaller departments and perhaps 8–12 pages for departments with more complex components or entities) to make it more accessible and likely to be read by a broad constituency within the department.
- Develop and frame your feedback as a consensus report from the review team.
- Include observations about the self-study report and from the site visit.
- Acknowledge departmental strengths and provide encouragement, where appropriate, to sustain them.
- Highlight accomplishments using examples from a diverse array of department stakeholders.
- Use comparison information (from, e.g., the AIP Statistical Research Center or the web page How Does Your Institution Compare?, peer institutions, and personal experience) when raising concerns, to provide context within the discipline and comparisons with peer or aspirant departments and institutions.
- Strengthen observations by paraphrasing (to protect anonymity) quotes from individuals.
- Validate observations whenever possible using multiple sources (preferably from distinct groups or with data obtained from the self-study report), or indicate the limitations of your sources when this is not possible, being careful to protect anonymity.
- Identify gaps, pitfalls, and/or inconsistencies in the department’s goals, objectives, and actions.
- Identify opportunities for the department to address concerns or capitalize on synergies within the department or institution.
- Prioritize recommendations. Identify what more could be accomplished with existing resources, identify what could be accomplished with additional resources, and indicate necessary steps to achieve these outcomes.
- Recognize that recommendations for additional resources must be backed by a clear rationale and potential for return on investment.
- Avoid making recommendations that are unrealistic within the department or institution’s particular context (e.g., recommending multiple new tenure lines), and suggest less resource-intensive alternatives, along with a discussion of the corresponding limitations.
- Suggest potential solutions within recommendations that address concerns raised in the report, to help the department see a path toward resolving issues.
- Address instructional, structural, and resource issues (e.g., personnel, budgets, facilities, curriculum, pedagogy) and social and climate issues, e.g., inclusivity, student spaces, mentoring, collegiality, cooperative versus competitive interactions, ethics.
- Consider encouraging the department to update or develop an action or strategic plan. See the section on How to Create and Use a Strategic Plan for more details.
- Elaborate on points made in, or discussions that accompanied, the oral report. (See 3.F.)
- Work to achieve a common style among coauthors.
- Endeavor to prepare a report that can be shared in its entirety with members of the department and administrators. Only in rare cases (e.g., evidence of serious misconduct or mention of interpersonal difficulties in the department) should you prepare a supplement of confidential advice to the department chair and/or upper-level administrators, keeping in mind that such reports do not always stay private, and that difficult issues are often more appropriately discussed in confidential verbal conversations.
- Indicate the limitations of your analysis, such as constraints of time and availability of information and individuals.
- Encourage a cyclic review process that includes the gathering and reviewing of data on an ongoing basis, annual review of initiatives and outcome measures, and periodic comprehensive reviews.
- Confirm the protocol for submitting the report (e.g., how and to whom to submit the report). If possible, provide a draft written report to the department and invite prompt corrections of factual errors.
- Submit the report within the time frame discussed initially, preferably within three weeks of the visit.
- Keep report information confidential. Do not quote or cite the report at any later time; the information and recommendations belong to the institution.