1997-98 Pilot Semester Evaluation Report

Ithaca Evaluation Group

August 27, 1998

Evaluation activities for the MentorNet project in 1998 have focused on creating tools for evaluation in 1998-99, which will be the first full year of mentoring partnerships. Primary data collection centered on the development and administration of an online exit questionnaire and the development of a preliminary coding scheme for content analysis of end-of-project feedback and email communications.

In May and June, all mentors and protégés matched prior to mid-March (n=184 pairs) were emailed invitations to complete a short online questionnaire concerning their experiences in the initial phase of mentoring. The questionnaires asked individuals to provide their names, but responses were transmitted to the evaluators, so some measure of confidentiality was provided. A followup reminder was emailed to nonrespondents approximately two weeks later. 60% of the mentors and 43% of the protégés completed the instrument, almost all using a web browser (a paper questionnaire was a rarely used alternative). Nonresponse was more common for pairs that were matched relatively late in the year (e.g., 55% for participants matched on 3/19 and 46% for participants matched after that date). Although we expected that response rates would be lower for schools that ended their school years early, this was not the case; schools with Commencement prior to May 23 had response rates indistinguishable from schools commencing after that date.

The questionnaire was intentionally brief, and took an estimated 5-10 minutes to complete. Participants were asked to estimate the number of mentoring messages per month they sent and received, and were queried concerning their comfort asking and responding to questions, interest in participating in MentorNet again, interest in recommending the program to others, assessment of three specific potential problem areas (delays in sending and receiving messages, difficulty discussing particular topics) and ratings of three measures of project effectiveness (mentor’s interest in the protégé, participants’ interest in meeting each other, and assessment of the overall quality of the mentor-protégé match. In addition, protégés were queried about their interest in continuing in their current major and their interest in working in industry following graduation. With the exception of the estimated number of messages sent and received, each of these quantitative measures used a five-point Likert scale ranging from strongly positive to strongly negative. The direction of the items was varied, and the summaries that follow first transformed the scales to equate "5" with "strongly favorable."

Mean participants’ assessments for all of these scales were favorable, ranging from "My partner was interested in me" (3.70 on a 5-point scale) to "Comfort asking questions" (4.46 on a 5-point scale). The only two quantitative assessments that approached neutrality (3.0) were items that asked how interested participants were in meeting their partners face to face (3.08), and a holistic assessment of the overall quality of the match (3.46). Closer analysis of the latter will require information on the quantitative closeness of matches to participants’ requests (data that will be collected during the 1998-99 matching process), as well as analysis of relationships between participant demographics and response patterns, which is ongoing. Although a larger pool of mentors and improvements in the matching algorithm will likely increase participant satisfaction with matches, three other actions that may also prove useful are the longer period of planned mentoring in 1998-99 (which will permit better identification of common areas of interest), a larger mentor pool, and - a somewhat speculative suggestion - streamlining the application process (we suspect that the extensive time required to complete the application and rate the priority of various matching criteria may have unrealistically raised participants’ expectations).

The online questionnaire also included three essay questions, which asked respondents to write briefly on topics that they discussed with their partner, insights they gleaned from the mentoring experience, and suggestions for improvement of the MentorNet project. A process of qualitative in vivo coding was used to generate a content analysis coding scheme for the data, and the 118 distinct codes were applied to analysis of all responses. Baselines have been communicated to MentorNet staff, and further analysis of the extant data will occur this fall; however, the primary use of these codes will be in the analysis of actual message traffic during 1998-99. Examples of commonly cited codes include "Balancing career and family" (reported topic of conversation), "Ementoring provided encouragement and support" (insight gleaned from the process), and "Extend the mentoring period to a full school year or longer" (suggestion for improvement).

At the recommendation of the project advisory board and staff, collection of primary data on email frequency and content was postponed until the second year of the project. This will permit Human Subjects review of the email-monitoring plan at participating universities.

Overall, the project appears to have had a very successful first year and the staff have identified a number of concrete steps for continuing improvements in recruitment, matching, and facilitation and monitoring of ementoring.