ANN DOWSETT JOHNSTON and MARY DWYER explain the methodology behind the annual Maclean’s survey
ANN DOWSETT JOHNSTON,MARY DWYERNovember142005
RANKING CANADIAN UNIVERSITIES
UNIVERSITY RANKINGS | Methodology
ANN DOWSETT JOHNSTON and MARY DWYER explain the methodology behind the annual Maclean’s survey
ANN DOWSETT JOHNSTON
The Maclean’s rankings offer students and parents a rare view into the ivory tower—and a unique opportunity to make informed decisions on the comparative strengths of public universities across Canada. On behalf of those readers, we ask a number of key questions related to the undergraduate experience. Where are the brightest firstyear students? Where will you find the smallest classes? Where are you most likely to have access to tenured faculty? Where will you find the richest library resources?
Which university has made the largest commitment to student services or scholarships and bursaries? Which university has the best reputation for quality and innovation? Maclean’s asks all this, and much more.
HOW THE RANKING IS DONE
In April, a detailed questionnaire was circulated to the universities. Throughout the spring and summer, we collected information on student and faculty awards from the administering agencies and sent out thousands of reputational surveys across the country. Throughout the fall, we spent several weeks collating and checking the data. Consulting statisticians from McDougall Scientific Ltd. performed the final calculations.
HOW WE PLACE THE UNIVERSITIES IN PEER GROUPS
Using such factors as research funding, diversity of offerings and the range of Ph.D. programs to define peer groupings, Maclean’s places each university in one of three categories. Primarily Undergraduate universities are largely focused on undergraduate education, with relatively few graduate programs. Those in the Comprehensive category have a significant amount of research activity and a wide range of programs at the undergraduate and graduate levels, including professional degrees. Medical-Doctoral universities offer a broad range of Ph.D. programs and research, as well as medical schools.
In reporting to Maclean’s, universities include all federated and affiliated institutions. The magazine does not rank schools with fewer than 1,000 full-time students or those with a strictly religious or specialized mission.
HOW WE EVALUATE
The universities in the three categories are treated as separate but equal. Maclean’s ranks the schools on a range of factors in six broad groupings (weightings are in parentheses). In total, Primarily Undergraduate universities are ranked on 22 performance measures, Comprehensive universities on 23 and Medical-Doctoral universities on 24— resulting in slightly different weightings for some performance measures.
(22% to 23% of final score)
Students are enriched by the calibre of their peers. For that reason, Maclean’s collects the incoming students’ average high-school grades (11%), and the proportion of those with averages of 75 per cent or higher (2%). This count includes only those students
whose secondary-school averages or CEGEP scores served as the basis of admission. With the exception of CÉGEP scores, no conversion formula is applied to incoming grade averages to adjust for provincial differences or varying admission'policies.
As a measure of drawing power, we count the proportion of out-of-province students in the first-year undergraduate class (1.5%). The percentage of international students in the first-year undergraduate class (0.5%) is measured as well, acknowledging the benefits such diversity brings to the classroom. The percentage of international students at the graduate level (1%) is also measured for Comprehensive and Medical-Doctoral universities.
In taking stock of retention rates (2%), Maclean’s asks for the percentage of firstyear students who return in second year, either full-time or part-time. While many factors can affect a student’s choice not to returnpersonal considerations, or a decision to transfer to a professional program unavailable
Cover I UNIVERSITY RANKINGS | >
at their home university—student retention, on the whole, reflects a university’s success in keeping its students on course. Maclean’s also measures graduation rates (2%): the percentage of full-time undergraduates in their second year who go on to graduate from the institution within one year of the expected time period.
In addition, Maclean’s collects data on the success of the student body at winning national academic awards (3%) over a fiveyear period.
CLASSES (17% to 18%)
The rankings take stock of the entire distribution of class sizes at the firstand secondyear levels (7.5% for Primarily Undergraduate universities, 7% for the other two categories), as well as the thirdand fourth-year levels (7.5% for the Primarily Undergraduate category, 7% for the others). Maclean’s measures the percentage of students in classes in each of the following class-size ranges: 1 to 25; 26 to 50; 51 to 100; 101 to 250; 251 to 500; 501 and higher.
Macleans also ranks universities on the percentage of first-year classes taught by tenured and tenure-track professors (3%).
In assessing the calibre of faculty, Maclean’s calculates the percentage of faculty with Ph.D.s (3%), and the number who win national awards (3%). In addition, the magazine measures the success of eligible faculty in securing grants from each of the three major federal granting agencies: Social Sciences and Humanities Research Council of Canada, Natural Sciences and Engineering Research Council of Canada, and the Canadian Institutes of Health Research, as well as the Canada Council. Maclean’s takes into account both the number and the dollar value received last year. Social sciences and humanities grants plus
Canada Council grants (5.5%) and medical/science grants (5.5%) were tallied as separate indicators.
This section examines the amount of money available for current expenses per weighted full-time-equivalent student (3.3%), as well as the percentage of the budget spent on student services (4.3%) and scholarships and bursaries (4.3%). When presenting their general operating budget, institutions deducted any funds used to pay off debt.
This section assesses the breadth and currency of the collection. Universities received points for the number of volumes and volume equivalents per number of full-time-equivalent students (4% for Primarily Undergraduate and Comprehensive, 3% for Medical-Doctoral). The total holdings measurement was used in the Medical-Doctoral category (1%), acknowledging the importance of extensive on-campus collections in those universities.
As well, Maclean’s measured the percentage of a university’s operating budget that was allocated to library services (4%) and the percentage of the library budget spent on updating the collection (4%). In acknowledging a shift from the traditional library model to an access model, Maclean’s captures spending on electronic resources in both the library expenses and acquisitions measurements.
This section reflects a university’s reputation with its own graduates, as well as within the community at large. For the reputational survey (16%), respondents rated the universities in three categories: Highest Quality, Most Innovative and Leaders of Tomorrow. Best Overall represents the sum of the scores.
REPUTATIONAL RESPONSE RATE
This year, Maclean’s solicited the opinion of 11,620 individuals across the country. They included university officials at each ranked institution, high-school principals and guidance counsellors from every province and territory, the heads of a wide variety of national and regional organizations, plus CEOs and recruiters at corporations large and small. The reputational survey is both regional and national, dividing the country into four key areas: the western provinces, Ontario, Quebec and the Atlantic provinces. All respondents completed a national survey; university officials, principals and guidance counsellors also completed regional ones.
WEIGHTINGS OF INDICATORS
Maclean’s ranks universities on 22,
23 or 24 performance measures, according to peer grouping, and then allocates the appropriate weights to
STUDENT BODY 22% to 23%
Average Entering Grade_11%
Proportion with 75% or Higher_2%
Proportion Who Graduate_2%
Out of Province (1st Year)_1.5%
International (1st Year)_0.5%
Student Awards 3%
Class Sizes: 1st and 2nd Year Level 7% to 7.5% Class Sizes: 3rd and 4th Year Level 7% to 7.5% Classes Taught by Tenured Faculty 3%
Faculty with Ph.D.s_3%
Awards per Full-time Faculty_3%
Social Sciences and Humanities Grants 5.5% Medical/Science Grants 5.5%
Scholarships & Bursaries_4.3%
Student Services 4.3%
Total Library Holdingsf_1%
Holdings per Student_3% to 4%
Reputational Survey 16%
•Comprehensive and Medical-Doctoral categories only fMedical-Doctoral category only
When looking at alumni support, institutions received points for the number— rather than the value—of gifts to the university over the past five years (3%). U]
The story you want is part of the Maclean’s Archives. To access it, log in here or sign up for your free 30-day trial.
Experience anything and everything Maclean's has ever published — over 3,500 issues and 150,000 articles, images and advertisements — since 1905. Browse on your own, or explore our curated collections and timely recommendations.WATCH THIS VIDEO for highlights of everything the Maclean's Archives has to offer.