SPECIAL REPORT

Reading The Rankings

A road map to the Maclean's methodology

VICTOR DWYER DIANE BRADY November 15 1993
SPECIAL REPORT

Reading The Rankings

A road map to the Maclean's methodology

VICTOR DWYER DIANE BRADY November 15 1993

Reading The Rankings

A road map to the Maclean's methodology

Two years ago, Maclean's published its first annual ranking of Canadian universities-and touched off a raging controversy. Focusing on the quality of arts and science faculties at 46 schools, the first ranking proved popular with readers-and contentious within the walls of the ivory tower. Last year, after extensive consultation with educational ex perts and university officials across the country, Maclean's further defined and examined each of its indicators of excellence in an expanded questionnaire that took into account the entire range of university faculties. As well, the universities were placed in one of three distinct peer categories. The 1993 ranking represents a consoli dation of that effort, as well as a striking recognition of the principle of accountability on the part of Canada's university community. As in past years, Maclean's editors maintained regular contact

with the university community and even expanded the scope of the survey. After submitting only a collective-and largely unanswered-questionnaire in 1992, the five largest campuses of the Université du Québec chose to participate this year. As well, the 1993 ranking features Canada's two newest universities, both in Ontario: North Bay's Nipissing University, until last November an affiliated campus of Laurentian University, and Toronto's Ryerson Polytechnic University, which offers a rare blend of academic re search and technical training. Carleton University in Ottawa and Memorial University in St. John's, Nfid., on the oth er hand, declined to take part-for differ ent reasons (page 68). This year, Maclean's worked to consoli date the ranking process, making only those changes needed to fine-tune the

NATIONAL REPUTATIONAL RANKING RANK HIGHEST QUALITY MOST INNOVATIVE LEADERS OF TOMORROW BEST OVERALL 1 McGILL WATERLOO WATERLOO WATERLOO 2 QUEEN'S McMASTER UBC UBC 3 UBC GUELPH CALGARY McMASTER 4 TORONTO QUEEN'S McMASTER McGILL 5 WATERLOO McGILL SIMON FRASER QUEEN'S 6 McMASTER UBC GUELPH CALGARY 7 ALBERTA SIMON FRASER McGILL GUELPH $ MONTREAL CALGARY VICTORIA TORONTO 9 GUELPH TORONTO QUEEN S SIMON FRASER 10 ACADIA SHERBROOKE YORK ALBERTA

questionnaire. Universities knew exactly what to expect and how to respond. In July, a 14-page questionnaire-containing 63 questionsand an accompanying 19-page User's Guide were sent to 53 universi ty presidents. (Institutions with a specialized mission or course selection were excluded, as were strictly religious institutions and those with fewer than 1,000 full-time students). As well, Maclean's sent each school a parallel electronic diskette, to facilitate data entry and provide a cross-check on all numbers. The schools were given six weeks to complete the survey, during which time Maclean's edi tors-along with several researchers assigned to the project-an swered queries. With its current institution-wide ranking, Maclean's judges uni versities only against those with a similar structure or mandate. Using such factors as research funding, program breadth and the number of PhDs granted as a gauge, universities are grouped in one of the following three categories:

MEDICAL/DOCTORAL: These are universities with a major commitment to PhD programs and research. All have medical schools, which set them apart due to the size of research grants. (Memorial University is the one exception: despite the presence of a medical school, its program mix is more comparable with the Comprehensive universities.)

COMPREHENSIVE: These institutions offer a significant amount of research activity and a wide range of programs—including professional degrees—at the graduate and undergraduate levels.

PRIMARILY UNDERGRADUATE: These schools are largely focused on undergraduate education, with relatively few graduate programs.

Because of their distinct mandates, the universities in the three categories are treated as separate but equal; they were not combined into a comprehensive list. Maclean’s ranks the schools in each category on a range of factors in the following broad groupings (with weightings in parentheses):

STUDENT BODY (20 per cent of final score):

The premise was that students are enriched by the academic calibre of their peers. As a result, Maclean’s collected the incoming students’ average high-school grades (12%), and the proportion of those with averages of 75 per cent or more (3%). The magazine also counted the proportion of out-of-province students in the first-year undergraduate class (1%) and the percentage of international students at the graduate level (1%), as a measure of drawing power. The student body section also includes graduation rates (2%)— the percentage of full-time undergraduate students in their second year (after the initial wave of first-year dropouts) who go on to graduate from the institution within one year of the expected time period. This indicator is just one example of the dramatic differences between schools: the lowest-ranking university graduated only 23 per cent of students; the highest, 91 per cent. In addition, Maclean’s collected data on student academic awards (1%) over the past five years.

CLASSES (18 per cent): For many universities, the Maclean’s survey prompted a firstever attempt to collect data on class size and quality. In addition to measuring the median size of first-year classes (3%), the rankings embrace the entire distribution of class sizes at the firstand second-year levels (6.5%), as well as the thirdand fourth-year levels (6.5%). Maclean’s also rated schools on the percentage of first-year classes taught by tenured and tenure track professors (3%), a measure of how much access students have to top faculty. As else where, there was a striking range: at the top school, 94 per cent of classes were taught by tenured or tenuretrack professors, compared with 42 per cent at the bottom-ranked school.

FACULTY (20 per cent): The rankings assessed the calibre of faculty by calculating the percentage of those with PhDs or equivalent (3%) and the number who had won national awards (6%). In addition, the ability of eligible faculty to secure grants from each of the three major federal granting agencies was assessed, along with a measure of both the number and the dollar value received last year. Because of wide differences in the costs of certain research, humanities grants (5.5%) and science/medical grants (5.5%) were judged in separate categories.

FINANCES (10 per cent): This section examined the amount of money available for current expenses at the university (3.3%), as well as the percentage of the budget spent on student services (3.3%) and scholarships (3.3%). When presenting their general operating budget, institutions had to deduct any funds used to pay off debt.

TOP MARKS

The three schools in each category with the highest average entering grades:

PRIMARILY

UNDERGRADUATE

1. Mount Allison 82.2

2. Wilfrid Laurier 81

3.Trent 79.3

COMPREHENSIVE

1. Victoria 84.5

2. Simon Fraser 83

3. Waterloo 82.6

MEDICAUDOCTORAL

1. Queen's 86.2

2. McGill 84.8

3.UBC 84

LIBRARY (12 per cent): This section of the survey assessed the size, cost and currency of the university’s collection. While students often have access to other university collections through computerized catalogues, the campus collection was treated as a critical resource for students. As a result, schools received points for the number of volumes and volume equivalents per student (4%), as well as the percentage of a university’s operating expenses that were allocated to library services (4%) and the percentage of the actual library budget that was spent on updating the collection in order to maintain current standards (4%).

REPUTATION (20 per cent): This section reflects a school’s reputation with its own graduates, as well as within the community at large. When looking at alumni support, schools received points for the number— and not the value—of gifts to the university (5%). The goal was to measure the proportion of alumni over the past five years who chose to give something back to their institution, regardless of the amount.

Within the reputational category, points were awarded on the basis of a survey of senior university officials and chief executive officers of major corporations across Canada (15%). The premise was that a school’s reputation for excellence and innovation affects its profile in the community and the ability of its graduates to find jobs. The CEOs assessed schools at the national level; university presidents and vice-presidents, academic, judged institutions nationally and within their own region; and a range of other university officials made regional comparisons only. Survey participants also named three “leaders of tomorrow.”

As in 1992, all data were calculated by Georges Lemaître, a Hull, Que.-based consultant and former senior analyst with Statistics Canada. Lemaître, who holds a graduate degree in mathematics from the Université de Paris, has extensive experience in data evaluation. For Maclean’s, Lemaître located examples of extreme changes from data reported in the 1992 survey, and directed researchers to double-check those changes with university officials. As well, he reconfirmed the statistical “robustness” of the Maclean’s ranking, ensuring that no single indicator unduly affected the overall result. Finally, he worked with university officials from Quebec to convert grades from their Z score system—in which applicants are graded according to where they place relative to the class average—into percentage grades.

In the end, the goal of the ranking is twofold. First to give Canadians a better picture of what universities are doing, and of how well they are doing it. More importantly, the comparison is designed to give students a critical tool to use in making one of the most important decisions of their lives. For the universities, meanwhile, the survey affords an opportunity for each to clarify its own vision—and to measure itself against its peers.

VICTOR DWYER DIANE BRADY