Outflanking the Rankings Industry:

When the U.S. News survey on academic reputation arrives this spring, presidents should throw it in the trash.

American higher education needs to take on the forces that are eroding our credibility.

Whipsawed by a cascade of critical public reports and rankings of dubious merit, we have become defensive, stifling our voices on large societal issues while grimly enduring probes into our quality and integrity.  Instead of seeking to evade this debate, presidents and trustees must be more passionate advocates on behalf of the mission and priorities of a truly higher education.

Let’s start by confronting, not enabling, the rankings industry.  Rankings are big business for commercial publishers, belying their solemn invocation of “public accountability” to justify these annual drives for improved profits.  More than the Spellings Commission report and similar public diagnoses, the rankings industry has affected the reputations and priorities of colleges and universities, leading some institutions to consider actions that are, in fact, harmful to the public interest — a prime example being limiting access for low-income students to ensure more competitive retention and completion rates. 

Moreover, far from helping consumers make better choices about college, rankings encourage the public’s infatuation with celebrity at the expense of substance.  Americans crave association with the team, car, movie, or school that shouts, “We’re No. 1!” regardless of whether there’s a rational relationship between the ranking and the results for the individual.

Before the next round of survey instruments hits our desks this spring, presidents and trustees should have a thoughtful discussion of the purpose, values, and ethics around institutional participation in commercial rankings.  Then, we should take direct action to produce better information for consumers, taxpayers, and regulators.

What Must Be Done.

Of course we should be highly transparent and publicly accountable — and we should take the lead in modeling effective methods for communicating constructive information about the effectiveness of the student experience on our campuses. Following are some points presidents and trustees should consider in discussing commercial rankings:

• Consumer information. Do American consumers really need U.S. News & World Report to tell them how to choose a college?  In a November 2006 panel discussion sponsored by the Washington think tank Education Sector, U.S. News editor Brian Kelly explained that “America’s Best Colleges” arose from a vacuum of consumer information, making the annual publication sound like a wholly altruistic venture.

It’s not. “America’s Best Colleges” sells a lot of magazines. Whether it is an effective guide for consumers depends upon how students and families use the information. The statistical information about schools can be interesting and, yes, useful, but focusing on the rankings themselves is a poor way to select a college. Those reading “America’s Best Colleges” might be justified in thinking that institutions ranked in the top tiers are the best places to obtain a college education.
But what is best for any given student is very different from what makes an institution competitive in the world according to U.S. News.

As Education Sector’s Kevin Carey has illustrated in his study “College Rankings Reformed” (www.educationsector.com), the U.S. News ranking criteria have more to do with “fame, wealth, and exclusivity” and far less to do with the amount of learning that may take place at a given institution. Instead, the magazine uses surrogate measures such as SAT scores, faculty salaries, spending per student, and the rate of alumni giving to pronounce the “quality” of a college or university.

At a time when tuition prices are hotly debated, it is ironic that an allegedly “pro-consumer” ranking system advantages institutions that spend a lot more money (higher faculty salaries at doctoral institutions, where faculty often have light teaching loads) at the expense of those that are more efficient with tuition dollars (modest faculty salaries at smaller “teaching” colleges that have far better track records for effective teaching).

In what must be the most shameful misuse of data in the rankings game, fully 25 percent of the U.S. News score is premised on an “American Idol”-style contest in which presidents and deans are asked to rate the “reputation” of one another’s institutions on a 1-to-5 scale. That most presidents and deans have little insight into actual learning results at other institutions doesn’t stop them from offering their opinions. In fact, a small percentage of those who respond to the reputational survey have acknowledged “gaming” the system by giving lower scores to their competitors.

Ten years ago, Stanford University President Gerhard Casper wrote to James Fallows, then the editor of U.S. News, as follows: “I am extremely skeptical that the quality of a university—any more than the quality of a magazine—can be measured statistically.  However, even if it can, the producers of the U.S. News rankings remain far from discovering the method.”

Casper’s words continue to ring true.  With more than 15 million students enrolled in American colleges and universities, we must firmly reject the notion that a very few institutions are “best” while the vast majority slog along as second, third, or (horrors!) fourth-tier losers.

Rankings may be fine for football teams and the Top 40 popular songs, but they seriously distort collegiate choices. What’s best for an aspiring biologist might be quite different for the young writer or the mid-career woman looking to complete her business degree.  Some of America’s best colleges — those that regularly produce measurable improvements in student learning — are stuck in the third and fourth tiers in the U.S. News rankings.  They don’t have a lot of money, but they have great faculty who love to teach. 

What can presidents and trustees do about this? First, we need to pay closer attention to the ways in which our marketing efforts address the true informational needs of prospective students. Winning basketball’s Final Four or having a Nobel Prize in physics are certainly marks of distinction, but unless a prospective student plays basketball or majors in a science, such achievements have little impact on his or her success in college. How does your institution’s admissions process help prospective students align their learning needs, abilities, and interests with the institution’s programs and pedagogies?

Second, we should look at the ease with which consumers can access data and information about our institutions. Contrary to the statements of U.S. News editors and other rankers, as well as the repeated assertions of Spellings Commission chair Charles Miller, consumers have ready access to a wealth of information about colleges and universities. However, some of the information is too obscure for consumers to understand easily. Creating and displaying information about costs, financial aid, learning outcomes, and student satisfaction in formats that consumers find accessible and useful should be a top priority of all colleges and universities. How long does it take to find your institution’s table of tuition and fees on your Web site?

Third, we need to make a stronger case to the public that there is no substitute for visiting the campus and talking with staff, faculty, and perhaps most important, current students. Face-to-face conversation is the best way not only for the institution to learn about prospective students’ interests and potential needs but also for students to assess whether a college aligns well with their academic and social goals and needs. Does your institution allow time for individual conversations with prospective students, or are they herded around in groups?

Higher education’s critics continue to insist that we should produce a magic bullet of information, readily available on the Internet, as the means to ensure that students and families are able to make an informed choice of where to attend college. Contrary to the analogy often stated by Education Secretary Margaret Spellings, choosing a college is not like buying a car over the Internet. But it is true that a sensible car buyer would not dream of plunking down thousands of dollars without first going for a test drive. What is the “test drive” experience on your campus?

• Accountability. Some of the rhetoric around rankings today implies that these beauty contests might serve as a formal means of public accountability for higher education.  Of course universities must be accountable, but rankings are the wrong means to achieve this objective.

Consider the Washington Monthly’s new rankings (www.washingtonmonthly.com).  They largely illustrate what happens when journalists decide to make the news rather than report it.  In the name of “accountability” for the investment of tax dollars in universities, the editors decided that the real question about “best” colleges should be the following: “What are reasonable indicators of how much a school is benefiting the country?”

The editors then came up with idiosyncratic criteria to measure outcomes in relation to three standards they devised.  So, to measure the “ethic of service to country,” they look at the percentage of students enrolled in ROTC and the percentage of alumni enrolled in the Peace Corps.  The value of this methodology to the consumer of higher education is obscure, while ignoring whatever other contributions universities might be making to their communities and the nation.

What can presidents and trustees do about the conflation of rankings and public accountability? We can shine a brighter light on the comprehensive accountability systems already in place through accreditation processes. We should go one step further and voluntarily embrace the concept of transparency when it comes to publication of all accreditation reports.

Some institutions resist this notion vigorously, though this contributes to the perception that higher education resists accountability. If ever there were an issue where the word “transparency” has real meaning, this is it. So what if some of the accreditation reports might be less than self-aggrandizing? A display of institutional humility while discussing problems in key performance areas might do wonders to restore credibility to our sector. (At Trinity, we publish our accreditation reports on our Web site, www.trinitydc.edu.)

• Institutional competitiveness.  Rankings have changed behaviors — of institutions, not consumers — and not necessarily for the better. Every spring, sure as the crocuses pop through my lawn, I begin to receive letters and publications from my presidential colleagues — a few at first, and then a deluge. My newfound friends are eager to regale me with the great achievements of Wonderful College and Exceptional State U. They are, of course, campaigning for my vote in the U.S. News reputational survey. Little do they know, I rip up my survey and throw it away! They should do the same, rather than indulging this codependent relationship with U.S. News. The amount of money spent on trying to influence peer votes could fund several more scholarships on each campus.

Fight the Power.

What can presidents and trustees do about the unseemly behaviors the rankings foster? Just say no—to pressure to take actions dictated by rankings rather than mission and strategic priorities, to spending precious institutional resources on campaigns for peer votes, and to answering any survey that asks for your opinion about another institution, especially when whatever you might know is based on gossip and newspaper stories.

Trustees can help presidents keep rankings in perspective. Rather than demanding to know why your university dropped several places in the rankings, or rather than celebrating an increase of a few notches, ask more meaningful questions about the data collected and reported in these magazines:

• What student achievements best represent our values?
• What are the typical questions families and prospective students ask of the admissions office? Are we providing the clearest, most effective information possible to them?
• What student-learning assessment tools do our faculty consistently use? Do we have a student-learning assessment plan and program that effectively capture and report student-learning outcomes?
• Do we track our year-to-year retention rates effectively? What do these results teach us about educational effectiveness?

Rankings may be inevitable parts of higher education’s reality today, but they need not be determinative of our reputations and purposes. I have spent enough time with the critics of higher education to know that our industry cannot continue to ignore the unrest in the general public. Working together and across institutions in a renewed climate of openness, transparency, and collaboration—not competition—presidents and trustees can rekindle the public’s imagination about the vitality and excellence of American higher education.

With renewed public confidence, we then can reclaim the voice we once had on the larger societal issues that need our brainpower and daring exercise of intellectual freedom.

-----------------------------------
 

Peter T. Ewell, “Do We Make the Grade?” Trusteeship Magazine, November/December 2006

George D. Kuh, “Seven Steps for Taking Student Learning Seriously,” Trusteeship Magazine, May/June 2005.