ACTA's Must-Reads
« Wall Street Journal gets it right | Main | A revealing debate »
Bridging the gap
It's not a matter of debate that American higher education is challenged. Tuition costs are skyrocketing while quality is, by a range of objective measures, declining. According to the American Institutes for Research, over half of graduating seniors at four-year colleges cannot summarize a newspaper editorial, and one in five cannot estimate whether a car has enough gasoline to reach the next filling station. ACTA's own research has shown that over 80% of elite college seniors can't pass a basic high school-level history test.
But while the fact that there are problems is widely acknowledged, the nature of those problems is hotly debated, as is--it follows logically--the question of how to address them. And too often, those debates are shaped along adversarial lines that prevent them from reaching constructive resolutions. This is particularly true in debates about assessment and accountability, where academic insiders tend to view external input and inquiry as intrusive and unwarranted.
Writing in Inside Higher Ed, Jeremy Penn, an assessment associate for the University of Nebraska at Lincoln's Program Excellence through Assessment, Research, and Learning (PEARL), captures the unfortunately territorial quality of so much of our national discussion about accountability and assessment in higher education:
This expectation for assessment as accountability has forced many faculty members and administrators to seek out ways to balance assessment for 'us', or assessment for 'improvement,' with assessment for 'them,' or assessment for 'accountability.' We do assessment for 'us' in our classrooms, to provide feedback to students on their progress, in our programs to provide direction for improvement efforts, for each other when we provide reviews of articles and of ourselves when we evaluate our own performance.
Conversely, assessment for 'them' is done in response to an external demand to prove 'how much students learn in colleges and whether they learn more at one college than another,' as the Spellings Commission put it in its final report.
When we perform assessment for 'us' we are not afraid to discover bad news. In fact, when we assess for 'us,' it is more stimulating to discover bad news about our students' performance because it provides clear direction for our improvement efforts. In contrast, when we perform assessment for 'them,' we try our best to hide bad news and often put a positive face on the bad news that we can't hide.
When we perform assessment for 'us' we do our best to create valid and reliable assessments but don't let the technical details, particularly when they are not up to exacting research standards, derail our efforts. When we perform assessment for 'them,' if there is any deviation from strict standards for validity, reliability, norming group selection, sampling approach, testing procedures or scoring techniques, we are quick to dismiss the results, particularly when they are unfavorable.
We know the 'us'--faculty members, students, department chairs, deans--and we know how to talk about what goes on at our institution with each other. Even amid the great diversity of institutions we often find a common core of experience and discover that we speak each other's language.
But the 'them' is largely a mystery. We may have some guesses about the groups that make up 'them'--parents, boards of regents, taxpayers, legislatures--but we cannot be sure because accountability is usually described generically, not specifying any particular group, and because our interaction with any of these groups is limited or nonexistent.
When we perform assessment for 'us,' we operate under a known set of possible consequences. Some of these consequences could be severe, such as a budget reduction or a reprimand from our superior, but in general the possible consequences are a known and acceptable risk.
When we perform assessment for 'them,' the consequences are much more terrifying because we do not control who uses these data or the purposes of their use. One of the uses of assessment for 'them' is for accreditation, which can bring particularly negative consequences. We wake up in the middle of the night with visions of newspaper headlines publicly disclosing our poor performance.
The us/them dynamic that Penn describes captures the manner in which discussions about students' educational needs have, at times, turned into territorial arguments. Feeling that their autonomy is threatened, academics have adopted an adversarial attitude towards parents, taxpayers, and others who wish to take part in those discussions. Our national dialogue about higher education has at times resembled a turf war centered on who does and does not have the right to ensure that colleges and universities are doing their jobs.
Penn's op-ed did not draw any commentary from Inside Higher Ed's usually vocal readers, a silence that is striking--and perhaps telling--given how readily IHE's readers tend to fall into exactly the sort of us/them debate Penn describes. But he does offer some excellent examples of schools that have tried to bridge the gap between 'us' and 'them' in useful and workable ways.
First, there is SUNY:
The State University of New York (SUNY) Assessment Initiative seeks to strike a balance between assessment for 'us,' or assessment for 'improvement,' with assessment for 'them,' or assessment for 'accountability'. The SUNY Assessment Initiative can be divided into two parts: assessment of general education and assessment within academic majors.
For assessment of general education, SUNY first developed a set of learning outcomes for general education programs at undergraduate degree-granting institutions. All SUNY institutions are required to use 'externally referenced measures' to determine whether or not their students are achieving in the areas of Critical Thinking, Basic Communication and Mathematics. However, to keep this approach in balance, the Assessment Initiative does not require all institutions to use the same measure. Rather, institutions can select from nationally-normed exams or rubrics developed by a panel that best represent their mission in the state. This holds institutions accountable for demonstrating student achievement in foundational areas but will not be used to 'punish, publicly compare, or embarrass students, faculty, courses, programs, departments or institutions either individually or collectively,' according to a description of the program.
Institutions are also required to perform local assessment of their general education programs. Institutions are held accountable for attending to the process of assessment--examining student learning on specific objectives through assessment and making decisions about ways to improve based on those data--by an external group called the General Education Assessment Review group (GEAR). GEAR, composed of primarily faculty members from SUNY institutions, reviews and approves campus assessment plans but not the actual assessment outcomes. In this way, SUNY documents say, 'emphasis is placed on assessment best practice without introducing an element of possible defensiveness campuses might feel if their assessment program does not yield evidence to support optimal student learning.'
Then there are Colorado State and the University of Nebraska at Lincoln:
At the institutional level, Colorado State University and the University of Nebraska-Lincoln partnered together to implement within their institutions the Plan for Researching Improvement and Supporting Mission (PRISM) and Program Excellence through Assessment, Research and Learning (PEARL), respectively. PRISM and PEARL engage faculty members in assessment of the academic major--assessment for 'us.' Faculty members select learning outcomes that are important for students in that major, perform assessment of student learning on those outcomes and then make improvements to their program based on those data. A panel of faculty members from each institution holds the academic majors accountable by reviewing assessment plans and encouraging the use of higher quality assessment practices.
To balance assessment for 'us' with assessment for 'them,' PRISM and PEARL utilize an online software system that allows for the classification of the academic major assessment activity for aggregation at higher levels. In this way the institutions can describe the kind of learning that is going on within the institution, the assessment instruments that are being used to examine that learning and the improvement activities that were performed in response to the assessment data.
For Penn, what makes these programs successful is their focus on student learning and the premise that assessment as a process must continually evolve.
The result in each case is a system of assessment that attends to outcomes as well as processes. Although there is debate about how successful these models are in practice (some have argued, for example, that the SUNY system is too self-referential to be a viable mechanism of assessment), they do make progress toward holding schools and departments accountable. As such, they represent the kinds of measures that ACTA has long been urging colleges and universities to devise and implement. Finally, they exemplify how, to return to Penn's formulation, accountability for 'us' can also be accountability for 'them'--and offer hope that all concerned can transcend an adversarial outlook that only interferes with our collective ability to realize the goal of improving higher education for all students.
Posted by acta online at June 29, 2007 12:25 PM
Trackback Pings
TrackBack URL for this entry:
http://www.goactablog.org/blog/mt-tb.cgi/376
Comments
Oh, great. PRISM. PEARL. "PRISM and PEARL utilize an online software system that allows for the classification of the academic major assessment activity for aggregation at higher levels." Sounds like more gobbledygook from the ed-school types. Just what we need, another multi-million dollar bureaucracy to go along with Diversity Office and all the other Offices. I guess every office for the leftwing fanatics will be balanced by one for the rightwing fantatics. Feeding "data" to the higher administration and feeding more rules and time-wasters to the leeches in the departments. It's not just the millions to run those offices, but the time they force real academics (and students) to waste on their nonsense.
Posted by: Mike at June 29, 2007 02:22 PM
"I guess every office for the leftwing fanatics will be balanced by one for the rightwing fantatics."
With respect to the assessment programs, I missed any mention of political affiliation. Where are you getting this?
Posted by: Federal Dog at June 29, 2007 04:31 PM
In principle, these offices are apolitical, including the Diversity office. Of course, the latter is hopelessly leftwing.
The assessment movement is coming mostly from elements of the political right, as far as I can tell. A good example is goacta. Of course, when it threatens to infringe on their pets, for example the independent conservative religious colleges or places like Hillsdale, they reconsider.
Eventually, I imagine an assessment program would probably be taken over by the left, simply because they have so much control in higher education. When the assessors start assessing how much progress students have made in achieving Diversity learning, and give up on assessing how much they've learned about American institutions, the right wingers will realize what a blunder they've made.
Posted by: Mike at June 30, 2007 12:38 PM
So people have proposed exempting conservative religious colleges and places like Hillsdale from assessment measures?
Posted by: Federal Dog at June 30, 2007 04:42 PM
I don't know how far things have gone as far as exempting the outliers. After all, the assessment measures are only proposals at this time. I do know the president of Hillsdale has written cogently about the dangers of greater federal interference in higher education along the lines pursued by the Bush administration. There was a brouhaha recently about making an exception to accreditation rules for a pet college of the right. The Spellings/Bush crew wouldn't go for it, and there was gnashing of teeth on the right. I don't remember the details, unfortunately, but you can probably find it with some hunting around on the web.
I think the outliers must realize, once they start to think about it, what the dangers are of greater federal oversight through assessment, accreditation, and so forth.
Posted by: Mike at July 1, 2007 11:40 AM
That begs the question of what to do about recent showings (e.g., the most recent NAEL) that colleges are graduating people among whom only 31% show basic literacy skills. As long as someone pays tuition, the kids get their diplomas, despite the fact that they simply cannot read and write. Ignoring the problem of incompetent and unethical "educators" has led to the bottom falling out on even minimal standards. Further, people who should be assuring quality have simply disregarded it for decades.
So what, if anything, do you propose? I mean, besides animosity towards people whose political views you dislike? By the way, I do not know what you are talking about with respect to this unidentified "pet college on the right." Less rhetoric and more facts would be helpful.
Posted by: Federal Dog at July 1, 2007 02:44 PM
OK, I'll do the work and figure out what I meant by "pet college on the right". It had to do with unhappiness on right over a Spellings action that would have killed off accreditation of colleges of a sort beloved by many on the right. Here's a link that will tell you more. By someone with certifiable rightward credentials:
http://article.nationalreview.com/?q=NzBjZDFmMjJhM2E4ODM3YzMxN2UwOTViMGZiMDlhNDY=
"Spellings Bee
The problem with standardized testing in higher education."
a quote:
Secretary Spellings is killing off one of the important reforms that followed from conservative critiques of higher education ... such programs don�t lend themselves to the simplistic �outcomes assessment� that Spellings wishes to impose across all of higher education."
Now -- the statistics you are talking about are very complex to interpret, as anyone finds who actually goes to the data tables. For example, you won't find data on recent college graduates, you'll find data on all college graduates aggregated. You will find that in most categories, most ethnic groups are rising when all education levels are aggregated. However, you will also find that college graduates of all ages as a distinct category are slipping.
What to make of this? Hard to say. It's a mixed bag, very mixed. A lot of it has to do with changing race and ethnic proportions in the country, a place very few people want to go when discussing educational attainment and competence.
In all likelihood, performance of recent college graduates, with all ethnic groups aggregated, is slipping. How about performance within groups? I don't know. Does anyone?
What to do about it? Maybe leave it alone. Maybe federal intervention will only be counterproductive. I'd say let people pick the programs that appeal to them. I doubt, for instance, that a national standardized test would improve the education of MIT electrical engineers. If there's anybody at MIT that wants to major in -- take your pick -- women's studies? Mayan engineering? -- I say let them, if MIT offers that program. If somebody wants to offer a Great Books program, I don't see any reason to put a stupid standardized test around their neck. Is someone unhappy that the thermodynamics taught locally isn't sufficiently multicultural? I think they should be invited to butt out.
If the local legislators think that some of the campuses are so lame they shouldn't survive, they are welcome to close them down, or try to impose whatever quality control scheme they want. I just hope they leave the good programs at the good campuses alone. The less the federal government gets involved, the better.
For myself and most of the people I know, we do our bit to teach at a high level and fight grade inflation. Forcing us to teach to a standardized test -- of what? -- will not help.
Posted by: Mike at July 1, 2007 03:22 PM
Pretty funny. I would imagine that no one can find this exception you're claiming for a "pet college of the right" because Wood's piece does not suggest such a thing.
If academics were competent and ethical, this discussion would not be taking place. The problem is that they are not necessarily either, and they are doing a lot of damage to a lot of people for the sake of personal profit. You propose that maybe we should just continue to ignore the problem. Many of my colleagues would agree with you: People should just stay silent and keep the money coming.
I have serious professional and ethical objections to that position. Students deserve better than to be used in that way. I guess it all boils down to whether or not you value education and people looking to faculty to acquire one. I do. For that reason, I do not believe that the problem should continue to be ignored.
Posted by: Federal Dog at July 1, 2007 04:35 PM
You can havigate the AALE website to find a list of colleges with which they are involved in accreditation:
http://www.aale.org/highered/decisions.htm
You can see that there a number of institutions that want to get going. So make that "pet colleges".
You seem to believe that you're involved with corrupt programs and people. Maybe you are. I don't regard myself and my immediate colleagues that way. Maybe it is better if the students and their parents figure out which is which. Or even the state boards and the trustees. I don't think the federal government can do it for them. Not all educational "problems" are soluble by dictate or industrial style assessment. Not even by the estimable Department of Education.
Posted by: Mike at July 2, 2007 10:12 AM
"Maybe it is better if the students and their parents figure out which is which."
No question about it. And they need some concrete basis on which to do so, which is why some idea of what students actually learn at any given college would be essential information. There are reasons that schools are fighting disclosure of such information.
Realistically, what arena of human activity is exempt from incompetence and corruption? Your declaration of purity is not persuasive because purity is not part of the human condition. Academics are in no way exempt from that condition, despite their heightened self-regard.
Posted by: Federal Dog at July 2, 2007 11:39 AM
"they need some concrete basis on which to do so"
Hint: If the course is on quantum electrodynamics, it's probably the real deal. If it's on Mayan quantum mechanics, it's probably not. No federal bureaucrat is needed, or probably competent, to help figure that out.
Look, I've read literally many hundreds, probably thousands, of student evaluations. I've read my share of complaints, but I have yet to see one asking for a standardized test to find out if they are learning anything. I've talked with lots of parents. Same thing. I've talked with a fair number of legislator types. Same thing. They are concerned about other things.
Maybe you need help over there in the law school. Have a talk with your dean, have a talk with the trustees.
I don't regard my declining to partake of self-flagellation as a "declaration of purity". I don't beat my wife either, by the way.
Posted by: Mike at July 2, 2007 12:24 PM
It's hard to evaluate your post. I said nothing about flagellation, of self or of others. You are too given to over-the-top rhetoric.
The most controversial thing that I am suggesting is that everyone is accountable to others, including academics. People outside the academy can readily determine when someone lacks basic skills that they should have acquired in college (actually, it is all too painfully obvious). There is nothing about being hired in an academic setting that exempts people from ethical requirements of accountability, especially when serious damage is being done by people who do not or cannot teach properly.
I have to wonder why people are so upset about any testing of basic literacy skills. If students are being properly trained, what is so threatening about such evaluation?
Posted by: Federal Dog at July 2, 2007 03:05 PM
By declining self-flagellation, I was alluding of course to the claim that I had made a declaration of purity and was blind to the failings of the human condition. I had meant to add something about an auto da fe, but I passed.
If you are really talking about a test of basic literacy, I believe something like that should be passed before matriculation. The SAT perhaps?
I don't think that's what we're talking about here, though. For example, take the last line of the first paragraph:
"ACTA's own research has shown that over 80% of elite college seniors can't pass a basic high school-level history test."
Aside from the question what it means to "pass" such a test -- this is a tendentious interpretation of a tendentious study -- I don't like the idea that ACTA or Margaret Spellings or Nancy Pelosi should be able to impose such a requirement on MIT engineers or University of Iowa chemistry students or Berkeley music majors (or Berklee music majors!)
If schools want to teach and market this kind of literacy, fine, I say more power to them. If they decline, I don't think it's any business of the Department of Education. ACTA or ISI can decry and harangue to their hearts content, and I might sympathize, but that's as far as it should go.
Posted by: Mike at July 2, 2007 07:30 PM