Many who have not learnt Reason, nevertheless live according to reason.
—Democritus of Abdera (5th-4th century BC)
More than a few readers will likely discern a tautology in our title phrase “assessment technologies.” Assessment instruments and methods are all technologies when technology is understood in its etymological sense as techne, craft or technique (Partridge, 1958, p. 698). When colleagues sheepishly confess, “Well, I do assessment, but I don’t use any technology,” I ask them to explain what it is they do, and inevitably a use of technology emerges. Even in its earliest Latinate sense assessment was technical. In Roman culture, an assessor provided the sovereign with technical advice regarding the value of things to be taxed (Steen, 1999). The assessor helped value the amount to be taxed, and he did so technically. This process of valuation is of interest today, for assessment instruments reflect what we value in learning, deem worthy of measurement. Hence it would be wise to make explicit, in each case, the technology of assessment, to better grasp how it reflects our values and the degree to which the technology determines the results of our studies—all for the sake of clarifying our values and refining our use of technology to serve learning. The confession by a colleague reveals the given or tacit presence of technology. Is there anything more slippery to put in perspective than technology, except perhaps one’s native language which by no means operates independently of everything technical?
Understanding through Research
Technologies of assessment are put in perspective in diverse ways by the authors in the present handbook. Many chapters assembled here demonstrate the curricular and institutional integration of instruments for effective assessment of thinking, writing, learning, and organizational structure in higher education. Other chapters evince a heightened methodological awareness with regard to the relative appropriateness and usability of technologies, what they lack, overlook, but also the opportunities they create. The professional research contained here represents diverse stakeholders whose broad constituency from composition studies to psychology to mathematics, liberal arts colleges to major research institutions, is heartening for those who promote the growth of assessment culture. Creativity receives attention in these pages, as do multi-modal practices and instruments familiar to a new generation of students. The result is to make instruments and methods of assessment more relevant, appear less intrusive and alien, more “ours” and less “theirs.” This research should help allay the pervasive concern, warranted or otherwise, among many faculty members that their daily practice has been commandeered by assessment imperatives in counter-intuitive ways. We know that teachers regularly practice assessment without calling it such. They assess intuitively as it were, putting truth before method. They also make adjustments in their teaching based on their assessments that experts call “closing the loop.” They call it something else—changing the lesson plan; updating the syllabus; refining learning objectives. The purpose of the research in this book, then, is to help educators make their own assessment efforts more articulate and systematic by demonstrating applied research in cases and initiatives. Such applied research is beneficial for readers and practitioners. The historian Droysen’s expression, forschend zu verstehen, “understanding through (re)search,” is apt here (Grondin, 1994, p. 81). It means that in research one relearns and becomes more methodologically explicit about what one already knows and does, creating a new resource. A case in point is when an utterly devoted instructor received help to formalize his grading criteria, carried around in his head for over a decade, at a rubric design workshop where the latest design research is learned and applied. He marveled at the rubric table he produced, which, now objectively sharable, he brought to class and distributed. His intent was not to officially inform his students how he graded and ranked them, but to ask their opinion how they might refine the criteria after a thorough discussion of learning objectives. The result was that the teacher and student collaboratively produced a rubric of which the students had co-ownership, and which raised their awareness of what they were in class to learn. This example confirms that the best teachers use assessment and evaluation to help students learn (Bain, 2004, p. 151).
Yet some educators will remain methodologically vague by choice, their pedagogy romantic, convinced that the intrinsic multiplicity (ambiguity) of the learning experience is immeasurable but true, and any attempt to reach an unequivocal assessment outcome is always already false. Romantics, it is important to remind ourselves (for some of us are them, or have been, or will be on occasion), are comfortable with inexplicit evidence—that which seems to be the case. They suspect that when an assessment technology like Accuplacer informs them of what is in fact the case, method has supplanted truth and technology has become sovereign.
Truth and Method
This sort of technical usurpation is allegorized in Kafka’s story, “A Visit to a Mine” (1976), narrated by miners whose work is interrupted by visiting engineers and inspectors—an accreditation team like WASC if you will, performing an audit of an academic program. “The management has issued some instructions or other about boring new galleries, and so the engineers arrived to make the initial survey.” The miners stare, transfixed by the procession of engineers and their machine replete with measuring instruments, an “extremely expensive apparatus.” The inspection group represents a different professional class. “One, a lively man with black hair, has eyes that take in everything. A second with a notebook makes jottings as he goes, looks around him, compares, notes down.” It is the ninth engineer, however, who embodies instrumental reason in his relation with a perambulator full of instruments.
Obviously he understands the apparatus thoroughly and seems to be really the man in charge of it. From time to time, without stopping the vehicle, he takes up a part of some instrument, peers through it, screws it open or shut, shakes and taps it, holds it to his ear and listens; and finally, while the man pushing the instruments usually stands still, he lays the small thing, which one can scarcely discern at a distance, back into its packing with great care. This engineer is a little domineering, but only in the service of his instruments. Ten paces ahead of the perambulator we have to give way to it at a wordless sign of his finger, even where there is no room for us to make way. (Kafka, 1976, p. 406)
This is not an “invitation to dialogue,” to invoke a phrase from the WASC Handbook of Accreditation (2001). The engineer is engrossed with his instruments and does not communicate with the miners except to soundlessly signal his obtrusive approach. To be sure, there can be a romance with measuring technology as much as with the immeasurable, but in the engineer’s case the relation exposes his domination. The intimate nature of his technical enthrallment, which separates him from others, is perceived through the respectfully objective gaze of the miners, imparting an air of absurdity to the otherwise sober behavior of the engineer. The almost fetish-like character of his relation to his tools is noteworthy, for they are his tools to measure their work space, tools with regard to which the miners can make no claim of ownership.
The reified behavior of the inspector is something before which many educators are most sensitive if not mistrustful. I have been using inspection, audit, and assessment interchangeably here to make a point about the relation between faculty members and assessment instruments and their representatives. I was the administrator when our Division of English and Applied Linguistics was subjected to an academic audit, so I have a concrete impression of the faculty response. The word “audit” is unpleasantly redolent of taxes; no one looks forward to an audit. We saw earlier that assessment is etymologically linked to tax collection. Some faculty members are wary not of mining machines or tax audits but assessment technologies such as Collegiate Learning Assessment (CLA) enthusiastically purchased by their universities. They have heard that Accuplacer and e-Write use an Artificial Intelligence program to produce their scores. For these educators the very idea of such systems and the uncritical bureaucracy that endorses them brings to mind the Skynet computer program in the “Terminator” films. This is writ large a narrative of the Verselbstandigung der Apparatur or autonomization of the apparatus (Adorno, 1998). The program takes over everything and destroys the world. As darkly comical as it sounds, this is not far from the sort of development feared by more than a few faculty members when confronted with assessment mandates and methods.
From Reactive to Active
“We’ve got to determine what happens, rather than adjust to what they do. It’s definitely an important game.” –University of Alabama football player
It would be pleasant to say that most educators have gotten involved with assessment because it is interesting and worthy of attention, sort of like a dissertation topic. And it is interesting and worthy of our attention, as the present book demonstrates. But it is rather the case that assessment sort of befalls many a teacher and administrator who in fact did not study it in graduate school. It comes down on many a head like climatic change, forcing a shift in lifestyle, or marches by in some official capacity that commands our attention, like the inspectors in Kafka’s “Visit to a Mine” who transfix the gazes of the miners with their somber procession. As a result, there is often little intellectual distance between educators and their assessment activity; they simply react to a mandate, grab a tool or template as one would an umbrella or pair of boots, mimic usage and do what needs to be done. They update a set of learning objectives without asking what the objective of learning is in the first place, a much tougher assignment. Pascal’s famous wager, Il faut parier, means, “You must do something; nonparticipation is not an option.” Well yes, but there is busy work and intellectual work, and most of us prefer the latter. The problem is that reactive behavior is not the most effective approach to a serious and complex state of affairs, or for that matter to an academic discipline. Nor does it bode well for the growth of a culture. It would be more active to take ownership of assessment and find in it an intellectually compelling set of problems and challenges for administrators, faculty members, and students to share. Several of our contributors argue this point, most notably Sean McKitrick, John Wittman, and Steve Culver. Motivation would thereby be more intrinsic, related to the matter at stake, and not mandated by the agenda of the newest political majority. Hitherto perfunctory tasks would point back to questions for reevaluating the conditions of learning and the possibilities of improvement in a time of scarce resources. Only in this way can we avoid the constricted cognition described by Collingwood (1978, p. 106): “He was trying to see only those elements in the situation which he already knew how to deal with, and was shutting his eyes to anything which might convince him that his ready-made rules were not an adequate guide to the conduct of life.” So it stands with assessment, whose technologies and assessments try to keep pace with yet inevitably lag behind the new epistemologies of student learning, webs of relations and ways of knowing so browsed, blogged, hypertexted, instant messaged, gamed, and hacked, as to be irreducible to any one identity, data set, or system. Sue Pieper and colleagues, and Deirdre Petttipiece and her group, are some of the authors in our book who look closely as these emerging technologies insofar as they impact or are integral to the assessment process.
The desire to turn from a reactive to active context for assessment is essentially the origin of the present book. In 2006, at the CCAS conference in Boston I heard many deans complain how difficult it is to get faculty members involved in assessment without rancor or resistance. One of the most heavily attended sessions was “Getting Faculty Excited about Assessment.” I realized something obvious, that for assessment to be done with enthusiasm by faculty members it would have to look like their other academic writing and have the same scholarly value, be integrated with their professional personae and rewarded on the scale of promotion. Hence it became my project to solicit substantial research papers in assessment for a book-length manuscript. Here would be my supporting evidence for saying to my colleagues, “Write up assessment projects not as internal memoranda, but with the flair, polish, and intellectual intensity you put into your scholarship.” This is apparently more difficult in practice than it sounds, for one rarely sees essays on assessment; it is rather the case that we have technical treatises and studies that do not, by any stretch, belong to the genre of the essay. A simple name change from “report” to “reflective essay,” recommended by WASC for purposes of documenting the institutional self-review process, does not automatically qualify the actual documentation being produced as an essay, at least as typically understood by its practitioners (Atkins, 2005). Nonetheless to even speak of an essay format for documenting educational assessment is a step in the right direction, and concretely confirms the sincerity of WASC’s stated intention “to maintain a posture of experimentation in the years ahead” (WASC, 2001).
Somewhat incredulous then, not unlike some of my colleagues, I wanted to see if assessment has depths. I knew that the place to look is in the research. Once I knew the answer, I could earnestly persuade colleagues to become engaged or involved with assessment, as is argued by several authors in the present book. Furthermore, few of my colleagues, however incredulous of assessment initiatives, would oppose the idea that as scholars we are still learning. Howard Gardner makes a good case for this in his description of what he calls “the disciplined mind,” which demonstrates “continuing mastery of one’s professional or employment roles, including the acquisition of additional disciplinary or interdisciplinary acumen” (Gardner, 2007). It would be beneficial then if assessment is viewed as an element of this new knowledge, which is very much conceived as an active learning process. In its active modality, alternative and authentic assessments become feasible (McMillan, 2001), as demonstrated in this handbook by Eubanks and Brunsden. This shift from passive to active awakens new kinds of thinking when one no longer resists but engages. One recalls here the protagonist in Kono Abe’s The Woman of the Dunes (1991) who at first seeks escape from a dwelling that is under constant threat of burial by sand dunes. He refuses to dig until he realizes that not digging sand is not a solution and the digging must be integrated harmoniously into his daily existence. At the point where his digging becomes active, his behavior evinces inventiveness; he builds a water collection device to make life more endurable.
Myth Interrupted
Happily, the chapters that appear here, arising from serious research, will disabuse the incredulous of their suspicions. As Henry James said in defense of his novels, “there are depths” to assessment, and, one has to say, a richness of conceptualization. I wanted to see if there are depths, not only shallows—not that shallows can’t be interesting and yield their own depths. But how many pre-and post tests based on ten items learned over fifteen weeks must we administer before we realize we need to swim more deeply to assess deep understanding, reasoning, and academic skills? But to pause for reflection, to gauge the relevance and appropriateness of a method is not as convenient as applying it and reaping data, however trivial. Let us remember what such methods do for us: as is said of myth, “it is a system of the elimination of arbitrariness” (Blumenberg, 1985, p. 43). Assessment, like myth, alleviates anxiety, gives us a semblance of measure and order where there seems disorder, simplicity where the situation is complex or nuanced.
In a recent essay in which Les Perelman seeks to put Collegiate Learning Assessment in perspective, he says the following about the Voluntary System of Accountability that was the bureaucratic precondition for the CLA: “This enterprise, however, of trying to measure and then compare the common benefits of a college education among widely differing educational institutions through two-hour standardized tests reaffirms the truth of H. L. Mencken’s observation that ‘For every complex problem there is an answer that is clear, simple, and wrong.’ These assessments are, at best, useless, and, at worst, subversive to the complex set of abilities that should inform undergraduate education. (Perelman, 2008, p. 1). It seems to me that what Perelman is doing is demythologizing assessment by warning us about blind faith in system-wide applications. Something like CLA is not a magic formula that is universally applicable. Bob Broad, who contributed the Foreword to this book, has done equally critical work with rubrics (2003). Another one of our contributors, Daniel Chambliss, moves in the same direction when he speaks of the individual as a neglected unit of analysis and when he examines the bureaucratic language that invokes assessment as a cure-all (2007). Likewise Stephen Ehrmann (1998) when he argues for a “unique uses” perspective to bring proportion to a paradigm that favors a uniform (average) impact perspective. Not solely enthralled with deriving data associated with what is average, he is encouraged by the fact that diverse outcomes are possible in the same course because of the different life paths students follow; a course is seen less as a set of fixed objectives than as a flexible learning opportunity. This is the case with delayed outcomes whose truth is untimely. Delayed outcomes can appear abysmal in present evidence, only becoming positive eight years hence, as when a student takes an advanced philosophy course as a sophomore, barely passes it, flops in exams and papers, yet is instilled with an obscure thirst for speculative inquiry that later matures into a passion and vocation in philosophy proper. A “C” student becomes a renowned teacher of philosophy. We have all heard of such cases.
Let us rephrase and say that the scholars cited above do not so much demythologize as
interrupt myth in assessment, to borrow a phrase from Jean-Luc Nancy (1991), prevent it from becoming a reified practice by practicing critique as integral to its effective implementation. Other contributors to this handbook, such as Asao Inoue, Mya Poe, and John Wittman, are forging ahead early in their careers with critically aware research as they focus on the related issues of race, cultural diversity and technology in assessment.
The KISS of Death
All of the scholars mentioned here, and many handbook contributors as well, deserve our attention because they practice critique while fully involved with the matter of assessment. By so doing they “desimplify” assessment, to borrow a catchword of the heyday of deconstruction. Much of the resistance to assessment arises from the problem that Mencken, cited by Perelman, observes above. The oft repeated slogan “KISS” (Keep it Simple Stupid) has done more harm than good, the kiss of death, further convincing already skeptical colleagues that “assessment culture” is less a watchword than an oxymoron. Many educators suspect assessment to be intellectually vacuous not only because of the KISS slogan with which it is associated, but because they often hear that it is focused on outcomes. Those who have privileged process over product are understandably perturbed. But it is not assessment per se that is profound or not, since it can take many forms and styles, but rather certain outcomes--say, a simple pre-and post test assessment project. These simple outcomes don’t condemn assessment as an activity; they merely challenge practitioners to invent new and more encompassing assessment instruments, such as the critical thinking and/or writing rubrics described in detail by Hedva Lewittes and Teresa Flateby, to name just two of our contributors.
Chapter Organization
The chapters have been organized in three sections. In all but a few chapters the research is applied, that is, methodologically worked out in concrete frameworks of data collection and analysis, either with actual data sets or literature searches associated with a concrete matter at stake. A focus on assessment design and instrumental qualification in an applied framework characterizes the chapters in Section One. Although many of the chapters here are case studies, most of those which explicitly declare themselves as such, and which were not primarily concerned with instrument design so much as implementation, comprise Section Two. It is always nonsensical to separate applied from theoretical chapters for many reasons, but those with more overarching or ostensibly critical viewpoints, as well as those focused on workplace and assessment culture issues, were placed in Section 3.
Section I
Assessment Technologies and Instruments
Chapter I describes a model for evaluating complex organizations or systems. The design assessment model proposed is a response to current notions of assessment. Purdue University’s strategic planning process provides a context for describing how design assessment takes place in a higher education setting.
Chapter II describes how critical thinking is assessed using two critical thinking learning outcomes that were required for the State University of New York’s General Education program. As part of this process faculty developed a rubric that delineated the criteria for rating critical thinking on a four-point scale.
Chapter III reviews literature over the past ten years regarding technology tools that are being used in higher education to assess student learning. Three commonly used technology tools are examined: electronic portfolios, course management systems, and audience response systems.
Chapter IV explains the Consensual Assessment Technique, discusses how it has been used in research on creativity, and explores ways it might be employed in assessment in higher education. Unlike other measures of creativity, such as divergent-thinking tests, the Consensual Assessment Technique is not based on any particular theory of creativity, which means that its validity (which has been well established empirically) is not dependent upon the validity of any particular theory of creativity.
Chapter V discusses creativity assessment as a means for evaluating skills required in higher education. Creativity is assessed in the context of the creative person, process, product and press or environment. A historical view of creativity assessment is addressed with a substantive approach to understanding the construct of creativity, its measurement and evaluation, and the broader implications for use in higher education settings.
Chapter VI conceptualizes writing assessment as a technology, accounting for the ways in which assessment dialectically constructs and is constructed by its historical environment. Seeing writing assessment as a technology provides a full account of assessment as an environment of conflict and social reproduction, but most importantly, it accounts for racial formations existing around it and because of it.
Chapter VII explores ways qualitative and quantitative methods are complimentary, as opposed to competing concepts. The chapter reviews the literature on qualitative, quantitative, and mixed method designs, then highlights successful examples of mixed-method assessment at a mid-sized, private university, presented in general frameworks which can be used on any campus.
Chapter VIII describes the development of the Cognitive Level and Quality of Writing Assessment online system. Beginning with needs identified in a learning community program, the system evolved from a classroom analytic writing and thinking assessment rubric to an online system for classroom assessment and instructional purposes.
Section II
Assessment Applications and Initiatives
Chapter IX reports on the process used to develop an outcomes assessment initiative for the Multimedia Writing and Technical Communication Program at Arizona State University. Discussed herein are the development of outcomes, the mapping of outcomes to the curriculum, the use of electronic portfolios to assess student writing using Phase 2 scoring procedures, and how results from the first three semesters of implementation are being used to evaluate and improve the program’s curriculum.
Chapter X details the Composition program self-assessment conducted at the University of Louisville in anticipation of a review conducted by the Southern Association of Colleges and Schools (SACS) in 2006. The chapter includes a comprehensive analysis of internal assessment rationale, theoretical foundations, methodologies, and results.
Chapter XI uses a case study to report assessment of three instructional delivery formats: (a) online, (b) distance, satellite campuses, and (c) traditional on-ground format. Student competencies on learning outcomes in a basic interpersonal communication college course were analyzed in a core assessment project (a course assignment portfolio) using a department-approved assessment rubric. This assessment effort provides an example of how faculty can employ assessment as part of a continuous improvement cycle.
Chapter XII presents a case study of a college classroom strategy that enables assessment and some remediation of student problem solving skills in mathematics. It reports the outcomes of implementing this technique and its associated processes in several lower-level mathematics courses in the calculus sequence at Penn State- Altoona.
Chapter XIII describes Coker College’s subjective performance assessment program to rate student thinking and communication skills. It uses a discussion of the epistemology of assessment to motivate an emphasis on direct observation by experts as the basis for “authentic” assessment for complex learning outcomes. The purpose of this chapter is to provide a philosophical framework and practical methods that can help institutions assess liberal arts learning outcomes.
Chapter XIV describes an assessment project conducted during the 2006-07 and 2007-08 academic years at the University of Guam in Elementary Japanese I and II. This chapter presents the results of this assessment and a reflection on the assessment process itself, including its rationale, methodology and consequences in terms of on-going and future assessment in the Japanese Studies program and Foreign Language instruction.
Chapter XV narrates an outcomes assessment study completed in a basic composition course at a small urban open admissions community college. The course was a pilot course designed in response to marginally remedial performance on a standardized writing instrument and solidly exempted performance on the standardized reading instrument. It will be shown how data was used to guide decision making about curricular change on our campus.
Chapter XVI reports research on the impact of different forms of peer observation and feedback on preservice teachers’ skills in analyzing classroom teaching and their perceptions of their experience with peer assessment. According to this study, peer observation and feedback is beneficial to preservice teachers’ learning. However, to maximize its effectiveness, a culture of peer assessment should be established in teacher education programs.
Chapter XVII offers a detailed description of an innovative senior capstone program developed by the Department of Communication Studies at Samford University. The program incorporates student-driven communication workshops and electronic portfolios, and uses qualitative and quantitative methods to assess Transformational Learning Abilities or TLAs. The program is shown to serves as a catalyst for student engagement, faculty development, and departmental transformation.
Section III
Assessing Assessment
Chapter XVIII argues that the trend favoring assessment initiatives of a system-wide scope such as program review and collegiate learning tend to overlook the specific, highly concrete learning experiences of individual students in the liberal arts. These individual cases offer a rich source of data. The insights that can be derived from a rigorous analysis of such individual experience can tell educators much about learning outcomes, teaching quality, and curricular effectiveness.
Chapter XIX introduces a variety of multi-modal writing assessment instruments and methods, and discusses potential ways to determine the value of the student texts produced with them. New instruments associated with computerized scoring and distance technology make multi-modal writing assessments readily available, affordable, replicable, and transferable, but the value of the texts produced must be carefully ascertained.
Chapter XX introduces methods that can be used to engage faculty in the assessment process, working within a shared governance structure in institutions of higher education. It begins by identifying assumptions about including faculty in the assessment process, placing special emphasis on social capital and networking theories often used in communication and sociological research. The chapter t identifies six methods that might be used to engage faculty strategically in the assessment process, and then used three case studies to help explain these methods.
Chapter XXI suggests five elements to consider when developing an environment for assessment that successfully engages faculty: structure of assessment; qualifications of those in assessment; focus of assessment conversations; faculty development; and linkages with other areas within the institution.
Chapter XXII argues that as primary stakeholders in writing program assessment, students and instructors need to be included proactively in assessment research. However, little research has been done to determine how to accomplish this methodologically even though assessment affects pedagogical practices, student populations, and public opinion about what constitutes good writing. Instead of traditional quantitative, psychometric research, this chapter argues that assessment practitioners need to utilize local opportunities to discover native needs.
Chapter XXIII is guided by evidence that the study of racial-ethnic group differences on educational tests has yielded a substantial body of research internationally in the last decade. It maps current research about racial-ethnic minorities and educational testing in the English-language educational assessment literature. From an initial search yielding 420 articles, 78 were identified for further analysis of the frameworks for defining race-ethnicity and discourses found in the research literature.
Chapter XXIV describes a method developed to assess the outcome of a “cultural familiarity” general education goal. Challenges in defining, measuring, and providing summary information on variables of interest are discussed. The authors review the process of developing their own “oral examination” assessment method, explain their rationale for using this particular method, and suggest that locally-developed methods – this one and others – may have particular benefits that make them especially useful for program review and revision.
Concluding the book there is a glossary of assessment instruments, many of which are not fully invoked, studied, or referenced in the preceding chapters, yet which are relevant to studies and conversations about assessment.
Not the Final Word
Most of the contributors to the present work understand that assessment is not the final word in the total effort to raise the quality of learning in our institutions of higher education, no more than the final exam is the high point for even the best students in a well attended and dynamic seminar. For my own colleagues, and for many faculty members I have met elsewhere, knowledge of the goals and methods of assessment helps them survive the visits of the accrediting agencies. End of story. Many faculty members go back to their teaching and research until the next round of visits. Understandably, no few think it is a passing fad; they have seen many fads over the years. Only so many construe and value assessment itself as an area in which they will conduct and publish research. The latter are the ones who have contributed to the present volume, and we are grateful to them for doing so.