Home | What's New | Articles Archive | Washington Archive | Resources Archive | Positions Archive | Reviews Archive | Links | Conferences | About Us
|
|
Home | What's New | Articles Archive | Washington Archive | Resources Archive | Positions Archive | Reviews Archive | Links | Conferences | About Us |
Education in the United States appears to have entered into a new era. While difficult for many people to appreciate at the time, the passage of the No Child Left Behind (NCLB) Act of 2001 (P.L. 107-110) is beginning to cast its shadow as a defining landmark event in American education. The emphasis on the scientifically-based evidence underlying educational practice has moved educational research into an arena with attention that it has seldom seen.
At the same time, the current budget climate in the United States, and in most states, has prompted increased scrutiny of technology expenditures. It is reasonable to anticipate that technology expenditures will be increasingly difficult in the days ahead. On the other hand, perhaps it is not surprising that attention is now focusing on the role of technology in testing (Education Week, 2003).
It is noteworthy to observe, within the context of increased accountability and desire to understand the value of technology investments, three national research centers have been established to advance an agenda that will substantially increase the knowledge base surrounding assistive technology and its effective use by individuals with disabilities.
First, the Office of Special Education Programs (OSEP) funded the National Assistive Technology Institute based at the University of Kentucky. This center is charged with conducting AT research, translating research into AT practice, and providing resources to improve the delivery of AT services. To learn more about this center, visit the NATRI home page: http://natri.uky.edu.
A second federal agency has also been concerned about assistive technology and has funded priorities to advance a research agenda concerning assistive technology outcomes. In October 2001, the National Institute on Disability and Rehabilitation Research (NIDRR) funded two, five-year, research centers to address the gap in data collection efforts concerning assistive technology outcomes, as well as the paucity of measurement instruments and strategies. The ATOMS (Assistive Technology Outcomes Measurement System) Project is based at the University of Wisconsin-Milwaukee and CATOR (Consortium for Assistive Technology Outcome Research) is housed at Duke University. Additional information about each of these projects can be found at their respective web sites: http://www.atoms.uwm.edu and http://www.atoutcomes.org.
While the profession waits for products and resources to be produced by the three federal projects, practioners will be increasingly called upon to answer the question: How do we know assistive technology is effective? Indeed, this is a timely and important question.
In contrast to the abundance of measures and indicators that profile the acquisition of technology, less information is typically available to reflect the impact of assistive technology. While I have previously addressed these issues in the context of high incidence disabilities (Edyburn, 2000), in this essay I will advance a framework for understanding the process of using evidence to inform decision-making about the impact of assistive technology.
I believe it is important to distinguish between claims of effectiveness and statements which can be supported by evidence and subsequently verified. That is, just because I say a device is effective for a specific child doesnt mean everyone will agree with my statement. As a result, I would suggest that all statements about assistive technology effectiveness be treated initially, simply, as a claim.
The routine scrutiny of each claim concerning the effectiveness of assistive technology is a nondiscriminatory professional practice. That means, vendors are not held to a higher standard simply because they are a for-profit organization versus the standard an independent engineer would need to meet to show the effectiveness of his product. Similarly, that means that parents are not required to present more proof than a professional to support a request for assistive technology devices and services.
To judge the validity of a claim that assistive technology is effective, we need to review some evidence. As illustrated in Table 1, I have come to view the challenges associated with the collection of evidence to support the effectiveness of assistive technology to be a developmental process that passes through distinct phases:(a) exploratory phase, (b) descriptive phase, and (c) empirical phase. In the sections that follow, I will elaborate on these concepts and describe the types of data that are often advanced in support of a claim.
Table 1 A Framework for Understanding the Use of Evidence to Inform Decision-making About the Impact of Assistive Technology Exploratory phase
Descriptive phase
Empirical phase
|
The exploratory phase is often associated with new product development. Two benchmarks can be observed in the exploratory phase: (a) intuition and (b) observation.
Evidence that is described with statements like I just know it works. requires deferring to the judgment of the individual, parent, teacher, or product designer. As a result, this type of evidence may be more properly referred to as intuition.
A second benchmark is observation. Evidence gathered from observers watching the assistive technology in use and drawing personal conclusions that It appears to help.
Neither of these two benchmarks use formal data collection procedures since the value of the assistive technology appears obvious. As a result, evidence gathered in the exploratory phase is seldom considered as convincing proof that the assistive technology is effective. However, experience in this phase may lead to more systemic data gathering efforts in subsequent phases.
The descriptive phase represents an advance in the type of evidence that is presented in support of an assistive technology effectiveness claim. Two benchmarks can be observed in the descriptive phase: (a) anecdotal evidence and (b) case studies.
Anecdotal evidence involves gathering stories and critical incidents that can be used to make an argument that the assistive technology is effective. Often these types of data are used to inform purchasing decisions when the assistive technology is inexpensive since the efforts required to collect data are commensurate with the cost of the technology (e.g., a hand-held spelling checker). The shortcoming of anecdotal evidence is that it is generally unconvincing due to the impressionistic nature of the data and the lack of comprehensive or systematic procedures for collecting data. Nonetheless, reports with anecdotal evidence have the potential to inform subsequent research by identifying key variables impacting the effective use of assistive technology.
Case studies represent another method of data collection. Generally, data collection focuses on a single individual. Since case studies typically employ systematic procedures for data collection and analysis, the results often present evidence that have some credibility in scientific circles; although the findings may or may not be replicable. Questions may also be raised concerning whether or not the observer was impartial.
Critics of evidence gathered in the descriptive phase often cite the inadequate sample size for basing decisions. However, the descriptive phase of evidence-based practice represents an important developmental benchmark in the search for reliable information about the effectiveness of assistive technology. As such, it signals a transition between sources of information which involve personal ways of knowing and more scientific approaches which involve systematic commitment to data collection and analysis.
The empirical phase typically involves well-designed research studies which seek to gather quantifiable data for statistical analysis. Three benchmarks can be observed in the empirical phase: (a) group studies, (b) research synthesis, and (c) meta-analysis.
Group studies are a basic tool for researchers who seek to identify differences based on specific treatments or interventions. While group design studies are common in general education, in special education, differences in disabilities have made this measurement technique of limited use. As a result, few group studies have been conducted in assistive technology due to fundamental challenges involved in identifying appropriate samples.
Research synthesis is a process of analyzing patterns across a body of research studies. Typically, this process is of limited use during the emergent period of growth of the research knowledge base (Edyburn, 1999). The requirements involving an adequate number of studies of which to make generalizations is a technical as well as a developmental problem for the field of assistive technology.
Meta-analysis is a specific statistical procedure that computes an effect size about an intervention across a body of research to try and understand the effectiveness of the intervention. While Swanson (1999) has reported an effect size for technology use by students with disabilities as .77 (highly effective), for the reasons noted in the preceding areas, to-date, meta-analysis has had limited application in the field of assistive technology.
Historically, empirical research has been the gold standard for arguments about proof concerning the effectiveness of a specific intervention. As a result, this standard has subsequently been incorporated into NCLB legislation.
Fuhrer (1999) provides a thoughtful summary of the historical context of assistive technology outcomes and describes his impressions concerning the relatively slow response of the profession to ask questions about the effectiveness of assistive technology or seek methods to measure the outcomes of assistive technology.
Where are we today? Its hard to say. Research on students with disabilities and how they use word processing tools is much more advanced than what we know about assistive technology in inclusive classrooms. If the types of evidence outlined in Figure 1 represents a developmental model, it appears the field is making progress in trying to use evidence in all three phases to inform decision-making. However, the gold standard of empirical evidence imposed by NCLB is based on group design studies. As a result, the answer to the question, "Where are we today?" is "In trouble." Research on the effectiveness of assistive technology is neither sophisticated nor adequate.
The emphasis on the scientific-based evidence in NCLB challenges the professional community to build the knowledge base about the effectiveness of assistive technology. Readers interested in learning more about No Child Left Behind are encouraged to consult the following web sites:
No Child Left Behind
http://www.nochildleftbehind.gov
No Child Left Behind: A Desktop
Reference
http://www.ed.gov/offices/OESE/reference.html
CEC NCLB Technical Assistance Briefing
http://www.cec.sped.org/pp/OverviewNCLB.pdf
Education Sciences Reform Act of
2002
http://www.ed.gov/legislation/EdSciencesRef/
I welcome your comments and reactions to this essay (email: edyburn@uwm.edu).
Education Week. (2003). Technology Counts 2003: Techs answer to testing. Available online: http://www.edweek.com/sreports/TC03/article.cfm?slug=35exec.h22
Edyburn, D.L. (2000). Assistive technology and students with mild disabilities. Focus on Exceptional Children, 32(6), 1-24.
Edyburn, D.L. (2000). 1999 in review: A synthesis of the special education technology literature. Journal of Special Education Technology, 15(1), 7-18.
Fuhrer, M. (1999). Assistive technology outcomes research: Impressions of an interested newcomer. Paper presented at the International Conference on Outcome Assessment in AssistiveTechnology, Oslo, Norway, November 25, 1999. Available online at: http://www.utoronto.ca/atrc/reference/atoutcomes/newcomer/index.html
Swanson, H.L., (1999). Instructional components that predict treatment outcomes for students with learning disabilities: Support for a combined strategy and direct instruction model. Learning Disabilities Research and Practice, 14, 129-140.
© 2003 ConnSENSE Bulletin