There has been growing interest in "teacher quality," a catch phrase for a host of teacher characteristics, including a teacher's content knowledge, classroom behavior, academic ability, advanced degree work, salary, and teacher education experiences.  Among the many characteristics under investigation as an indicator of teacher quality has been teacher certification.  This study examines the effects of different kinds of teacher certification on student achievement.  Reviews of this issue may be found in Darling-Hammond, 1999 and 2002; Evertson, Hawley & Zlotnik, 1985; and Ashton, Crocker, & Olejnik, 1986. 

In Arizona, a state with a strong commitment to standards based reform, policies were needed to ensure that quality teachers would be available for students to meet the new and more rigorous mandated standards.  Thus the Arizona Educator Proficiency Assessment (AEPA) was developed as a tool in the state certification process to ensure the quality of new teachers.  One part of the test purports to measure teachers' professional knowledge, including pedagogy, teaching methods, and educational theory.  The second part of the test covers content knowledge, either elementary content, or for secondary teachers, a subject specific content area.  A passing score on the test, clearance by the police of any criminal record, and an accredited university's recommendation that a person is prepared to work as a classroom teacher earns a regular certificate to teach from the State. 

Arizona's efforts are part of a national movement to improve the quality of teachers through assessments like the AEPA ("Quality Counts," 2000; Higher Education Reauthorization Act, 1998).  But not every district in the state or nation can find regularly certified teachers, giving rise to other policies that appear to work against the goal of increased teacher quality.  For example, in Arizona and elsewhere, attempts at improving the quality of the teaching force seem contradicted by the continuing practice of issuing emergency certification (see "Quality Counts," 2000; Olson, 2000).  Critics of hiring uncertified teachers ask whether complex, standards—based reforms can be enacted with teachers who are, to varying degrees, untrained.  Supporters of hiring uncertified teachers claim that the advantages of traditional teacher education programs are unproven, and some question, as well, whether such training is even necessary.  Stated in its simplest form as a research problem the question is: "Do students taught by teachers with emergency certification learn as much or achieve as well as students who are taught by regularly certified teachers?" An answer to this simple question would inform us whether policies designed to improve teacher quality are being undermined by the simultaneous adoption of policies that allow the use of uncertified teachers.

The dilemma associated with using uncertified teachers is not limited to Arizona where, currently, one out of six teachers are estimated to be uncertified (Go, 2002).  For example, the Chicago "Sun Times" (Rossi & Grossman, 2002) reports an audit by the Chicago Board of Education showing that 22 percent of teachers in the system's 81 probationary schools—those with the greatest academic needs and the lowest test scores—were not fully qualified to teach.  These were teachers missing what the state calls "initial'' or "standard'' certificates.  Other teachers were found with certificates, but they were teaching subjects they were not certified to teach.  In all, 900 teachers, about one of every five in Chicago's worst-performing public schools appeared unqualified to teach during the school year 2001-2002.  New York State appears to be no different.  Lankford, Loeb and Wycoff (2002) report that in a recent school year, in some New York schools, less than half the teachers held certification for the courses they taught.  These schools were invariably urban and serving the poor. 

With the passage of the No Child Left Behind Act of 2001 (HR1), federal law will require schools to have a "highly qualified" teacher in every classroom by 2005-2006.  Thus research on the effectiveness of uncertified and certified teachers takes on added significance as the designation of teachers as uncertified and certified becomes entwined with the evaluations of teachers so that the highly qualified can be distinguished from those less qualified.

The review that follows explores the difference between licensure and certification, reasons for teacher certification, the role of certification in the professionalizaton of the teaching workforce, on-the-job training for teachers, and the assessment of beginning teachers' competency.  After those topics are discused briefly, research that bears on a broad range of certification issues and teacher effectiveness is discussed in somewhat more detail.  These areas are all highly contested and interpretations of this literature are, more than usual, intertwined with the ideology of the researchers (Cochran-Smith & Fries, 2001). 

Licensure and Certification

Teacher certification, at its core, is based on the need to ensure that every public school teacher has had rigorous screening and training and been judged qualified to teach.  Certification is designed to protect the public from harm.   But there is a difference between certification and licensure.  Lawyers, cosmetologists, and physicians represent a few of the many professions that require a practitioner to hold a license in order practice their profession.  The goal of licensing tests is to set a minimum level of competency.  Professions that require licensure make it illegal for someone without a license to practice that occupation (Pyburn, 1990).  A person without a law or cosmetology license would be committing a crime if caught practicing law or working in a hair-dressing salon.  No such legal protection is afforded the public when it comes to education.  Teachers without certification are simply not allowed to use the title of "certified teacher" but there are no legal impediments for teaching without certification.  This difference between certification and licensure allows states to issue emergency certificates but not emergency licenses.  Issuance of these emergency certificates produces considerable moral difficulty.  A newspaper report on Chicago public schools makes this case dramatically (Rossi & Grossman, 2002).  The journalists document that at Howland elementary school, one of Chicago's poorest:

55 percent of teachers were "not fully certified" to teach all of their students.... That included four of six teachers in high-stakes classrooms, where kids must repeat a grade if they don't do well on annual tests. All four held substitute certificates, although two were in teacher preparation programs. Assigning uncertified substitutes to low-scoring kids who face high-stakes tests "should be illegal," said Kati Haycock, head of the Education Trust, a Washington, D.C., research and advocacy group. "That's educational malpractice." However, in Chicago, no policy governs who can teach such students.

The "Philadelphia Inquirer" wondered what the city of Philadelphia was going to do about the same problem (Hundreds of teachers, 2002).  President George W. Bush had just asked America to ensure that there was a highly qualified teacher in every classroom.  But Philadelphia has 30,000 students in classrooms where teachers are uncertified and the city cannot figure out how to solve that problem.  President Bush did not mention that while Governor of Texas, during the 1996-97 school year, he allowed 760,000 of the state's 3.8 million students to be taught by uncertified teachers.  Nor did he note that the students with uncertified teachers were found not to do as well on the state achievement tests as did students in the classes of regularly certified teachers (Students of certified, 1999).  President George W. Bush has now passed on to the nation the problem that Governor George W. Bush could not solve.

Reasons for Certification

Those who defend the process of teacher certification claim it is a necessary component in the development and maintenance of the profession of teaching, as well as the means by which the state can ensure the quality of those who enter the profession.  Wise (1994a) notes, however, that there are two methods of controlling entry into the profession, professional control and popular control. 

Professional control allows the teaching profession to monitor who becomes a teacher.  By specifying standards for certification and through various political mechanisms, the profession controls the quality of teachers who enter the profession.  When professional control is present we often see teacher input in the design of teacher certification tests.  On the other hand, popular control allows public demand to control who is placed in classrooms, with much less concern for their qualifications.  Emergency certificates to teach during times of "shortages" are an example of popular control.  Wise (1994b) advocates professional control as the primary means to allow the promotion of teaching within the economic sector.  He believes that without certification teaching becomes a trade rather than a profession. 

Among other characteristics, a profession is also defined as possessing a distinct body of knowledge and having control of the education and licensing of its members (Pratte & Rury, 1991; Burbules & Densmore, 1991).  Labaree (1992) describes professionalization as the ability to demonstrate formal knowledge and to have autonomy in the work place.  He explains that any occupational group:

...must establish that it has mastery of a formal body of knowledge that is not accessible to the layperson and that gives it special competence in carrying out a particular form of work.  In return, the group asks for a monopoly over its area of work on the grounds that only those certifiably capable should be authorized to do such work and to define appropriate practice in the area (p. 125).

Both Wise (1994b) and Roth (1994) fear that demand-based policies allowing for uncertified teachers can be devastating to the profession.  They argue that such policies are likely to reduce the quality of teaching, lower the livable wage of teachers, and change the resources that are spent on and in schools.  In effect, downgrading the importance of certification and training prevents teaching from meeting one of the criteria by which an occupational group calls itself a profession.

[A] shift in locus of preparation [from university programs to alternative certification programs] moves teaching in the direction of trade.  On-the-job training is not characteristic of a profession.  Dismissing the requirement of professional preparation and a credential prior to practice is also uncharacteristic of a profession (Roth, 1994, p. 267).

This battle over control of training is not new.  For over 150 years who certifies teachers and how that certification is to be done has been a topic of intense debate.  At all times, as might be expected, professional educators have fought to control the process, using medicine and law as their models (Angus, 2001).

On-the-job Training and Teacher Certification

To counter the argument that teachers can learn all that is required to be effective on the job, Darling-Hammond (2000), Howey and Zimpher (1994), and others argue that there is an inadequate amount of supervision and training provided to novice teachers by schools.  Principals and veteran teachers who could serve as mentors generally do not have the required skills, training, or time to provide novices with quality supervision for on-the-job training.  With few exceptions, school districts do not now have access to the additional resources needed for the training of teachers, and it is unlikely that such resources can be obtained.  Hawley (1992), articulating the views of many others, claims also that there is a body of subject-matter content and subject-matter method, as well as skills and pedagogical knowledge, that needs to be learned prior to teaching.  He and other teacher educators argue that it is unlikely that someone without training in subject matter methods could get in front of a class of students and be a successful teacher.  This group of scholars rejects the idea that effective teaching can be learned on the job.

Ordinarily, certification should assure the public that a minimal level of competency has been achieved, thereby insuring that unqualified people are not practicing the profession.   Darling-Hammond  (2000) believes the extant data supports that claim.  For example, in an analysis of state level data she found the percent of new teachers in a state who were uncertified correlated negatively with performance on six different state assessments conducted by the National Assessment of Educational Progress.  These correlations ran from –.40 to –.63.  She found equally large positive correlations for the percent of teachers in a state having regular certification and a major in the field in which they were teaching, again using six NAEP data sets.  These correlations ran between +.61 and +.80.  Even for state level data the correlations Darling-Hammond found seem to be quite large, allowing her to assert with confidence that students achieve better when they have certified teachers as instructors.

Certification and Competency Testing for Novice Teachers 

There are disputes about what should be covered on teacher certification exams because there is conflict about what is necessary for teachers to know in order for them to be effective.  Nevertheless, and despite the problem of defining these areas unambiguously, teacher certification exams currently focus on measuring basic skills, content and pedagogical knowledge (Kearns, 1984).

In 42 states, "Candidates [for certification] are required to pass one or more tests of basic skills, general knowledge, subject matter knowledge, or teaching knowledge" (Mitchell, Robinson, Plake, & Knowles, 2001, p. 44).  But critics maintain that the basic skills certification tests "measure verbal and mathematical achievement at about the 10th grade level.  And many states set their passing scores so low that virtually anyone can succeed" (Olson, 2000).  Even Minnesota, usually one of the highest scoring states in the nation on standardized tests, has a teacher basic skills test that currently passes over 99% of the applicants (Scores needed, 2002).  The Arizona basic skills assessment was also of this kind.  It had only a 1% failure rate (Crehan, Hess, Lawrence, & Sabers, 2000).  This basic skills test was abandoned, in part, because of its low failure rate and also because of adverse impact on some of Arizona's minority group candidates.  The low failure rates nationwide suggest that teacher competence testing in basic skills areas is much more about symbolic politics (the need to appear as if standards have been put in place), and a lot less about genuine and systematic attempts to upgrade the quality of the profession. 

In the area of pedagogical skills and methods, the test developers often determine what good teaching looks like based on some definition of teacher quality.  But teacher quality often is defined as having the necessary knowledge, skills, abilities and behaviors of a good teacher, a circular definition providing little guidance.   Moreover, ideas about quality change from one decade to the next, as well as from one test developer to another, and the criteria for measuring teacher quality (the knowledge, skills, and abilities a teacher possesses) is not readily agreed on from person to person (Mitchell, Robinson, Plake, & Knowles, 2001). 

The National Research Council (Mitchell, Robinson, Plake, & Knowles, 2001) reviewed past and present definitions of teacher quality and competency, finding that past definitions of teacher quality emphasized teachers' virtue.  In its more modern form teachers are still expected to be role models for students, representing high standards of personal behavior and expected to transmit worthy cultural and education values.  With no diminishment over time, it is still assumed that effective teachers possess certain personality traits, including enthusiasm, curiosity, and compassion, as well as democratic values.  And, as always, effective teachers have been thought talented in teaching the prescribed curricula, an increasing concern after states have invested heavily in the development of standards and accountability systems.  

Today, most definitions of teacher competence from which assessments follow, are the product of three organizations, namely, the National Board for Professional Teacher Standards (NBPTS), the Interstate New Teacher Assessment Support Consortium (INTASC) and the National Council for Accreditation of Teacher Education (NCATE).  The National Research Council (Mitchell, Robinson, Plake, & Knowles, 2001) notes that all three sets of standards of quality promoted by these organizations examine teaching in the light of student learning, a relatively new concern.  While a focus on learning seems obvious to parents and politicians, this is a much more problematic an issue then it appears to be.  The three organizations also acknowledge that teachers' actions or performances depend on many different kinds of knowledge as well as the dispositions to use that knowledge.  And they recognize that teachers must also work with others to support the learning and success of all students.  The standards of competence described by the three organizations relate to a teacher's commitment to students and their students' learning, requiring that teachers:

• should act on the belief that all students can learn;

• should have deep subject matter knowledge about the substance and structure of their disciplines;

• need to manage and monitor student learning, identify learning goals and choose from teaching styles to meet these goals;

• need to be reflective about their teaching, evaluate their decisions and experiences to make adjustments in their teaching;

• must be part of a larger community consisting of school staff, parents, and the broader non-parent community. 

As the National Research Council notes, the standards currently in use "...illustrate the wide range of knowledge, skills, abilities and dispositions that contemporary educators believe competent teachers must possess and demonstrate in the classroom" (Mitchell, Robinson, Plake, & Knowles, 2001, p. 31).  Given the wide range and lengthy descriptions of the knowledge, skills, abilities and dispositions that the various assessors of teachers' competence have considered necessary to begin to teach, it seems anomalous that arguments to drop teacher education or to provide emergency certificates would have any credibility at all.  Test developers find effective classroom teaching to be extremely complex, requiring the coordination of many different kinds of skills and dispositions, many of which cannot be observed directly.  It is hard to imagine that an occupation with these characteristics can be effectively learned on-the-job.

This very same occupational complexity also handicaps the developers of teacher certification testing, leading Sabers to note that, "test developers and researchers haven't done a good job of telling the public that they can't measure everything with a test" (in Crehan, et al., 2000).  The public believes that a certification exam will eliminate poor teachers from schools and in essence guarantee that teachers who pass these exams are of high quality.  But we do not yet have such tests.  At present, it is fair to say that many aspects of teaching cannot be assessed by using a multiple choice or essay exam, and if performance tests of teaching were used such testing would be prohibitively expensive (Crehan, et al, 2000). 

Validity problems with certification tests

Research on Certified Teachers and Student Achievement

Three major areas of research are salient for understanding the importance of certification.  First is the research on the effects of certification regarding teachers' content knowledge, particularly mathematics and science knowledge, as it affects student achievement.  A second area of research deals with the effects of certification regarding a teachers' pedagogical knowledge, and its effects on student achievement.  It is clear that the federal government is having troubles deciding on the relative importance of these two areas, paying lip service to the latter but more often endorsing the former.  As "Washington Post" reporter Jay Mathews notes (2002), first the Bush administration pushes through an education bill that demands a "highly qualified" teacher in every classroom.  Then the administration releases a report arguing that the nation's education schools spend too much time on classroom methodology.  Mathews points out that mixed messages are being sent to the public.  But in fact, they aren't very mixed. 

Education Secretary Roderick Paige and other Education Department officials claim that schools of education need to spend less time on pedagogical issues and spend more time worrying about whether teachers understand what they teach.  The current mantra of federal educational administrators seems to be "You can't teach what you don't know" (Mathews, 2002).  Not mentioned explicitly, but implied, is that schools of education should have little role in the training of teachers.  Secretary Paige's comments are all the more puzzling from some one who advocates evidence- based research.  His own Department of Education recently requested a review of "rigorous empirical research" on teacher preparation (Wilson, Floden & Ferrini-Mundy, 2002).  The authors of this government commissioned report concluded that subject matter knowledge is not sufficient for effective teaching to take place. 

[The studies reviewed] suggest that the subject matter preparation....prospective teachers currently receive is inadequate for teaching toward high subject matter standards, by anyone's definition.  [Without training in pedagogy] it appears that prospective teachers may have mastered basic skills but lack the deeper conceptual understanding necessary when responding to student questions and extending lessons beyond the basics (p.192).

The third area of research focuses on two sub-areas that both deal with traditional certification and alternatives to it.  One of these areas of research is on the effects of regularly certified teachers teaching in or out of their area of expertise.  In this literature in-field vs. out-of-field teaching performance is compared, such as when an English teacher is assigned to teach algebra.  Out-of-field teaching can be viewed as teaching without the appropriate certification to do so.  The second sub-area is concerned with the effects of alternatively certified teachers in comparison to traditionally certified teachers.  Present government policy has decided that alternate means to certification are appropriate, with officials claiming that:

[T]here is no evidence that lengthy preparation programs achieve [their] goals any better than streamlined programs that quickly get talented teachers into the classroom....Requiring excessive numbers of pedagogy or education theory courses acts as an unnecessary barrier for those wishing to pursue a teaching career (Mathews, 2002).

Our evaluation of this literature, reviewed in more detail below, is that there is sufficient evidence to conclude that 1) subject-matter knowledge is an important, but not sufficient, factor in a certified teacher's success with mathematics and science students in the upper grades; 2) that teachers who have training in pedagogy outperform teachers without such training; and 3) that traditionally certified teachers teaching in their area of certification outperform both certified teachers teaching out-of-field and alternatively certified teachers.  The data on these issues, however, is certainly not unequivocal, and dissenters to all these conclusions exist (see Ballou & Podgursky, 1999; Peck, 1989; Miller, McKenna, & McKenna, 1998).  We look at these literatures next.

Teacher Subject-matter Knowledge

Researchers have usually, though not always shown that having a major or minor in mathematics or science is beneficial to student achievement in those content areas.  Hawk, Coble, and Swanson (1985) provide research on that issue obliquely, by comparing in-field and out-of-field teaching; concluding that student achievement, for general mathematics as well as algebra, is greater for students who are taught by teachers certified in mathematics (in-field teachers, possessing a major or minor) than is the achievement of students taught by teachers certified in some other content area (out-of-field teachers, neither a major or minor in the area).  The researchers hypothesize that the greater success of these in-field teachers' appears to be their greater ability to successfully impart content specific knowledge to students, as compared to their out-of-field counterparts.  It is important to note that these studies compare teachers who hold a standard teaching certificate in their subject area (indicating specialized content knowledge training) with teachers who also hold a standard teaching certificate in another subject area (indicating a lack of specialized content knowledge training).  The study supports the case for certification in a content area, and suggests deficiencies can be expected among those who are teaching in areas for which they are not prepared. 

From their research Goldhaber and Brewer (1996) conclude that "in mathematics and science, teacher subject-specific training has a significant impact on student test scores in those subjects" (p. 206).  These same researchers go on to say that their results suggest that it is subject-specific training, not teacher ability that leads to such findings.  These authors believe that achievement in technical subjects can be improved by a cessation of out-of-field teaching.  The generalizability of these results to the humanities and for teachers in the primary grades is unknown.

Monk and King (1994) also evaluated subject-matter preparation and student performance.  In an earlier analysis Monk (1994) had found that there was a "positive relationship between the number of subject-related courses in a teacher's background and subsequent performance gains of these teachers students within the indicated subject area" (as cited in Monk & King, 1994, p. 36).  Continued investigation of this phenomena revealed interactions, among them, that, "low-pretest students' performance gains in mathematics were more sensitive to the mean level of their previous teachers preparation than were the high-pretest students" (p. 56).  This suggests that lower achieving students will profit more from teachers who are well prepared in their subject matter, than might better achieving students.  Thus policies that promote uncertified teachers as the instructors of the poorest and the lowest achieving students, which is the way those policies are usually realized, may be particularly harmful. 

Byrne (1983) provides a review of thirty studies that relate teachers' subject matter knowledge, measured by subject knowledge exams or coursework taken, to student achievement.  These results were contradictory.  A majority of the studies showed a positive relationship (17), while a large number (14) showed that no relationship existed.  Byrne does not provide more than a tally analysis of the studies included, which is insufficient given the capabilities of meta-analytic research.  A re-analysis using meta-analysis would be helpful.

The National Teacher Exam (NTE) was once used as a measure of subject matter knowledge and was extensively studied.  It measured both subject matter content knowledge and pedagogical knowledge.  Quirk, Witte, and Weinberg (1973) found only a single study, Lins (1946), in which NTE scores were correlated positively with students' average gain in performance from pretest to posttest.  But this finding must be reconsidered in light of the work of Strauss and Sawyer (1986).  They analyzed the determinants of average student performance on standardized exams and found that a "1% increase in teacher quality, ceteris paribus, as measured by standardized test scores [NTE], is accompanied by a 3-5 % decline in the level of failure or rate of failure of students on high school competency examinations" (p. 41).  Simply put, increased scores by teachers on the NTE exams, reflecting increased subject matter knowledge and increased pedagogical knowledge, decreased student rates of failure.  Research on the PRAXIS tests, successor to the NTE, has been conducted by Gitomer, Latham, and Ziomek (1999), and will be reported below. 

Teachers' advanced degrees are another indicator of subject matter competency.  Goldhaber and Brewer (1996), as part of the study reported above, also found that teachers' degree level is significantly related to student achievement in the area in which the degree was obtained.  However, when a general production function model is used, teachers with master's degrees appeared to be no more effective than teachers without advanced degrees.  Results varied depending on the statistical models that were used to analyze teacher effects.  Goldhaber and Brewer (2000) report that mathematics students who have teachers with bachelors or masters degrees in mathematics have higher test scores than students of teachers without these degrees.  They report, however, that there is no comparable impact of degree in science.  Fetler (1999) confirms the findings in mathematics, noting that "Schools with more experienced and more highly educated mathematics teachers tend to have higher achieving students" (p. 13).   But of course, higher achieving students have access to better schools, and thus these kinds of studies require caution when interpreting them.

Ehrenberg and Brewer (1994), however, report unambiguously that teachers' degree level does matter.  "The greater the percentage of teachers with at least a masters degree...the higher black students' scores are [on measures of mathematics, reading, and vocabulary that are associated with the High School and Beyond study]" (p. 10).  On the other hand, with a Texas sample, Rivkin, Hanushek, and Kain (2000), note just the opposite, namely, that there "is no evidence that a masters degree raises teacher effectiveness" (p. 3).  However, these researchers also report "teacher quality is a very important determinant of the quality of education" (p. 30).  Thus the conclusions in this report appear contradictory.

Kain (1998) also studied this issue.  He reports that in his earlier research, teachers accounted for at least 7% of total variation in student achievement, indicating that a set of teacher characteristics, including certification and training, affect student achievement. 

To analyze the affect of teacher degree on student achievement, Goldhaber and Brewer (2000) used data from the National Educational Longitudinal Study of 1998 (NELS: 88), which provided information about students and their teachers in 10th and 12th grades.  Students were surveyed as well as tested on one or more mathematics, science, English/writing and history exams.  The researchers found that "having a degree in education has no impact on student science test scores and, in mathematics, having a BA in education actually has a statistically significant (at the 10% level) negative impact on mathematics scores of students" (pp. 138-139).  Goldhaber and Brewer's research focused on secondary grades.  The complexity of the content taught in secondary mathematics classes is undoubtedly greater than that taught in elementary school, so advanced training in mathematics may be required to effectively transmit that content.  That same depth of subject matter knowledge may not be required for elementary teaching.

There is some support for this hypothesis, although research on the effects of advanced degrees and subject matter majors for primary grade students is scanty.  One such study, however, was published by the National Center for Educational Statistics (NCES) (as cited in Hawkins, Stancavage, & Dorsey, 1998).  These results concerned fourth-grade mathematics students who took the National Assessment of Educational Progress (NAEP).  The researchers note that "fourth-grade students who were taught by teachers with an undergraduate or graduate minor in mathematics or mathematics education did not perform better on the 1996 mathematics assessment than students whose teachers had an undergraduate or graduate major in education" (p. 12).  But the NCES report also states that, unlike fourth- grade, eighth-grade students who were taught by a teacher with a major in mathematics outperformed students taught by teachers with majors in education.

One conclusion to be drawn from these data is that a teacher's depth of knowledge in a subject matter influences students' achievement more in the upper grades than the primary grades.  But the broader conclusion might be that as the content in a subject matter area becomes more complex, teachers need a much deeper knowledge of that subject matter area to help their students learn at high levels.  It is not grade level per se, we think, but the complexity of the ideas to be taught that requires the specialized subject matter knowledge of a teacher.  Thus the claim made by TFA, that an individual with a subject matter major from an elite college who elects to teach elementary school without certification is bringing great strength to the schools, may be questioned.  It seems probable that after basic college level content is mastered, after some threshold of competency in a subject matter domain is crossed, as it is for most college majors, then pedagogical training for teaching in the elementary grades is more important to success than is content knowledge.  Support for this interpretation comes from Rowan, Correnti and Miller (2002), in their study of teacher subject matter competency in the early grades.  We look at this issue next.

Professional Knowledge/Pedagogical Content Knowledge

Ferguson and Womack (1993) found that the amount of education coursework teachers completed explained about 16% of the variance in teaching performance, as measured by supervisor evaluations; this was more variance accounted for than with teachers' content knowledge, as measured by NTE specialty scores.  This research suggests that education coursework is a strong predictor of teaching effectiveness, over and above grade point average in a teachers' major and their NTE specialty scores.  In their review of research on this same issue Ashton, Crocker, and Olejnik (1986) also found education coursework to have a significant relationship to teacher performance.

More recent research by Wenglinsky (2002), on the link between teacher quality and student performance, supports the belief that teacher inputs do influence student performance.  He notes that the greatest influence on student's achievement comes from classroom practices and the professional development that supports them.  Wenglinsky's research indicates that "regardless of the level of preparation students bring into the classroom, decisions that teachers make about classroom practices can either greatly facilitate student learning or serve as an obstacle to it" (p. 7).  That is, teacher pedagogical decisions and activities (which are separate from but not unrelated to teacher subject matter knowledge) independently make a difference in student achievement. 

Rowan and colleagues (Rowan, Correnti & Miller, in press) reached similar conclusions.  These researchers found relatively large effects on young students that could be attributed to teachers, independent of school, social class, previous achievement, and so forth.  For any given year, looking at a single score, at a single point in time, teachers accounted for 4% to 18% of the variance in student's reading and mathematics achievement.  This yielded effect sizes of .21 to .42.  Across years, looking at student growth, the effects of teachers on students were magnified.  Analysis of the teachers' effects on student growth in reading yielded effect sizes of from .77 to .88.  The teachers' effects on growth in mathematics achievement were equally impressive, yielding effect sizes of between .72 and .85.  The effect of teachers' characteristics on student achievement growth, across time, is roughly three times greater than they are on student achievement status measured at only one point in time.  When searching for which teacher characteristics make a difference, these investigators found that the most consistent predictor of young children's achievement was teacher experience.  Experience was found to be a much better predictor of student achievement than was subject matter competency.  Here again we see the relative importance of pedagogy over content knowledge in influencing the achievement of young children.

What is often not discussed in research reports connecting some teacher quality variable and student achievement is that the great bulk of a teacher's pedagogical training and understanding of beneficial classroom practices is provided in their teacher training programs.  Clearly experience matters; but that means that preparation to profit from experience must matter as well.  And that suggests that the experience gained from intensive student teaching, over a sufficient time period, might also matter.  Such experiences are provided as a matter of course in most traditional teacher certification programs, and are missing from most alternative and emergency certification programs.  Without adequate teacher training, then, emergency certified teachers and other under-certified teachers could retard student learning as they engage in teacher learning.  We examine teacher experience in more detail next.

Teacher Experience

The point of this discussion about experience is that virtually all university teacher certification programs include both early field experiences and student teaching in their curriculum to provide experience to novice teachers.  While we have little empirical evidence to determine what kinds and amounts of experience are the most beneficial, it seems likely that teacher induction programs that offer little or no experience will be deficient.  This is a criticism of the TFA program and any other programs supporting emergency or alternative certification that allow un-experienced and inexperienced teachers to become classroom instructors.  Reviewing similar literature for the Department of Education, Wilson, Floden and Ferrini-Mundy (2002) conclude that the parts of the teacher education experience that make a difference are unknown, but that "the research suggests that there is value added by teacher preparation (p. 194)."  They also state that clinical experiences and field-work, such as that provided through student teaching, are seen as powerful forces—maybe the most powerful force—in programs of teacher preparation.

Interestingly, if a state policy provides for emergency certification to teach for only a short period of time, they may do a disservice to students, since it is through experience that teachers acquire their competency.  The logic is this: It may be wrong to employ emergency certified teachers, but to dismiss them solely on the basis that they served two years, the maximum for an emergency certificate in some states, is to negate and reject how much they may have learned in that time.  On a case-by-case basis, it may be better to decide if an emergency teacher has been reflective about his or her experience and thereby learned enough to be effective.  It may compound the original error to dismiss them after a short period of time.

The review of research on content knowledge, pedagogical knowledge, and experience, given above, focuses on where these abilities and characteristics of teachers are to be learned, and in what mix, but there is no major dissent about their importance for student learning.  Wenglinsky (2002) makes this case best using data from the National Assessment of Educational Progress (NAEP) to examine the role teachers and their instructional practices play in  influencing student achievement.  He summarizes his findings as follows: "The study finds that the effects of classroom practices, when added to those of other teacher characteristics, are comparable in size to those of student background, suggesting that teachers can contribute as much to student learning as the students themselves."

Appropriate Assignment of Certified Teachers

In some curricula areas such as bilingual and special education, science, and mathematics, there is a teacher shortage.  This appears to be caused by increased student enrollments, retirements due to the  "graying" of the teaching profession, increased turnover by teachers due to increased difficulties in carrying out their jobs, and the increased remuneration for mathematicians and scientists for work in other fields.  If faced with difficulty filling positions, school boards and administrators think of three solutions: they hire less qualified teachers, they assign teachers trained in another subject area to teach in the understaffed areas, or they make extensive use of substitute teachers. 

There are many problems with the teacher shortage explanation for out-of-field teaching.  First of all, this explanation does not provide an answer for why large numbers of out-of-field teachers exist in fields like English and social studies, two areas of teaching that have long had a surplus of teachers.  Another problem with this explanation is that it has only been within the past few years that schools have had trouble filling teaching openings, while the problem of out-of-field teaching has been with us for a significantly longer period of time.  Finally, the teacher shortage is based on the assumption that there are too few teacher candidates.  But in fact, the biggest difficulty is that teachers are choosing to leave their profession for other jobs (Ingersoll, 1999a).  Ingersoll (1999a) comments, "Out-of-field teaching is common, I believe, because it is not only legal but also more convenient, less expensive and less time consuming than the alternatives" (p. 7). 

Ingersoll (1996) evaluated data from the School and Staffing Survey to determine the proportion of teachers who teach out-of-field.  He provides data indicating that one-fifth of public school students enrolled in 7th-12th grade English courses are taught by teachers who did not have at least a minor in English or some other closely related field.  Of the students enrolled in 7th-12th grade mathematics courses, more than a quarter are taught by teachers without a minor in mathematics or mathematics education.  The results are less drastic in other areas of course work.  In Arizona, 35% of teachers in English, 39% of teachers in Math, 35% of teachers in Social Studies and 27% of teachers in Science were assigned to teach secondary courses without a major in that subject area ("Quality Counts," 2000). 

Research also supports the belief that out-of-field teaching is related to levels of school poverty (Ingersoll, 1996; 1999b; Haycock, 2001).  Ingersoll (1996) reports, " in no fields did high-poverty schools have less out-of-field teaching than did low poverty schools, while in several fields, students in high poverty schools received distinctively more out-of-field teaching than in low poverty schools" (p. 5).  This trend is similar for students who are placed within different educational tracks in their courses.  High track students are exposed to less out-of-field teaching than low track students (Goldhaber & Brewer, 2000; Rivkin, et al, 2000, Ingersoll, 1996), while "minority and poor students are disproportionately placed in lower track and lower achievement courses, [which] critics claim are taught by the least qualified" (Ingersoll, 1996, p. 1).   Darling-Hammond (1997b) reports that in the most heavily minority schools and inner cities less than 50% of the teachers in mathematics and science are licensed and have a degree in the subject they teach. Darling-Hammond remonstrates that throughout the country we have the least qualified teachers teaching the most disadvantaged students, while the most qualified teachers are teaching the most advantaged students.

At the secondary level the relationship between in-field teaching and student achievement is stated forcefully by Hawk, Coble, and Swanson (1985).  They conclude that:

In field certified math teachers know more math and show evidence of using more effective teaching practices than their out-of-field counterparts.  Further, and most important, students of in field certified math teachers achieve at a higher level than do students taught by out-of-field teachers (p. 15).

In short, a certified teacher teaching in the field for which they were prepared performs better than when assigned to areas for which they were not prepared.  Preparation matters.

Alternative Routes to Teaching

Advocates of alternative certification, however, claim that they provide teachers for urban and rural schools and in specific shortage areas, i.e., mathematics and science.  Zumwalt (1991) summarizes research on several alternative certification programs and reports that they do attract teachers who are more willing to work in rural or urban poor districts than traditionally trained teachers.  McKibbin and Ray (1994) also report that alternative certification programs attract people with subject matter majors like mathematics and science who are interested in teaching, but not interested in traditional teacher certification. 

Alternative certification is also seen as a cost-effective way to train people who did not or will not enroll in conventional undergraduate or graduate education programs.  Such programs are cheaper (Zumwalt, 1991), as might be expected from programs that are shorter in duration and provide less instruction, supervision and assessment of their students.

Proponents of alternative certification claim that these programs attract better quality candidates who are more academically able than those who attend traditional certification programs (Kanstroom & Finn, 1999).  Participants of these programs are generally people who have majored in traditional academic subjects rather than education.  It is a major assumption of alternative programs that subject matter content knowledge is more important to teaching than is education related coursework (Jelmberg, 1996).  But some studies show that the teachers in alternative routes to certification have high drop-out rates from both the programs of instruction and from actual teaching. (Wilson, Floden & Ferrini-Mundy, 2002).  Other studies show that alternative certification recruits in mathematics and science have lower grade point averages than recruits in traditional teacher education programs (Stoddart, 1992).  Moreover, to date, these alternatively certified teachers have not demonstrated strong skills in their content area.  Furthermore, teachers from alternative routes to education, including TFA teachers, when compared to those trained in more traditional teacher education programs, report many more problems with their preparation programs.  For example, on 39 of 40 different questions the TFA teachers rated their preparation more poorly than did those who were trained in more traditional programs. The self-confidence and sense of efficacy of those prepared in traditional programs was higher than for those who came to teaching through alternative programs (Darling-Hammond, Chung, & Frelow, 2002).

When these facts about alternative routes to teaching are added to research that debunks the belief that subject matter knowledge is more important than education related coursework (Darling-Hammond, 2000; Gitomer, Latham, and Ziomek, 1999; Monk and King, 1994; Wilson, Floden & Ferrini-Mundy, 2002), we see evidence of an educational policy that must be seriously questioned.

Proponents of alternative certification also make the claim that traditional certification programs attract mostly twenty-something, white women, a problem because the students of the public schools in urban areas are not often white, and some commonality and similarity in life experience is probably a better background for teaching positions in these more difficult settings.  Proponents of alternative certification correctly note that they often attract a more diverse group of candidates, specifically men, older adults, minorities and retried military personnel (Bradshaw & Hawk, 1996; Eifler & Potthoff, 1998; Hawley, 1992; Houston, Marshall, & McDavid, 1993; Keltner, 1994; Kennedy, 1991; Kwiatkowski, 1999); Natriello & Zumwalt, 1993; MacDonald, Manning, & Gable, 1994; and Stoddart, 1993). Wendy Kopp (1994), Teach for America founder, notes that short alternative certification programs allow young adults who are unsure of their career options an opportunity to serve students.

Alternative certification may actually be an improvement over simple emergency certification, which allows almost anyone with a bachelors degree to teach without any preparation to speak of.  But some researchers (Bradshaw & Hawk, 1996; Berry, 2001) are critical of the level of professional knowledge demonstrated by alternatively certified teachers.  Alternatively certified teachers tend to have a limited view of curriculum and a lack of understanding of student ability as well as motivation; they experience difficulty translating content knowledge into meaningful information for students to understand; they are less effective planners of instruction; and they tend not to learn about teaching through their experiences.  Research is also critical of the supervision and mentoring support that is given to preservice teachers in most alternative certification programs (Smith, 1991; Darling-Hammond, 1992; McKibbin & Ray, 1994; Bradshaw & Hawk, 1996; Jelmberg, 1996; Miles-Nixon & Holloway, 1997; Berry, 2001).  It appears likely, from the extant research and criticism, that poorly run and short alternative certification programs, particularly those that do not provide much classroom experience and supervision, may actually not be any better than simply hiring emergency certified teachers with no teacher education experiences. 

Teach for America

Four separate evaluations found that TFA's training program did not prepare candidates to succeed with students, despite the noticeable intelligence and enthusiasm of many of the recruits.  Most criticism of a corps member's teaching behavior (classroom management was the greatest area of concern, followed by insufficient knowledge of the fundamentals of teaching and learning) was qualified by the cooperating teachers' perceptions of limitations of the program in providing the corps member with adequate practice or theory to be successful (Darling-Hammond, 1997a, p. 310).

From an interview study by Stevens and Dial (1993), TFA teachers apparently decide to teach because they like working with children; they didn't have other options; and they felt that TFA was their best alternative given their "circumstances and indecisiveness at the time" (p. 70).

Jonathan Schorr (1993), a former TFA teacher, describes the inadequate training and preparation that he and other TFA teachers received prior to being placed into schools.  He notes, "just eight weeks of training ... is not enough for teachers" (p. 316).  Schorr admits, "I was not a successful teacher, and the loss to the students was real and large" (p. 318).  Schorr offers the first-hand experience that makes Darling-Hammond (1994; 1997a; 2001) quite critical of TFA, specifically due to the program's limited training of candidates, lack of evaluation, and the fact that such a program perpetuates the placement of poorly trained teachers with the most needy students in the nation. 

Raymond, Fletcher, and Luque (2001) conducting research for the Center for Research in Education Outcomes (CREDO), released a report evaluating the Teach for American program in Texas.  The report compares scores on the Texas Assessment of Academic Skills (TAAS) of students taught by TFA teachers and non-TFA teachers, and lauded the performance of TFA teachers.  However, the National Commission on Teaching and America's Future released a response outlining serious concerns with the research (see Darling-Hammond, 2001 and 2002).  The most important of the criticisms is that the performance of the TFA teachers was never compared to the performance of regularly certified teachers.  The comparison used to assess the TFA teachers was other uncertified teachers, some of whom didn't even have four-year college degrees. 

It should also be noted that when we tried to access the data for this report, we were informed from both the primary researcher and the Texas school district responsible for the data that it was not available for independent review.  We were told that the data was not the property of the researchers who reported the study, nor did it belong to the district, and that neither had a complete data set to provide for independent analysis.  In our opinion, therefore, it is appropriate to regard this report as irrelevant, given that the comparison used to assess TFA teachers was faulty, the data are not available for verification or replication by other scientists, and the report has not been published in a peer-reviewed journal.

Traditional Teacher Certification

The research martialed in support of prepared and certified teachers includes research demonstrating that teachers hired without preparation or only partial training experience difficulty meeting the needs of the students in their classrooms.  Such individuals have more difficulties than fully prepared teachers do in accomplishing their day-to-day job requirements (Darling-Hammond, 1997a).  Prospective teachers apparently perform better if they have completed a teacher preparation program focused on content knowledge, pedagogical course work (including learning theories, developmental theories, theories of motivation and issues of student assessment) and practice teaching.  Although variations in the philosophy, implementation and quality of teacher education programs are enormous, research nevertheless suggests that many versions of this form of preparation are successful in providing adequately trained teachers for the complexity of classroom instruction (Ashton & Crocker, 1987; Darling-Hammond, 1992; Wilson, Floden, & Ferrini-Mundy, 2002).

McDiarmid and Wilson's (1991) research is relevant to this point.  They demonstrated that subject matter knowledge is not sufficient to prepare teachers for teaching the concepts in these fields to students.  They did this by evaluating mathematics majors in alternative certification programs that stressed subject matter knowledge and found that those teachers had strongly held misconceptions about the subject matter and the appropriate ways to teach the content.  Their results indicated that these teachers were unable to integrate their subject matter knowledge with teaching practices to allow for effective instruction.  In effect, because they were lacking in education coursework, they were unable to provide the appropriate instruction to students.

Another aspect of good quality certification programs is that they provide experiences for the preservice teachers in classrooms both under direct supervision and with continued mentoring.  Darling-Hammond (1992) notes that skills need to be learned in context, where they can be practiced under supervision.  The student teaching experience allows the preservice teacher to transform information from coursework in order to learn its character in the context of the real world of teaching in classrooms.  Jelmberg (1996), cited above, compared traditionally certified teachers with alternatively certified teachers' performance based on their experience.  His results showed that experienced teachers from traditional certification programs are rated higher on instructional skills and planning by their principals, and perform better, than did experienced teachers who came from alternative certification programs.

Emergency certification

Emergency certificates are issued to prospective teachers who have met some, but not all of the requirements for state certification.  Minimum requirements are often a bachelor's degree and a passing score on a certification exam.  Emergency certificates are issued for a limited time period, usually one to two years.  Some states allow for these to be renewed, while others states issue a one time only, nonrenewable certificate.  In 1998 data about certification waiver rates were available from 39 states (U.S. Department of Education, 1999).  Sixteen of the 39 states had waiver rates greater than 2% of their teaching population, eight had rates over 5% while some had rates as high as 17%.

Emergency certified teachers are more likely to be hired in already low performing schools, schools that serve low SES students, schools in rural and inner city areas, and for positions that are hard to fill (Darling-Hammond, 1997a; Darling-Hammond, 2001; Ingersoll, 2001; Gitomer, Latham, & Ziomek, 1999).  Since high poverty schools are more likely to have high rates of out-of-field teaching as well as more emergency certified teachers, it may that some states are failing to provide the "adequate education" that most state constitutions require (Ingersoll, 2001; Hirsh, Koppich & Knapp, 2001; Rivkin, et. al., 2000).  And now that federal government has waded in on this issue, requiring a highly qualified teacher in every classroom, the competency of teachers with emergency certification is sure to be questioned further.

Emergency certification is justified on the basis of three arguments.  First, it is argued, that there is a teacher shortage requiring that states emergency certify teachers to provide enough teachers for every classroom.  But the shortage may not be as severe as many claim (Feistritzer, 1994; Ingersoll, 1997, 2001; Ballou, 1996; Hirsh, Koppich, & Knapp, 2001; Hardy, 1998).  The National Center for Education Statistics indicates that the teaching force will grow to more than 3 million teachers by the year 2008 (US Department of Education, 1999).  But it is a misconception that colleges of education will need to train millions of new teachers to meet the needs of school districts.  Darling-Hammond (cited in Hardy, 1998) believes that this potential shortage is not universal, claiming that "there are districts that experience difficulty hiring qualified teachers, but overall, we have a surplus of teachers" (p.20).  The teacher shortages are seen in subject areas like mathematics and science; in the service areas for special needs populations, such as special education and bilingual education; and shortages exists in rural areas and in inner city school districts (Wayne, 2000; Natriello & Zumwalt, 1993; Hardy, 1998; Hirsch, Koppich, & Knapp, 2001).  

The projected teacher shortage is also based on assumptions of increased student enrollments and an aging workforce. But these assumptions have similarly been questioned.  Research evaluating the Survey of Recent College Graduates (Ballou, 1996) has shown that:

In every year there were at least twice as many [qualified] applicants as there were persons hired in full-time public school positions.  Far from indicating that the nation faces a teacher shortage, these data show that the teacher labor market as a whole has been in a chronic state of excess supply, though shortages may arise in specific locations and subject areas (p. 101).

The imbalance of teacher supply and demand at the root of school staffing problems is neither synonymous with, or primarily due to, teacher shortages in the technical sense of a deficit in the quantity of qualified candidates.  Rather than insufficient supply, the data indicate that school staffing problems are primarily due to excess demand, resulting from a 'revolving door'—where large numbers of teachers depart their jobs for reasons other than retirement.  Thus the solution...does not primarily lie in increasing supply, but rather in decreasing demand (p. 501).

A second argument for emergency certification is that there are many people who would teach, but do not, because standard certification requirements prohibit them from doing so.  Thus opponents of traditional teacher education programs call for the removal of certification requirements, claiming that there is no "special body of knowledge" that teachers need to know in order to be successful.  These advocates for the abolition of requirements claim that what needs to be learned by new teachers can be learned in the first year of teaching (Roth, 1994; Kanstroom & Finn, 1999; Finn & Kanstroom, 2000).  In fact, anecdotal evidence claiming that teachers learn more in the first year of teaching than they do in their education courses is quite strong.  Armed with this knowledge, it is then argued that a person who holds a college degree, in possession of some level of content knowledge, and with some limited experience teaching youth, is competent enough to begin to teach.  Such beliefs drive the movement against certification despite research that argues against this position (McDiarmid & Wilson, 1991).

Traditional certification programs are rejected by many adults who would be interested in teaching as a second career but who will not or cannot engage in time consuming and expensive regular teacher certification programs.  Proponents of alternative certification believe that these adults, called career transitioners, have skills that they have learned in their other employment that could be used to teach children (MacDonald, Manning, & Gable, 1994).  Additionally, some believe that adults have unique life skills and experiences that can be useful to students (Zumwalt, 1991).  Research does indicate that alternative certification programs attract an older and more diverse population, through their more flexible schedules, less stringent requirements, and so forth; however, it is unclear from this research that certification needs to be waived in order to recruit a more diverse teaching population with many life skills and employment experiences (Natriello & Zumwalt, 1993; Bradshaw & Hawk, 1996; Hawley, 1992).

The third argument for emergency certification makes use of the long-standing lack of confidence by state officials and the general public in the quality of the teachers who graduate from colleges of education.  Too often colleges of education are perceived to attract less able students, thus producing under qualified teachers.  This is simply not true.  Research supports the assertion that the academic quality of students entering colleges of education is quite good.  For example, Gitomer, Latham, and Ziomek (1999) showed in their analysis of ACT and SAT scores that students from colleges of education were as academically skilled as students with other college majors.  They also reported that traditionally certified teachers have the highest passing rates on certification exams (PRAXIS I and II) compared to alternatively certified and emergency certified teachers, even though they appeared to be similar in initial achievement, based on SAT scores. They concluded that traditional certification (having training in teaching methods, pedagogy and practice in teaching) makes a difference on licensure.  They attribute the better performance of traditionally certified teachers to the training and instruction, provided by colleges of education.

Summary of Related Research

With regard to experience through teacher education course work and by means of learning on the job, the research suggests that student achievement is affected in positive ways.  The powerful effects of content knowledge, pedagogical knowledge, and experience, contribute to success in teaching in one's own field.  When teaching out-of-field, such knowledge and skill is of less use and teacher effectiveness is compromised. 

When we examined alternative routes to teaching we found them to be quite variable.  Still, in comparison to traditional teacher education programs (that are also quite variable) the alternative programs take less time, are less costly, attract more diverse students, but also record higher drop-out rates.  Because they usually take much less time, alternative programs may suffer from under-preparing students for the classroom, a problem compounded by the lack of supervision and support given by the hiring districts.  Teach for America, as one alternative route to teaching, appears to suffer from the ordinary and typical problems inherent in the designs of such programs.  While criticisms abound, there is a substantial body of literature suggesting that traditional teacher education programs, warts and all, seems to provide more competent appearing teachers whose students perform better.

  The research suggests that emergency certified teachers are probably the least prepared to do well.  Unfortunately, such teachers are typically hired into districts with the hardest to teach students.

After reviewing this literature it seemed as if the conditions necessitating out-of-field teaching and the hiring of alternate and emergency certified teachers could easily be modified, eliminating the need for these practices to exist.  In the past, however, neither politicians nor school boards had the will to do that.  But under the new federal No Child left behind Act of 2001, school districts will have to have a "highly qualified" teacher in every classroom or loose federal funding.  It will be an interesting few years as ways to define "highly qualified" and the related term "competent" are worked on to meet the letter of the law. The research in these areas is also likely to be reinterpreted in light of that law.  In the near future, surely both the definitions of, and the research associated with the idea of teacher quality, will find their ways into courtrooms of America.  This study may help the courts in thinking about what it means to have a highly qualified teacher in the classroom.

Research Design

Sample 

Among the under-certified teachers in this study are some from an alternative route to traditional teacher training, the program "Teach for America (TFA)."  Teach for America is a popular alternative certification program.  Its mission calls for placing energetic, bright, but unqualified teachers into poor, urban school districts (Darling-Hammond, 1994).  The instructional effectiveness of under-certified teachers in general, and the TFA teachers in particular, is of considerable interest to the policy community.

District Selection.   Arizona school district superintendents listed by the department of education as participating in the Federal Teacher Shortage Loan Deferment Program (Arizona Department of Education, http://ade.az.gov/researchpolicy/ts) were invited to participate in this study.  This federal program requires the State Department of Education to generate a ranking of school districts in the state by the percent of under-certified teachers in each district.  This list of school districts provided a convenient population from which to obtain a sample.  Of the school districts receiving a request to participate (N=24 for the 1998-1999 dataset, N=12 for the 1999-2000 dataset) five school districts responded positively.  These five school districts represent 20.8% and 16.6%, respectively, of the school districts designated by the state as having severe teacher shortages.  All five school districts chosen for this study were included in the Department of Education's 1998-1999 classification of school districts.  Only two of these school districts were also included in the Department of Education's 1999-2000 classification of school districts with severe shortages.  All of the participating school districts shared similar characteristics.  They all serve inner city student populations, largely minority, and all participate in the Teach for America (TFA) program.  All the schools in these districts have difficulty filling teaching positions.

Methods of data collection.  The five participating school districts provided lists of new hires for the 1998-1999 and 1999-2000 school years.  In some cases this list contained information about the teachers' certification status, while in other cases further research was necessary to obtain this information.  We were granted permission to access individual teacher personnel files in order to collect the necessary data on the school where the teacher was employed, the grade level taught, the teachers' certification status, their highest degree earned, the date and institution where there degree was earned, their age (determined from year of their birth), and teaching experience. 

Of the teachers included in the dataset, 64% had no prior teaching experience, a judgment based on their hire date, employment history, resume and application. The majority of newly hired teachers were recent graduates from college.  Some of the new teachers, however, had delayed their entrance into teaching for many years after graduation, but they had no prior teaching experience indicated in their personnel files.  Teachers were removed from the sample if they taught kindergarten, first grade, art, music or special education, grades and subjects not assessed by the Stanford Nine (SAT 9).

The assessment departments of each school district provided test scores aggregated at the classroom level.  Included were the teachers' class totals as raw scores, scaled scores, grade equivalent, national percentile rank, stanine, national normal curve equivalent, as well as class percentile rank and class stanine.  Individual student scores were not needed or provided.  Additionally, state level SAT 9 data was later obtained from the Research and Policy division of the State Department of Education in order to confirm the accuracy of the SAT 9 data collected from the school districts.  State level data was aggregated, by teacher name, for each of the school districts.  In the event of a discrepancy between the two sets of data, we opted to use the state level data.  This discrepancy occurred with only one of the school districts data files.

Matching Procedure

In the five districts studied, 293 teachers were hired in 1998-1999 and 1999-2000 who met the requirements for inclusion in this study.  In order to participate teachers personnel files were required to contain the demographic data necessary for analysis, as well as classroom level SAT 9 scores.  Teachers' data files were matched using SPSS procedures to sort the files.  Teachers in each district were first categorized on the basis of their certification status, under-certified teachers (labeled emergency, provisional or temporary certified teachers by their districts) constituted one group, certified teachers made up the other group.  Teachers from each group were then matched based on grade level and highest degree attained.  Teachers for whom no matches could be found were removed from the analysis. 

Matches were made using the following rules: 1) matches were first made within the same school, 2) matches were made within the same school district, and 3) matches were made between similar school districts.  The first and second matching rules serve to minimize exogenous variables associated with student achievement scores, e.g., socio-economic status, school characteristics, curriculum, etc.  It is assumed that teachers in the same school teach similar students, an imperfect but reasonable assumption.  The identical assumption can be made about schools within the same district boundaries, since Arizona school district boundaries are based on relatively homogenous geographic areas.  Conversations with district personnel, in the course of collecting the data at the district offices, provided no evidence that the certified or under-certified teachers were "tracked" in any way.  The assignment of teachers to schools, and to classrooms within schools, appears to have been unbiased.  Similarly, we have no reason to believe that class size or student ability was different in any way for the certified or under-certified teachers in our sample.

The cross-district matching of teachers, however, is more problematic than the within district matching.  We made these matches based on the "sameness" of the two districts.  Sameness was determined using data collected from the Education Finance Statistics Center, a subdivision of the National Center for Education Statistics (U.S. Department of Education, 2001).  Using public school district financial records for the latest year available, 1996-1997, data about each of the participating school districts were collected for: student teacher ratio, administrative ratio, median income, percent of householders with high school graduates, percent of non-white children, percent of limited English proficiency, and percent of children in poverty.  School districts that shared similar characteristics were matched to one another.  This procedure matched the Osborn Elementary School District to the Creighton Elementary School District, and matched the Roosevelt Elementary School District to both the Nogales Elementary School District and the Murphy Elementary School District.  These are not pseudonyms, but the actual names of the Arizona school districts that generously helped us in this study.

Synthesizing data from the various state departments of education, GreatSchools.net also provided data that were used to match schools having similar characteristics.  Sameness matches were also made based on: AIMS reading and math scores, SAT 9 reading and math scores, teacher experience, percent of teachers with masters degrees, attendance rate, open enrollment, percent free and reduced lunch, and ethnic breakdown within the school district.  These data support the matches that were made using the NCES finance data with the exception that the Murphy School District was found to more closely match the Creighton School District.  Teacher matching across districts, therefore, was accomplished by finding similar school districts serving similar student populations with similar economic bases.  The assertion that teachers in these different school districts are sensibly matched is well substantiated, but we acknowledge that the nature of the matching procedures used constitutes a potential flaw in studies such as this.  Random assignment of under-certified and certified teachers to classes within districts would have provided a stronger design, but this was not possible.  It is important to note, however, that the matches in this study were made with out any knowledge of the teachers' class scores on the SAT 9.

Descriptive analyses were conducted on the complete sample of teachers in the data set (N= 293).  These results are provided in Table 1.

Note: N = 293 

From this population, teachers for whom no matches were found were removed from the analysis.  The initial matching procedures produced 116 pairs of teachers (N=232 individual teachers) out of the total teacher pool of 293, thereby using 79% of the original dataset.  We undertook analyses of the quality of the matching procedures that were used.  These are reported on, below, and each analysis is based either on the entire sample drawn (N = 293), or on the pairs that were created by the matching procedure that was used (N = 232).

However, the process of matching produced 28 mismatched pairs consisting of teachers who did not share class score data for the same test administrations.  This occurred, for example, when one teacher in a pair had scores for 1998 and the other teacher had scores for 1999.   Additional matches were then made based on all of the above matching rules, but eliminating cross year matches.  This finally resulted in N=109 matched pairs, using 74% of the original data set.  The appendix to this study contains descriptive information about the 109 pairs of teachers who comprise the sample for this study. The data are also available as a Microsoft Excell spreadsheet..

Matching in primary schools across grades 3-8, however, created a problem inherent in archival studies such as this one.  To design this study sensibly we needed teachers of self-contained classes.  If departmentalization (more than one teacher working with the class) were occurring we would have problems inferring a teacher's affect on student learning.  But we have no knowledge of what went on in every school at these upper grade levels.  We were told, however, that these schools used little departmentalization, and that the teachers for whom we had files were the classroom teachers of record for the district and the state.  The 218 teachers in our sample were, therefore, the teachers designated by administrators as those responsible for their student 's achievements on the SAT 9 tests.  Since these were the responsible teachers we included all the matches from grades 3 to 8.  Nevertheless, because we worried about the issue of departmentalization in the upper two grade levels, we ran separate analyses.  One set of analyses was done with the full sample of 109 pairs of certified and under-certified teachers, and another set of analyses was done with a reduced sample, eliminating all the pairs of teachers in the 7th and 8th grade.  The appendix to this report describes all 109 pairs of teachers by grade level and thus identifies which pairs were eliminated from the second analysis.  In the discussion that follows, when the second analysis using the smaller sample of pairs of teachers from grades 3-6 produces results different from the analysis of the entire sample, we will note these differences.

Instruments

The Stanford Achievement Test, Ninth Edition (SAT 9), a nationally norm referenced standardized test is used by all districts in the state of Arizona and was used, as well, in this study.  The test assesses student achievement in reading, math and language arts (Harcourt Brace, 1997).  The SAT 9 is believed by the State Department of Education to relate to Arizona academic standards, which teachers use as a guide to instruction.  It is claimed that The SAT 9 tests between 70-80% of the material outlined in the state's academic standards (ADE personal communication, 2001).  This relationship is strongest for the reading and mathematics subtests, and in grades 2 through 8.  The language subtest of the SAT 9 is not as well related to state standards because it does not require a writing sample of students, an ability that is promoted in the standards.  The analysis of the State Department of Education suggests that the SAT 9 is a reasonable indicator of student achievement, perhaps more for reading and mathematics than for language.  Furthermore, in Arizona the test is often used as an indicator of teacher and school effectiveness.

Scoring.  Once teachers were matched to one another their Stanford Achievement Test, Ninth Edition (SAT 9) scores were aggregated at the class level.  For each pair of teachers their mean National Normal Curve Equivalent (NCE) scores for SAT 9 reading, math and language were analyzed.  The NCE scores are a type of normalized standard score resulting form the division of the normal curve into 99 equal units.  This score is traditionally used for research purposes, enabling researchers to interpret differences in NCEs more readily because of the equal-interval nature of the NCE scores.  Differences between NCEs obtained by different groups have the same meaning regardless of what part of the scale is referenced.  For the purposes of this study individual student scores were not collected and thus cannot be reported.

Analyses of Matching Procedures

The credibility of the matching procedure for pairing uncertified and certified teachers is important in the interpretation of the results of this study.  Therefore, we undertook some analyses to explore that issue.  We began by looking at the similarity of the SAT 9 test scores in each school and district to determine their comparability to each other, a check on the level of "sameness" of each school and district to one another.  These analyses were conducted using NCE scores on the mathematics, reading and language sub-tests of the SAT 9, for both the 1998-1999 and 1999-2000 data sets.  Alpha levels were set at p= .05.

Matching Analysis 1.  To answer the question about whether students' test scores on the SAT 9 are different as a function of which school they attend, a one-way analysis of variance (ANOVA) was conducted on the entire data set (N=293) to evaluate the relationship between the school assignments and student achievement scores.  The independent variable was teachers' school assignment; the dependent variable was mean student achievement scores of these teachers as measured in Normal Curve Equivalents (NCE) for reading, mathematics and language in 1998-1999 and 1999-2000.  The ANOVA was not significant with the exception of mathematics scores in the 1999 sample.  These results are provided in Table 2.  The results indicate that, overall, the schools from whom teachers in the sample came showed no statistically significant differences from each other in terms of mean NCE scores on the SAT 9, except for the special case of mathematics scores in the 1999 sample, F (41,190) = 1.65, p = .01.  It is unlikely that there are inherent differences in the schools that could bias the findings of subsequent analyses.