Read EOR_Voc_McRel.indd text version

Web Version

Final Evaluation Report A Study of the Effects of Harcourt Achieve's

Elements of Reading: Vocabulary

December 2005

Submitted By: Helen Apthorp, Ph.D. Mid-continent Research for Education and Learning 2550 S. Parker Road, Suite 500 Aurora, CO 80014

This page left blank intentionally.

Final Evaluation Report A Study of the Effects of Harcourt Achieve's

Elements of Reading: Vocabulary

Table of Contents

Acknowledgments .........................................................................................................................iv Introduction .................................................................................................................................... 1 Evaluation Approach and Design ................................................................................................. 1 Method ............................................................................................................................................ 1 Supplemental Vocabulary Product .............................................................................. 1 Study Measures ............................................................................................................ 2 Study Procedures .......................................................................................................... 4 Settings ......................................................................................................................... 4 Participants ................................................................................................................... 5 Results ............................................................................................................................................ 6 Instructional Contexts.................................................................................................. 6 Student Performance .................................................................................................... 8 Level of Implementation ............................................................................................ 11 Conclusion .................................................................................................................................... 11 References .................................................................................................................................... 12 Appendix A Grade 3 Student Demographics by Site................................................................. 13 Appendix B ERDA Subtest Reliability Coefficients for Grade 3 .............................................. 14 Appendix C Multivariate Analysis Results for GMRT Vocabulary and Comprehension Extended Scale Scores ................................................................. 15

List of Figures and Tables Figure 1. Frequency of student engagement in vocabulary development activities in treatment and control classrooms (seven teachers/classrooms per group)............ 7 Figure 2. ERDA Vocabulary Composite by time and Group. ...................................................... 9 Figure 3. ERDA Word Identification by time and group. ............................................................ 9 Figure 4. Mean GMRT Vocabulary ESS for treatment and control.......................................... 10 Table 1. Defining Criteria for Levels of EOR Vocabulary Implementation ............................... 3 Table 2. Early Reading Diagnostic Assessment (ERDA) Subtests ............................................. 4 Table 3. Relating Level of Implementation and Student Performance .................................... 11

A Study of the Effects of Harcourt Achieve's Elements of Reading: Vocabulary

iii

Acknowledgments

This evaluation study could not have been possible without the close collaboration between McREL, Harcourt Achieve, and all of the study participants in the schools. I especially want to thank the site coordinators and teachers who responded consistently and with good humor to McREL's varied requests for data. I would be remiss if I did not acknowledge the ongoing support from members of the Harcourt Achieve staff who worked closely with us and provided continued support and feedback throughout the study process. Also, I gratefully acknowledge the contributions of colleagues and staff at McREL, including the generous help of McREL's Research and Evaluation Program Coordinator, Robyn Alsop, two of McREL's Research Associates, Carolyn Woempner and Becky Van Buhler. Several research and evaluation colleagues provided guidance, tips, and thoughtful reviews, including Dr. Sheila Arens, Dr. Zoe Barley, Dr. LeAnn Gamache, and Dr. Kerry Englert. I am grateful to Dr. Stephanie Wilkerson-Baird, who developed the project's design and evaluation plan.

iv

Final Evaluation Report

Introduction

Increasing vocabulary knowledge is a basic part of education, both as a means and as an end. (Nagy, 1988, p. 1) The ways for teachers to enhance vocabulary development are expanding. Word lists include more sophisticated and fun words. The words are introduced in good literature. Test-like workbooks are less common, and instead, students study new words with graphic organizers, drawing, pantomimes and/or oral and written retellings of related experiences. Context-rich vocabulary instruction is interesting and engaging, but how does it actually work? How do teachers incorporate context-rich vocabulary instruction in their practice, and to what extent is it effective for enhancing student vocabulary development? In Elements of Reading (EOR) Vocabulary, Harcourt Achieve presents a supplemental product that encourages children to take pleasure in language and to repeatedly hear and use new vocabulary in a variety of contexts (Jordan, 2003). Authored by Isabel L. Beck and Margaret B. McKeown, EOR Vocabulary engages students' oral competence to develop sophisticated vocabulary repertoires for academic success. Mid-continent Research for Education and Learning (McREL) provided Harcourt Achieve with independent evaluation services designed to assess the effectiveness of EOR Vocabulary. McREL's evaluation employed an experimental, randomized control trial to measure the impact of teachers' use of this product on student's vocabulary and reading achievement. This report presents findings regarding the impact and implementation of EOR Vocabulary as a supplemental product used by third-grade classroom teachers during the 2004­ 2005 school year. In addition, McREL's evaluation examined how teachers viewed the utility of EOR Vocabulary, integrated it with their core reading program, and leveraged its value through their own professional wisdom and experience.

Evaluation Approach and Design

To examine the effectiveness of EOR Vocabulary, McREL's evaluation used an experimental design, randomly assigning Grade 3 teachers to treatment or control, with a combination of quantitative and qualitative methods. Combining a randomized control trial with quantitative and qualitative data collection and analysis allows for a full understanding not only of how the product is implemented but also why and how it might influence student achievement. McREL's approach recognized the need to collect information from multiple sources in order to ensure the credibility of findings and to bring important discrepancies to light. The evaluation was designed to address the following questions: 1. What is the effect of using EOR Vocabulary as a supplemental product on student achievement? 2. Do students of teachers who use EOR Vocabulary perform better in vocabulary knowledge and reading than students whose teachers do not supplement their basal curriculum? 3. How effective is this EOR Vocabulary among different subgroups of students? 4. How do teachers integrate into and/or supplement their reading/language arts instruction with EOR Vocabulary?

Method

This study examined the effect that teacher use of EOR Vocabulary had on third-grade students' vocabulary and reading achievement. The implementation period for this study was the 2004­ 2005 school year. This section provides a description of the methods employed in the study, including the EOR Vocabulary product, measures, procedures, settings, and participants.

Supplemental Vocabulary Product

The EOR Vocabulary supplemental product was developed specifically to meet, and in direct response to, the challenge of the No Child Left

A Study of the Effects of Harcourt Achieve's Elements of Reading: Vocabulary

1

Behind (NCLB) Act. It allows all instruction and practice of vocabulary words to be done orally, building foundational knowledge essential to reading comprehension and fluency. Drs. Isabel L. Beck and Margaret B. McKeown developed EOR Vocabulary based on their extensive research and teaching experience. EOR Vocabulary provides "thoughtful, interactive encounters with words and frequent practice with word meanings and their uses, and also prompts children to take the words they learned beyond the classroom" (Beck & McKeown, 2004, p. T6). EOR Vocabulary is designed for use in daily 20-minute whole-group instruction and to present ample opportunities that engage student responses throughout the day, week, and year, during and beyond the language arts period, and in and out of the classroom.

challenging discussion prompts; for example, pantomimed guessing games, sentence starter prompts, cloze activities, or judging if situations are or are not examples of a word/concept. Day 4 activities engage students in completing graphic organizers with the words/concepts and writing about their own experiences with the new words.

Ample Practice and Assessment

Throughout each week, the student workbook activities provided varied practice, and the Teacher's Guide includes a host of suggested prompts to encourage children's spontaneous use of the words in and out of the classroom. Day 5 involves a week's-end assessment in a multiple choice format and a cumulative review involving open-ended questions using words from previous and current lesson. A Word Watcher Chart is provided that is a blend between a pocket chart to hold the week's word cards and a visual record of word learning. On the erasable poster part, students and teachers tally the number of times each week's word is used.

Robust Vocabulary Selection

The content of EOR Vocabulary is comprised of what Beck, McKeown, and Kucan (2002) refer to as second-tier words, the kinds of "vivid" words teachers encourage their students to use in place of "tired" words. Second-tier words describe with more precision, people and situations with which children "already have some familiarity" (p. 17); for example, "required" instead of "have to" or "benevolent" instead of "kind." Second-tier words, selected for EOR Vocabulary, "are not the most basic or common ways of expressing ideas, but they are familiar to mature language users as ordinary" (Beck et al., 2002, p. 18).

Study Measures

Teacher Survey

The teacher survey was designed to collect information about instructional practices in both treatment and control classrooms and information on teachers' professional experience and preparation. Instructional practice items asked about frequency of use of generic practices (e.g., "I evaluate whether individual students are sufficiently progressing.") and practices specific to developing vocabulary (e.g., "In my classroom, students write about their own experiences in relation to vocabulary words.").

Context-rich, Explicit and Interactive Instruction

In classroom kits specially designed for each grade (kindergarten through Grade 5), EOR Vocabulary provides an anthology of stories for read-alouds as the kick-off to each weekly lesson plan. Each weekly lesson plan targets seven words in the story and offers a sequenced set of activities. On Day 1, vocabulary is introduced in the context of the story, and a Fun With New Words letter is sent home with students. On Days 2 and 3, teachers define each word with Word Snapshot Photo Cards and lead students in Word Chats. The Photo Cards contain dramatic images in real-life contexts that help children visualize and personalize each vocabulary word. The Word Chats involve increasingly

Site Visit Protocols

Classroom observation guidelines and protocols were developed for researchers to record the nature (grouping, type of student activity, materials used, etc.) and content of instruction in both treatment and control classrooms. Guidelines for interviewing teachers postobservation were developed to ensure that unique circumstances were identified and that teachers provided clarification, rationale and feedback regarding instructional materials and activities. Interview guidelines were developed for principals and site coordinators to obtain their

2

Final Evaluation Report

feedback on product usage. Field notes were handwritten or typed and identified by date and school and teacher identification number; summaries of classroom observations were written from field notes.

Teacher Activity Logs

An EOR Vocabulary activity log was developed for teachers to complete weekly. Teachers recorded the date, the EOR Vocabulary lesson number and story title used that week, and the extent to which each key instructional activity was implemented. An open-ended question also asked about the helpfulness of the Teacher's Guide and lesson materials.

Four levels of EOR Vocabulary implementation were defined according to three indicators, including (a) appropriate use of key activities observed during site visit, (b) use of key activities reported in weekly logs, and (c) use of the intervention stories and vocabulary words. In addition to appropriate observed use, further criteria distinguishing each level of implementation are identified and presented in Table 1.

Early Reading Diagnostic Assessment (ERDA)

Selected subtests of the Early Reading Diagnostic Assessment, Second Edition (ERDA; Psychological Corporation, 2003) served as pre- and posttests of student vocabulary and reading skills. The ERDA, designed for use with students in kindergarten through third grade, provides a comprehensive diagnostic assessment of early reading and reading-related skills. Subtests selected for use in this study are identified in Table 2. Reliability

Implementation Monitoring

Monitoring implementation of EOR Vocabulary was conducted using data from two sources: treatment teachers' weekly logs and researcher classroom observations during March site visits.

Table 1. Defining Criteria for Levels of EOR Vocabulary Implementation

Level of EOR Vocabulary Implementation Very Low Low Moderate High

a

Criteria in addition to appropriate observed use Key activities

a

Number (%) of EOR Vocabulary stories/units used 10­12 (50% or less) 13­18 (54% - 75%) 19­20 (79% - 83%) 21­24 (more than 86%)

Four or fewer of six key activities used the majority of weeks Five of six key activities used the majority of weeks Six of six key activities used the majority of weeks Six of six key activities used the majority of weeks, AND EOR Vocabulary words are used in extended instructional activities beyond the prescribed EOR Vocabulary lesson activities.

EOR Vocabulary Key Activities: 1. At least 3 word cards displayed 70% of weeks 2. At least 5 minutes with Photo Cards majority of weeks 3. At least 5 minutes in Word Chat majority of weeks 4. Graphic organizer, writing or student workbook used majority of weeks 5. Used Word Tally to monitor word use 6. Cumulative review used 50% or more weeks

Table 2. Early Reading Diagnostic Assessment (ERDA) Subtests

ERDA Tests Used as Pre- and Posttests Full Vocabulary Composite Brief Composite (Receptive and Expressive Vocabulary) Synonyms Word Definitions Multiple Meanings Word Reading

A Study of the Effects of Harcourt Achieve's Elements of Reading: Vocabulary

3

evidence for critical subtests in the present study supported their measurement coherence with splithalf reliability coefficients for each subtest .65 or greater (see Table B-1 in Appendix B). Additionally, skill analyses and patterns of intercorrelations between the ERDA subtests support the content and construct validity of the measures as intended (Psychological Corporation, 2003).

Training

In addition to the study orientation, treatment teachers participated in 1 hour of training in the use of EOR Vocabulary. Teachers received and unpacked their EOR classroom kit and were provided a brief overview of the product design, rationale, and key components. The 5-day lesson sequence was explained and all core activities described and an explanation of their expected benefits for students provided. The organization and parts of the Teacher's Guide and weekly units were previewed and a question and answer period concluded the session. Each treatment teacher and site coordinator were encouraged to contact Harcourt Achieve's Educational Support Services for additional consultation or support in use of EOR Vocabulary and were provided telephone numbers to call for this purpose.

Gates-MacGinitie Reading Test (GMRT)

The Gates-MacGinitie Reading Test, Fourth Edition (GMRT), Level 3, was selected as a normreferenced, posttest only measure of student reading achievement. Two subtests were used: Reading Vocabulary and Comprehension. GMRT Vocabulary is designed to measure knowledge of printed word meaning. GMRT Comprehension is designed to measure reading comprehension using materials students are expected to learn to read and questions that students should be able to answer from their reading. The passages are both fiction and nonfiction and include material from both the social and natural sciences. The GMRT Vocabulary and Comprehension subtests have strong internal consistency as reflected in high Kuder-Richardson (K-R) coefficients. These K-R 20 coefficients are .93 and .92 for Level 3 Vocabulary and Comprehension, respectively.

Test Administration and Scoring

Trained examiners administered and scored the selected subtests of the ERDA to individual students. ERDA pretests were administered in September 2004 in Site A and in October 2004 in Site B. ERDA posttests were administered in May 2005 in Site A and in June 2005 in Site B. Administration of the GMRT posttest was coordinated by the district and school study coordinators. Classroom teachers administered the GMRT in May 2005 in Site A and in June 2005 in Site B. GMRT tests were scored by Riverside Publishing.

Study Procedures

Once school participation agreements were finalized, teachers at each grade level within each school were randomly assigned to treatment or control. Reading coaches or reading specialists at each school served as study coordinators to facilitate data collection and communication between participants and McREL researchers. District-level coordinators also assisted with data collection and adherence to district policies. Both treatment and control teachers participated in a 1-hour, on-site study orientation. Treatment teachers, control teachers, and site coordinators received an honoraria payment relative to the level of their responsibilities for the study. At the conclusion of the study, schools received an additional gift certificate for Harcourt Achieve materials and classroom EOR kits for the control classrooms (in addition to the EOR kits provided to treatment teachers at the beginning of the study).

Settings

The study of the effectiveness of EOR Vocabulary was conducted in 15 third-grade classrooms in seven schools across the three districts. One district comprised Site A, was located in a southern state, and was classified as a "Small Town" by the National Center for Education Statistics. Two neighboring districts, both classified as "Urban Fringe of Mid-Size City," comprised Site B and were located in a northeastern state. At each site, elementary schools were identified by districtlevel staff to ensure that schools met the following selection criteria: Title I status or 35% and higher students eligible for free or reduced-price lunch, current implementation of one of two basal reading programs (McGraw Hill or Scott Forseman),

4

Final Evaluation Report

relatively high attendance and historically low student mobility rates, and teacher comfort with teacher-level random assignment.

Site A

Four schools in Site A participated in the EOR Vocabulary study, each with an average enrollment of 226 students. The majority (98.5%) of students in these four schools are Black or African American, and 1.5% are White. Ninety-four percent of students in these four schools are eligible for free or reducedprice lunch (FRL). The average student­teacher ratio in the participating schools is 15 students per teacher. District revenue per student is $6,362. McGraw-Hill Reading was the basal reading program in place in Site A. The instructional approach was standards-based. Elementary schools used the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) to assess and monitor student progress in beginning reading. Teachers and reading coaches participated in a statewide professional development program in reading instruction for each element of reading. The program was in its 2nd or 3rd year. The focus was on phonics and phonemic awareness in prior years and on oral reading fluency during 2004-2005, the time of the study. Vocabulary instruction had not yet been covered in this statewide professional development program.

running records and Fountas and Pinnell (1996) book levels. Teachers and teacher leaders participated in professional development and networking as part of a local school­university collaborative called the Balanced Literacy Consortium.

Participants

Teachers

Fifteen third-grade classroom teachers participated in this study, 7 in Site A (4 treatment and 3 control teachers) and 8 in Site B (4 treatment and 4 control).1 Teachers, across both control and treatment, had an average of 14 years teaching experience. Teachers' preparation to teach reading included, on average, 8 or 9 courses in reading from both undergraduate and graduate training. The control and treatment groups of teachers did not significantly differ at any grade level in either years of experience or reading coursework.

Students

Of the 291 student participants in the EOR Vocabulary study, with complete demographic data, 164 (56%) were female and 127 (44%) were male. For Grade 3, across treatment conditions, for the 298 students where ethnicity data were provided, 148 (49%) were African American, four (1%) were Asian/Pacific Islander, 12 (4%) were Hispanic, and 138 (46%) were White. The number of students in the EOR Vocabulary study (Grade 3) who qualified for free or reducedprice lunch was 250 (83.3%). Seven (7%) percent of student participants were special education students, and 1.5% of students were categorized as limited English proficient (LEP). A significant difference between treatment and control occurred when examining the special education/non­special education status. The treatment group had a proportionately greater number of special education students (19 or 11.5%) compared to the control group, which had three (2%).

1

Site B

Three schools in Site B participated in the EOR Vocabulary study, each with an average enrollment of 518 students. The majority (82%) of students in these three study schools are White, with 10% of students considered Black or African American, and 8 % from other ethnic backgrounds. Each of the three schools are Title I schools. The average teacher­ student ratio is 13.9 students per teacher. The average yearly revenue per student is $12,489.50. McGraw-Hill Spotlight on Literacy (two schools) and Scholastic Literacy Place (one school) were the basal reading programs in place in Site B. The instructional approach was balanced literacy with Guided Reading groups in all schools to provide responsive systematic instruction based on and to enhance individual reading development. For assessment and monitoring, all Site B schools used

Two of the three third-grade teachers in one Site A school exchanged students for reading to provide small, ability group instruction. This cross-class grouping arrangement precluded use of the planned teacher/classroom as unit of random assignment; and therefore, only one third-grade teacher/ classroom was eligible to participate at this school.

A Study of the Effects of Harcourt Achieve's Elements of Reading: Vocabulary

5

A breakdown of the number and percentage of students in each demographic category for thirdgrade student characteristics is provided in Table A-1 in Appendix A. The total number of students within each category varies slightly based on available data in that category.

such as writing, graphic organizers, and discussion. Weekly log and observational data confirmed the survey results. In both Site A and B, weekly log and observational data indicated that all treatment teachers used EOR Vocabulary with fidelity, introducing weekly vocabulary words with a story from the product's anthology, explaining word meaning with the support of each word's Photo Card, engaging in Word Chat, and encouraging children's spontaneous use of the new words. In both Sites, use of EOR Vocabulary was lively and rewarding. To summarize, teachers responded enthusiastically to the EOR Vocabulary program, stories, activities and materials. Teachers found the teacher's manual helpful (e.g., "I like the Teacher's Manual because it gives clear directions on how to present this program."). The Photo Cards were fun, interesting, and very beneficial as teachers indicated in the following comments: "The Photo Cards are my favorite part of the program because it gives the students a visual look at the words." "The photo cards are the students' favorite part." (Site A) "Students that are lower than others use the Photo Cards to figure out the meaning of the word." "Students are able to relate to several of the pictures." (Site A) "I feel the best part of the program is the Photo Cards because they give real examples of each word." (Site A) "Students love the word cards...enjoyed the Photo Cards, especially originality." (Site B) "The Photo Cards are very helpful in developing the students' interest in learning new vocabulary words." (Site A) Participants also enjoyed the Word Chat ideas and other discussion or game prompts as indicated by numerous and positive comments in the teacher logs. A few sample comments follow:

Results

The evaluation questions for this study focus on (a) student reading performance in classrooms with and without the use of EOR Vocabulary, (b) possible differential effects on student subgroups, and (c) classroom implementation of EOR Vocabulary. This section presents analyses and results addressing each of these questions based on implementation monitoring and student reading performance. First, to better understand the instructional contexts of the study classrooms, evaluators analyzed data collected through the teacher survey, interviews, and classroom observations. Descriptive findings regarding instructional contexts are summarized below.

Instructional Contexts

Analyses of the midyear teacher survey data revealed that EOR Vocabulary treatment teachers, compared to control teachers, reported more frequent use of research-based vocabulary development activities. As seen in Figure 10, 70% of treatment teachers or more reported students (a) wrote about their own experience, (b) completed graphic organizers, and (c) participated in discussions in relation to vocabulary words at least weekly compared to proportionally fewer control teachers. With regards to more traditional vocabulary building activities (i.e., vocabulary workbooks) frequency of use was less distinct in the treatment and control classrooms. In both conditions, 100% of teachers reported at least weekly use of vocabulary worksheets or workbooks (see Figure 1). Based on survey data, the instructional contexts were different between treatment and control with treatment teachers, as would be expected when using EOR Vocabulary, reporting more frequent use of interactive vocabulary development activities,

6

Final Evaluation Report

Figure 1. Frequency of student engagement in vocabulary development activities in treatment and control classrooms (seven teachers/classrooms per group).

(a) Write about own experience in relation to words

Treatment

Control 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Daily Weekly Monthly or less

(b) Graphic Organizers about Vocabulary Words

Treatment

Control 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Daily Weekly Monthly or less

(c) Discuss Vocabulary Words

Treatment

Control 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Daily Weekly Monthly or less

(d) Vocabulary Wordsheets/Workbooks

Treatment

Control 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Daily Weekly Monthly or less

A Study of the Effects of Harcourt Achieve's Elements of Reading: Vocabulary

7

"Students enjoyed the activity describing the situation. This activity allows acting out the word for better understanding." (Site A) "Students love to divide in groups and make a game out of the Word Chat activities." (Site A) "Many didn't know what the Special Olympics were, so the story lent itself to a good discussion afterwards." (Site B) Teachers using EOR Vocabulary saw immediate benefits for their students. Their students kept finding and using the words taught. Two different teachers commented: "Their vocabulary is expanding each day" and "The students actually use their vocabulary words in conversation." Treatment teachers' students asked more questions about words (e.g., "What does phenomenal mean?") than students in previous years. Their students commented on their principal's use of words during daily announcements (e.g., when the principal announced, "There will be a brief faculty meeting after school," children said, "That means it's not going to last long."). Students took pleasure in bringing the words home with them. At home, students noticed their words in their older siblings' homework, and, as reported by teachers, students "taught their parents new words." During Word Chat and Graphic Organizer activities observed during the visit, children made the following statements using EOR Vocabulary: "Remember, teacher, yesterday when you comforted him, you were soothing him." (Site A) In response to the teacher's request for an example of integrity, a student offered, "When in the grocery store, I find money under the benches and give it to the cashier or give it to charity." (Site B) Teachers and students used the EOR Vocabulary words throughout the day and over time. "Already today, three people have used one of our vocabulary words," announced a Site B treatment teacher during the site visit. "Integrity, we've talked about this in Character Ed.," another teacher was observed

explaining to her class. In Site A, EOR Vocabulary words were used in practice activities to prepare students for upcoming state assessments. At one school, "boasting" was encountered in the practice workbook for the upcoming state assessment. At another school, a treatment teacher regularly included the week's EOR Vocabulary words in practice writing prompts for the upcoming assessment. Clearly, the instructional contexts were different between treatment and control classrooms. The next question addressed by the analyses is whether or not there were related differences between treatment and control in terms of student performance.

Student Performance

At the beginning of the study (September 2004), there were important differences between sites in terms of both school demographics (over 90% FRL rates in Site A schools compared to under 30% FRL rates in Site B schools) and student knowledge and skills. A 2-factor (group and site) multivariate analysis of variance was conducted on pretest ERDA subtest scores. Results revealed significant effects of Site (F (6, 281) = 21.39, p < .001).2 Average pretest ERDA scores were significantly lower in Site A compared to Site B for all six subtests. For example, for the pretest ERDA Full Vocabulary Composite, mean pretest score in Site A was 37.19 (n = 124) compared to 46.67 in Site B (n = 166). When converted to percentile ranks, these Site A and B pretest mean scores are at the 9th and 39th, or near 40th, percentile, respectively. Therefore, given the significant differences in pretest levels of vocabulary knowledge, treatment effects on student performance were analyzed separately within each site.

Changes from Pretest to Posttest

Vocabulary growth was examined using the ERDA Full Vocabulary Composite score, a composite measure representing both receptive and expressive vocabulary, synonym knowledge, definitional skills, and awareness of words' multiple meanings. As can

2

Results of the multivariate analysis of variance on ERDA pretest scores also revealed nonsignificant effects for group (F (6, 281) = 1.497, p = .179) and nonsignificant group by site interaction effects (F (6, 281) = 1.102, p = .361) indicating that random assignment resulted in comparable treatment and control groups in terms of ERDA pretest performance.

8

Final Evaluation Report

be seen in Figure 2, average ERDA Full Vocabulary Composite scores, in both sites in both treatment and control classrooms, increased from pretest to posttest. Also shown in Figure 2, in the left-hand plot, in Site A, the treatment gains on ERDA Full Vocabulary Composite were greater than the control gains. The steeper slope of the treatment compared to the control line connecting pre- and posttest ERDA Full Vocabulary Composite scores represents a statistically significant group by time interaction effect in Site A (F (1, 114) = 8.324, p < .01). Using Cohen's d, the effect size for this interaction was moderate (ES = 3 .54). The effect size for the treatment group's pre- to

posttest vocabulary gains was also moderate (ES = .52) compared to a negligible effect size for the control group's gains (ES = .06). With regards to ERDA Word Identification, a measure of the ability to identify printed words in isolation, students in both sites in both treatment and control improved from pretest to posttest. Furthermore, as suggested by the near parallel lines in Figure 3, there were no significant group by time

3

In estimating effect sizes, we used partial eta squared as the proportion of variance (PV) and the following formula for computing Cohen's d: effect size (ES) = Square Root of (4(PV)/(1-(PV)).

Figure 2. ERDA Vocabulary Composite by time and Group.

Site A Vocabulary Composite

55 55 51.58 50 Mean Raw Score Mean Raw Score 50 47.93 49.82 45 46.80

Site B Vocabulary Composite

45 41.42 40 37.64 38.14 35 36.88

40

35

30

Pretest

Posttest

30

Pretest

Posttest

Control (n=74)

Treatment (n=42)

Control (n=74)

Treatment (n=76)

Figure 3. ERDA Word Identification by time and group.

Site A Word Identification

42 40 Mean Raw Score 37.85 38 36 34.21 34 33.48 32 30 32 30 38 36 34 42 40.21 40 38.07 39.24

Site B Word Identification

36.40

36.38

Pretest

Posttest

Pretest

Posttest

Control (n=47)

Treatment (n=77)

Control (n=76)

Treatment (n=74)

A Study of the Effects of Harcourt Achieve's Elements of Reading: Vocabulary

9

Mean Extended Scale Score (ESS)

interaction effects meaning that EOR Vocabulary did not have a significant impact on word identification in either site. This finding does not diminish the potential value of EOR Vocabulary since improved printed word identification is not an explicitly intended outcome of EOR Vocabulary. Improved reading vocabulary and comprehension, however, are explicitly intended outcomes. The next section of the results addresses EOR Vocabulary's effectiveness for reading vocabulary and comprehension.

Figure 4. Mean GMRT Vocabulary ESS for treatment and control.

Site A GMRT Vocabulary Performance

460

457.89

455

450

445

Comparing Treatment and Control Posttest Performance

The Gates-MacGinitie Reading Test, Fourth Edition (GMRT), Level 3, was used as a norm-referenced test of reading achievement. Two GMRT subtests, Reading Vocabulary and Comprehension, were administered at posttest. To compare treatment and control student performance on these posttests, the GMRT Vocabulary and Comprehension posttest scores were analyzed, within each site, using a multivariate analysis of variance with group as the independent factor. In Site A, there were significant treatment effects on GMRT Reading Vocabulary but not on GMRT Comprehension. Students in classrooms taught by teachers using EOR Vocabulary compared to students in classrooms not taught by teachers using EOR Vocabulary, on average, performed significantly higher on GMRT Vocabulary (F (2, 127) = 8.452, p < .005). Treatment and control mean scores are shown graphically in Figure 4. The effect size for the treatment effect on GMRT Vocabulary is moderate (.51). In Site B, neither treatment nor control groups performed significantly higher on either GMRT Vocabulary or Comprehension. Results of each of the Site A and Site B analyses comparing GMRT posttest performance for treatment and control, included mean group scores, are presented in Appendix C. Thus, at posttest, significant effects for treatment on GMRT performance were found in Site A but not in Site B. This finding is consistent with the pattern of results reported earlier with respect to ERDA performance. Additionally, evaluators examined whether student attendance may have influenced the potential

442.17

440

435

430

Control (n=48)

Treatment (n=82)

impact of EOR Vocabulary. Attendance rates did not differ between treatment and control groups; nor, based on Pearson correlation analysis, was attendance significantly correlated with posttest performance (Pearson r's ranged from .03 to .29 with all p-values greater than .05).

Subgroup Analyses

Gender subgroups (boys and girls) and ability subgroups were identified. Insufficient numbers of students in special education and Limited English Proficient (LEP) groups precluded analyses regarding these other subgroups. With regard to ability, students were categorized into achievement levels based on their pretest performance at the beginning of the study. The ERDA Technical Manual defines three levels of achievement as follows: Score at or below the 29th percentile Need for intensive support and instruction to reach grade level Score between 30th and 69th percentile On grade level achievement Score at or above the 70th percentile Proficient Proficient skills and at or above grade-level achievement

Below Basic

Basic

10

Final Evaluation Report

Three Vocabulary Ability and three Reading Ability subgroups were formed using the above criteria as applied to individuals' ERDA Full Vocabulary Composite pretest and ERDA Word Identification pretest scores, respectively. Within each site, a 3factor (group, gender, ability) multivariate analysis of variance was applied to the ERDA Vocabulary and Word Identification and GMRT Vocabulary and Comprehension posttests. There were no significant group by gender or group by ability interaction effects in either site indicating that EOR Vocabulary did not differentially affect students based on gender or ability.

in Table 3, as Level of Implementation increased, so did class averages for ERDA Full Vocabulary Composite, but the same pattern was not evident between teachers' level of implementation and class averages on either GMRT test. Other variation in implementation of EOR Vocabulary also was evident. Site A teachers used the Word Watcher tally infrequently reporting an average of seven instances of words taught used in the classroom per week. Site B teachers used the Word Watcher tally frequently, reporting an average of 19 instances of words taught used in the classroom per week. Two teachers in Site A, in particular, extended the time spent on the EOR Vocabulary words each week, either by included the words in the weekly spelling list or by practice writing prompts. Teachers in Site B were especially challenged to find time for EOR Vocabulary. Time allocated to reading/language arts already was filled with multiple Guided Reading groups, assessments, and related reading and writing activities.

Level of Implementation

In Site A, use of EOR Vocabulary had a positive impact on students' vocabulary. In this section, variation in teachers' use of EOR Vocabulary in and across both sites was examined. The focus was on whether or not patterns of use and relationships between use and student achievement would surface. Four levels of EOR Vocabulary Implementation were defined (see Table 1): (1) Very Low, (2) Low, (3) Moderate, and (4) High. As seen in Table 3, no treatment teachers were rated as Very Low Implementers, three teachers were rated as Low or Moderate Implementers, and two teachers were rated as High Implementers. Teachers were similarly distributed across Levels of Implementation in each site. Higher levels of implementation were distinguished by more comprehensive, frequent, and extended use of EOR Vocabulary key activities and stories/units. Evidence relating Level of Implementation and student performance, however, was mixed. As seen

Conclusion

Using randomized control trials, the effectiveness of Elements of Reading Vocabulary as a supplemental product for improving student vocabulary and reading performance was examined. Fifteen Grade 3 teachers and their students (291) participated in the study in two sites, A and B. Teachers randomly assigned to treatment in both sites used EOR Vocabulary with fidelity over the 2004­2005 school year. Pre- and posttests were administered to students in both treatment and control classrooms.

Table 3. Relating Level of Implementation and Student Performance

Level of EOR Vocabulary Number of teachers Implementation Site A Site B Low Moderate High

a

Adjusted mean posttest class average ERDA Full Vocab 44.702 47.07 47.47 GMRT Vocab 491.04 468.81 472.02

a

GMRT Comprehension 478.90 472.67 475.67

2 1 1

1 2 1

Class averages were adjusted to control the influence of differences between classes on Fall 2004 ERDA Word Identification and Vocabulary Composite.

A Study of the Effects of Harcourt Achieve's Elements of Reading: Vocabulary

11

In Site A, EOR Vocabulary had a significant and positive impact on student vocabulary development. Significant group by time interaction effects were found on ERDA Full Vocabulary Composite, a composite measure of oral vocabulary. Compared to negligible pre- to posttest gains for students in control classrooms (ES = .06), pre- to posttest gains for students in classrooms where teachers used EOR Vocabulary were sizable (ES = .52). In Site A, the effectiveness of EOR Vocabulary also was evident in reading vocabulary as measured by a norm-referenced achievement test. At posttest, treatment students performed significantly higher than control students on GMRT Reading Vocabulary. The effect were sizable (ES = .51). These effects are stronger than the average effect of other interactive vocabulary instruction interventions. In a meta-analytic review of vocabulary instruction effects, Stahl and Fairbanks (1986) reported an average effect size of .30. In Site B, treatment students performed as well as control students in vocabulary development and reading achievement. Therefore, in Site B, EOR Vocabulary was beneficial but not more effective than the vocabulary instruction and related experiences that were provided in control classrooms. Four levels of EOR Vocabulary implementation were defined, ranging from Very Low to High, and applied to a combination of observational and weekly log data. Treatment teachers in both sites used EOR Vocabulary at one of the top three levels of implementation. More frequently than control teachers, treatment teachers encouraged student use of vocabulary words in interactive writing and discussion activities, including graphic organizers, games and pantomime, and at home. Teachers in both sites were especially enthusiastic about the Photo Cards helping develop student interest in learning new words. Teachers in Site A extended opportunities for students to use the vocabulary words by including them in weekly spelling lists and writing prompts. Teachers saw immediate benefit for their students, commenting, for example, "students actually use their vocabulary words in conversation." These and other reports

provide supportive evidence that EOR Vocabulary accomplishes what the developers intended it to accomplish, create enthusiasm about words and engage students in frequent, interactive encounters with words in and out of the classroom.

References

Beck, I. L., & McKeown, M. G. (2004). Elements of Reading vocabulary teacher's guide level C. Austin, TX: Steck-Vaughn. Beck, I. L., McKeown, M. G., & Kucan, L. (2002). Bringing words to life: Robust vocabulary instruction. New York: Guilford Press. Fountas, I. C., & Pinnell, G. S. (1996). Guided reading: Good first teaching for all children. Portsmouth, NH: Heinemann. Jordan, G. (2003). Florida Center for Reading Research: Elements of Reading: Vocabulary. Retrieved January 3, 2005, from http://www.fcrr.org/ FCRRReports/PDF/vocaublary-elements.pdf Lipsey, M. W. (1990). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage. MacGinitie, W. H., MacGinitie, R. K., Maria, K., & Dreyer, L. G. (2000). Gates-MacGinitie reading tests­ Manual for scoring and interpretation. Itasca, IL: Riverside. Nagy, W. E. (1988). Teaching vocabulary to improve reading comprehension. Urbana, IL: ERIC Clearinghouse on Reading and Communication Skills. Psychological Corporation, a Harcourt Assessment Company. (2003). Early Reading Diagnostic Assessment (ERDA) administration manual (2nd ed.). San Antonio, TX: Author. Stahl, S. A., & Fairbanks. M. M. (1986). The effects of vocabulary instruction: A model-based metaanalysis. Review of Educational Research, 56(1), 72110.

12

Final Evaluation Report

Appendix A

Table A-3. Grade 3 Student Demographics by Site

Site A Control Gender Female Male Ethnicity African American Asian/Pacific Islander Hispanic White English language learners/ limited English proficiency ELL/LEP Formerly ELL/LEP Non­ELL/LEP Special education LD or speech impaired Other disability Nondisabled Poverty status Free lunch Reduced-priced lunch Non­FRL (n = 49) 61.2% 38.8% (n = 49) 100% 0% 0% 0% (n = 49) 0% 0% 100% (n = 49) 0% 0% 100% (n = 49) 98.0% 0% 2.0% Treatment (n = 74) 52.7% 47.3% (n = 84) 97.6% 0% 0% 2.4% (n = 84) 0% 0% 100% (n = 84) 8.3% 2.4% 89.3% (n = 84) 86.9% 4.8% 8.3% Control (n = 86) 57.0% 43.0% (n = 85) 11.8% 1.2% 2.4% 84.7% (n = 64) 1.6% 3.1% 95.3% (n = 84) 3.6% 0% 96.4% (n = 85) 45.9% 23.5% 30.6% Site B Treatment (n = 82) 56.1% 43.9% (n = 80) 8.8% 3.8% 7.5% 80.0% (n = 59) 1.7% 0% 98.3% (n = 80) 8.8% 3.8% 87.5% (n = 82) 57.3% 23.2% 19.5%

A Study of the Effects of Harcourt Achieve's Elements of Reading: Vocabulary

13

Appendix B

Table B-1. ERDA Subtest Reliability Coefficients for Grade 3

EOR Vocabulary Full Vocabulary Composite Brief Composite (Receptive and Expressive Vocabulary Synonyms Word Definitions Multiple Meanings Word Reading Split-half reliability coefficient .87 .65 .70 .65 .65 .85

14

Final Evaluation Report

Appendix C

Table C-1. Multivariate Analysis Results for GMRT Vocabulary and Comprehension Extended Scale Scores

Site A Treatment (n = 82) GMRT Vocabulary GMRT Comprehension M SD M SD 457.89 31.48 457.04 28.73 Control (n = 48) 442.17 26.54 457.94 28.96 F 8.452 .030 p .004 .864 ES (Cohen's d) .51 .00

Site B Treatment (n = 71) GMRT Vocabulary GMRT Comprehension M SD M SD

2

a

Control (n = 77) 494.60 34.05 481.16 41.23

F 2.107 .326

p .149 .569

ES (Cohen's d) .24 .09

485.80 39.61 485.04 41.53

Note. In estimating effect sizes, and eta as the proportion of variance (PV), we used the following formula for computing Cohen's d: effect size (ES) = square root of (4(PV)/(1-(PV)). From Design Sensitivity: Statistical Power for Experimental Research, by M. W. Lipsey, 1990, Newbury Park, CA: Sage.

A Study of the Effects of Harcourt Achieve's Elements of Reading: Vocabulary

15

©2006 Harcourt Achieve All rights reserved. Printed in the U.S.A. 10M/1454/NY/HOR/02-06 9994114255 IND 9994114298 10-PK

16

www.HarcourtAchieve.com Final Evaluation Report 1.800.531.5015

Information

EOR_Voc_McRel.indd

20 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate

1228411


You might also be interested in

BETA
untitled
Assisting Students Struggling with Reading: Response to Intervention and Multi-Tier Intervention in the Primary Grades
limSYC