Read ESL ACCUPLACER Writing Pilot Project text version

ESL ACCUPLACER Writing Pilot Project (Fall 01 thru Fall 02) Purpose of the Project

The Assessment Office in cooperation with the ESL section of the English department at TMCC conducted a study involving ESL students enrolled in the writing courses for the period from Fall 2001 through Fall 2002. The purpose of the project was to collect the data to: 1) validate the writing placement instrument currently used by TMCC on a pilot basis to assess and advise ESL students; 2) determine writing cut scores for ESL assessment and placement . The results of the study will be used to establish the placement cut scores to assess writing abilities of incoming ESL students, as well as to aid in the process of defining competencies across the ESL curriculum and measure overall student mastery of these competencies.

Instrument

Students were given a LOEP (test of Limited English Proficiency) writing section of the ACCUPLACER on-line placement tests and were asked to compose an on-line writing sample that was scored electronically.

Population

A total of 203 students were tested from the Fall 2001 to Fall 2002. This produced a valid sample of 176 students for the statistical analysis. The majority of students were enrolled in the writing courses English 112 & 113.

Methodology

The LOEP on-line writing test was administered to students during the first week of each semester on their regular class time. The essay topics were randomly administered to students from the three prompts available. Students were asked to compose an on-line writing sample in response to a given prompt within 1 hour. The essays were electronically scored by ACCUPLACER. The second part of the project involved 4 human readers, ESL instructors form the TMCC English department, blind-scoring the hard copies of the same essays using the same scoring rubric (see the scoring rubric attached). In addition, the readers assigned placement to each one of the writing samples based on the existing ESL curriculum. The student essays were assessed using five main characteristics: 1) Organization 2) Focus, Development, and Support 3) Vocabulary 4) Sentence Structure 5) Mechanics The data were analyzed for concordance between the ACCUPLACER electronic scoring and human readers. The placement data were analyzed in conjunction with the competencies represented by each score and their correlation to the levels of proficiency expected in each one of the ESL writing courses currently offered. Based on the conducted analysis, a preliminary recommendation on the writing cut scores and ESL placement policies was produced.

1

Office of Institutional Research and Assessment, 12/11/02

Funding

Funding for this project was provided by the TMCC Office of Institutional Research and Assessment. All four readers were given a stipend to conduct the assessment of student essays. The staff of the Institutional Research and Assessment Office conducted the testing. Elena Bubnova, Director of the Assessment Office and Chris Scally, the ESL coordinator conducted data collection and analysis in conjunction with the ESL faculty.

Major Findings

Overall, the data collected provided enough information to generate preliminary recommendations for ESL on-line writing assessment utilization and application of cut scores for placement into existing ESL courses. When the electronic scores produced by ACCUPLACER are compared to those given by human readers, the readers appear to be more stringent graders. This is acceptable given that there is a need to account for test anxiety that students are experiencing when taking the on-line examination. A significant degree of variability in ratings is found between readers. There was a high number of occurrences when ACCUPLACER electronic scores matched at least one reader's score. Most frequently occurring score appeared to be `4' which correlated to the most frequently assigned placement into English 113, which in turn, corresponded to the highest number of participants who were enrolled in English 113 at the time of testing. Two options are proposed with regards to using the ACCUPACER LOEP on-line writing test to assess ESL students and place them into writing courses: Option 1 If a student scores between 76 and 105 on the LOEP Reading Skills test, then administer on-line writing. Students receiving a score of 0 to 2 on the writing sample will be recommended to sign up for English 081 sequence. Students receiving a score of 3 to 6 will be recommended to sign up for English 112. Option 2 If a student scores between 106 and 114 on the LOEP Reading Skills test, then administer on-line writing. Students receiving a score of 0 to 4 on the writing sample will be recommended to sign up for English 112. Students receiving a score of 5 to 6 will be recommended to sign up for English 113.

2

Office of Institutional Research and Assessment, 12/11/02

The following notes are helpful when interpreting the data in summary tables

General Notes

27 student essays had to be eliminated from the study due to incomplete or inadequate essays (written on irrelevant topic), erroneous student identification numbers. A number of students had multiple essays due to their continuous enrollment in ESL courses over the course of the project. These students were included in the analysis and treated as separate participants since they were enrolled in different courses each time when tested.

Highlights

A total of 176 student essays from 7 different ESL courses were used in the analysis 68% of the time ACCUPLACER electronic scoring matched at least one reader's score The majority of participants (56%) were enrolled in English 113 at the time of testing (see Table A). Most frequently occurring score given by ACCUPLACER is `4'. This corresponds to English 113, the most frequently occurring placement assigned by readers. Most frequently occurring score given by readers is `3' (see Table B). Course placement distribution based on the ACCUPLACER scores was as follows: o score of `1': extremely rare o score of `2' : equal split between ENG 081 and ENG 112 o score of `3': predominantly ENG 112 o score of `4' equally split between ENG 112 and ENG 113 o score of `5': predominantly ENG 114 o score of `6': none See attached tables for detailed data and charts for recommended placement options.

Future Recommendations

Introduce the new placement recommendations for the next testing period starting February 1, 2003 Conduct a follow-up study in a year to validate the cut scores and student success

3

Office of Institutional Research and Assessment, 12/11/02

ESL ACCUPLACER Writing Pilot Project Fall 2001 thru Fall 2002 Fall 01 Total tested: Usable samples: Fall 02 Total tested: Usable samples: Total tested overall: Total usable sample:

59 46

157 157 203 176

TABLE A: Actual Enrollment Distribution Among Students Tested course 52 81 85 103 112 113 114 Total # 1 1 1 9 53 98 13 176 % 1% 1% 1% 5% 30% 56% 7% 100%

TABLE B: Score Distribution (# of occurances) Score scored by Accuplacer Readers (Ave) 1 21% 2 3% 19% 3 28% 38% 4 47% 32% 5 18% 10% 6 3% 1%

TABLE C: Placement Distribution by Readers' Ratings Course Placement ENG 112

# %

Readers' score 1 2 3 4 5 6

ENG 081 # % 21 43% 7 5% -

ENG 113

# % #

Total

%

16 28 134 54 1 -

100% 57% 95% 50% 3% -

54 36 1

50% 97% 100%

16 49 141 108 37 1

100% 100% 100% 100% 100% 100%

Fall02 ESL WS Pilot Summary

Office of Institutional Research and Assessment, 12/11/02

Information

ESL ACCUPLACER Writing Pilot Project

4 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate

328345


Notice: fwrite(): send of 201 bytes failed with errno=32 Broken pipe in /home/readbag.com/web/sphinxapi.php on line 531