Read Microsoft PowerPoint - Keynote dibels composite score and benchmark goals handout.pptx text version

Developing Benchmark Goals and Cut-points for Risk: Odds f Achieving Subsequent Odd of A hi i S b t Reading Goals

DIBELS Summit Albuquerque, Albuquerque NM February 15, 2011

Roland H. Good III Dynamic Measurement Group, Inc. University of Oregon Kelly Powell-Smith Ruth A. Kaminski Dynamic Measurement Group, Inc Group Inc..

Significant Advances in Reading Assessment to Inform Instruction

1. Research Based DIBELS Composite Score. The DIBELS Composite Score combines multiple DIBELS Next scores into a single composite th t b t predicts and measures i l it that best di t d outcomes. 2. Research Validated Benchmark Goals and Cut Points for Risk. Benchmark goals and cut points for risk are empirically validated based on the odds of achieving future reading g goals. 3. Extraordinary Control of Text Readability. Passages at each grade level are developed, researched, and arranged to provide maximum control of text difficulty. difficulty 4. DIBELS Instructional Grouping Worksheets, DIBELS Survey. 5. And, First Sound Fluency replaces Initial Sound Fluency; NWF-Whole Words Read and Daze are added. And more!

1

January 14, 2011 Hershey, PA

February 15, 2011

Albuquerque, NM

2

Benchmark Goal Study Research Q R h Questions ti

· The Benchmark Goal Study was designed to address three research questions:

1. What levels of performance on DIBELS Next assessments predict a student is likely to score at or above the 40%ile on selected outcome measures? 2. What levels of performance on DIBELS Next assessments predict a student is unlikely to score at or above the 40%ile on selected outcome measures? 3. 3 What are the correlations between DIBELS Next assessments and the Group Reading Assessment and Diagnostic Evaluation (GRADE), a criterion measure of reading proficiency that includes comprehension?

February 15, 2011 Albuquerque, NM

Participants

· · · Students recruited for the study were from 13 schools in five school districts representing five US regions. Participating school districts had a median of 10 years experience using DIBELS. Kindergarten through 6th grade students participated in DIBELS Next assessments (n = 3,816 total; 433 to 569 per grade). The percentage at benchmark ranged from 65% 79% across grades and times of year. Subsamples of students participated in testing with an external criterion measure (Group Reading Assessment and Diagnostic Evaluation; GRADE) (n = 1257 total; 103 to 219 per grade). The GRADE subsample was 50% female on average across grades grades.

Albuquerque, NM

·

3

February 15, 2011

4

Participant Demographics

Participant Demographics

Figure 1 R i l/Eth i B k Fi 1. Racial/Ethnic Background d 5

Figure 2: Parent-Reported Level of Education

February 15, 2011 Albuquerque, NM

February 15, 2011

Albuquerque, NM

6

Participant Demographics

Measures: DIBELS Next

· The measures included all DIBELS Next assessments. DIBELS Next assessments include: · Letter Naming Fluency · First Sound Fluency · Phoneme Segmentation Fluency · Nonsense Word Fluency Correct Letter Sounds and Whole Words Read · Oral Reading Fluency Words Correct, Accuracy, and Retell. · Daze Adjusted Score (DIBELS-maze)

Figure 3: Parent-Reported Household Income

February 15, 2011 Albuquerque, NM

· DIBELS Composite Score

7

February 15, 2011 Albuquerque, NM

8

Measures: Group Reading Assessment and Diagnostic E l ti (GRADE) Di ti Evaluation

· · Un-timed and group administered. Appropriate for students in preschool through grade 12 Five components and 16 subtests. Subtests combine to form the following composites: · Phonemic Awareness, Early Literacy Skills, Comprehension,

Vocabulary, and Total Test. · We used the Total Test Raw Score for analyses.

Procedures: Data Collection

· All Data were collected during the 2009-2010 school g year

· DIBELS Next assessments were administered at regular benchmark intervals by trained school personnel using standardized procedures. · GRADE testing occurred in the spring at the end of the G f year and was conducted across two to three sessions. Total testing time ranged from 60 to 90 minutes. The g g GRADE was administered by trained school personnel and onsite coordinators.

·

The GRADE has excellent reliability and validity for its y y intended purposes.

· Reliability ranges from .77 to .98. · Correlation coefficients range from .69 to .86 with other groupand individually-administered achievement tests.

February 15, 2011

Albuquerque, NM

9

February 15, 2011

Albuquerque, NM

10

DIBELS Composite Score

· The DIBELS Composite Score is an important advance in DIBELS Next and a cornerstone of DIBELS Next Benchmark Goals and Cut Points for Risk Risk. · The DIBELS Composite Score is based on an improper linear model described by Robyn Dawes in a 1979 article titled The Robust Beauty of Improper Linear Models in Decision Making. ­ Identify variables that are strongly related to an important outcome. ­ Standardize each variable so it has a mean of 0 and a standard deviation of 1. ­ Sum or average the variables to obtain a composite.

Dawes, R. (1979). The Robust Beauty of Improper Linear Models in Decision Making. American Psychologist, 34, 571-582.

February 15, 2011 Albuquerque, NM

Robyn Dawes (1979) Unit-Weighted Improper Linear Model (ILM)

Unit-Weighted Improper Linear Model

ILM

X 1i X 1 X 2 i X 2 X 3 i X 3 X 4 i X 4 s1 s2 s3 s4

Unit-Weighted Improper Linear Model with Rearranged Terms

s1 ILM constant X 1 i

s1 s s X 2i 1 X 3i 1 X 4i s2 s3 s4

Where: X1 = DORF Words Correct X2 = DORF Accuracy X3 = DORF Retell X4 = Daze

11

February 4, 2011 Coronado, CA

12

For Example: Third Grade, Beginning of Year DIBELS Composite Score

s1 ILM constant X 1 i 4.46 X 2 i 2.02 X 3 i 5.76 X 4 i DCS X 1 i accuracy _ value 2 X 3 i 4 X 4 i l

· The DIBELS Composite Score is a scale transformation p of an improper linear model with all the predictive value and generalizability of the improper linear model. · We used integer weights to approximate the improper sed eights appro imate linear model weights. · We used a lookup table for DORF Accuracy to p y approximate the improper linear model weights.

February 15, 2011 Albuquerque, NM

Improper Linear Model Weight for Daze

· For DIBELS Daze the improper linear model weight varies between 3.8 and 5.8 for different grades 58 and times of year. · An integer value of 4 is a reasonable approximation and fit for all grades and times of year year.

Improper Linear Model Weight for Daze 5.76 4.62 4.37 5.53 5.32 5 32 4.37 4.53 4.60 3.79 4.45 3.85 3.99 3 99 4.49

Grade 3 3 3 4 4 4 5 5 5 6 6 6

Time of Year Beginning of Year Middle of Year End of Year Beginning of Year Middle of Year Middle of Year End of Year Beginning of Year Middle of Year End of Year Beginning of Year Middle of Year End of Year End of Year Median

13

February 15, 2011

Albuquerque, NM

14

DORF Accuracy Lookup Values V l

· The DORF Accuracy Lookup Values y p approximate the improper linear model weights for 85% accuracy through 100% accuracy. accuracy · Accuracy below 85% is treated as unacceptable accuracy, essentially contributing 0 to the DIBELS Composite Score.

Why not use a Proper Linear Model?

· In a proper linear model, variables are combined into a composite using weights that are optimized to provide the "best" composite. · For example, a linear regression model predicting end of year GRADE Total Raw score is a proper linear model. ­ Weights are optimized to provide the "best" prediction of GRADE Total Raw Score · "best" is the maximum explained variance · For the target measure: GRADE Total Raw Score · For the specific sample of subjects used to estimate the model.

February 15, 2011

Albuquerque, NM

15

February 15, 2011

Albuquerque, NM

16

Why not use a proper linear model?

· Dawes (1979) argues that an improper linear model can provide a better, more generalizable prediction for: ­ different ways of defining "best" including best best", discrimination of groups (i.e., linear discriminant function analysis), best representation of total variance in the variables (i.e., principle components analysis), best representation of shared variance (i.e., factor analysis), etc. ­ different measures of the same or closely related constructs, or ­ different sample of subjects. · The goal of the DIBELS Composite Score is generalized prediction for many ways of defining "best", many reading best measures, and many different groups of students.

February 15, 2011 Albuquerque, NM

Comparison of Linear Regression Model and DIBELS Composite Score

· · Target Measure: The GRADE Total Raw Score administered works The DIBELS Composite Score at end of year was used to estimate weights in a the generalization measure. g better for linear regression model. Generalization Measure: The endAs an improper linear model it isScore of year DIBELS Composite designed to (reading proficiency) is a different measure of the same construct work better for other measures as well. that was not used to estimate weights in the linear regression model.

The proper linear model works better for the target measure ­ it has to. But the DIBELS Composite Score works almost as well. well Linear Regress Model predicting GRADE Total Raw Score (Target measure) from beginning of year DIBELS Next measures Beginning of year DIBELS Composite Score

Target g Measure .78

Generalization Measure .86

.73

.88

17

February 15, 2011

Albuquerque, NM

18

Why Should I use the DIBELS Composite Score? S ?

· Why can't I just use DORF Words Correct? I have been can t using DIBELS a long time and DORF Words Correct works very, very well. ­ Research Rationale · DIBELS Composite Score works even better. ­ Ed Educational R ti ti l Rationale l · DIBELS Composite Score represents a range of different reading behaviors required for overall reading proficiency.

Research Rationale R ti l

· DIBELS Composite Score explains more variance in reading outcomes than DORF Words Correct alone. · Median 9% more, range 3% to 17%. · DORF Words Correct alone is good, DIBELS Composite Score is better.

19

February 15, 2011

Grade and Time of Year Grade 1 Middle of Year Grade 1 End of Year Grade 2 Beginning of Year Grade 2 Middle of Year Grade 2 End of Year Grade 3 Beginning of Year Grade 3 Middle of Year Grade 3 End of Year Grade 4 Beginning of Year Grade 4 Middle of Year Grade 4 End of Year Grade Beginning f Year G d 5 B i i of Y Grade 5 Middle of Year Grade 5 End of Year Grade 6 Beginning of Year Grade 6 Middle of Year Grade 6 End of Year

Albuquerque, NM

DORF Words Correct Predicting GRADE Total 0.64 0.75 0.69 0.76 0.73 0.66 0.67 0.66 0 66 0.76 0.76 0.75 0.69 0 69 0.64 0.66 0.64 0.59 0.61

DIBELS Additional Composite Variance Score Explained Predicting by DIBELS GRADE Composite Total Score 0.70 8% 0.77 4% 0.75 8% 0.80 5% 0.75 3% 0.73 10% 0.78 15% 0.75 0 75 13% 0.80 5% 0.80 6% 0.80 8% 0.76 0 76 11% 0.76 17% 0.77 17% 0.71 9% 0.68 12% 0.73 16%

February 15, 2011

Albuquerque, NM

20

Educational Rationale

Reading at an adequate rate. Reading orally for meaning. Reading silently for meaning. With a high degree of accuracy.

Educational Rationale

· What we choose to assess communicates to students what we want them to learn, communicates to teachers what we want them to teach, and communicates to administrators what we want them to prioritize prioritize. · We think whenever we ask a student to read for fluency and accuracy we should also ask them to engage what they have read for meaning.

Students who are at or above benchmark on the DIBELS Composite Score are reading for meaning at an adequate rate and with a high degree of accuracy accuracy.

February 15, 2011 Albuquerque, NM

21

February 15, 2011

Albuquerque, NM

22

But I just don't think Retell is measuring R di C i Reading Comprehension. h i

· Research Rationale ­ Strong evidence that DORF Retell is measuring reading comprehension in a reliable and valid way. · Educational Rationale ­ Retell is a comprehension skill. ­ Equally applicable to narrative text and expository text. ­ Cannot be guessed with high background knowledge, g g g g , vocabulary, or general verbal reasoning skills.

Research Rationale R ti l

· Strong, robust, and g, , stable correlations with GRADE Reading Comprehension outcomes in the .50s to .60s. · About as strong as any two high quality measures of reading comprehension. · Inter-rater reliabilities of .92 to .99 for median of three forms.

23

February 15, 2011

Grade and Time of Year Grade 1 Middle of Year Grade 1 Middle of Year Grade 1 End of Year Grade 2 Beginning of Year Grade 2 Middle of Year Grade 2 End of Year G d 2E d f Y Grade 3 Beginning of Year Grade 3 Middle of Year Grade 3 End of Year Grade 4 Beginning of Year Grade 4 Middle of Year Grade 4 End of Year Grade 5 Beginning of Year Grade 5 Beginning of Year Grade 5 Middle of Year Grade 5 End of Year Grade 6 Beginning of Year Grade 6 Middle of Year G d 6 Middl f Y Grade 6 End of Year

Albuquerque, NM

Correlation of Retell with GRADE GRADE Comprehension Composite Total 0.55 0 55 0.57 0 57 0.40 0.41 0.53 0.53 0.54 0.54 0.52 0 52 0.52 0 52 0.53 0.55 0.57 0.60 0.53 0.57 0.59 0.56 0.62 0.60 0.63 0.61 0.61 0 61 0.59 0 59 0.63 0.60 0.65 0.64 0.55 0.48 0.59 0 59 0.56 0 56 0.56 0.51

February 15, 2011

Albuquerque, NM

24

Educational Rationale

· Reading Comprehension involves many skills

Educational Rationale

From Big Ideas in Beginning Reading, University of Oregon, 2002-2004

From Big Ideas in Beginning Reading, University of Oregon, 2002-2004

· Retell is one. A foundational one. If you can't talk about what you have j t read, you cannot b ild more h t h just d t build advanced skills.

25

February 15, 2011 Albuquerque, NM

February 15, 2011

Albuquerque, NM

26

DIBELS Composite Score Summary

· For each grade and time of year, the DIBELS Next measures that correlate highly with later outcomes are combined into a DIBELS Composite Score. Each measures is weighted so that all contribute approximately equally to the DIBELS Composite Score. ­ Weighted scores have approximately equal standard deviations. g pp y q The DIBELS Composite is constructed to highly correlate with a broad, general range of reading outcomes. The DIBELS Composite Score represents a rich and broad sample of behavior.

For Example: Third Grade DIBELS Composite Score. Benchmark Goal: 220

DORF Words Correct

99 78 64 104 345

·

39 16 98

· ·

· The DIBELS Composite Score conveys that all of the aspects of reading proficiency are critical ­ a student whose DIBELS Composite Score is At or Above Benchmark is reading accurately, at an adequate rate, and attending to the meaning of the passage.

February 15, 2011 Albuquerque, NM

27

February 15, 2011

Albuquerque, NM

28

What is the Purpose of Benchmark Goals and S d Screening f Ri k i Ed i for Risk in Education? ti ?

Different standards, procedures, and requirements are necessary if our purpose is:

Fourth Grade Reading Outcomes on the 2007 National Assessment of Educational Progress National (public N ti l ( bli school) percent of fourth grade students scoring below (pp. 16, 52-53) 34% Nation ( bli ) N ti (public) percent t of fourth grade students from diverse backgrounds scoring below (pp. 54 & 57) 54%, 51%, 49%, 50%

1. To quickly identify students that are likely to need additional support to prevent later academic difficulty. · To specify important and meaningful future goals--a level of skills at a point in time where we can change the odds to being in favor of an individual's meeting subsequent goals.

2. To accurately identify students who are the true Tier 3 students or who have a true learning disability early.

We are troubled by the purpose of identifying true Tier 3 students. We think the future i not set. Ti 3 i thi k th f t is t t Tier is not a characteristic of the student. There are no true Tier 3 students. pp Tier 3 is a level of support necessary for the student to make adequate progress. No fate but what we make.

Skill Level Basic

Skill level definition Basic denotes partial mastery p y of prerequisite knowledge and skills that are fundamental for proficient work at a given g grade.

Proficient Proficient represents solid academic performance. Students reaching this level have demonstrated competency over challenging subject matter.

68%

86%, 83%, 80%, 83%

Our purpose is to prevent reading difficulty and enhance reading outcomes by providing targeted, differentiated instruction early.

February 15, 2011 Albuquerque, NM

29

Note: Students from diverse backgrounds includes students identified as Black, Hispanic, American Indian/Alaska Native, and eligible for free/reduced-price school lunch. From DIBELS Summit 30 data reported in Lee, Grigg, & Donahue (2007).

Goal: Adequate Reading Skills

· Adequate reading skills should generalize across different state, national, and published reading tests. · Ad Adequate reading skills are not a normative d i i t di kill t ti decision, b t but are a socio-political judgment. · The 40th percentile or above on a high q p g quality, nationally y, y norm-referenced test can serve as an approximation for adequate reading performance. · St d t at or above th 40th percentile on a hi h Students t b the til high quality, nationally norm-referenced test are on track to be rated Basic or above on NAEP. · We used the Group Reading Assessment and Diagnostic Evaluation (GRADE) in our initial research to provide an initial approximation of adequate reading skills.

February 15, 2011 Albuquerque, NM

Third Grade DIBELS Composite Score for End of Year (DCS3e) and GRADE Total Raw Score (gtotr3e)

· Group Reading Assessment and Diagnostic Evaluation (GRADE) is a high quality nationally norm referenced test quality, norm-referenced test. · .75 correlation to GRADE Total Raw Score at End of Year

97 87

40th Percentile on GRADE

gtotr3e

77 67 57 47 37 50 150 250 350 DCS3e

End f E of Y 450 d 550Year

DIBELS C Composite it Score Benchmark Goal: 330 32

31

February 15, 2011

Albuquerque, NM

Building Futures by Changing Odds Outcomes Driven Decisions

Evidence Base, Score Level, Likely Need for Support

Outcome: Looking back Screening: Looking forward

Odds f hi i Odd of achieving Likely Lik l need for support to df tt subsequent early achieve subsequent early literacy goals Score level literacy goals Likely to Need Core Support 80% to 90% At or Above Benchmark scores at or above th t b the benchmark goal Likely to Need Strategic Support 40% to 60% Below Benchmark scores below the benchmark goal and at or b h k l d t above the cut point for risk Likely to Need Intensive Support 10% to 20% Well Below Benchmark scores below the cut point for i k f risk

Beginning of Year Benchmark

Beginning of Year Cut Point for Risk

Sep

Oct

Nov

At or Above Benchmark Odds are 80% to 90% g q of Achieving Subsequent Early Literacy Goals Below Benchmark Odds are 40% to 60% of Achieving Subsequent Early Literacy Goals Well Below Benchmark Odds are 10% to 20% of Achieving Subsequent Early Literacy Goals

Dec Jan Feb Mar

End of Year Benchmark Goal

End of Year Cut Point for Risk

Apr

May

The fundamental rationale for benchmark goals and screening decisions is based on th odds of achieving subsequent early lit i b d the dd f hi i b t l literacy goals. l

Albuquerque, NM

February 15, 2011

33

February 15, 2011

Albuquerque, NM

34

Primary Design Specifications for DIBELS Goals and Cut Points for Risk

· Primary Specification: At or Above Benchmark Decision on initial (screening) DIBELS assessment should provide favorable odds (80% -- 90%) of achieving subsequent reading outcomes outcomes. Benchmark Goal should provide a level where we are reasonably confident the student is making adequate progress. Below Benchmark Decision on initial DIBELS assessment should provide 50 ­ 50 odds (40% -- 60%) of achieving subsequent reading outcomes. Below the Benchmark Goal but above the Cut Point should provide a zone of uncertainty where we don't know if the p y student is making adequate progress or not. Well Below Benchmark Decision on initial DIBELS assessment p ( ) g q should provide low odds (10% -- 20%) of achieving subsequent reading outcomes ­ unless intensive intervention is implemented. Below the Cut Point should provide a zone where we are reasonably confident the student will not make adequate progress -- unless we provide additional support.

Albuquerque, NM

Secondary Specifications for Benchmark G l and C t P i t B h k Goals d Cut Points

· Marginal p g percents for the p predictor close to marginal g percents for the outcome. ­ The sample for the Benchmark Goal Study was a relatively hi h performing sample. l ti l high f i l ­ We tried have them appear equally high performing on DIBELS Next and the GRADE. · Logistic Regression Analysis ­ Logistic regression predicted odds of about 60% or better at the exact goal score. ­ Logistic regression predicted odds of about 40% or below at the exact cut point for risk score score.

35

February 15, 2011 Albuquerque, NM

·

·

February 15, 2011

36

Other Considerations DIBELS Goals and Cut Points

· Other considerations ­ Receiver Operator Characteristic Curve (ROC) analysis with large area under curve ­ Other metrics for decision utility · sensitivity, · specificity, · percent correct classification, · kappa ­ Coherent pattern of goals across measures and grades.

Setting Benchmark Goals and Cut Points for Risk

1. Examine scatterplot illustrating the relation between the screening assessment (earlier assessment or predictor) and the outcome assessment (later assessment). ­ DIBELS is a step-by-step model, so the outcome of one step by step step is the predictor of the next step. 2. Examine the table of counts for each zone of the scatterplot. 3. Primary: C 3 Pi Consider odds of students with each screening id dd f t d t ith h i decision achieving goal. 4. Secondary: Consider marginal percents 5. Secondary: Consider logistic regression analysis 6. Other: Consider ROC curve and decision utility metrics 7. Other: C 7 Oth Consider the overall pattern of goals and cut points. id th ll tt f l d t i t

37

February 15, 2011 Albuquerque, NM

February 15, 2011

Albuquerque, NM

38

Example Analysis Detail

Third Grade DIBELS Composite Score for Beginning ( g g (DCS3b) to Middle of Year ( ) (DCS3m) )

608 508

At or Above Benchmark Likely to need intensive support Likely to need strategic support Likely to need core support

308 Below Benchmark 208

Well Below 108 Benchmark

D DCS3m

408

4 20 70

104

22 16 9

204

324 21 4

304 DCS3b 404 504

40

8 4

February 15, 2011 Albuquerque, NM

39

February 15, 2011

Albuquerque, NM

Third Grade DIBELS Composite Score for Beginning (DCS3b) to Middle of Year (DCS3m)

DCS3b Screening Decision:

Likely to need intensive support Likely to need strategic support Likely to need core support

Primary consideration: Odds of achieving goal

DCS3b Screening Decision:

Likely to need intensive support Likely to need strategic support Likely to need core support

DCS3m Outcome:

At or Above Benchmark Below Benchmark Well Below Benchmark Marginal Total M i lT t l

Marginal total

DCS3m Outcome:

At or Above Benchmark Below Benchmark Well Below Benchmark Marginal Total M i lT t l

Marginal total

4 20 70 94

22 16 9 47

324 21 4 349

350 57 83 490

4 20 70 94

22 16 9 47

324 21 4 349

350 57 83 490

· Pi Primary consideration: Odd of achieving outcome goal. id i Odds f hi i l · Secondary consideration: Marginal Percents

February 15, 2011

Albuquerque, NM

41

· Core support beginning of year screening decision: 324 of 349 students achieve the middle of year goal, or 93% odds. · Strategic support: 22 of 47 students achieve the goal, or 47% odds. · Intensive support: 4 of 94 students achieve the goal, or 4% odds.

February 15, 2011 Albuquerque, NM

42

Also Considered Marginal Percents

DCS3b Screening Decision:

Intensive support decision Strategic support decision Core support decision Marginal total Marginal percent

608 Moving Odds: Logistic Regression 508 408 308 208 108

1 0.8

At or Above Benchmark Below Benchmark Well Below Benchmark Marginal Total Marginal Percent

4 20 70 94 19%

22 16 9 47 10%

324 21 4 349 71%

350 57 83 490

71% 12% 17%

· Percent At or Above Benchmark at beginning of year is very close to the percent At or Above Benchmark in the middle of the year. · Desirable for the screening decision to identify about the same percent of students that are expected on the outcome.

February 15, 2011 Albuquerque, NM

DCS3m

DCS3m Outcome:

8 4

0.6 0.4 0.2 0 86 136 186 236 286

104

204

304 DCS3b

404

504

43

February 15, 2011

Albuquerque, NM

44

Logistic Regression Estimates Odds of Adequate Outcomes for each Score O t f hS

· Blue diamonds are moving proportion with adequate ith d t outcome. · Red line is logistic regression estimated odds of adequate outcomes.

February 15, 2011

DIBELS is a Step-by-Step Model: Beginning to Middle; Middle to End; ... g g ; ;

· Mastering each step puts the odds in favor of mastering the next step. ­ At or Ab Above B Benchmark: Odd are generally 80% t 90% h k Odds ll to of achieving subsequent benchmark goals and important reading outcomes. Student is likely to make adequate progress with effective core instruction instruction. ­ Below Benchmark: Odds are generally 40% to 60% of achieving subsequent benchmark goals and important reading outcomes Student is likely to need strategic outcomes. support to make adequate progress. ­ Well Below Benchmark: Odds are generally 10% to 20% of achieving subsequent benchmark goals and important reading outcomes. Student is likely to need intensive support to make adequate progress. Contiguous Continuity. Each step is a continuous process with a Continuity strong linkage. Each step is contiguous with the next step.

Albuquerque, NM

1 0.8 0.6 0.4 04 0.2 0 86 136

60% estimated odds of adequate outcomes for the score exactly at the Benchmark Goal: higher scores, higher odds 30% estimated odds of adequate outcomes for the d t t f th score exactly at the Cut Point for Risk, lower scores, lower 186 odds. 236 286

·

45

Albuquerque, NM

February 15, 2011

46

End of Year Benchmark Goals

40

28 --

58 13

47 90% 15

87 97% 27

100 97% 30

19

*Word Use Fluency--Revised (WUF-R) is available as an experimental measure from http://dibels.org/. For Nonsense Word Fluency, the first number is the Correct Letter Sounds goal, and the second number is the Whole Words Read Goal. For DIBELS Oral Reading Fluency, the first number is the Words Correct goal, the second number is the Accuracy goal, and the third number is the Retell goal. For example, at the end of second grade a student should be able to read 87 words with 97% accuracy and retell of 27 words relevant to the passage. Third grade benchmark goals are illustrated, benchmark goals for grades 4 through 6 are available.

47

February 15, 2011 Albuquerque, NM

48

Third Grade DIBELS Composite Score for Beginning of Year (DCS3b) and Middl of Y fY d Middle f Year (DCS3 ) (DCS3m)

· .91 correlation to DIBELS Composite Score at Middle of Year

4% Odds

608 508

DCS3 3m

Third Grade DIBELS Composite Score for Middle of Year (DCS3m) and E d of Y fY (DCS3 ) d End f Year (DCS3 ) (DCS3e)

· .90 correlation to DIBELS Composite Score at End of Year 43%

8% Odds

612

47% Odds

93% Odds Middle of Year Goal: 285

512 412

DCS S3e

Odds

91% Odds End of Year Goal: 330

408 308 208 108 8 4 104 204 304 DCS3b 404 504

312 212 112 12 12 212 DCS3m 412 612

Beginning of Year Goal: 220

February 15, 2011 Albuquerque, NM

Middle of Year Goal: 285 49

February 15, 2011 Albuquerque, NM

50

Third Grade DIBELS Composite Score for End of Year (DCS3e) and GRADE Total Raw Score (gtotr3e)

· .75 correlation to GRADE Total Raw Score at End of Year 48%

7% Odds

97 87

gtotr3e

Receiver Operator Characteristic Curve

· Larger area under the g curve indicates favorable trade off of sensitivity and specificity. specificity · Decision points in the upper left bend of the curve indicate a favorable balance of sensitivity and specificity. specificity

1.00 .80 60 .60 .40 .20 .00 .00 00 .20 20 Benchmark Goal ROC, AUC = .90 Cut Point for Risk ROC, AUC = .87 .40 40 .60 60 .80 80 1.00 1 00

Odds

90% Odds

40th Percentile on GRADE

77 67 57 47 37 50 150 250 350 DCS3e 450 550

Receiver Operator Characteristic (ROC) curves.

End of Year DIBELS Composite Score Goal: 330

February 15, 2011 Albuquerque, NM

51

February 15, 2011

Albuquerque, NM

52

Other Decision Utility Metrics End f Thi d Grade E d of Third G d

We are troubled by the terminology. We think a "True Positive" is actually a student for whom we were not effective in ruining the prediction. A "False Positive" is P iti " i a student t d t for whom we have changed the future.

At or Above Well Below Benchmark outcome B B h k t Benchmark outcome h k t

Core support decision True Negative False Negative True Positive False Positive Sensitivity Specificity Negative Predictive Pow er Positive Predictive Pow er Accurate Classification Kappa Intensive support decision Core support decision Intensive support decision

Early Intervention and Prevention are Active Ingredients Between Screening and O t I di t B t S i d Outcomes

· The effectiveness of the school-wide system of y instruction can change the odds. ­ Differences in the effectiveness of Tier 1 instruction and Tier 2 & 3 intervention change the underlying relation between screener and outcome. ­ Less effective school-wide system Tier 1 instruction can decrease the odds of achieving subsequent early literacy goals for students who are at or above benchmark. ­I Increasing the effectiveness of Tier 2 & 3 intervention i th ff ti f Ti i t ti can increase the odds of achieving subsequent early literacy goals for students who are at risk.

123 14 37 13 .73 .90 .90 .74 .86 .63

134 26 25 2 .49 .99 .84 .93 .85 .56

132 5 18 32 .78 .80 .96 .36 .80 .39

154 6 17 10 .74 .94 .96 .63 .91 .63

Screening Decision Outcome

February 15, 2011

Role Variable Predictor DCS3e Criterion gtotr3e

Goal 330 83

Cut Point Description 280 DIBELS Composite Score, Grade 3, End of Year 71 GRADE Total Test, Grade 3, End of Year

Albuquerque, NM

53

February 15, 2011

Albuquerque, NM

54

Building Futures

· Key Point: The student's outcome is unknown and not fixed at the time of the screening. Instead, the outcome is the result of the targeted differentiated instruction and targeted, intervention we provide as a direct result of the screening information. · Our instructional goal is to ruin screening predictions · For Example: If a child screens as at high risk on a measure of early literacy skills in Kindergarten, we know Kindergarten they are likely to need additional instructional support to be successful. Their later outcome, their reading skills in first grade for example, are a direct result of the targeted, differentiated instruction and early intervention that we provide.

February 15, 2011 Albuquerque, NM

55

Information

Microsoft PowerPoint - Keynote dibels composite score and benchmark goals handout.pptx

14 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate

238887