Read Microsoft PowerPoint - Portfolio Assessment 6-22-08-ea changes.ppt text version

Portfolios: A Nontraditional Approach to Assessment of Competency

ADEA CCI Liaisons Conference June 23-24, 2008 23Chicago, IL

Cynthia C. Amyot, BSDH, Ed.D. University of Missouri-Kansas City MissouriSchool of Dentistry

Cynthia C. Gadbury-Amyot GadburyProfessor and Director Distance Education & Faculty Development UMKC School of Dentistry, 650 E. 25th Street, KC, MO 64108 Phone: 816-235-2054 816- 235Email: [email protected]

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

1

Objectives

· Provide rationale for the "why" behind using nontraditional assessment measures in competency-based education. · Compare and contrast traditional faculty-centered assessment measures with portfolios as a studentcentered nontraditional assessment measure. · Discuss a systematic approach (how to) for implementing portfolio assessment along with methods for evaluation. · Discuss barriers to implementing portfolio assessment of student competency and strategies for overcoming potential barriers.

Knowledge-based Society

· The knowledge worker of the future will be that person who has the capacity to continually learn new concepts and modes of operations through critical thinking and problem solving · The knowledge worker will have the "tools" to make evidence-based decisions · The knowledge to be acquired will be practical in nature and specialized in scope (Drucker, 1994)

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

2

The GOOD News!

· The knowledge worker's performance depends on formal education and theoretical knowledge.

­ Education will become the center of the knowledge society and the school its key institution!

The BAD News!

Average reading literacy scores of 15-year-olds, by country: 2000

Finland Canada New Zealand Australia Ireland Korea United Kingdom Japan Sweden Austria, Belgium, Iceland Norway, France United States NCES, 2006 546 534 529 528 527 525 523 522 516 507 505 504

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

3

The BAD News!

Average math literacy scores of 15-year-olds, by country: 2003

Finland Korea Netherlands Japan Canada Belgium Switzerland Australia New Zealand Czech Republic

Iceland, Denmark, France, Sweden, Austria, Germany, Ireland, Slovak Republic, Norway, Luxembourg, Poland, Hungary, Spain

544 542 538 534 532 529 527 524 523 516 515-485 483

United States NCES, 2006

The BAD News!

Average science literacy scores of 15-year-olds, by country: 2003

Finland, Japan Korea Australia Netherlands Czech Republic New Zealand Canada Switzerland France Belgium 548 538 525 524 523 521 519 513 511 509

Sweden, Ireland, Hungary, Germany, Poland, Slovak Republic, 506-495 Iceland United States NCES, 2006 491

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

4

· The process of increasing metacognition, self-awareness, and responsibility for one's learning is largely atypical in American education, where students frequently do only what they need to get by and instructors coerce them with the threat of poor grades

Burch, 1999

Bottom Line

· Era of assessment and accountability in education · Citizens are concerned over....

­ Financial well-being ­ Economic security

· Pressure has increased on those individuals (faculty) and institutions (education) that might influence the outcome

· Resistance by education/educators is seen as suspect

Campbell et al, 2007; Palomba & Banta, 1999

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

5

Commission on Dental Accreditation

· Standards for Predoctoral Dental Education

­ 2-7 The dental school must define the competencies needed for graduation, which must be focused on educational outcomes. ­ 2-8 The dental school must employ student evaluation methods that measure the defined competencies.

Assessment - Operationalized

· Assessment

­ Process that focuses on student learning and involves review ­ Reflects on practice in a planned and careful way

· Palomba & Banta believe overarching goal of assessment in higher education should be to enable educators to examine whether the curriculum makes sense in its entirety and whether students, as a result of their collective experiences in education, have the knowledge, skills and values that graduates should possess

Palomba & Banta, 1999

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

6

Traditional Assessment Measures In Dental Education

· · · · GPA Paper and Pencil Tests National Dental Board Examination Regional Clinical Exams

Competency

· Competent students have been defined as those individuals who are qualified to begin working independently because they can think critically and apply critical thinking skills to today's complex work environment.

Chambers & Glassman, 1997

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

7

So Why Go Beyond Traditional Assessment ­ Why Non-traditional Assessment Measures??

· If the focus of an academic program is on the creation of products or performances, then assessment should draw on these natural results of the educational process (Palomba & Banta, 1999)

Assessment of Competency

· Palomba and Banta state that authentic (nontraditional) assessment is the process of using student activities or products, as opposed to tests or surveys, to evaluate students' knowledge, skills, and development. · In academic programs like dentistry that develop complex, integrated skills, this form of assessment is appealing because it requires students to display their skills in a way that is more direct and thorough than that provided by traditional paper and pencil tests.

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

8

Assessment of Competency

· Research has suggested that the evaluation of competency is best attained through the use of authentic assessment

­ Chambers suggests that portfolios are one form of authentic assessment where actual examples of student work is displayed

Chambers & Glassman, 1997 Wiggins, 1993

Some Paradoxes in CompetencyBased Education (JDE, 2008)

· Study concluded:

­ Some dental educators have adopted the learner-centered view of education required of competency-based education, while others have adopted only some of the language while retaining a teacher- or disciplinecentered view

Licari & Chambers, 2008

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

9

Some Paradoxes in CompetencyBased Education (JDE, 2008)

· The concept of competency-based education understood and valued among administrators and others responsible for curriculum design and innovation · Faculty members and students were less familiar with the concept......

Think Aloud!

· Rationale for non-traditional, authentic assessment measures

­ Your thoughts, questions????

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

10

Portfolios - Operationalized

· A focused purposeful collection of student work that documents evidence of traditional and nontraditional sources of student learning, progress, and achievement over time (Arter & Spandel, 1992;

MacIsacc & Jackson, 1994)

· The construction of a portfolio involves gathering a body of evidence of one's learning and competence (Lyons, 1988)

­ Student-centered versus faculty-centered

Portfolio Pedagogy

· Portfolio pedagogy is characterized by two processes (Yancy, 1992):

­ Reflection ­ Inquiry

· Reflective portfolio use stimulates students to develop new understanding and appreciation of their experiences, recognize links between different aspects of these experiences and formulate insights to be tested in future actions · Constructivist Learning Theory (Fosnot, 1997)

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

11

The "How To" of Portfolio Assessment of Student Competency

Paramount

· Needs a champion(s) at your institution · Needs the support of administration

­ Kneka Smith, M.P.H. Associate Dean for Education ­ John W. Thurmond, D.D.S., M.S., Director, AEGD

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

12

Steps 1.

What To Do

What It Improves

Determine the programmatic/clinical Validity competencies (Determine what students need to know by the end of the educational program) Determine how competencies are assessed, what courses cover what (Create a table) Establish criteria standards and rubrics Validity

2.

3.

Reliability

4.

Offer choices of ways to demonstrate competencies (Portfolio table) Obtain multiple measures of each competency (Portfolio table)

Fairness

5.

Reliability & Validity

Process

· Introduce portfolio project early in curriculum

­ Completion of some competencies prior to final semester when portfolio is due

· Deadlines help improve overall quality of portfolio · Faculty advising throughout process

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

13

Capstone Project UMKC

Dental Hygiene Program

· Final grade for portfolio incorporated into Senior Seminar course (45% of grade)

Development of Portfolios

· Students should choose work which stands as evidence of attainment of the program competencies ­ Just as a curriculum vitae or resume, all claims made in the portfolio MUST be supported by evidence · Faculty have provided a table (Portfolio Table can be found in article...) which lists required items of inclusion and several ideas for additional portfolio entries based on course and clinic assignments, projects, testing, etc.

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

14

UMKC Portfolio Format

· Notebook format · Includes a Table of Contents · An introduction is necessary to orient the reader ­ Purpose statement

UMKC Portfolio Format

· Notebook divided into sections, one for each competency · Each section begins with listing the specific competency and an explanation of the items included in that section · Self-reflection follows at the end of each section discussing how the items included in that section demonstrate attainment of that particular competency ­ No statements should be made that aren't supported by evidence in terms of items

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

15

Development of and Demonstration of Critical Thinking and Problem Solving Skills

· Reflection requires critical thinking and problem solving skills

­ ADEA CCI ­ IOM Report ­ Etc.

Reflection

· Reflection is believed to focus on professional attributes and offer directions for improvement by identifying strengths and weaknesses;

Driessen et al., 2005

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

16

Reflection ­ What is gained

· In one study, mentors saw reflection as a method of identifying causes in order to answer the question of why things are as they are.

Driessen et al., 2005

· Mentors see writing reflective reports as fostering critical attitudes toward one's own performance and offer directions for improvement;

Driessen et al., 2005

How to Stimulate Reflection

· Stimulate reflection with Favorable Conditions:

­ Good introduction ­ Student ownership ­ Clear structure ­ Appropriate use

Wade and Yarbrough, 1996

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

17

UMKC Portfolio Format

· A global self-reflection concludes the portfolio ­ overall evaluation of what the student has learned ­ how this new knowledge applies to their future goals ­ evidence of growth during educational process ­ Insight into life long learning ­ etc.

Hallmark of Competence

· Chambers states that one of the characteristics of a competent individual is the capacity to accurately assess competence

Licari & Chambers, 2008

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

18

Barriers

· · · · Concerns related to validity and reliability Time Student Resistance Faculty Resistance

Validity and Reliability Concerns

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

19

Grading Rubric

· Allows assessment to be more objective and consistent · Clarifies assessment criteria in specific terms · Shows the student how their work will be evaluated and what is expected · Promotes student awareness · Provides useful feedback regarding effectiveness of instruction · Provides benchmarks against which to measure and document progress

Primary Trait Analysis - Rubric

Description of Portfolio Traits 1. Is the student able to illustrate growth and professional development to the reader? Components · llustrates continued development and growth over time, i.e., ability to read, analyze and apply scientific literature in decision making process · Demonstrates increased use of professional language over time · Illustrates heightened professionalism, humanitarianism and ethical behavior Comments Rating Not at Some all 1 2 Most All

3

4

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

20

Excellent Resource For Creating Rubrics

· By Stevens and Levi

· http://www.introductiontorubrics.com/

Rubric ­ Another Example

Excellent Average Weak

Preliminary lit search

-All possible primary sources listed -All possible secondary sources are listed -Formatting follows assigned style

-Some primary sources listed -Some secondary sources listed -Includes all relevant info but follows no format

-No primary sources listed -No secondary sources listed -No recognizable format style and information partial

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

21

Performance Assessment

· Two general conclusions about inter-rater reliability predominate

­ When tasks are the same for all students and scoring procedures well defined, inter-rater reliability tends to be high (Dunbar, 1991; Gadbury-Amyot et al., 2003) ­ When different students respond to different tasks, select their own task, or produce unique products, inter-rater reliability tends to be low (Koretz et al., 1994)

Inter-rater Reliability

· Faculty accounted for very little variability or error (1.28 percent) in portfolio measurement demonstrating that faculty members were consistent and calibrated in their use of the scoring rubric. Gadbury-Amyot, 2003 · The fact that very little error is attributed to faculty raters has been reported throughout the portfolio literature in both small- and large-scale studies. Nazier, 1997; LeMahieu, 1995

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

22

Performance Assessment

· Carefully constructed scoring rubrics and intensive training session for raters are essential elements for producing ratings that are reliable (Brennan, 2000)

Validity and Reliability of Portfolio Assessment of Competency

· "A single score obtained on one occasion on one test with a single administrator, is not dependable (valid or reliable)."

­ This is what makes portfolio assessment of competency such a dependable measure!! (Shavelson and Webb, 1991)

· Gadbury-Amyot et al. (2003)

­ Only work currently in dentistry looking at this very important aspect

· Generalizability coefficient of .81

­ Analogous to reliability coefficient

· Theoretical and empirical evidence for validity using Messick's unified framework

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

23

Predictive Validity of DH Competency Assessment Measures

· Gadbury-Amyot et al. (2005)

­ Predictive validity of dental hygiene competency assessment measures on one-shot clinical licensure examinations (CRDTS)

· Explore the ability of four variables (GPA, Clinic GPA, NBDHE, Portfolios) to predict one-shot clinical licensure exam scores. · Results

­ Two-factor solution obtained (DH cognition and DH clinical performance) ­ Factor scores used in a linear predictive model to exam ability of predicting outcome on clinical licensure exam ­ Shared contribution of both factors only accounted for 13.9% ­ Disturbing lack of concordance between previously validated or accepted measures of dh student competency (GPA, NBDHE, Portfolios) and one-shot clinical licensure exam

Time As A Barrier

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

24

Student and Faculty Resistance As A Barrier

Large-Scale Portfolio Assessment

· Exemplary Examples

­ Truman State University ­ Portfolio Project

· http://assessment.truman.edu/components/portfoli o/

­ Alverno College ­ Diagnostic Digital Portfolio

· http://ddp.alverno.edu/

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

25

Accountability Measures for Practicing Dentists

· Continuing Competency/Quality Assurance

­ Minnesota

· http://www.dentalboard.state.mn.us/ · http://www.dentalboard.state.mn.us/Portals/3/SelfA ssessment/SelfAssessment2006.pdf

Objectives Revisited

· Provide rationale for the "why" behind using nontraditional assessment measures in competency-based education. · Compare and contrast traditional faculty-centered assessment measures with portfolios as a studentcentered nontraditional assessment measure. · Discuss a systematic approach (how to) for implementing portfolio assessment along with methods for evaluation. · Discuss barriers to implementing portfolio assessment of student competency and strategies for overcoming potential barriers.

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

26

References

· · · · · · · · · · · Arter, JA & Spandel, V. (1992). Using portfolios of student work in instruction and assessment. Educational Measurement, 11(1), 36-44. Boud, D., Keogh, R. and Walker, D. Reflection: Turning Experience into Learning. London: Kogan Page; 1985. Burch, CD (1999). Inside the portfolio experience: The student's perspective. English Education, 32(1): 34-39. Campbell JP, DeBlois PB, and Oblinger DA. (2007). Academic analytics a new tool for a new era. Educause Review:41-57. Chambers, DW & Glassman, P. (1997). A Primer on Competency-Based Evaluation. J Dent Educ, 61(8), 651-666. Driessen, E.W., van Tartwijk, J., Overeem, K., Vermunt, J.D. and van der Vleuten, C.P.M. Conditions for successful reflective use of portfolios in undergraduate medical education. Med Educ 2005;39:1230-1235. Drucker PF. (1994) The age of social transformation. The Atlantic Monthly, 274 (Nov.):53-80. Fosnot, CT. (1996). Constructivism: Theory, perspectives, and practice. New York: Teachers College Press. Friedman TL. The world is flat (updated and expanded): a brief history of the twenty-first century. 1st ed. New York: Farrar, Straus and Giroux, 2006. Gadbury-Amyot C, Kim J, Palm R, Mills GE, Noble E, Overman P. Validity and Reliability of Portfolio Assessment of Competency in a Baccalaureate Dental Hygiene Program. J Dent Educ, 67(9): 991-1002, 2003. Gadbury-Amyot C, Holt L, Overman P, Schmidt C. Implementation of Portfolio Assessment in a Competency-Based Dental Hygiene Program. J Dent Educ, 64(5):375-380, May 2000.

References

· · · · · · · · · · Lemke M and Gonzales P. (2006). U.S. students and adult performance on international assessment of educational achievement: Findings from the condition of education 2006 (NCES 2006-073). U.S. Department of Education, Washington, DC: National Center for Educational Statistics Licari FW & Chambers DW. (2008). Some paradoxes in competency-based dental education. J Dent Educ, 72(1):8-18. Lyons, N. (1988). Constructing narratives for understanding: Using portfolio interviews to scaffold teacher reflection. In N. Lyons (Ed.), With Portfolio in Hand: Validating the New Teacher Professionalism. New York: Teachers College Press. MacIsacc, D. & Jackson, L. (1994). Assessment process and outcomes: Portfolio construction. New Directions for Adult and Continuing Education, 62, 63-72. Palomba, C., & Banta, T. (1999). Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education. San Francisco: Jossey-Bass. Pink D. (2005) A Whole New Mind: Moving From the Information Age to the Conceptual Age. Riverhead Hardcovers. Stevens DD, Levi AJ. (2005). Introduction to rubrics. Stylus Publishing, Sterling, Va. Wade, R.C., and Yarbrough, D.B. Portfolios: A Tool for Reflective Thinking in Teacher Education? Teaching and Teacher Educ 1996;12(1):63-79. Wiggins, G. (1993). Assessment: Authenticity, context, and validity. Phi Delta Kappan, 75(3), 200214. Yancy, KB. (1992). Teachers' stories: Notes toward a portfolio pedagogy. In K. B. Yancy (Ed.), Portfolios in the writing classroom: An introduction (pp.12-20). Urbana: National Council of Teachers in English.

This material is the intellectual property of Cynthia C. Amyot, Ed.D. ([email protected]).

27

Information

Microsoft PowerPoint - Portfolio Assessment 6-22-08-ea changes.ppt

27 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate

949964