Read PCPI Position Statement: Evidence Base Required text version

APPROVED BY THE PCPI, JUNE 26, 2009

Physician Consortium for Performance Improvement® (PCPI) Position Statement The Evidence Base Required for Measures Development

Table of Contents

Frequently Asked Questions............................................................................................................ Purpose ................................................................................................................................... Summary of Recommendations....................................................................................................... Context and Need for this Position Statement......................................................................................... Goals for this Position Statement....................................................................................................... Recommendations to PCPI Members, Staff, and Work Groups ..................................................................

Table 1: Evidence and consensus development processes used in guideline development.............................................

1 4 4 4 5 6 7 9 10 10 11 12 12 13

14 15 16 17

Recommendations to Guideline Developers.......................................................................................... Framework for Determining Acceptability of Guidelines and Other Evidence Review Documents as Basis for PCPI Measures Development........................................................................................................................

Table 2: Evidence Base Variables and Timeline ............................................................................................. Table 3: Additional Conditions for Acceptance of Alternative Evidence Review Documents....................................... Table 4: Additional Conditions for Acceptance of Guideline Recommendations Based on Expert Opinion........................

Implementation Process for PCPI Evaluation of Guidelines .......................................................................... Dissemination and Evaluation .........................................................................................................

Appendix 1: Algorithm for PCPI Evaluation of Guidelines....................................................................................... Appendix 2: The Conference on Guideline Standardization (COGS) Checklist for Reporting Clinical Practice Guidelines ............ Appendix 3: Appraisal of Guidelines Research & Evaluation (AGREE) Instrument......................................................... Appendix 4: National Guideline Clearinghouse (NGC) Inclusion Criteria.....................................................................

© 2009 American Medical Association. All Rights Reserved.

APPROVED BY THE PCPI, JUNE 26, 2009

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

PCPI Position Statement ­ The Evidence Base Required for Measures Development Frequently Asked Questions

Q. Physician performance measures developed by the PCPI are described as "evidence-based." What does this mean? The measure development methodology of the PCPI is based in part on the principles of evidencebased medicine, the practice of which involves the integration of individual clinical expertise with the best available clinical evidence from systematic research. Research evidence is typically reviewed in clinical practice guidelines and synthesized into clinical recommendations, from which "evidence-based" performance measures are derived. In addition, the PCPI has now expanded what is acceptable as the evidence base for measures to include documented quality improvement (QI) initiatives or implementation projects that have demonstrated improvement in the quality of care, as outlined in the Criteria for Topic Selection approved by the PCPI Executive Committee in January 2009. (This latter category of evidence is not subject to the guideline criteria outlined in this document.) Is it easier to derive performance measures from some guidelines than from others? When guideline developers use evidence rating schemes, which assign a "grade" or "level" to the evidence based on the type and quality of the research, it is easiest for measure developers to identify the strongest evidence on which to base their measures. Some guideline developers separately assign grades to their clinical recommendations based on the degree of confidence in the benefits of the recommended procedure or treatment. Unfortunately, not all guideline developers use such rating schemes and, among those that do, there is limited standardization in the rating schemes chosen. Why does the PCPI need to set criteria for the acceptability of guidelines? As the PCPI continues to work with its members to develop performance measures for every medical specialty, it is apparent that--in addition to the inconsistencies noted among evidence rating schemes--the quality of the research evidence available for each clinical topic and the process used for developing clinical recommendations from the evidence are also extremely variable. By disseminating the criteria described in the Position Statement, the PCPI seeks to promote greater consistency and rigor in guideline development methodology and to facilitate the current measure development process for PCPI work groups. The PCPI is also beginning to explore new models and methodologies for measure development, to which the criteria described in this Position Statement may not be fully applicable. When guidelines are used as the basis for PCPI measures, are only guidelines listed in the National Guideline Clearinghouse (NGC) considered acceptable? The PCPI Position Statement describes evidence base criteria for clinical practice guidelines, not limited to guidelines included in the NGC. These criteria will initially also be applicable to evidence review documents other than guidelines (eg, clinical review articles, treatment protocols or algorithms), provided all the "required" criteria and certain additional conditions are met. After an initial period of two years, an enhanced set of criteria designating additional "required" and "high priority" elements will take effect. (Additional "required" elements will likely be designated in future revisions of the criteria; the ultimate goal will be to require that guidelines routinely provide all "high priority" elements.) Thereafter, guidelines accepted as the evidence base for PCPI measures will be limited to those with the methodological rigor and other attributes consistent with the new criteria. Inclusion in the NGC improves the accessibility of guidelines for clinicians and other users, but will not be required 1

A.

Q. A.

Q. A.

Q.

A.

APPROVED BY THE PCPI, JUNE 26, 2009

54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106

for use by the PCPI. The PCPI may reconsider this policy if significant changes are made to the mission of the NGC or its inclusion criteria. Q. If no acceptable evidence base (guideline, qualifying alternative evidence review document, or qualifying QI initiative) is available for a clinical topic, will the PCPI consider basing measures on individual research studies? The PCPI has not created the infrastructure or developed the methodology needed for assessing the quality of individual research studies. It is also not unusual for individual studies to reach different conclusions on the same research questions. The PCPI is not equipped to reconcile such discrepancies and would therefore be unable to identify the evidence base on which to base performance measures. Will any guidelines other than those developed by American medical specialty societies be acceptable as the basis for PCPI measures? Guidelines developed by national, American physician organizations and federal agencies (such as the National Institutes of Health and the Centers for Disease Control and Prevention) are preferred for use in PCPI measure development. However, guidelines and other evidence documents developed by non-American organizations, as well as non-physician organizations (eg, a nursing association), are also acceptable and should be evaluated for consideration at the discretion of individual work groups. Are guideline recommendations based on expert opinion acceptable under the PCPI criteria? Expert opinion is generally included among the various types of evidence that may be considered by guideline developers in the formulation of clinical recommendations. However, the various schemes used for rating the quality or strength of evidence generally place expert opinion at the low end of the scale. That being the case, guideline recommendations based on expert opinion may be considered as a basis for PCPI measure development, but only if the additional criteria specified in the Position Statement are met. How did the PCPI select its criteria for the acceptability of guidelines? The PCPI criteria are derived from and consistent with several earlier initiatives to establish and evaluate quality characteristics for guidelines, including the Conference on Guideline Standardization (COGS) and the AGREE (Appraisal of Guidelines Research and Evaluation) Collaboration. The NGC Inclusion Criteria have also been taken into consideration. Who decides if a guideline or an alternative evidence review document meets PCPI criteria? Guidelines proposed as the basis for measure development will be reviewed by PCPI staff against the PCPI criteria; a summary and recommendations on the topic proposed for measure development and the acceptability of the guidelines will be presented to the PCPI Executive Committee. The Executive Committee will approve or disapprove the proposed topic based on the staff recommendations or, if additional information is needed, will refer its decision to the PCPI Measures Development, Methodology, and Oversight Advisory Committee (MDMO-AC). The MDMO-AC will work with staff to gather additional information as needed and as described in the Position Statement. Some PCPI members may consider developing measures independently and submitting them to the PCPI for approval, consistent with the process approved by the PCPI in 2007. Under these circumstances, will PCPI members be expected to use evidence base criteria identical to those described in the Position Statement? 2

A.

Q.

A.

Q. A.

Q. A.

Q. A.

Q.

APPROVED BY THE PCPI, JUNE 26, 2009

107 108 109 110 111 112 113 114 115

A.

The approved process for PCPI review and approval of measures developed independently by PCPI voting members requires that the measure developer "include all elements of the PCPI's measure development process (eg, ... consideration of existing guideline recommendations and the strength of evidence...)" A PCPI member developing measures would therefore be expected to apply standards for the selection of guidelines that are generally comparable to the criteria described in the Position Statement, subject to the determination of the PCPI Executive Committee.

3

APPROVED BY THE PCPI, JUNE 26, 2009

116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164

PCPI Position Statement ­ The Evidence Base Required for Measures Development

Purpose

This position statement is provided to establish a framework for consistent and objective selection of clinical practice guidelines from which PCPI work groups may derive clinical performance measures. This framework is intended to guide PCPI members and staff in the selection of measurement topics, to facilitate measure development, and to promote greater consistency in guideline development methodology.

Summary of Recommendations

As described in detail in the sections to follow, this position statement provides recommendations for: · Use of a defined set of criteria (categorizing guideline attributes as "required," "preferred," or "acceptable") for determining acceptability of guidelines for PCPI use in measure development · Work group consideration of all types of evidence reviewed in guidelines, including expert opinion · Acceptance of guideline recommendations based on expert opinion as the basis for PCPI measure development when additional conditions are met · Temporary acceptance of suitable alternative evidence review documents (not including individual studies) as basis for PCPI measures, until new PCPI requirements take effect on July 1, 2011 and when additional conditions are met · Use of discretion by work groups when considering guideline attributes that are not "preferred" · Implementation of an enhanced set of PCPI criteria to take effect on July 1, 2011, adding requirements for guideline content and methodology, identifying some "high priority" elements, and ending PCPI acceptance of alternative evidence review documents other than guidelines · In subsequent, periodic reassessments of the PCPI criteria, gradual upgrading of all "high priority" elements to become "required" elements, driving a progressive increase in the rigor of guideline development over time · Implementation of a process for guideline evaluation by PCPI staff and approval or non-approval of a proposed measurement topic by the PCPI Executive Committee, or further evaluation of the topic if needed · Dissemination of the position statement to PCPI members and other guideline developers through various presentations and communications · Interim and full reassessments of the PCPI guideline requirements within two years of implementation; subsequent reassessments to be conducted at regular intervals

Context and Need for this Position Statement

In the years since the PCPI began working with its members to develop evidence-based performance measures for physicians, the demand and potential uses for PCPI measures have expanded and evolved: · · · · Private and public payers for health care services, including the Centers for Medicare and Medicaid Services (CMS), have launched pay-for-reporting or pay-for-performance initiatives. Specialty certification boards are integrating measures into their "maintenance of certification" programs. The original PCPI focus on developing measures for quality improvement (QI) has shifted to the development of measures suitable for QI and accountability. Originally designed for physician use only, PCPI measures are now developed and specified for potential use by other health care professionals as well.

These changes have created a need for measures applicable to each medical specialty. At the same time, it has become apparent that evidence-based guidelines are lacking for clinical topics that may lack a strong research base. For some specialties and clinical topics, only guideline recommendations based on expert 4

APPROVED BY THE PCPI, JUNE 26, 2009

165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216

opinion are available; for other topics, evidence may only be available without a published guideline or in a guideline without clear clinical recommendations. Where evidence-based clinical recommendations are available in guidelines, the processes used for identifying the evidence and for synthesizing it into recommendations can be extremely variable. 1 Additional factors suggest that the PCPI must assess the wisdom of developing new models for measure development and may need to become less reliant on the availability of strictly evidence-based guidelines: · · · Heightened interest in several new measure types, including structural measures, patient safety measures, and measures designed for team- or system-level accountability (eg, care coordination measures) Particular interest among some PCPI constituents in measures of clinical efficiency An expressed need for the PCPI to become more responsive to the measurement priorities of measure implementers (eg, CMS and private payers), the National Quality Forum, and other PCPI constituents for which the evidence base may not be the primary consideration

In response to these evolving needs and priorities, the PCPI has expanded the scope of what will be considered acceptable as an evidence base for measures. For any proposed measure development topic, the Criteria for Topic Selection approved by the PCPI Executive Committee in January 2009 describes the evidence base requirements as:

· ·

One or more national, widely-accepted clinical guidelines that meet the standards set forth in this position statement, OR One or more documented quality improvement (QI) initiatives or implementation projects that have demonstrated improvement in the quality of care (based on measures of access, processes, outcomes or the patient experience of care)

The guideline evaluation criteria outlined in this document are not intended for use in assessing QI initiatives or implementation projects proposed as the evidence base for potential measure topics. As the PCPI expands its definition of an acceptable evidence base and begins to explore the development of new types of measures, it must still provide its members and work groups with clear guidance for the new development and maintenance of measures derived from evidence-based guidelines. Even as new measurement models and methodologies are created, consistency in the existing PCPI methodology must be maintained. To that end, this position statement presents recommendations for the review and evaluation of guidelines and other evidence review documents based on an objective set of content and process criteria. The criteria include current PCPI requirements for the evidence base as well as an enhanced set of requirements and other prioritized elements scheduled to take effect two years after the approval of this document. Additional requirements will likely be specified in subsequent reassessments of the criteria. In thus giving notice to guideline developers and progressively "raising the bar" for guideline content and methodology, the PCPI also seeks to promote greater consistency in guideline development and to facilitate future measure development.

Goals for this Position Statement

· · · · · Provide recommendations to PCPI medical specialty society members and staff proposing topics for measure development and to PCPI work groups Establish a set of PCPI criteria for the evaluation of guidelines and other evidence review documents proposed as the basis for PCPI measure development Describe a process for implementing the review of proposed guidelines against the PCPI criteria Provide notice to guideline developers on PCPI requirements for guideline content and methodology, including an enhanced set of criteria to take effect after a two-year notice period Establish a plan and timeframe for dissemination and evaluation of the new guideline review process 5

APPROVED BY THE PCPI, JUNE 26, 2009

217

Recommendations to PCPI Members, Staff, and Work Groups

Policy recommendations: · A comprehensive set of source, content, and process criteria will be used to determine the acceptability of guidelines for PCPI use in measure development. · "Required" guideline elements include disclosure of potential conflicts of interest, description of methods used for scientific literature search, types of evidence reviewed, etc.; other guideline attributes are initially labeled "preferred" or "acceptable." (Enhanced set of guideline criteria taking effect in two years is described under Recommendations to Guideline Developers; see page 9.) · The full set of criteria may not be applicable to all measure topics. The PCPI Executive Committee will determine when exceptions are appropriate, based on the recommendations of PCPI staff and, if applicable, the Measures Development, Methodology, and Oversight Advisory Committee (see Implementation Process, page 12). · PCPI work groups are encouraged to consider all types of evidence reviewed in guidelines, including expert opinion. · Additional conditions must be met for acceptance of guidelines with recommendations based on expert opinion (eg, use of a formal consensus development process). · Alternative evidence review documents will be temporarily acceptable for PCPI use if additional conditions are met (including unavailability of a relevant guideline), until new PCPI requirements take effect on July 1, 2011. · Work groups should use discretion in considering the use of evidence review documents with attributes other than those identified in the criteria as "preferred." The PCPI evidence base criteria are outlined in Tables 2-4 on the following pages. An algorithm for the use of these criteria is provided in Appendix 1. The criteria are based in part on prior work by the Conference on Guideline Standardization (COGS), which sought to establish and evaluate quality characteristics for guidelines. 2 (The complete list of guideline characteristics selected for the COGS Checklist is shown in Appendix 2.) The PCPI criteria are also consistent with the quality attributes identified for guidelines by the AGREE (Appraisal of Guidelines Research and Evaluation) Collaboration. 3 (See Appendix 3) The National Guideline ClearinghouseTM (NGC) Inclusion Criteria were also taken into consideration in developing the PCPI criteria. 4 (See Appendix 4) The full set of criteria listed in Tables 2-4 is intended to apply primarily to measure topics limited to individual clinical conditions and for which clinical processes and outcomes are the focal points of measurement. For more broadly-defined measure topics (eg, care coordination) or when measures are needed for quality issues other than clinical process and outcome (eg, structural measures), only a subset of the criteria may be applicable. The following recommendations expand upon the criteria in the tables and are intended to assist PCPI members, staff, and work groups in applying the criteria consistently to measurement topics and evidence review documents under consideration. Aside from those few elements designated "required," no single criterion in the tables should be used to determine the acceptability of a guideline for PCPI use; rather, the applicable set of criteria should be considered in its entirety. 1. Types of evidence: The research evidence that guideline developers typically review in formulating their recommendations may range from randomized controlled trials (RCTs) to expert opinion, with the former type of evidence generally "preferred" as a basis for measure development. (See Table 1) When recommendations based on expert opinion are available for a measurement topic in addition to other, "preferred" types or levels of evidence, PCPI work groups are encouraged to use all available types of evidence for measure development, with greater weight given to the "preferred" evidence. 5 Work groups are also encouraged to allow a variety of stakeholders (ie, clinicians as well as purchasers, payers, and consumers of health care) to participate in the consideration of the evidence available for use in developing measures. 6

218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249

APPROVED BY THE PCPI, JUNE 26, 2009

250 Table 1: Evidence and consensus development processes used in guideline development Consensus development process used Types/levels of evidence used*

PREFERRED

Formal consensus methods**

(with or without evidence ratings)

Informal (unstructured) expert consensus

LESS PREFERRED, BUT MAY BE ACCEPTABLE

Multiple randomized controlled trials (or metaanalysis of such trials) Single randomized trial, or nonrandomized studies Cohort studies Case-control studies Observational studies/ Cross-sectional studies/ Case series/ Case reports Non-quantitative systematic review of primary studies Expert opinion/ Clinical experience

LESS PREFERRED, BUT MAY BE ACCEPTABLE

NOT ACCEPTABLE

* In addition to the level of evidence reviewed (and possibly rated), many guideline developers also identify a "Strength of recommendation," depending on the clarity of benefit (or harm) of the procedure or treatment studied ** "Formal consensus methods" include: Delphi Method, Nominal Group Technique (NGT), "Modified NGT" or "modified Delphi," Consensus Development Conference, other formal consensus methods 6 Also, it should be noted that multiple types/levels of evidence may be used as the basis for different recommendations within an individual guideline.

251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272

2. Other variables: PCPI members and staff are advised to consider all "evidence base variables" listed in Table 2 (divided into Source Criteria and Content and Process Criteria) when determining the acceptability of each evidence document. Among the criteria listed as "Effective Immediately," note that some items are identified as "required;" others are labeled "preferred" or "may be acceptable." 3. Use of expert opinion-based recommendations: Table 4 lists an additional PCPI requirement (ie, in addition to those listed in Table 2) for the acceptance of guideline recommendations based on expert opinion in developing measures: the use of a formal consensus method for reaching consensus on the recommendations. (Additional information and guidance on the use of "formal" and "informal" consensus methods is provided in Table 1.) For other guideline elements relevant to the use of recommendations based on expert opinion, Table 4 adds a "high priority" tier to the PCPI criteria. Elements designated "high priority" are particularly desirable to maximize the usability of a guideline for measure development; PCPI work groups are therefore discouraged from using guidelines lacking any elements so labeled. However, given the variability among guideline development methodologies, the PCPI recognizes that setting a requirement for all desirable guideline elements remains unrealistic at this time. "High priority" elements specified for the use of recommendations based on expert opinion are: a rating/grading of the strength of recommendations, and an indication that the anticipated benefits associated with the recommendations outweigh the potential risks.

7

APPROVED BY THE PCPI, JUNE 26, 2009

273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325

Despite PCPI acceptance of a guideline with opinion-based recommendations (per the criteria in Tables 2 and 4), the relevant work group should use discretion in choosing to base performance measures on expert opinion rather than other types of evidence that may be available for the measurement topic. 4. Evidence base documentation: The documentation for completed PCPI measures must always include information on the evidence base for the measures, including, where applicable, the use of recommendations based on expert opinion. 5. Use of alternative sources: Recognizing that research evidence may be available for some clinical topics without a published guideline, the PCPI will temporarily accept qualifying alternative sources as the basis for measure development: documents other than guidelines (eg, clinical review articles, treatment protocols or algorithms) that provide clinical recommendations based on a review and synthesis of evidence from multiple research studies. These alternative evidence review documents may be accepted as the basis for measure development only until the enhanced PCPI requirements take effect two years after approval of this document. Table 3 lists additional PCPI requirements (ie, in addition to those listed in Table 2) for the use of evidence from such alternative sources. In addition to the requirement for the use of a formal consensus method, note that an alternative evidence review document may be used only if there is no guideline available for the measurement topic. Again, a description of anticipated benefits and risks (weighted in favor of the benefits) is labeled "high priority." 6. Individual studies: In the absence of either a guideline or a suitable alternative source for a measure topic (or another acceptable evidence base, as described above), the PCPI will not consider basing measures on individual research studies. The PCPI methodology is not designed for assessing the quality of individual studies, nor does it enable the PCPI to reconcile the conflicts possible among conclusions reached on the same research questions by multiple studies. 7. Balanced representation; pre-release review: When information is provided by guideline developers on the size and composition of their consensus panels, consensus-based recommendations from broad-based, multi-stakeholder groups with balanced representation will be preferred by the PCPI over those from small, single-specialty groups. The PCPI will also favor guideline development processes that include the solicitation of public comments on draft guideline recommendations. 8. Work group discretion: When information is lacking, PCPI staff or work groups may consider contacting guideline developers for details regarding their consensus panel composition and consensus development process. Based on all information provided by the guideline developer and at the discretion of the PCPI work group, any guideline (or any subset of the clinical recommendations within) may be excluded from consideration as a basis for measure development. 9. National origin: Guidelines developed by organizations outside of the United States may be used for PCPI measure development and should be evaluated for consideration at the discretion of the work group. 10. Professional origin: Guidelines developed by regional or local physician organizations or by nonphysician organizations (eg, a nursing association) may be considered as a basis for PCPI measures and ­ relative to guidelines developed by national physician organizations or with physician participation ­ should be evaluated for consideration at the discretion of the work group.

8

APPROVED BY THE PCPI, JUNE 26, 2009

326

Recommendations to Guideline Developers

Policy recommendation: Two years after approval of this position statement, an enhanced set of guideline requirements and "high priority" elements will take effect and the PCPI will no longer accept evidence review documents other than guidelines for measures development. As noted previously, one objective of this position statement is to "raise the bar" for guideline content and methodology. The PCPI proposes to achieve this goal by implementing an enhanced set of criteria, with additional "required" and "high priority" elements, to take effect two years after the approval of this document. The far-right column of Table 2, labeled "Effective July 1, 2011," lists the enhancements to the criteria. Of particular importance, as stated previously and in Table 2 under Source Criteria, is the change in PCPI acceptance of evidence review documents other than guidelines. The changes taking effect in two years will effectively eliminate such alternative sources from consideration as the basis for measures development. The PCPI hopes with this change (and those described below) to promote increased commitment to and greater consistency in guideline development. In the Content and Process Criteria section of Table 2, the new requirements taking effect in two years will be for the use and description of an evidence rating scheme and the use and description of a method for grading the strength of recommendations. However, as explained in the footnote to these elements, the two rating schemes are intended to be alternative requirements; ie, guidelines will only be required to provide one or the other. Although many guidelines document the use of both rating schemes, either may suffice for measure development purposes. In fact, an analysis of guidelines cited as the basis for existing PCPI measures demonstrates that many guideline developers provide only a strength of recommendation rating, but that this rating is in most instances based on the quality of evidence found. 7 The PCPI therefore considers it reasonable to set a requirement in two years for either evidence ratings (with the rating method described) or recommendation ratings (with the rating method described). This flexible approach to the two rating schemes will be subject to further evaluation in future reassessments of the guideline criteria, as described below. Additionally, Table 2 lists various other elements (eg, use of a formal consensus method, description of potential benefits and harms) that will be elevated from "preferred" to the "high priority" tier in two years. As explained above, PCPI work groups will be discouraged from using guidelines lacking any "high priority" elements, but are advised to consider the applicable criteria set in its entirety when considering the acceptability of each guideline. The current inconsistencies across guidelines suggest that labeling many of these elements as "required" may still be unrealistic in two years, but the "high priority" designation may serve to convey to guideline developers the value of these elements in measure development. To promote continued improvement and consistency in guideline development over time, subsequent reassessments of the PCPI guideline criteria will gradually upgrade the "high priority" elements to become "required" elements. The ultimate goal of this iterative process of raising expectations for guideline content will be to drive the routine inclusion of all desired guideline elements and to progressively achieve a significant increase in the rigor of guideline development.

327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367

9

APPROVED BY THE PCPI, JUNE 26, 2009

368 369 370 371 372 373

Framework for Determining Acceptability of Guidelines and Other Evidence Review Documents as Basis for PCPI Measures Development

Table 2: Evidence Base Variables and Timeline For PCPI staff and work group consideration when reviewing guidelines & other evidence review documents Source Criteria 1. Type of document Guideline Alternative evidence review document (eg, clinical review article, treatment protocol or algorithm)

(see Table 3 for additional conditions effective for such alternative sources)

Effective Immediately Preferred

May be acceptable for first 2 yrs. if conditions met (Table 3)

Effective July 1, 2011 Required Not acceptable

2. Availability, currency, and acceptance Guideline/alternative source available in English language Guideline/ alternative source developed, reviewed, or revised within past 5 years Guideline included in National Guideline Clearinghouse (NGC) Accepted and/or endorsed for use nationally 3. Professional origin National physician organization * Regional, local, or other physician organization* Non-physician organization 4. National origin U.S. organization Non- U.S. organization 5. Developer Responsible organization(s); names/credentials/potential conflicts of interest disclosed Balanced representation on consensus panel 6. Funding source/sponsor Funding source and role identified; potential conflict of interest disclosed 374 Content and Process Criteria 1. Evidence collection Methods used for systematic search of scientific literature described 2. Evidence descriptions and rating a. Type of evidence (RCTs, cohort studies, observational studies, expert opinion) described

(see Table 4 for additional conditions effective for expert opinion)

Required Required Preferred Preferred Preferred May be acceptable May be acceptable Preferred May be acceptable Required Preferred Required Effective Immediately Required Required Preferred Preferred

Required Required Preferred Preferred Preferred May be acceptable May be acceptable Preferred May be acceptable Required Preferred Required Effective July 1, 2011 Required Required Required Required

(OR #3b-c below) §

b. Quality or strength of evidence rated c. Methods used for rating evidence quality described

Physician organizations described in NGC criteria include medical specialty associations and relevant professional societies For consideration at the discretion of work group Non-physician organizations described in NGC criteria include public or private organizations, government agencies at the Federal, State, or local level health care organizations or plans § Requirement met with either evidence rated & rating method described OR recommendations rated & rating method described (or both). Not applicable to recommendations based on expert opinion ­ see Table 4.

*

10

APPROVED BY THE PCPI, JUNE 26, 2009

375 Content and Process Criteria (continued) 3. Recommendation statements a. Distinct recommendation statements provided b. Strength of recommendation rated c. Methods used for grading strength of recommendation described d. Public comments solicited on draft recommendation statements 4. Evidence synthesis How evidence was used to create recommendations (e.g., evidence tables, meta-analysis) described 5. Consensus development process Consensus development process described Formal consensus method used Informal consensus method used 6. Update plan Plan to update guideline described; expiration date for current version stated 7. Potential benefits and harms Anticipated benefits and potential risks associated with recommendations described 8. Patient preferences Role of patient preferences discussed when recommendation involves personal choice or values 376 377 378 379 380 381 Effective Immediately Required Preferred Preferred Preferred Preferred Effective July 1, 2011 Required Required Required Preferred

High priority

(OR #2b-c above) **

Preferred (guidelines only) Preferred (guidelines only) May be acceptable (guidelines only)

High priority High priority

May be acceptable

Preferred

Preferred (guidelines only)

High priority High priority

Preferred

High priority

Table 3: Additional Conditions for Acceptance of Alternative Evidence Review Documents ­ Effective upon PCPI approval of Position Statement and through the 2-year notice period only

Effective Immediately (and until July 1, 2011) for Alternative Evidence Review Documents

1. Consensus development process Consensus development process described Formal consensus method used Informal consensus method used 2. Potential benefits and harms Anticipated benefits and potential risks associated with recommendations described; benefit must > risk 3. No guideline available for the measurement topic 382 383

Required Required Not acceptable High priority Required

Requirement met with either evidence rated & rating method described OR recommendations rated & rating method described (or both). Not applicable to recommendations based on expert opinion ­ see Table 4. "Formal consensus methods" include: Delphi Method, Nominal Group Technique (NGT), "Modified NGT" or "modified Delphi," Consensus Development Conference, other formal consensus methods6

**

11

APPROVED BY THE PCPI, JUNE 26, 2009

384 385 386 387

Table 4: Additional Conditions for Acceptance of Guideline Recommendations Based on Expert Opinion ­ Effective upon PCPI approval of Position Statement (Note: These criteria will remain in effect after the 2-year notice period.) Effective Immediately for Expert Opinion 1. Recommendation statements Strength of recommendation rated Methods used for grading strength of recommendation described 2. Consensus development process Consensus development process described Formal consensus method used Informal consensus method used 3. Potential benefits and harms Anticipated benefits and potential risks associated with recommendations described; benefit must > risk High priority High priority Required Required Not acceptable High priority

388 389 390

Implementation Process for PCPI Evaluation of Guidelines

Policy recommendation: Before a work group is convened for a proposed measurement topic, PCPI staff will evaluate guidelines available for the topic against the approved criteria and will provide a recommendation to the PCPI Executive Committee for approval or non-approval of the topic. A protocol is described for further evaluation of the topic if deemed necessary by the Executive Committee.

391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413

All documentation proposed as the evidence base for a potential measure topic will be assembled for review before a work group is convened for that topic. Using the PCPI criteria, an evaluation of guidelines or alternative evidence review documents proposed as the basis for measure development (if applicable) will be conducted by PCPI staff. Final determinations on the acceptability of guidelines will be made by the PCPI Executive Committee or referred for further evaluation, as follows: 1. Staff review the guidelines available for a potential measurement topic against the PCPI-approved criteria and provide a summary and recommendations on the topic to the PCPI Executive Committee (EC). In most instances, this will take place before co-chairs for the topic have been selected. 2. For measure topics other than individual clinical conditions (eg, care coordination) or measures focused on quality issues other than clinical process and outcome (eg, structural measures), staff may recommend that the PCPI-approved criteria are not applicable. In such circumstances the EC may, at its discretion, assess the topic against an appropriate subset of the criteria or refer the topic for further evaluation, as per the process described below. 3. Based on their evaluation of the guidelines against the criteria (if applicable), staff will recommend that the EC: a) approve the topic, b) not approve the topic, or c) refer the topic for further evaluation, due to a lack of available information, a lack of consensus among EC members, etc. If the EC selects option c (whether or not that was the staff recommendation), the referral will generally be to the Measures Development, Methodology, and Oversight Advisory Committee (MDMO-AC).

"Formal consensus methods" include: Delphi Method, Nominal Group Technique (NGT), "Modified NGT" or "modified Delphi," Consensus Development Conference, other formal consensus methods6

12

APPROVED BY THE PCPI, JUNE 26, 2009

414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435

4. The MDMO-AC will work with staff to gather additional information as may be needed to reach a decision on the topic. The MDMO-AC will NOT be involved in topic approval routinely, but only if a 2nd level review is needed. a. If deemed necessary by the MDMO-AC, and in concert with staff, additional internal or external advisors with appropriate content expertise may be consulted to assist with guideline evaluation and topic approval. b. At the discretion of the EC, an alternative approach may be to assign the secondary guideline evaluation to a sub-committee of the EC. c. An additional alternative may be to commission a review by an organization external to the PCPI. 5. The EC may also approve a topic with limitations to the scope of the work, based on the adequacy of the evidence base. Again, the MDMO-AC will be consulted only if the EC feels that additional review is needed. 6. Limiting the guideline evaluation process to the EC and the MDMO-AC would shield the decisionmaking on topics from potential biases and avoid the added complexity of an independent review group. 7. Only after the topic is approved through this process will co-chairs be confirmed and a work group convened.

Dissemination and Evaluation

Policy recommendations: After PCPI approval, the position statement and its recommendations will be disseminated widely and information provided proactively to PCPI members and other guideline developers. Interim and full reassessments of the guideline requirements will be conducted within two years of implementation (by July 1, 2011), with subsequent reassessments to be conducted at regular intervals. After the position statement has received PCPI approval, PCPI staff will disseminate the document widely through various means, which may include: posting the position statement on the AMA web site and developing it into an article for publication in a peer-reviewed journal; delivering presentations to individual medical specialty societies and other PCPI members that have expressed interest in developing a guideline; and communicating directly with other guideline developers (eg, NHLBI, ICSI) regarding current PCPI criteria for guidelines and the new requirements taking effect on July 1, 2011. To adjust the guideline requirements as may be needed based on experience from initial implementation, a full reassessment will take place immediately before the enhanced requirements take effect; an interim reassessment will be conducted after the first year. After the initial two-year implementation period, subsequent reassessments of the guideline requirements will be conducted at regular intervals.

436 437 438 439 440 441 442 443 444 445 446 447 448 449

13

APPROVED BY THE PCPI, JUNE 26, 2009

Appendix 1: Algorithm for PCPI Evaluation of Guidelines*

Effective Immediately Evidence Base

Effective after 2-year Notice Period Evidence Base

QI Initiative or Implementation Project

Guideline

Alternative evidence review document

QI Initiative or Implementation Project

Guideline

Table 2

Tables 2 & 3

Table 2

Recommendations based on expert opinion?

Recommendations based on expert opinion?

No

Yes

No

Yes

Table 4

Table 4

Executive Committee Review (See Implementation Process) *See Tables 2-4 and supporting text for additional information

Executive Committee Review (See Implementation Process)

450 14

APPROVED BY THE PCPI, JUNE 26, 2009

451 Appendix 2: The Conference on Guideline Standardization (COGS) Checklist for Reporting Clinical Practice Guidelines2

Topic 1. Overview material 2. Focus 3. Goal 4. Users/setting 5. Target population 6. Developer 7. Funding source/sponsor 8. Evidence collection 9. Recommendation grading criteria 10. Method for synthesizing evidence 11. Prerelease review 12. Update plan 13. Definitions 14. Recommendations and rationale 15. Potential benefits and harms 16. Patient preferences 17. Algorithm 18. Implementation considerations Description Provide a structured abstract that includes the guideline's release date, status (original, revised, updated), and print and electronic sources. Describe the primary disease/condition and intervention/service/technology that the guideline addresses. Indicate any alternative preventive, diagnostic or therapeutic interventions that were considered during development. Describe the goal that following the guideline is expected to achieve, including the rationale for development of a guideline on this topic. Describe the intended users of the guideline (e.g., provider types, patients) and the settings in which the guideline is intended to be used. Describe the patient population eligible for guideline recommendations and list any exclusion criteria. Identify the organization(s) responsible for guideline development and the names/credentials/potential conflicts of interest of individuals involved in the guideline's development. Identify the funding source/sponsor and describe its role in developing and/or reporting the guideline. Disclose potential conflict of interest. Describe the methods used to search the scientific literature, including the range of dates and databases searched, and criteria applied to filter the retrieved evidence. Describe the criteria used to rate the quality of evidence that supports the recommendations and the system for describing the strength of the recommendations. Recommendation strength communicates the importance of adherence to a recommendation and is based on both the quality of the evidence and the magnitude of anticipated benefits or harms. Describe how evidence was used to create recommendations, e.g., evidence tables, meta-analysis, decision analysis. Describe how the guideline developer reviewed and/or tested the guidelines prior to release. State whether or not there is a plan to update the guideline and, if applicable, an expiration date for this version of the guideline. Define unfamiliar terms and those critical to correct application of the guideline that might be subject to misinterpretation. State the recommended action precisely and the specific circumstances under which to perform it. Justify each recommendation by describing the linkage between the recommendation and its supporting evidence. Indicate the quality of evidence and the recommendation strength, based on the criteria described in 9. Describe anticipated benefits and potential risks associated with implementation of guideline recommendations. Describe the role of patient preferences when a recommendation involves a substantial element of personal choice or values. Provide (when appropriate) a graphical description of the stages and decisions in clinical care described by the guideline. Describe anticipated barriers to application of the recommendations. Provide reference to any auxiliary documents for providers or patients that are intended to facilitate implementation. Suggest review criteria for measuring changes in care when the guideline is implemented.

COGS Checklist also referenced in:

Rosenfeld RM, Shiffman, RN. Clinical practice guidelines: A manual for developing evidence-based guidelines to facilitate performance measurement and quality improvement. Otolaryngology-Head and Neck Surg 2006;135:S1-S28. Methodology manual for ACC/AHA guideline writing committees. American College of Cardiology Foundation and American Heart Assn., Inc 2006. Available at: http://acc.org/qualityandscience/quality/quality.htm Accessed November 2007. Institute of Medicine (IOM). 2008. Knowing What Works in Health Care: A Roadmap for the Nation. Washington, DC: The National Academies Press. Prepublication copy, available at: http://www.nap.edu/catalog.php?record_id=12038#toc

15

APPROVED BY THE PCPI, JUNE 26, 2009

452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499

Appendix 3: Appraisal of Guidelines Research & Evaluation (AGREE) Instrument3 Scope and Purpose 1. The overall objective(s) of the guideline is (are) specifically described. 2. The clinical question(s) covered by the guideline is (are) specifically described. 3. The patients to whom the guideline is meant to apply are specifically described. Stakeholder Involvement 4. The guideline development group includes individuals from all the relevant professional groups. 5. The patients' views and preferences have been sought. 6. The target users of the guideline are clearly defined. 7. The guideline has been piloted among target users. Rigour of Development 8. Systematic methods were used to search for evidence. 9. The criteria for selecting the evidence are clearly described. 10. The methods used for formulating the recommendations are clearly described. 11. The health benefits, side effects and risks have been considered in formulating the recommendations. 12. There is an explicit link between the recommendations and the supporting evidence. 13. The guideline has been externally reviewed by experts prior to its publication. 14. A procedure for updating the guideline is provided. Clarity and Presentation 15. The recommendations are specific and unambiguous. 16. The different options for management of the condition are clearly presented. 17. Key recommendations are easily identifiable. 18. The guideline is supported with tools for application. Applicability 19. The potential organisational barriers in applying the recommendations have been discussed. 20. The potential cost implications of applying the recommendations have been considered. 21. The guideline presents key review criteria for monitoring and/or audit purposes. Editorial Independence 22. The guideline is editorially independent from the funding body. 23. Conflicts of interest of guideline development members have been recorded.

AGREE Instrument also referenced in: Rosenfeld RM, Shiffman, RN. Clinical practice guidelines: A manual for developing evidence-based guidelines to facilitate performance measurement and quality improvement. Otolaryngology-Head and Neck Surg 2006;135:S1-S28. Institute of Medicine (IOM). 2008. Knowing What Works in Health Care: A Roadmap for the Nation. Washington, DC: The National Academies Press. Prepublication copy, available at: http://www.nap.edu/catalog.php?record_id=12038#toc

16

APPROVED BY THE PCPI, JUNE 26, 2009

500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528

Appendix 4: National Guideline Clearinghouse (NGC) Inclusion Criteria4 Definition of Clinical Practice Guideline The National Guideline ClearinghouseTM (NGC) employs the definition of clinical practice guideline developed by the Institute of Medicine (IOM).

·

Clinical practice guidelines are systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances. [Institute of Medicine. (1990). Clinical Practice Guidelines: Directions for a New Program, M.J. Field and K.N. Lohr (eds.) Washington, DC: National Academy Press. page 38].

Criteria for Inclusion of Clinical Practice Guidelines in NGC All of the criteria below must be met for a clinical practice guideline to be included in NGC. 1. The clinical practice guideline contains systematically developed statements that include recommendations, strategies, or information that assists physicians and/or other health care practitioners and patients make decisions about appropriate health care for specific clinical circumstances. 2. The clinical practice guideline was produced under the auspices of medical specialty associations; relevant professional societies, public or private organizations, government agencies at the Federal, State, or local level; or health care organizations or plans. A clinical practice guideline developed and issued by an individual not officially sponsored or supported by one of the above types of organizations does not meet the inclusion criteria for NGC. 3. Corroborating documentation can be produced and verified that a systematic literature search and review of existing scientific evidence published in peer reviewed journals was performed during the guideline development. A guideline is not excluded from NGC if corroborating documentation can be produced and verified detailing specific gaps in scientific evidence for some of the guideline's recommendations. 4. The full text guideline is available upon request in print or electronic format (for free or for a fee), in the English language. The guideline is current and the most recent version produced. Documented evidence can be produced or verified that the guideline was developed, reviewed, or revised within the last five years.

1

Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 1999;281(20):1900-5. 2 Shiffman RN, Shekelle P, Overhage M et. al. Standardized reporting of clinical practice guidelines: A proposal from the conference on guideline standardization. Ann Intern Med 2003;139:493-8. 3 The AGREE Collaboration. 2001. Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument. London, United Kingdom: The AGREE Research Trust. www.agreecollaboration.org. Last accessed: February 2008. 4 National Guideline Clearinghouse, Inclusion Criteria. http://www.guideline.gov/about/inclusion.aspx. Accessed November 2007. 5 "EBM [evidence-based medicine] does not mean that decision-makers should only consider randomized trials, but it does mean that higher-quality studies have more influence on decisions than do other types of information and evidence." In: Tunis, SR. Reflections on science, judgment, and value in evidence-based decision making: a conversation with David Eddy. Health Affairs 26, no. 4 (2007): w500-w515. 6 Murphy MK, et al. Consensus development methods, and their use in clinical guideline development. Health Technology Assessment 1998; Vol. 2: No. 3. Available at: http://www.ncchta.org/fullmono/mon203.pdf. 7 Documentation for PCPI measure sets available at: http://www.ama-assn.org/ama/pub/category/4837.html.

17

Information

PCPI Position Statement: Evidence Base Required

18 pages

Find more like this

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate

1070658


You might also be interested in

BETA
jep_631.fm
Microsoft Word - anti-ccp_testing_for_rheumatoid_arthritis
Acknowledgements
New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1)