Not playing with a full DEC: why development and evaluation committee methods for appraising new drugs may be inadequate
BMJ 1999; 318 doi: https://doi.org/10.1136/bmj.318.7196.1480 (Published 29 May 1999) Cite this as: BMJ 1999;318:1480
This article has a correction. Please see:
- Nick Freemantle, senior research fellow,
- James Mason (jmm7{at}york.ac.uk), senior research fellow
- Correspondence to: Dr Mason
- Accepted 10 February 1999
The consultation document A First Class Service: Quality in the New NHS1 heralded the introduction of the National Institute for Clinical Excellence (NICE). A key task of the institute is to provide rapid appraisal of new drugs in the period before licensing. New products may be accepted or refused NHS reimbursement, or they may be allocated “continuing research status.”
The review process described in A First Class Service and developed in a subsequent discussion paper2 mirrors that used for regionally funded development and evaluation committee (DEC) reports. Development and evaluation committee reports are produced by an independent arbitration committee comprising senior clinicians and others, who review the quality of available evidence, give explicit consideration to cost utility estimates, and make recommendations about interventions. Reports are published by NHS Research and Development (www.epi.bris.ac.uk/rd). Are the methods used by the development and evaluation committee up to the task?
Summary points
The National Institute for Clinical Excellence will appraise 30-50 drugs and technologies each year to inform decisions on whether these should be accepted or refused NHS reimbursement
The appraisal process will mirror that of regional development and evaluation committees, but this may lead to poor decision making since the methods used are inappropriate
The cost utility method of ranking treatments is based on strong assumptions and selective use of available evidence
New drugs should be appraised in terms of physical outcomes that mean something to doctors and patients
Evaluating new health technologies
Because of limited resources and medical cost inflation, new drugs can place a considerable strain on prescribing budgets.3 New and expensive drugs can benefit patients, but only if worthwhile advances can be differentiated from costly ways of achieving equivocal results. However, new drugs present particular difficulties; they have usually been evaluated in relatively few clinical trials in which measures of effectiveness are narrow—and there is often uncertainty about their value to patients. The risk of rare, unexpected, and untoward side effects is highlighted by the regularity with which licensed drugs are withdrawn from the market for safety reasons. This happened recently with the antipsychotic drug sertindole.
“Those conducting appraisals should have the complete findings of trials used for licensing new products”
Trial results are often published after the drugs have been introduced, and may not include enough data on all the outcomes. In some reviewers can access substantial amounts of unpublished data,4 but the lack of success of the trials amnesty shows that this is not general.5 Retrieving unpublished data is a difficult but highly important task faced by the development and evaluation committee and will be a core task for the institute. In the appraisal process, it is proposed that pharmaceutical companies should submit trial results for consideration four months before licensing.But will all the data be required, or will a selection determined by the sponsoring company be sufficient?
Opportunity costs
The most important question is not whether a drug is expensive and can be afforded, but whether it is a worthwhile use of resources, given the competing claims on these resources. High quality evaluation of new drugs and technologies will enable clinicians and their advisors to prioritise emerging drugs so that they can either move quickly to incorporate true advances in practice or reject change where benefits remain uncertain. For example, angiotensin converting enzyme inhibitors have been shown to improve the length and quality of life of patients with heart failure, and they seem cost effective compared with other treatments.6 However,the overall profile of diagnosis,drug acquisition, and monitoring, and the use of other disease related resources, means that greater prescribing of angiotensin converting enzyme inhibitors requires additional funding. This is the rather crude but realistic sense in which a drug may be deemed cost effective.
Modelling the cost per QALY
The most striking feature of the appraisal methods used in development and evaluation committee reports is the explicit modelling of a cost per quality adjusted life year (QALY). The QALY tries to collapse the effects of a multidimensional phenomenon (aspects of quality of life) onto a single scale. This enables comparisons to be made across disease and treatment areas, and aids planning and resource allocation. However, ranking the value of treatments using cost per QALY is controversial, not least because of the strong assumptions that are often concealed.7 This is particularly likely when, as with new drugs, there is limited evidence.
The development and evaluation committee report considering donepezil hydrochloride for treating patients with Alzheimer's disease allowed that the “range of cost-utility estimates is wide, ranging from approximately £21,000 to £200,000 per QALY,” based on data acknowledged to be “limited.”8 The pooled estimate of the effect on cognitive function of donepezil compared with placebo in the three trials available to the committee suggested a shift of about 2.8 points on the Alzheimer's disease cognitive subscale (95% confidence interval 2.2 to 3.4).9 The development and evaluation committee equated this finding to a “three to six month delay in progression” when deriving a QALY score (figure); however,the drug trials obtained data directly on quality of life, and these showed no measurable improvement.9 It is simply not possible to square the arbitrary cost per QALY estimates or the conclusions in the committee's report with the results provided by the trials.
Donepezil hydrochloride trials: pooled standardised weighted mean difference (95% confidence interval) for a quality of life scale, random effects model9
Similarly, the development and evaluation committee report examining the place of olanzapine in treating schizophrenia based an estimated cost per QALY for one year on trials that did not exceed six weeks of properly randomised follow up.11 Although this report found “good evidence of excellent value for money,” claims of appreciable cost saving or a measurable gain in QALYs when olanzapine was compared with conventional treatment seem optimistic when all the available published reports on atypical antipsychotic drugs are considered. It is alarming how few of the assumptions made in the reports mentioned here could be validated from an evidence based approach.
Undertaking more valid appraisals
In the context of the National Institute for Clinical Excellence, a number of fundamental changes need to be made to development and evaluation committees' methods. Those conducting appraisals should have the complete findings of trials used for licensing new products.It is reasonable for the NHS to expect such information from the pharmaceutical industry, unreasonable to expect clinicians to prescribe without such evidence, and important that the potential value of new treatments receives timely and independent scrutiny. This will avoid the sort of situation that occurred with the report on donepezil hydrochloride, where evidence of quality of life measured in trials and used for licensing purposes was not apparently available to the appraisers.
Producing systematic profiles
The objective of appraisal should be to enable clinicians and patients to reach informed decisions about treatments. A cost per QALY framework which requires substantial assumptions is the wrong approach.Instead, the institute's appraisals should produce a systematic profile of the attributes of treatment (effectiveness, tolerability,safety, health service delivery issues,resource implications, and cost), and comment on the absence or inadequacy of critical information. The profile should include best estimates of effect over a range of relevant outcomes.12 With such a profile, it should be possible to establish the “ball park”value for money of new treatments without reference to the questionable assumptions required to model an estimate of cost per QALY. Additionally, this approach will ensure that research priorities for those drugs which are categorised as “continuing research status” are identified correctly to further our understanding rather than build a better model.
“The government's stated aims describe a shift from clinical freedom towards care driven by protocol”
More extensive review
Rather than by examining a single drug, considerable information could be gained by considering other related products, particularly those with a similar pharmacological action. The development and evaluation committee report on donepezil hydrochloride could usefully have considered trials of tacrine, which has a similar pharmacological action but is associated with greater side effects. The olanzapine report should have considered the broader range of atypical antipsychotic drugs. Some of these have been licensed for use for several years, and properly randomised longer term data are available.Although this principle implies a more extensive review, it would strengthen the validity of recommendations.
“Horizon scanning” is inefficient
A First Class Service suggests that the horizon scanning approach, by which new and emerging agents are identified early and plans made for their assessment, should be extended.1 This may be an inefficient use of resources that could be better deployed in the rapid appraisal of drugs. Problems with horizon scanning are exemplified by experience with drugs such as lubeluzole, whose further development was halted because of poor results of phase III trials. Before lubeluzole was withdrawn, “new drugs for acute stroke” had been identified as a priority for the NHS Health Technology Assessment Programme in the expectation that lubeluzole would be licensed. An alternative approach could be for the National Institute for Clinical Excellence to require structured outline pharmacoeconomic impact profiles for all new products at a specified time (for example, one year) before the intended date of marketing to the NHS, so that major challenges to prescribing budgets can be anticipated.
Including the right people
Finally, rapid appraisal of new technologies conducted by the institute should include representation from relevant NHS health professionals. For example, if a new or existing drug will mostly be prescribed in primary care, it is important to include general practitioners in the interpretation of the research evidence.
Conclusion
Guidance provided by the National Institute for Clinical Excellence is one aspect of a range of schemes aiming to promote clinical governance and to make NHS organisations accountable for the quality of services provided. The belief underpinning clinical governance is that acceptable and unacceptable variations in health care are separable. However, audit to ensure that the right patient gets the right treatment at the right time is unrealistic, and simple performance measures are likely to be misleading. Evidence establishing the worth of many treatments currently provided by the NHS is inadequate. The government's stated aims suggest a shift from clinical freedom towards care driven by didactic protocol; it should give careful consideration to the dual issues of measurability and enforceability implicit in these aims. If health professionals are to respond appropriately to the concept of clinical governance, it is essential that methods of assessment are sound and transparent, reflecting both the available evidence and the uncertainties.
Footnotes
-
Competing interests None declared.