Read 022_eval_bas.pdf text version

Methods in The Evaluation of Publicly Funded Basic Research

A Review for OECD

Erik Arnold Katalin Balázs

Technopolis Ltd. 3 Pavilion Buildings Brighton BN1 1EE UK Tel: +44 1273 2043 20 Fax: +44 1273 747 299

A copy of this report is available from

The Evaluation of Publicly Funded Basic Research

Erik Arnold and Katalin Balázs Technopolis Brighton, UK March 1998 0 Introduction The evaluation of publicly funded research and technological development (RTD) has grown in most OECD countries during the 1990s. Immediate reasons include budget pressures and an increasing desire for accountability in the use of taxpayers' money. An underlying factor is probably the public skepticism that has been increasing since the 1960s about the previously privileged status of science and the value of technocracy. Declining or level funding for science and technology has increased the need for selection and priority setting tools which enable more strategic science and technology policies to be developed and therefore an allocation of public RTD funds which more closely reflects national economic and social priorities. The growth in evaluation has therefore gone hand in hand with `technology foresight' exercises, which aim to set the strategic directions for national RTD policies which are expected to generate unique national patterns of comparative advantage. In both the evaluation of industrially applied RTD and in technology foresight, there has been a pragmatic process of gradually improving the imperfect tools available. A craft, if not a science, of evaluation has evolved which is widely exploited in making policy for applied RTD, including innovation. This craft is based on relatively simple mental models of the relationship between applied research and application, combined with a large body of tacit knowledge and experience of how to manage this link in practice. Good practice involves the `agile' use of multiple imperfect evaluation tools to reach policy-relevant conclusions.1


Arnold, E. and Guy, Ken, `Technology Diffusion Programme and the Challenge for Evaluation,' Paper for OECD Conference on Policy Evaluation Practices in Innovation and Technology, Paris 26-27 June 1997


It is harder to see similar progress in policy evaluation of basic science. Various efforts have been made to improve ways to evaluate the quality of basic research. Some of these involve modifying peer review, which is science's own internal quality-control mechanism. Others, like increasingly refined bibliometrics, provide a more external view of quality. As in the evaluation of more applied RTD programmes, none of these methods works adequately on its own.2 However, the difficult step of evaluating the socio-economic impact of basic research has simply not been taken at a level of specificity which is useful for most policy making. Thus, for example, econometric approaches can tell us that more science is broadly good for the economy, but not which science. A key reason for this lack of specificity is that the link between basic research and the economy is very poorly understood. We have been forced to reject the simplicities of the various `linear' models of the science-economy relationship that prevailed in the 1950s and 1960s. These have been replaced by less predictive, more complex models and a discussion of National Innovation Systems3 which improves our understanding that the state needs to act in a holistic way in order to improve national economic performance but, again, offers little help in telling us how to set individual priorities. Evaluators of basic science trying to measure the impact of different scientific programmes on the economy today face a practical choice · · We can stay with the traditional evaluation focus on quality in basic science, and accept that we will then be able to say very little about impact at all We can try to stretch the tools used in applied and industrial technology evaluation into a shape where they can identify links between micro-level scientific activities and meso- or macro-level socioeconomic effects. To this alternative it has to be said that some of these tools are stretched to breaking point already in their existing applications Or we can try to understand at the overall level what kind of links there are between science and economic activity, and expect to use a combination of strategic planning and evaluation to manage the system. This involves lower ambitions for impact evaluation, with a focus on trying to measure only the measurable and therefore probably to say more about the plausibility of impact than to demonstrate it conclusively


2 3

Martin, Ben , `The use of Multiple Indicators in the Assessment of Basic Research,' Scientometrics Vol. 36, No 3, 1996, pp. 343-362 See, for example, Bengt-Åke Lundvall, National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning, London: Pinter, 1992


Given the state of the evaluative arts, the third option appears the most reasonable for the time being - even if it is not necessarily the most popular with those decision-makers who would like to make portfolio choices based on simple estimates of rates of return to alternative investments. In the longer term, it is clear that more research needs to be done not only on developing evaluation methodologies for the basic sciences but on reducing our ignorance of the mechanisms which connect basic research with society. In this short report, we first consider why it is useful to evaluate basic research at all, discussing separately the questions of evaluating quality and evaluating impact. Second, we set out what appears to be known today about the ways in which basic science creates socioeconomic impacts. Third, we discuss the relevance of different tools for the task of evaluating basic research. Finally, we draw some conclusions about the state of the art and what should be done to improve it. 1 Why Evaluate Basic Research? Internal quality evaluation is an integral part of doing basic science. However, as scientists increasingly work on problems defined by social or economic needs rather than following wherever intellectual curiosity and the scientific community may lead them, so the need to evaluate the use of research resources as means to solve these problems has risen. More recently, with basic science itself becoming increasingly resourceconstrained, the same pressure to make best us of resources means policy makers are looking for ways to evaluate the impact even of basic research. 1.1 Evaluation of Quality Since science makes progress through an iterative process of theorising, experiment, publishing and criticism4 then evaluation of quality is deeply embedded in the scientific process itself. Every journal article, every academic promotion and every research proposal is reviewed by scientific peers. The scientific community decides what is, and what is not, science and dominates the process of allocating resources among scientists. In this sense, science is very much socially constructed5 and cannot be done without internal evaluation by the scientific community. In basic science, traditional quality evaluation by research funders applies this logic wholesale rather than at the level of the individual scientist or experiment, for example in deciding about resource allocation to programmes or institutions. It can be thought of as simply an extension of the scientific quality control process itself, since both use similar methods and affect resource allocation.

4 5

Karl Popper, Objective Knowledge: An Evolutionary Approach, London: OUP, 1972 T S Kuhn, The Structure of Scientific Revolutions: 2nd edition, Chicago: University Press, 1970



Characteristics of Basic Research This view of science as an essentially self-regulating community with strong quality control procedures integrated into the knowledge production process is, however, only valid for part of the scientific endeavour. Gibbons et al6 argue that science is splitting into two modes of knowledge production. Mode 1 is traditional disciplinary science, especially basic science. The important recent changes have happened in Mode 2, which includes not only the practice of applied science in universities and other research institutions but also the generation of research-based knowledge elsewhere in society. The boundary between the traditional, university-based applied sciences and the other parts of Mode 2 has been broken down by the massive output of qualified scientists and engineers (QSEs) from the universities into the economy since the post-War expansion of higher education, and by changes in research funding systems which deliberately promote research across institutional boundaries. Scientists vary in the extent to which they stick to one Mode or move between Modes

Mode 1 is discipline-based and carries a distinction between what is fundamental and what is applied; this implies an operational distinction between a theoretical core and other areas of knowledge such as the engineering sciences, where the theoretical insights are translated into applications. By contrast, Mode 2 knowledge production is transdisciplinary. It is characterised by a constant flow back and forth between the fundamental and the applied, between the theoretical and the practical. Typically, discovery occurs in contexts where knowledge is developed or put to use, while results - which would have been traditionally characterised as applied - fuel further theoretical advances.7

We have summarised the distinctions between the two modes of production in Exhibit 1. Mode 2 does not simply apply results obtained in Mode 1. Distinct types of knowledge are also created in the process of doing Mode 2 work. One of the reasons why the Mode 1/Mode 2 distinction is interesting is that it focuses on how knowledge is produced, rather than on why.

6 7

Michael Gibbons, Camille Limoges, Helga Nowotny, Simon Schwartzman, Peter Scott and Martin Trow, The New Production of Knowledge, London: Sage 1994 Gibbons et al, p19


Exhibit 1 Distinctions Between Modes 1 and 2 Mode 1

Problems set and solved in the context of the (academic) concerns of the research community Disciplinary Homogeneous Hierarchical, tending to preserve existing forms of organisation Internal quality control

Mode 2

Problems set and solved in the context of application

Transdisciplinary Heterogeneous Heterarchical, involving more transient forms of organisation Quality control is more socially accountable

The economics profession in general thinks about the results of research as information, as if the results could costlessly be assimilated by any potential user. Callon has pointed out8 that for some important results of basic science this is not true. In many cases, only large and affluent companies have the complementary assets in terms of particular investments, capabilities and personnel needed to put specific scientific results to economic use. It is therefore arguable that at least some of the public investment in basic science leads to private rather than public returns. One of the functions of Mode 2 work can therefore be to appropriate outputs from Mode 1. Impact evaluation of basic research will need to take this into account. The logic of Mode 1 comes from its internal organisation and control mechanisms. Its institutions tend to be centralised and stable. In terms of education, Mode 1 tends to provide `basic training' and a disciplinary `entry ticket' (such as a PhD) for people to qualify as credible researchers in either Mode. In contrast, Mode 2 work tends to be transient. It forms and re-forms around applications problems. Calling on different disciplines and locations at different times, it is hard to centralise. Since Mode 2 work is performed in an applied, social context, it is normally subject to social and economic evaluation, and not solely to traditional quality reviews by scientific peers. To the occasional irritation of those used to the Mode 1 tradition, this means that relatively frequent evaluation - in part by nonscientists - is normal in Mode 2 work, and has become part of a `new social contract' between science and society.


Michel Callon, `Is Science a Public Good?' Science, Technology and Human Values, 19, pp 395-424



The Growing Requirement to Evaluate Basic Research There are at least five reasons why evaluation of publicly funded basic research has become increasingly important from the mid 1980s.9 First is that the economic environment is changing. Globalisation, growing competition and new technological challenges raise the priority of securing technological and economic impacts from basic research. At the same time, almost all industrialised countries face constraints on public expenditure. Many countries have cut spending on research, which made it more difficult to respond to the new challenges by fostering new scientific areas as opposed to supporting existing ones. Second, this pressure on funding has been intensified by the growing cost of scientific instrumentation, facilities and infrastructure. As a result, a number of countries have experienced a perceived crisis of underresourcing capital equipment in the universities. Decisions about spending large sums on the very sophisticated equipment needed to do frontier research in some disciplines (particle physics and astronomy are the most notorious, but the problem is much more widespread than this) have become hard to take. Decision makers are naturally looking for criteria and evidence to help them move beyond responding to scientific lobbying in the process of selection and priority setting. These decisions about allocating society's resources require ex ante evaluation not only of scientific potential but also of the socio-economic relevance of the research. Third, there is a more general but growing concern with the accountability, effectiveness and efficiency of public expenditure. Government anxiety about `value for money' has stimulated a critical approach to existing evaluation indicators and methodology and prompted a search for a more complete understanding of the role of basic research in economic performance. Fourth, problems are emerging with peer review which, since 1945, has been the principal mechanism for determining quality and resource allocation in basic research. Peer review was reliable while government funding was rising and a major objective was to identify and foster new areas. The method has become less satisfactory when it has to deal with potential loss of resources, declining fields and groups. A fifth reason for the growing significance of evaluating basic research is the growing role of international organisations in public funding, especially the European Union which justifies even its basic research funding as a contribution to European competitiveness. The comparative


The first four are drawn in part from Martin, Ben `The use of Multiple Indicators in the Assessment of Basic Research,' Scientometrics Vol. 36, No 3, 1996, pp. 343-362


novelty of this role for the European funding for basic research has become another stimulus to develop evaluation methodology further, in order to be able to measure the direct and indirect impacts at firm, national and European levels. 2 The Socio-economic Impacts of Basic Research It would be nice if we could simply pick up the micro-analytic tools of Mode 2 project impact evaluation and apply them to measuring basic research. However, the outputs of basic research are inherently more complex than those of Mode 2 work and more distant from application. Traditional innovation theory does not help much. However, SPRU has argued on the basis of a literature survey that there are six key mechanisms which link basic research to industrial growth. We have identified a seventh. These provide an initial model of linkage, though their relative importance must vary between branches of industry. Recognition of these links may play a useful part in future evaluation strategies for basic research, though better models need to be developed over time. 2.1 Can we use Mode 2 Evaluation Tools to Evaluate Basic Research? If evaluators are to try to measure the socio-economic impacts of basic research, we need a model of the relationship between the research and the impacts: to be able to explain the mechanisms that connect them. In problem-oriented, Mode 2 work it is conceptually easy to explain these mechanisms. Exhibit 2 shows an example. It considers NUTEK's ITYP programme, which co-funds R&D projects with companies in the Swedish service sector, aiming to · · · Increase productivity in services through better use of Information Technology (IT) Increase the professional skills of workers in services through IT Increase quality and competitiveness in Swedish services


Exhibit 2 Categories of Programme Results: ITYP Programme

ITYP Outputs

· Developed products, eg software · Developed tools · Intellectual property · Research reports · Demonstrations · Dissemination conferences and demonstrations, eg ITYP Forum · Information exchange among programme participants · Project participants' own dissemination activities · Experienced or trained people


· Changes in the company's willingness and ability to tackle service productivity and professionalism using IT · Changes in the company's IT and service capabilities · Financial return on project investment - Entrepreneur - NUTEK · Licence and patent royalties · Student and trained personnel movements


· Changes in other firms' awareness and willingness to tackle service productivity and professionalism using IT · Changes in other firms' IT and service capabilities· Raised services quality in a branch or in the economy · Improved Swedish economic performance · Increased competitiveness · Increased Swedish services exports

The Exhibit differentiates between three categories10 of effects, each logically more remote from the programme itself · · · Outputs: the technical results of the projects, such as software tools, management techniques Outcomes: the direct effects of the projects, such as new jobs created, increased productivity, measurable increases in workplace safety Impact: the wider effects of the programme on the society, e.g. faster diffusion of technology, increased service sector competitiveness

In principle, outputs cause outcomes, and outcomes cause impacts. Individual projects tend to focus on obtaining outputs, while programmes are about obtaining outcomes and impacts. Clearly, the further we look to the right in the Exhibit, the more remote the connections are between the projects and their effects, the more factors outside the programme come into play and the more difficult it is to say that an ITYP project alone was responsible for a result.


Some evaluation traditions use only two categories, typically with names like `direct effects' and `indirect effects', where the indirect category contains both outcomes and impacts. One of the reasons we prefer to use three categories is that this allows us to distinguish between outcomes, which can result in internal returns to project participants, and impacts, which are externalities or spill-overs to the rest of society


Specifying a model of the relationship between a basic (Mode 1) research project or programme and socioeconomic impacts is more difficult because all the impacts are indirect and because basic research is multidimensional in terms of its nature and outputs.11 These dimensions are at least · Scientific contributions to the stock of knowledge, such as new knowledge in an established field, opening up new areas of research or influencing other scientific fields. Outputs may include theoretical or empirical results, methodological progress or developing research instrumentation, or even the setting of new agendas in research or in policy Educational contributions in terms of skills and educated personnel, including research students. This includes codified knowledge in teaching and textbooks as well as the tacit knowledge embodied in people Technological contributions enabling the development of new improved products and processes Cultural contributions to society more broadly


· · 2.2

Does the Innovation Literature Help? If the literature has something to say about the mechanisms which translate basic research into socio-economic impact, this should be visible in the successive generations of `innovation models' developed in the post-War period. These have characterised innovation as increasingly complex and bound up with socioeconomic factors such as market linkage and match with the available infrastructure.12 The startling achievements of physics during the Second World War had made clear the immense power of science, reinforcing belief in science as a force for social change. The 1950s and 1960s saw significant efforts in many countries to build up their university systems and, often, dedicated research institutions. There were many reasons for this, including an increasingly democratic view of education as well as a belief that this growth would hasten economic reconstruction and development. Underlying these efforts was the now-traditional `linear' view of the innovation process as `pushed' by science. The policy implication of the linear model is simple: if you want more innovation (and therefore economic development), you fund more science.

11 12

Martin, Ben I b i d, 1996 Rothwell, R., `Successful Industrial Innovation: Critical Factors for the 1990s', R&D Management, 3, p 221-239, 1992


During the 1950s, the technology-push model of innovation dominated. Then, thanks to the empirical work of those such as Carter and Williams,13 Schmookler14 and Myers and Marquis,15 more emphasis came to be placed on the role of the marketplace in innovation. This led to market-pull or need-pull models of the innovation process. In the late 1970s, Mowery and Rosenberg16 largely laid the intellectual argument to rest by stressing the importance of coupling between science, technology and the marketplace. Their coupling model constituted a more or less sequential process linking science with the marketplace (via engineering, technological development, manufacturing, marketing and sales), but with the addition of a number of feed-back loops and variations over time in the primacy of 'push' and 'pull' mechanisms. Rothwell17 has charted this succession of innovation models into the 1990s (see the numbers in the Theory part of Exhibit 3, which crudely tracks the post-war development of innovation policy and theory). Exhibit 3 Post-War Shifts in Theory, Subsidy and RTD Policy

3 Theory 2 1 Subsidy Focus Technology Push Needs Pull 4 5 Coupling, Complex Systems

Big Cos, National Champions


Tax Incentives Foresight New programme forms Funding reforms University reforms


Collaborative programmes Economic, military competition Build up Universities, RIs, RAs 1950s

13 14 15 16

Commercialise RIs, RAs 1970s 1980s 1990s



Carter, C. and Williams, B., Industry and Technical Progress, Oxford University Press, 1957 Schmookler, J., Invention and economic growth, Harvard University Press, 1966 Myers, S. and Marquis, D.G., Successful Industrial Innovation, National Science Foundation, 1969 Mowery, D.C. and Rosenberg, N., `The Influence of Market Demand upon Innovation: A Critical Review of Some Recent Empirical Studies', Research Policy, Vol 7, April 1978 Rothwell, R., `Successful Industrial Innovation: Critical Factors for the 1990s', R&D Management, 3, p 221-239, 1992


His fourth and fifth generation models are essentially increasingly complex refinements of the third generation `coupling' model. Unfortunately, this `push-pull' debate about innovation focused mostly on the direction of influence in the innovation process. The models involved do not tell us how basic research links to industrial application. At the macroeconomic level, there have been several attempts to link basic research (or, often, science more generally) with industrial change. Mansfield's is one of the key studies18 looking at the relation between academic research and industrial benefits. He claims a 28% social rate of return on investment in academic research. More recently, Griliches has modelled the effects of different types of R&D on the performance of 1 000 companies in the USA, using unpublished data on R&D expenditure from the US Census. His model recognises that it takes time for R&D expenditure to have an effect on business performance, so it relates the stock of R&D built up by companies to output. In effect, Griliches treats R&D as a cumulative investment.19 However, even these recent studies are beset by difficulties of method and in attributing causation. There are no reliably accurate methods for estimating the value for money from publicly funded basic research. Nonetheless, the weight of evidence is that - at the aggregate level, and in large economies - the direct economic returns alone to science are much better than those available in more traditional investments, such as banking or the stock exchange. Pioneers of the new economics of knowledge approached the same issues by breaking with the mainstream in economics.20 Noting that cost-benefit evaluation of basic research investment has limitations and some undesirable consequences, David et al. proposed an alternative approach based on information theory.21 This emphasises the role of information generation and learning in the basic research process, information linkages between basic and applied research activities, and the incentives for the rapid and widespread dissemination of the knowledge emerged from basic research activities. They also underline that cost-benefit analysis may tend to focus mainly on `successes,' and moreover may fail to measure all the essential complementary inputs required to commercialise technological advantages deriving from basic research.

18 19 20


Mansfield, E. `Academic Research and Industrial Innovation' Research Policy, Vol 20, 1991, pp 1-20. Griliches, Z. (1995) `RD and Productivity' in Paul Stoneman (ed.) Handbook of the Economics of Innovation and Technological Change, Oxford: Blackwell, 1995 Peter Swann (1996) The Economic Value of Publicly Funded Basic Research. A Framework for Assessing the Evidence, Report for Branch 4, Technology and Innovation Division DTI, Manchester: PREST April 1996 David, P., Mowery, D. and Steinmuller, W. `Analysing the Economic Payoffs from Basic Research' Economics of Innovation and New Technology Vol 2 No 1 1992, pp73-90


Accordingly David has argued that it is better if economic analysis of the returns to basic research focuses more clearly on the productivity of the basic research activity itself, rather than the task of tracing economic benefit. David and Foray examine what they call the `distribution power' of an innovation system, which means its capability to ensure the timely access by innovators to relevant stocks of knowledge.22 They identified various characteristics of a successful distribution system, which include the capabilities of the adjacent domains of education and training, financing and industrial organisations. 2.3 Identifying the Mechanisms Linking Basic Research to Industrial Activity A recent literature review by SPRU on the relationship between publicly funded basic research and economic performance drew some rather more operational lessons and identified different types of benefits from publicly funded basic research.23 These different forms of benefits are interconnected and mutually supporting. · · · · · · Increasing the stock of useful information; New instrumentation and methodologies; Skills, especially skilled graduates; Access to networks of experts and information; Complex technological problems solving; Creation of firms, `spin-off' companies.

In a recent evaluation24 of Irish basic science, we have empirically confirmed the use of these mechanisms and identified a seventh category: Access to facilities, such as instrumentation. This often plays a role in the barter of ideas and resources which typifies scientific networks. Increasing the stock of useful information is the most obvious output of basic research. This leads to an expectation that any economic impacts of basic science are very long term. However while this is indeed partly the case, there are other sorts of knowledge and information outcomes of basic research which are assessable in shorter time scale. The economically useful output of basic research is above all codified information, which is a public good accessible by companies and other potential users. Users need to accumulate absorptive capacity to be able to access and understand new scientific results. Absorptive capacity includes a large amount of tacit knowledge and information about availability of new results. Moreover a

22 23


David, P, and Foray, D, Accessing and Expanding the Science and Technology Base, Paris: OECD, 1994 Ben Martin, Ammon Salter et al, The Relationship Between Publicly Funded Basic Research and Economic Performance, report to HM Treasury, Brighton: Science Policy Research Unit, 1996 Erik Arnold and Ben Thuriaux, Forbairt Basic Research Gants Scheme: An Evaluation, Dublin: Forfás, 1998


large element of new knowledge is tacit and is distributed though informal channels and personal contacts. Distribution of tacit knowledge and accumulation of absorptive skills are crucial for economic progress. New instrumentation and methodologies represent the `capital goods' of the scientific research industry.25 Their transfer to the commercial sector can create the basis for production as well as for industrial research activity. Huge amounts of effort and expertise may be expended on developing instrumentation and the equipment needed to do basic research. The European Centre for Nuclear Research (CERN) is an extreme example, where major engineering feats have been required in order to provide the equipment needed for experimentation in physics. However, it seems hard to predict when new instrumentation will be crucial to scientific research and when that research can safely be done using small-scale or commercially obsolete instruments. Equally, it is not always clear when instrumentation devised for research purposes will make the transition to industry. Almost all the major production equipment used in semiconductor manufacture originated in this way, while the bulk of the advances made in medical instrumentation in this century have originated in medical schools and research hospitals, not in the instruments companies. But, as with the information that emerges from basic science, there are often very long lags in adoption, and by no means all research instruments find their way into commercial use. Skills, especially skilled graduates, may well form the key short-term link between basic science and industry. Those firms needing graduates for R&D or in some other technical functions, actively seek research-trained people. "As far as companies are concerned, formal qualifications are ... evidence of researchers' tacit ability to acquire and use knowledge in a meaningful way. This attitude of mind ... is a most important contribution to new product development."26 Graduates of taught courses, such as typical first degrees, cannot offer this evidence. Graduates and post-graduates do not simply transfer from universities to companies the state of learning at the time when they passed their exams. Studies of basic-science roots of knowledge used in innovations show that industrial R&D workers use their education to keep up with the state of knowledge.27 So an important aspect of basic science education is to implant a capability for continuous learning in graduates and, therefore, among their employers.

25 26 27

Nathan Rosenberg, `Scientific Instrumentation and University Research,' Research Policy, 21, 1992, pp381-390 Jacqueline Senker, "Tacit Knowledge and Models of Innovation," Industrial and Corporate Change, 4, pp425-477 Michael Gibbons and Ron Johnston, `The Role of Science in Technological Innovation', Research Policy, 3, 1974, pp220-242; C Lyall, The 1993 White Paper on Science and Technology: Realising our Potential or Missed Opportunity? MSc dissertation, Science Policy Research Unit, University of Sussex, Brighton, 1993


Access to networks of experts and information is important for company R&D departments. It would be impossibly uneconomic for them to generate themselves most of the knowledge and information they use, so they need a strong `search' function to identify and absorb external knowledge. Formal and informal participation in scientific networks is therefore important. Derek de Solla Price pointed out28 that research scientists tend to organise themselves in global "invisible colleges," made up of people who are advancing the frontiers of knowledge. Within these colleges, discussions, draft papers, conferences and bilateral exchanges of various sorts provide members with privileged and early access to new knowledge. Networking and exchanging information is becoming more crucial between public sector research and companies even beyond competition. While competitors used to find pre-competitive areas of cooperation in the 1990s we notice strategic co-operation and competition at the same time what has recently been dubbed as `copetition'. Companies employ a number of means to be involved in such invisible colleges or networks, where this is important. For example, Pilkington's last head of research was a part-time professor of chemistry at Liverpool University. Hicks29 has shown that corporate R&D departments increasingly publish relatively basic research results in order to create an `entry ticket' to international scientific networks. This allows them to access technical opportunities in the science base, including the recruitment of skilled graduates who also carry with them bodies of tacit knowledge. The PACE Study30 emphasised the importance to firms of monitoring developments in public research, and the widespread use of informal, networking mechanisms to access these developments. The evidence is therefore that many R&D-performing companies consider links to basic science as important and are prepared to back up this perception of importance with resources devoted to monitoring. The importance of this type of monitoring and learning is also reflected in how companies participate in Mode 2 `pre-competitive, collaborative' R&D programmes. Complex technological problem solving is another way in which basic science contributes to the economy - in the sense of enabling the application of the stock of (basic) knowledge to industrial needs. It is not normally the basic scientists but others who put the stock of knowledge to use directly. Senker and Faulkner have shown that firms in hightechnology industries seek relations with the science base in two ways

28 29


Derek de Solla Price, Little Science, Big Science, New York: Columbia UP, 1963 Diana Hicks, `Published Papers, Tacit Competencies and Corporate Management of the Public/Private Character of Knowledge,' Industrial and Corporate Change, Vol 4, 1995, pp401-424 A Arundel, G van de Paal and L Soete, PACE Report: Innovation Strategies of Europe's Largest Firms: Results of the PACE Survey for Information Sources, Public Research, Protection of Innovations and Government Programmes, Final Report, Maastricht: MERIT, University of Limburg, 1995


· ·

As a source of new knowledge in specialist fields of science and engineering - it is vital that companies engaged in innovative branches keep up with developments at the leading edge of research As a source of practical help and assistance, often in response to specific problems, and frequently in the area of experimental methodologies and research instrumentation, for example in interpreting results from test equipment

These companies do not expect or use a meaningful flow of inventions from basic science. Rather, the contribution of publicly-funded research is made up of small, "invisible" flows, the cumulative effect of which is very significant.31 There is a geographical element to these flows. Because the major multinationals continue to do most of their research at or close to headquarters, there is no systematic relationship observable through the proxy of patenting in the USA between the locations in which they do R&D and the scientific strengths of host countries.32 Creation of `spin-off' companies is often thought to be a major benefit of research in academia and research institutes, yet the empirical evidence for this is at best mixed. A great part of the science park movement has been founded on the idea that there is a substantial pool of untapped ideas in the research sector which can be nurtured into commercial reality through new firm creation. Reality does not always live up to these expectations.33 In some cases, science parks come to be populated by those who find it attractive to be near a university rather than those who are genuinely exporting and commercialising scientific capabilities from it.34 Growth rates of such firms tend to be low.35 US research suggests a positive correlation between university research and firm growth in the electronic equipment sector, but - surprisingly - finds no significant relation in instruments.36 It is likely that the reason for this paradox is the extremely skewed nature of success and the correspondingly high death rates of spin-off firms. Government and other programmes to promote invention typically


32 33 34



J Senker and W Faulkner, `Public-private research linkages in advanced technologies,' paper presented at the Indo-British Seminar on Industry-Institute Interaction, British Council Division New Delhi, March 6-7 1995 Pari Patel, `Are large firms internationalising the generation of technology?' IEEE Transactions on Engineering Management, 1996 D Massey, P Quintas and D Wield, High-Tech Fantasies: Science Parks in Society, Science and Space, London: Routledgee, 1992 Ken Guy, Erkko Autio, Tomi Laamanen, Bill Wicksteed, Tero Kivisaari, Vesa Jutila, The Evaluation of the Otaniemi Science Park Cluster, Technopolis, Brighton, 1995 Rikard Stankiewitz, `Spin-off Companies from Universities,' Science and Public Policy, Vol 21, No 2, 1994; Erkki Autio, Simplistic and Generative Impacts of New Technology-Based Firms in Innovation Networks, Doctoral Dissertation, Institute of Industrial Management, Helsinki University of Technology, 1995 Bania, Edwards and Fogarty, 1993


involve filtering out the overwhelming majority of ideas in order to find a handful of commercially feasible ones. Only a small minority of the feasible ideas goes on to become profitable. However, large number of spin off companies are not based on new products but rather on knowledge and skills accumulated in the public sector. There is some evidence that although these firms may not always keep strong contact with their academic roots or create entirely new products, they nonetheless contribute to regional economic performance. There is indeed a link between science funding - including basic science and the creation of new knowledge-based firms. These firms may contribute to job creation and to knowledge and technology transfer in the relevant region.37 In understanding how these categories of academic-industry links work, it is important to note that the relative importance of direct and indirect transfers differs a great deal between industrial sectors. Direct linkages visible through patent and citation data are clearest between basic chemistry and the chemicals industry. Half the links identified by Pavitt for the USA were in this field, with another 20 - 30 percent linking basic science with electrical and electronic products. In contrast, less than 10 percent of the measured linkages were with non-electrical machinery, automobiles and aerospace - which together employ almost half the qualified scientists and engineers in the USA.38 This latter group is more engineering-oriented and its links with basic science are more indirect. The `PACE' study explored the way individual scientific disciplines relate to branches of industry. In Exhibit 4, we have classified the disciplines and industrial sectors and shown the intensity of the links found in the PACE survey of large firms. The `Overall' column in the Exhibit shows the proportion of the companies sampled which indicated that publicly funded research in the science shown during the past ten years was `very important' to their technology base (scoring 5 or more on a 7-point scale where 1= not important and 7=very important). As might be expected from the 10-year horizon chosen, the applied and transfer sciences were generally more important to them than the basic sciences.

37 38

Balázs, K. `Academic Entrepreneurs and their Role in Knowledge Transfer' STEEP Discussion Paper No 37. SPRU, University of Sussex, November 1996 Keith Pavitt, `National policies for technical change: Where are there increasing returns to economic research?' Paper prepared for the Colloquium on Science, Technology and the Economy, organised by the US Academy of Sciences at the University of Irvine, 1995


Exhibit 4 Importance of Publicly Funded Research in Last 10 Years to Companies' Technology Base

Engineering-Based Other Science-Based

Transfer/Applied Science · Materials 47 · Computer 34 · Mechanical Eng 34 · Electrical Eng 33 · Chemical Eng 29

77 60 64 73


72 47 56

63 56 70 47 64 53 47 47 78 46 55 60 46

Basic Science · Chemistry 29 · Physics 19 33 · Biology 18 · Medicine 15 · Mathematics 9 20 20 Source: Arundel et al, 1995; Technopolis analysis

33 64 27 25

78 25 25 33 15 13 71 85

52 18

46 17 29

The remaining columns show the top-four scoring industries when the same responses are analysed at the individual sector level. The numbers in the columns show the percent of firms saying the relevant science was `very important'. This confirms Pavitt's analysis that the engineeringbased industries tend to lean heavily on the applied and transfer sciences. Chemical engineering has a markedly different pattern of links with industry from the other transfer disciplines because it is effectively provides `engineering to the science-based industries'. Three of the basic sciences (chemistry, biology and medicine) have strong links are with the corresponding science-based industries. Physics and mathematics are different: they provide underpinnings to engineering rather than supporting their own unique industries. Among the engineering-based industries, computing is unusual in using a good deal of both transfer and basic sciences, presumably in part because it has a high electronics content. Existing research shows, then, that the mechanisms through which basic science contributes to technological problem-solving are not always direct. We cannot simply think of basic science as putting new information into a bucket, into which applied scientists and engineers dip for ideas. Rather, the flows look more like those shown in Exhibit 5 - which should be regarded as broadly illustrative rather than as being definitive.


Exhibit 5 Knowledge Flows in Technological Problem-Solving

Applied and Transfer Sciences

· · · · ·

Materials Science Computer Science Mechanical Engineering Electrical and Electronic Engineering Chemical Engineering

Engineering Industries

Basic Sciences

· · · · ·

Chemistry Physics Biology Medicine Mathematics

Science-Based Industries

Major flow

Minor flow

Equally, these flows are only possible when appropriately qualified and experienced people are available in industry to translate and develop the tacit knowledge needed to exploit the codified flows shown in the Exhibit. 2.4 Implications for Evaluating Basic Research Macro-level econometric approaches to understanding the impacts of basic science on the economy suggest that there is a big, positive relationship between doing basic research and economic growth but are incapable of giving us enough detail about which piece of science does what to let us choose between alternative scientific investments. On the other hand, the multiplicity and indirectness of the links between basic research and industrial impact means that we cannot rely on the type of project-focused approach often used in evaluating the socio-economic impact of more problem-oriented work. We rejected what we might call the `simple bucket theory of linkage': the idea that basic research puts ideas into a `bucket' and that applied scientists and engineers dip into the bucket when they need to solve problems. At the very least, it seems that there are multiple buckets (or buffers) in the system, multiple hose pipes connecting them and that the hoses feed both back and forward (Exhibit 6).


Exhibit 6 The Innovation Chain

Basic Research Applied Research Experimental Development Development Engineering Standards & Certification Diffusion & Adoption

The buckets or buffers in the system are not, however, just the stages in the innovation chain illustrated in the Exhibit but also comprise many of the actors and institutions involved. For example, a basic researcher's thinking may suddenly become useful in the applied sciences once the results and meaning of several new experiments are combined with old theory and some data from a different field of science. Progress in basic research may only become possible once an instrument company has solved a particular engineering problem, and so on. If evaluators are to say anything about basic research at a sufficient level of detail to be useful, it is nonetheless the micro approach which must be pursued. A macro approach simply aggregates away the level of detail which is needed for policy making. As we will show in the next section, existing tools are not particularly good at evaluating the impacts of basic research, though some of them are rather better at evaluating its quality. The agenda for basic research evaluation must therefore be both to improve our crude models of the links between basic science and the economy and to improve evaluation tools further.



Tools and Techniques `Evaluation' is sometimes discussed as if it were a unified concept or as if the field were somehow evolving towards a single true `theory' of evaluation. We reject this `one size fits all' approach. Rather, we view evaluation as offering a set of tools and techniques which need selectively to be adapted and used in order to answer questions about performance. Behind any evaluation is a customer who needs to know something usually in order to take a policy decision. The techniques traditionally used to evaluate basic research are internally focused on the science itself. Evaluating basic science in terms of socioeconomic impact introduces several new issues. Existing evaluation tools help resolve some of these, but major technical problems remain.


Who Needs Evaluation? The type of evaluations considered here are normally launched by policy makers and others who need answers to specific questions. Exhibit 7 illustrates the four main stages of the evaluation design process and relates them to each other and to subsequent phases. Feedback loops make the whole process iterative rather than strictly sequential. Understanding should precede the formulation of strategy, but the actual process of choosing a strategy itself often improves understanding - and similarly for the other stages. Critically, also, the same relationship exists between the design and the conduct stages. Exhibit 7 Evaluation Design Steps

Understanding the Context

Developing an Evaluation Strategy

Specifying Evaluation Tactics

Establishing an Operational Agenda

Conducting the Evaluation

Initiating Follow-on Actions

Design Phase

Understanding the context of the evaluation involves unravelling user needs. The degree of novelty of the type of intervention involved is key · · · Understanding why and for whom an evaluation is being commissioned Identifying data sources and interrogation techniques likely to be useful An appreciation of rationale, history and objectives is necessary if an appropriate evaluation is to be designed and implemented 20

Choosing an evaluation strategy is largely a case of deciding upon the central issues which are to be investigated and the overall approaches needed to explore them. This calls for an understanding of · · · The range of issues which can feasibly be included within the scope of an evaluation The factors which can influence the choice of issues The general approaches which can be used to explore them

To a large extent, understanding the context of an evaluation and the initiative to be evaluated helps define the issues to be explored. Other factors affecting the importance of particular issues are the nature, timing and ambition of the evaluation. An important dimension which helps define the overall evaluation strategy is the balance between formative and summative ambitions. Specifying evaluation tactics involve characterising the system to be investigated, and choosing the procedures and techniques to be used. Care has to be taken in the specification of the system to be evaluated, particularly delineation of the system boundaries; the relationship of the system to the environment in which the system is located; and the definition of all relevant sub-systems and their relationships with each other and with the system as a whole (Exhibit 8). Exhibit 8 Systems and their Environments

Environment System Boundary Sub-system Sub-system Relationships System-Sub-system Relationships System-Environment Relationships

Many of the evaluation issues defined in an evaluation strategy phase can be described in terms of relationships between system variables. For example the ratio of expected outputs (Oe) to actual outputs (Oa) from a


system is a measure of goal attainment or Effectiveness. Exhibit 9 shows how some critical evaluation issues can be defined using a very simple systems model and a conceptual calculus relating basic input and output system variables. Exhibit 9 A Conceptual Evaluation Calculus

System Variables Ia Ie Oa Oe = = = = Input achieved Input expected Output achieved Output expected I a/I e Oa/Oe Oa /Ia Evaluation Issues = = = Economy Effectiveness Efficiency

Ie S Ia



Choices of system variables to be used are normally dictated by three factors · · · Decisions made in the strategy phase of the evaluation design process between output- and/or process-oriented approaches The set of issues, variables and indicators which the strategy phase of the evaluation design process has determined to be of most relevance The set of resource constraints within which the evaluation has to function

Once these matters are decided, setting the operational agenda is a matter of routine project management. 3.2 Internally-Focused Evaluation of Basic Research Martin recently summarised39 the three `internal' dimensions of basic research which are conventionally evaluated: scientific activity, production and progress. Scientific activity is concerned with the consumption of inputs - resources - and is related to such factors as the number of scientists involved, the level of funding, the number of support staff and the scientific equipment required. Evaluation of the reasonableness of resource consumption is


Martin, Ben, I b i d, 1996


based on peers' judgements, comparative analysis of indicators of resources and measuring against constraints (typically budget constraints). Scientific production refers to the body of scientific results embodied in both research publications and other types of less formal communications between scientists. Numbers of scientific publications - that is, articles reporting substantive research results published in peer-reviewed journals - are a reasonable measure of the amount of scientific production, though not necessarily of its value to science. Scientific progress refers to the extent to which scientific activity results in substantive contributions to scientific knowledge. Although some output indicators are fairly closely linked with scientific production, their relationship with scientific progress is more complex. However, indicators of scientific progress are more relevant than others to assessing scientists' success in fulfilling the primary goal of basic research, the production of new scientific knowledge. As we indicated in Section 1.1, the evaluation of research quality is a prerequisite for the operation of a scientific community (see, for example, Weinberg40 ). Many definitions of research quality focus on actual or potential contributions to scientific progress. However quality of research is also a relative concept depending on time, the state of the field in question and the characteristics of the evaluator.41 As an indicator on its own, scientific quality tells us little that allows us to make absolute comparisons between fields or to predict impact `downstream.' However, we have found that the scientific quality of projects in Mode 2 R&D programmes is well correlated to their effectiveness42 and this suggests it may be worth looking for a broader link. Evaluations of these aspects of research handle the scientific dimensions of basic research using the values of the scientific community. They consider the impact of basic research on other researchers and research results. However, such internally-focused evaluation cannot tell us enough to be useful about success and failure in the real world, about the efficiency of public expenditure, or about decisions about research areas. These depend on the wider social and economic context of basic research.

40 41 42

Weinberg, A. `Values in Science: Unity as a criterion of scientific choice,' Minerva, Vol. 22. No. 1, 1984 pp1-12 Luukkonen-Gronow, T `Scientific research evaluation: a review of methods and various contexts of their application,' R&D Management 17, 3, 1987,pp. 201-221 see, for example, Erik Arnold and David Gann, Evaluation of the IT-BYGG Programme, Stockholm: NUTEK, 1995; Erik Arnold, Peter Cochrane, Anne Lavery, John O'Reilly, Sven-Olof Öhrvik and Matt Staton, Teltec Ireland: An Evaluation, Dublin: Forfás, 1996



Evaluation of Basic Research in a Social and Economic Context In order to step up to this bigger evaluation challenge, we have to move from the comparatively narrow set of issues considered in internallyoriented evaluation to the more generic questions addressed in Mode 2 evaluation. Since publicly funded basic research is a creature of public policy, policy frameworks, objectives, priorities and decisions need to be included as evaluation issues. These determine research strategies, funding levels, choice and style of research management and links to the wider environment, thus influencing social and economic impacts. The ability of basic research to affect economic performance depends not only on investment in the research itself and on the quality of research performance but also on several other factors, notably the absorptive capacity of those who can potentially exploit the results of the research, employ people who have done it and otherwise interact with the basic research community. Evaluation of basic research in the wider social and economic context can therefore also involve considering other parts of the innovation process. Exhibit 10 shows a set of generic evaluation issues which spans the larger set of questions that need to be tackled in socio-economic evaluation of basic research. Exhibit 10 Generic Evaluation Issues for a Policy Action Issue

Appropriateness Economy Effectiveness Efficiency Efficacy Process Efficiency Quality Impact Additionality Displacement Process Improvement Strategy


Was it the right thing to do? Has it worked out cheaper or more expensive than we expected? Has it produced the expected effects? What's the return on investment (ROI)? How does the ROI compare with expectations? Is it working well? How good are the outputs? What has happened as a result of it? What has happened over and above what would have happened anyway? What hasn't happened which would have happened in its absence? How can we do it better? What should we do next?


To address these issues, we need to measure and explore various system variable. Exhibit 11 extends slightly the simple model in Exhibit 9 to show some of the other relevant variables. Exhibit 11 A Simple Systems Model and Variables of Interest

Policy Resources Outputs Outcomes Impacts

Structure and Operation of t h e Process being Evaluated


Policy variables characterise policy decisions on basic research - objectives, strategy, long term policy aims. Evaluation is first of all an assessment of policy realisation looking at other variables against policy objectives. Resource variables characterise the origins of the many inputs into a system, including public funding for the given project or program, other funding (e.g. private company sources), scientists and skilled personnel, research instruments and available infrastructure, capabilities of the research group or company environment. Structure variables help characterise the system of basic research itself, describing the research organisation, structure of research group and its relationships. Process variables help define the way in which the system operates and functions and how it is organised and managed. Output variables describe the direct results of the research. These can be scientific research results, educational outputs, technological results which may enable the creation of new products, processes, creating new intellectual property and increasing technological awareness. Traditional basic research evaluation approaches collect quantitative and qualitative data on outputs and match these with quality judgements. Outcomes variables describe the changes and benefits which result from the availability of the outputs. Many of these are benefits internal to the research-performing organisation. Evaluation of outcomes involves multiple indicators and techniques. 25

Impact variables characterise the effect outputs and outcomes have on the broader environment. This can include the impact on the scientific community as well as other non-research effects, for example on educational provision and needs. The social and economic impacts of basic research include increased productivity, innovation capabilities and competitiveness. Not all the variables can be used to illuminate all the issues. Exhibit 12 suggests which variables should be tackled in order to address the generic issues. Exhibit 12 How System Variables Address Evaluation Issues Issues

Policy Appropriateness Economy Effectiveness Efficacy Process Efficiency Quality Effects Additionality Displacement Process Improvement Strategy

System Variables

Resources Structure, Process Outputs Outcomes Impacts

However, it makes no sense to attempt this breadth of evaluation in the abstract. The more structured and formulated the policy objectives of basic research funding are, the more clearly focused evaluation can be. If the policy aim is to improve competitiveness in general terms, the evaluation tools we have today will not be up to the job. If policy is more focused on outputs or outcomes directly - e.g. improving knowledge dissemination, networking or distributing new process technology - evaluation can focus on these variables. 3.4 The Usefulness of Alternative Evaluation Tools Because it is a professional literature, much of the technical evaluation literature focuses on the use of individual tools and techniques. No strategy for systematic tool development is visible, though the resulting patchwork suggests a strategic approach might be a good idea. In this section, we briefly review the available tools and techniques and discuss their usefulness in the context of evaluating basic research. 26

Quantitative data - e.g. bibliometrics, citation and co-word analysis, and patents - tackle research outputs and quality of research. While they are often supposed to be more objective than qualitative methods, they can turn out to be less reliable since they measure only certain narrow aspects of research and its exploitation. Thus, qualitative information is indispensable to complement numerical data. Qualitative approaches may start with collecting quantitative data and may produce results which are partly quantified. Qualitative approaches like peer reviews, expert panels, spectral surveys, and case studies need to be structured and designed to bring in the relevant information. Case studies involve examining a limited number of specific cases or situations which the evaluator anticipates will be revealing in order to understand the dynamics within specific settings. Case studies are powerful where a `learning' evaluation needs to be done - where a new kind of intervention is being tried or where actors are inexperienced in such intervention. The approach allows evaluators to learn as they work, gradually building up a model of how the intervention works. Ideally, this then provides the basis for more structured approaches such as user questionnaires. Bibliometric indicators describe properties of the scientific literature through the "application of mathematics and statistical methods to books and other media communications".43 The construction of databases suitable for bibliometric studies has made bibliometric indicators easily available and this has established their use for evaluation purposes.44 However it is an essential problem that most publications make only a very incremental addition to knowledge, while only a very few make a major contribution. Nevertheless, scientometric studies continue to make far more use of this indicator than any other. Further criticisms of bibliometrics related to the databases themselves which are overwhelmingly Anglo-Saxon, discriminating against small countries and non-English publications. Further, bibliometrics discriminates against new emerging scientific fields. The main uses of bibliometrics are to compare large-scale patterns, such as inter-country differences in publication patterns, or to map the development of a field over time. For small-scale phenomena - such as individual basic research programmes the smaller numbers of publications involved means that comparisons of numbers of publications are close to impossible to interpret. The related technique of citation analysis is based on the fact that most articles, notes, reviews, corrections and correspondence published in scientific journals contain citations. Citation indexing and analysis is built on the assumption of linkages or relationships between the citing and the cited documents and that all linkage are of equal value. The aim of citation analysis is to estimate the varying contributions to scientific

43 44

Pritchard, A `Statistical bibliography or bibliometrics" Journal of Documentation, Vol 25, No 4, 1969 pp348-349 Luukkonen-Gronow, T, i b i d, 1987


progress made by different publications. However there are serious `technical problems' and also substantive conceptual problems with the use of citation, such as critical citations of mistaken work, the failure to cite early classics, variations in citations rates across fields and type of paper (e.g. methodological, empirical or theoretical papers), and the `halo' effect.45 Co-word analysis is beginning to be used as a mechanical but rigorous way to analyse the content of papers in relation to other scientific publications. Both co-citation and co-word analysis are useful for mapping scientific fields. They provide relational indicators which group papers to show the structure and dynamics of science. As with most quantitative methods which are built on models, results depend on the starting assumption and on the selection of variables involved. Peer evaluation is based on scientists' perceptions of contributions by others and is influenced partly by the magnitude of those contributions and partly by other factors. It covers a wide range of methods in which peers express opinions. However peer review needs to be guided and modified according to the particular evaluation's objectives and methods. Peers are important in research quality evaluation since they represent the key reference point: namely, the state of the art of the given scientific field. Peer review is one of the most common qualitative evaluation techniques. Many government science and technology funding agencies use peer review as a primary mean of evaluation. It is increasingly used in combination with other indicators, especially in Mode 2 evaluations. However there are problems with peer evaluations too. Political and social pressures within the scientific community may affect the way scientists assess the contribution of their peers. Peer review depends on finding neutral peers. Modern research is, however, increasingly competitive and at the same time tend to be concentrated in smaller number of large centres which makes it difficult to find truly neutral peers in the more capital-intensive areas of research. Different peers in different cognitive and social locations may evaluate differently. Peers cannot have perfect information, therefore they base their assessment on limited and imperfect information. In general, it is our experience that peer reviewers tend to give most weight to scientific quality in forming judgements. However peer review means only the involvement of peers professionals, who represent, quality and values of their field. Peers are important contributors in evaluation, since they are the experts and remain the most reliable, `rounded' source establishing scientific quality. There is a risk that peers' evaluation is biased, although in practice this can be reduced by using structured guidelines, evaluation strategy and strict evaluation management.


Martin, Ben, I b i d, 1996


Peer review techniques which go beyond quality of science and include issues concerning social and economic benefits are described in the evaluation literature as modified peer review. In this type of evaluation, scientific experts are supported or work with specialist evaluators and other non-peer members of the evaluation team - such as social scientists, economists or business people, and potential users - to be able to assess outcomes and impact. Modified peer review's key weakness is the small number of individuals who can be involved. In many types of evaluation, it is necessary to use complementary techniques in order to compensate for this. Patents analysis has much the same strengths and weaknesses as bibliometrics. It is very useful to mapping R&D activities and for understanding large-scale events in innovation and for comparing aspects of the performance of economies and branches. At the level of smaller events - such as basic science or R&D programmes - the lack of data-points makes it difficult to say much that is statistically meaningful about performance. Questionnaire methods and surveys are among the most important tools at evaluators' disposal. These mix qualitative and quantitative methods. Once a model has been built of the intervention to be evaluated, surveys allow hypothesis testing and detailed exploration of both process and impacts. In many cases, especially in evaluating wider impact of basic research, one aspect of surveys effectively involves asking people about their perceptions of how well they performed a particular task and how they assessed results. Replies, need to be interpreted accordingly. These methods provide a more systematic review, gathering wide number of experts, clients and potential users and overcoming restrictions of modified peer reviews with a limited number of participants. Survey results can be tested and validated by follow up interviews and case studies. Cost-benefit analysis and related financial approaches which calculate rates of return or returns on investment have the strongest theoretical foundation for the task. However, their limits are also widely understood. Mansfield46 used a backward tactic tracing innovation back to their academic source. He used information from the firms, an estimate on he relationship between certain academic research and total sales of a relevant product. On the basis of Mansfield's work, the BETA group at the University of Strasbourg attempts to calculate direct benefits at company level.47 Their methodology makes a distinction between technological, commercial,



Mansfield, E `Academic Research and Industrial Innovation,' Research Policy 20, 1991, pp1-12; and `Academic Research and Industrial Innovation: A Further Note' Research Policy 21, 1992, pp295-296 Bach, L. And Lambert, G. (1993) "Evaluation of the Economic Effects of Large R&D Programmes: The Case of the European Space Programme" Research Evaluation, 2,


organisational and work factor effects and quantifies these in terms of `added value'. Managers are asked to estimate the contribution to sales from these four categories. The cost benefit ratio is calculated as the ratio of indirect benefits to the total payments. Although this method is intended to be quantitative and objective it includes multiple subjective estimates of outcomes. Both benefits and costs are in practice difficult to estimate, notably where these are indirect rather than direct. Direct costs of a specific research project or programme should be complemented by an estimate of costs of infrastructure and human resources on the one hand and by an estimate of the additional factors playing role in research commercialisation. Many of the social and economic benefits of basic research may not surface in exploitable products and processes but feed back into the knowledge and capabilities of the researchers. Many of the benefits are intangible, and cannot be assessed by cost-benefit analysis.48 Many people in the European RTD evaluation community have balked at using cost-benefit techniques because the numbers they produce can be extremely misleading. The main reason why they have not been widely used in RTD evaluation is that they involve large uncertainties and methodological problems while at the same time producing authoritativelooking numbers. A central problem is that the state invests in RTD programmes on behalf of society in order to reap the externalities: namely, the benefits which are not captured by the direct beneficiaries of the programme but which leak away to society more generally. Unfortunately, while cost-benefit techniques are moderately good at counting the internal benefits of projects - for example the benefits to a firm of participating in a technology transfer programme - they systematically ignore these external benefits. This is a pity, because it is just these external benefits that matter to the state in its role as investor. The problem of multiple causality is another of the difficulties in counting benefits. One way to reduce the problems in counting benefits is to be less ambitious: to count what can easily be counted; and to ignore the rest. This immediately makes the analysis less useful for the high-level policy maker who must choose between alternative investments. If we do costbenefit analyses of two programmes - A and B - leaving out the hard-tocount benefits we can no longer compare the results. We do not know whether we are counting the same proportion of total benefits in each case. If it turns out we are counting 10% of the benefits of A and 90% of the benefits of B, we will probably end up with poor policy recommendations. This is especially dangerous if, say, A is a basic science programme whose benefits are inherently difficult to monetarise, and B is an applied science or innovation programme whose monetary benefits may be no bigger but which are easier to estimate in financial terms. Even


Luukkonen-Gronow, T. 1987, i b i d


if we use the same method, we cannot simply assume that we are counting the same proportion of the benefits in each case. The programmes may use quite different mechanisms. The parts of the economy in which they operate may have quite different characteristics in terms of their ability to exploit externalities.49 In addition, different evaluators may - knowingly or accidentally - use different methods for counting benefits. It is still possible to do cost-benefit analysis with a lower level of ambition. We can ask: If we limit ourselves to counting that which is countable and take a systematically conservative view of benefits, can we still find enough benefits to justify the state's investment in an RTD programme? This is a useful half-way house. The discipline of numbers provides a good check on the evaluation process overall. But it has to be recognised that evaluations which take this approach are incommensurable. Logically, they cannot be used simply as inputs to an arithmetic portfolio management process, where policy makers choose the programmes with the highest rates of return. This is, in any case, not a sensible policy choice process. If national innovation systems really are systems, then the parts of the system are interdependent. Investing in the high-rent parts of the system and ignoring the apparently low-rent parts will not necessarily increase the GDP. It is more likely to help the innovation system to collapse. Each evaluation technique has strengths and weaknesses which suggest that it is most important in the course of the evaluation to select the right assessment techniques which are capable of addressing the evaluation objective. Evaluation of phenomena as complex as basic research and innovation require a systems approach and a group of evaluation techniques which complement each other. Exhibit 13 summarises strengths and weaknesses of individual tools and techniques. Exhibit 14 gives an impressionistic rating of how well the various tools perform in evaluating basic research as compared with applied R&D. It is striking how many weaknesses can be identified, and this underscores the importance of using multiple techniques in parallel in order to increase our confidence in evaluation results to acceptable levels. Single-tool evaluations are inherently unreliable.


Industrial structure, varying technological capabilities, different degrees of appropriability of different technologies are among the factors that could account for this kind of difference



Conclusions Conclusions in this field - the evaluation of basic research - must be very preliminary. While a fair amount of work has been done in the area over the years, the resulting techniques are not very good for moving beyond internal evaluation of basic science to understanding its socio-economic effects. Some of this weakness may be rectified by further efforts in the evaluation community. Some of it may be a result of the inherent difficulty of the problem. For the time being, we are more or less forced to use the existing techniques with care, while not claiming too much for them. Available techniques will not give a complete answer on the social impacts of basic research. However, tactical use of evaluations and evaluation techniques as part of larger strategic planning exercises will allow some of the strategic assumptions to be checked. Clearly, a research agenda is needed. First, our ability to understand and model the relationship between basic research and the economy is too weak. More research is needed here - not only into the basic research itself but also into the capabilities in other parts of the innovation system that allow its outputs to be put to use. Second, in parallel, we need to continue to experiment with evaluation tools in order to calibrate our (hopefully, growing) understanding of this relationship. In the meantime, decision-makers will continue to ask simple questions about how to understand priorities and the effectiveness of alternative programme approaches. It is the responsibility of the evaluation community not to give simplistic answers.


Exhibit 13 Strengths and Weaknesses of Individual Evaluation Tools (1) Tool

Case Studies + + + + + -

Strengths and Weaknesses

Help understanding of complex processes Explore situations where interesting variables not predefined Can be structured and subjected to analysis Provide `how to' understanding, for example of success-factors Highly dependent on evaluators' skills and experience Expensive if done in large numbers Hard to incorporate into routine monitoring Generate limited quantitative information Useful for understanding some macro-level patterns, especially in more basic types of research Unhelpful where populations are small, as in a single research programme Focus on academic research. Under-values other types Differences in propensity to publish among disciplines and at different stages of the `innovation cycle' Number of publications not a reliable indicator for quality or scientific progress Meaning of citation counts ambiguous: Positive or negative citation? Increasing share of collaborative, multi-authored publications in journals is hard to evaluate Bias towards English-language journals Under-represents new and emerging fields Gratuitous co-authoring and co-citation as authors manipulate counts. `What you measure is what you get' Useful for mapping fields Immature technique. Meaning unclear for evaluation purposes Informed, `rounded' judgement, especially of scientific quality Can be systematised, checked and analysed to increase confidence in results Typically over-influenced by scientific quality criteria Qualitative, judgement basis leaves it open to criticism Problems of criterion-referencing and differing cultural behaviours `Group think' and social dominance effects within panels Risk of `prisoner's dilemma' behaviour by peers Hard to apply to commercially-sensitive work Useful for understanding some macro patterns and problems relating to programme appropriateness Inherently unlikely to captures outputs from basic research Patents indicate neither technical nor economic success Variations in national patent systems Variations in patenting propensities between countries, branches of industry and individual companies/institutions Tell nothing about non-patented or non-patentable aspects


Co-word Analysis Peer Review

+ + + -

Patents Analysis

+ -


Exhibit 13 Strengths and Weaknesses of Individual Evaluation Tools (2) Tool

User Surveys + + + + + -

Strengths and Weaknesses

Can provide a nuanced, quantified understanding of programme Collects direct process experience as well as indicators Can test and generalise case study and other findings Enables estimation and description of key impacts Provides quality control of programme management Subject to positive bias, reflecting users' appreciation of receiving resources and optimism about impacts Requires a rigorous and systematic analysis of costs and effects Forces the construction of a model of the action being evaluated It is possible to generate coherent but widely differing benefit/ cost ratios Focus on money creates a risk of under-valuing non-monetary costs and benefits Typically does not handle externalities/spill-overs Costly to collect the needed detailed benefit and cost data Difficult to deal adequately with multiple causality Deals poorly with multi-step causality

Cost/Benefit Analysis etc

+ + -

Exhibit 14 Usefulness of Various Tools in Basic Research and R&D/Innovation Programme Evaluation Tool

Case Studies Bibliometrics Co-word Analysis Patents Analysis Peer Review User Surveys Cost/Benefit Analysis etc

Basic Research


Key: Shading of the spheres denotes increasing usefulness



35 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate


Notice: fwrite(): send of 202 bytes failed with errno=104 Connection reset by peer in /home/ on line 531