Innovation policy programs and instruments are very ambitious. Measuring their effects must focus broadly and not narrowly on their added value.

Many countries have developed ambitious public policies, programs and instruments that promote innovation in the economy, and the solution of complex societal problems. The logic behind this is the expectation that public programs can boost and complement the insufficient levels of investment by private actors. In other words, public programs and public investment seek to act as a catalyzer, complementing private levels of investment and increasing the overall levels of investment in an economy. Understanding this complementary role is essential, because public investment must never crowd out or substitute private investment.

This logic is very important to keep in mind. Particularly when measuring the effects of these public policies and programs. Generally speaking, there are two possible perspectives on this; a narrow and a broad perspective. The narrow perspective measures the effects of these public programs in terms of the private rates of return for the individual firms/organizations that took part in individual projects. Here the perspective is to measure whether the public investment did result in a concrete advancement for that individual firm, compared with a sample of firms which did not obtain public support. This approach is valid because it seeks to examine value-creation at the individual level of the participant, aggregating it subsequently. It is valid and methodologically sound. But it is insufficient for two reasons:

Firstly, we know from previous studies, that the distribution of outcomes of these programs is highly skewed (Scherer and Harhoff, 2000). Meaning that, few of the firms receiving public support have positive outcomes. Yet, it is important that those outcomes are so excellent and relevant, that they are likely to have a significant impact in the overall economy. In other words, it is the quality of those few outcomes that is important for the economy, and not just the quantity of the outcomes as such.

Secondly, there is something called social rates of return, which includes not only the private rate of return of the individual firms, but also the knowledge and market spillovers of these R&D activities.The significance of these spillovers was formulated in a seminal work by Audretsch and Feldman in the 1990s (Audretsch and Feldman, 1996). Spillovers are the unintended and wider transfer of market and knowledge benefits to the economy and society. The creation of new forms of knowledge, new skills and competences (know-how), new and improved products that benefit the consumers, etc, have broad effects into the economy, well beyond the individual firm.

The broad perspective measures the effects of these public programs in terms of the social rates of return of the R&D activities, which includes not only the private rate of returns, but also these wider considerations too. Admittedly, measuring these broad effects is not easy, at it requires evaluators with a solid expertise in the technologies and knowledge-area in question, and it requires as well longer time frames for the evaluation. However, this broader effect perspective is very important. We must avoid simplistic and too short-term oriented conclusions about the effects of public programs, written by generalist evaluators without real insights into the scientific and technical fields in question.

References:
Audretsch DB and Feldman MP. (1996) R&D Spillovers and the Geography of Innovation and Production. The American Economic Review 86: 630-640.

Scherer FM. and Harhoff D. (2000) Technology policy for a world of skew-distributed outcomes. Research Policy 29: 559-566.