Assessment drawing
The methodology developed for the ex post evaluation
uses an integrated approach, quantitative and qualitative based, through the
application of the econometric evaluation of an ex post cost-benefit analysis
(CBA) on measurable effects, and the impact assessment based on theory, through
the reconstruction of causation chain on unmeasurable effects; the evaluation
takes into account the complexity of the large investment project and the
medium-long-term time horizon that characterizes its exercise and effects in
the context.
In particular, the Cost-Benefit analysis applies an
appropriate analytical tool for impact assessment because it can provide
quantification and monetization of some of the short and long-term effects
produced by the project, at the same time providing a framework to identify the
crucial aspects of its results and performance. The evaluation based on theory
is instead characterized by the observation of each phase of implementation of
the logic of intervention, aimed at identifying causal relationships and
mechanisms of change, for the identification of the how and why the
intervention works, focusing on the understanding of the determinants and
causation chains of the process leading to the appreciating of the difficult
monetization effects. It is placed within the framework of the Theory of Change
which concerns the opening of the so-called “black box” in the search for an
understanding of the generative mechanisms of change and the reconstruction of
the underlying theory by the evaluator “Weiss (1995)”. These approaches fit
into the framework of generative causation “Stern et al. (2012)” [1], which
bases inference on mechanisms that explain the effects, and identifies the
application approaches in theory-based evaluation (TBE) and realistic evaluation
“Pawson and Tilley (1997)” [2,3].
The tools and the main qualitative techniques of
investigation, used in application of the impact assessment based on theory,
consist in conducting interviews with stakeholders, focus groups, surveys on
samples of commuters, research on institutional authorities and service
providers archives, search for press articles, suitably combined and
interpreted according to the type of contribution to be made to the evaluation
process. The questions that guide the evaluation process are articulated below:
What change can be
observed as a result of project implementation?
To what extent can the
changes observed be attributed to the project?
Are there any unexpected
effects?
What mechanisms explain
the impacts? What are the key factors of the causal chain that explain the
mechanisms?
What kind of short and
long-term effects can be identified?
What is the minimum time
needed for a long-term effect to manifest and stabilize?
The purpose of the ex post evaluation is to assess the
degree of effectiveness, efficiency and relevance of the results achieved
compared to those expected, using all the available resources, taking into
account also the unexpected and unplanned effects, to understand the internal
and external dynamics of the program [4].
The evaluation analysis develops substantially through
the following three dimensions:
The
"WHAT": this dimension refers to the types of long-term effects that
can be observed following the implementation of the project. The classification
of effects can be represented by the following categories:
- economic growth;
- quality of life and
well-being;
- environmental
sustainability;
- distributive impacts.
The "WHEN":
this dimension refers to the point in the life cycle of the project where the
effects materialise for the first time (short-term horizon) and stabilise
(long-term effects). It discusses the moment of the post-intervention time
cycle in which the evaluation is carried out and the role it can play in
relation to the implementation of the project.
The "HOW": this
dimension implies the development of a reasoning on the factors, both internal
and external to the project, that have determined the observed causal chain of
effects and have influenced the performance of the observed project. For this
purpose it is possible to identify six stylised determinants of project
results: relation to context; selection process; project design; forecasting
capacity; project governance; managerial capacity. The interrelations of these
determinants and their influence on the effects of the project is crucial to
understand the final performance.
Methodological model and
evaluation approach
The methodology adopted consists of four main phases:
- Mapping
of the project effects;
- Measurement
of the project effects;
- Understanding
the project effects;
- Final
assessment and conclusions.
The first step is mapping the effects. In principle,
transport infrastructures can achieve a variety of short- and long-term
effects, which must be appropriately identified together with the investigation
of temporal dynamics and causal chains. This phase responds to the dimension of
analysis of the "what" and also observes the "when", the
temporal dimension of the effects.
The second phase aims to measure the relevant effects.
This is done through the combination of econometric analysis (quantitative)
supported by retrospective CBA and qualitative assessment. In particular,
effects that are not quantifiable through monetary assessments will be
qualified and evaluated through qualitative analyses resulting from the
application of the specific methodologies adopted.
The third phase concerns the understanding of the
effects or better the dimension of the "how", aiming to reconstruct
the logical framework of the intervention by identifying the elements
responsible for the causal chain of the effects detected. The final evaluation
integrates qualitative and quantitative evidence to focuses the most relevant
effects derived from the implementation of the project, highlighting the
project impacts. The figure (Figure 4) represents the schematic conceptual
framework of the adopted methodology.
The description of the four methodological phases is
shown below:
- Mapping
of the project effects
The
first phase, concerning the mapping of long-term effects, identifies the
potential impacts on social welfare by public investment projects on transport
systems, and, for the project being evaluated, the following have been selected
[5]:
Effects
on economic growth:
the first category of effects concerns
economic growth, which is declined through productive effects. In the specific
case of the transport project, most of the effects are related to the
reductions in production costs, the increase in accessibility and attractiveness
of the territory in which the transport project is implemented. There are
several groups of actors on which the economic effects can impact (users,
producers, managers/promoters of the infrastructure project). Direct indicators
that can measure effects on economic growth are:
- travel
time;
- vehicle
operating costs;
- reliability
of travel time;
- revenue
of the service producer (surplus of the producer).
Effects on
quality of life and social well-being: in
line with the theories of “Dasgupta (2001)” and “Stiglitz et al. (2009)”, the
concepts of quality of life and social well-being refer to factors that affect
social development and satisfaction as well as the perception of users and
society in general [6,7]. The main direct effects on the quality of life, in
this project case, are:
quality
of service (speeding up, comfortable seats, wifi, etc.);
safety
prevention (safety);
security
protection (security);
noise.
Effects
on environmental sustainability:
the concept of environmental
sustainability refers to the possibility of ensuring the needs of the current
generation without compromising the environmental conditions of future
generations. Transport infrastructures can influence environmental
sustainability in two opposite ways: on the one hand, the expansion of
transport infrastructure is negatively linked to environmental sustainability
as it could generate greater environmental pollution and reduction of natural
resources, and, on the other, more efficient and technologically modernised
solutions, provided by improved infrastructure, can help to achieve
environmental pollution reduction targets. The main effects of the transport
project are related to the contribution to the mitigation of the following
factors:
- climate
change (GHG greenhouse gas emissions);
- air
pollution.
Effects
on distributional impacts:
distributive
impacts can relate to social cohesion and territorial cohesion. The lowering
transport costs can mitigate social inequality of the weakest social groups,
while the improving of accessibility can inhance the territorial cohesion of
the peripheral areas, contributing to the bridging of territorial accessibility
gaps.
Measurement of the project effects
With regard to the
measurement of effects, the cost benefit analysis (CBA) is the methodological
approach adopted, most desirable from the point of view of “quantitative”
analysis, for the following reasons:
is the most desirable
quantitative method to investigate the detail elements required to isolate the
impact of a single project;
is a reliable tool for
presenting project benefits and externalities in monetary terms;
measures the impacts in
terms of welfare changes, the CBA being based on the welfare economy. This
makes it possible not only to rank projects, but also to formulate conclusions
about their social desirability.
The methodological approach is completed with the
qualitative evaluation of those effects that cannot be measured with the CBA,
enriching the evaluative research with the consideration of the variety of
long-term productive effects of the contribution to the impact on the social
welfare.
The methodology used for the application of the CBA is
set out in the Community Guide adopted in respect of investment projects
financed by the European Cohesion Policies , assuming that the projects
selected for ex post evaluations have been functional for at least five years,
with the implications of reducing any bias by an optimistic approach, given the
assessability of actual observed data, without making a comparison between the
two CBA ex ante and ex post, rather analyzing the contributions of the project
with respect to medium-long term impacts.
General
principles in methodology application
The ex post evaluation is carried out five years after
the completion of the works of renewing and modernizing rail infrastructure, to
assess its medium-long-term impacts, therefore, adding the average period of
completion of the work to the first five years of operation, will be quantified
analytically the historical cash flows that the project has accumulated,
including the actual project implementation and operating costs, and revenues
recorded during the first five years of operation. In the remaining years,
included in the time window following the current year, within the time horizon
(30 years), instead, operating cash flows forecast data will be used relative
to the management phase already started, that is the costs of the service, the
costs of the infrastructural management in relation to usury, the railway
service revenues. The approach used for the ex post CBA is that of the
differential analysis between the design scenario and the non-intervention
scenario and other possible alternative design scenarios; in particular the
non-intervention scenario is the counterfactual scenario without the
intervention on the railway line, where the cash flows of the railway
management are considered retrospectively and prospectively, which, in addition
to tariff revenues, include maintenance costs, functional renewal (Business As
Usual), or even planned investments in extraordinary maintenance and
operational improvement, for the functional operation of the service (Scenario
Do minimum). The counterfactual scenario is characterised by significant costs
to be incurred in terms of extraordinary maintenance and hydrogeological risk
mitigation, higher than those that occur in the intervention scenario
characterized by improvements in infrastructure and increased safety and
resilience to environmental risks.
The implementation of the railway requalification
project involves the reduction of operating costs compared to the
non-intervention scenario, both for the infrastructure operating costs and for
the service management costs, also the number of the human resources employed
is reduced for effect of the increase of the service level of renewed
infrastructure; the modernization of the transport system also allows a
significant reduction of costs of extraordinary maintenance. Based on these
circumstances, the savings obtained in the differential operating costs
translate into an increase in differential revenues in the calculation of the
financial and economic performance of the project. The increase of the tariff revenues
derives also from the forecasted increment of railway transport demand
consequent to the upgrading intervention.
The main benefits generated by the implementation of
the project are the following:
- Travel
time savings;
- Savings
on the operating costs of traffic diverted by road;
- Savings
on the operating costs of the railway service;
- Savings
on external transport costs: noise, air pollution, greenhouse gases, safety.
If the sum of the positive and negative components
results in a favourable final balance, measured in the amount of avoided
externalities in millions of euro/year, this translates into net benefits for
the community. The process of discounting cost and benefit items leads to the
definition of the economic net present value (ENPV), the economic internal rate
of return (EIRR) and the benefit/cost ratio. The results show that the
intervention scenario is characterised by a positive socio-economic feasibility
assessment if the economic indices are characterised by the positive value of
ENPV, the value of the TRIE higher than the social discount rate used for the
prospective analysis, and the benefit/cost ratio higher than one.
The objectives pursued by the qualitative analysis are
identified in the following:
describe the project with
a critical focus on its identification;
- analyze the
socio-economic context;
- reconstruct the
decision-making process;
- evaluate possible
alternative options;
gather evidence on the
non-quantifiable effects and factors influencing design performance.
The following qualitative evaluation focuses on the
effects produced by the realization of the railway project, not measured by the
cost-benefit analysis, and any unexpected effects. The approach, unlike
quantitative methods, produces an explanatory narrative of how things should
logically function to produce the desired change. A precondition to the
realization of the process of reconstruction of the theory of change is that
the evaluator works collaboratively and as a facilitator, among a wide range of
stakeholders, in a typically constructivist approach. The evaluator, once the
reconstructed “theory of change” ensures that the intervention logic is
acceptable to stakeholders, appropriates the map of the program resulting from
the stakeholder involvement process and, using the most appropriate data
collection techniques, monitors and analyses the progress of the intervention
programme and the deployment of its short-medium term effects, under the
integration of the evaluation evidence.
Understanding
of the project effects
Once the effects of the project have been identified
and measured, and the causation chain of the different categories of short and
long-term effects has been investigated, the next step of the methodological
approach involves the understanding of the elements and mechanisms, both
internal and external to the project, that have determined the succession of
the causes and effects determining the performance of the project. The
interactions of these determinants may reinforce or mitigate one effect over
another. In addition, each determinant can contribute, whether positively or
negatively, to generating, accelerating or curbing certain short or long-term
effects (Table 1).