Performance Measurement and Evaluation of Public-Private Partnerships: A Systematic Literature Review

Model

SLR is a specific methodology, to locate studies already carried out in certain areas of knowledge, select and evaluate contributions, analyze and synthesize data resulting in a report that makes it possible to understand what is known and what is not about a certain topic or area of knowledge [8]. Performing an SLR in the MOS knowledge area is particularly challenging given its fragmented nature [16]. “The complicated state of MOS research makes it tough to know what we know, especially as specialization spawns research communities that often don’t and sometimes can’t talk with each other” [17], which leads the Investigators to take positions related to management practices and organizational decisions, for which there is no evidence aggregation or at least some of the evidence are under study [17].

For the elaboration of the present SLR, the model proposed by Denyer D, et al. [8] was followed, which contemplates five essential steps for the SLR elaboration, within the scope of business sciences, particularly MOS. The first step comprises the research question formulation, the second the location of the studies to include, the third the selection and evaluation of the selected studies, the fourth step the analysis and synthesis of the selected information, and finally, description of the results obtained in the previous steps. Figure 1 encodes the summary of the SLR process used in this article.

Regarding the approach adopted for aggregating and synthesizing information, we decided to group our sample of articles by categories (clusters). Cluster analysis is a technique for grouping multivariable that has the main objective of grouping objects based on their characteristics. There are numerous terminologies for the analysis of clusters, used in different areas of knowledge (psychology, biology, sociology, management and engineering), including Q Analysis, Typology Construction, Classification Analysis, and Numerical Taxonomy [18]. Taking into consideration the scope under investigation, an analytical classification of each group of variables was performed, which corresponds to individual contributions to the research question. While addressing the main topic, it becomes clear that two main clusters must be defined: measurement frameworks and evaluation systems.

Click to enlarge

Source: Adapted from Denyer D, et al. [8]. Figure 1: SLR process summary.

Articles Identification

By defining the research question, the criteria for including primary studies in the SLR become clear. The question guides the review, defining which studies to include, which research strategy to identify and the relevant primary studies to be extracted from each investigation [19].

Thus, the present article tries to answer the following research question: “How to measure and evaluate performance in PPP model?” The aim is to identify both conceptual studies in the PPP model performance evaluation and as well, empirical studies where the model performance was, in fact, measured. Bearing in mind the research question, it was necessary to locate, select and evaluate scientific articles that allow addressing the research question. The article search was carried out in March 2022. 64 articles were found and analyzed, published or available in the electronic platform “Web of Science” database, between 1998 and 2021, divided by 14 reference journals. The list of articles analyzed represents the catalog of studies, considered most relevant for the area of “Performance Measurement and Evaluation”, adopting a managerial lens concerning PPP.

The results obtained took into account the application of research chains, grouping keywords and applying research conventions in the selected database WoS (Web of Science - The world’s largest publisher-neutral citation index and research intelligence platform). By adopting and combining Boolean operators with complex searches, a final string was built: Topic: (Performance) AND Topic: (Measurement) AND Topic: (public-private), the result being combined with a filter: Document Types = (ARTICLE) AND Languages = (ENGLISH). The terms searched as “topics” aimed to ensure that the results extracted from the WoS were predominantly relevant to this study and contribute to answering the research question.

Because of the research string applied, we obtained a comprehensive list of articles (primary contributions), which helped us to address the research question previously formulated. The complete articles were downloaded from the WoS database, whenever available. The articles that are not available for a full consultation, we recorded their abstract, in order to assess their pertinence to be part of the selectable articles.

Articles Selection and Evaluation

We gathered and stored the 64 articles found, using the Mendeley citation software tool, which allowed the identification and ordering of the articles. Subsequently, we proceeded to read and analyze the abstracts for the set of articles, to guarantee their relevance to the objectives of our study Denyer D, et al. [8]. propose four alternative principles to be applied when preparing an SLR, in the MOS studies area. Adopting the principles proposed by the author, the articles initially identified (64 articles) were tested, firstly, for their nature, regarding their transparency, inclusiveness and then concerning to the explanation provided and heuristic, regarding transparency, and taking into account the selected database (“Web of Science”), all 64 articles were initially inclusive in terms of transparency. The exclusion criteria were limited to the explanation provided by the article, and whether it contributes to meet the objective of the investigation, resulting in a final selection of 35 articles identified in Table 1.

Source: Own elaboration. Table 1: Final selected articles.

Evaluation Systems & Frameworks

Liu HJ, et al. in their research, call into question the conventional model of performance evaluation based on three dimensions (cost, time and quality), commonly called Iron Triangle. These three dimensions are evaluated ex post, that is, at the end of the project execution process. They concluded, through an interview method, that the application of a “PMS - Process Management Life-Cycle” model accurately reflects the parties’ intentions in evaluating the expected (exante) performance of PPP projects [28].

The proposed model accommodates the typical nuances of a dynamic business environment, incorporating performance measures, as well as a learning mechanism throughout the operation’s life cycle. The same investigation also concludes, empirically, that performance is normally measured during the construction and operation phases, using the metrics time, cost and quality. The authors suggest that the adoption of a measurement approach based on processes and oriented to the parties will be more suitable to evaluate the model´s performance. In the same identified the current scarcity of evaluation models, conceptualizing a model called Performance Measurement Framework that reveals itself, theoretically, as a reliable and robust model which can be used to guarantee a successful execution of PPP

projects, throughout each phase of its life cycle [31].

The proposed model provides governments and private entities with a tool that allows them to measure and control the performance of their future projects. Still, regarding evaluation models [48], were the first to propose a framework based on six dimensions: cost and benefit, social and political impacts, skills, collaboration, and performance measurement. Their framework had, as its main objective the assisting of public managers in determining accountability regarding the life cycle of the PPP project (ex post). For the author, the strategic thinking, on the manager’s side about the net gains for the public, obtained through the offer of the model, and the careful consideration of the six dimensions of accountability should result in a management structure. Social and political risks, costs and benefits, impact measurement, knowledge, collaboration, and performance must be monitored continuously throughout the life cycle of the partnership, thus leading to a PPP management structure that reflects a culture of public responsibility.

Grossman SA, et al. presents a performance measurement model, to accurately evaluate the PPP model, in a complete and comprehensive way, evaluating the metrics Quality of Life, Return on Investment and Management Capacity. For the author, a pragmatic performance evaluation, complete knowledge of the PPP nature, a balanced score of the referred metrics and growth capacity to develop the partnership, is not only necessary to effectively evaluate the performance, but also addresses the true nature of PPP [42]. For Saeed AM, et al. [29] there is no consistent approach that allows the performance of PPPs to be evaluated, presenting a framework (ex-post), for performance measurement developed based on criteria pointed out by the “NSW Auditor-General”, to develop a better performance in the evaluation and auditing practices, being, however, a framework limited to PPP projects for education. The presented framework aims to increase the taxpayers’ responsibility for the content and use of PPPs by the public partner. In the same line of framework proposals [30], set out to answer the question “how do performance measurement systems vary according to different PPP project configurations?”, proposing a performance evaluation structure for PPP projects based on services, based on four dimensions (financial, investment, process and result) and also a set of KPIs, applied to each dimension, to measure the partnerships success.

Several authors have carried out case studies, to assess the real performance of PPP models in progress or already completed [37]. Used three different perspectives to measure the success of four PPP projects in the area of transport:

project management perspective, stakeholder perspective, and contractual management perspective, combining them in a holistic way and obtaining contradictory results in what concerns the final result. The authors opted for a qualitative comparative analysis approach, proposing, in the end, a simple methodology to measure the success of PPP projects in the transport sector.

Gordon C, et al. [38] compared the contract signed by the Australian government, called Sydney Metro Contract, regarding the negotiated performance structure with the current contracts in force for public bus, train and boat transport in Sydney, concluding a slowdown in the performance results in requirements contractually established for the “Sydney Metro” project [26]. presented a case study where they describe a model for transferring assets at the end of the PPP contract. The authors decided to approach the water distribution sector in China, starting their investigation with a literature review in order to assess the status quo of asset transfer regime management in China. The authors present the results hierarchically, using the language method IDEF0, concluding that the model adopted for the water sector is deficient in several aspects, proposing a dynamic management model for the transfer of assets at the end of PPP projects. With the proposed model, it is intended to instruct public and private managers about mechanisms that aim to smooth the phase of transferring assets to other PPP projects that end their concession period.

Tijssen RJW, et al. [44] presents a series of empirical studies, where they investigated the performance of several PPP projects, identifying metrics applied in several PPC (Public-Private Co-publications) articles and performance indicators identified by several reference authors. The conclusions of the study point out that the applications of strategic and innovation metrics must take into account the different propensities of the PPCs of the research fields and related industrial sectors.

Performance Measurement (KPIs)

The present SLR makes it clear that, currently, academics disagree about the indicators used to measure and evaluate the performance of PPP models [23, 24, 34]. argue that the traditional KPI’s used: cost, time and quality, are not comprehensive enough to measure the performance of the PPP model.

The investigation was based on a PPP project developed in Australia, using a semi-structured interview methodology, concluding the existence of a great propensity for the use of TCQ (Time, Cost, and Quality) indicators during the design and construction phases of the projects. However, there was a broad consensus that the traditional approach to TCQ indicators is too simplistic in terms of capturing the inherent complexities of social infrastructure PPPs. The results of the investigation indicate that the measurement of the life cycle performance of PPP projects based on processes strategically translates VfM, and that it should be introduced to replace traditional ex ante and ex post evaluations [24]. Go further, and propose a new five-dimensional model, to evaluate the PPP model more effectively; carrying out a questionnaire- based investigation of PPP managers in China, concluding that the proposed model reflects the real performance of the project during its life cycle.

In the same Liu J, et al. [34], indicate VfM as one of the most relevant indicators to measure the success of the PPP model throughout its life cycle. Previously Love PED, et al. [35], concluded that a more comprehensive approach (project life cycle), will be necessary to “prove the future” performance of the model and ensure the delivery of VfM to the public sector, proposing the adoption of a Building Information Modelling (BIM) model for this purpose. For Jackson A, et al. [51], the ten principles of good performance measurement practices: conceptualization, stakeholder approach, clear balance, ownership, usability, precision, contextualization, dynamism and VfM, are not easily achievable and require adjustments that are difficult to implement in performance measurement systems, especially when trying to apply them to PPP contracting models.

Yuan J, et al. [45], proposes a conceptual model composed of five performance measures and 48 indicators, which give indications that the PPP model is highly influenced by the public sector’s hiring, design and planning actions, concluding the need for convergence between the interests of the public sector and private partner [36]. identified, as a critical factor for the successful implementation of PPP projects, an effective measurement of project performance. The authors carried out a comprehensive review of the normative literature associated with PPPs (for example, critical success factors, public sector roles, selection of dealerships, risk management, cost and time and finance issues) and propose a theoretical framework for the life cycle dynamics. This structure provides the public sector and private entities with an insight into effective and efficient performance measurement in the context of PPP infrastructure projects.

Yong Kim S, et al. [20] argues that ineffective performance evaluation is one of the leading causes of failure for PPP projects. The authors aimed to analyze the importance of performance measurement for the success of PPP infrastructure projects using five criteria:

strategies, processes, capabilities, stakeholder contribution, and stakeholder satisfaction. The results showed that the performance measurement in the partnership phase was the most important component in the success of PPP infrastructure projects.

Conceptual - Theoretical Analysis

In a more theoretical approach [47], carried out a study where they concluded that the influence, in complex PPP contracts, of the parties and the transaction environment were not properly recognized in the literature [33]. showed that the sanctions imposed, contractually, negatively influence the performance of the PPP model; however, they also demonstrated that the characteristics of the clause are the key factor for a positive performance of this type of models.

Sundararajan SK, et al. [32] addressed the question of uncertainty in this type of contracting, studying a dynamic capital structure that can add significant value to PPP projects, when the uncertainty variable is a problem. They concluded, through numerical tests, that the flexibility of the capital structure can mitigate the risk of deficit in the final result of the project.

Tang L, et al. [21] argued that it is necessary to comprehensively consider the project performance under different states to conduct scientific performance measurement. The author emphasis the research on the process of construction, operation, and handover of Civil Air Defense PPP projects. It was concluded that government plays an important role in Civil Air Defense PPP projects, when resources are limited and that the government should give priority to measures such as reducing tax rates and increasing subsidies to ensure project benefits.

Wang D, et al. [22] examined the evolution of stakeholder influence and the trans-period effect (TPE) of process performance of PPPs. The authors conclude that the performance in the five phases (e.g. initiation and planning, procurement, construction, operation and transition) of PPPs exhibits significant trans-period effect. The stakeholder network varies in different phases. The most influential stakeholder is a public authority, followed by a public initiator and a private consortium.

Following the model of representation used by Morais F, et al. [56], we chose to summarize the different areas of research and mutual interrelations, illustrated in (Figure 3) [57, 58, 59, 60, 61, 62, 63, 64].

Click to enlarge

Source: Own elaboration. Figure 3: PPP performance measurement and evaluation critical factors.

Evaluation Systems & Frameworks

In order to gather consensus and gauge a model or framework, as the most correct, to evaluate and measure PPP’s performance, it is crucial to invest in a segmented investigation, which empirically studies the various models when applied to PPP projects of an identical nature. It would be important to assess whether a given model or framework can be suitable when applied, for example, only to PPP projects implemented under a DBO (design, build, operate) regime in the road infrastructure scope or to a PPP projects implemented under the BOT (build, operate, transfer) regime in the health infrastructure scope. In other words, it is translated to the matrix elaboration where, on the one hand, the PPP model is drilldown into its various typologies, and on the other, it is divided into procurement scope (Infrastructure, Services, Healthcare etc.), applying, for each alignment, the various models/frameworks proposed in the literature. The proposed matrix could assist towards a validation, if in fact it is possible to associate a given measurement and evaluation model, with a specific PPP typology vs. scope, in a consistent and transversal way.

Performance Measurement (KPIs)

It becomes clear that, among academics, VfM is the most reliable indicator to measure PPP performance. Since VfM can be predicted before the PPP project execution (exante) and also confirmed after the conclusion (ex post), it is crucial to devote more efforts to empirical studies to conclude if in fact VfM is delivered according to the initial predictions. And also to conclude to each kind of PPP project this indicator can be sustainably applied. Cost indicator is also referred as a critical factor to determine the performance measurement, but it is not clear how the indicator can be applied and delivered when PPP projects are conducted in countries facing budgets constraints, resulting from central decisions (ex. European Central Bank) or temporary financial crises and at the same time must address the infrastructure gaps (recent data released by Global Infrastructure Hub, shows that $97 trillion is required to meet the world’s total infrastructure needs by 2040). It is important to empirically test if the cost indicator can or not be disregarded in situations when the private sector is the only option for governments.

Conceptual - Theoretical Analysis

The contractual wording and terms are often pointed out as a positive influence on the PPP outcome, but still, no empirical studies are indicating that the several worldwide PPP contracts are following a standard legal format in order to accommodate all the nuances present in this type of procurement and to enhance the performance of this kind of projects. Before the empirical studies, it is important to elaborate a conceptual analysis in order to propose a contractual framework that will be suitable to all PPP projects, despite their nature and objective, that will mitigate this risk of getting a negative influence of applying penalties resulting from non-compliances of the signed contracts and thus increasing the possibilities of a positive performance.