study to develop an upgraded single market scoreboard as a … · 2019-09-18 · centre for...
TRANSCRIPT
Centre for Strategy & Evaluation Services LLP
Westering House 17 Coombe Road
Otford, Kent TN14 5RJ United Kingdom
E: [email protected] T: +44 (0) 1959 525122
Study to Develop an Upgraded Single Market Scoreboard as a Governance Tool for the Single Market
An Inception Report to the Directorate-General for Internal Market, Industry, Entrepreneurship and SMEs of the European Commission
765/PP/GRO/IMA/19
Implementing Framework Contract 575/PP/2016/FC
18 September, 2019
i
Contents
1 Introduction 1
1.1 Structure of the inception report ................................................................................... 1
1.2 Study objectives and scope ............................................................................................ 1
1.3 Study background and Issues ......................................................................................... 2
1.3.1 Tracking progress in Single Market implementation – the Single Market Scoreboard ..... 2
1.4 Progress update – Phase 1 ............................................................................................. 4
1.4.1 Literature, sources and stakeholder review (task 1) ......................................................... 4
2 Methodology – Phases 2 and 3 7
2.1 Overview of methodology ............................................................................................. 7
2.2 Phase 2 - Data identification, collection and mapping .................................................... 8
2.2.1 Task 2. Mapping of existing tools for monitoring Single Market policies .......................... 8
2.2.2 Task 3. Identification of available data from different sources ......................................... 8
2.2.3 Task 3.2 Stakeholder consultations ................................................................................... 9
2.2.4 Timing of consultations .................................................................................................... 13
2.2.5 Data quality ...................................................................................................................... 14
2.3 Phase 3 - Data analysis ................................................................................................. 14
2.3.1 Task 4 Identification of gaps ............................................................................................ 15
2.3.2 Analysis of the stakeholder consultation feedback ......................................................... 15
2.3.3 Task 5. Definition of new indicators including end user’s perspective indicators ........... 16
2.3.4 Task 6. Recommendations on indicators ......................................................................... 18
2.3.5 Task 7. Presentation of upgraded scoreboard ................................................................. 18
3 Project Management 19
3.1 Detailed workplan and timeline .................................................................................... 19
3.2 Reporting and meetings ............................................................................................... 19
3.3 Assumptions................................................................................................................. 21
4 Next steps 22
4.1 Finalisation of Phase 1 .................................................................................................. 22
4.2 Launch phase 2 ............................................................................................................ 22
Annex 1 - Bibliography 23
ii
Tables Table 2.1: Stakeholder groups for the targeted online survey ................................................................... 11 Table 2.2: Interview programme structure ................................................................................................. 12
Figures Figure 2-1: Methodology and workplan ........................................................................................................ 7 Figure 3-1: Detailed workplan – SMS 2.0 .................................................................................................... 20
1. Introduction
1
1 Introduction
This document sets out the (draft) inception report for the “Study to develop an upgraded Single Market Scoreboard as a governance tool for the Single Market”. The study was contracted through Framework Contract 575/PP/2016/FC for DG GROW and is being led by CSES with support from Idea Consult, Oxford Research and Trilateral Research.
1.1 Structure of the inception report
The inception report sets out the detailed work plan and methodological approach to be adopted. Section 1 sets out the study objectives, scope and background. It presents progress made since the kick-off meeting and takes account of the findings from the preliminary desk research and review of data/information sources. Section 2 describes the methodology that will be used for the remainder of the study. Section 3 on project management provides the detailed project plan, reporting-related tasks and assumptions. Section 4 outlines the next steps.
The main report (maximum 20 pages) is complemented with more detailed and practical aspects which are annexed. The annexes include:
• A bibliography (Annex 1);
• Summary of Kick-off meeting (Annex 2)
• The contacts database (Annex 3). This will be updated on an ongoing basis during the study.
1.2 Study objectives and scope
The overall study objective is to provide advice and input in order to upgrade the Single Market Scoreboard to better monitor the state of the Single Market in its four dimensions: free movement of goods, services, people and capital.
The specific objectives are to:
• assess the existing Single Market Scoreboard and its accompanying indicators;
• map these against policy needs;
• identify new and relevant indicators – covering legal and economic issues and taking stock of the situation for users; and
• recommend how to improve the user-experience within an upgraded online scoreboard.
The study also needs to incorporate the international dimension of the Single Market, by identifying possible indicators for benchmarking/comparing market integration within the EU and within its major trading partners, including what it delivers for citizens and businesses.
In terms of identifying indicators, the ToR stresses that the study will need to look at:
• the causality and significance of Single Market policies and their implementation by Member States in relation to the identified indicators;
• data quality and stability, including how data could be collected periodically and interpreted, possible links with other relevant existing scoreboards at EU level (e.g. the European Innovation Scoreboard); considering how “real time” data could be collected and integrated into data analytics and data visualisation over the longer term;
• data analytics and data mining, to help identify barriers within the Single Market across sectors,
1. Introduction
2
by checking different sources of information within EU institutions and beyond and;
• data visualisation, to improve the Scoreboard’s user-friendliness and presentation of the most meaningful data.
1.3 Study background and Issues
The Single Market, established in 1993, supports goods, services, people and capital to move more freely between 32 European Union and EFTA countries. The Commission identifies the Single Market as one of the greatest achievements of the European project, essential in increasing the success and prosperity of European Union (“EU”) citizens and businesses.
Through its Single Market Strategy, the Commission is continuing to develop the Single Market and its performance, in particular through measures that aim to:1
• Enable the balanced development of the collaborative economy
• Help SMEs and start-ups to grow
• Improve the opportunities for businesses and professionals to move across borders
• Address restrictions in the retail sector
• Prevent discrimination against consumers based on nationality or place of residence
• Modernise our standards system
• Create more transparent, efficient and accountable public procurement
• Consolidate Europe’s intellectual property framework
• Ensure a culture of compliance and smart enforcement to help deliver a true Single Market
The Single Market Strategy complements other key Commission initiatives such as the Investment Plan for Europe, the Capital Markets Union (“CMU”), the Digital Single Market (“DSM”), the Energy Union and labour mobility.
1.3.1 Tracking progress in Single Market implementation – the Single Market Scoreboard
The Single Market Scoreboard (“SMS”) is one of the mechanisms which tracks progress in the implementation, enforcement and performance of the EU Member States (and sometimes EEA) vis-à-vis the Single Market.
Introduced in 1997, its purpose and scope have evolved over time, such that it now aims at giving an overview of the practical management of the Single Market. The Scoreboard organises data, information and analysis by: stage in the governance cycle; Member State performance; governance tool; policy area – for a small sub-set of policy areas; state of trade integration and market openness.
The SMS contains a range of indicators and analysis, updated and published annually. The data is principally based on a combination of official statistics, online (public) data portals and studies. As a monitoring tool, the aim is that the indicators and analysis provide the evidence-base, trends and comparison as a basis for action, allowing for better implementation and enforcement of Single Market rules.
Among the ways in which the SMS can be upgraded include:
• Development of additional indicators that provide greater utility for assessment of policy performance and subsequent action, including underscoring the causality and significance of
1 https://ec.europa.eu/growth/single-market/strategy_en#cross_border
1. Introduction
3
single market policies;
• Broadening the coverage of policy areas, sectors and help avoid emerging barriers in strategic sectors (for example digital, artificial intelligence or the circular economy) or where there is further potential for economic integration such as services, taxation or network industries;
• Extending the set of indicators and analysis to better reflect the quality of transposition of directives;
• Complementing the existing regulatory/input-oriented indicators with more perception-based indicators to have a broad and coherent picture of how the Single Market works on the ground for citizens and business, including to cover the social dimension and issues related to trust;
• Making the scoreboard easier to use, more accessible to existing audiences, and more appealing to new audiences – drawing on best practice in scoreboard development and presentation; and
• Investigating the degree to which selected indicators can be compared with Europe’s main competitors.
Taking an example as to the debate on how data visualisation might be improved, during the kick-off meeting (KoM), the extent to which the existing ‘traffic light’ colour-coded system for reporting on data remains fit for purpose was discussed. This was introduced to report on the transposition deficit, and on the percentage of Directives where national transposition led to subsequent infringement proceedings in the early stages of the scoreboard’s development, and was subsequently rolled out across other areas. However, its ongoing pertinence may be questionable, given that it does not necessarily fit with some of the other governance tools and policy areas in the SMS.
The possibility of extending the scoreboard to include new areas such that it could feed into the European Semester reporting mechanism was also raised. The importance of reviewing the SMS to consider the micro-economic not only the macro-economic dimension was also highlighted.
A further aspect discussed at the KoM was the extent to which good practices can be identified and lessons learned from the development of other scoreboards by the Commission and international organisations (e.g. the OECD, WB). In particular, the success of the (relatively new) Justice Scoreboard, and the relevance of some of its indicators to the SMS was stressed. The rapid acceptance of the Justice Scoreboard and indicators contained therein was given as an example of good practice. Furthermore, the Digital Economy and Society Index (DESI) Scoreboard2, a composite index, was cited as an example of a scoreboard that is attractively presented in an engaging way. A particular good practice is the use of easily understandable ‘headline’ indicators. However, it was also recognised that there are drawbacks of the use of composite indicators, and it was not suggested to go down this avenue, but rather to draw on other elements of good practice from DESI.
Whilst our clear focus will be on reviewing the existing scoreboard and in parallel, identifying possible new indicators, and on strengthening the policy focus of the SMS, the benchmarking of existing scoreboards offers the potential not only to hone in on useful indicators for the SMS, but also alternative means of presenting data and information in a more visually appealing way. Consideration will therefore be given not only to using benchmarking to derive good practices for the website on which data is presented, but also in terms of the scope for standalone PDF scoreboards, with attractive visuals (graphs, charts, Figures) that could help to modernise the presentation of the SMS.
The target audience for the SMS was also discussed. Whilst it was acknowledged that this remains broadly unchanged, and should focus on EU and national policy makers (including politicians), national
2 https://ec.europa.eu/digital-single-market/en/desi - DESI summarises relevant indicators on Europe's digital performance
and tracks the evolution of EU member states in digital competitiveness, across five main dimensions: Connectivity, Human Capital, Use of Internet, Integration of Digital Technology and Digital Public Services.
1. Introduction
4
and regional authorities, journalists and others with a more technical understanding of the different dynamics of the IM, equally there is a need to provide interesting and engaging data and information that helps to facilitate external communications on progress against the four freedoms of the IM using the SMS as a mechanism.
A possibility floated by our study team at the KoM was the need to investigate whether the indicators could be streamlined, with differentiation between a much smaller set of headline indicators (possibly structured differently, for example, around the four freedoms rather than the governance tools, policy areas etc. with a more numerous set of broader operational indicators of managerial use, reporting on the many different single market-related initiatives, networks and cross-border cooperation activities that feed into the single market. Getting the structure right, and considering different possibilities in terms of how data might be presented, will be an essential precursor, before including additional policy areas and indicators to reflect the evolution and modernisation of the IM.
1.4 Progress update – Phase 1
During Phase 1, a number of preparatory tasks have been carried out so as to structure the study assignment effectively. These are briefly summarised in bullet point form below: • A kick-off meeting (KoM) was held on 17 July 2019 with the client and wider Commission services
participating in the Steering Group (SG). A summary of the meeting was prepared by CSES and commented by the Commission. The final version is annexed to this report (Annex 2).
• A draft letter of introduction was prepared by CSES and sent to DG GROW. When signed, this will be used to facilitate the interview programme and wider stakeholder consultations.
• A small number of Commission officials from different DGs in the SG have been contacted, and scoping interviews have taken place.
• A detailed workplan has been developed. While no major changes have been made to the methodology following the kick-off meeting, the sequencing and timing of sub-tasks has been refined to take into account comments from the Commission.
• A template and guidance for the literature review has been created in Excel. The specialised research software Zotero is being used by the study team to support the literature search and review. Further details are provided below.
• An excel database has been created to clearly and systematically store information at various levels of granularity, allowing users to see 1) the ‘big picture’ of existing tools and indicators within the SMS and 2) the detailed information and metadata for each of the indicators.
• A contacts database has been developed which expands on the list of stakeholder groups in our proposal. Some additional names have been suggested by the Commission. The contacts database will be treated by our team as a work in progress, since those interviewed may have useful suggestions as to which other stakeholders from the target audience for the SMS should be added.
1.4.1 Literature, sources and stakeholder review (task 1)
An extensive review of the literature and other sources has been conducted to ground the study in a thorough understanding of the Single Market, its status and challenges concerning integration, the main related EU policies and the different data sets and scoreboards that could be exploited to upgrade the Single Market Scoreboard.
Starting with the publicly available sources outlined in the ToR, our study team has identified further relevant literature, documentation, data and information sources. We have done this using different techniques in parallel, such as Google, Google scholar searches and a snowball citation-chasing approach to check that the very latest literature on the subject has been included.
1. Introduction
5
The sources include EU regulations, directives and strategies, Council conclusions and Commission communications. They also include a wide range of professional and academic research studies linked not just to existing areas of the SMS, but also to policy areas not currently covered. Numerous other scoreboards and official statistics have also been identified. Furthermore, the Commission services have shared a number of data sources at the kick-off meeting and in the inception stage which have been added to the bibliography as appropriate.
The identified sources are detailed in the bibliography in Annex 1.
• The initial findings from the desk research and review of data/ information sources suggest that there are an extensive range of data sources available on different aspects of the existing SMS and on the single market more broadly.
• Regarding the existing SMS, we have mapped the main webpages, indicators and report outputs linked to the 13 governance tools and 4 policy areas identified in the SMS. In addition, we have begun benchmarking other scoreboards against the SMS, and mapping data sources available in new areas that could potentially be covered by the SMS in future.
• A finding was that the type of data varies widely, depending on the specific area concerned. Across some of the governance tools, the information available is relatively managerial in nature, i.e. pertains to the operations of a network and the frequency of the use of particular services, such as the CPC Network and Solvit.
• It was also noticeable that in some areas of the SMS, although there is reporting data available through the SMS, there is much less analytical literature than in the past. For example, the transposition deficit used to attract a lot of attention from researchers, academics and policy makers, and there were conferences, research and working papers, and studies on this topic, for example in the early 2000s, when the deficit was higher, and non-timely implementation of EU legislation was more an issue. Likewise, interest was maintained in the post-EU enlargement period. However, since just after the mid-2000s, there appears to have been a considerable decline in the level of interest and attention in the research and evaluation community and among policy makers in the issue. This may paradoxically be due to the success of SM implementation in terms of reducing the problem of the deficit, and this is an example of how strong performance in a particular area of the SM may lead to a decline in interest.
• The desk research has already pointed to the rapid evolution in the relevance (and/ or non-relevance) of particular indicators. Taking postal services as an example, a key SM performance indicator has for some years been the percentage of items posted that arrive the following day. However, digitalisation has led to a major reduction in conventional letters being sent (e.g. due to instant e-communications), but paradoxically, the rapid growth of e-commerce has led to an increase in the number of packages being sent. Therefore, structural changes in the market have led to changes in the ongoing relevance of the performance indicators selected. At the same time several Member States postal services have questioned whether the frequency of home delivery can be reduced below the minimum service required in the postal services directive e.g. by derogation. This is a good example as to why modernisation is needed to keep pace with the evolution in the different dimensions of the internal market itself.
• From our initial literature search and discussions, it has become apparent that there is an over-focus in terms of relative weighting and number of areas covered on the governance tools, and insufficient attention on reporting on policy areas (and in structuring these according to the 4 freedoms). The Single Market Scoreboard currently identifies relevant legislation as including “measures considered to have an impact on the functioning of the internal market, as defined in Articles 26 and 114(1) of the Treaty on the Functioning of the European Union (TFEU). This includes the four freedoms (freedom of movement of persons, goods, services and capital across borders within the EU), and supporting policies with a direct impact on the Single Market, such as taxation,
1. Introduction
6
employment, culture, social policy, education, public health, energy, consumer protection, transport, environment (except nature protection), information society and media.” 3 The extent and scope of this description is noteworthy. The study team will therefore seek to agree with the Commission which are the main policy areas that it will be important to cover in an upgraded SMS and the extent to which if these areas were reported on, sufficient linkages can be made to delivering on the single market. The stakeholder consultations will also be crucial in this regard and will help to arrive at an SMS 2.0 that is relevant and focused, but also extended beyond the current emphasis on Governance Tools.
• In some areas, the causal links between a given policy area and progress towards single market objectives was unclear, for instance FDI and trade data. This depends on the data sources available and the specific indicators on which data is being reported, where there may not always be a Single Market dimension.
• Regarding scoreboard presentation, this initial phase has underlined the large number and wide variety of different scoreboards that exist. For example, in the JRC draft inventories being prepared, there are close to 70 scoreboards and in the data hub list being prepared by the JRC Competence Centre on Composite Indicators and Scoreboards (COIN), there are more than 500. Rather than reviewing all of these, for the purposes of this study it will be necessary to draw lessons from a sub-set of the most relevant scoreboards including those mentioned at kick-off meeting. Already from these there is a wide variety of different approaches, ranging from the very quantitative and highly interactive scoreboard prepared on the DG MARE Blue economy indicators (which allows users to select indicators and their combinations), to the more static approach adopted but with additional qualitative analysis in a pdf document for the Justice Scoreboard. The Digital Economy and Society Index (DESI) and European Innovation Scoreboard both categorise their shortlisted indicators into a small number of dimensions e.g. connectivity, human capital, and – based on an analysis of performance – the European innovation scoreboard also groups Member States into innovation leaders, moderate innovators, etc. Each approach requires a different investment, in terms of set-up, maintenance and annual update.
Once the literature review has been finalised, an output will be a “conceptual and mapping overview” of the Single Market indicating its objectives, associated policies and corresponding data-sets. This will provide a building block for the subsequent finalisation of a framework, criteria and heuristics tool for assessing the relevance, coherence and effectiveness of the SMS indicators to measure performance in Single Market policy implementation, identify gaps and support the definition of indicators.
Stakeholders identified
Section 2.2 provides a draft plan for the stakeholder consultations, including the stakeholder groups identified. It outlines clearly the scope, objective and activities, methods and suggested stakeholder groups. The timing and sequencing of stakeholder consultations is such that this can help guide the gap analysis for the existing SMS from an end-user viewpoint as well as validate new policy areas and indicator suggestions. Further feedback as to which stakeholders should be targeted through the consultations during either data collection or analysis phases will also be welcomed and integrated.
3 https://ec.europa.eu/internal_market/scoreboard/performance_by_governance_tool/infringements/index_en.htm
2. Methodology – Phases 2 and 3
7
2 Methodology – Phases 2 and 3
This section sets out an abbreviated version of the methodology and explains how the study team will carry out the practical research tasks required in Phases 2 and 3. Accompanying information is provided in annex and in the original Technical Proposal (the latter sets out a detailed research framework for undertaking the study).
2.1 Overview of methodology
The assignment will be carried out in three phases as outlined in Figure 2.1.
Figure 2-1: Methodology and workplan
During the first weeks since the kick-off meeting was held, the focus has been on ensuring that the study is grounded in a thorough understanding of the key study issues through familiarisation with, and a review of key documentation sources. The research activities and progress made during Phase 1 were summarised in Section 1.4 – progress update. The rest of this section therefore focuses on the work to be undertaken during Phases 2 and 3. Once feedback has been received on the inception report and the associated methodology, we will commence the core data collection phase and put in place a robust framework for the data analysis.
Inception Report and meeting (Month 1)
Draft Final Report (Month 5) Final Report (Month 6)
• Task 4 Identification of gaps
• Task 5. Definition of new indicatorsincluding user's perspective indicators
• Task 6. Recommendation of indicators
• Task 7. Presentation of upgraded scoreboard
• D3 - Draft Final Report (Month 5)
• M4 - Draft Final Report meeting
• D4 - Final Report (Month 6)
• D5 - Executive summary and supporting data (D6) (Month 6)
Phase 2
Data identification, collection and mapping
Phase 1
Inception Phase
• M1 - Kick-off meeting
• Task 1 - Literature, sources and stakeholders review
• Develop data collection and analysis tools/framework
• Finalisation of methodological framework & workplan
• D1 - Inception Report (Month 1)
• M2 - Inception Report meeting
Interim Report (Month 3)
Months 0-1 Months 2-3 Months 4-6
• Task 2. Mapping of existing tools for monitoring Single Market policies
• Task 3. Identification of available data from different sources
• Stakeholder consultations
- Targeted consultation
- Interview programme
- Focus Group
• D2 - Interim report, including task 4 (Week 10)
• M3 - interim meeting (presentation of the interim report)
Phase 3
Data analysis
2. Methodology – Phases 2 and 3
8
2.2 Phase 2 - Data identification, collection and mapping
Phase 2 involves data identification collection and mapping, building on the initial assessment of key documentation during Task 1, but taking this further through in-depth analysis and review of existing and possible new data sources. This will then be complemented by stakeholder consultations to accumulate key informant opinions on the use, utility and additional needs from the Single Market Scoreboard in terms of data, presentation and usability. Specifically, our plan includes 1) targeted stakeholder consultations 2) an interview programme with maximum 45 stakeholders and 3) focus group discussions.
Each task is considered in turn.
2.2.1 Task 2. Mapping of existing tools for monitoring Single Market policies
Task 2 will involve mapping the existing tools for monitoring single market policies as desk research, and will build on the desk research already undertaken in Task 1. In particular, we will conduct a detailed review of the identified tools and map these tools, their sources and indicators and other relevant information.
This will help build a comprehensive picture of the strengths and weaknesses of the existing Single Market Scoreboard and facilitate preparation for tasks 3 and 4 on identifying additional sources and performing gap analysis, respectively.
The tools and indicators will be mapped against an overall framework for the Single Market and its different elements and policies as prepared in the framework, criteria and heuristics tool. Looking ahead to the gap analysis, particular attention will also be given to tools used to monitor single market polices but that are not currently integrated into the Single Market Scoreboard, and tools beyond those used for monitoring single market policies. We will also pay specific attention to identifying data sources that have the potential to be integrated into the Scoreboard as real-time data sources in the longer-term.
The information will be clearly and systematically stored in an Excel database.
2.2.2 Task 3. Identification of available data from different sources
2.2.2.1 Task 3.1 Detailed identification, review and analysis of data from different sources
Task 3 involves broadening the research to identify available, relevant data from additional sources, to analyse the quality of data coming from these sources as well as to assess the feasibility of using the data periodically for the purposes of the Single Market Scoreboard.
We will conduct a detailed review of the identified additional sources and, as in Task 2, store this information clearly and systematically in the Excel database.
Different data and information sources will be investigated in this task, including:
• Existing data sources relating to the SMS Scoreboard – data from across the current set of governance and policy indicators.
• European Commission, ECB, EIB and other EU-related – this includes existing scoreboards such as the Digital Economy and Society Index (DESI), the EU Justice Scoreboard, European Innovation Scoreboard, Consumer Markets Scoreboard.
• The OECD – for instance, the Product Markets Regulation database or Government at a Glance
• The World Bank – for example Doing Business
• Commercial datasets – such as micro-level firm data from Orbis for possible consideration of econometrics, the FT’s FDI Markets database.
2. Methodology – Phases 2 and 3
9
The literature search in Annex 1 provides further examples.
The examples of existing data and information sources identified in the original proposal include a combination of perception-based surveys and micro-level firm statistics as well as examples of real-time data. The international dimension (beyond the EU) is also available through some of these sources.
2.2.3 Task 3.2 Stakeholder consultations
In Task 3.2 we will carry out consultations with key stakeholders, keeping in mind the requirements to make the scoreboard a more useful tool for a (broader) professional audience. In particular, the following different types of consultations will be undertaken:
• Task 3.2.1 – Design a detailed implementation plan for a consultation based on an agreed strategy
• Task 3.2.2 – Targeted stakeholder consultations (online surveys)
• Task 3.2.3 – Interview programme
• Task 3.2.4 – Focus group discussions
2.2.3.1 Task 3.2.1. – Design a detailed implementation plan for a consultation based on an agreed strategy
The first step in managing the stakeholder consultations will be to finalise the overall strategy for conducting the consultations. The Tender Specifications suggest that the EC would take the lead in this regard. There is a logic to this, in that unlike many studies for DG GROW, where there is a strong focus on consulting with external stakeholders using a variety of mechanisms (e.g. online questionnaires, interviews), here, many of the interviewees will be from across the Commission services, such as those involved in the data for policy initiative, Units already contributing data to the SMS, and those that may be brought in in future within the context of an expanded scoreboard.
We would therefore welcome any further suggestions on stakeholders to be contacted/ interviewed, building on those already identified in the contacts database. During Phase 1, the client made various useful suggestions, such as providing contact details for EC officials involved in a variety of mapping and new initiatives that could be relevant to the future SMS, as well as for internal market specialists within the permanent representations. Members of the IMAC will be added as and when their contacts become available. This will be an iterative process.
In accordance with the Better Regulation guidelines, we will ensure that the draft detailed implementation plan for the consultation, the draft questionnaires and the selection of stakeholders are drafted and submitted to the Commission in a timely manner. Following any comments, the implementation plan, draft online questionnaires and interview guides will then be finalised and agreed.
Below we outline the draft plan for stakeholder consultations.
Consultation scope and objectives
We propose that stakeholders be consulted for several purposes:
• to collect primary data, where a need is identified for additional information to strengthen the evidence base available from secondary data
• to obtain key informant opinions on the use, utility and additional needs from the Single Market Scoreboard in terms of data, presentation and usability. This will help guide and inform the possible selection of important policies, policy areas, indicators usability and information so as to inform the data analysis phase.
2. Methodology – Phases 2 and 3
10
• to test and validate the analyses from different data sources with a selection of the targeted stakeholders.
Consultation activities, methods and tools
We propose to focus the stakeholder consultation process on collecting data via targeted consultations, a wide-ranging interview programme and selected focus groups to test and validate the analyses.
In accordance with our standard practices, attention will be paid to ensuring an appropriate balance between the different types of stakeholders consulted: • By type of stakeholder – e.g. policy makers, national authorities/administrations, professional
organisations at EU level, representatives of businesses including SMEs and start-ups, consumer organisations, academics/researchers, journalists, thinktanks, organisations representing the interests of citizens;
• By geography to ensure appropriate geographic coverage and a balance of small and large Member States as well as EU level organisations such as Business Europe or the European Consumer Organisation
• By sector / policy area to attain representative coverage of different sectors, including strategic sectors of interest for upgrading the scoreboard
The consultation data collection tools will contain questions directly linked to the needs of the gap analysis so that the results can be easily fed into Task 4.
In designing and structuring the detailed implementation plan for the stakeholder consultation, CSES will take into close consideration the relevant parts of the Commission’s Better Regulation guidelines.
The way in which each of the proposed consultation tools will complement one another, and the way in which particular consultation methods might help to address possible information and evidence gaps will be also be specified in the finalised implementation plan. It should be noted that wherever potential gaps in the availability of information and data are identified (both in relation to particular consultation methods, and overall once the different consultations tools used to gather information have been carried out, we will propose mitigation methods to overcome any gaps. A list of the mitigation measures already identified was provided in the risk assessment in our proposal (Section 5.2). These will be integrated into the detailed implementation plan.
2.2.3.2 Task 3.2.2. – Targeted stakeholder consultations (online surveys)
The next step will be to undertake targeted stakeholder consultations. This will involve in particular the development of questionnaire-based surveys which will be launched once approved using the Commission’s EU Survey tool or, as an alternative, using SurveyMonkey, which could help to mitigate the risk of time delays.
The purpose of the targeted consultations will be to gather opinion-based feedback from different categories of stakeholders concerning their expectations of the Single Market Scoreboard. This includes opinions on the use, utility and additional needs from the Single Market Scoreboard in terms of data, presentation and usability, as well as important policies, policy areas and possible indicators. The consultation data collection tools will contain questions directly linked to the needs of the gap analysis so that the results can be easily fed into Task 4.
Since, as far as we know, there is no existing database of SMS users that we can draw upon as a sampling frame for statistical random sampling purposes, we propose a non-probability sampling method. Additional efforts will be made to ensure that the respondents are qualitatively representative of the stakeholder population by gathering responses from a mixture of Member States and stakeholder organisation types.
2. Methodology – Phases 2 and 3
11
In our proposal, we suggested that the online surveys remain open for 4 weeks. This is the maximum time that the survey could remain open for if preliminary analysis is to be integrated into the interim report. It assumes that both the draft questionnaires and contacts database can be rapidly finalised, noting that little to no buffer exists in the sequencing of tasks.
To facilitate quantitative analysis of the opinion-based survey feedback, we will use mainly closed ended questions. However, respondents will have the opportunity to explain several of their responses in open text boxes, which will allow for more technical feedback to be provided. Additionally, a small number of open questions will be included.
The suggested stakeholder groups are outlined in Table 2.1.
Table 2.1: Stakeholder groups for the targeted online survey
Stakeholder type Stakeholder group
EU/international public administration
• Steering committee members
• DG DIGIT, DG SEC GEN, DG EMPL, DG FISMA, DG ENV, DG TAXUD, DG SANCO …
• Joint Research Centre
• Eurostat
National public administration • Ministries of economy, industry
• National Statistical Offices
National contact points /networks for aspects of the scoreboard
• Contacts for the transposition & infringements chapters;
• Networks such as European Enterprise Network (EEN), EURES, ECC-NET, SOLVIT centres, Product contact points, Product contact points for construction, National assistance centres for professional qualifications, National contact points of cross-border healthcare, ECC-Net, Points of single contact, Online dispute resolution, European Judicial Network in civil and commercial matters (EJN-civil), etc.
Representative EU and national (professional) organisations
• Business associations and SME associations: Business Europe, Euro-chambers, SMEUnited, Eurocommerce
• European Start-up Network
• European Trade Union Confederation (ETUC)
Consumer and civil society organisations
• Consumer organisations e.g. BEUC, consumers international
• Social Platform
• Member organisations of the EESC liaison group
Academics/ researchers • Selected researchers identified during the literature review
Journalists • News associations e.g. Associated Press, Reuters, Politico
Any further feedback as to which stakeholders should be targeted through the consultations during either data collection or analysis phases will also be welcomed and integrated.
The questionnaires will also need to be customised to different stakeholder groups. We will also develop suggested text for consultation launch page, as required and pilot the questionnaires.
Once the targeted online survey has been launched, we will monitor the response rate and take steps to improve response rates if necessary.
2.2.3.3 Task 3.2.3 – Interview programme
An interview programme will be undertaken with key stakeholders with a direct or indirect interest in the Single Market.
The purpose of the interview programme will be to:
Undertake stakeholder screening to:
o Obtain feedback from stakeholders that are identified as existing key users of the SMS that
2. Methodology – Phases 2 and 3
12
can provide in-depth feedback;
o Obtain feedback from stakeholders that have a strong interest in the new policy areas that the SMS could be extended to cover
• Obtain qualitative feedback related to the relevance, coherence and effectiveness of the Single Market Scoreboard for measuring the performance of the Single Market policy implementation
• Acquire feedback on possible policy and sector areas that should be covered in an upgraded Single Market Scoreboard
• Solicit stakeholder input concerning the presentation and usability features desirable in an upgraded scoreboard
• Follow-up with respondents to the targeted (online) consultations that are willing to be interviewed
The proposed structure of the interview programme is provided below:
Table 2.2: Interview programme structure
Stakeholder type Examples Number Notes
National Administrations / Organisations from EU Member States
• Ministries of economy, industry,
• National Statistical Offices, etc.
5 National actors using, assessed by and providing some data used in the SMS
DGs with policy links to the Single Market
• Steering committee members
• DG DIGIT, DG SEC GEN, DG EMPL, DG ENV, DG TAXUD, DG SANCO, DG FISMA
• Joint Research Centre
• Eurostat
10 European data provider, DGs with a potential stake in an upgraded SMS, expert in the development of composite indicators – and research on the SMS.
Selected EU and national (representative) organisations
• Business associations and SME associations e.g. Business Europe, Euro-chambers, SMEUnited, Eurocommerce
• European Start-up Network
• European Trade Union Confederation (ETUC)
13 Stakeholders representing employers and employees with a view on the Single Market. Business Europe has published a strategy paper on the opinion on the EC Communication on the Single Market in a changing world.
NGOs and civil society organisations
• Consumer organisations e.g. BEUC, consumers international
• Social Platform
• Member organisations of the EESC liaison group
5 NGOs representing organisations and citizens linked to the four freedoms.
Other Networks • European Enterprise Network (EEN, EURES, ECC-NET
10 Each of the networks listed in the ToR has a stake in the SMS and may also provide and use data and results.
Other interviewees identified via the online survey
• Journalists, academics 2
Total 45
2. Methodology – Phases 2 and 3
13
*To be confirmed and only if agreed as being relevant by the client. Further details regarding the structure of the interview programme and an explanation as to how the interviews will be conducted is now provided.
Following agreement on the stakeholder types, we will first identify a suitable set of target interviewees and integrate these in a contacts database and will then ask for feedback from the client as to whether there are any stakeholders missing that should be included. A further source of information for identifying suitable interviewees will be to analyse the information received through the targeted online consultations. A question will be included asking whether respondents are willing to be contacted and interviewed as part of a follow-up process.
The final selection of interviewees will need to be agreed with the Commission Services prior to launching the interview programme.
In structuring the interview programme presented in the above table, we will ensure not only an appropriate balance in the selection of interviewees across different categories of stakeholder, but also in terms of ensuring an appropriate geographical balance across the interview sample as a whole. Appropriate geographical balance will also be reflected in the selection of associations, enterprises and NGOs.
The interviews to be carried out will be undertaken as semi-structured interviews, customised to the different categories of stakeholders mentioned above. Initially we will conduct mainly face to face interviews to gain a full appreciation of the issues. Subsequently, phone conference interviews will be prioritised.
We will produce a Specific Privacy Statement on data protection and privacy explaining how the data will be collected and explaining their rights under the GDPR. For reasons of preserving the impartiality of third-party contractors in conducting independent technical studies, we can summarise the key messages from particular stakeholder types and once anonymised, provide these to the Commission services.
2.2.3.4 Task 3.2.4 – Focus group discussions
We propose to use a focus group setting to test and validate the analysis work, and in particular the mapping of existing data (Task 2), new data identified (Task 3) and gap analysis (Task 4) performed. The first focus will collect participants in an informal IMAC meeting, foreseen for 24 October 2019. This will focus on the tasks that are covered in the Interim Report. We propose, to the extent possible to use this occasion to get feedback on preliminary thinking concerning new indicators, including user perspective indicators, identified in Task 5.
In addition, we will organise targeted online focus group discussions where needed, in order to validate analysis regarding the gap analysis, possible data sources and/or definitions of new indicators. We will invite selected stakeholders who are experts in particular tools or policy areas that are relevant for the SMS. The detailed composition of the groups will be proposed following the analysis of stakeholder feedback via the survey and interview programme. The timing of the targeted focus groups will mainly coincide with the timing of task 5, where new indicators will be defined and which follows the gap analysis. This can be conducted efficiently via e.g. Skype with a small number of stakeholders.
2.2.4 Timing of consultations
The timing and sequencing of the consultation is clearly laid out in the detailed project plan in section 3.1. This approach allows for stakeholders both to help guide the gap analysis for the existing SMS from an end-user viewpoint and to validate new policy areas and indicator suggestions.
2. Methodology – Phases 2 and 3
14
2.2.5 Data quality
All of the data collected as well as data sources and indicators mapped during Phase 2 of the study will clearly and systematically stored and described. As far as the indicators are concerned, we will aim to establish whether the indicators are RACER (relevant, accepted, credible, easy to understand and robust), in line with the Better Regulation guidelines TOOL #41. As a minimum, this detailed information will contain those areas underlined in section 8.1.4 of the Request for Services, namely:
List 1: information on data collected in the SMS and as part of this study
• Objective of the exercise
• Target population and sampled population
• Data (variables) to be collected
• Degree of precision i.e. are there some missing data or breaks in time series?
• Collection mode, i.e. how will the data be collected (by email, web platforms, dedicated application)
• Periodicity/frequency of a process i.e. is it a one-off exercise or a regular one?
• Validation
• Publication format i.e. in which electronic (open) format will the data be published (plain text CSV files,. Excel, R, or Stata files)
• Metadata i.e. what background information about the data needs to be disclosed to the public:
Data collection methodology;
Target population;
Sampled population;
Glossary and definitions of indicators/variables and their respective measurement units;
Codes, acronyms, flags used (those should normally be harmonised with Eurostat codes, e.g. two-letter country codes);
The timing and frequency of data collection;
The publication date;
Limitations, confidentiality issues, disruptions of methodology etc.
Contact point for potential questions and comments from the public.
In addition, we will tag the data sets and indicators as appropriate by governance area, results chain, policy area and sector, such that users can easily obtain comprehensive information to help assess the quality of the data, but also easily map this to different areas of the Scoreboard and to Single Market implementation.
At this stage, we will consider to what extent the data identified has the potential to be integrated into the Scoreboard in real-time over the longer-term.
2.3 Phase 3 - Data analysis
The data analysis phase includes a gap analysis, defining and recommending new indicators and providing recommendations to improve the presentation of the upgraded scoreboard.
2. Methodology – Phases 2 and 3
15
Throughout the analysis phase the study team will utilise a combination of methods in order to analyse the data. We will support findings and recommendations by explaining the degree to which these are based on opinions, analysis and objectively verifiable evidence. Where opinions are the main source, the degree of consensus and the steps taken to test the opinions will be given.
2.3.1 Task 4 Identification of gaps
In Task 4, we will conduct the gap analysis, with the aim of identifying possible alternate indicators / data sources needed to give a more complete picture of the state of the Single Market and its four freedoms, as well as extending it to additional policy areas.
Task 4 will flow naturally from the previous Tasks, and will involve an analysis of the framework, criteria and heuristics tool developed and an assessment of the relevant consultation feedback.
We consider this task as a multidimensional assessment of the “identification of gaps”, including a gap analysis of the:
1. The broad type of indicator e.g. regulatory/policy, economic, users of the Single Market;
2. Policy areas not current covered by the SMS e.g. network industries, data economy, circular economy etc;
3. Related to the above, key policy sub-areas or critical issues not monitored currently;
4. Whether the indicators fully, partially, or to a limited extent measure the desired policy areas / concepts to be monitored;
5. Consideration of whether the indicators represent the optimal measures of internal market performance i.e. if the measures are causally and significantly linked to key Single Market policies and can demonstrate their effects / impacts.
Therefore, Task 4 will involve a detailed review of this tool to generate a reporting-output specifying “where the gaps exist”, while providing suggestions on how the “gaps can be filled” by new or modified indicators.
This reporting output will be structured according to the criteria / assessment points described above indicating where the main gaps are, while providing analytical commentary assessing the relevance, coherence and effectiveness of the SMS in terms of its gaps and whether the introduction of the alternate indicators / data sources can strengthen the system.
Moreover, the gap analysis will score the coverage of the policy areas by the current SMS indicators and also the alternate possible indicators, to get an idea of possible improvements.
We will also examine the relevant parts of the consultation feedback within this reporting-output. The consultation feedback will present the stakeholder views in terms of whether: their needs are being met by the current SMS; the type of indicators they require; their key areas for improvement of the existing SMS, including additional policy areas.
2.3.2 Analysis of the stakeholder consultation feedback
2.3.2.1 Analysis of the questionnaire responses to the targeted consultation
Having cleaned the data from the online survey, we will analyse the data to provide a clear set of findings and conclusions from the targeted consultation. This will involve aggregated and disaggregated analysis, for instance differentiating between:
• Responses by category of stakeholder;
• Responses by EU Member State
2. Methodology – Phases 2 and 3
16
• Part of the Single Market Scoreboard e.g. Governance tool, sector, policy area, indicator.
• New policy area
We will also provide background information regarding the response to the survey, explain data limitations encountered and give an indication of the degree of consensus in the responses - either through disaggregating the feedback into groups, or through statistics that provide a measure of spread – e.g. the standard deviation or the interquartile range.
Quantitative opinion-based responses received through the targeted consultation will be triangulated against qualitative feedback gathered through the desk research and other forms of consultation i.e. interviews and where appropriate focus groups. At this point additional desk research will be conducted so as to complement the findings from the consultations with official statistics and studies, where possible.
The consultation outcomes will be analysed by our study team and included in the interim report.
2.3.2.2 Analysis of the interview feedback
The next step will be to undertake an analysis of the interview feedback, drawing on the detailed interview notes that will be produced by members of our team. All interview feedback will be collected and stored on a single platform, ensuring cross-cutting analysis of all the feedback. Most of the data collected in the interview programme will be qualitative so the feedback will be analysed and presented in a structured and thematic way. However, some quantitative statistics will also be calculated based on the feedback to identify, for example, the degree of consensus across stakeholder groups.
The research team will also search for themes that have a broader pattern of meaning, respecting the European Commission standards for data protection when analysing responses. The methods used and analytical building blocks (coding system, tags and sub-tags) will be made transparent to readers by providing supporting text in the main report to this effect, clarifying how the findings and conclusions were made.
2.3.2.3 Analysis of the Focus groups discussions
We suggest focus group work to test and validate the analysis work including on the new indicators identified in Task 5. To the extent possible this will be included as part of the informal IMAC meeting following the interim report. We will also conduct selected online focus groups or email exchanges concerning “showcased indicators” as needed (see Task 5).
We will prepare detailed focus group notes and store them on a single platform, ensuring cross-cutting analysis of all the feedback. Most of the data collected in the focus groups will be qualitative so the feedback will be analysed and presented in a structured and thematic way.
2.3.3 Task 5. Definition of new indicators including end user’s perspective indicators
In Task 5, we will define new indicators, including those from the perspective of users such as business and citizens. The indicators defined need to be relevant, as well as causally and significantly linked to Single Market policies demonstrating the real effect of those policies.
We will define the users in two ways:
• Users of the internal market (i.e. citizens and businesses); in the context of the SMS the aim is to identify the obstacles they are facing and monitor the extent to which they can fulfil their EU rights;
• Current users of the Single Market Scoreboard are also those stakeholders that use the indicator results to support their organisational and research activities (i.e. including the stakeholders
2. Methodology – Phases 2 and 3
17
participating in the consultation for this study);
The process to defining the new indicators will involve the following steps:
1. Responding to the needs of end users: the appropriate aspects of the consultation feedback and needs will be considered and integrated into the process to define new indicators;
2. Making links between the Task 2 results around possible tools/datasets and the Single Market intervention logic and relevant policy areas to determine the possibilities for indicator definition;
3. Indicator concept exploration: investigate the specific elements of the policies that can be measured using the tools/datasets reviewed including new types of (user perspective) indicators, and measures to monitor the four freedoms, but also examining if the existing SMS indicators can be improved, considering the methods to be employed;
4. Indicator concept validation considering whether the selected variable alone is sufficient, if a derived indicator should be developed, if estimates are needed e.g. via econometrics/data analytics methods, or if a composite indicator could be appropriate. 4 Then involving:
a) Internal indicator concept validation between the project team members to determine the extent to which the indicators proposed measure appropriately the relevant policies/concepts considering the extent to which they are causally and significantly linked;
b) External indicator concept validation with DG GROW and selected stakeholders to obtain 360-degree feedback on the suggested indicators considering their strengths and weaknesses in measuring the effects of the suggested policy areas and to agree a subset of indicators to be showcased further by the study.
5. Data analysis and showcasing of “prime” indicators defined: for prime indicators identified in the validation exercise provide in-depth case research and testing of possible indicators and related methods, plus visualisations, demonstrating the “prime” indicators that would add significant value in meeting the needs of SMS end users if it were upgraded;
2.3.3.1 Explanation of the data analytics / econometric methods that could be suggested by the indicator showcasing exercise
A series of methods will be reviewed in-depth for the definition of the indicators selected. The explanation of the methods will be indicator specific.
In some instances, a variable used to represent an indicator may not be subject to further transformation if it already represents a meaningful measure of the performance of a relevant policy area. For pragmatic reasons, linked to time and the acceptability criterion in the RACER criteria, we will concentrate our attention on data and indicators that already exist.
However, in some cases, we may need to derive indicators by combining different variables or investigate other data analytics or econometric methods. Indicative methods include cluster analysis, factor analysis, principal components analysis or latent class analysis, social network analysis, regression, latent growth modelling or sentiment analysis.
4 As suggested by the briefing report to the EP IMCO committee ‘Towards Indicators for Measuring the Performance of the Single Market’, composite indicators to measure the internal market performance at a high level across economic and regulatory areas are likely not to produce an insightful measure of performance. However, the SMS could contain already ‘accepted’ composite indicators in specific sub-areas of policy, if appropriate.
2. Methodology – Phases 2 and 3
18
All research methods are associated with advantages / disadvantages, limitations and risks. These will be flagged-up for all indicators defined with further in-depth assessments of these aspects explored in the indicator showcasing exercise. In doing so, we will develop a series of criteria so that these items can be examined evenly for each indicator.
2.3.4 Task 6. Recommendations on indicators
In Task 6, we will recommend the set of indicators to be included in the upgraded Single Market Scoreboard. This set of indicators will allow for an annual benchmarking of Member States in the most relevant Single Market areas. It will be based on:
1. A synthesis analysis of all study outputs;
2. Research on how “real time” information can be obtained from the relevant data-sources (the data sources’ potential for providing real-time updates is one of the aspects that we will investigate when conducting tasks 1-3);
3. An assessment of the relevance, effectiveness and coherence of the proposed and existing indicators;
4. Defining a final set of recommendations validated by stakeholders.
2.3.5 Task 7. Presentation of upgraded scoreboard
In this task we will make recommendations and give ideas on how to 1) improve the user-experience and user-friendliness of the current scoreboard 2) provide easy and simple access to the most meaningful data, for example through a dashboard displaying the most relevant areas for the Single Market.
Recommendations will be provided in areas including:
• Improving the user experience and providing simplicity in accessing quantitative and qualitative data. We will provide recommendations concerning a range of different data visualisations, such that users can understand the performance and challenges of the implementation of the Single Market Scoreboard, viewing and comparing charts, maps, time series, selecting single or multiple variables
• Interactivity and multiple data-set integration. Among the options we will consider and make recommendations on are: making the chart interactive, ranking countries, allowing users to visualise where the data came from, combining data from multiple datasets.
• Cloud-based visualisation dashboards that can be updated in real-time. We will investigate the possibility of using open data sources with an Application Program Interface (API) and cloud-based platforms to build data visualisation tools. Similar solutions could be foreseen using Eurostat’s JSON and UNICODE Web Service.
The recommendations will also be based upon reflections regarding the structure and dimensions of the existing SMS, the use of traffic lights or other ways to signal performance levels, catering to different user needs in terms of level of detail and technical information (through for example headline and operational/managerial indicators) as well as the various platforms and communication channels.
3. Project Management
19
3 Project Management
3.1 Detailed workplan and timeline
The detailed workplan in Figure 3-1 outlines our proposed approach to successfully deliver this study. It takes into account issues discussed at the kick-off meeting. This workplan will be finalised following any remaining comments from the Commission at the inception report meeting.
3.2 Reporting and meetings
During the remaining phases of this study, we will produce/integrate the following reporting-related outputs:
• An Interim report D2 (N+10 weeks), summarising the state of play, highlighting results to date and problems encountered (Tasks 2, 3 and 4), and setting out the workplan to project completion.
• Following the Interim Report and informal IMAC meetings, integrate feedback to improve the quality, preliminary conclusions and the proposed workplan.
• A Draft Final Report D3 (Month 5), which will address all the key study issues identified in the ToR and include sound analysis, draft conclusions and recommendations
• Following the Draft Final Report and IMAC validation meetings, the analysis and recommendations will be refined to bolster the report’s technical quality.
• A Final Report D4 (Month 6), which will take into account the comments and feedback obtained from the meeting and subsequent reviews of the report.
• Alongside the final report, CSES will submit an Executive summary (D5) an abstract in English, French and German and supporting data (D6).
3. Project Management
20
Figure 3-1: Detailed workplan – SMS 2.0
PROJECT PLAN: SINGLE MARKET SCOREBOARD
Contract signature 16/07/2019 July August September October November December JanuaryWeek 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Week 9 Week 10 Week 11 Week 12 Week 13 Week 14 Week 15 Week 16 Week 17 Week 18 Week 19 Week 20 Week 21 Week 22 Week 23 Week 24 Week 25 Week 26 Week 27
Task ID What 15/07 22/07 29/07 05/08 12/08 19/08 26/08 02/09 09/09 16/09 23/09 30/09 07/10 14/10 21/10 28/10 04/11 11/11 18/11 25/11 02/12 09/12 16/12 23/12 30/12 06/01 13/01
P1 Phase 1
0.1 Kick-off meeting - 17/07/2019 Ω
0.2 Team leadership/project management
0.3 Prepare letter of introduction for signature by EC
0.4 Finalise draft work plan
0.5 Prepare short note for informal IMAC meeting
1 Task 1. Literature, sources and stakeholders review (N+3 weeks)
1.01 Review of lit. and other sources
1.02 Create contacts database
1.03 Stakeholder identification
1.04 Develop draft implementation plan for stakeholder consultations
1.05 Develop a framework, criteria and heuristics tool
1.06 Develop tools to facilitate data analysis
1.07 Quality Assurance
1.08 Inception report - 05/08/19
1.09 Inception report meeting - 09/08/2019 Ω
P2 Phase 2
2 Task 2. Mapping of existing tools for monitoring Single Market policies
2.01 Detailed review of the identified tools
2.02 Store info in Excel database
3 Task 3. Identification of available data from different sources
3.1 Task 3.1 Detailed identification of available data from different sources
3.1.01 Detailed review of additional data sources
3.1.02 Store info in Excel database
3.2 Task 3.2 Stakeholder consultations
3.2.01 Finalise implementation plan for stakeholder consultations
3.2.02 Develop online questionnaires for targeted consultations
3.2.03 Identify survey sample
3.2.04 Pilot survey
3.2.05 Conduct survey
3.2.06 Produce data quality note on survey
3.2.07 Semi-structured interview guides by stakeholder category
3.2.08 Produce privacy statement linked to GDPR
3.2.09 Finalise list of interviewees
3.2.10 Conduct interviews
3.2.11 Summarise key messages from interviews
3.2.12 Prepare focus group plans and guides
3.2.13 QA and backstopping
P3 Phase 3
4 Task 4. Identification of gaps
4.01 Analyse stakeholder consultation feedback
4.02 Perform gap analysis for SMS
4.03 Interim report - 09/10/2019
4.04 Interim report meeting - 16/10/2019 Ω
4.05 Informal IMAC meeting - 21/10/2019 Ω
5 Task 5. Definition of new indicators including end user’s perspective indicators
5.01 Develop indicator definition inventory
5.02 Explore indicators concepts
5.03 Internal validation discussion/meeting on indicator concepts
5.04 Hold focus group to validate indicators
5.05 In-depth review of 'prime' indicators
6 Recommendation on indicators
6.01 Research on how real-time info can be obtained from relevant data sources
6.02 Synthesis analysis of study outputs
6.03 Develop recommendations
6.04 Validate and present recommendations
7 Presentation of upgraded scoreboard
7.01 Investigate existing dashboards/scoreboards
7.02 Review and analyse SMS data-set including new indicators
7.03 Provide proof of concept of real-time DV
7.04 Provide recommendations on DV of a new SMS
7.05 Draft Final Report - 16/12/2019
7.06 Draft Final Report meeting 19 or 20/12/2019 Ω
7.07 Revise draft report
7.08 Quality Assurance and backstopping
7.09 Submit Final Report, Exec summary and supporting data - 16/01/2020
7.10 Present report at IMAC meeting - TBC
Total
Legend
Phases
Workflow/ activity
Meetings Ω
Deliverables
Annex 1 - Bibliography
21
3.3 Assumptions
The are several study challenges and issues to be considered in the process of defining indicators for the SMS 2.0. A main challenge in upgrading the SMS is that it implies a broader measurement of the single market, not only as a policy strategy, but also as a concept in itself. Given the multifaceted nature of the Single Market, the definition of the concept to be measured is complex and associated strategic choices will be needed to decide what should not be included as well as what should.
While the existing SMS indicators measure the transposition and enforcement of the existing policies, there is a gap around examining the quality of the application of the legislation typically assessed through qualitative evaluations. In this regard, it will be important that databases that provide value judgements on performance of the policies can be identified and not just information on the outputs that do not suggest the need for action by the Member States. Examples like the World Bank’s Doing Business database helps to overcome such challenges by providing information on the extent of the regulatory and administrative burden for companies when entering and operating in different country markets by compiling data from multiple sources.
Further assumptions we are making to perform this study in the six months available include:
• We will continue to get timely access to data/information, noting that the Commission services have provided information rapidly since the contract began.
• We will need to balance the resources allocated to assessing and improving indicators for existing governance tools and policy areas with identifying new policy areas and associated indicators. In the detailed project plan, we have shifted additional resources towards identifying new policy areas and indicators.
• To keep the survey open for four weeks and integrate preliminary results into the Interim Report, we will need to launch the survey during the holiday period. The interview programme will also need to be launched rapidly and relies on some stakeholders being available in August. Alternatively, the survey could be kept open for a shorter period, but which runs the risk of reducing the number of survey responses.
• Another option is to delay the interim report and interim report meeting, for example by two weeks, which would allow time for additional stakeholder responses, gap analysis and early new indicator concepts to be developed ahead of the informal IMAC meeting. This is not expected to influence the timing of delivery of the final report since the project is currently front-loaded, while the phase 3 tasks have more time and fewer parallel interdependencies (see detailed project plan).
• Task 7 in the ToR requires that we make recommendations and give ideas on how to present new and existing data within the upgraded scoreboard, using visualisation tools ensuring that it is more innovative and user friendly. We will provide recommendations as per our proposal and ensure that we provide suggestions regarding structural aspects of an SMS 2.0. We will attempt a data visualisation prototype for an indicator/indicator concept subject to resource availability.
Annex 1 - Bibliography
22
4 Next steps
A series of next steps will be required to finalise the methodology and practical elements of the data collection and analysis framework as part of the structuring phase. In summary, the next steps are:
4.1 Finalisation of Phase 1
The following Phase 1 sub-tasks proposed in our offer will be finalised shortly:
• A framework, criteria and heuristics tool for assessing the relevance, coherence and effectiveness of the existing and proposed SMS indicators to measure performance in Single Market policy implementation and identifying gaps (matrix). This tool will be shared with the client for comment.
• Finalise the initial version of the contacts database, although this will remain a work in progress.
• Finalisation and approval of this inception report, following feedback from the European Commission’s DG GROW.
4.2 Launch phase 2
The immediate tasks to launch are tasks 2-4 as well as preparation for the informal IMAC meeting planned for October 24th. The interim report will be submitted by 9th October with a meeting to discuss the report scheduled for October 16th.
The draft research tools to facilitate data collection and analysis have been finalised. These were: (1) online questionnaires for the targeted consultations (2) a series of customised and semi-structured interview guides for the different categories of stakeholder and (3) a series of focus group plans and guides and (4) the qualitative tools to analyse key patterns in interview notes and focus groups.
The online survey was launched on 6th September and will remain open until 27th September.
Annex 1 - Bibliography
23
Annex 1 - Bibliography
The following bibliography builds on the initial document examples provided on our technical offer. It will continue to be developed and analysed during Phase 2.
SMS area SMS sub-area Title Author
General. General An easier consumer journey when buying online | BusinessEurope
An EU New Year's Resolution: Keep Boosting the Single Market Berkjem, Johan &
Harbour, Malcolm
Annual Report on European SMEs 2017/2018 Special Background Document on the internationalisation of SMEs
Business services: Konzepte, Technologien, Anwendungen; 9. Internationale Tagung Wirtschaftsinformatik, Wien, 25. - 27. Februar 2009. Bd. 2: ...
Consumer Conditions Scoreboard - Trust in e-commerce dramatically increases - 2017 Edition
Eurasian Economic Perspectives: Proceedings of the 20th Eurasia Business and Economics Society Conference -
Bilgin, Mehmet Huseyin; Danis, Hakan; Demir, Ender; Can, Ugur
Indicators for measuring the Performance of the Single Market - Building the Single Market Pillar of the European Semester
Pelkmans, Jacques
Indicators for the internal market? An unfinished business Saltelli, Andrea
justice_scoreboard_2019_en.pdf
Making EU trade in services work for all
Mapping the cost of non-Europe, 2014-19 study Hiller, Wolfgang;
Europäisches Parlament
Opentender - Opentender Portals digiwhist
Rebooting the Single Market: The Top Priority for Eu Growth Harbour, Malcolm
Study to monitor the business and regulatory environment affecting the collaborative economy in the EU final report.
Karanikolova, Kristina; Naumanen, Mika; Juskevicius, Ricardas; Rabuel, Lison; Porsch, Lucas; European Commission; Directorate-General for Internal Market, Industry, Entrepreneurship and SMEs
Tenders Electronic Daily (TED) (csv subset) – public procurement notices - ecodp.common.ckan.site_title
THE EU DIGITAL SINGLE MARKET FROM A CONSUMER STANDPOINT: HOW DO PROMISES MEET MEANS?
Havu, Katri
Towards Indicators for Measuring the Performance of the Single Market Pelkmans, Jacques
Scoreboards Consumer markets scoreboard
Consumer conditions scoreboard
EU Monitoring Framework for the Circular Economy
Regional Ecosystem Scoreboard (Innovation and Entrepreneurship)
European Innovation Scoreboard (also the Regional Innovation Scoreboard)
EU Justice Scoreboard
Euro indicators/PEEI Scoreboard
Annex 1 - Bibliography
24
EU Open Data
European Enterprise Network
European Central Bank financial integration in Europe Annual Report
European Citizens Consultations (conducted by the European Movement International)
Tenders Electronic Daily (TED)
Public procurement database
Product Markets Regulation database
Government at a Glance
OECD business statistics
Regulatory Policy Outlook
Country and thematic reports on public procurement and digital government
Doing Business in guides
Worldwide Governance Indicators
Business Environment and Enterprise Performance Surveys
World Economic Forum Global Competitiveness Report
Institutional Profiles Database
Bertelsmann Transformation Index
Gallup world poll, with UN Sustainable Development Solutions Network
KETs database (publicly available)
Eurostat SBS
Eurostat Prodcom
Amadeus/ Orbis
FDI Markets database - Financial Times (FT)
Governance Tool
Administrative cooperation between national authorities
Consumer Protection Cooperation Network
Consumer protection in the EU: policy overview : in-depth analysis. Valant, Jana; EP;
Directorate-General for Parliamentary Research Services
EC Consumer Law Howells, Geraint G.;
Wilhelmsson, Thomas; Wilhelmsson, Thomas
EMOTA Position Paper on the European Commission Consultation "Towards a Coherent European Approach to Collevtive Redress", SEC (2011) 173 Final
EMOTA
Ensuring consumer rights are properly enforced: Revising EU Consumer Protection Cooperation
European Commission
Networks to Enforce European Law: The Case of the Consumer Protection Cooperation Network
Poncibò, Cristina
Regulation (EU) 2017/ of the European Parliament and of the Council of 12 December 2017 on cooperation between national authorities responsible for the enforcement of consumer protection laws and repealing Regulation (EC) No 2006/2004
Annex 1 - Bibliography
25
Assistance services for citizens and businesses
2018 Annual Report on Intra-EU Labour Mobility
Access to public procurement contracts in EU: perspective of SMEs Bobowski, Sebastian;
Gola, Jan; Szyd\lo, Wojciech
Altius - E-procurement in Belgium: Towards a complete automation of public procurement?
Gellezak, Justin
Answer to a written question - No Polish-language version of e-Certis brochure - E-006587/2011
COMMISSION IMPLEMENTING DECISION (EU) 2018/ 170 - of 2 February 2018 - on uniform detailed specifications for data collection and analysis to monitor and evaluate the functioning of the EURES network
COMMISSION IMPLEMENTING REGULATION (EU) 2016/ 7 - of 5 January 2016 - establishing the standard form for the European Single Procurement Document
Commission Implementing Decision (EU) 2018/170 of 2 February 2018 on uniform detailed specifications for data collection and analysis to monitor and evaluate the functioning of the EURES network (Text with EEA relevance. )
Commission Implementing Regulation (EU) 2016/7 of 5 January 2016 establishing the standard form for the European Single Procurement Document (Text with EEA relevance)
Correct Application of EU Law by National Public Administrations and Effective Individual Protection: the SOLVIT Network
Lottini, Micaela
Developments in Public Procurement in Hungary–Opportunities and Challenges Rigó, Csaba Balázs
Directive 2014/24/EU of the European Parliament and of the Council of 26 February 2014 on public procurement and repealing Directive 2004/18/EC
European Commission
Directive 2014/24/EU of the European Parliament and of the Council of 26 February 2014 on public procurement and repealing Directive 2004/18/EC Text with EEA relevance
Do Invisible Borders Still Restrict Consumer Access to Services in the EU? ECC-Net
eCertis
European Commission
e-Certis KOLINKOVA, Eliska
Employers’ skill preferences across Europe: between cognitive and non-cognitive skills
Kureková, Lucia Mýtna; Beblavý, Miroslav; Haita, Corina; Thum, Anna-Elisabeth
Employment: Commission proposes to improve EURES job search network Andor, László
EU social security coordination
EURES European
Commission
European Consumer Centres Network
European Single Procurement Document (ESPD) ALEKSANDROVA, Zhenya
European single procurement document and eCertis Anonymous
Governance Issues in the EU's e-Procurement Framework Khorana, Sangeeta;
Ferguson-Boucher, Kirsten; Kerr, William A.
Governance through Cooperation: Solvit System and its Role in the Correct Implementation of the European Law by the National Public Administrations
Mătușescu, Constanța; Mares, Christopher; Gilia, Claudia
Annex 1 - Bibliography
26
Home | Europass
Labour Mobility in Europe: An Untapped Resource? Barslund, Mikkel;
Busse, Matthias; Schwarzwälder, Joscha
Labour Mobility in Europe: An untapped resource? CEPS Policy Brief No. 327, March 2015
Barslund, Mikkel; Busse, Mathias; Schwarzwälder, Joscha
Labour Mobility in the Euro Area: Cure or Curse for Imbalances? auf dem Brinke,
Anna & Dittrich, Paul-Jasper
Moving & working in Europe - Employment, Social Affairs & Inclusion - European Commission
Network of legal experts (MoveS) - Employment, Social Affairs & Inclusion - European Commission
New Skills Agenda for Europe - Employment, Social Affairs & Inclusion - European Commission
Points of Single Contact European
Commission
Posting of workers - Consilium
REGULATION (EU) 2016/ 589 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL - of 13 April 2016 - on a European network of employment services (EURES), workers' access to mobility services and the further integration of labour markets, and amending Regulations (EU) No 492/ 2011 and (EU) No 1296 / 2013
Regulation (EU) 2016/589 of the European Parliament and of the Council of 13 April 2016 on a European network of employment services (EURES), workers' access to mobility services and the further integration of labour markets, and amending Regulations (EU) No 492/2011 and (EU) No 1296/2013 (Text with EEA relevance)
REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL on the review of the practical application of the European Single Procurement Document (ESPD)
Single Market Scoreboard – eCertis
Skills and qualifications - Employment, Social Affairs & Inclusion - European Commission
Social Agenda
Social Scoreboard - Competence Centre on Composite Indicators and Scoreboards - EU Science Hub - European Commission
SOLVIT European
Commission
The European Commission Single Market Scoreboard Fernández Moriana, Vanessa; Vida, Alexandra Melissa; De Lombaerde, Philippe
The European Single Procurement Document Telles, Pedro
The Solvit National Coordination Mechanisms – an Organizational Analysis Moldoveanu,
George; Năstase, Bogdan
THE USE OF DIGITAL PUBLIC SERVICES BY ROMANIANS, IN LIGHT OF THE DIGITAL SINGLE MARKET.
ANIELA, BĂLĂCESCU
Will a European Unemployment Benefits Scheme Affect Labour Mobility? Alcidi, Cinzia;
Barslund, Mikkel; Busse, Matthias; Nicoli, Francesco
Annex 1 - Bibliography
27
Your Europe European
Commission
Your Europe Advice European Commission
ECC-net indicators on the SMS
ECC home page
ECC 10 year anniversary report
ECC - legal role
Your Europe SMS Indicators
Your Europe Portal
Single Digital Gateway
Your Europe Advice Portal
Your Europe Advice SMS Indicators
European Consumer Action Service Portal
SOLVIT SMS Indicators
SOLVIT Portal
SOLVIT Action Plan 2017
Recommendation on Principles Governing SOLVIT 2013
Recommendation on Principles for Using SOLVIT 2013
Points of Single Contact DG GROW Webpage
Services Directive setting up the PSCs
PSC Charter
PSC SMS Indicators
An assessment of the PSCs 2015
Formal and informal cooperation between EC and MS
2019 Communication from Commission — Modification of the calculation method for lump sum payments and daily penalty payments proposed by the Commission in infringements proceedings before the Court of Justice of the European Union
European Commission, DG GROW
Best practices in the internal market - European Commission European
Commission, DG GROW
Comparing transposition of EC law in an enlarged Union - the swift, the slow, and the slack.
ECPR
Complaint Form for breach of EU law – European Commission European
Commission, DG GROW
EPC WORKING PAPER No.25 Making Europe work: improving the transposition, implementation and enforcement of EU legislation, June 2006
Lorenzo Allio and Marie-Hélène Fandel, EPC
European Commission - PRESS RELEASES - Press release - January 2019 infringements package: key decisions
European Commission, DG GROW
Infringement Decisions European
Commission, DG GROW
Infringement procedure European
Commission, DG GROW
Infringement Proceedings in EU Law Luca Prete
Annex 1 - Bibliography
28
Infringements European
Commission, DG GROW
Ministerial Transposition of EU Directives: Can Oversight Improve Performance? Radoslaw Zubek*,
Katarina Staronova
Monitoring implementation of EU directives European Commission
European Commission, DG GROW
Monitoring the implementation of EU law: tools and challenges Ballesteros, Marta
Rule of law infringement procedures A proposal to extend the EU’s rule of law toolbox
Petra Bárd and Anna Śledzińska-Simon, CEPS (CEPS Papers in Liberty and Security in Europe, No. 2019-09, May 2019)
Sectors at Different Speeds: Analyzing Transposition Deficits in the European Union Frans van Waarden,
University College, Utrecht University and Markus Haverland, Erasmus University Rotterdam - Faculty of Social Science
Single Market Scoreboard – Transposition European
Commission
European Commission, DG GROW
THE EU TRANSPOSITION DEFICIT Paper Haverland, Markus
Single Market Scoreboard – EU Pilot
Communication from the Commission - EU law: Better results through better application
Annual reports on monitoring the application of EU law
IMI Portal
IMI Regualtion
IMI indicators on the SMS
PolicyArea Circular and low carbon economies
2030 climate & energy framework European Commission
A European strategic long-term vision for a prosperous, modern, competitive and climate neutral economy
European Commission
A European strategy for plastics in a circular economy European
Commission
A monitoring framework for the circular economy European Commission
Accelerating the transition to the circular economy: improving access to finance for circular economy projects
European Commission
Advancing to a Circular Economy: three essential ingredients for a comprehensive policy mix
Milios L.
Analysis of certain waste streams and the potential of Industrial Symbiosis to promote waste as a resource for EU Industry
IDEA Consult, Bilsen et al
Annex 1 - Bibliography
29
Binding annual greenhouse gas emission reductions by Member States from 2021 to 2030 contributing to climate action to meet commitments under the Paris Agreement and amending Regulation (EU) No 525/2013
European Commission
Cases of implementing resource efficient policies by the EU industry - Final Report IDEA Consult, Bilsen
et al
Circular economy European Commission
Delivering the circular economy: a toolkit for policymakers Ellen Macarthur
Foundation
Development and implementation of initiatives fostering investment and innovation in construction and demolition waste recycling infrastructure
IDEA Consult, Bilsen et al
Environment and Internal Market European
Commission
EU Emissions Trading System (EU ETS) European Commission
European Circular Economy Stakeholder Platform
Exchange of good policy practices promoting Innovative/Green Business Models IDEA Consult, Bilsen
et al
Fourth report on the State of the Energy Union European Commission
Governance of the energy union and climate action (EU) Regulation 2018/1999 European
Commission
Growth Within: a circular economy vision for a competitive Europe Ellen Macarthur Foundation
Identifying Levers to unlock Clean Industry - background report IDEA Consult, Bilsen
et al
Identifying Levers to unlock Clean Industry - summary report IDEA Consult, Bilsen et al
Integrating Rather than Juxtaposing Environmental Policy and the Internal Market Hans H. B. Vedder
Interrogating the circular economy: the moral economy of resource recovery in the EU
Gregson, N., Crang, M., Fuller, S., Holmes, H.
Movements of waste across the EU's internal and external borders EEA
Our Vision for A Clean Planet for All: Economic Transition European
Commission
Regulatory barriers for the Circular Economy Technopolis Group in consortium with Fraunhofer ISI, Thinkstep and Wuppertal Institute.
Report EC on the implementation of the Circular Economy Action Plan European
Commission
Staff working document with details on the 54 actions included in the action plan European Commission
Study on the Energy Saving Potential of Increasing Resource Efficiency – Final Report Oeko Insitut, IDEA
Consult, …
Technical Report accompanying the Study on the Energy Saving Potential of Increasing Resource Efficiency
Oeko Insitut, IDEA Consult, …
The Circular Economy and Benefits for Society Wijkman, A.,
Skånberg, K.
Towards a circular economy: A zero waste programme for Europe European Commission
Annex 1 - Bibliography
30
Towards the circular economy Ellen Macarthur
Foundation
Treating Waste as a Resource for EU Industry ECSIP Consortium
Understanding the Circular Economy in Europe, from Resource Efficiency to Sharing Platforms: The CEPS Framework
Taranic, I., Behrens, A., & Topi, C.
Upgrading the Single Market: more opportunities for people and business European
Commission
Collaborative Economy
An empirical analysis of factors affecting sharing economy growth Virginija Grybaitė, Jelena Stankevičienė
European Perspectives on Participation in the Sharing Economy Andreotti et. al
Impact of the Sharing Economy in Germany BMBF
Measuring the collaborative economy: a case study in tourism statistics Dr Vasiliki Benaki-
Kyprioti
Multidisciplinary Framework on Commons Collaborative Economy decode
Sharing Economy Index Epicenter
Shifting Perceptions of Collaborative Consumption National League of Cities
Single Market Scoreboard: Collaborative Economy European
Commission
Study to Monitor the Economic Development of the Collaborative Economy in the EU Technopolis, VVA, Triconomics
The Collaborative Economy Nesta
The Collaborative Economy Based Analysis of Demand: Study of Airbnb Case in Spain and Portugal
Pedro R. Palos-Sanchez1 and Marisol B. Correia
The Collaborative Economy: Unlocking the power of the workplace crowd Deloitte
The Feasibility of Measuring the Sharing Economy Michael Hardie
The Impact of the Collaborative Economy on the Labour Market Willem Pieter De
Groen and Ilaria Maselli
The Sharing Economy: New Opportunities, New Questions Credit Suisse
Timbro Sharing Economy Index Timbro
Understanding the Sharing Economy World Economic
Forum
Environment A measure of protection: A baseline report on performance measurement in the Scottish Environment Protection Agency
SEPA
Assessing Environmental Performance in the GMs GMS-CEP
Environment, Energy, and Economic Development RAND Corporation
Environmental indicator frameworks to design and assess environmental monitoring programs
Tomás B. Ramos , Sandra Caeiro & João Joanaz de Melo
Environmental Indicator Report 2017 EEA
Environmental Indicator Report 2018 EEA
Environmental Indicators EEA
OECD
Environmental Measures and Modeling: Monitoring Programmes EPA
Environmental metrics: government’s approach to monitoring the state of the natural environment
National Audit Office (UK)
Environmental Performance Index Yale
Annex 1 - Bibliography
31
Environmental Performance Report 2017 National renewable
Energy Laboratory (USA)
Environmental Performance Review of Germany OECD
Environmental Performance Review: Albania UNECE
Environmentl Indicators: A Systematic Approach to Measuring and Reporting on Environmental Policy Performance in the Context of Sustainable Development
World Resources Institute
Ideas for defining environmental objectives and monitoring systems for a results-oriented CAP post 2020
Institute for European Environmental Policy
National Environment Monitoring Swedish EPA
Financial markets
Banking and financial services European Parliament
BetterFinance European
Commission - European Commission
Capital markets union Financial Times
Capital Markets Union: An Action Plan of Unfinished Reforms Constantin Gurdgiev,
Trinity College, Dublin C, April 2019, with Tracy Lee Lyon, Alexandra Cohen, Margaret Poda and Matthew Salyer (Middlebury Institute of International Studies at Monterey (MIIS); GUE/NGL Group, European Parliament, Policy Analysis Paper, March 2019
Capital Markets Union: Creating a stronger and more integrated European financial supervisory architecture, including on anti-money laundering
European Commission
Communication from the Commission on anti-money laundering (COM(2018) 645 final. Strengthening the Union framework for prudential and anti-money laundering supervision for financial institutions.
European Commission
Communication from the Commission on capital market union: Progress on building a single market for capital for a strong economic and monetary union (SWD(2019) 99 final)
European Commission
Financial services policy | Fact Sheets on the EU | European Parliament European
Parliament, Lexalert
Legislative train schedule European Commission
Rebranding Capital Markets Union European Parliament
Regulation (EU) No 345/2013 of the European Parliament and of the Council of 17 April 2013 on European venture capital funds
European Commission
Summary overview of EU policies on banking and financial services European
Commission
The EU needs to pick up the pace on capital markets union CEPS
The Free Movement of Capital and Financial Services (Cambridge Scholars) Graeme Baber, Cambridge Scholars
The Single Market: Financial Services and the Free Movement of Capital
Annex 1 - Bibliography
32
General Trade in Goods and Services SMS Indicators
DG GROW Single Market for Goods
DG GROW Single Market for Services
PRODCOM Trade in Goods Statistics
Eurostat Balance of Payments
Eurostat international trade in services
Network industries
2015-12-analysis-and-recommendation-on-collection-of-data-in-the-field-of-general-aviation-in-europe.pdf
Air Smith, John
Air transport: market rules | Fact Sheets on the European Union | European Parliament
COMMUNICATION FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS Progress towards completing the Internal Energy Market
Directive (EU) 2019/944 of the European Parliament and of the Council of 5 June 2019 on common rules for the internal market for electricity and amending Directive 2012/27/EU (Text with EEA relevance.)
Energy consumer rights user_administrator
Energy Union Data & analysis user_administrator
Energy Union indicators user_administrator
Evaluation of the Performance of Network Industries Providing Services of General Economic Interest
Buelens, Christian
EVALUATION of the Regulation (EC) No 1008/2008 on common rules for the operation of air services in the Community
Executive summary of the evaluation of the Regulation (EC) No 1008/2008 on common rules for the operation of air services in the Community
Internal energy market | Fact Sheets on the European Union | European Parliament
Internal market Smith, John
Market Smith, John
Market Functioning in Network Industries Electronic Communications, Energy and Transport
Brons, Martijn; Bucher, Anne; Europäische Kommission; Generaldirektion Wirtschaft und Finanzen
Market functioning in network industries: electronic communications, energy and transport
Market legislation user_administrator
Monitoring progress towards the Energy Union objectives - Concept and first analysis of key indicators
Monitoring progress towards the Energy Union objectives – key indicators
Network industries: efficient regulation, affordable and adequate services Centre on
Regulation in Europe
Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL on common rules in the field of civil aviation and establishing a European Union Aviation Safety Agency, and repealing Regulation (EC) No 216/2008 of the European Parliament and of the Council COM/2015/0613 final - 2015/0277 (COD)
Rail Market Monitoring (RMMS) BORMANS, Yves
Rail transport | Fact Sheets on the European Union | European Parliament
Recast of the first railway package (Dir 2012/34/EU) Anonymous
Annex 1 - Bibliography
33
Regulation (EU) 2019/943 of the European Parliament and of the Council of 5 June 2019 on the internal market for electricity (Text with EEA relevance.)
Regulation of network industries in the European Union and in Central and Eastern Europe
Major, Iván; Kiss, Károly M.
REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL: Sixth report on monitoring development of the rail market
Standing Up for Competition: Market Concentration, Regulation, and Europe’s Quest for a New Industrial Policy |
The Future of Rail: Regulation & Competition for an Innovative Industry
The Future of the Digital Single Market
Towards Interoperability within the EU for Electricity and Gas - Data Access and Exchange (Report)
Anonymous
Towards Interoperability within the EU for Electricity and Gas Data Access & Exchange
European Smart Grids Task Force; Expert Group 1 – Standards and Interoperability; Working Group on Data Format & Procedures
What do we want to achieve? Anonymous
Postal Services
Development of cross-border e-commerce through parcel delivery : final report. Union, Publications Office of the European
Directive 2008/6/EC of the European Parliament and of the Council of 20 February 2008 amending Directive 97/67/EC with regard to the full accomplishment of the internal market of Community postal services
Directive 98/5/EC of the European Parliament and of the Council of 16 February 1998 Smith, Rhona
ERGP PL (17) 36 Report on Core Indicators 15 December 2017_.pdf
ERGP Report on the outcome of the ERGP public consultation on the evolution of the Universal Service Obligation
Maraschin, Emma
EU postal legislation, implementation and enforcement Anonymous
How customer demands are reshaping last-mile delivery | McKinsey
IPC Global Postal Industry Report
IPC Postal Sector Sustainability Report 2018
Main developments in the postal sector (2010-2013). Union, Publications
Office of the European
Main developments in the postal sector (2013-2016). Union, Publications
Office of the European
Postal sector studies Anonymous
Postal statistics Anonymous
Statistics | Internal Market, Industry, Entrepreneurship and SMEs | European Commission
Public Procurement
Benchmarking Public Procurement 2017 World Bank Group
Buying Green! Handbook European
Commission
Commission Implementing Regulation (EU) 2016/7 of 5 January 2016 establishing the standard form for the European Single Procurement Document (Text with EEA relevance)
European Commission
Annex 1 - Bibliography
34
Corruption in Public Procurement: Finding the Right Indicators Joras Ferwerda,
Ioana Deleanu, Brigitte Unger
Determinants of direct cross-border public procurement in EU Member States Zornitsa Kutlina-
Dimitrova, Csilla Lakatos
Digiwhist public procurement data Digiwhist Project
Directive 2014/24/EU of the European Parliament and of the Council of 26 February 2014 on public procurement and repealing Directive 2004/18/EC (Text with EEA relevance) Directive 2014/24/EU of the European Parliament and of the Council of 26 February 2014 on public procurement and repealing Directive 2004/18/EC (Text with EEA relevance)
European Commission
EU Open Data Portal European
Commission
Evaluating the Effectiveness of Public Procurement Performance Management Systems in Local Governments
Andrea Stefano Patrucco, Davide Luzzini & Stefano Ronchi
Measuring procurement performance in Europe Anthony Flynn
New ways to measure institutionalised grand corruption in public procurement U4
Opentender Digiwhist Project
Preferential Public Procurement for Small and Medium Enterprises DCED
Project Performance Indicators as an Innovative Tool for Identifying Sustainability Perspectives in Green Public Procurement
Konstantinos I.Vatalis, Odysseus G.Manoliadis, Dimitrios G.Mavridisa
Public Procurement for a Circular Economy European
Commission
Public Procurement Indicators 2015 European Commission
Public Procurement Toolbox OECD
Public Procurement Topic Guide Transparency
International
Single Market Scoreboard: Public Procurement European Commission
SMEs in Public Procurement: Practices and Strategies for Shared Benefits European
Commission
Strategic Public Procurement European Commission
TED dataset: Public Procurement Notices European
Commission
Social 2018 Annual Growth Survey and Joint Employment Report - Council conclusions (15 March 2018)
Coordination of Social Security Systems in Europe Giubboni, Stefano;
Iudicone, Feliciano; Mancini, Manuelita; Faioli, Michele
Document database - Employment, Social Affairs & Inclusion - European Commission
EU social indicators - Employment, Social Affairs & Inclusion - European Commission
EU social indicators dataset - Employment, Social Affairs & Inclusion - European Commission
EU social security coordination
Annex 1 - Bibliography
35
Free movement - EU nationals - Employment, Social Affairs & Inclusion - European Commission
https://composite-indicators.jrc.ec.europa.eu/social-scoreboard Social Scoreboard
https://ec.europa.eu/social/main.jsp?catId=737&langId=en&pubId=8183&furtherPubs=yes
Indicators' sub-group - Employment, Social Affairs & Inclusion - European Commission
Portfolio of EU Social Indicators for the Monitoring of Progress Towards the EU Objectives for Social Protection and Social Inclusion
Social Protection Committee; Indicators Sub-group
Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL amending Regulation (EC) No 883/2004 on the coordination of social security systems and regulation (EC) No 987/2009 laying down the procedure for implementing Regulation (EC) No 883/2004 (Text with relevance for the EEA and Switzerland) - Analysis of the final compromise text with a view to agreement
Social Protection Committee - Employment, Social Affairs & Inclusion - European Commission
Professional Qualifications
Professional Qualifications SMS Indicators
Professional Qualifications Directive 2005
Regulated Professions Database
Regulated Professions Interactive Map
Digital single market
The Future of the Digital Single Market Alexandre De Streel, Univ of Namur
Digital Single Market: bringing down barriers to unlock online opportunities European
Commission
Shaping the Digital Single Market European Commission
Eurobarometer survey on new rules ending unjustified geo-blocking and cross-border portability
European Commission
Factsheet on the DSM: A digital single market for the benefit of all Europeans European
Commission
A Europe that protects: EU reports on progress in fighting disinformation ahead of European Council
European Commission
Commission's report shows that targeted investment and robust digital policies boost Member States' performance
European Commission
Digital Single Market: Europe announces eight sites to host world-class supercomputers
European Commission
New EU rules on e-commerce European
Commission
Building a European data economy European Commission
Connectivity for a European Gigabite society European
Commission
Digital skills and jobs European Commission
Digital skills training blueprints for upskilling SME employees and unemployed persons
BlueSpecs, et al.
Artificial Intelligence, Real Benefts Viola R., European
Commission
Digital Agenda Scoreboard key indicators — Digital Scoreboard - Data & Indicators European Commission
The Data Monitoring Tool European
Commission
Annex 1 - Bibliography
36
2019 DESI Report - Electronic communications markets overview per Member State (Telecom chapters)
European Commission
TradeMarket
FDI European Union Foreign Direct Investment [1999 - 2019] [Data & Charts] CEIC Data
FDI Regulatory Restrictiveness Index - OECD OECD
Foreign direct investment and its drivers: a global and EU perspective ECB
Foreign direct investment report: continuous rise of foreign ownership of European companies in key sectors
European Commission
Foreign Direct Investment Screening in the EU Degreef, C
Intra-EU trade in goods - recent trends - Statistics Explained Eurostat
Market access: Trade barriers OECD
Measures Of Restrictions On Inward Foreign Direct Investment For OECD Countries Golub, Stephen, S.
The determinants of Foreign Direct Investment F Mistura, C Roulet
The determinants of foreign direct investment into European transition economies Alan A bevin,S Estrin
Trade Policy COMMISSION STAFF WORKING DOCUMENT on FOREIGN DIRECT INVESTMENT IN THE
EU Following up on the Commission Communication "Welcoming Foreign Direct Investment while Protecting Essential Interests" of 13 September 2017
European Commission
Scoreboard - Eurostat Eurostat
Services Trade Restrictiveness Index - Trends up to 2019 OECD
Services Trade Restrictiveness Index Regulatory Database OECD
The OECD Services Trade Restrictiveness Index provides a snapshot of services regulatory regimes to help policymakers to assess reform options.
OECD
Trade in Value Added - OECD OECD
World Integrated Trade Solution World bank
37