results of the c3s user satisfaction survey 2019...c3s user satisfaction survey 2019 final 1 . about...

Results of the C3S User Satisfaction Survey 2019 January 2020

Issued by: Data Support Team, ECMWF

Date: 31/01/2020

REF.: C3S User Satisfaction Survey 2019 final.docx

Copernicus Climate Change Service

Contents

Executive Summary ......................................................................................... 3

About the survey ................................................................................................ 1

Target audience ............................................................................................... 1

The questionnaire ............................................................................................ 1

Q1: Where are you based? .................................................................................. 3

Results ........................................................................................................... 3

Recommendations ........................................................................................... 5

Q2: Which sector does your organisation belong to? ............................................... 6

Results ........................................................................................................... 6


Q3: Overall, how satisfied are you with the Copernicus Climate Change Service? ........ 8

Results ........................................................................................................... 8


Q4: How useful are the following C3S products for you, and how satisfied are you with them? ............................................................................................................. 10

Results ......................................................................................................... 10

Recommendations ......................................................................................... 12

Q5: How useful is, and how satisfied are you with the following aspects of the C3S Service ............................................................................................................ 13

Results ......................................................................................................... 13


Q6: How useful is, and how satisfied are you with the following functions of the Climate Data Store (CDS)? ............................................................................................ 17

Results ......................................................................................................... 17


Q7: How likely are you to recommend C3S products and services to others? ............ 19

Results ......................................................................................................... 19


Q8: How do you use C3S (e.g. your user story)? .................................................. 20

Results ......................................................................................................... 20

Q9: What changes in C3S products and services would have the most beneficial impact on your work? .................................................................................................. 21

Results ......................................................................................................... 21


Q10: What additional information would you consider useful to be included with the datasets in the Climate Data Store (CDS)? .......................................................... 21

Results ......................................................................................................... 21

Q11: How do you prefer to find out about changes to C3S (e.g. new dataset, system maintenance, etc)? ........................................................................................... 22

Results ......................................................................................................... 22

Q12: Do you have any other suggestions on how we can improve C3S for you? ....... 22

Results ......................................................................................................... 22

Comparison with results from previous C3S User Surveys ...................................... 24

Results ......................................................................................................... 24


Executive Summary Users of the Copernicus Climate Change Service (C3S) are overall satisfied with the service provided by C3S and rate the service with an overall 4.2 out of 5.

C3S serves a global user community: around two thirds (64%) of survey participants are based outside the European Union, and 36% in the EU. Within the EU, about 60% of participants are from only five countries (Germany, Italy, UK, Spain, France). Around 70% of all participants are academics and researchers, and within the EU 60% work in this sector, with the majority of other users equally based in the private and public sector (18% each). From 2018 to 2019 the country and sector demographics remain largely unchanged.

The utility of C3S data products and services varies significantly: climate reanalysis data (mainly ERA5 datasets) are by far the most popular and receive the highest satisfaction ratings. Other C3S products and services are increasing in popularity (such as seasonal forecast datasets), although some are of rather specialist interest (monthly sea ice maps) and user interest remains low. Users of these products are mostly satisfied though. As in previous years, the survey participants raised many ideas and suggestions for C3S, in particularly in relation to the Climate Data Store (CDS). It is expected that these user requirements will continue to inform the development of the CDS.

Users also find supporting services useful, especially the CUS helpdesk, data access mechanisms (in particular the CDS API), and the available documentation. However, the results suggest that data retrieval from the CDS and the CDS Toolbox remain an important area for improvement.

The C3S newsletter and new CUS Forums should be used to strengthen the impact of user communications, and to help build a C3S user community.

The demographics, usage patterns and satisfaction ratings show that uptake in the private and public sectors is continuing to grow.

The previous year’s survey had identified several areas for improvement. The 2019 survey confirmed that two of them still require improvement, while highlighting the challenges associated with delivering ‘state of the art’ climate data and information efficiently. These areas are (a) the footprint of C3S in Eastern Europe (b) the value and impact of C3S in the private sector and public administration.


C3S User Satisfaction Survey 2019 Final 0

C3S significant events relevant to User Support for the period 01/06/2018 - 31/05/2019

• Launch of Climate Data Store (CDS) in June 2018 • SF data enhancement (DWD + CMCC) in November 2018 • Launch of User Learning Service in December 2018 • Web-API seasonal forecast ended on 08/01/2019 (now via CDS access) • ERA5 1979-1999 data released in CDS in June 2018 • Web-API ERA5 access ended on 05/03/2019 (now via CDS access) • Launch of the State of the European Climate in April 2019 • CDS FAQs moved over to Knowledge Base in April 2019 • Launch of CUS Forum on 08/05/2019 • ERA5 monthly means released via CDS in May 2019 • EFAS data released via CDS in May 2019 to build on existing CDS capabilities



About the survey

Target audience The purpose of the C3S User Satisfaction Survey 2019 was to understand how satisfied C3S users are with the services provided, to collect feedback, analyse them and help improve C3S services.

The survey targeted all users who, in the 12 months preceding the survey, accessed C3S data products or who interacted with Copernicus User Support, and for whom email addresses were available. This demographic comprised 28570 distinct users.

The survey was deployed via an online questionnaire, in English, which was open between 1st July and 22nd July 2019. C3S invited all 28570 identified users by email to participate in the survey.

In total, 2102 users participated (7.4% of addressed users). This compared with 2018 figures of 1323 participating users (6.8% of addressed users).

For further information please contact [email protected].

The questionnaire The following questionnaire was used:

Q1: Where are you based? (List of countries and territories)

Q2: Which sector does your organisation belong to? (Public authorities / University and Research / Commercial or private sector / International organisation or NGO / Other)

Q3: Overall, how satisfied are you with the Copernicus Climate Change Service? (1 = not satisfied, 5 = very satisfied)

Q4: How useful are the following C3S products and services for you, and how satisfied are you with them?

• Climate Projections (e.g. CMIP5) • Reanalysis (ERA-Interim, ERA5, UERRA) • Seasonal forecasts data and charts • Satellite observations (e.g. SSTs, Albedo, etc) • Sectoral climate indices

Q5: How useful is, and how satisfied are you with the following aspects of the C3S service?

• C3S website • Climate Data Store (CDS) • Climate bulletins and European State of the Climate • Available data formats • Documentation • Knowledge base • C3S User Forum • Helpdesk / Support • Mailing list



• C3S User Learning Service (ULS) • Newsletter

Q6: How useful is, and how satisfied are you with the following functions of the Climate Data Store (CDS)?

• CDS Data Catalogue (presentation/navigation) • CDS Data Overviews • CDS Data download form • CDS Documentation • Automated data download (CDS API) • CDS Toolbox

Q7: How likely are you to recommend C3S products and services to others?

• Very likely • Likely • Unlikely • Very unlikely

Q8: How do you use C3S (e.g. your user story)?

Q9: What changes in C3S products and services would have the most beneficial impact on your work?

Q10: What additional information would you consider useful to be included with the datasets in the Climate Data Store (CDS)?

Q11: How likely are you to recommend C3S products and services to others?

• Mailing list • Social media (e.g. Twitter) • C3S User Forum • C3S Newsletter • Other (please specify)

Q12: Do you have any other suggestions on how we can improve C3S for you?

Q13: Please provide your email address if you agree to be contacted for further discussion regarding your user requirements



Q1: Where are you based?

Results The survey participants are distributed across the globe, with the largest numbers in South and East Asia (Figure 1).

36% of participants are based in the European Union and 64% are from outside the EU.

Figure 1: Geographical distribution of all survey participants, where >= 3% of total (total 2099 responses)

From 2018 to 2019 the share of EU-based responses increased significantly from 28% to 36% (Table 1).



2017 2018 2019

EU 384 (30%) 373 (28%) 750 (36%)

Non-EU 914 (70%) 939 (72%) 1352 (64%)

Total 1298 (100%) 1312 (100%) 2102 (100%)

Table 1: Regional distribution of survey participants

Among EU users, most are based in Germany, Italy and the UK. The Eastern European counties again stand out as having very few users (Figure 2).

Figure 2: Geographical distribution of EU-based survey participants (750 responses)



Recommendations As seen in 2018, C3S uptake is relatively high in Western Europe and still much lower in Eastern Europe. Future C3S user uptake activities should again consider targeting specifically users in Eastern Europe.



Q2: Which sector does your organisation belong to?

Results A significant majority of survey participants (70%) work in academia and research, followed by 14% in the public sector, and 11% in the commercial sector (Figure 3). Looking only at EU-based responses we see a more even distribution, with public sector and commercial entities having 18% each (Figure 4). The sectoral distributions show that the C3S products continue to find significant uptake among commercial and public-sector users.

The global results for the commercial sector show an improvement on the previous year’s results (Figure 5). In the EU the user base remains to be mostly academic (Figure 6), albeit with an increase in commercial users from 15% to 18%. At the same time, public sector use remained at around the same level.

Figure 3: Responses by sector, 2019, all responses (2094 responses)

Figure 4: Responses by sector, 2019 EU responses (747 responses)

Figure 5: Responses by sector, 2018, all

responses (1307 responses)

Figure 6: Responses by sector, 2018, EU




Figure 7: Responses by sector, 2017, all


Figure 8: Responses by sector, 2017, EU


Recommendations The strong and increasing interest from the EU private sector in the C3S products over the past 3 years is encouraging and suggests that C3S is increasingly seen as a useful resource in this sector.



Q3: Overall, how satisfied are you with the Copernicus Climate Change Service?

Results Over 83% of survey participants give the service 4 or 5 stars out of 5 (Figure 9), indicating high utility and quality of C3S products and services. At the time of the survey, C3S offered data via the Climate Data Store (CDS); these datasets were predominantly ERA5, and Seasonal Forecasts.

Figure 9: Overall satisfaction, all responses (1084 responses)

The percentage of users rating the C3S as 4 stars or higher was 83% (Figure 9), however EU-based survey participants gave a slightly lower rating (Figure 10). Please see the following sections for further details.

Selected comments:

“It is an essential ingredient in almost any project I work on ”

“We regularly download ERA5 data for wind resource analysis purposes and for wind farm performance analysis. We are very satisfied with these data which correlate

quite well with our on-site wind speed measurement”



Figure 10: Overall satisfaction, EU-based users (387 responses)

Recommendations As in 2018, the results indicate that C3S products and services are considered useful by the targeted user communities. The current service strategy appears to meet users’ needs and should be continued.



Q4: How useful are the following C3S products for you, and how satisfied are you with them?

Results C3S is most widely known for the numerical data it provides, primarily reanalysis data. At the time of the survey the C3S flagship product was the ERA5 dataset.

Users of these datasets are highly satisfied with the products provided (Figure 11).

A total of 53% of participants rate reanalysis data as essential, a figure which is up

from 48% and 40% in 2018 and 2017 respectively. This shows that these data from C3S meet a very significant demand from users.

Figure 11: Utility and satisfaction with products, all responses (1055 responses)



In terms of satisfaction, 86% of users are very satisfied with reanalysis data, consistent with survey results from 2017 and 2018.

For Seasonal Forecasts, the utility rating decreased slightly from 47% to 43% (for the categories “very useful” or “essential”), and satisfaction from 69% to 66% (for the categories “very satisfied” or “extremely satisfied”).

Looking only at responses from EU-based survey participants, the results and the distributions are almost identical to those of the global user base. (Figure 12)

Figure 12: Utility and satisfaction with products and services, EU-based users (374 responses)



Recommendations It was found that most survey participants require reanalysis data (particularly ERA5) and are highly satisfied with the available data. Other important data products have recently been added to the CDS, and work is ongoing to publicise these to a wider audience.

User satisfaction is rated as lower for some C3S products, such as Sectoral Climate Indices. However, where users express dissatisfaction it is mostly about data access mechanisms rather than the data itself.

Selected user comments:

“All this data is wonderful. It is hard to keep pace with what is available..”

“the download and queue-time for era5 data is very long.”

“We appreciate the transparency of CDS with the ERA5 data and its known issues.”



Q5: How useful is, and how satisfied are you with the following aspects of the C3S Service

Results This question relates not to C3S data, but to supporting services like documentation and communications.



Figure 14: Utility and satisfaction with supporting services, all survey participants (938 responses)

As with 2018, the survey participants rate technical criteria like data formats and download mechanisms most important (Figure 14). This is not entirely surprising, since the population of the survey deliberately included only actual data users.

Satisfaction with the supporting services is generally high throughout. However, it should be noted that for some services (such as C3S User Forum, C3S User learning, Climate Bulletins), satisfaction and awareness is relatively low.

Compared to the previous year, awareness and satisfaction of all supporting services increased.

Many of the free text comments submitted on this question highlight performance and availability issues with data access through CDS. A number of these relate to the time taken for the retrieval of data from ECMWF MARS (such as ERA5 model level data), and the desire from users to be able to see the CDS queue and the status of CDS datasets, in order to plan their work effectively. These comments are fed back to the CDS developers in order to improve the service to users.

Looking only at EU-based users (Figure 15), a similar pattern of usefulness and satisfaction is seen, although there is room for improvement (such as with the user awareness of the C3S User Forum).


“All services, APIs, and documentation is good. However, bulk-data requests from CDS are too slow”

“Forum is very helpful”



Figure 15: Utility and satisfaction with supporting services, EU-based survey participants (318 responses)



Recommendations C3S data users have seen improvements in data access, however some performance issues remain. User feedback regarding the CDS should be evaluated with improving the system for users.

For communications and user support services, there is scope for making the services more relevant for users and raising user awareness. In particular, the C3S Forum should be increasingly promoted as a way of engaging with users to help ensure that C3S continues to meet their needs.



Q6: How useful is, and how satisfied are you with the following functions of the Climate Data Store (CDS)?

Results This question relates to current user satisfaction with the CDS infrastructure itself

Figure 16: Usefulness and satisfaction all survey participants (840 responses)



While survey participants are mostly satisfied with the CDS infrastructure (Figure 16), users would like data retrieval through the CDS API to be faster, in particular for the very large data volumes of ERA5 and Seasonal Forecasts.

It should be noted that the CDS API is intended for ‘expert’ users and can be challenging to use for new/non-expert users. Over recent years, Copernicus User Support have provided extensive support to these non-expert users on a 1-1 basis.

Recommendations The CDS data platform provides GUI-driven data access, resolving many usability issues which previously challenged users. However, many datasets remain hosted in external repositories, e.g. ERA5 model level data on MARS and access times can be an issue. Emphasis should be put on speedy access to these external repositories. Also, the ongoing development work on the Toolbox should improve the situation in the future.



Q7: How likely are you to recommend C3S products and services to others? This question was aimed at assessing the success of C3S in providing high quality, relevant datasets to users

Results

Figure 17: User recommendations, all survey participants (973 responses)

It is notable that 96% of respondents would recommend C3S to others, which is a significant endorsement of C3S in the eyes of users.

Recommendations The high percentage of users who would recommend C3S data and service to others indicates the success of C3S thus far. The feedback from users who were ‘Unlikely’/’Very unlikely’ to recommend C3S services could be investigated to see if there were common issues which could be addressed.



Q8: How do you use C3S (e.g. your user story)?

Results This is an open question. Among the survey population (404 responses) there is a strong positive sentiment towards C3S. This is showing that users are deriving actual value from what is provided by C3S.


“I build machine learning models that take weather data as input. CDS provides me with rich datasets for training such models.

“To inform humanitarian decision making ahead of the season

“I needed ERA5 data for a project looking at the marine ecosystem in the subtropical Atlantic. The ERA5 data were very easy to access and download, thank you!.”



Q9: What changes in C3S products and services would have the most beneficial impact on your work?

Results This is an open question. In general, users were positive about C3S, although there were a number of common themes in the comments made by users which are worth noting. In particular,

• the performance of the CDS and Toolbox • simplified data access methods • making the catalogue easier to browse (API for catalogue) • improved spatial and temporal resolution of the datasets

Q10: What additional information would you consider useful to be included with the datasets in the Climate Data Store (CDS)?

Results This is an open question. Survey participants’ answers cover a wide range of additional requirements, ranging from more documentation to online tutorials and additional datasets. No clear picture of user requirements emerges from these results.

A sample of the responses include:

• “Estimates of a file query size in gigabytes.” • “The inclusion of the observations that are assimilated in the reanalysis. This

would allow to explore the observed data that drives the model and understand a bit better the relation between the observations and the model outputs.“

• “More example requests, specifically a model level ERA5 example api request.” • “Clear information more clear about temporal, spatial resolutions, parameters,

etc.”

Regarding this last point, CUS has noted this feedback and is continually reviewing and improving the documentation we provide -and how we make this important information available to users.



Q11: How do you prefer to find out about changes to C3S (e.g. new dataset, system maintenance, etc)?

Results

Figure 18: Preferred method of communication (803 responses)

While the mailing list was seen to be the preferred way for users to find out about changes, although there is a resource implication for CUS in managing these lists and ensuring that the right messages reach the right people, as some users may not be interested in all of the announcements. Also mailing lists can be a ‘one way’ communication channel, which does not encourage users to interact with each other.

Regarding this last point, there was an encouraging level of interest in the new C3S forum. The CUS are keen to develop and promote the C3S Forum further, in order to establish a ‘self-supporting’ C3S user community.

Q12: Do you have any other suggestions on how we can improve C3S for you?

Results This is an open question. A total of 187 users provided a response.

Sample comments from the users include:

• “quick API function to get single point time series text file format (csv). for example: get_time_serie(lat = 45.2, long = 1.2, from = '2010-01-01 00:00', to = '2019-01-01 00:00' features = ['U100', 'V100', 'temp2m'] data_product = 'XXXX' ) I am actually surprise it did not already exist. lots of non 'weather professional' business might need this simple form of data to do some analysis.”



• “The CDS toolbox needs an automated way to download files, e.g. by submiting a processing script via python API and downloading the results, instead of needing to manually click a link.”

• “Catalogue API. Knowing when data is available without having to browse the website manually.”

• “The data download is too slow, need to add more resources.”



Comparison with results from previous C3S User Surveys

Results The graphics below show the comparison of results from the C3S User Survey’s undertaken in 2017, 2018 and 2019 for a number of selected areas.

User satisfaction remains high, despite the rapid increase in the number of datasets being served by the CDS and their increasing complexity. The increase in the number of C3S users responding to the survey is also encouraging (although the percentage of users responding remains at a similar level to 2018). The strategy employed for the user survey is one area we could investigate further, in order to see if we can improve user engagement with the process and retrieve more representative user feedback.



Figure 19: User Satisfaction rating (%)

Figure 20: Number of surveys sent out

Figure 21: Number of user responses (all)

Figure 22: Number of user responses (all, %)

Figure 23: Number of user responses (EU, %)

Figure 24: Number of user responses by country

(EU, %)



Figure 25: Number of user responses by sector

(all, %)

Figure 26: Number of user responses by sector

(EU, %)


climate.copernicus.eu copernicus.eu ecmwf.int

ECMWF Shinfield Park Reading RG2 9AX UK

results of the c3s user satisfaction survey 2019...c3s user satisfaction survey 2019 final 1 . about...

Documents