Data & Analytics Framework: how public sector can profit from its immense asset, data

RAFFAELE LILLO Chief Data Officer@ Digital Transformation Team

—

Who we are and what we do• Digital Transformation Team• Projects

Data & Analytics Framework (DAF)• Goals• Ok, but what is DAF?• Projects• Some Architectural Highlights

Q&A

—

DAF

Digital Transformation Team

The “operating system” of the country: a series of fundamental blocks upon which services for citizens, the Public Administration, and enterprises are built with modern digital products.

Vision

Make public services for citizens and businesses accessible in an easy manner, via a mobile first approach, with reliable, scalable and fault tolerant architectures, based on clearly defined APIs; support the different central and local government departments in making the best and most data driven decisions, thanks to the adoption of big data and machine learning techniques.

Mission

—

● DAF: Data & Analytics F.● ANPR● Security - Resp. Disclosure● SPID● PagoPA● API Ecosystem● E-Procurement● Developer Community● Digital Citizenship● LexDatafication● ...

Projects

● 7 "Comuni": Bari, Firenze, Milano, Palermo, Roma, Torino, Venezia

● Software House in-house: Sogei, InfoCamere, ACI Informatica, IPZS

● PAC: Agenzia Entrate, ISTAT, MISE, MIUR, ANAC, CdC…

● PAL: Roma, Milano, Torino, Firenze...

Partners

DAF: Data & Analytics Framework

Data & Analytics Framework (DAF) is a combination of: ● A Big Data Platform to centralize and store (data lake),

manipulate and standardize (data engine), re-distribute (API & Data Applications) data and insights.

● A Data Team (data scientists + data engineers) which uses and evolve the Big Data Platform to analyze data, create ML models and build data applications and data viz.

Give us data and a platform...

Interoperability (aka Get out of the Silos!)Public data is… public and all PP.AA. should have access to it

Democratizing Data (aka Open Data, API & Data Viz)Data should be open (when legally possible), accessible by anyone (and

anything) and insightful

Data Products (aka deliver value & insights)Machine Learning in interconnected software applications

Crowdsourcing (aka data is everywhere, let’s help us out)Citizens (esp. civic hackers) contribute to the surfacing of knowledge

… and we shall move the PA

Organizational and Managerial ChallengeCentral Data Office and federated analytics teams

Human ResourcesData Scientists & Data Engineers to get knowledge from data

TechnologyThis is the least complicated one, but still fundamental.

Legislative ChallengeBalancing Privacy and Public Interest

Data Driven Policy needs… Data (and Data Scientists)

Introduction of DAF in Piano Triennale 2017-2019DAF is one of the building blocks of the official document setting the strategy for

digitalization of the PA, and signed by the Prime Minister

DAF prototype developmentTD started the development of the platform from scratch around March ‘17, and

released an Alpha version the first week of October ‘17

Experimental phaseWe started working with a selected number of PA to showcase DAF, test it and

listen to PA’s needs so to fine-tune the platform before final release

Institutionalization of DAFIntroduce by law the role of a central data office for the entire PA

Our Strategy

Mission: Data driven decision making in efficient waysSupport PA at all levels to implement informed policies, both ex ante (policy

formulation) and ex post (policy monitoring and fine tuning).

Centralize common & non-domain specific tasksProvide general purpose data platform once and for all, efficiency in standard

data processes, let PA focus on domain specific tasks / analysis

Economy of scope towards a center of excellenceReach proper dimension to develop and acquire expensive and idiosyncratic

capabilities, and share them with all PA

Design and coordinate implementation of Data PoliciesHelp interoperability and usage of state-of-the-art standards and processes in

data management and analysis. Stimulates research and collaboration.

End Goal: Chief Data Office for the PA

High-level Architectural Design

Data Ingestion

Persistence & Offline analysis (analysis, data processing, model training)

Real-Time data processing

Operational DB for API and Data applications

Communication layer - API

(Less) High-Level Architectural Design

Hadoop cluster for distributed persistence and processing

Kubernetes cluster manages dockerized microservices and external applications.

Core Managers: microservices managing core functionalities of DAF

External applications natively integrated in DAF

Unique identity management system, integrated with HDFS

Dataportal - Public Version

https://dataportal.daf.teamdigitale.it

Dataportal - Private Version

https://dataportal-private.daf.teamdigitale.it

Data Sources

People Firms Smart Cities

Fiscality Pension

Education

Healthcare Cultural events

Public Transportation

Corporate events

Energy Traffic

Corporate info

Balance Sheet

Crime

Real estate registry

Machine Learning Based Applications (aka Data Products)Lex Datafication & Citizen Assistant, Fraud Detection, Citizen

Recommendation Engine, Spending Check, Leading Indicators, etc.

Data VisualizationThematic dashboards and infographics for citizens and firms

API for Interoperability and Open DataEasy and standard access to data within PP.AA. and citizens

And much more… The limit is imaginationSmart city, analysis for data driven policy making, etc.

Use Cases (examples)

It… Could… Work!

Raffaele LilloChief Data Officerraffaele@teamdigitale.governo.itTwitter, Medium: @lilloraffa

—

Grazie!Cooperate with us, please :)

Websitehttp://teamdigitale.governo.itForumhttps://forum.italia.it/c/dafTwitter#DatiPubblici #DAFGoogle Group - Open Data & Analyticsdata@teamdigitale.governo.it