Transcript
Page 1: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

ThisprojecthasbeenfundedwiththesupportoftheH2020ProgrammeoftheEuropeanUnionãCopyrightbytheOpenGovIntelligenceConsortium

.

OpenGovIntelligence

FosteringInnovationandCreativityinEuropethroughPublicAdministrationModernizationtowardsSupplyingandExploiting

LinkedOpenStatisticalData

Deliverable3.2

ReportonOpenGovIntelligenceICTtools-firstrelease

Editor(s):

ArkadiuszStasiewicz(NUIG)

MohammadWaqar(NUIG)

ResponsibleOrganisation: NUIG

Version-Status: V1.0

Submissiondate: 31/10/2016

Disseminationlevel: PU

Ref. Ares(2016)6209154 - 31/10/2016

Page 2: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page2of39

DeliverablefactsheetProjectNumber: 693849

ProjectAcronym: OpenGovintelligence

ProjectTitle:Fostering Innovation and Creativity in Europe through PublicAdministrationModernizationtowardsSupplyingandExploitingLinkedOpenStatisticalData

TitleofDeliverable: D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Workpackage: WP3–ICTtoolsdevelopment

Duedateaccordingtocontract: 31/10/1016

Editor(s):ArkadiuszStasiewicz(NUIG)

MohammadWaqar(NUIG)

Contributor(s):

DimitriosZeginis(CERTH)

EvangelosKalampokis(CERTH)

EfthimiosTambouris(CERTH)

KonstantinosTarabanis(CERTH)

ArkadiuszStasiewicz(NUIG)

MohammadWaqar(NUIG)

MohamedAdel(NUIG)

BillRoberts(SWIRRL)

RickMoynihan(SWIRRL)

Reviewer(s): BillRoberts(SWIRRL)

Approvedby: AllPartners

Abstract: This deliverable provides the description of the prototypes

ofsoftware components delivered as a result of the firstOpenGovIntelligenceICTtoolsdevelopmentstage.

Page 3: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page3of39

KeywordList: software,components,linkeddata,statisticaldata,ICTtools

Page 4: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page4of39

Consortium Role Name ShortName Country

1. Coordinator CentreforResearch&Technology-Hellas CERTH Greece

2. R&Dpartner DelftUniversityofTechnology TUDelft Netherlands

3. R&Dpartner NationalUniversityofIreland,Galway NUIG Ireland

4. R&Dpartner TallinnUniversityofTechnology TUT Estonia

5. R&Dpartner ProXMLbvba ProXML Belgium

6. R&Dpartner SwirrlITLimited SWIRRL UnitedKingdom

7. PilotPartner TraffordCouncil TRAF UnitedKingdom

8. PilotPartner FlemishGovernment VLO Belgium

9. PilotPartner MinistryofInteriorandAdministrativeReconstruction MAREG Greece

10. PilotPartner MinistryofEconomicAffairsandCommunication MKM Estonia

11. PilotPartner MarineInstitute MI Ireland

12. PilotPartner PublicInstitutionEnterpriseLithuania EL Lithuania

Page 5: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page5of39

RevisionHistoryVersion Date Revisedby Reason

0.1 30/08/2016 NUIG Firstdraft

0.2 15/09/2016 NUIG Generalupdates

0.3 14/10/2016 NUIG ToolsDescription(draft)

0.4 26/10/2016 NUIG Architecturedetails

0.5 28/10/2016 NUIG Generalupdates

0.6 29/10/2016 NUIG Generalupdates

0.7 30/10/2016 NUIG Addressedreviewcomments

1.0 31/10/2016 CERTH SubmissiontoEC

Statementoforiginality:This deliverable contains original unpublishedwork except where clearly indicated otherwise.Acknowledgement of previously publishedmaterial andof thework of others has beenmadethroughappropriatecitation,quotationorboth.

Page 6: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page6of39

TableofContentsDELIVERABLEFACTSHEET......................................................................................................................2

CONSORTIUM.......................................................................................................................................4

REVISIONHISTORY...............................................................................................................................5

TABLEOFCONTENTS............................................................................................................................6

LISTOFFIGURES...................................................................................................................................7

LISTOFABBREVIATIONS.......................................................................................................................8

EXECUTIVESUMMARY..........................................................................................................................9

1 INTRODUCTION..........................................................................................................................10

1.1 SCOPE...................................................................................................................................................101.2 AUDIENCE..............................................................................................................................................101.3 STRUCTURE............................................................................................................................................10

2 ARCHITECTUREOVERVIEW.........................................................................................................11

2.1 DATAPROVISION.....................................................................................................................................112.2 DATAPLATFORM.....................................................................................................................................142.3 PROCESSLAYER.......................................................................................................................................152.4 SERVICEDESIGN......................................................................................................................................162.5 SERVICEPROVISION..................................................................................................................................182.6 MANAGEMENT........................................................................................................................................18

3 OPENGOVINTELLIGENCEICTTOOLS-FIRSTRELEASE...................................................................20

3.1 JSONAPIFORDATACUBESPECIFICATION...................................................................................................203.2 JSONAPIFORDATACUBEIMPLEMENTATION..............................................................................................213.3 TABLE2QBANDGRAFTER..........................................................................................................................283.4 DATACUBEBUILDER................................................................................................................................303.5 DATACUBEEXPLORER..............................................................................................................................333.6 OLAPBROWSER.....................................................................................................................................36

4 CONCLUSION..............................................................................................................................38

REFERENCES.......................................................................................................................................39

Page 7: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page7of39

ListofFiguresFIGURE 1 OGI ARCHITECTURE ............................................................................................................................... 13FIGURE 2 CUBE BUILDER DESKTOP UI ................................................................................................................... 30FIGURE 3 DATA CUBE EXPLORER - DATASET SELECTION ...................................................................................... 34FIGURE 4 DATA CUBE EXPLORER - SAMPLE VISUALISATION ................................................................................. 34

Page 8: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page8of39

ListofAbbreviationsThefollowingtablepresentstheacronymsusedinthedeliverableinalphabeticalorder.

Abbreviation Description

API ApplicationProgrammingInterface

CSV CommaSeparatedValues

ICT InformationandCommunicationTechnologies

LOSD LinkedOpenStatisticalData

OLAP OnLineAnalyticalProcessing

RDF ResourceDescriptionFramework

UI UserInterface

URI UniformResourceIdentifier

W3C WorldWideWebConsortium

WP WorkPackage

Page 9: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page9of39

ExecutiveSummary

The present document is the deliverable “D3.2 – Report on OpenGovIntelligence ICT tools – firstrelease” (referred to as D3.2). It provides detailed information about the result of the first stageoftheOpenGovIntelligencedevelopment.ThegoalofWP3 (“ICT toolsdevelopment”) istodeveloptheOpenGovIntelligence ICT tools asasuite of open source and commercial tools. ThedevelopedICT toolswill support theOpenGovIntelligence frameworkcreatedatWP2 (“Frameworkcreation”)andwillenable:

(a) thecreationofLinkedOpenStatisticalData(LOSD)fromvarioussources,(b) theexpansionofLOSDwithdatasetsfromexistingsources,(c) theexploitationofLOSDfortheco-productionofpublicservices.

The first release of the OpenGovIntelligence ICT tools cover the tools regarding the creationandexploitation of LOSD. With regards to the creation of LOSD, OpenGovIntelligence designedanddeveloped ICT tools thatenable the transformationofpublicsectordata tostandardmachinereadable forms (specifically toRDF data cubes) and the validation of the generated RDF.Withregards to the exploitation of LOSD, OpenGovIntelligence designed and developed ICT toolsthatenablevisualisationandanalysisofstatisticaldata.

The tools presented in this deliverable will be evaluated during the pilot implementations stage.Basedontheoutcomes,theexistingtoolswillbeimproved.MoreovernewtoolswillbedevelopedinordertocoverremainingfeaturesoftheOpenGovIntelligenceFramework.

Page 10: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page10of39

1 IntroductionThis section introduces the background of thework carried out inWP3 “ICT tools development”.Sub-section 1.1 presents the scope and the objectives of the current document, sub-section 1.2describes the intendedaudience for thisdocument,whilesub-section1.3outlines thestructureofthedocument.

1.1 ScopeThis reportdocuments the ICT toolsdevelopedduring the firstphaseofWP3.Toguide readers inunderstanding the context of these tools, Section 2 presents the high level architecture of theOpenGovIntelligenceFramework,showingtheoverallorganizationof layers,componentsandtheirinteractions. This first release focuses on the development of tools thatwill enable creation andexploitationofLOSD.InsubsequentstagesfurthertoolsareplannedcoveringexpansionofexistingdatasetsandexploitationofLOSDforco-productionofpublicservices.

1.2 AudienceThe intended audience for this document is the OpenGovIntelligence consortium, in particularpartnerorganisationsresponsibleforthedevelopmentofpilottrials,theEuropeanCommission(EC)andthosewhoareinterestedinchallengesandneedsforopening-upandexploitingLOSDfortheco-productionofinnovativedata-drivenservicesongovernments.

1.3 StructureThestructureofthedocumentisasfollows:

• Section2presentstheOpenGovIntelligenceArchitectureandprovidesdescriptionsofeachoftheindividualcomponents.

• Section3providesadetaileddescriptionoftheindividualOpenGovIntelligenceICTtoolsdevelopedduringthefirstdevelopmentstageofOpenGovIntelligenceproject;

• Finally,Section4concludesthereportandoutlinesthefuturedevelopmentplan.

Page 11: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page11of39

2 ArchitectureOverview

TheOGIArchitecture forLOSDanddata-drivenpublicservicesenablesstakeholders tocollaboratetowards the production of innovative data-driven public services by exploiting Linked Open Datatechnologiesandstatisticaldatasets.Thisistheinitialversionofthearchitecturethatwillbetestedand refined during the first round of pilot evaluations. The architecture is presented in Error!Referencesourcenotfound.anditisorganisedasfollows:

• Thearchitectureisdividedintofivelayers:(i)DataProvision,(ii)DataPlatform,(iii)Process• Thearchitectureisdividedintofivelayers:(i)DataProvision,(ii)DataPlatform,(iii)ProcessLayer,(iv)ServiceDesign,and(v)ServiceProvision.

• Keymanagementresponsibilitiesaresharedacrossalllayers.

• Eachlayerhasasetofcomponentsthatperformstasksspecifictothatlayer.

Thearchitecturewillguidethepilot implementations,aswellastheotherfuture implementationsoftheOGIsoftwareinotherprojects.

Eachpartofthearchitectureisdescribedindetailinthefollowingsections.

2.1 DataProvisionTheDataProvisionLayerimplementsfundamentalfunctionalitiesneededtocreatehighqualityLODandthussupporttheexecutionandscalabilityoftheServiceDesign.TheDataProvisionservicesRDFcreation, schema, code lists, vocabularies andmetadatamanagement services in order to enabletheextensibility and scalability and assure the data quality of the proposed OpenGovIntelligenceapproach.

2.1.1 LODCreationTheLinkedOpenDataCreationgroupincludesthefollowingmodules:

2.1.1.1 MetadataManagement

Metadataisessentialtolearnwhatdataisavailableandunderstandandinterpretthestatisticaldata-withoutitthedataismeaningless.

Themetadatacanbecategorisedas:

• Structuralmetadata-codeorconceptlists,dimensiondefinitions,unitsofmeasure,etc.• Referencemetadata-informationonmethodologyandquality• Processmetadata-descriptionofproductionprocesses,orprocessmetrics• Provenance-descriptionofthesourceofinformation,originofthedata,digitalrights,etc.

OpenGovIntelligence metadata is represented as Linked Data, as it will be used as a medium todiscover aggregated cubes that are stored in dispersed data repositories.MetadataManagement

Page 12: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page12of39

willbecarriedoutbymappingtheavailabledescriptive,structuralandadministrativecharacteristicsofthedigesteddatasetusingtheDataCatalogVocabulary(DCAT1)vocabulary,whichisawidespreadW3Cstandardfordescribingdatasets.

2.1.1.2 VocabulariesManagement

Re-use of accepted controlled vocabularies (for example those from W3C, Eurostat or nationalstatisticalorganisations)isoneofthemostimportantprinciplesofLinkedData.Itenablesthelinkingofdifferentcubesattheschemalevel(i.e.measures,dimensions,attributes).Consequently,thereisa need for managing controlled vocabularies that could be reused across different datasets.VocabulariesManagement service will be responsible for creating, storing, searching, discoveringandreusingexistingcontrolledvocabularies.

2.1.1.3 CodeListsManagement

Code Lists are commonly used among RDF data cubes in order to assure unambiguous definitionofthe dimensions within a dataset. Moreover, code lists may have a hierarchical structure.CodeListsManagementservicewillberesponsibleforcreating,storing,searching,discoveringandreusingexistingcodelists.

2.1.1.4 SchemaManagement

Data gathered during the cube building and modelling process using RDF Data Cube vocabularyallows to store information at the schema level (i.e. dimensions, attributes,measures). The reuseofthe vocabularies and the code lists enables linking of disparate cubes. Schema Managementservicewillberesponsibleforcreating,storing,searching,discoveringandreusingexistingschemas.

2.1.1.5 RDFCreation

TheRDFCreationprocess isresponsiblefortransformationofthedata(i.e.publicsectorstatisticaldata) to LinkedData from various existing source formats. This process enables search, discoveryandqueryingofrelevantdata. InparticulartheRDFcreationserviceisresponsiblefordefinitionofthemodel to represent the data, then transforming the data to fit thatmodel. The core of themodelistheRDFDataCubeVocabulary2.

2.1.1.6 CubeBuilder

TheCubeBuilderserviceprovides interfacesthatallowsausertomaprawdatatoRDFDataCubestructure by identifying dimensions, measures and attributes in the raw data. With the supportofother components (e.g. SchemaManagement, Metadata Management) it maps code lists andvocabularies.

1 https://www.w3.org/TR/vocab-dcat/ 2 https://www.w3.org/TR/vocab-data-cube/

Page 13: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page13of39

Figure1OGIArchitecture

Page 14: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page14of39

2.1.2 QualityAssuranceThe Quality Assurance group is a set of activities, standards and processes that will ensuresthevalidation aswell as the high quality of the generated LOSD and theirmetadata and includesthefollowingmodules:

2.1.2.1 Validation

Validation component is responsible for the validation of data cubes. It achieves this by verifyingtheschema requirements and detection of incorrect or missing values. Validation is performedonbothdatacubeschemamodelsandcreateddatacubeobservations.

2.1.2.2 QualityAssurance

Quality Assurance component is responsible for ensuring the quality of data and associatedmetadata.Thecomponentwilluseseriesofteststhatcanbeapplied inordertocheckthequalityofthedatasetandflagthe information.Thisprocess tracksqualityofdatacubesandensuresthatonlyhighqualitydataismaintainedinsidethesystem.

2.2 DataPlatformThe Data Platform Layer is responsible for the data storage and provides a catalogue service foravailable datasets that lets userssearch for aggregated cubes overmany distributed repositories,basedontheirmetadata.Itconsistsofthefollowingdatadrivenprocesses:

2.2.1 CubeCatalogueTheDataCubeCatalogueprovidesan interface thatenablesbrowsingandexplorationof thedatacubesbasedonmetadataassociatedwiththedatacubes.Itlistsavailabledatasetsandtheusercansearchandfilterresultsbasedonkeywords,tagsandcategoriesetc.

2.2.2 CubeDiscoveryCube Discovery is the starting point for the services like Visualisation, Analysis, Mashups, etc.Itallowsuserstonavigatethroughthedatasetsandexplorethecontentsofthedatasets.

2.2.3 QueryingThe Querying Service provides a user interface for writing and running SPARQL queries andpreviewingtheresults.Additional featuresmight includesyntaxhighlighting,acatalogueofstoredqueries,autocompleting,etc.

2.2.4 SearchingTheSearchingcomponentprovidesa setof simpleandadvancedsearchand indexingcapabilities,lookingatthedata itself,not just itsmetadata.Searchservice isessentialforfindingrelevantdatainlargerepositoryofdatasets.

Page 15: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page15of39

2.2.5 LinkingTheLinkingcomponenthelpsinlinkingdatatointernalandexternallinkeddataresources.

2.2.6 StorageThe Storage component is responsible for persistence of the data and information. In particularitwill store Data cubes outputted from the Data Provision Layer, mashups, outputsoftheenrichmentprocesses,analyticsresults,indexedpartsoftheRDF,metadata,userprofiles,etc.

2.2.7 LinkedStatisticalDataCloudLinked Statistical Data represents the Linked Open Data available on the web (externaltotheStoragecomponent).

2.3 ProcessLayerTheProcessLayerallowsanendusertointeractwiththesystem.Itconsistsofthefollowing:

2.3.1 BrowsingThe Browsing component enables navigation through available data cubes, data cubes metadataandexaminationofthedatacubeobservations'values.

2.3.2 VisualizationsThe Visualization component involves meaningful interactive visualizations from the availabledatasets.

2.3.3 ExpansionTheExpansioncomponentenablestheprocessofexpandingadatacubewithcompatibledatafromelsewhere.Ifthecompatibiityrulesaresatisfied,thiscanbedoneinthefollowingways:

• CubeExpansionbyaddingadditionalmeasures,• CubeExpansionbyaddingadditionalvaluetoselecteddimension.

The linking ofmeasures of aggregated cubes results in larger cubes describing the samemeasurewhile the linking of the dimensions of different aggregated cubes facilitates data analyticsandvisualizationservices.

2.3.4 Mashups

TheMashups component let users combine data frommultiple sources and create new servicesdisplayedinasinglegraphicaluserinterface.

Page 16: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page16of39

2.3.5 StatisticalAnalysisStatisticalAnalysismodule is responsible for rangeof statistical analysis functions and algorithms.Itrangesfrombasic(e.g.standarddeviationofameasurevaluesagainstadimension)toadvanced(e.g.anomalydetectionusingmultiplemethodologies)methodswhichusercanchoosefrom.

2.3.6 MachineLearning

This component will use machine learning algorithms in order to mine and extract meaningfulinformationaboutthedataandhiddenpatternsaswellasintroduceintuitivevisualisations.

2.3.7 Enrichment

TheEnrichmentcomponentwillenablelinkingofdatacubeswithexternalresources(e.g.DBpedia)and combination of data across statistical and non-statistical datasets. Thisprocess will allowtoperformaricherdataanalysisandenhancedvisualizations.

2.3.8 AggregationThis component will compute aggregations across dimensions and hierarchies in selected datacubes. Aggregated values will be used for visualization, statistical analysis and machine learningpurposes,thusenablingenhancedoperations(e.g.roll-up/drill-down,dimensionreductionetc.).

2.4 ServiceDesignService Design implements the main functionalities offered by OpenGovIntelligence and enabletheco-designofdata-drivenpublicservices.Itconsistsofthefollowing:

2.4.1 ServiceDesignTheServiceDesignGroupofcomponentshelpsstakeholderswithproperICTskillstobeconnectedtodatasourcesinordertodesignnewservicesthatattemptstoaddresspublicneeds.Itincludesthefollowingmodules:

2.4.1.1 IdeaCreation

The Idea creation component will support workflow for idea creation process aligned withOpenGovIntelligence framework. Platform will allow involvement of more than one stakeholdergroupinthecreationprocessandsupportitbycombiningcomponentsavailableintheProcesslayerandtheDataPlatformlayer.

2.4.1.2 Prototyping

Prototyping enables rapid creation of the services prototypes and its dissemination toshowcaseproposedsolutions.

Page 17: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page17of39

2.4.1.3 Evaluation

The Evaluation module allow to validate designed prototypes, collect feedback, improvementsuggestionsandadditionalfeaturesrequests.

2.4.1.4 Deployment

The Deployment component is responsible for the publication of the developed and validatedsolution.

2.4.2 CollaborationSpaceThiselementwillhelptoovercomespaceandtimedifferentialsandfacilitatetheco-designofdata-drivenpublicservicesusingelectroniccommunicationsandgroupwarebystakeholderscomingfromanycountryand/ortimezone.

2.4.2.1 Feedback

Feedback involvesmechanisms thatwill beused to collect andmanage theneedsof the citizens,businessesandpublicauthoritiesregardingtheinnovativedata-drivenpublicservices.

ItwillincludemechanismsandtoolsthatwillallowtheOpenGovIntelligencestakeholderstoprovidecomments(positiveornegative),preferences,propositionsand/orneedsregarding:

• datasourceusedbytheservice(i.e.enhancementofLOSDwithnewdata,updatedversionofthedataetc.).

• exploitation methods adopted in the co-produced public service (i.e. new types of dataanalysesorvisualizationthatisnotsupportedbyaprovidedservice)

• newservices.Forexampleacompanycouldaskfortheco-productionofanewinnovativeservicethatwillserviceitsneeds.

2.4.2.2 Discussion

Discussioncomponentenablesstakeholders(co-designersoftheservices)tosharethoughts,discussnew ideas, tobrainstorminsessions,chatandshareassets inordertodesignnewworkflowsthatcouldgeneratenewservices.

2.4.2.3 Rankings

Rankingscomponentletsstakeholderstorateandranktheservicesbasedoni.e.usabilityandvalueadded.

2.4.2.4 Exploration

This module enables exploration of the functionalities offered by OpenGovIntelligence platform,availableservices,workflowsandreusablecomponents.

Page 18: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page18of39

2.5 ServiceProvisionTheServiceProvisionlayerimplementstheco-provisionofthepublicservicesthatwerepreviouslyco-designedintheServiceDesignlayer.Atthislayerallmodulesareconcernedwithenduseroftheplatformservices.

2.5.1 ServiceProvision

2.5.1.1 ServicesCatalogue

ServicesCatalogueenablesdiscoveryoftheservicesdeployedattheplatform. Itwillprovideasetofsimple and advanced features for finding single service or group of services (e.g. by categoryorbytypeoftheservice).

2.5.2 ServiceImplementation

TheServiceImplementationGroupincludesthefollowingmodules:

2.5.2.1 ServiceImplementation

This module allows end users of the platform to adopt and implement selected service usingOpenGovIntelligenceplatform.

2.5.2.2 ServiceLiveness

Themainresponsibilityofthismoduleismonitoringofthedeployedservice,providingperformanceandusagemetricsandgeneratingreports(i.e.errorsandavailability)formaintenancepurposes.

2.5.2.3 FeedbackCollection

Thiscomponentprovidesmechanismforcollectionof thefeedbackfromendusersof theservicesprovidedbyOpenGovIntelligenceplatform.Theoutcomesmaybeusedforuserexperienceanalysisandservicesevaluationforcontinuity,enhancementsoreliminationdecisions.

2.5.2.4 ServiceEvaluation

Service Evaluation component will evaluate deployed services based on multiple analysis factors(e.g. service liveness and feedback) in scheduled periods and provide factors, that may helptodecidewhethertocontinue,enhanceorwithdrawparticularservice.

2.6 ManagementThe Management Layer provides cross-platform functionalities such as logging, configuration,security, API etc. It is responsible for overall management and smooth functioningoftheOpenGovIntelligenceplatform.

Page 19: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page19of39

2.6.1 ExternalToolsOpenGovIntelligence platform provides APIs to external tools (i.e. data processing pipelinesorexternalvisualisationlibraries)toletthemuserintegrateOGIcapabilities.

2.6.2 Monitoring

This component will provide information about liveness of the platform, usage performance,numberofuseroftheplatformandsoon.

2.6.3 LoggingSystemlogswillbeusedfortrackingplatformperformanceandresolveordebugissuesifoccurred.

2.6.4 SecurityThis component role is to ensure general security of the platform and provides authentication,authorizationandusermanagementcapabilities.

2.6.5 SystemConfiguration

Systemconfigurationisresponsibleforthemanagementofconfigurationdata.

Note: OpenGovIntelligence project will not develop software tools for every componentofthearchitecture or an integrated system thatmatches this architecture as awhole.Moreover,somepartsofthearchitecturemightendupbeinghumanprocessesandsomeotherinstantiationsof someother components-tools. Existing tools available for the purposewill re-usedwhere everpossible and new tools will be developed when required. We have already identified anumberoftools that can be re-used for example in collaboration space. This list of tools is includedindeliverableD2.1–OpenGovIntelligenceframework–firstrelease.

Page 20: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page20of39

3 OpenGovIntelligenceICTtools-firstreleaseThis section provides a detailed description of the individual OpenGovIntelligence ICT toolsdevelopedduringthefirstdevelopmentstageoftheOpenGovIntelligenceproject.

Wedescribethefunctionalityandimplementationofeachofthetools.

In the first implementation stage, we have developed tools supporting data conversionandexploitationof thedata. Thenext releaseofOpenGovIntelligencewill includeadditional toolsandwillsupportalargerpartoftheOpenGovIntelligenceframework.

3.1 JSONAPIforDataCubespecificationTheJSON-QBAPIisastandardisedwayfordatauserstoprogrammaticallyselectandretrievedatafromamultidimensionalstatisticaldatacube.Itisdesignedprimarilyforusebydeveloperswithoutspecialist knowledge of statistics or specialist knowledge of the underlying RDF Data Cuberepresentation. It supplements existing more complex ways of accessing data, such as viatheSPARQL query language. The objective is to increase the ease of use of statistical datainvisualisationsandapplications.

The OpenGovIntelligence is developing an openly licensed specification and implementationoftheAPI. The longer term aim is that this could be offered by awide range of data publishersasamethodofexploitingtheirdataandsocreatinggreater impact. Datapublisherscouldchoosetouse theOpenGovIntelligence implementation,orcouldcreate theirownsoftware thatcomplieswith the API specification, perhaps taking advantage of particular features of their own dataplatform.

3.1.1 FunctionalityDescriptionTheJSON-QBAPIprovidesprogrammaticaccess todataheld in theopenstandardRDFDataCubeformat,returningsubsetsofthedatainaformthatiseasyformostprogrammerstoworkwith.

3.1.2 ImplementationDescription

Therequirementsforthetoolaretoprovideaccesstomultidimensionalstatisticaldatainawaythatisfamiliartoabroadgroupofwebdevelopers,inparticulartomakeiteasytocreatecharts,mapsand other visualisations. The design favours simplicity over completeness. As well as being easytouse, it should be relatively easy for data publishers to implement, and so have themaximumchanceofbecomingwidelyusedstandard.

3.1.3 AvailabilityThespecificationcanbeaccessedathttps://github.com/OpenGovIntelligence/json-qb.

Page 21: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page21of39

3.1.4 LicenseThelicenceofthespecificationisCC-BY4.0https://creativecommons.org/licenses/by/4.0/

3.1.5 Futuredevelopmentplan

TheAPI is initsfirstexperimentalversion. Feedbackwillbegatheredfromusersofdataandusedtoenhancethedesignandimplementation.

3.1.6 RoleinthearchitectureJSONAPIforDataCubespecificationcanberepresentedasQueryingcomponentinDataPlatformLayer.

3.2 JSONAPIforDataCubeimplementationThiscomponentistheimplementationoftheJSONAPIforDataCubespecificationprovidedbytheOpenGovIntelligence project. It aims to provide an easy to use API for web developers that usestatisticaldatastoredintheformofRDFDatacubes.TheAPIimplementationcanbeinstalledontopofanyRDFrepositoryandofferbasicandadvancedoperationsonRDFDatacubesassuming that:i)theyarestoredusing theRDFDataCubeVocabulary, ii) they followaspecificapplicationprofile(common practices) and iii) are accessible through a SPARQL endpoint. The results are alwaysreturnedinJSONformat.

3.2.1 FunctionalityDescriptionTheAPI implementationoffers functionalityusedbyother components through simpleRESTcalls.Inthe next tables we describe each function. Specifically, we provide the name of the function,ashort description, the URI (including the REST method used e.g. GET), the parameters thatthefunction requires as input, a sample JSON response and the SPARQL query that is executedtogettherequiredresults.

Name Getallavailablecubes

Description ReturnsalltheavailablecubesofanRDFrepository.

URI GET/cubes

Parameters -

Response {"cubes":[{"URI":"http://example.com/cube/unemployment",

Page 22: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page22of39

"label":"Unemploymentrate"},{"URI":"http://example.com/cube/population","label":"Population"}....]}

SPARQLquery

PREFIXqb:http://purl.org/linked-data/cube#PREFIXskos:http://www.w3.org/2004/02/skos/core#selectdistinct?cube?labelwhere{?cuberdf:typeqb:DataSetOPTIONAL{?cubeskos:prefLabel|rdfs:label?label.}}

Name Getdimensions

Description Returnsallthedimensionsofacube

URI GET/dimensions

Parameters dataset:theURIofthecube

Response {"dimensions":[ {"URI":"http://example.com/dimension/refArea", "label":"Reference area" }, {"URI":"http://example.com/dimension/refPeriod", "label":"Reference period" }.... ] }

SPARQLquery

PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select distinct ?dim ?label where { <http://example.com/cube/unemployment> qb:structure ?dsd. ?dsd qb:component ?cs. ?cs qb:dimension ?dim. OPTIONAL{?dim skos:prefLabel|rdfs:label ?label.}}

Name Getmeasures

Description Returnsallthemeasuresofacube

Page 23: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page23of39

URI GET/measures

Parameters dataset:theURIofthecube

Response {"measures":[ {"URI":"http://example.com/measure/employmentRate", "label":" Employment Rate " }, {"URI":"http://example.com/measure/unemploymentRate", "label":" Unemployment Rate” }.... ] }

SPARQLquery

PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select distinct ?measure ?label where { < http://example.com/cube/unemployment> qb:structure ?dsd. ?dsd qb:component ?cs. ?cs qb:measure ?measure. OPTIONAL{?measure skos:prefLabel|rdfs:label ?label.}}

Name Getattributes

Description Returnsalltheattributesofacube

URI GET/attributes

Parameters dataset:theURIofthecube

Response {"attributes":[ {"URI":"http://example.com/attribute/unit", "label":"Unit of measure" }, {"URI":"http://example.com/attribute/obsStatus", "label":"Observation status" }.... ] }

SPARQLquery

PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select distinct ?attribute ?label where { < http://example.com/cube/unemployment> qb:structure ?dsd. ?dsd qb:component ?cs. ?cs qb:attribute ?attribute.

Page 24: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page24of39

OPTIONAL{?attribute skos:prefLabel|rdfs:label ?label.}}

Name Getdimensionvalues

Description Returnsallthevaluesofadimensionthatappearataspecificcube

URI GET/dimension-values

Parameters dataset:theURIofthecube

dimension:theURIofthedimension

Response {"values":[

{"URI":"http://example.com/value/female", "label":"Female" }, {"URI":"http://example.com/value/male", "label":"Male" }.... ] }

SPARQLquery

PREFIX qb: http://purl.org/linked-data/cube#

PREFIX skos: http://www.w3.org/2004/02/skos/core#

select distinct ?val ?label where {

?observation qb:dataSet <http://example.com/cube/unemployment >.

?observation < http://example.com/dimension/sex> ?val.

OPTIONAL{?val skos:prefLabel|rdfs:label ?label}}

Name Getattributevalues

Description Returnsallthevaluesofanattributethatappearataspecificcube

URI GET/attribute-values

Parameters dataset:theURIofthecubeattribute:theURIofthedimension

Page 25: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page25of39

Response {"values":[ {"URI":"http://example.com/value/count", "label":"Count" }, {"URI":"http://example.com/value/ratio", "label":"Ratio" }.... ] }

SPARQLquery

PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select distinct ?val ?label where { ?observation qb:dataSet <http://example.com/cube/unemployment >. ?observation <http://example.com/attribute/unit> ?val. OPTIONAL{?val skos:prefLabel|rdfs:label ?label}}

Name Getslice

Description Returnsasetofobservationsofcubethatmatchtoparticularvaluesofselecteddimensions

URI GET/slice

Parameters dataset:theURIofthecube0ormoredimensionidentifiersandtheallowedvaluese.g. http://example.com/dimension/sex =http://example.com/value/female

Response {"observations":[ {"URI":"http://example.com/observation/1", "http://example.com/dimension/refArea ":" http://example.com/Greece", "http://example.com/dimension/refPeriod ":" http://example.com/2015", "http://example.com/measure/unemploymentRate": 0.24 }, {"URI":"http://example.com/observation/2", "http://example.com/dimension/refArea ":"http://example.com/Greece", "http://example.com/dimension/refPeriod ":"http://example.com/2016", "http://example.com/measure/unemploymentRate": 0.26

Page 26: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page26of39

},.... ] }

SPARQLquery PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select ?obs ?refArea ?refPeriod ?unemployment where { ?obs qb:dataSet <http://example.com/cube/unemployment >. ?obs <http://example.com/dimension/sex> <http://example.com/value/female>. ?obs <http://example.com/dimension/refArea> ?refArea. ?obs <http://example.com/dimension/refPeriod> ?refPeriod. ?obs <http://example.com/measure/unemploymentRate> ?unemployment. }

Name Gettable

Description Returns a 2D table representation of the cube’s observations that match toparticularvaluesofselecteddimensions

URI GET/slice

Parameters dataset:theURIofthecubecol:theURIofthedimensionusedforthecolumnsofthe2Dtablerow:theURIofthedimensionusedforrowsofthe2Dtable0ormoredimensionidentifiersandtheallowedvaluese.g. http://example.com/dimension/sex =http://example.com/value/female

Response The response is in json-stat format e.g. {"class" : "dataset", "id" : ["http://example.com/dimension/refArea","http://example.com/dimension/refPeriod"], "size" : [10,5], "dimension" : { "http://example.com/dimension/refArea" : { "category" : { "index" : ["http://example.com/Greece",

Page 27: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page27of39

"http://example.com/UK",…] }}, "http://example.com/dimension/refPeriod" : { "category" : { "index" : ["http://example.com/2015", " http://example.com/2016",…] } }…}, "value" : ["0.24", "0.26" ….] //values are stored in row-major order }

SPARQLquery PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select ?unemployment where { ?obs qb:dataSet <http://example.com/cube/unemployment >. ?obs <http://example.com/dimension/sex> <http://example.com/value/female>. ?obs <http://example.com/dimension/refArea> ?refArea. ?obs <http://example.com/dimension/refArea> ?refPeriod. ?obs <http://example.com/measure/unemploymentRate> ?unemployment. } ORDER BY ?refArea ?refPeriod

3.2.2 ImplementationDescription

TheimplementationoftheAPIusesthefollowingtechnologies:

1. Jersey3frameworkfortheimplementationoftheRESTfulservices.JerseyisanimplementationoftheJAX-RS4ReferenceImplementation.

2. SPARQL5forqueryingtheRDFrepositories.3. Rdf4j6JavaframeworkforprocessingRDFdata4. Json-statjavalibrary7tocreateandconsumejson-statdata5. Gson8JavalibrarytoserializeJavaObjectsintotheirJSONrepresentation

3 https://jersey.java.net/ 4 https://jax-rs-spec.java.net/ 5 https://www.w3.org/TR/rdf-sparql-query/ 6 http://rdf4j.org/ 7 https://github.com/statisticsnorway/json-stat.java 8 https://github.com/google/gson

Page 28: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page28of39

3.2.3 AvailabilitySource code is available at GitHub: https://github.com/OpenGovIntelligence/json-qb-api-implementation

3.2.4 LicenseThesoftwareisavailableasopensourceundertheApacheLicense2.0.

3.2.5 Futuredevelopmentplan

Theplansfornextyeardevelopmentinclude:

1. Support pagination of results. In the case of big cubes this will enable the limiting ofreturnedresult.

2. Support advanced filtering of dimension values. For example, enable filtering by ranges,patterns,filteringbygeographyetc.

3. Supportmultilinguality. In somecases, cubescontain labels inmultiple languages.TheAPIshouldbeextendedtosupportthem.

4. Support hierarchical data. In some cases, dimensions have hierarchical structure. The APIshouldbeextendedtosupporthierarchicaldata.

3.2.6 RoleinthearchitectureJSON API for Data Cube implementation can be represented as Querying component in DataPlatformLayer.

3.3 Table2qbandGrafterGrafter is an open source software library to support Extract Transform Load processes, withaparticular emphasis on transforming tabular data to RDF. Its development has been supportedbytheEUFP7DaPaaSandOpenCubeprojects,andfurther improvementshavebeenimplementedas part of OpenGovIntelligence. These include bug fixes, performance improvements and bettererrormessages.

Grafter has been used to implement a process to convert statistical data to RDFData Cube fromatabular inputfile inCSVformat,thatfollowsaparticulartemplate. Wehavecalledthissoftware‘Table2qb’(pronounced‘tabletocube’).

The input template has one statistical observation per row, with a column for each dimension,attribute and measure. Column headers and cell contents are mapped to appropriate URIsforrepresenting thedata inRDFDataCube. The format is designed tobe simple for statisticianstogenerate,withouttheneedforprogrammingskillsandwithouttheneedforknowledgeofRDF.

Page 29: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page29of39

WorkisinprogresstomakethemappingfromtablecontentstoURIsmoreflexibleandconfigurable,toimprovethevalidationofuserinputsandtodefineastandardwayofassociatingmetadatawiththeinputfile.

3.3.1 FunctionalityDescriptionGrafterprovidesabasisforawiderangeofdatatransformationtasks,suchascleaningtabulardataandtransformingitintoRDF.

The Table2qb tool, implemented with Grafter, takes data in a specific tabular structure, eitherasaCSV or Excel file, and converts it into an RDF Data Cube: representing the data as a seriesofobservations with dimensions, attributes and measures, and generating the associated DataStructureDefinition.

3.3.2 ImplementationDescription

Grafter is implemented using the Clojure programming language. Clojure is a functionalprogramming language that runson the JavaVirtualMachine. Thismakes itefficient,able to runonallcommonoperatingsystems,andabletomakeuseofthelargerangeofexistingJavalibraries.

Table2qb is implemented using Clojure, and the Grafter library in particular. The prime designcriterion forTable2qb is that it shouldbesimple fornon-programmers togenerate the inputdataintherequiredformat,forexamplebyusingExcelorotherspreadsheetsoftware.

3.3.3 AvailabilityGrafter can be obtained from https://github.com/Swirrl/grafter. Table2qb is also on GitHubathttps://github.com/OpenGovIntelligence/table2qb.

3.3.4 LicenseThesoftwareisavailableasopensourceundertheEclipsePublicLicense.

3.3.5 Futuredevelopmentplan

FurtherworkonboththeTable2qbtoolandthesupportingGrafterlibraryisplannedoverthenextyear.ThiswillinvolvetestingandevaluatingTable2qbwitharangeofusersandarangeofdatasets.Thelessonslearnedwillbeusedtoimprovetheeaseofuseandflexibility.Itisalreadyknownthatimprovedvalidationanderrorreportingisahighpriorityareatoworkon.

3.3.6 RoleinthearchitectureTable2qb and Grafter can be represented as both: RDF Creation and Cube Builder componentsinDataProvisionLayer.

Page 30: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page30of39

3.4 DataCubeBuilderDataCubeBuildingenablesRDFcreationbythetransformationofdatasourcesinnon-RDFformatstoRDFDataCubes(inparticularconversionoftheCSVfiles).

CreatedDataCubesmaybestoredinthelinkeddataspaceinordertolinkandenrichthedatacubeswithexternallinkedknowledgebasedonpre-selectedlinkabledimensions.

RDFDataCubecreationprocessisusingthecorrespondingschemastructure.

3.4.1 FunctionalityDescription DataCubeBuildercanbeusedthroughthefollowinginterfaces:(A)DesktopUI,(B)CommandLine,(C)WebApplicationandas(D)WebService.

Application user is configuring the conversion process by passing required arguments throughthedesired interface. Data Cube Builder will output the RDF Data Cube regarding the argumentspassed.

3.4.2 ImplementationDescription

DataCubeBuildingfunctionalitiesarewritteninJavaandareextendingTarql(statisticalobservationcapturing tool) by adding the corresponding cube schema. At this stage tool support data cubeschemasfortheMarineInstitutepilotdatasets.Jframe,JcommanderandSparkjavaareusedtoofferinterfacesfortheCubeBuilder.

Figure2CubeBuilderDesktopUI

Page 31: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page31of39

(B)CommandLine,

Usingthefollowingsetofarguments:

Arguments -Usage:OGIEU[options]Options:--csvFilePath,-csvCSVinputfileLocationandname--dimOrMeasures,-lCustomizedDimandMeasuresareNotAvailableatthisstage--help,-help,-hHelpDefault:false--marineDatasetName,-schemaDataSetSchemaCurrentlySupporting(IWaveBNetwork30Min,IMI_EATL_WAVE,IrishNationalTideGaugeNetworkandIWBNetwork)--qbFileName,-qbNCubeoutputfileLocationandname--qbPath,-qbCubeoutputfileLocationandname--run,-rWhichmainclasstoRun!(cmd,webserviceordesktop)--serialization,-formatOutputCubeRDFserlizationformat(turtleorntriples)Default:turtle

Example $mvn exec:java -Dexec.args="-run:cmd -csv:src/main/resources/data/IWBNetwork.csv -schema:IWBNetwork -format:turtle -qb:src/main/resources/output/ -qbN:IWBNetwork.ttl"

(D)WebService.

Arguments csv=inputFileNameAndLocation

schema=marineInstituteDatasetId

serialization=turtle

qbPath=outputFileLocation

qbName=outputFileName

Page 32: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page32of39

Example /cubeBuilderAPI/cubeBuilderArgs?csv=inputFileNameAndLocation&schema=marineInstituteDatasetId&serialization=turtle&qbPath=outputFileLocation&qbName=outputFileName

SampledatatransformationoutputfromrowdatasetsinCSVformattoRDFcubes:

Dataset IrishNationalTideGaugeNetwork.csv

CSVstructure

longitude,latitude,altitude,time,station_id,Water_Level,Water_Level_LAT,Water_Level_OD_Malin,QC_Flag

degrees_east,degrees_north,m,UTC,,m,m,m,

-6.0683,53.3915,0.0,2016-08-09T00:00:00Z,HowthHarbour,2.902,2.902,-0.06,0.0

Cubiqstructureofobservations

OGI:observations_51fa99a5-cb7b-4f76-afc2-a4016f9da8f0

rdf:typeqb:Observation;

OGI:longitude"-6.0683";

OGI:latitude"53.3915";

OGI:altitude"0.0";

OGI:time"2016-08-09T00:00:00Z";

OGI:year"2016";

OGI:month"2016-08";

OGI:day"2016-08-09";

OGI:station_id"HowthHarbour";

OGI:water_Level"2.902";

OGI:water_Level_LAT"2.902";

OGI:water_Level_OD_Malin"-0.06";

OGI:qC_Flag"0.0";

qb:dataSetOGI:IrishNatoinalTideGaugeNetwork_ds.

Page 33: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page33of39

3.4.3 Availability Thesoftwarecanbeobtainedfromhttps://github.com/OpenGovIntelligence/data-cube-builder

3.4.4 License ThesoftwareisavailableasopensourceundertheApacheLicense2.0.

3.4.5 Futuredevelopmentplan The next development phase of Data Cube Builderwill involve development of schema for otherpilotspartners(e.g.LithuanianPilot)andfacilityofcustomschemausage.

Moreover,supportforotherrawdatasourcesformatstobedigestedbytheDataCubeBuildereg.JSONandTSVisplanned.

3.4.6 RoleinthearchitectureData Cube Builder can be represented as both: RDF Creation and Cube Builder componentsinDataProvisionLayer.

3.5 DataCubeExplorerDataCubeExplorer is responsible for listing,discovering,analysingandvisualizationdata fromthecubes. Itseversas integrateduser interfacefortheOGIplatform.Thetool isofferingcustomizablestatisticalanalysisandmachinelearningservicesusingDataCubesRDFasinputdata.

3.5.1 FunctionalityDescriptionExploration,DataCubeExplorationandVisualizationareservedthroughwebapplicationbuildwithDjango9(High-LevelPythonWebFramework).

API, Receiving multiple kinds of requests eg. Data Cube Creation, and Data Cube Queries thenroutingthoserequeststotheresponsiblemoduleforprocessing.

3.5.2 ImplementationDescription

Data Cube exploration uses pivottable.js visualization and for computing summary of dataset.ItiswritteninJavaScriptandDjangoframeworkforserversideoperations.

Userisablecanchooseoneoftheavailabledatasetandselectvisualisationtemplate(i.e.barchart)andconfigureitbyselectingdimensionsandfunctions(i.e.aggregationorsum).

9 https://www.djangoproject.com/

Page 34: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page34of39

Figure3DataCubeExplorer-Datasetselection

Figure4DataCubeExplorer-SampleVisualisation

Inthecurrentversionofthetool,additionalAPIwasdevelopedinordertoquerythedataseore.

Type Description FunctionName

PresetQueries ListavailableLinkedCubes

JSONArray LqbQueryingForLqbSpaces(String limit)

List Linked Cubemetadata

JSONArray LqbQueryingForDimAndMeasures(String marineDatasetURI)

Page 35: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page35of39

Retrieve Data ofcertainLinkedCube

JSONArray LqbQueryingForLqbData(String marineDatasetURI,String limit)

Customized/ on-demandQueries

Send Sparql QueryoverLinkedCubes

JSONArray LqbDirectQuerying(String sparqlQuery)

DataCubeAPIfunctionalities,currentlyavailableforDataCubeExplorationAPIs:

3.5.3 AvailabilityThesoftwareisavailableathttps://github.com/OpenGovIntelligence/data-cube-explorer

3.5.4 LicenseThesoftwareisavailableasopensourceundertheApacheLicense2.0.

3.5.5 Futuredevelopmentplan

Next step in the tool development will enable the usages of statistical and machine learningfunctionsandalgorithmsover thedatacubedimensionsandattributesvalues (starting frombasic

APIlink Description

listingavailablecubes

/cubeQueryingAPI/listLqbs?limit=numberOfRetrievedRecords

retrievecubemetadata

/cubeQueryingAPI/LqbMeta?dsuri=marineInstituteDatasetURI

retrievedataofcertaincube

/cubeQueryingAPI/listdataofLqb?dsuri=marineInstituteDatasetURI &limit=numberOfRetrievedRecords

sparqlendpoint /cubeQueryingAPI/cubeQueryingArgs ?query=sparqlQueryToExecuteAgainstLinkedCubeSpace

Page 36: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page36of39

statisticalexplorationfunctionstoamorecomplexmachinelearningpredictivealgorithms).WEKA10libraryisacandidateforprovidingthedataminingcapabilities.

Useof JSONAPI forDataCube as communication channelwithdata store is oneof theprioritiesinthedevelopmentcycle.

3.5.6 RoleinthearchitectureData Cube Explorer can be represented as multiple components inProcess Layer: Browsing,VisualisationandStatisticalAnalysiscomponents.

3.6 OLAPBrowserThe OLAP Browser enables the exploration of RDF data cubes by presenting each time a two-dimensional slice of the cube as a table. The OLAP Browser is based on the “OpenCube OLAPBrowser”11. “OpenCube OLAP Browser” was developed as a widget of Information Workbenchplatform12, in thisnew implementationweaimtodisengage fromthe limitations that InformationWorkbench imposed and implement a more powerful and user friendly standalone tool.ThelimitationsthatInformationWorkbenchincludethe“poor”userinterface,badperformance,useofspecificRDFrepositories,bundledwithInformationWorkbenchetc.

TheOLAPBrowserbuildsuponthefunctionalityofferedbytheJSONQBRESTAPI implementation.Specifically,itcallstheAPIfunctionsandvisualizesthereturnedJSONresults.OLAPBrowsersupportvisualizationofdatarepresentedinjson-stat13format

3.6.1 FunctionalityDescriptionCurrentlytheOLAPBrowsersupportsthefollowingfunctionalities:

1. Listall theavailablecubesofanRDFrepository.Makeuseof theGET /cubes functionprovidedbytheJSONQBRESTAPI.

2. Presentthestructure(measures,dimension,attributes)ofaselectedcube.MakeuseoftheGET /dimensions,GET /measuresandGET /attributes functionsoftheJSONQBRESTAPI.

3. Enabletheselectionofthedimensions(rowsandcolumns)ofthe2Dtabletobevisualized.

10 http://www.cs.waikato.ac.nz/ml/weka/ 11 http://opencube-toolkit.eu/opencube-olap-browser/ 12 https://www.fluidops.com/en/products/information_workbench/ 13 https://json-stat.org/

Page 37: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page37of39

4. Enable filtering of fixed dimensions/attributes (i.e. the dimensions/attributes of the cubethat are not shown in the 2D table).Make use of theGET /dimension-values andGET /attribute-valuesfunctionsoftheJSONQBRESTAPI.

5. Presentsintablethevaluesofatwo-dimensionalsliceofanRDFdatacube.MakeuseoftheGET /table function of the JSON QB REST API. The OLAP Browser presents the tablebasedonthejson-statresultsreturnedbytheAPI.

3.6.2 ImplementationDescription

TheimplementationoftheAPIusesthefollowingtechnologies:

1. D3.js14JavaScriptlibraryforeasilymanipulatingandvisualizingtheJSONdatareturnedbytheJSONQBRESTAPI.

2. JQuery15 a cross-platformJavaScriptlibrarytosimplifytheclient-sidescriptingofHTML.3. AjaxtomakeRESTcallstotheJSONQBRESTAPI,receiveresponsesandupdatetheweb

pagewithoutreloading.4. Json-statJavaScriptlibrary16toparseandconsumejson-statdata

3.6.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-olap-browser

3.6.4 LicenseThesoftwareisavailableasopensourceundertheApacheLicense2.0.

3.6.5 Futuredevelopmentplan

Theplansfornextyeardevelopmentinclude:

1. Enabletheintegratedviewofmultiplecompatiblecubesontheflywithouttheneedtocreateanewmergedcube

2. Performroll-upanddrill-downOLAPoperations3. Performdimensionaddition/reductionOLAPoperation

3.6.6 RoleinthearchitectureOLAPBrowsercanberepresentedasmultiplecomponentsinProcessLayer:Browsing,VisualisationandStatisticalAnalysiscomponents.

14 https://d3js.org/ 15 https://jquery.com/ 16 https://github.com/badosa/JSON-stat

Page 38: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page38of39

4 ConclusionIn this deliverable, we have described the components delivered as the result of the firstOpenGovIntelligence development phase. The next release of OpenGovIntelligence will consistofalarger number ofthetools supporting a more complete part of the OpenGovIntelligenceframework.Existingtoolsavailableforthepurposewillre-usedwhereeverpossibleandnewtoolswillbedevelopedwhenrequired.

The next stage of the project will involve evaluation of the delivered components in use casesorganizedbythepilotpartners.

Page 39: OGI D3.2 Report on OpenGovIntelligence ICT tools - … · D3.2 – Report on OpenGovIntelligence ICT tools - first release Page 4 of 39 Consortium Role Name Short Name Country 1

D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease

Page39of39

References

[1]D1.1OpenGovIntelligencechallengesandneeds

[2]D2.1OpenGovIntelligenceframework-firstrelease

[3]D3.1OpenGovIntelligenceICTtools-firstrelease

[4]D4.1PilotsandEvaluationplan


Top Related