ThisprojecthasbeenfundedwiththesupportoftheH2020ProgrammeoftheEuropeanUnionãCopyrightbytheOpenGovIntelligenceConsortium
.
OpenGovIntelligence
FosteringInnovationandCreativityinEuropethroughPublicAdministrationModernizationtowardsSupplyingandExploiting
LinkedOpenStatisticalData
Deliverable3.2
ReportonOpenGovIntelligenceICTtools-firstrelease
Editor(s):
ArkadiuszStasiewicz(NUIG)
MohammadWaqar(NUIG)
ResponsibleOrganisation: NUIG
Version-Status: V1.0
Submissiondate: 31/10/2016
Disseminationlevel: PU
Ref. Ares(2016)6209154 - 31/10/2016
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page2of39
DeliverablefactsheetProjectNumber: 693849
ProjectAcronym: OpenGovintelligence
ProjectTitle:Fostering Innovation and Creativity in Europe through PublicAdministrationModernizationtowardsSupplyingandExploitingLinkedOpenStatisticalData
TitleofDeliverable: D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Workpackage: WP3–ICTtoolsdevelopment
Duedateaccordingtocontract: 31/10/1016
Editor(s):ArkadiuszStasiewicz(NUIG)
MohammadWaqar(NUIG)
Contributor(s):
DimitriosZeginis(CERTH)
EvangelosKalampokis(CERTH)
EfthimiosTambouris(CERTH)
KonstantinosTarabanis(CERTH)
ArkadiuszStasiewicz(NUIG)
MohammadWaqar(NUIG)
MohamedAdel(NUIG)
BillRoberts(SWIRRL)
RickMoynihan(SWIRRL)
Reviewer(s): BillRoberts(SWIRRL)
Approvedby: AllPartners
Abstract: This deliverable provides the description of the prototypes
ofsoftware components delivered as a result of the firstOpenGovIntelligenceICTtoolsdevelopmentstage.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page3of39
KeywordList: software,components,linkeddata,statisticaldata,ICTtools
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page4of39
Consortium Role Name ShortName Country
1. Coordinator CentreforResearch&Technology-Hellas CERTH Greece
2. R&Dpartner DelftUniversityofTechnology TUDelft Netherlands
3. R&Dpartner NationalUniversityofIreland,Galway NUIG Ireland
4. R&Dpartner TallinnUniversityofTechnology TUT Estonia
5. R&Dpartner ProXMLbvba ProXML Belgium
6. R&Dpartner SwirrlITLimited SWIRRL UnitedKingdom
7. PilotPartner TraffordCouncil TRAF UnitedKingdom
8. PilotPartner FlemishGovernment VLO Belgium
9. PilotPartner MinistryofInteriorandAdministrativeReconstruction MAREG Greece
10. PilotPartner MinistryofEconomicAffairsandCommunication MKM Estonia
11. PilotPartner MarineInstitute MI Ireland
12. PilotPartner PublicInstitutionEnterpriseLithuania EL Lithuania
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page5of39
RevisionHistoryVersion Date Revisedby Reason
0.1 30/08/2016 NUIG Firstdraft
0.2 15/09/2016 NUIG Generalupdates
0.3 14/10/2016 NUIG ToolsDescription(draft)
0.4 26/10/2016 NUIG Architecturedetails
0.5 28/10/2016 NUIG Generalupdates
0.6 29/10/2016 NUIG Generalupdates
0.7 30/10/2016 NUIG Addressedreviewcomments
1.0 31/10/2016 CERTH SubmissiontoEC
Statementoforiginality:This deliverable contains original unpublishedwork except where clearly indicated otherwise.Acknowledgement of previously publishedmaterial andof thework of others has beenmadethroughappropriatecitation,quotationorboth.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page6of39
TableofContentsDELIVERABLEFACTSHEET......................................................................................................................2
CONSORTIUM.......................................................................................................................................4
REVISIONHISTORY...............................................................................................................................5
TABLEOFCONTENTS............................................................................................................................6
LISTOFFIGURES...................................................................................................................................7
LISTOFABBREVIATIONS.......................................................................................................................8
EXECUTIVESUMMARY..........................................................................................................................9
1 INTRODUCTION..........................................................................................................................10
1.1 SCOPE...................................................................................................................................................101.2 AUDIENCE..............................................................................................................................................101.3 STRUCTURE............................................................................................................................................10
2 ARCHITECTUREOVERVIEW.........................................................................................................11
2.1 DATAPROVISION.....................................................................................................................................112.2 DATAPLATFORM.....................................................................................................................................142.3 PROCESSLAYER.......................................................................................................................................152.4 SERVICEDESIGN......................................................................................................................................162.5 SERVICEPROVISION..................................................................................................................................182.6 MANAGEMENT........................................................................................................................................18
3 OPENGOVINTELLIGENCEICTTOOLS-FIRSTRELEASE...................................................................20
3.1 JSONAPIFORDATACUBESPECIFICATION...................................................................................................203.2 JSONAPIFORDATACUBEIMPLEMENTATION..............................................................................................213.3 TABLE2QBANDGRAFTER..........................................................................................................................283.4 DATACUBEBUILDER................................................................................................................................303.5 DATACUBEEXPLORER..............................................................................................................................333.6 OLAPBROWSER.....................................................................................................................................36
4 CONCLUSION..............................................................................................................................38
REFERENCES.......................................................................................................................................39
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page7of39
ListofFiguresFIGURE 1 OGI ARCHITECTURE ............................................................................................................................... 13FIGURE 2 CUBE BUILDER DESKTOP UI ................................................................................................................... 30FIGURE 3 DATA CUBE EXPLORER - DATASET SELECTION ...................................................................................... 34FIGURE 4 DATA CUBE EXPLORER - SAMPLE VISUALISATION ................................................................................. 34
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page8of39
ListofAbbreviationsThefollowingtablepresentstheacronymsusedinthedeliverableinalphabeticalorder.
Abbreviation Description
API ApplicationProgrammingInterface
CSV CommaSeparatedValues
ICT InformationandCommunicationTechnologies
LOSD LinkedOpenStatisticalData
OLAP OnLineAnalyticalProcessing
RDF ResourceDescriptionFramework
UI UserInterface
URI UniformResourceIdentifier
W3C WorldWideWebConsortium
WP WorkPackage
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page9of39
ExecutiveSummary
The present document is the deliverable “D3.2 – Report on OpenGovIntelligence ICT tools – firstrelease” (referred to as D3.2). It provides detailed information about the result of the first stageoftheOpenGovIntelligencedevelopment.ThegoalofWP3 (“ICT toolsdevelopment”) istodeveloptheOpenGovIntelligence ICT tools asasuite of open source and commercial tools. ThedevelopedICT toolswill support theOpenGovIntelligence frameworkcreatedatWP2 (“Frameworkcreation”)andwillenable:
(a) thecreationofLinkedOpenStatisticalData(LOSD)fromvarioussources,(b) theexpansionofLOSDwithdatasetsfromexistingsources,(c) theexploitationofLOSDfortheco-productionofpublicservices.
The first release of the OpenGovIntelligence ICT tools cover the tools regarding the creationandexploitation of LOSD. With regards to the creation of LOSD, OpenGovIntelligence designedanddeveloped ICT tools thatenable the transformationofpublicsectordata tostandardmachinereadable forms (specifically toRDF data cubes) and the validation of the generated RDF.Withregards to the exploitation of LOSD, OpenGovIntelligence designed and developed ICT toolsthatenablevisualisationandanalysisofstatisticaldata.
The tools presented in this deliverable will be evaluated during the pilot implementations stage.Basedontheoutcomes,theexistingtoolswillbeimproved.MoreovernewtoolswillbedevelopedinordertocoverremainingfeaturesoftheOpenGovIntelligenceFramework.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page10of39
1 IntroductionThis section introduces the background of thework carried out inWP3 “ICT tools development”.Sub-section 1.1 presents the scope and the objectives of the current document, sub-section 1.2describes the intendedaudience for thisdocument,whilesub-section1.3outlines thestructureofthedocument.
1.1 ScopeThis reportdocuments the ICT toolsdevelopedduring the firstphaseofWP3.Toguide readers inunderstanding the context of these tools, Section 2 presents the high level architecture of theOpenGovIntelligenceFramework,showingtheoverallorganizationof layers,componentsandtheirinteractions. This first release focuses on the development of tools thatwill enable creation andexploitationofLOSD.InsubsequentstagesfurthertoolsareplannedcoveringexpansionofexistingdatasetsandexploitationofLOSDforco-productionofpublicservices.
1.2 AudienceThe intended audience for this document is the OpenGovIntelligence consortium, in particularpartnerorganisationsresponsibleforthedevelopmentofpilottrials,theEuropeanCommission(EC)andthosewhoareinterestedinchallengesandneedsforopening-upandexploitingLOSDfortheco-productionofinnovativedata-drivenservicesongovernments.
1.3 StructureThestructureofthedocumentisasfollows:
• Section2presentstheOpenGovIntelligenceArchitectureandprovidesdescriptionsofeachoftheindividualcomponents.
• Section3providesadetaileddescriptionoftheindividualOpenGovIntelligenceICTtoolsdevelopedduringthefirstdevelopmentstageofOpenGovIntelligenceproject;
• Finally,Section4concludesthereportandoutlinesthefuturedevelopmentplan.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page11of39
2 ArchitectureOverview
TheOGIArchitecture forLOSDanddata-drivenpublicservicesenablesstakeholders tocollaboratetowards the production of innovative data-driven public services by exploiting Linked Open Datatechnologiesandstatisticaldatasets.Thisistheinitialversionofthearchitecturethatwillbetestedand refined during the first round of pilot evaluations. The architecture is presented in Error!Referencesourcenotfound.anditisorganisedasfollows:
• Thearchitectureisdividedintofivelayers:(i)DataProvision,(ii)DataPlatform,(iii)Process• Thearchitectureisdividedintofivelayers:(i)DataProvision,(ii)DataPlatform,(iii)ProcessLayer,(iv)ServiceDesign,and(v)ServiceProvision.
• Keymanagementresponsibilitiesaresharedacrossalllayers.
• Eachlayerhasasetofcomponentsthatperformstasksspecifictothatlayer.
Thearchitecturewillguidethepilot implementations,aswellastheotherfuture implementationsoftheOGIsoftwareinotherprojects.
Eachpartofthearchitectureisdescribedindetailinthefollowingsections.
2.1 DataProvisionTheDataProvisionLayerimplementsfundamentalfunctionalitiesneededtocreatehighqualityLODandthussupporttheexecutionandscalabilityoftheServiceDesign.TheDataProvisionservicesRDFcreation, schema, code lists, vocabularies andmetadatamanagement services in order to enabletheextensibility and scalability and assure the data quality of the proposed OpenGovIntelligenceapproach.
2.1.1 LODCreationTheLinkedOpenDataCreationgroupincludesthefollowingmodules:
2.1.1.1 MetadataManagement
Metadataisessentialtolearnwhatdataisavailableandunderstandandinterpretthestatisticaldata-withoutitthedataismeaningless.
Themetadatacanbecategorisedas:
• Structuralmetadata-codeorconceptlists,dimensiondefinitions,unitsofmeasure,etc.• Referencemetadata-informationonmethodologyandquality• Processmetadata-descriptionofproductionprocesses,orprocessmetrics• Provenance-descriptionofthesourceofinformation,originofthedata,digitalrights,etc.
OpenGovIntelligence metadata is represented as Linked Data, as it will be used as a medium todiscover aggregated cubes that are stored in dispersed data repositories.MetadataManagement
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page12of39
willbecarriedoutbymappingtheavailabledescriptive,structuralandadministrativecharacteristicsofthedigesteddatasetusingtheDataCatalogVocabulary(DCAT1)vocabulary,whichisawidespreadW3Cstandardfordescribingdatasets.
2.1.1.2 VocabulariesManagement
Re-use of accepted controlled vocabularies (for example those from W3C, Eurostat or nationalstatisticalorganisations)isoneofthemostimportantprinciplesofLinkedData.Itenablesthelinkingofdifferentcubesattheschemalevel(i.e.measures,dimensions,attributes).Consequently,thereisa need for managing controlled vocabularies that could be reused across different datasets.VocabulariesManagement service will be responsible for creating, storing, searching, discoveringandreusingexistingcontrolledvocabularies.
2.1.1.3 CodeListsManagement
Code Lists are commonly used among RDF data cubes in order to assure unambiguous definitionofthe dimensions within a dataset. Moreover, code lists may have a hierarchical structure.CodeListsManagementservicewillberesponsibleforcreating,storing,searching,discoveringandreusingexistingcodelists.
2.1.1.4 SchemaManagement
Data gathered during the cube building and modelling process using RDF Data Cube vocabularyallows to store information at the schema level (i.e. dimensions, attributes,measures). The reuseofthe vocabularies and the code lists enables linking of disparate cubes. Schema Managementservicewillberesponsibleforcreating,storing,searching,discoveringandreusingexistingschemas.
2.1.1.5 RDFCreation
TheRDFCreationprocess isresponsiblefortransformationofthedata(i.e.publicsectorstatisticaldata) to LinkedData from various existing source formats. This process enables search, discoveryandqueryingofrelevantdata. InparticulartheRDFcreationserviceisresponsiblefordefinitionofthemodel to represent the data, then transforming the data to fit thatmodel. The core of themodelistheRDFDataCubeVocabulary2.
2.1.1.6 CubeBuilder
TheCubeBuilderserviceprovides interfacesthatallowsausertomaprawdatatoRDFDataCubestructure by identifying dimensions, measures and attributes in the raw data. With the supportofother components (e.g. SchemaManagement, Metadata Management) it maps code lists andvocabularies.
1 https://www.w3.org/TR/vocab-dcat/ 2 https://www.w3.org/TR/vocab-data-cube/
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page13of39
Figure1OGIArchitecture
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page14of39
2.1.2 QualityAssuranceThe Quality Assurance group is a set of activities, standards and processes that will ensuresthevalidation aswell as the high quality of the generated LOSD and theirmetadata and includesthefollowingmodules:
2.1.2.1 Validation
Validation component is responsible for the validation of data cubes. It achieves this by verifyingtheschema requirements and detection of incorrect or missing values. Validation is performedonbothdatacubeschemamodelsandcreateddatacubeobservations.
2.1.2.2 QualityAssurance
Quality Assurance component is responsible for ensuring the quality of data and associatedmetadata.Thecomponentwilluseseriesofteststhatcanbeapplied inordertocheckthequalityofthedatasetandflagthe information.Thisprocess tracksqualityofdatacubesandensuresthatonlyhighqualitydataismaintainedinsidethesystem.
2.2 DataPlatformThe Data Platform Layer is responsible for the data storage and provides a catalogue service foravailable datasets that lets userssearch for aggregated cubes overmany distributed repositories,basedontheirmetadata.Itconsistsofthefollowingdatadrivenprocesses:
2.2.1 CubeCatalogueTheDataCubeCatalogueprovidesan interface thatenablesbrowsingandexplorationof thedatacubesbasedonmetadataassociatedwiththedatacubes.Itlistsavailabledatasetsandtheusercansearchandfilterresultsbasedonkeywords,tagsandcategoriesetc.
2.2.2 CubeDiscoveryCube Discovery is the starting point for the services like Visualisation, Analysis, Mashups, etc.Itallowsuserstonavigatethroughthedatasetsandexplorethecontentsofthedatasets.
2.2.3 QueryingThe Querying Service provides a user interface for writing and running SPARQL queries andpreviewingtheresults.Additional featuresmight includesyntaxhighlighting,acatalogueofstoredqueries,autocompleting,etc.
2.2.4 SearchingTheSearchingcomponentprovidesa setof simpleandadvancedsearchand indexingcapabilities,lookingatthedata itself,not just itsmetadata.Searchservice isessentialforfindingrelevantdatainlargerepositoryofdatasets.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page15of39
2.2.5 LinkingTheLinkingcomponenthelpsinlinkingdatatointernalandexternallinkeddataresources.
2.2.6 StorageThe Storage component is responsible for persistence of the data and information. In particularitwill store Data cubes outputted from the Data Provision Layer, mashups, outputsoftheenrichmentprocesses,analyticsresults,indexedpartsoftheRDF,metadata,userprofiles,etc.
2.2.7 LinkedStatisticalDataCloudLinked Statistical Data represents the Linked Open Data available on the web (externaltotheStoragecomponent).
2.3 ProcessLayerTheProcessLayerallowsanendusertointeractwiththesystem.Itconsistsofthefollowing:
2.3.1 BrowsingThe Browsing component enables navigation through available data cubes, data cubes metadataandexaminationofthedatacubeobservations'values.
2.3.2 VisualizationsThe Visualization component involves meaningful interactive visualizations from the availabledatasets.
2.3.3 ExpansionTheExpansioncomponentenablestheprocessofexpandingadatacubewithcompatibledatafromelsewhere.Ifthecompatibiityrulesaresatisfied,thiscanbedoneinthefollowingways:
• CubeExpansionbyaddingadditionalmeasures,• CubeExpansionbyaddingadditionalvaluetoselecteddimension.
The linking ofmeasures of aggregated cubes results in larger cubes describing the samemeasurewhile the linking of the dimensions of different aggregated cubes facilitates data analyticsandvisualizationservices.
2.3.4 Mashups
TheMashups component let users combine data frommultiple sources and create new servicesdisplayedinasinglegraphicaluserinterface.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page16of39
2.3.5 StatisticalAnalysisStatisticalAnalysismodule is responsible for rangeof statistical analysis functions and algorithms.Itrangesfrombasic(e.g.standarddeviationofameasurevaluesagainstadimension)toadvanced(e.g.anomalydetectionusingmultiplemethodologies)methodswhichusercanchoosefrom.
2.3.6 MachineLearning
This component will use machine learning algorithms in order to mine and extract meaningfulinformationaboutthedataandhiddenpatternsaswellasintroduceintuitivevisualisations.
2.3.7 Enrichment
TheEnrichmentcomponentwillenablelinkingofdatacubeswithexternalresources(e.g.DBpedia)and combination of data across statistical and non-statistical datasets. Thisprocess will allowtoperformaricherdataanalysisandenhancedvisualizations.
2.3.8 AggregationThis component will compute aggregations across dimensions and hierarchies in selected datacubes. Aggregated values will be used for visualization, statistical analysis and machine learningpurposes,thusenablingenhancedoperations(e.g.roll-up/drill-down,dimensionreductionetc.).
2.4 ServiceDesignService Design implements the main functionalities offered by OpenGovIntelligence and enabletheco-designofdata-drivenpublicservices.Itconsistsofthefollowing:
2.4.1 ServiceDesignTheServiceDesignGroupofcomponentshelpsstakeholderswithproperICTskillstobeconnectedtodatasourcesinordertodesignnewservicesthatattemptstoaddresspublicneeds.Itincludesthefollowingmodules:
2.4.1.1 IdeaCreation
The Idea creation component will support workflow for idea creation process aligned withOpenGovIntelligence framework. Platform will allow involvement of more than one stakeholdergroupinthecreationprocessandsupportitbycombiningcomponentsavailableintheProcesslayerandtheDataPlatformlayer.
2.4.1.2 Prototyping
Prototyping enables rapid creation of the services prototypes and its dissemination toshowcaseproposedsolutions.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page17of39
2.4.1.3 Evaluation
The Evaluation module allow to validate designed prototypes, collect feedback, improvementsuggestionsandadditionalfeaturesrequests.
2.4.1.4 Deployment
The Deployment component is responsible for the publication of the developed and validatedsolution.
2.4.2 CollaborationSpaceThiselementwillhelptoovercomespaceandtimedifferentialsandfacilitatetheco-designofdata-drivenpublicservicesusingelectroniccommunicationsandgroupwarebystakeholderscomingfromanycountryand/ortimezone.
2.4.2.1 Feedback
Feedback involvesmechanisms thatwill beused to collect andmanage theneedsof the citizens,businessesandpublicauthoritiesregardingtheinnovativedata-drivenpublicservices.
ItwillincludemechanismsandtoolsthatwillallowtheOpenGovIntelligencestakeholderstoprovidecomments(positiveornegative),preferences,propositionsand/orneedsregarding:
• datasourceusedbytheservice(i.e.enhancementofLOSDwithnewdata,updatedversionofthedataetc.).
• exploitation methods adopted in the co-produced public service (i.e. new types of dataanalysesorvisualizationthatisnotsupportedbyaprovidedservice)
• newservices.Forexampleacompanycouldaskfortheco-productionofanewinnovativeservicethatwillserviceitsneeds.
2.4.2.2 Discussion
Discussioncomponentenablesstakeholders(co-designersoftheservices)tosharethoughts,discussnew ideas, tobrainstorminsessions,chatandshareassets inordertodesignnewworkflowsthatcouldgeneratenewservices.
2.4.2.3 Rankings
Rankingscomponentletsstakeholderstorateandranktheservicesbasedoni.e.usabilityandvalueadded.
2.4.2.4 Exploration
This module enables exploration of the functionalities offered by OpenGovIntelligence platform,availableservices,workflowsandreusablecomponents.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page18of39
2.5 ServiceProvisionTheServiceProvisionlayerimplementstheco-provisionofthepublicservicesthatwerepreviouslyco-designedintheServiceDesignlayer.Atthislayerallmodulesareconcernedwithenduseroftheplatformservices.
2.5.1 ServiceProvision
2.5.1.1 ServicesCatalogue
ServicesCatalogueenablesdiscoveryoftheservicesdeployedattheplatform. Itwillprovideasetofsimple and advanced features for finding single service or group of services (e.g. by categoryorbytypeoftheservice).
2.5.2 ServiceImplementation
TheServiceImplementationGroupincludesthefollowingmodules:
2.5.2.1 ServiceImplementation
This module allows end users of the platform to adopt and implement selected service usingOpenGovIntelligenceplatform.
2.5.2.2 ServiceLiveness
Themainresponsibilityofthismoduleismonitoringofthedeployedservice,providingperformanceandusagemetricsandgeneratingreports(i.e.errorsandavailability)formaintenancepurposes.
2.5.2.3 FeedbackCollection
Thiscomponentprovidesmechanismforcollectionof thefeedbackfromendusersof theservicesprovidedbyOpenGovIntelligenceplatform.Theoutcomesmaybeusedforuserexperienceanalysisandservicesevaluationforcontinuity,enhancementsoreliminationdecisions.
2.5.2.4 ServiceEvaluation
Service Evaluation component will evaluate deployed services based on multiple analysis factors(e.g. service liveness and feedback) in scheduled periods and provide factors, that may helptodecidewhethertocontinue,enhanceorwithdrawparticularservice.
2.6 ManagementThe Management Layer provides cross-platform functionalities such as logging, configuration,security, API etc. It is responsible for overall management and smooth functioningoftheOpenGovIntelligenceplatform.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page19of39
2.6.1 ExternalToolsOpenGovIntelligence platform provides APIs to external tools (i.e. data processing pipelinesorexternalvisualisationlibraries)toletthemuserintegrateOGIcapabilities.
2.6.2 Monitoring
This component will provide information about liveness of the platform, usage performance,numberofuseroftheplatformandsoon.
2.6.3 LoggingSystemlogswillbeusedfortrackingplatformperformanceandresolveordebugissuesifoccurred.
2.6.4 SecurityThis component role is to ensure general security of the platform and provides authentication,authorizationandusermanagementcapabilities.
2.6.5 SystemConfiguration
Systemconfigurationisresponsibleforthemanagementofconfigurationdata.
Note: OpenGovIntelligence project will not develop software tools for every componentofthearchitecture or an integrated system thatmatches this architecture as awhole.Moreover,somepartsofthearchitecturemightendupbeinghumanprocessesandsomeotherinstantiationsof someother components-tools. Existing tools available for the purposewill re-usedwhere everpossible and new tools will be developed when required. We have already identified anumberoftools that can be re-used for example in collaboration space. This list of tools is includedindeliverableD2.1–OpenGovIntelligenceframework–firstrelease.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page20of39
3 OpenGovIntelligenceICTtools-firstreleaseThis section provides a detailed description of the individual OpenGovIntelligence ICT toolsdevelopedduringthefirstdevelopmentstageoftheOpenGovIntelligenceproject.
Wedescribethefunctionalityandimplementationofeachofthetools.
In the first implementation stage, we have developed tools supporting data conversionandexploitationof thedata. Thenext releaseofOpenGovIntelligencewill includeadditional toolsandwillsupportalargerpartoftheOpenGovIntelligenceframework.
3.1 JSONAPIforDataCubespecificationTheJSON-QBAPIisastandardisedwayfordatauserstoprogrammaticallyselectandretrievedatafromamultidimensionalstatisticaldatacube.Itisdesignedprimarilyforusebydeveloperswithoutspecialist knowledge of statistics or specialist knowledge of the underlying RDF Data Cuberepresentation. It supplements existing more complex ways of accessing data, such as viatheSPARQL query language. The objective is to increase the ease of use of statistical datainvisualisationsandapplications.
The OpenGovIntelligence is developing an openly licensed specification and implementationoftheAPI. The longer term aim is that this could be offered by awide range of data publishersasamethodofexploitingtheirdataandsocreatinggreater impact. Datapublisherscouldchoosetouse theOpenGovIntelligence implementation,orcouldcreate theirownsoftware thatcomplieswith the API specification, perhaps taking advantage of particular features of their own dataplatform.
3.1.1 FunctionalityDescriptionTheJSON-QBAPIprovidesprogrammaticaccess todataheld in theopenstandardRDFDataCubeformat,returningsubsetsofthedatainaformthatiseasyformostprogrammerstoworkwith.
3.1.2 ImplementationDescription
Therequirementsforthetoolaretoprovideaccesstomultidimensionalstatisticaldatainawaythatisfamiliartoabroadgroupofwebdevelopers,inparticulartomakeiteasytocreatecharts,mapsand other visualisations. The design favours simplicity over completeness. As well as being easytouse, it should be relatively easy for data publishers to implement, and so have themaximumchanceofbecomingwidelyusedstandard.
3.1.3 AvailabilityThespecificationcanbeaccessedathttps://github.com/OpenGovIntelligence/json-qb.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page21of39
3.1.4 LicenseThelicenceofthespecificationisCC-BY4.0https://creativecommons.org/licenses/by/4.0/
3.1.5 Futuredevelopmentplan
TheAPI is initsfirstexperimentalversion. Feedbackwillbegatheredfromusersofdataandusedtoenhancethedesignandimplementation.
3.1.6 RoleinthearchitectureJSONAPIforDataCubespecificationcanberepresentedasQueryingcomponentinDataPlatformLayer.
3.2 JSONAPIforDataCubeimplementationThiscomponentistheimplementationoftheJSONAPIforDataCubespecificationprovidedbytheOpenGovIntelligence project. It aims to provide an easy to use API for web developers that usestatisticaldatastoredintheformofRDFDatacubes.TheAPIimplementationcanbeinstalledontopofanyRDFrepositoryandofferbasicandadvancedoperationsonRDFDatacubesassuming that:i)theyarestoredusing theRDFDataCubeVocabulary, ii) they followaspecificapplicationprofile(common practices) and iii) are accessible through a SPARQL endpoint. The results are alwaysreturnedinJSONformat.
3.2.1 FunctionalityDescriptionTheAPI implementationoffers functionalityusedbyother components through simpleRESTcalls.Inthe next tables we describe each function. Specifically, we provide the name of the function,ashort description, the URI (including the REST method used e.g. GET), the parameters thatthefunction requires as input, a sample JSON response and the SPARQL query that is executedtogettherequiredresults.
Name Getallavailablecubes
Description ReturnsalltheavailablecubesofanRDFrepository.
URI GET/cubes
Parameters -
Response {"cubes":[{"URI":"http://example.com/cube/unemployment",
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page22of39
"label":"Unemploymentrate"},{"URI":"http://example.com/cube/population","label":"Population"}....]}
SPARQLquery
PREFIXqb:http://purl.org/linked-data/cube#PREFIXskos:http://www.w3.org/2004/02/skos/core#selectdistinct?cube?labelwhere{?cuberdf:typeqb:DataSetOPTIONAL{?cubeskos:prefLabel|rdfs:label?label.}}
Name Getdimensions
Description Returnsallthedimensionsofacube
URI GET/dimensions
Parameters dataset:theURIofthecube
Response {"dimensions":[ {"URI":"http://example.com/dimension/refArea", "label":"Reference area" }, {"URI":"http://example.com/dimension/refPeriod", "label":"Reference period" }.... ] }
SPARQLquery
PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select distinct ?dim ?label where { <http://example.com/cube/unemployment> qb:structure ?dsd. ?dsd qb:component ?cs. ?cs qb:dimension ?dim. OPTIONAL{?dim skos:prefLabel|rdfs:label ?label.}}
Name Getmeasures
Description Returnsallthemeasuresofacube
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page23of39
URI GET/measures
Parameters dataset:theURIofthecube
Response {"measures":[ {"URI":"http://example.com/measure/employmentRate", "label":" Employment Rate " }, {"URI":"http://example.com/measure/unemploymentRate", "label":" Unemployment Rate” }.... ] }
SPARQLquery
PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select distinct ?measure ?label where { < http://example.com/cube/unemployment> qb:structure ?dsd. ?dsd qb:component ?cs. ?cs qb:measure ?measure. OPTIONAL{?measure skos:prefLabel|rdfs:label ?label.}}
Name Getattributes
Description Returnsalltheattributesofacube
URI GET/attributes
Parameters dataset:theURIofthecube
Response {"attributes":[ {"URI":"http://example.com/attribute/unit", "label":"Unit of measure" }, {"URI":"http://example.com/attribute/obsStatus", "label":"Observation status" }.... ] }
SPARQLquery
PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select distinct ?attribute ?label where { < http://example.com/cube/unemployment> qb:structure ?dsd. ?dsd qb:component ?cs. ?cs qb:attribute ?attribute.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page24of39
OPTIONAL{?attribute skos:prefLabel|rdfs:label ?label.}}
Name Getdimensionvalues
Description Returnsallthevaluesofadimensionthatappearataspecificcube
URI GET/dimension-values
Parameters dataset:theURIofthecube
dimension:theURIofthedimension
Response {"values":[
{"URI":"http://example.com/value/female", "label":"Female" }, {"URI":"http://example.com/value/male", "label":"Male" }.... ] }
SPARQLquery
PREFIX qb: http://purl.org/linked-data/cube#
PREFIX skos: http://www.w3.org/2004/02/skos/core#
select distinct ?val ?label where {
?observation qb:dataSet <http://example.com/cube/unemployment >.
?observation < http://example.com/dimension/sex> ?val.
OPTIONAL{?val skos:prefLabel|rdfs:label ?label}}
Name Getattributevalues
Description Returnsallthevaluesofanattributethatappearataspecificcube
URI GET/attribute-values
Parameters dataset:theURIofthecubeattribute:theURIofthedimension
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page25of39
Response {"values":[ {"URI":"http://example.com/value/count", "label":"Count" }, {"URI":"http://example.com/value/ratio", "label":"Ratio" }.... ] }
SPARQLquery
PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select distinct ?val ?label where { ?observation qb:dataSet <http://example.com/cube/unemployment >. ?observation <http://example.com/attribute/unit> ?val. OPTIONAL{?val skos:prefLabel|rdfs:label ?label}}
Name Getslice
Description Returnsasetofobservationsofcubethatmatchtoparticularvaluesofselecteddimensions
URI GET/slice
Parameters dataset:theURIofthecube0ormoredimensionidentifiersandtheallowedvaluese.g. http://example.com/dimension/sex =http://example.com/value/female
Response {"observations":[ {"URI":"http://example.com/observation/1", "http://example.com/dimension/refArea ":" http://example.com/Greece", "http://example.com/dimension/refPeriod ":" http://example.com/2015", "http://example.com/measure/unemploymentRate": 0.24 }, {"URI":"http://example.com/observation/2", "http://example.com/dimension/refArea ":"http://example.com/Greece", "http://example.com/dimension/refPeriod ":"http://example.com/2016", "http://example.com/measure/unemploymentRate": 0.26
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page26of39
},.... ] }
SPARQLquery PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select ?obs ?refArea ?refPeriod ?unemployment where { ?obs qb:dataSet <http://example.com/cube/unemployment >. ?obs <http://example.com/dimension/sex> <http://example.com/value/female>. ?obs <http://example.com/dimension/refArea> ?refArea. ?obs <http://example.com/dimension/refPeriod> ?refPeriod. ?obs <http://example.com/measure/unemploymentRate> ?unemployment. }
Name Gettable
Description Returns a 2D table representation of the cube’s observations that match toparticularvaluesofselecteddimensions
URI GET/slice
Parameters dataset:theURIofthecubecol:theURIofthedimensionusedforthecolumnsofthe2Dtablerow:theURIofthedimensionusedforrowsofthe2Dtable0ormoredimensionidentifiersandtheallowedvaluese.g. http://example.com/dimension/sex =http://example.com/value/female
Response The response is in json-stat format e.g. {"class" : "dataset", "id" : ["http://example.com/dimension/refArea","http://example.com/dimension/refPeriod"], "size" : [10,5], "dimension" : { "http://example.com/dimension/refArea" : { "category" : { "index" : ["http://example.com/Greece",
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page27of39
"http://example.com/UK",…] }}, "http://example.com/dimension/refPeriod" : { "category" : { "index" : ["http://example.com/2015", " http://example.com/2016",…] } }…}, "value" : ["0.24", "0.26" ….] //values are stored in row-major order }
SPARQLquery PREFIX qb: http://purl.org/linked-data/cube# PREFIX skos: http://www.w3.org/2004/02/skos/core# select ?unemployment where { ?obs qb:dataSet <http://example.com/cube/unemployment >. ?obs <http://example.com/dimension/sex> <http://example.com/value/female>. ?obs <http://example.com/dimension/refArea> ?refArea. ?obs <http://example.com/dimension/refArea> ?refPeriod. ?obs <http://example.com/measure/unemploymentRate> ?unemployment. } ORDER BY ?refArea ?refPeriod
3.2.2 ImplementationDescription
TheimplementationoftheAPIusesthefollowingtechnologies:
1. Jersey3frameworkfortheimplementationoftheRESTfulservices.JerseyisanimplementationoftheJAX-RS4ReferenceImplementation.
2. SPARQL5forqueryingtheRDFrepositories.3. Rdf4j6JavaframeworkforprocessingRDFdata4. Json-statjavalibrary7tocreateandconsumejson-statdata5. Gson8JavalibrarytoserializeJavaObjectsintotheirJSONrepresentation
3 https://jersey.java.net/ 4 https://jax-rs-spec.java.net/ 5 https://www.w3.org/TR/rdf-sparql-query/ 6 http://rdf4j.org/ 7 https://github.com/statisticsnorway/json-stat.java 8 https://github.com/google/gson
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page28of39
3.2.3 AvailabilitySource code is available at GitHub: https://github.com/OpenGovIntelligence/json-qb-api-implementation
3.2.4 LicenseThesoftwareisavailableasopensourceundertheApacheLicense2.0.
3.2.5 Futuredevelopmentplan
Theplansfornextyeardevelopmentinclude:
1. Support pagination of results. In the case of big cubes this will enable the limiting ofreturnedresult.
2. Support advanced filtering of dimension values. For example, enable filtering by ranges,patterns,filteringbygeographyetc.
3. Supportmultilinguality. In somecases, cubescontain labels inmultiple languages.TheAPIshouldbeextendedtosupportthem.
4. Support hierarchical data. In some cases, dimensions have hierarchical structure. The APIshouldbeextendedtosupporthierarchicaldata.
3.2.6 RoleinthearchitectureJSON API for Data Cube implementation can be represented as Querying component in DataPlatformLayer.
3.3 Table2qbandGrafterGrafter is an open source software library to support Extract Transform Load processes, withaparticular emphasis on transforming tabular data to RDF. Its development has been supportedbytheEUFP7DaPaaSandOpenCubeprojects,andfurther improvementshavebeenimplementedas part of OpenGovIntelligence. These include bug fixes, performance improvements and bettererrormessages.
Grafter has been used to implement a process to convert statistical data to RDFData Cube fromatabular inputfile inCSVformat,thatfollowsaparticulartemplate. Wehavecalledthissoftware‘Table2qb’(pronounced‘tabletocube’).
The input template has one statistical observation per row, with a column for each dimension,attribute and measure. Column headers and cell contents are mapped to appropriate URIsforrepresenting thedata inRDFDataCube. The format is designed tobe simple for statisticianstogenerate,withouttheneedforprogrammingskillsandwithouttheneedforknowledgeofRDF.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page29of39
WorkisinprogresstomakethemappingfromtablecontentstoURIsmoreflexibleandconfigurable,toimprovethevalidationofuserinputsandtodefineastandardwayofassociatingmetadatawiththeinputfile.
3.3.1 FunctionalityDescriptionGrafterprovidesabasisforawiderangeofdatatransformationtasks,suchascleaningtabulardataandtransformingitintoRDF.
The Table2qb tool, implemented with Grafter, takes data in a specific tabular structure, eitherasaCSV or Excel file, and converts it into an RDF Data Cube: representing the data as a seriesofobservations with dimensions, attributes and measures, and generating the associated DataStructureDefinition.
3.3.2 ImplementationDescription
Grafter is implemented using the Clojure programming language. Clojure is a functionalprogramming language that runson the JavaVirtualMachine. Thismakes itefficient,able to runonallcommonoperatingsystems,andabletomakeuseofthelargerangeofexistingJavalibraries.
Table2qb is implemented using Clojure, and the Grafter library in particular. The prime designcriterion forTable2qb is that it shouldbesimple fornon-programmers togenerate the inputdataintherequiredformat,forexamplebyusingExcelorotherspreadsheetsoftware.
3.3.3 AvailabilityGrafter can be obtained from https://github.com/Swirrl/grafter. Table2qb is also on GitHubathttps://github.com/OpenGovIntelligence/table2qb.
3.3.4 LicenseThesoftwareisavailableasopensourceundertheEclipsePublicLicense.
3.3.5 Futuredevelopmentplan
FurtherworkonboththeTable2qbtoolandthesupportingGrafterlibraryisplannedoverthenextyear.ThiswillinvolvetestingandevaluatingTable2qbwitharangeofusersandarangeofdatasets.Thelessonslearnedwillbeusedtoimprovetheeaseofuseandflexibility.Itisalreadyknownthatimprovedvalidationanderrorreportingisahighpriorityareatoworkon.
3.3.6 RoleinthearchitectureTable2qb and Grafter can be represented as both: RDF Creation and Cube Builder componentsinDataProvisionLayer.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page30of39
3.4 DataCubeBuilderDataCubeBuildingenablesRDFcreationbythetransformationofdatasourcesinnon-RDFformatstoRDFDataCubes(inparticularconversionoftheCSVfiles).
CreatedDataCubesmaybestoredinthelinkeddataspaceinordertolinkandenrichthedatacubeswithexternallinkedknowledgebasedonpre-selectedlinkabledimensions.
RDFDataCubecreationprocessisusingthecorrespondingschemastructure.
3.4.1 FunctionalityDescription DataCubeBuildercanbeusedthroughthefollowinginterfaces:(A)DesktopUI,(B)CommandLine,(C)WebApplicationandas(D)WebService.
Application user is configuring the conversion process by passing required arguments throughthedesired interface. Data Cube Builder will output the RDF Data Cube regarding the argumentspassed.
3.4.2 ImplementationDescription
DataCubeBuildingfunctionalitiesarewritteninJavaandareextendingTarql(statisticalobservationcapturing tool) by adding the corresponding cube schema. At this stage tool support data cubeschemasfortheMarineInstitutepilotdatasets.Jframe,JcommanderandSparkjavaareusedtoofferinterfacesfortheCubeBuilder.
Figure2CubeBuilderDesktopUI
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page31of39
(B)CommandLine,
Usingthefollowingsetofarguments:
Arguments -Usage:OGIEU[options]Options:--csvFilePath,-csvCSVinputfileLocationandname--dimOrMeasures,-lCustomizedDimandMeasuresareNotAvailableatthisstage--help,-help,-hHelpDefault:false--marineDatasetName,-schemaDataSetSchemaCurrentlySupporting(IWaveBNetwork30Min,IMI_EATL_WAVE,IrishNationalTideGaugeNetworkandIWBNetwork)--qbFileName,-qbNCubeoutputfileLocationandname--qbPath,-qbCubeoutputfileLocationandname--run,-rWhichmainclasstoRun!(cmd,webserviceordesktop)--serialization,-formatOutputCubeRDFserlizationformat(turtleorntriples)Default:turtle
Example $mvn exec:java -Dexec.args="-run:cmd -csv:src/main/resources/data/IWBNetwork.csv -schema:IWBNetwork -format:turtle -qb:src/main/resources/output/ -qbN:IWBNetwork.ttl"
(D)WebService.
Arguments csv=inputFileNameAndLocation
schema=marineInstituteDatasetId
serialization=turtle
qbPath=outputFileLocation
qbName=outputFileName
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page32of39
Example /cubeBuilderAPI/cubeBuilderArgs?csv=inputFileNameAndLocation&schema=marineInstituteDatasetId&serialization=turtle&qbPath=outputFileLocation&qbName=outputFileName
SampledatatransformationoutputfromrowdatasetsinCSVformattoRDFcubes:
Dataset IrishNationalTideGaugeNetwork.csv
CSVstructure
longitude,latitude,altitude,time,station_id,Water_Level,Water_Level_LAT,Water_Level_OD_Malin,QC_Flag
degrees_east,degrees_north,m,UTC,,m,m,m,
-6.0683,53.3915,0.0,2016-08-09T00:00:00Z,HowthHarbour,2.902,2.902,-0.06,0.0
Cubiqstructureofobservations
OGI:observations_51fa99a5-cb7b-4f76-afc2-a4016f9da8f0
rdf:typeqb:Observation;
OGI:longitude"-6.0683";
OGI:latitude"53.3915";
OGI:altitude"0.0";
OGI:time"2016-08-09T00:00:00Z";
OGI:year"2016";
OGI:month"2016-08";
OGI:day"2016-08-09";
OGI:station_id"HowthHarbour";
OGI:water_Level"2.902";
OGI:water_Level_LAT"2.902";
OGI:water_Level_OD_Malin"-0.06";
OGI:qC_Flag"0.0";
qb:dataSetOGI:IrishNatoinalTideGaugeNetwork_ds.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page33of39
3.4.3 Availability Thesoftwarecanbeobtainedfromhttps://github.com/OpenGovIntelligence/data-cube-builder
3.4.4 License ThesoftwareisavailableasopensourceundertheApacheLicense2.0.
3.4.5 Futuredevelopmentplan The next development phase of Data Cube Builderwill involve development of schema for otherpilotspartners(e.g.LithuanianPilot)andfacilityofcustomschemausage.
Moreover,supportforotherrawdatasourcesformatstobedigestedbytheDataCubeBuildereg.JSONandTSVisplanned.
3.4.6 RoleinthearchitectureData Cube Builder can be represented as both: RDF Creation and Cube Builder componentsinDataProvisionLayer.
3.5 DataCubeExplorerDataCubeExplorer is responsible for listing,discovering,analysingandvisualizationdata fromthecubes. Itseversas integrateduser interfacefortheOGIplatform.Thetool isofferingcustomizablestatisticalanalysisandmachinelearningservicesusingDataCubesRDFasinputdata.
3.5.1 FunctionalityDescriptionExploration,DataCubeExplorationandVisualizationareservedthroughwebapplicationbuildwithDjango9(High-LevelPythonWebFramework).
API, Receiving multiple kinds of requests eg. Data Cube Creation, and Data Cube Queries thenroutingthoserequeststotheresponsiblemoduleforprocessing.
3.5.2 ImplementationDescription
Data Cube exploration uses pivottable.js visualization and for computing summary of dataset.ItiswritteninJavaScriptandDjangoframeworkforserversideoperations.
Userisablecanchooseoneoftheavailabledatasetandselectvisualisationtemplate(i.e.barchart)andconfigureitbyselectingdimensionsandfunctions(i.e.aggregationorsum).
9 https://www.djangoproject.com/
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page34of39
Figure3DataCubeExplorer-Datasetselection
Figure4DataCubeExplorer-SampleVisualisation
Inthecurrentversionofthetool,additionalAPIwasdevelopedinordertoquerythedataseore.
Type Description FunctionName
PresetQueries ListavailableLinkedCubes
JSONArray LqbQueryingForLqbSpaces(String limit)
List Linked Cubemetadata
JSONArray LqbQueryingForDimAndMeasures(String marineDatasetURI)
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page35of39
Retrieve Data ofcertainLinkedCube
JSONArray LqbQueryingForLqbData(String marineDatasetURI,String limit)
Customized/ on-demandQueries
Send Sparql QueryoverLinkedCubes
JSONArray LqbDirectQuerying(String sparqlQuery)
DataCubeAPIfunctionalities,currentlyavailableforDataCubeExplorationAPIs:
3.5.3 AvailabilityThesoftwareisavailableathttps://github.com/OpenGovIntelligence/data-cube-explorer
3.5.4 LicenseThesoftwareisavailableasopensourceundertheApacheLicense2.0.
3.5.5 Futuredevelopmentplan
Next step in the tool development will enable the usages of statistical and machine learningfunctionsandalgorithmsover thedatacubedimensionsandattributesvalues (starting frombasic
APIlink Description
listingavailablecubes
/cubeQueryingAPI/listLqbs?limit=numberOfRetrievedRecords
retrievecubemetadata
/cubeQueryingAPI/LqbMeta?dsuri=marineInstituteDatasetURI
retrievedataofcertaincube
/cubeQueryingAPI/listdataofLqb?dsuri=marineInstituteDatasetURI &limit=numberOfRetrievedRecords
sparqlendpoint /cubeQueryingAPI/cubeQueryingArgs ?query=sparqlQueryToExecuteAgainstLinkedCubeSpace
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page36of39
statisticalexplorationfunctionstoamorecomplexmachinelearningpredictivealgorithms).WEKA10libraryisacandidateforprovidingthedataminingcapabilities.
Useof JSONAPI forDataCube as communication channelwithdata store is oneof theprioritiesinthedevelopmentcycle.
3.5.6 RoleinthearchitectureData Cube Explorer can be represented as multiple components inProcess Layer: Browsing,VisualisationandStatisticalAnalysiscomponents.
3.6 OLAPBrowserThe OLAP Browser enables the exploration of RDF data cubes by presenting each time a two-dimensional slice of the cube as a table. The OLAP Browser is based on the “OpenCube OLAPBrowser”11. “OpenCube OLAP Browser” was developed as a widget of Information Workbenchplatform12, in thisnew implementationweaimtodisengage fromthe limitations that InformationWorkbench imposed and implement a more powerful and user friendly standalone tool.ThelimitationsthatInformationWorkbenchincludethe“poor”userinterface,badperformance,useofspecificRDFrepositories,bundledwithInformationWorkbenchetc.
TheOLAPBrowserbuildsuponthefunctionalityofferedbytheJSONQBRESTAPI implementation.Specifically,itcallstheAPIfunctionsandvisualizesthereturnedJSONresults.OLAPBrowsersupportvisualizationofdatarepresentedinjson-stat13format
3.6.1 FunctionalityDescriptionCurrentlytheOLAPBrowsersupportsthefollowingfunctionalities:
1. Listall theavailablecubesofanRDFrepository.Makeuseof theGET /cubes functionprovidedbytheJSONQBRESTAPI.
2. Presentthestructure(measures,dimension,attributes)ofaselectedcube.MakeuseoftheGET /dimensions,GET /measuresandGET /attributes functionsoftheJSONQBRESTAPI.
3. Enabletheselectionofthedimensions(rowsandcolumns)ofthe2Dtabletobevisualized.
10 http://www.cs.waikato.ac.nz/ml/weka/ 11 http://opencube-toolkit.eu/opencube-olap-browser/ 12 https://www.fluidops.com/en/products/information_workbench/ 13 https://json-stat.org/
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page37of39
4. Enable filtering of fixed dimensions/attributes (i.e. the dimensions/attributes of the cubethat are not shown in the 2D table).Make use of theGET /dimension-values andGET /attribute-valuesfunctionsoftheJSONQBRESTAPI.
5. Presentsintablethevaluesofatwo-dimensionalsliceofanRDFdatacube.MakeuseoftheGET /table function of the JSON QB REST API. The OLAP Browser presents the tablebasedonthejson-statresultsreturnedbytheAPI.
3.6.2 ImplementationDescription
TheimplementationoftheAPIusesthefollowingtechnologies:
1. D3.js14JavaScriptlibraryforeasilymanipulatingandvisualizingtheJSONdatareturnedbytheJSONQBRESTAPI.
2. JQuery15 a cross-platformJavaScriptlibrarytosimplifytheclient-sidescriptingofHTML.3. AjaxtomakeRESTcallstotheJSONQBRESTAPI,receiveresponsesandupdatetheweb
pagewithoutreloading.4. Json-statJavaScriptlibrary16toparseandconsumejson-statdata
3.6.3 AvailabilitySourcecodeisavailableatGitHub:https://github.com/OpenGovIntelligence/qb-olap-browser
3.6.4 LicenseThesoftwareisavailableasopensourceundertheApacheLicense2.0.
3.6.5 Futuredevelopmentplan
Theplansfornextyeardevelopmentinclude:
1. Enabletheintegratedviewofmultiplecompatiblecubesontheflywithouttheneedtocreateanewmergedcube
2. Performroll-upanddrill-downOLAPoperations3. Performdimensionaddition/reductionOLAPoperation
3.6.6 RoleinthearchitectureOLAPBrowsercanberepresentedasmultiplecomponentsinProcessLayer:Browsing,VisualisationandStatisticalAnalysiscomponents.
14 https://d3js.org/ 15 https://jquery.com/ 16 https://github.com/badosa/JSON-stat
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page38of39
4 ConclusionIn this deliverable, we have described the components delivered as the result of the firstOpenGovIntelligence development phase. The next release of OpenGovIntelligence will consistofalarger number ofthetools supporting a more complete part of the OpenGovIntelligenceframework.Existingtoolsavailableforthepurposewillre-usedwhereeverpossibleandnewtoolswillbedevelopedwhenrequired.
The next stage of the project will involve evaluation of the delivered components in use casesorganizedbythepilotpartners.
D3.2–ReportonOpenGovIntelligenceICTtools-firstrelease
Page39of39
References
[1]D1.1OpenGovIntelligencechallengesandneeds
[2]D2.1OpenGovIntelligenceframework-firstrelease
[3]D3.1OpenGovIntelligenceICTtools-firstrelease
[4]D4.1PilotsandEvaluationplan