oodida: on-board/off-board distributed data analytics for ...solution to this problem is oodida...

OODIDA: On-board/Off-board DistributedReal-Time Data Analytics for Connected Vehicles

Gregor Ulm∗

Fraunhofer-Chalmers ResearchCentre for Industrial Mathematics

Gothenburg, [email protected]

Simon Smith∗Fraunhofer-Chalmers Research

Centre for Industrial MathematicsGothenburg, Sweden

[email protected]

Adrian Nilsson∗Fraunhofer-Chalmers Research


[email protected]

Emil Gustavsson∗Fraunhofer-Chalmers Research


[email protected]

Mats Jirstrand∗Fraunhofer-Chalmers Research


[email protected]

AbstractA fleet of connected vehicles easily produces many giga-bytes of data per hour, making centralized (off-board) dataprocessing impractical. In addition, there is the issue of dis-tributing tasks to on-board units in vehicles and processingthem efficiently. Our solution to this problem is OODIDA(On-board/Off-board Distributed Data Analytics), which isa platform that tackles both task distribution to connectedvehicles as well as concurrent execution of tasks on arbitrarysubsets of edge clients. Itsmessage-passing infrastructure hasbeen implemented in Erlang/OTP, while the end points use alanguage-independent JSON interface. Computations can becarriedout inarbitraryprogramming languages.Themessage-passing infrastructure of OODIDA is highly scalable, facili-tating the execution of large numbers of concurrent tasks.

Keywords Distributed systems, Concurrent computing, Er-lang

1 IntroductionBig data in the automotive industry is of increasing concern,considering that connected vehicles may produce large vol-umes of data per hour. When dealing with a fleet of vehicles,centrally processing such data is impractical, if not infeasible.However, with OODIDA (On-board/Off-board DistributedData Analytics), which is a platform that facilitates the distri-bution and concurrent execution of real-time data analyticstasks in a heterogeneous system,we can conveniently process

∗Also with Fraunhofer Center for Machine Learning, Gothenburg, Sweden.

Permission tomakedigitalorhardcopiesofpartorallof thiswork forpersonalor classroom use is granted without fee provided that copies are not made ordistributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for third-party componentsof this workmust be honored. For all other uses, contact the owner/author(s).—, —, —© 2020 Copyright held by the owner/author(s).

EdgeOnboard computations

Computation manager

Task Task ...

Task handlers

User

Frontend App Backend App

Server App

Custom-code storage

Client App

Cloud

Offboard computations

Computation manager

Task Task ...

Assignment handlers

Sensor manager

Custom-code storage

Figure 1.OODIDA has a modular architecture. The backend,server, and client apps as well as the assignment andtask handlers have been implemented in Erlang. The userfrontend, computation manager and sensor manager havebeen implemented in Python.Modules or functions that carryout tasks can be implemented in an arbitrary programminglanguage. The sensor manager (bottom) interfaces with thevehicle’s CAN bus.

vehicle telemetry data as batches or pseudo-realtime streamsclose to the data source. This is a step towards facilitating ef-ficient big data analytics in vehicular networks. The majorityof the computational work is carried out on client devices(on-board), so-called edge devices, and only a supplementarypart on the server (off-board).

1

arX

iv:1

902.

0031

9v2

[cs

.PL

] 6

Feb

202

0

—, —, — G. Ulm et al.

OODIDAuses avirtual privatenetwork for communication.It connects data analysts with a large number of on-boardunits (OBUs). While our system could be used for generaldistributed data processing tasks, it has been designed fordata exploration and rapid prototyping in the automotive do-main, targeting a fleet of reference vehicles. An architecturediagram is provided in Fig. 1:m users interact with a centralcloud application that is connected with n client nodes. Themain feature of our system is distributing and concurrentlyprocessing decentralized data analytics tasks of arbitrary du-ration. We provide a JSON interface for external applications.In addition, error handling mechanisms address the issue ofintermittent availability of client devices.This paper presents the architecture and design of OOD-

IDA as well as use cases that illustrate how this frameworksolves real-world problems. It has been deployed to test vehi-cles and used for processing live real-time data. Due to legalagreements that prevent us frompublicly sharing source codeartifacts, we need to keep some details under wraps, however.From a conceptual perspective, OODIDA is noteworthy forapplying the paradigm of lightweight concurrent process-ing, via the programming language Erlang, to the automotivedomain for real-time data analytics.The remainder of this paper starts off with relevant back-

ground information in Sect. 2. Subsequently, we provide adetailed description of OODIDA in Sect. 3, with a particularfocus on how this system handles distributed concurrent ex-ecution of multiple assignments. As an intermission, Sect. 4shows examples of how our general-purpose system for dis-tributed real-time data analytics has been applied to variousproblems in the automotive domain.We substantiate the prac-tical usefulness of our systemwith an empirical evaluation inSect. 5, before concluding with an overview of related workin Sect. 6, which discusses various alternatives and why wedid not use them, and potential future work in Sect. 7.

2 BackgroundIn this section, we provide some background information onautomotive big data. First, we summarize challenges com-monly associated with big data, covering volume, velocity,and variety of data but also the issue of data privacy (Sect. 2.1).Second, we look at examples of decentralized data processing,covering real-world use cases in our domain (Sect. 2.2). Third,we highlight scalability, based on transparent distributionand concurrent execution on client devices and the server,as a key to solving automotive big data problems at the fleetlevel (Sect. 2.3). Fourth, we clarify relevant terms (Sect. 2.4).

2.1 Big Data ChallengesThe most immediate issue with data generated by connectedvehicles is its immense volume and velocity, potentially ex-ceeding 20 gigabytes per hour and vehicle [3]. It is impracticalto continually transfer suchvolumes of data to a central server

for processing, particularly at the fleet level. Communicationprotocols such as 5G and beyond will increase available band-with. However, data generation outpaces those increases, andvehicular networks cannot realistically be expected to be sta-ble. Thus, processing data on the edge and sending results incertain intervals is a more stable approach than continuallysending data, even amounts that could conveniently be trans-ferred via thenetwork, to a central server for processing.Withtime-critical information, transferring data to a central serverfor subsequent processing may not be viable as the overheadof data transmission negatively affects processing times com-pared to processing data close to its source. There is also theissue of the variety of big data due to collecting a multitudeof signals. Thus, the heterogeneity of data rules out a one-size-fits-all approach. An additional challenge, but one thatis externally imposed, is data privacy [1, 19]. A recent exam-ple is the General Data Protection Regulation (GDPR) of theEuropean Union [5], which places heavy restrictions on datahandling. On-board processing sidesteps data privacy issuesthat would have to be addressed with central data collection.

2.2 Decentralized Big Data ProcessingThemain purpose of OODIDA is tomake automotive big dataeasier to handle by processing it closer to its source, withthe goal of limiting the amount of data that needs to be pro-cessed on a central server. To illustrate this idea, we presentcategories of practical use cases.We start with use cases where the central server only col-

lects results from clients, without further off-board process-ing. A straightforward example is filtering on the client, e.g.monitoring or anomaly detection. Retaining only values thatfulfill a given predicate may allow us to discard most data. Asimilar case is sampling, where a randomly chosen subset ofdata is retained. However, such tasks may require additionalprocessing. A related example is MapReduce [4]. Considerthe standard MapReduce problem of determining word fre-quency: clients first map eachwordw of the input to the tuple(w,1). Afterwards, they reduce all those tuples to their respec-tive count c , i.e. (w,c). This is the data that is sent to the server,which, in turn, reduces all incoming data to the final value(w,c ′), where c ′ specifies the total count of w in the input.Large-scale MapReduce is hardly the goal of OODIDA, butsome of its use cases fit this paradigm very well (cf. Sect. 4).

A more complex use case is federated learning [13], whichis an example of distributedmachine learning. In this case, theserver maintains a global machine learning model, which issent to clients. Each client trains themodelwith local data andsubsequently sends the updated model to the server, whichcomputes an average of the received updated local models. Inpractice, there is likely a limited number of iterations, basedon the total errorof theglobalmodel.Once it falls belowaspec-ified threshold value, training is considered complete. Subse-quently, the final global model is sent to the user. Federatedlearning is an active area of research,with a particular interest

2

OODIDA: On-board/Off-board Distributed Real-Time Data Analytics —, —, —

from the mathematical optimization community. Examplesinclude work on general distributed optimization [9] and thealternating direction of multipliers (ADMM) [10]. Relevantfor our case is that large-scale optimization on edge devicesusing deep federated learning has been shown as feasible [6].Data processing tasks could be one-off with a fixed dura-

tion or they may run indefinitely long. In some scenarios, itmakes sense to repeatedly perform narrowly defined tasks,for instance getting a status update every n seconds, whichcan be modeled as an assignment with multiple iterations.Federated learning is merely a more complicated example ofthis approach as the results of iteration i are used as inputof iteration i+1. Another useful addition is the emulation ofstream processing by iteratively processing batches of data.The shorter the iterations are, the closer we get to pseudo-realtime stream processing.

2.3 Scalability at the Fleet LevelIn order to handle big data problems at the fleet level, we needan effective means of task distribution. Of course, we alsohave to have the ability to issue multiple assignments to over-lapping subsets of clients. This necessitates that client devicesare able to concurrently execute tasks.Meanwhile, the centralserver has to remain responsive even as the workload on thesystem increases. That being said, there is a clear limitationto the amount of work the system needs to perform as weare not targeting a large fleet of production vehicles. Instead,the goal is to execute an analytics platform on a private cloudthat connects to OBUs in test vehicles, so-called referencevehicle computational nodes (RVCNs). This facilitates rapidprototyping of data analytics methods which may eventuallybe executed on OBUs in production vehicles.

2.4 Some TerminologyAs this paper does not exclusively address a computer scienceaudience, we would like to clarify a few relevant terms. Theactormodel is amathematicalmodel for describing concurrentcomputations, developed by Hewitt [8]. In it, independentactors send and receive messages, making it straightforwardto model concurrent computations. The most prominent lan-guage related to the actor model is Erlang, in which we haveimplemented themessage-passing infrastructure of OODIDA.There is some conceptual confusion surrounding the termsconcurrent and parallel, however. While even computer scien-tists may use them interchangeably, we follow, for instance,Harper [7] and Marlow [12], and understand concurrencyas the simultaneous execution of nondeterministic computa-tions and parallelism as the simultaneous execution of deter-ministic computations. An example of the former is lock-freegarbage collection, one of the latter is matrix multiplication.Aswearenot going to shareErlang source codeartifacts, thereis no need to explain syntactic details. However, we do fre-quently use the terms processes and process identifiers (PIDs).Erlangprocessescommunicatewitheachotherbyexchanging

messages. By knowing the PID of a process, another processis able to send messages to it. Incoming messages are storedin a process mailbox and handled in the order they arrive in.

3 SystemDescriptionIn this section we describe the OODIDA platform in detail,starting with an overview (Sect. 3.1) and execution scenar-ios (Sect. 3.2). The main part is the discussion of the centralcloudapplication (Sect. 3.3).Twobriefnoteaddresseerrorhan-dling (Sect. 3.4) and the back-end on client devices (Sect. 3.5).

3.1 OverviewOODIDA is a platform for distributed real-time analytics forthe automotive domain. Its input is the specification of anassignment, that, among others, specifies the task to be car-ried out by the selected subset of clients as well as an optionaltask for the central cloud server. The task is a specification ofthe actual computation, while an assignment also containsinformation on how tasks are to be carried out. Users specifyassignmentswith thehelpof anexternal library that facilitatestheir creation and verification. On client devices, a externalapplications perform analytics tasks. These can be third-partylibraries. Our system is able to concurrently execute multipleapplications on client devices. The number of users, clients,and tasks is only limited by the computational power of thehardware the system is executed on. Referring to Fig. 1, therelevant parts for the subsequent presentation of the mes-sage passing infrastructure are the backend application of theuser (u), the server application (b) as well as the assignmenthandler (b ′) of the cloud, and the client application (c) withits accompanying task handlers (c ′) on the edge. The sensormanager provides an interface to a third-party applicationthat reads data from the vehicle’s CAN bus. The ability toexecute user-defined code is hinted at in the this figure, butdiscussed elsewhere (cf. Sect. 6).

3.2 Execution ScenariosIn this subsection we describe various usage scenarios. Thefocus is not on specific assignments, but on their distributionand execution instead. We look at three scenarios: one finitetask,multiple finite tasks, andmultiple indefinitely long tasks.For simplicity, we consider only one user and three client de-vices, although there could be arbitrarily many of both. Simi-larly, our system does not restrict the number of concurrentlyexecuted assignments. The used short-hands (u, b, etc.) forthe various components were introduced in Sect. 3.1. The de-scription is based on whole-fleet assignments. A closing notehighlights an important difference for sub-fleet assignments.The most basic scenario consists of submitting an assign-

mentwith onefinite task for all clients (cf. Fig. 2a): cloudnodebhas spawned an assignment handler b ′, which divides the as-signment into tasks and sends a corresponding message toeach of the client nodesx ,y, andz. In general, each client node

3

—, —, — G. Ulm et al.

u b b′

x′x

y

z

y′

z′

(a)Whole-fleet assignment (1 task/client)

u

b′

x′

x′

b

x

y

zb′

y′

y′

z′

z′

(b)Whole-fleet assignment (2 tasks/client)

Figure 2. These two basic execution scenarios start with user nodeu sending an assignment to cloud node b, which spawnsan assignment handler b ′ that divides the assignment into tasks. Red nodes are permanent, while blue nodes are temporary.In (a) each each of the client nodes x ,y and z spawns a task handler, i.e. x ′,y ′, and z ′, that interacts with external applicationson the client device. In (b) no separate task handler is spawned by client nodey, as it is not in the set of addressed clients.

ci runson itsownOBU.Uponreceivinga taskdescription fromb ′, each client process ci spawns a task handler c ′i , which com-municates with an external application a. Upon completionof the task, the results are sent from a tob ′ via c ′. Nodeb ′ per-forms optional off-board computations. Afterwards, resultsare sent tob and, finally, to the user nodeu. Executing just onefinite task at a time on some or all of the available clients hassome practical utility, butmany use cases that necessitate con-current execution ofmultiple assignments. For instance, a clientthat is training amachine learningmodelwith local data couldconcurrently monitor a range of sensors for critical values forthe purpose of performing anomaly detection tasks. Figure 2bshows a whole-fleet assignment with two concurrent tasksper client. The cloudprocess has spawned two instances of theassignment handler b ′, i.e. one handler per assignment. Eachof the three client processes has likewise spawned one taskhandler per received task, which means that all client nodesare processing both tasks. Having illustrated these two cases,it is easy to see how additional concurrent assignments arehandled. There are also indefinitely long assignments, whichare a variation of the previous scenarios as tasks could as wellrun for an unspecified duration. This is the case when execu-tion is not tied to a fixed number of iterations but to a stoppingcriterion, such as reaching a desired threshold value for thevalidation error of an artificial neural network. Tasks can alsobe set up to run until an interrupt from the user has been re-ceived. It is also possible to set up an assignment consisting ofan indefinite number of iterations. This generates a stream ofdata as intermediate results are produced after each iteration.When a sub-fleet is addressed, client nodes that do not

carry out any task are isolated from the system. OODIDAuses targeted messages that are sent only to affected clients,as opposed to broadcasting to all clients. A task handler isonly spawned when there is a task to perform. Furthermore,

for each concurrent assignment, a subset can be chosen ar-bitrarily. It is irrelevant for the system if they overlap or not,albeit there are use cases, such as A/B testing, for which theuser should select non-overlapping subsets.

3.3 Implementation DetailsThis subsection details the message-passing infrastructureof OODIDA, and how it facilitates scalability with concur-rent data analytics workloads. The sequence diagram in Fig. 3captures the simple case of non-concurrent execution withone user, one client, and one task. In the description below,the fully concurrent case is described, however. The systemconsists of one central cloud nodeb,m user nodes, andn clientnodes that are all intended to communicate via virtual privatenetwork. Nodeb thus provides a connection between the user,i.e. a data analyst specifying assignments on one end, anda multitude of OBUs on the other. The underlying messagepassing infrastructure has been implemented in Erlang/OTP.Our presentation mainly focuses on this part of the system.We address the external user application only in passing, butprovide some details on the client application (cf. Section 3.5).In a production system, eachOBU is supposed to execute onlyone client node. This limit is not enforced, which facilitates analternative operational mode, i.e. simulating a large numberof client devices on a more powerful workstation.

Assignment specification. Assignments are JSON objects,containing key-value pairs. Among others, they indicate thechosen on-board and off-board computations and the signalsto read. The sampling frequency is specified inHz. Dependingon the chosen computations there may be contextual param-eters to set. The assignment specification also indicates thenumber of samples to take, e.g. 1000 samples at 1 Hz, andwhether intermediate results are required. The latter is pos-sible by setting a number of iterations. Lastly, we need to

4


User Cloud Assignment Handler Client Task Handler Worker

assignmentspawn handler

assignmentsplit assignment into tasks

taskspawn handler

tasktask

onboard compute

onboard resultonboard result

terminate

aggregate onboard results

offboard compute

final resultterminate

final result

Figure 3. Simplified sequence diagram of OODIDA. The system handles multiple users which can all send multiple assignmentsthat can address arbitrary subsets of clients, whereas this diagram only shows one user and one client. On top clients executetasks concurrently. The system can also perform iterative work on the task and assignment level. In contrast, this diagramshows an assignment consisting of a task that contains one iteration.

choose clients. In addition, there is a keyword for whole-fleetassignments. For sub-fleet assignments, the user can eitherchoose a number of clients or a list of IDs, not PIDs, of clients.

Server loop. Oneachuserworkstations, a front-end f is avail-able that interfaceswith a Python library. It generates and val-idates assignment specifications in the JSON format. Once anassignment specification has been generated, it is processedvia the user nodeu and sent via the network to nodeb, whichis executed on an internal private cloud. There, the executionof a single assignment is described as the following loop. Thesteps are prefixed by the node (b or b ′) that executes them.

1. b: Await assignment specification2. b: Read assignment specification3. b: Spawn assignment handler b ′4. b ′: Divide assignment into tasks5. b ′: Distribute tasks to the chosen subset of client nodes6. b ′: Await task completion; clients send results to b ′7. b ′: Upon completion, aggregate results of client devices8. b ′: Perform optional off-board processing and forward

results to b9. b: Send assignment results to user nodeu10. b: Go to step 1However, b is ready to process incoming assignment speci-

fications (steps 1 to 3) or forward results from the assignment

handler b ′ to the user (step 9), regardless of the status of anyother active assignment because b ′ covers steps 4 to 8 con-currently. Furthermore, b can spawn an arbitrary number ofassignment handlers b ′, which independently await resultsand process them further. For each new incoming assignment,b spawns a new assignment handler. OODIDA can handletasks that run for an unspecified amount of time, which canbe manifested in multiple ways. For instance, an assignmentcould consist of one iteration and terminate based on userintervention. Alternatively, it could consist ofk rounds. Theserepeat steps 4 to 8 of the hypothetical loop abovek times. Theduration may be left unspecified, implying that the assign-ment runs until a predefined stopping criterion has been met.

User node (incoming). Every instance of user node u hastwo main components, a process user that communicateswith nodeb and a process await that reacts to input sent fromthe front-end f , such as assignment specifications or statusrequests. We ignore the latter in our presentation, however.When an assignment specification arrives, the process awaitdecodes it, extracts all relevant information, turns it into anErlang tuple tagged as assignment, and sends it to the pro-cess user for further processing. Having a separate processawait dedicated to handling incoming assignments enablesthe process user to remain responsive at all times.

5

—, —, — G. Ulm et al.

Cloud node (incoming). The process user forwards the rel-evant content of an assignment specification to the processcloud of node b. Its state contains the list of currently activeclients, pairing their IDs and PIDs.1 The goal is to send taskspecifications to the chosen client nodes. However, we can-not block the execution of the process cloud until we havereceived a result because the user may want to issue anotherassignment in the meantime. This necessitates concurrentexecution, which is addressed as follows. The process cloudawaits assignments fromtheprocessuser. If there is currentlyno client process available due to no vehicles having beenregistered in the system, the assignment is dropped (and theuser informed of that outcome). Otherwise, the process cloudspawns an assignment handler b ′ for the current assignmentthat sends task specifications to the chosen client nodes andmonitors task completion. Each incoming assignment leadsto spawning a separate node b ′.The assignment handler is a key component for enabling

concurrent processing of assignments. It receives as argu-ments the list of currently active clients, the subset of clientscspecified by the assignment, the assignment configuration,and the PID of the process cloud. It first determines the PIDsof the clients specified in the assignment configuration andafterwards sends the specification of the on-board task toeach client node. The next step is to block execution for thisinstance ofb ′ until it has received results from all client nodes(cf. Sect. 3.4 for error handling). This is possible due to thecloud process spawning a new assignment handler processfor each incoming assignment.We continue at this point afterdiscussing the client node.

Client node (incoming and outgoing). The client node cwaits for task descriptions to arrive. There is one processclient per client node. Whenever this process receives a tu-ple that is tagged as a task, the task specification is extractedand turned into a JSON object.2 This is followed by spawninga task handler c ′, which communicates with an external ap-plication a that carries out the specified task. For each newincoming task, a new task handler c ′ is spawned. Thismirrorsspawning a separate assignment handler b ′ for each new as-signment on the cloud node b and it likewise ensures that theclient node remains responsive. The first objective of c ′ is toforward the task specification to a and await the completionof that task. It is possible to have different external applica-tions process different tasks, but we only consider the case ofone external application. Most tasks consist of reading valuesfrom a set of sensors for a given interval and performing acomputation on it. The result of this computation is sent to c ′.

1IDs are static,while PIDsmay changewhena client node reconnects. For thisreason, the user addresses client nodes with their IDs instead of their PIDs.2Decoding is not performed by the task handler because some tasks arestatus requests or special instructions, e.g. a shutdown command. Theseare handled by the client node; to simplify the architecture of the system,the client node processes all incoming JSON objects.

An assignment may consist of multiple iterations. How-ever, this is hidden from the client as it treats each iterationin isolation. In other words, the client is both oblivious towhether it performs an assignment with multiple iterationsor just one, and also to the iteration it is currently working on.Furthermore,c ′ does not communicatewithb at all. Instead, itdirectly sends results to b ′. This reduces the communicationand computation overhead of the client node. After sendingthe result of the task to b ′, c ′ terminates.

Cloud node (outgoing). Once b ′ has received the resultsfrom all involved task handlers c ′i , it performs the specifiedoff-board computation. If the current assignment consists ofmultiple iterations, b ′ calls itself recursively. This leads toone more round of sending out task specifications to clientnodesci andwaiting for the results.On the other hand, if thereare no iterations left, the final result is sent to b. Afterwards,b ′ terminates. This is the end of a cascade effect: First, all taskhandlers c ′i send the results to b

′ and terminate themselves.Then, b ′ performs the off-board computation, sends the re-sults to b, and terminates itself as well. Lastly, b sends theresults of the assignment to the user nodeu.

User node (incoming). On the user node, the default ap-proach is that a JSON object with the result of the assignmentis sent to the front-end application f . These results can befurther manipulated in f , for instance by integrating theminto computational workflows where the results of one as-signment are used as input for a later one (cf. Sect. 4).

3.4 Error HandlingSo far, we have made the assumption that all assignmentscomplete and that all client nodes are available all the time.Of course, it is overly optimistic to assume that the networkis always stable. To account for real-world disturbances, weuse various Erlang/OTP mechanisms for fault-tolerance suchas links and monitors. If the connection to a client node hasbeen lost, it is given a set duration of time to reconnect. Aslong as a disconnected client node is alive, it actively tries toreconnect to the cloud node. Should this fail, its results areexcluded from the current iteration of the assignment. For allremaining iterations of the assignment, OODIDA attemptsto send a task description also to clients that were lost duringprevious iterations. This enables them to rejoin this assign-ment in case they have become available again. The errorhandling mechanisms are preliminary (cf. Sect. 7).

3.5 Client Back-EndThe external client application, referred to as a worker inFig. 3, has been implemented in Python, but there is also afunctional Go prototype. These applications consume tasksspecifications, perform computational work, and send theresults as JSON objects to their task handler. Workers areintegrated with a third-party application that provides accessto the vehicle’s CAN bus data. OODIDA could be run with

6


Figure 4. Geo-fencing usecase screenshot (best viewedin color). This shows how the user front-end facilitatesthe intuitive definition of geo-fences as a polygon can bedrawn graphically. A geo-fence works in conjunction withanalytics code that processes the detected vehicle coordinates.Geo-fences can be arbitrarily complex, however. In thosecases, a definition in code may be preferable.

multiple applications on client devices. An example is delegat-ing stream processing tasks to an external stream processingengine, e.g. Apache Edgent.3

4 Practical Use CasesOODIDA is a general-purpose system. The examples beloware intended to illustrate examples of how this system can beused for practical data analytics tasks. We cover straightfor-ward tasks such as random sampling and anomaly detection,but also more complex ones, such as predictive maintenance,MapReduce-like computations, and federated learning. Lastly,we show how a data analyst can define control flows that usethe results of an assignment as the input for a subsequentone. These are all implemented examples, albeit predictivemaintenance is a proof-of-concept at this point.

Sampling. One of the challenges of big data is its immensevolume. A straightforward approach to handling it is to col-lect a random sample that amounts to only a fraction of theoriginal data. The assumption is that the random sample hasproperties similar to the original data. In order to do this withOODIDA, the data analyst needs to set up a sampling taskwith a list of signals and specify the amount of data in percentthat should be collected. This is the on-board task. A fittingcorrespondingoff-board task is the collectionof values, pairedwith the ID of the client they originated from.

Anomalydetection. The goal of anomaly detection is to findoutliers indata, e.g. determining if a givendatapoint is outsidethe range a machine learning model predicts. Concretely, as-sume we have a probability distribution of our data and wantto detect values that are considered statistically improbable.

3The homepage of the Apache Edgent project is https://edgent.apache.org/.

Thus, the data analyst selects a model and sets the desired pa-rameters as well as the list of signals to monitor. This definesthe on-board task. A fitting off-board task is collecting the de-vice IDs of the targeted OBUs and their corresponding values.

Predictive maintenance. Using an existing machine learn-ing model that is tied to a list of signals, it is possible to detectcomponents that may be in need of maintenance. As a sim-plified example, consider a model for braking pad lifetimethat uses braking pad characteristics, the speed of the vehicle,vehicle weight, current precipitation, force of braking androad condition, which is a lookup based on the current GPScoordinates of the vehicle. The on-board task is detecting ifbraking efficiency deviates from expected values. The corre-sponding off-board task consists of collecting values and IDsof vehicles that deviate from the model.

Geo-fencing OODIDA has been designed for use with com-mercial vehicles, which entails several use cases for geo-fencing, such as intrusion detection. Geo-fences can be set upgraphically in the user interface (cf. Fig. 4) or defined in code.Dependingon the complexity of the use case, the computationusing a geo-fence could be based on predefined methods ordefined as custom code (cf. Sect. 7). Conceptually, use casesbuilt upon geo-fencing are examples of anomaly detection.Corresponding code runs constantly on all targeted vehiclesand when a corresponding event triggers, an alert containingthe vehicle ID and relevant additional data is sent from theclient to the server and forwarded to the user.

MapReduce-like tasks. While running large-scale MapRe-duce jobs on a fleet of connected vehicles may sound out-landish, there are nonetheless use cases where a MapReduce-like approach is helpful. For instance, consider driver-specificspeed-distribution histograms. The corresponding on-boardtask consists of mapping velocity signal values to a bin value,resulting in (bini ,1) and subsequently reducing these pairsto their sums (bini ,sum) as well as normalizing them. Fittingoff-board task are collecting those histograms or averagingthem to one histogram of the entire fleet of vehicles.

Federated learning. Amuchmore complex use case is feder-ated learning, an approach to machine learning where clientdevices train a model on local data. One round consists ofsending a global model to a subset of clients. Afterwards,client devices train the model with local data and send theirupdated local model to a central computer that averages thereceived local models to a new global model. This is repeateduntil either the error drops below the threshold value ϵ , i.e.the desired maximal validation error of the machine learn-ing model, or the maximum number of iterations is reached.In OODIDA, the assignment handler b ′ that is spawned bycloud node b initializes the model with arbitrary parametervalues and afterwards sends it to the chosen client nodes fortraining. After concluding this training pass, the task han-dlers send the updated local model to b ′. Subsequently, b ′

7

https://edgent.apache.org/

—, —, — G. Ulm et al.

creates a new global model based on the updated local modelsand evaluates it with a validation data set. If the resultingerror exceeds ϵ , training continues by sending the new globalmodel to clients for training, and so on. Otherwise, trainingis considered complete and the model is sent to the user.

Controlflows. Above,wehavediscussed isolated tasks.How-ever, the user front-end also facilitates control flows, wherethe results of one assignment serve as the input of a sub-sequent one. For example, consider the case of predictivemaintenance, for which the user issues a whole-fleet assign-ment. We may start with a model that does not require manyresources, does not report false negatives, but has a smallrisk of reporting false positives. After this assignment hasconcluded, we filter out the client IDs of all positives, which islikely a fraction of all active client IDs. Thenwe deploy a com-putationallymore demandingmodel that has a reduced risk ofreporting false positives to those clients. The previous exam-ple uses conditional branching. Other control flows may bebased on loops that iterate through a sequence of whole-fleetassignments. Such tasks can be set up to concurrently targetnon-overlapping subsets of clients, which enablesA/B testing.

5 EvaluationThekeypart ofOODIDA is itsmessage-passing infrastructure,whichwas designed to facilitate concurrentworkloads.As theworker applications of our system can be arbitrarily replaced,we do not include experiments on the performance of edgedevices as these would be little more than evaluations of thechosenhardware. Furthermore, asmanyuse casesofOODIDAinclude third-party libraries such as scikit-learn, evaluationsbased on such use caseswould be equivalent to evaluating thelibraries themselves and reveal little of the scalability of OOD-IDA itself. Indeed, themessage-passing infrastructure is taxedregardless of howmuch computational power has been usedon edge devices to arrive at certain results. To give a simpleexample, the size of amessage from the client to the cloud thatcontains an average of a given number of vehicle speed valuesis unaffected by the preceding work that was performed onthe client to produce that result. A result that was arrivedat by taking one value as its input requires as much space,i.e. a single floating-point number, than one that resulted fromaveraging a billion input values. Consequently, the scalabilityand responsiveness of the message-passing infrastructure ofOODIDAwas evaluated based on tests that stress the abilityof a cloud server as well as clients to handle concurrent work-loads. The primary focus is on quantifying the ability of thesystem to remain responsive while concurrently processingassignments. Below, we describe our hardware and softwareenvironment (Sect. 5.1) and the experiments we conducted(Sect. 5.2), which is followed by their results (Sect. 5.3) and adiscussion (Sect. 5.4).

5.1 Hardware and Software SetupWe used three quad-core PC workstations as well as Rasp-berry Pi client devices. Workstation 1 contains an Intel Corei7-6700k CPU (4.0GHz), workstation 2 and 3 an Intel Core i7-7700k CPU (4.2GHz). These workstations are equipped with32GBRAMeach, of which 23.5 GBweremade available to thehosted Linux operating system. They run onWindows 10 Pro(build 1803) and execute OODIDAonUbuntu Linux 16.04 LTSin VirtualBox 6.0 (workstation 1) or VirtualBox 5.2 (worksta-tions 2 and 3). For the client stress test, we used a Raspberry Pi3 Model B with a quad-core BroadcomARMv7 CPU (1.2 GHz)and 1 GB RAM. It runs Raspbian Linux 9 with kernel 4.14.52.This CPU is similar to the CPU of the RVCNs our industrialcollaborators use.

5.2 ExperimentsOODIDA is designed to run in a heterogeneous system,whereapowerful central server orchestrates computationally boundclient devices. Our experiments therefore explored the scal-ability of (a) a single client device and (b) the entire system.For (a) the mode of connection to the system is irrelevant be-cause the client does not communicatewith the cloud process.However, for (b) the difference between Ethernet and wire-less connectivity affects the maximal network throughput.Ethernet connectivity is not indicative of real-world usage ofOODIDA, but it allows us to estimate the peak performanceof our system.

Client devices. In order to evaluate the scalability of the cho-senclienthardware,weexecutedOODIDAwithoneuser.Userandcloudnodeswere executedonworkstation1,while aRasp-berry Pi device executed one client node. We measured CPUand RAM utilization as the number of concurrently executedlightweight sampling tasks increases. For this experiment, wechose a sampling rate of 1 Hz of input consisting of syntheticCAN bus data. We used Python 3.5.3 and Go 1.11.3.

System. The goal of thewhole-system experiment is to deter-mine how OODIDA reacts to surges of activity, with a focuson how large concurrent workloads affect the response timeof the cloud process. We looked at this issue from two angles:(a) incoming assignments from the user side as well as (b)incoming results from clients. In (a) the user and cloud nodeswere executed on workstation 1. We issued a batch of 100assignments and measured how the response time, i.e. thedifference between assignment creation and processing, isaffected as subsequent assignments are generated. A surgeof such an extent is unrealistically large, so this approach ex-plores the theoretical limits of our system. In (b),workstations2 and 3 simulated 50 clients each, for a total of 100, for whichwe issued an assignment with an indefinite number of taskiterations. In three different scenarios, they execute either 1, 5,or 10 tasks concurrently per client. We collected 100,000 pro-cessing times of a full task iteration per scenario. A full task

8


0 50 100 150 200 250 300Number of tasks

0

20

40

60

80

100CP

U &

Mem

ory

utiliz

atio

n [%

]MemoryCPU

(a) Client implemented in Python

0 50 100 150 200 250 300Number of tasks

0

20

40

60

80

100

CPU

& M

emor

y ut

ilizat

ion

[%]

MemoryCPU

(b) Client implemented in Go (proof-of-concept)

Figure 5. Concurrency stress test of the Python and Go client applications, showing CPU and RAM utilization while increasingthe number of concurrently executed sampling tasks at a sampling rate of 1 Hz of a one-dimensional input stream. The plotsshow the mean and standard deviation of five runs. Over 250 tasks can be processed concurrently in Python before hittinghardware limitations. The spike in CPU utilization at ∼90% memory is due to memory swapping. In contrast, the Go clientuses ∼20% of the available RAMwith the same workload. At 300 concurrent tasks, CPU utilization is ∼18% with Python and∼10% with Go. Python forks duplicate some memory, which is not an issue with lightweight goroutines in Go.

iteration startswhen the assignment handler splits the assign-ment into tasks for clients and ends when it has collected theresults of all clients. The round-trip time between cloud andclient workstations is included as well, which sidesteps clocksynchronization issues between workstations. In order to putmore demand on the cloud node, we implemented dummyclientprocesses inErlang thatproducea result immediately af-ter receiving a task specification, leading to a sustained surgeof incoming results. Using 100 clients may seem low, but thisis already a multiple of the number of RVCNs OODIDA istargeting. Also, the number of clients is not nearly as relevantas the number of incoming results. Running 10 tasks concur-rently on one client is arguably excessive. Yet, the workloadof 100 clients running ten tasks is numerically equivalent to,for instance, 500 clients concurrently executing two tasks. Ofcourse, it is also the case that real-world taskswould normallynot immediately produce a result. Thus, again, the stress weare putting our system under far exceeds real-world demands.

5.3 ResultsClient devices. We determined RAM and CPU utilizationwith increasing numbers of concurrently processed assign-ments, the results of which are shown in Fig. 5. These plotsstart with zero assignments, showing that the base utilizationof the Python client application is less than 1% of the CPUand 20.5% of RAM.With the Go client application, base CPUutilization is likewise less than 1%,while base RAMutilizationamounts to18.8%. Fromthatpointonward,RAMconsumptionincreases linearly in Python, while CPU utilization fluctuates,but increases mostly linearly. Past around 250 assignments,the Python client becomes less responsive. At ∼90% of RAM

utilization, CPU utilization spikes due to memory swapping.CPU utilization is close to 20% at 300 concurrent tasks. In con-trast, the Go client application shows a very modest increaseof RAM consumption to ∼20%, while the CPU load increaseda lot more slowly to ∼10%.

System. The results of the evaluationof the systemare shownin Fig. 6. The median and standard deviation of five runs inFig. 6a indicate thatevenwhensubmittinganexcessively largebatch of 100 assignments, the difference between assignmentcreationby theuser and assignment processingon the cloud isless than 40ms on average, with a slight upward trend. The re-sultsof theevaluationofa surgeof incomingwork fromclientsand its effect on assignment iteration completion times of thecloud process are shown in Fig. 6b and Table 1.We see that thesystemremainshighly responsiveevenas theworkload rampsupdrastically. In all three scenarios themeasurements approx-imate a log-normaldistribution.With1, 5, and10concurrentlyexecuted tasks per client and 100 clients, themean completiontimeand standarddeviationof 100,000 iterations,withmedianand 95th percentile value in parentheses, are 2.7±2.1ms (2ms,6ms), 5.3±5.0ms (4ms, 12ms), and 8.2±6.1ms (6ms, 20ms).

5.4 DiscussionClient devices. The Python client application exhibits a con-stant increase of RAM consumption that outpaces CPU uti-lization.Newprocesses are forks that share somememory.Wealso tried spawning processes, but this led to an even greaterincrease in RAM utilization due to not sharing any memory.Another alternative would have been using threads, but weruled this option out from the start. Despite the drawbacks

9

—, —, — G. Ulm et al.

0 20 40 60 80 100Number of issued assignments

0

10

20

30

40

50

Resp

onse

tim

e [m

s]

(a) Surge processing times of assignments (user)

0 20 400

5000

10000

15000

20000

25000

30000

35000

40000

0 20 40 0 20 400.0 0.2 0.4 0.6 0.8 1.0Roundtrip time [ms] with 1, 5, and 10 concurrent tasks

0.0

0.2

0.4

0.6

0.8

1.0

(b) Processing times with 95th percentiles (100 clients)

Figure 6. The stress test of the OODIDA cloud node shows that (a) it can easily handle a surge of incoming assignments.Processing times, based on five runs, which are illustrated by mean and standard deviation, only marginally deteriorate. In(b) it comfortably handles large numbers of concurrent assignments. The histograms show the processing times of 100,000results per scenario, measured from assignment creation on the cloud until the reception of results from the client.

Table 1. Results of the system stress test (cf. Fig. 6b) with100 clients performing 1, 5, and 10 concurrent tasks withan indefinite number of iterations. The data are based on100,000 completed assignment iterations each. All entriesare given in milliseconds.

Concurrent Tasks/Client 1 5 10Mean 2.7 5.3 8.2Standard deviation 2.1 5.0 6.1Median 2 4 695th percentile 6 12 20

of forked processes in Python, the number of (simple) assign-ments that can be executed concurrently is rather high, farexceeding 200. Thus, even devices as resource-constrained asthe oneswe used are powerful enough for complex real-worldscenarios.Again, the issue is notwhether theunderlyinghard-ware can execute certain tasks, which are highly dependentonparticular use cases andhardware capabilities, butwhetherthe OODIDA client can remain responsive as the number ofconcurrent tasks on an OBU increases.

System. The main takeaway of our system tests is that OO-DIDA is able to handle concurrent workloads easily. We havefound that congestion due to incoming assignments fromusers is a non-issue. Even when a batch of 100 assignmentsarrives at once, which is a figure that is far above real-worldrequirements, the cloud node keeps up very well. The slightupward trend observed in the data as the number of assign-ments increases is practically irrelevant with a response timeofaround40ms in themean.Whensimulating theworkloadofan, as of yet, unrealistically largenumberof clients, the system

remains highly responsive. As the number of incoming re-sults increases, due a greater concurrent workload on clients,iteration completion times exhibit a modest deterioration. Tosingle out the most extreme case: even with 1000 incomingresults per iteration — 100 clients executing 10 tasks concur-rently for 100 iterations — the 95th percentile is at only 20 ms.Thus, the bottleneck of the system is the performance of clienthardware, considering the workload its message-passing in-frastructure is able to handle. Furthermore, the central cloudhardware is a regular workstation with a modest number ofCPU cores. Thus, we are quite far from reaching hardwarelimits on that front even if system demands increased by or-ders of magnitude. We would also like to reiterate that theseresults hold regardless of the work that is performed on theclient as it was excluded from our measurements.

6 RelatedWorkWhenwe began developing OODIDA in 2016, there was nomature open-source data analytics framework available thatcould easily be modified for the demands of the automotiveindustry. To our knowledge, at the conclusion of this projectin early 2020, this still holds. In addition, aswould be expected,there isnoopen-sourcedataanalytics framework for analyticsin vehicular networks available that could be usedwith the ve-hicles of our project partners. In themeantime, potential com-mercial alternatives such asNVIDIA’s EGXorAmazon’sAWSIoTGreengrass have been released. However, even if they hadbeen available in 2016, they could not have been used for avariety of reasons, such as vendor lock-in, total cost of owner-ship, or the problem that some of the data OODIDA processesis too sensitive to be stored on servers owned by a third party.

10


The problem of using networked devices for data analyt-ics has been tackled from different angles. Google’s MapRe-duce [4] uses clusters of commodity PCs for batch-processingof large-scale finite tasks. An implementation of MapReducein Erlang/OTP and Python, DISCO, was described byMund-kur et. al [14].WithMapReduce, a master process keeps trackof task completion. While the master may supervise mul-tiple assignments, worker machines are only assigned onespecific task along with a corresponding batch of data. InOODIDA, data is generated on clients. They furthermore con-currently execute a multiple of assignments, which could bebased on data batches as well as data streams. Apart fromthose conceptual differences, MapReduce was designed towork with commodity PCs in a data center instead of dis-tributed computationally-bound edge devices. However, dueto its inherent flexibility, our system can performMapReduce-like tasks as well (cf. Sect. 4).As some of our use cases require the ability to process in-

finite data streams, part of our work could be seen as relatedto the many stream processing platforms that emerged as aresponse to limitations of MapReduce. Of course, the focusshould not necessarily be onMapReduce, as there are decadesof research behind stream processing [18]. OODIDA can per-form complex stream processing tasks, similar to use casesthat are tackled in systems like Apache Storm [20] or ApacheSpark [23], but with the caveat that our system is intended toorchestrate CPUs of limited computational power instead ofdata centers housing industry-grade server hardware. How-ever, using our system for stream processing would be ratherlimiting. While one could write, for instance, an applicationfor filtering values from an infinite data stream, it would bemore practical to use an external stream processing engine(SPE) instead, but preferably one that has been designed forthat kind of hardware, and interface it with the client node.Due to hardware limitations, using standard SPEs like Sparkwas infeasible due to their comparatively steep performancerequirements. One exception is Apache Edgent, which targetsresource-constrained devices. Nonetheless, building a clientapplication on top of Edgent seemed to limiting as our frame-work targets general-purpose analytics task. The ApacheEdgent project has been retired in the meantime, which illus-trates one of the risks of wanting to use available off-the-shelfcomponents for internal projects, i.e. the lack of control overdevelopment. Furthermore, even if Apache Edgent was stillactive, it would arguably be difficult if not impossible to getthe project maintainers to make changes that accommodateour needs. As a consequence, it would be necessary to main-tain a local fork, which would eventually erode the benefitsof using an existing solution.OODIDA emerged from ffl-erl, a framework for feder-

ated learning in Erlang [22]. The main conclusion of thatproject was the general feasibility of using Erlang/OTP forlarge-scale task distribution. However, it also confirmed that

it is not prudent to rely entirely on Erlang/OTP as the perfor-mance for numerical computations may not be favorable. Inaddition, there are practical limitations, such as the limitednumber of skilled Erlang/OTP programmers on the labormar-ket or the fact that extensive machine learning libraries existforotherprogramming languages, suchas thePython librariesscikit-learn [16] andKeras [2]. Both aspects, but especially thelatter one, point to the importance of interoperability and thebenefits of providing a language-independent JSON interfacefor external applications.We furthermorehaveusedOODIDA,or derivatives of it, for research in machine learning. This in-cludes a performance evaluation of our own implementationsof various federated learning algorithms [15] as well as ongo-ing work on federated learning of neural decision forests [17].

7 FutureWorkOODIDA is a fully functional prototype that has been de-ployed to both stationary and wirelessly connected OBUswith recorded CAN bus data in an experimental setup as wellas prototype vehicleswith live data.Deploying to a larger pop-ulation of prototype vehicles, followed by a roll-out to a smallfleet of commercial vehicles, is an obvious next step. In addi-tion, to provide amore fleshed out solution for the automotiveindustry,we intend to implement further distributed data ana-lytics algorithms or, when suitable third-party solution exists,interface with them through the external client application.

The central server is currently a single point of failure. Fora large-scale deployment, this issue has to be addressed byadding redundant servers. Similarly, as the number of clientdevices grows, so does the expected consumption of networkbandwidth. Consequently, by replacing the JSON interfacewith Protocol Buffers, which use binary data, we may be ableto reduce network bandwidth consumption by more than50% [11]. On a partly related note, network reliability alsoaffects error handling. Our current solution (cf. Sect. 3.4) issensible as long as the network is reasonably stable. As thiscannot be generally assumed, we intend to explore alterna-tive approaches. Other work that is still ongoing includes anexperimental feature for rapid prototyping, i.e. the ability ofthe user to define and execute custom on-board and off-boardcomputations, expressed as Python modules, without hav-ing to restart any part of the system [21]. This is relevant forexploratory experiments as it sidesteps the time-consumingand potentially disruptive deployment process.

AcknowledgmentsThis research was financially supported by the project On-board/Off-board Distributed Data Analytics (OODIDA) in thefunding program FFI: Strategic Vehicle Research and Innova-tion (DNR 2016-04260), which is administered by VINNOVA,the Swedish Government Agency for Innovation Systems. Ittook place in the Fraunhofer Cluster of Excellence "Cognitive

11

—, —, — G. Ulm et al.

Internet Technologies." Vincenzo Gulisano (Chalmers Uni-versity of Technology), Hans Svensson (Quviq), and DanielLee (Google) provided insightful comments on a draft of thispaper.

References[1] DeyanChenandHongZhao. 2012. DataSecurity andPrivacyProtection

Issues in Cloud Computing. In Computer Science and Electronics Engi-neering (ICCSEE), 2012 InternationalConferenceon,Vol. 1. IEEE, 647–651.

[2] François Chollet et al. 2015. Keras: Deep Learning Library for Theanoand Tensorflow. URL: https://keras. io/k 7, 8 (2015).

[3] Riccardo Coppola and Maurizio Morisio. 2016. Connected Car:Technologies, Issues, Future Trends. ACMComputing Surveys (CSUR)49, 3 (2016), 46.

[4] Jeffrey Dean and Sanjay Ghemawat. 2008. MapReduce: Simplified DataProcessing on Large Clusters. Commun. ACM 51, 1 (2008), 107–113.

[5] Regulation (EU). 2016. Regulation (EU) 2016/679 of the EuropeanParliament and of the Council of 27 April 2016 on the Protection ofNatural Persons with Regard to the Processing of Personal Data andon the Free Movement of such Data, and Repealing Directive 95/46/EC(General Data Protection Regulation). Official Journal of the EuropeanUnion L119 (2016), 1–88.

[6] CorentinHardy, ErwanLeMerrer, andBrunoSericola. 2017. DistributedDeep Learning on Edge-Devices: Feasibility via Adaptive Compression.arXiv preprint arXiv:1702.04683 (2017).

[7] Robert Harper. 2016. Practical Foundations for Programming Languages.Cambridge University Press.

[8] Carl Hewitt, Peter Bishop, and Richard Steiger. 1973. Session 8Formalisms for Artificial Intelligence A Universal Modular ACTORFormalism for Artificial Intelligence. In Advance Papers of theConference, Vol. 3. Stanford Research Institute, 235.

[9] Jakub Konečny, H Brendan McMahan, Daniel Ramage, and PeterRichtárik. 2016. FederatedOptimization: DistributedMachine Learningfor On-Device Intelligence. arXiv preprint arXiv:1610.02527 (2016).

[10] Qunwei Li, Bhavya Kailkhura, Ryan Goldhahn, Priyadip Ray, andPramod K Varshney. 2017. Robust Federated Learning Using ADMMin the Presence of Data Falsifying Byzantines. arXiv preprintarXiv:1710.05241 (2017).

[11] Kazuaki Maeda. 2012. Performance Evaluation of Object Serializa-tion Libraries in XML, JSON and Binary Formats. In 2012 SecondInternational Conference on Digital Information and CommunicationTechnology and Its Applications (DICTAP). IEEE, 177–182.

[12] SimonMarlow. 2012. Parallel and Concurrent Programming in Haskell.InCentral European Functional Programming School. Springer, 339–401.

[13] H Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agueray Arcas. 2016. Federated Learning of Deep Networks Using ModelAveraging. (2016).

[14] Prashanth Mundkur, Ville Tuulos, and Jared Flatow. 2011. Disco: aComputing Platform for Large-Scale Data Analytics. In Proceedingsof the 10th ACM SIGPLANWorkshop on Erlang. ACM, 84–89.

[15] Adrian Nilsson, Simon Smith, Gregor Ulm, Emil Gustavsson, andMats Jirstrand. 2018. A performance evaluation of federated learningalgorithms. In Proceedings of the Second Workshop on DistributedInfrastructures for Deep Learning. ACM, 1–8.

[16] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, VincentMichel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, PeterPrettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn:Machine Learning in Python. Journal of Machine Learning Research12, Oct (2011), 2825–2830.

[17] Anders Sjöberg, Emil Gustavsson, Ashok Chaitanya Koppisetty, andMats Jirstrand. 2019. Federated Learning of Deep Neural DecisionForests. In International Conference on Machine Learning, Optimization,and Data Science. Springer, 700–710.

[18] Robert Stephens. 1997. A Survey of StreamProcessing. Acta Informatica34, 7 (1997), 491–541.

[19] Omer Tene and Jules Polonetsky. 2011. Privacy in the Age of Big Data:a Time for Big Decisions. Stan. L. Rev. Online 64 (2011), 63–69.

[20] Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Karthik Ramasamy,Jignesh M Patel, Sanjeev Kulkarni, Jason Jackson, Krishna Gade,Maosong Fu, Jake Donham, et al. 2014. Storm@ Twitter. In Proceedingsof the 2014 ACM SIGMOD International Conference on Management ofData. ACM, 147–156.

[21] Gregor Ulm, Emil Gustavsson, and Mats Jirstrand. 2019. Active-CodeReplacement in the OODIDAData Analytics Platform. arXiv preprintarXiv:1903.09477 (2019).

[22] Gregor Ulm, Emil Gustavsson, and Mats Jirstrand. 2019. FunctionalFederated Learning in Erlang (ffl-erl). In Functional and ConstraintLogic Programming, Josep Silva (Ed.). Springer International Publishing,Cham, 162–178.

[23] Matei Zaharia,Mosharaf Chowdhury,Michael J Franklin, Scott Shenker,and Ion Stoica. 2010. Spark: Cluster Computing With Working Sets.HotCloud 10, 10-10 (2010), 95.

12

oodida: on-board/off-board distributed data analytics for ...solution to this problem is oodida...

Documents