NanoLambda Implementing Functions as a Serviceat All Resource Scales for the Internet of Things

Gareth George Fatih Bakir Rich Wolski and Chandra KrintzComputer Science Department

Univ of California Santa Barbara

AbstractmdashInternet of Things (IoT) devices are becoming in-creasingly prevalent in our environment yet the process ofprogramming these devices and processing the data they produceremains difficult Typically data is processed on device involvingarduous work in low level languages or data is moved to thecloud where abundant resources are available for Functionsas a Service (FaaS) or other handlers FaaS is an emergingcategory of flexible computing services where developers deployself-contained functions to be run in portable and secure con-tainerized environments however at the moment these functionsare limited to running in the cloud or in some cases at the ldquoedgerdquoof the network using resource rich Linux-based systems

In this paper we present NanoLambda a portable platformthat brings FaaS high-level language programming and familiarcloud service APIs to non-Linux and microcontroller-based IoTdevices To enable this NanoLambda couples a new minimalPython runtime system that we have designed for the least capa-ble end of the IoT device spectrum with API compatibility forAWS Lambda and S3 NanoLambda transfers functions betweenIoT devices (sensors edge cloud) providing power and latencysavings while retaining the programmer productivity benefits ofhigh-level languages and FaaS A key feature of NanoLambda isa scheduler that intelligently places function executions acrossmulti-scale IoT deployments according to resource availabilityand power constraints We evaluate a range of applications thatuse NanoLambda to run on devices as small as the ESP8266 with64KB of ram and 512KB flash storage

Index TermsmdashIoT serverless cloud functions edge computingmicrocontrollers portability

I INTRODUCTION

The Internet of Things (IoT) provides a data collection andcomputing fabric for physical objects in our environment Assuch it enables developers to build new classes of applica-tions that integrate with and act upon the world around usHowever with these applications come new data processingand software development challenges Specifically developersmust either port their code to large numbers of highly resource-constrained heterogeneous devices (eg sensors actuatorsand controllers embedded with microcontrollers) or movetheir data on volatile high latency and energy-consuminglong-haul networks to perform computation on resource-richcloud infrastructures

Functions as a Service (FaaS) is a technology originallydeveloped by cloud providers to ease the development ofscalable cloud and web services [6] [9] [11] [21] [22] [26][34] Because of its simple event-driven programming modeland autoscaled deployment FaaS is increasingly used for IoTapplications [8] [12] [13] [23] [36] With FaaS developers

decompose their applications into multiple functions each re-sponsible for some short-lived piece of processing A functionis typically a small single-entry code package (50 megabytesor less) written in a high level language (eg Python nodejsor Java) Thanks to their small size and self-contained designthese functions are easily invoked and scaled automaticallyby the FaaS platform in response to incoming events (dataarrival messages web requests storage updates etc) Manyfunctions potentially belonging to different tenants execute ona single server using Linux containers to provide secure iso-lation and a consistent execution environment Together thesefeatures allow the application developer to focus on the func-tion logic using a single familiar programming environmentwithout having to port code to heterogeneous architectures toconsider where functions execute or to manage how and whento scale their applications

To support IoT some FaaS platforms are now able to movedata from devices at the ldquoedgerdquo of the network to the cloud [8][13] [23] Given high latency and intermittent connectivity ofthese networks other FaaS alternatives offer edge extensionsthat instead move the functions to the edge (which are typicallymuch smaller in size than data) to consume and operate on dataldquonearrdquo where it is produced Performing computations at theedge reduces application response latency and device powerconsumption eg many IoT devices are battery powered andradio use consumes significant energy [16] [19] [20] [24][33] [35] [36]

Although a step in the right direction these edge-awareFaaS systems still impose a significant burden on IoT de-velopers Specifically FaaS functionality in most cases islimited to Linux-based edge systems ndash precluding the useof a uniform programming methodology on non-Linux de-vices (ie microcontroller-based sensors and embedded sys-tems) To integrate such devices existing platforms use mul-tiple technologies and protocols [10] [12] [13] [16] [23][24] For example an IoT applications developer that usesAWS Lambda [9] and Greengrass (ie AWS Lambda atthe edge) [24] for edge-cloud interoperation must configureMQTT [15] FreeRTOS IoT SDK [25] AWS Lambda AWSGreengrass and a persistent storage service (typically S3 [4]or DynamoDB [3]) in addition to the function itself ThusIoT application development requires significant expertise ina myriad of programming technologies and styles that isfurther complicated by differing systems capabilities resourcerestrictions quality of service and availability [10] [16]

CSPOT attempts to overcome these challenges via a low-level distributed operating system that implements a FaaS ex-ecution environment for C-language functions across all tiersof scale in IoT [36] CSPOT provides the same event-drivendeployment and execution model across systems ranging fromclouds to microcontrollers to hide heterogeneity It exports adistributed storage abstraction that functions use to persist andshare state CSPOT supports execution of functions writtenusing high level programming frameworks and cloud APIsvia Linux containers and C-language bindings However theselatter capabilities (ie high-level language support functionisolation and interoperability with other FaaS systems) doesnot extend to microcontrollers or any non-Linux system

The key challenge with extending these capabilities tomicrocontroller-based systems which all FaaS systems mustface is supporting the vast heterogeneity of such devicescoupled with their limited computational resources Sensorsand embedded systems implement different hardware architec-tures operating software and peripherals which make themdifficult to program to provide abstractions for and to leverageusing existing APIs and programming frameworks Limitedcompute storage and battery resources severely constrainwhat programs run on these devices Finally most devices lackthe physical hardware (eg memory protection units sufficientvolatile and non-volatile storage etc) that is required to runconventional FaaS software stacks

In this paper we overcome these challenges to extend FaaSuniformly to the least capable of IoT devices To enablethis we design and implement a distributed FaaS platformcalled NanoLambda that runs over and integrates with CSPOT By doing so we leverage CSPOTrsquos FaaS programming anddeployment model its fault resilient distributed storage ab-straction and cloud and edge portability NanoLambda raisesthe level of abstraction of CSPOT to support execution andisolation of Python functions on non-Linux systems By doingso NanoLambda makes Python FaaS functions ubiquitous andportable across all tiers of IoT (across sensors edge systemsand clouds) and facilitates programmer productivity acrossthese heterogeneous systems

NanoLambda is also unique in that it is tailored to the FaaSprogramming model and mirrors the APIs of popular cloud-based FaaS systems By optimizing resource use and powerconsumption NanoLambda facilitates efficient execution ofPython functions on severely resource constrained devices Byimplementing popular FaaS and storage APIs from the de factostandard AWS Lambda [9] NanoLambda brings cloud-basedFaaS familiarity to IoT programming and facilitates reuse ofexisting FaaS applications in IoT settings

The NanoLambda platform consists of EdgeCloud compo-nents (called NanoLambda CloudEdge) and microcontroller(ie on-device) components (called NanoLambda On Device)that interoperate NanoLambda CloudEdge runs on Linux-based systems employs a containerized runtime executesFaaS functions directly (those targeting the local host) andprovides a translation service for functions that target neigh-boring non-Linux devices The translator transforms Python

functions emulating any AWS Lambda and Simple StorageService (S3) API calls using compact and efficient bytecodecode packages that are executed by NanoLambda On DeviceOur design of the translator is sufficiently small and efficientto run on even resource-constrained Linux-based edge systems(eg single board computers like the Raspberry Pi)

NanoLambda On Device is designed without dependencyon Linux interfaces or a memory protection unit for functionisolation To enable this it integrates a from-scratch novelredesign of a Python interpreter called IoTPy IoTPy definesits own bytecode language (precluding the need for recom-pilation for different devices) and code packaging system(which the NanoLambda CloudEdge translator targets) IoTPyalso implements FaaS- and device-aware optimizations thatefficiently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices Our design of IoTPy is motivated by our observationthat existing Python interpreters including MicroPython [30]are too resource intensive for increasingly ubiquitous IoTdevices that we wish to target (eg the Espressif ESP8266)With the combination of these components NanoLambdaachieves AWS compatibility and efficient FaaS execution end-to-end

In the sections that follow we describe NanoLambda andthe design and implementation of its components and featuresWith this hybrid edge-device service ensemble this papermakes the following contributions

bull We explore the feasibility and performance of a FaaSservice for running high level Python functions onhighly resource restricted devices (eg microcontrollers)We evaluate the efficacy of our system using a ldquorealworldrdquo implementation We also provide microbench-marks which delve into the performance trade-offs ofrunning FaaS on-device and show that for a variety ofsuitable applications it can provide improved latency andpower efficiency

bull We demonstrate portability and the possibility of ubiquityndash executing the program on device at the edge or inthe cloud ndash using CSPOT [36] a distributed runtime(designed to support FaaS) to implement a multi-scalecompatibility service for FaaS that is API-compatiblewith AWS Lambda and S3

bull We present a simple scheduler that uses this portability toselect locations for function execution based on latencyconstraints and we show that the scheduler outperformseither remote-only execution or on-device-only executionacross a range of problem sizes

bull We achieve isolation between untrusted FaaS functions onnon-Linux IoT devices using a high-level language virtualmachine for Python as an alternative to heavyweightcontainerization

As a result this work enables the use of a single FaaS pro-gramming model and set of APIs across all resource scales inan IoT deployment while leveraging the considerable existingcode base and technological footprint of AWS Lambda

2

II RELATED WORK

Functions as a Service (FaaS) is a cloud-computing platformwith which developers write code without consideration forthe hardware on which it runs Applications are broken downinto distinct units of functionality and packaged as functionsthat can be distributed and updated on the fly Functions areexecuted in lightweight containers which provide portabilityscalability and security functions are automatically scaledacross multiple machines while execution is isolated fromother tenants of those instances A number of major cloudproviders such as AWS Lambda [9] Google Cloud Functions[22] and Microsoft Azure Functions [11] now offer hostedFaaS platforms

Because of its automatic scaling and convenience FaaS hasseen increased adoption in IoT in recent years AWS IoTfor example leverages FaaS functions extensively in order toallow users of the platform to attach functions that handle IoTevents [17] Serverless [34] a leading industry toolchain forFaaS programming estimates that IoT makes up 6 [1] ofthe use of FaaS in industry

An emerging development in IoT data processing is theprospect of FaaS at the edge for data processing Commercialofferings such as AWS GreenGrass [24] and open sourceApache OpenWhisk [6] make it possible for users to host theirown FaaS deployments on low-power devices By runningfunctions at the edge the hope is that handlers can berun closer to where data is produced making better use oflocally-available processing and reducing latency penalties andtransfer costs The authors of [31] explore the concept of aserverless platform for real-time data analytics at the edge anddiscuss the benefits and challenges of the approach

CSPOT is an open source low-level distributed operat-ing system for IoT that executes a uniform FaaS modelacross devices ranging from servers in the cloud to resource-constrained sensors and microcontrollers [36] CSPOT exportsa distributed storage abstraction called a WooF for persistentstorage of FaaS function objects WooFs are append-onlylogs with built-in event tracking and repair capabilities [28]WooFs serve both the role of state storage and executionlog ndash appending data to WooFs optionally can trigger theexecution of an event handler that responds to the appendeddata Functions (ie event handlers) are implemented as Linuxexecutables allowing for flexibility of language choice andlibraries A pool of docker containers is maintained to runhandlers ndash by running only one handler in a container atany given time functions are isolated from one another Thisenables development of complex applications and provides thesecurity portability and scalability expected of FaaS systems

For Linux systems CSPOT provides an execution envi-ronment and AWS API compatibility (for S3 and Lambdaservices) for Python functions However this support does notextend to microcontrollers (which we address with the workherein) To execute C-language functions on microcontrollersCSPOT uses a threaded multitasking system with direct mem-ory addressing This support is available for ARM AVR and

Espressif microcontrollers FaaS functions in CSPOT must becompiled and installed on a per-device basis Our work enablesPython functions (including Python AWS Lambda functions)to be executed over CSPOT including on microcontrollersWe do so using a combination of automatic translation at theedge and interpretation on-device to enable both portabilityand efficiency

When functions can be made highly portable as is the casewith FaaS interesting possibilities for improving performanceemerge such as scheduled execution The authors of [7]propose a framework for scheduling execution among multipleFaaS providers The authors develop an extensible API forexecuting functions across multiple providers as well as ametrics database that tracks execution statistics from theseservices Once a model of function performance has been builtfor a given provider this information can be provided to a user-defined scheduler eg a Python function which selects theprovider with the minimum latency The authors find that byemploying a scheduler they can achieve a 200 faster averageround-trip execution time compared to executing on any oneservice provider This serves as a promising example of howthe portable nature of FaaS functions can provide applicationdevelopers with more flexible execution options and improvedperformance

III NanoLambda DESIGN AND IMPLEMENTATION

At present writing applications for IoT means programmingfor APIs and libraries specific to each device writing functionsthat are ldquolocked inrdquo to particular cloud provider APIs andusing only those devices that are supported by these platformsIt is our objective to support a common programming modeleverywhere extending from IoT devices to the edge to thecloud In service of this goal we opted to design our systemaround CSPOT a low-level framework for supporting IoTFaaS CSPOT provides the foundation for our high-level FaaSruntime with powerful cross-platform primitives for securefunction (event handler) execution and data persistence

However existing cloud-based FaaS offerings that have seenadoption in industry use high level languages such as Pythonfor their handlers These high-level languages are particularlyadvantageous for portability and programmer productivityThe CSPOT service while capable of running with minimaloverhead on a wide range of severely resource-constraineddevices defines its own API Thus while CSPOT is itselfportable it cannot execute FaaS functions written for existingcloud services such as AWS Lambda

It is our view that in order to provide the same conveniencecompatibility and security as FaaS handlers executing inthe cloud a FaaS system for resource-restricted non-Linuxdevices should have the following properties

1) Ease of development ndash we wish for NanoLambda tobe familiar to developers already using FaaS ideallyallowing existing FaaS functions to run with little tono code changes

3

CSPOT BackendObject Store

Python 36CSPOTHandler

CSPOT WooFInvocation

Log IoTPyCodeAPItranslation

and packaging

service

NanoLambda CloudEdge NanoLambda On Device

IoTPYon deviceand remoteexecution

capabilities

CSPOT WooF measurement

logs

Sensors amp drivers custom OS with monitoring C++ threads

Lambda APIEmulationService

S3 APIEmulationService

Fig 1 Service diagram outlining the interactions of the NanoLambda components that deliver FaaS for IoT Shaded components are those added to CSPOTby NanoLambda NanoLambda CloudEdge runs on Linux hosts in the cloud or at the edge NanoLambda On Device runs on microcontrollers and non-Linuxsystems

2) Portability ndash we want portable application code so thatthe same implementation can run in the cloud or on thedevice with little to no additional developer effort

3) Small code and memory footprint ndash NanoLambda shouldleave as much storage and memory space as possibleavailable for libraries and C extensions

4) Security ndash IoT devices can be particularly vulnerableto buffer overflow stack overflow and other classes ofexploits [2] [29] which we must protect against

Figure 1 shows the architecture of the system we havedeveloped to meet these goals We deliver on our goals withtwo core systems built on CSPOT

bull NanoLambda CloudEdge shown on the left in the figureprovides FaaS handler execution to Linux capable devicesas well as object storage It doubles as a centralizedrepository from which IoT devices can request functions

bull NanoLambda On Device shown on the right in the figureprovides on-device handler execution capabilities withIoTPy a Python interpreter tailor-made for FaaS handlerexecution

This separation of concerns is primarily driven by the needsto tailor our device implementation to best perform withinthe constraints of low power non-Linux IoT devices Thisallows us to offload complex and processing-intensive tasksto NanoLambda CloudEdge while keeping NanoLambda OnDevice space- and power-efficient

A NanoLambda CloudEdge ServiceFigure 1 illustrates the NanoLambda CloudEdge service

on the left The core of this service consists of two RESTAPI servers which export (i) an object store compatiblewith the Amazon Simple Storage Service (S3) [4] and (ii)a FaaS service that deploys FaaS functions written for AWSLambda [9] over CSPOTndash in the same way they are deployedover AWS

To enable this the servers leverage multiple CSPOT primi-tives This includes an object store and event model in whichhandlers are triggered in response to object updates CSPOTobjects are fixed-size records in append-only logs Handlersexecute inside Linux containers allowing for concurrent ex-ecution of handlers as isolated processes The NanoLambda

CloudEdge API servers build upon these primitives andCSPOT support for a subset of the S3 and AWS Lambda APIs(for Linux systems) to unify deployment and execution ofAWS Lambda Python functions across Linux and non-Linuxsystems

Specifically the S3 API Emulation Service provides awrapper for the CSPOT append-only object store It storesblobs to CSPOTrsquos fixed-size log records by first splitting theminto 16KB chunks and then linking these chunks together withmetadata records It then stores a mapping between the blobrsquoskey and the location of the first chunk separately in an indexlog Blobs are retrieved via a sequential scan of the indexlog to find their location Once located they are read back inchunks and streamed to the requester This service exports thisfunctionality via the Amazon S3 REST API [5]

The Lambda API Emulation Service manages functiondeployment and facilitates function execution in CSPOT Theservice stores the configuration and code package using theS3 API Emulation Service making it a single store for allpersistent data By offloading state management to a cen-tralized object store these services make it possible to runload-balanced instances across multiple physical machines toimprove performance and scalability

AWS Lambda compatibility is provided via a specialCSPOT handler binary which embeds a Python interpreterWhen an function invocation request is received NanoLambdaCloudEdge appends an invocation record to CSPOTrsquos objectstore This append creates an event which invokes this handlerin an isolated docker instance The handler reads the invocationrecord and checks for the presence of the Python code forthe handler If necessary the code is fetched from the objectstore Once ready the Python interpreter invokes the handlerfunction with the payload from the invocation record andwrites the result to a result log for return to the requester

To support handler execution on device NanoLambdaCloudEdge implements an IoTPy translator (cf Section III-C)which compiles and packages Python handlers for execu-tion on neighboring microcontroller devices This translationservice enables IoT devices to fetch handler code from thesame function repository used by the Lambda API Emulation

4

Service The service compiles handlers on-demand into acompact binary representation that can be sent to the devicefor execution Fetch requests check that the code size andmemory requirements (specified in the function configuration)fit within the resource constraints of the requesting device Ifrejected it is possible for the cloudedge device to execute thefunction on behalf of the IoT device

B NanoLambda On Device

NanoLambda On Device is runs Python handlers on non-Linux IoT devices As shown in Figure 1 (on right) the serviceleverages CSPOT WooF logs on-device for persistent storageand event invocation As for Linux systems FaaS handlers areinvoked in response to data appended to storage logs

As seen in the figure the typical model for NanoLambdaOn Device is that data is produced by threads monitoringsensors on the device and appending it to objects in the objectstore It is also possible for data to be remotely deliveredfor processing over CSPOTrsquos network API Each append tothe object store is processed by CSPOTrsquos event model whichruns a C-language handler function with the new data Similarto the NanoLambda CloudEdgersquos support for Python usinga special Linux handler binary with an embedded Pythoninterpreter we extend NanoLambda CloudEdge with Pythonsupport by registering a special C-language handler functionwhich interprets the handlerrsquos name to be a Python Lambdafunction deployed via the NanoLambda CloudEdge serviceWhen this function is invoked it triggers the handler withthe payload provided by the newly appended object in aninstance of the IoTPy VM By leveraging CSPOTrsquos existingfunctionality we are able to offer Python 3 Lambda supporton device

C IoTPy

To ease development and portability we choose Python asour FaaS programming language because its bytecode is easilyportable and its simple virtual machine implementation canbe easily ported to IoT devices Likewise we package FaaSfunctions using the package format defined for AWS Lambdafor compatibility purposes

To achieve a small code and memory footprint we optedto implement IoTPy our own extremely lightweight Pythoninterpreter built from the ground up with embedding in mindWe found existing solutions such as MicroPython [30] to beunsuitable because they left little additional space for featuressuch as our FaaS runtime and the drivers for our networkingand sensors On our most resource-constrained devices suchas the ESP8266 our binary must be able to fit in as little as512KB of flash memory

IoTPy is written to be as unintrusive as possible with afocus on allowing the Python interpreter to be treated like alibrary To keep IoTPy small we opted to omit the Pythonparserlexer on the device Instead these tasks are offloadedto the NanoLambda CloudEdge service This benefits us in anumber of ways

1) bytecode compilation as well as optimization steps canbe performed by a full Python 36 implementation onNanoLambda CloudEdge service

2) memory use is reduced since we are only shippingcompact bytecode representation to the device and

3) a simpler virtual machine implementation can be used ondevice ndash by delegating compilation to the NanoLambdaCloudEdge service our IoTPy is able to limit its func-tionality to implementing the core language features ofthe Python virtual machine (VM)

Together these capabilities mitigate the additional complexityof maintaining a simple Python bytecode VM To enablethese capabilities we make some tradeoffs with regard toprogram storage and architectural simplicity IoTPy provides aconvenient CC++ interface for extending Python with nativefunctions We currently provide implementations for only themost common Python libraries and emulation of only the AWSS3 and Lambda deployment services The libraries includecommon math functions JSON operations APIs for accessingdevice peripherals basic networking and interfaces for theAPIs provided by NanoLambda CloudEdge IoTPy cannothowever load standard Python packages

When comparing the size of the generated binary files forIoTPy and MicroPython we find that the code for the IoTPylibrary uses 290KB of flash storage whereas MicroPythonrequires 620KB of flash storage Thus the IoTPy imple-mentation leaves us with almost 300KB of additional storageavailable for programs drivers and the FaaS runtime

D Python VM Security and Isolation for FaaS Handlers

IoT devices are often so resource-constrained that theylack security features such as memory protection units toimplement separate address spaces for isolated execution ofuser code Additionally conventional FaaS isolation achievedby Linux features like container-based isolation is simply notpossible on devices like these The authors of [2] and [29] findvarious attacks many of which rely on out-of-range memoryaccess to exploit IoT devices and execute code or update theirfirmware with malicious payloads Yet it is essential that IoTdevelopers be able to reprogram and update their devices inthe field

By placing the majority of application logic in FaaS han-dlers we reduce potential attack surface area A secure Pythonvirtual machine implementation can ensure that the bytecoderunning within it cannot access memory out of range orotherwise escape the permissions allowed to the handler bythe built-in functions provided to it Python handlers can thenbe isolated from one another by running in separate instancesof the Python VM This isolation is similar to traditional Linuxcontainers

E Deploying and Running Functions

By leveraging NanoLambda CloudEdge and its compatibil-ity layer for Amazonrsquos AWS Lambda APIs we minimize theeffort to port functions to the IoTPy platform In many casesfunctions run directly on IoTPy without code changes

5

programmer deploys code toNanoLambda CloudEdge service

with aws cli

NanoLambdaCloudEdge

1 stores code in object store service2 compiles and caches compact

bytecode representation on-demand

when IoT devices find a handler is not cachedlocally they request bytecode representation

from NanoLambda service

Fig 2 Life cycle of a function deployment from the application developerrsquoscommand line to the NanoLambda CloudEdge service to the IoT devicerunning NanoLambda On Device

Function deployment is performed by developers using theaws-cli or the aws-sdk (using a NanoLambda CloudEdgedevice as the cloud target) meaning existing tools capable ofdeploying to AWS Lambda can also deploy to NanoLambdaCloudEdge with few changes Deploying a new version ofan application is done with the ldquoaws lambda create-functionrdquocommand which accepts the application code package as azip file as well as other configuration parameters includingthe function entry-point its memory limit and Python version(Python 36 is currently supported) The deployment processchecks code size and resource limits (as is done in AWSLambda) and also checks for use of unsupported AWS APIs inthe code (eg only AWS REST S3 API is currently supported)The validated function is stored in the S3 API EmulationService as described above

Function execution on device is performed by IoTPy Localinvocations are accelerated by an interpreter cache whichallows the IoTPy interpreter for a given handler to be reusedif it is still available from a previous invocation This isdone to avoid the relatively expensive process of reinitializingPython built-ins as well as functions and globals defined inthe bytecode

In Figure 2 we show the deployment and function downloadprocess mapped out for NanoLambda On Device When theinterpreter is still running (ie because it is already running afunction) execution involves simply setting up a new Pythonframe with the call stack for the function and invoking thefunction in the cached interpreter When no interpreter isrunning the NanoLambda service opens a TCP connectionto NanoLambda CloudEdge and serializes a request for thefunction using flatbuffers [18] a fast and memory efficientbinary protocol for network serialization When NanoLambdaCloudEdge receives this request the function is fetched fromour object store and compiled on the fly using the Linuximplementation of Python 36 and the Python dis package toextract generated bytecode

F Execution Offloading CapabilitiesOur NanoLambda On Device service provides direct han-

dler code compatibility with the NanoLambda CloudEdgeservice We achieve this by adopting the same AWS Lambda-compatible handler format

NanoLambda On Device supports local execution of han-dlers on microcontrollers like the ESP8266 and the CC3220SFNanoLambda CloudEdge leveraging CSPOTrsquos tiered-cloudapproach extends these execution options to include edgecloud nodes like the Raspberry Pi Zero as well as in-cloudsolutions like CSPOT running on AWS EC2 Lastly becauseboth services use a common function format which is code-compatible with AWS Lambda our handlers can be easilytransferred between the services or even executed on AWSLambda itself Selecting to develop application logic in com-partmentalized functions using a high-level language lends thedeveloper immediate portability benefits which grant her thepower to choose where code should be run to best utilizeresources or achieve desired performance

IV EVALUATION

In this section we evaluate NanoLambda on two exampleFaaS applications for sensor monitoring We provide a perfor-mance analysis of NanoLambda using micro-benchmarks toexpose a detailed analysis of its performance characteristicsunder its various configurations Additionally we show howNanoLambdarsquos portability can be leveraged to implement alatency-aware scheduler that migrates execution between thedevice and the edge cloud to improve performance across arange of problem sizes

A Automatic Light Control ApplicationWe show how NanoLambda might be used in an auto-

matic lighting control system ndash for example to automati-cally turn on walkway lighting at night based on ambientlight levels The hardware setup for this experiment is aphotoresistor connected to the analogue-to-digital converter(ADC) pin of an ESP8266 microcontroller This processoris extremely resource-constrained with 80KB of user dataRAM and 512KB of flash storage for code The only device-specific C code written for this experiment is a simple samplerthread which measures the voltage on the ADC pin of themicrocontroller and writes it out to an invocation log at a rateof 5 samples per second

In response to each log append (analogous to publishing toan MQTT queue) an IoTPy handler is invoked to analyze thenew data and send commands to the light system if necessaryThe implementation of the handler for this automatic lightcontrol experiment is shown in Figure 3

def new_light_lvl(payload ctx)if payload[0] gt 100

deviceset_pin_high(2)else

deviceset_pin_low(2)

Fig 3 Handler function for the light control application

6

In Table I we show invocation latencies for various config-urations of the NanoLambda runtime The ldquoNo Server Cacherdquoconfiguration represents the true ldquocold startrdquo cost of invokinga new AWS Lambda function for the first time This numberincludes both the time taken for the server to compile theLambda function and to deliver it to the device The LocalCache configuration shows us the typical amortized runtimeof the application where frequently invoked Lambda functionswill typically be already available on the device We cansee that the performance difference is substantial requiringalmost 32x as much time to invoke a function that is not incache This is largely due to the effects of both compilationtime and the cost of network latency Compilation and codetransfer each take roughly 100 milliseconds on the networktested We include ldquoNo Local Cacherdquo to show the startup costof a function shared by multiple devices In this scenario afunction typically has already been accessed and compiled bythe server but has not yet been delivered to a given deviceWe expect this to be representative of the startup cost for real-world deployments

We include a C implementation of the same handler functionas a baseline It is worth noting that this is on the order of20x faster than our configuration with local caching This is asubstantial overhead but it is representative of what we wouldexpect for the performance difference between an interpretedlanguage without JIT and a compiled language like C [32]Ultimately it is up to the application developer to determinewhen the flexibility and ease of deployment with Python andthe FaaS ecosystem outweigh its overhead

Configuration Latency in msNanoLambda Local Caching 67 msNanoLambda No Local Cache 1195 msNanoLambda No Server Cache 2204 msNanoLambda C Handler 03 ms

TABLE IAVERAGE LATENCY FOR 100 ITERATIONS OF THE HANDLER FUNCTION

B Predictive Maintenance Application

In this experiment we look at the detailed performance ofa NanoLambda application with more demanding processingrequirements compared to the previous example Predictivemaintenance techniques use sensors to detect part failures orother maintenance requirements

Our experiment considers motor maintenance by analyzingthe vibrations off of a motor as measured by an accelerometerIt is possible to detect if the motor or a connected part hasfailed by monitoring for changes in the distribution of themagnitude of vibrations picked up by the accelerometer Ourapplication detects these changes with a Python implementa-tion of the KolmogorovndashSmirnov (KS) test used to comparea reference distribution against real time measurements froman accelerometer The test provides the probability that theempirical distribution matches the reference distribution Apossible failure can be flagged if this probability drops belowa tuned threshold

list containing a reference distribution from accelerometerreference = []

def kstest(datalist1 datalist2) omitted see implementation from gistgithubcomdevries11405101

def new_accel_sample(payload ctx)global referencetransformed = []for record in payload

transformedappend(magnitude(record)))

prob = kstest(transformedreference)

return prob

Fig 4 The implementation of the handler providing the KS test results

In Figure 4 we show the implementation of our KS testhandler The handler is provided with a JSON payload con-taining a list of recent accelerometer data as tuples of (x y z)acceleration These records are normalized to a transformedarray of floating point magnitudes and then passed to anldquooff-the-shelfrdquo open-source implementation of the KS test byChristopher De Vries [27] This KS test implementation iscomputationally intensive and scales linearly with the inputproblem size

In Table II we show the performance of a real-worldinstallation The hardware configuration is a CC3220SF pro-cessor with 1MB of program flash and 256KB of RAM Thisprocessor is wired to an accelerometer monitored by a samplerthread (much like the photoresistor experiment) which collectsdata from the accelerometer 5 times per second and appends itto a data log Every 32 samples a handler is invoked to analyzerecent data The handler is always invoked on device andshows the performance of a more complex handler running onIoTPy The more complex function requires longer executiontime 147ms as well as a longer startup time of 436ms tocompile

We found this handler to be sufficiently computationallyintensive to be a good candidate for offloading to an edge clouddevice To this end we next present a scheduler for offloadingcomputationally-intensive tasks to remote FaaS providers

C Saving Power With Execution OffloadingIn Figure 5 we show the power used by the CC3220SF

microcontroller for KS problem sizes 20 and 80 over the

Configuration Latency (ms) Power Use (mJ) Memory Use KBNanoLambda Local Caching 2094 ms 2123 mJ 236KBNanoLambda No Local Cache 7290 ms 8500 mJ 217KB

TABLE IIAVERAGE TIMESPOWER CONSUMPTION OVER 100 ITERATIONS OF THE

HANDLER FUNCTION ON THE CC3220SF MICROCONTROLLER WITH A KSPROBLEM SIZE OF 32 IN EACH CONFIGURATION

7

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600Po

wer D

raw

in (m

W)

Power Use Over Local Request Lifecycle for Problem Size 20Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 20Remote Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w in

(mW

)

Power Use Over Local Request Lifecycle for Problem Size 80Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 80Remote Request

Fig 5 Comparison of power draw over request lifecycle for various KSproblem sizes using both remote and local strategies

life cycle of a request We observe here that with the ex-ception of some power spikes due to background activity onthe microcontroller the power used by the local request iscontinuous from the beginning of the invocation until the resultis available and so it grows with the size of the problem (plussome constant overhead from the FaaS runtime) With remoteexecution we see just over twice the peak power draw of thelocal execution strategy but we see a trough in the middle ofthe request life cycle where power use drops to near zeroThis is because the device is free to completely idle whilewaiting for results to return from the server Therefore wepay a large but short-lived power cost during transmissionbut when enough computation can be performed in the cloudwe can save power overall relative to local execution

D Predictive Maintenance Application with a Naive Offload-ing Scheduler

In this application we take advantage of our code compat-ibility between NanoLambda CloudEdge and NanoLambdaOn Device to extend our predictive maintenance applicationrunning on device with the capability to dynamically offloadcomputation to the edge

Our scheduler offloads computation when its algorithmestimates that better latency can be achieved by transferringexecution to the cloud We model execution latency by check-ing the runtime of the last execution on NanoLambda OnDevice and comparing it against the round trip latency of thelast invocation on NanoLambda CloudEdge The service withthe lowest total latency is selected to run the handler

However as conditions change execution-time measure-ments can become stale Much like the authors of [7] weperiodically retest each possible execution strategy to allow thescheduler to sense when the relative performance has changedeg as a result of evolving network conditions or changesin problem complexity In this experiment our scheduler isconfigured to retest the losing strategy 1 out of every 16invocations

To benchmark the scheduler we measure the average in-vocation latency for a range of synthetic problems from 5data points to 100 data points We examine problem sizes atmultiples of 5 with 1000 trials at each size

Figure 6 shows average invocation latency as a functionof problem size for the scheduler a remote-only invocationstrategy and a local-only invocation strategy Local invoca-tions provide very consistent runtime which scales linearlywith the problem size Remote invocation on the other handis highly subject to network latency and congestion as wecan see from occasional spikes The scheduled invocationsappear to offer the best of both worlds scaling linearly untilthe latency exceeds the average network latency at which pointthe strategy switches to remote invocation

Figure 7 shows average power per invocation as a functionof problem size for the scheduler the remote-only invocationstrategy and the local-only invocation strategy We see similartrends as the previous the latency graph ndash remote executionuses relatively constant power while local execution powerconsumption scales with execution time We see again thatthe scheduler offers the best of both strategies Interestinglyhowever even though the scheduler begins to switch to remoteinvocations at problem size 40 we do not observe powersavings from this until the problem size reaches 60

A more detailed breakdown of the choices made by thescheduler strategy are shown in Figure 8 The top of thischart shows the execution latency while the bottom shows thenumber of calculations run locally versus the number offloadedover the network Initially all calculations run on device forsmall problem sizes however once the problem size exceeds40 it becomes intermittently worthwhile to run over thenetwork when latency is low After the problem size exceeds60 almost all invocations run remote as the speed differencebetween on-device execution and remote execution outweighsthe latency cost of transferring data over the network Weobserve however that this naive scheduling algorithm leavessome room for improvement First with a fixed re-testinglatency we allocate more cycles to re-testing than may benecessary As we can see in this graph the scheduler isalways spending 601000 cycles re-testing the losing strategyWhen there is a large difference in performance this can be

8

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 6 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 7 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

a substantial overhead Second the scheduler has difficultyswitching to remote invocations when optimal as it is subjectto temporary network spikes

E An Improved Scheduler for Predictive Maintenance

Learning from the performance of our naive scheduler wenext construct a model adjusts to short-lived network spikesand that reduces the amount of time spent retesting To thisend we devise a new scoring function that incorporates anexponential term to adjust dynamically the re-testing intervalIn addition we use the median latency of recent requests as away of filtering outlier measurements

We define the new score function to be

score = median(H)minus (1 + r)d

Where H is the invocation latency history for the strategy(local or remote) being scored r is the decay rate which isset to r = 12 and d is the decay term ndash the number of

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)

Avg Invocation Latency vs Problem Size for Naive Scheduler

20 40 60 80 100ks problem size

20

40

60

80

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Naive Scheduler

remote invocationslocal invocations

Fig 8 Top average invocation latency as a function of KS problem size forthe offloading scheduler Bottom local invocation count vs remote invocationcount selected by the offloading scheduler for each problem size

invocations since this strategy was last selected We take themedian over a window of 3 samples which serves to filterout any one unusually fast or slow result allowing us to moreeasily recover from lost packets or other short-lived networkspikes

In Figure 9 we can see how this invocation strategy breaksdown in terms of request distribution When compared withthe naive schedulerrsquos strategy breakdown in Figure 8 weobserve that the improved scheduler begins to switch schedul-ing strategies slightly earlier than the naive scheduler It isinteresting to see that the improved scheduler with its noisefiltering transitions smoothly from local to remote invocationsas the problem size gradually increases When problem sizesare in the range from approximately 50 to 65 the schedulermixes local and remote invocations as a result of variations innetwork latency etc

With the improved scheduler the transition to remote exe-cution is much smoother and completes much earlier than theswitch-over seen in the naive strategy The naive strategy failsto fully commit to remote execution until a problem size of 75or 80 whereas the improved scheduler has mostly switchedover by the time the problem size reaches 65 Even when thelatency difference between remote execution is large as is thecase with problem sizes greater than 80 the naive schedulerappears to occasionally switch back to local execution This ismost likely a result of network timeouts which go unfiltered in

9

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

CSPOT attempts to overcome these challenges via a low-level distributed operating system that implements a FaaS ex-ecution environment for C-language functions across all tiersof scale in IoT [36] CSPOT provides the same event-drivendeployment and execution model across systems ranging fromclouds to microcontrollers to hide heterogeneity It exports adistributed storage abstraction that functions use to persist andshare state CSPOT supports execution of functions writtenusing high level programming frameworks and cloud APIsvia Linux containers and C-language bindings However theselatter capabilities (ie high-level language support functionisolation and interoperability with other FaaS systems) doesnot extend to microcontrollers or any non-Linux system

The key challenge with extending these capabilities tomicrocontroller-based systems which all FaaS systems mustface is supporting the vast heterogeneity of such devicescoupled with their limited computational resources Sensorsand embedded systems implement different hardware architec-tures operating software and peripherals which make themdifficult to program to provide abstractions for and to leverageusing existing APIs and programming frameworks Limitedcompute storage and battery resources severely constrainwhat programs run on these devices Finally most devices lackthe physical hardware (eg memory protection units sufficientvolatile and non-volatile storage etc) that is required to runconventional FaaS software stacks

In this paper we overcome these challenges to extend FaaSuniformly to the least capable of IoT devices To enablethis we design and implement a distributed FaaS platformcalled NanoLambda that runs over and integrates with CSPOT By doing so we leverage CSPOTrsquos FaaS programming anddeployment model its fault resilient distributed storage ab-straction and cloud and edge portability NanoLambda raisesthe level of abstraction of CSPOT to support execution andisolation of Python functions on non-Linux systems By doingso NanoLambda makes Python FaaS functions ubiquitous andportable across all tiers of IoT (across sensors edge systemsand clouds) and facilitates programmer productivity acrossthese heterogeneous systems

NanoLambda is also unique in that it is tailored to the FaaSprogramming model and mirrors the APIs of popular cloud-based FaaS systems By optimizing resource use and powerconsumption NanoLambda facilitates efficient execution ofPython functions on severely resource constrained devices Byimplementing popular FaaS and storage APIs from the de factostandard AWS Lambda [9] NanoLambda brings cloud-basedFaaS familiarity to IoT programming and facilitates reuse ofexisting FaaS applications in IoT settings

The NanoLambda platform consists of EdgeCloud compo-nents (called NanoLambda CloudEdge) and microcontroller(ie on-device) components (called NanoLambda On Device)that interoperate NanoLambda CloudEdge runs on Linux-based systems employs a containerized runtime executesFaaS functions directly (those targeting the local host) andprovides a translation service for functions that target neigh-boring non-Linux devices The translator transforms Python

functions emulating any AWS Lambda and Simple StorageService (S3) API calls using compact and efficient bytecodecode packages that are executed by NanoLambda On DeviceOur design of the translator is sufficiently small and efficientto run on even resource-constrained Linux-based edge systems(eg single board computers like the Raspberry Pi)

NanoLambda On Device is designed without dependencyon Linux interfaces or a memory protection unit for functionisolation To enable this it integrates a from-scratch novelredesign of a Python interpreter called IoTPy IoTPy definesits own bytecode language (precluding the need for recom-pilation for different devices) and code packaging system(which the NanoLambda CloudEdge translator targets) IoTPyalso implements FaaS- and device-aware optimizations thatefficiently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices Our design of IoTPy is motivated by our observationthat existing Python interpreters including MicroPython [30]are too resource intensive for increasingly ubiquitous IoTdevices that we wish to target (eg the Espressif ESP8266)With the combination of these components NanoLambdaachieves AWS compatibility and efficient FaaS execution end-to-end

In the sections that follow we describe NanoLambda andthe design and implementation of its components and featuresWith this hybrid edge-device service ensemble this papermakes the following contributions

bull We explore the feasibility and performance of a FaaSservice for running high level Python functions onhighly resource restricted devices (eg microcontrollers)We evaluate the efficacy of our system using a ldquorealworldrdquo implementation We also provide microbench-marks which delve into the performance trade-offs ofrunning FaaS on-device and show that for a variety ofsuitable applications it can provide improved latency andpower efficiency

bull We demonstrate portability and the possibility of ubiquityndash executing the program on device at the edge or inthe cloud ndash using CSPOT [36] a distributed runtime(designed to support FaaS) to implement a multi-scalecompatibility service for FaaS that is API-compatiblewith AWS Lambda and S3

bull We present a simple scheduler that uses this portability toselect locations for function execution based on latencyconstraints and we show that the scheduler outperformseither remote-only execution or on-device-only executionacross a range of problem sizes

bull We achieve isolation between untrusted FaaS functions onnon-Linux IoT devices using a high-level language virtualmachine for Python as an alternative to heavyweightcontainerization

As a result this work enables the use of a single FaaS pro-gramming model and set of APIs across all resource scales inan IoT deployment while leveraging the considerable existingcode base and technological footprint of AWS Lambda

2

II RELATED WORK

Functions as a Service (FaaS) is a cloud-computing platformwith which developers write code without consideration forthe hardware on which it runs Applications are broken downinto distinct units of functionality and packaged as functionsthat can be distributed and updated on the fly Functions areexecuted in lightweight containers which provide portabilityscalability and security functions are automatically scaledacross multiple machines while execution is isolated fromother tenants of those instances A number of major cloudproviders such as AWS Lambda [9] Google Cloud Functions[22] and Microsoft Azure Functions [11] now offer hostedFaaS platforms

Because of its automatic scaling and convenience FaaS hasseen increased adoption in IoT in recent years AWS IoTfor example leverages FaaS functions extensively in order toallow users of the platform to attach functions that handle IoTevents [17] Serverless [34] a leading industry toolchain forFaaS programming estimates that IoT makes up 6 [1] ofthe use of FaaS in industry

An emerging development in IoT data processing is theprospect of FaaS at the edge for data processing Commercialofferings such as AWS GreenGrass [24] and open sourceApache OpenWhisk [6] make it possible for users to host theirown FaaS deployments on low-power devices By runningfunctions at the edge the hope is that handlers can berun closer to where data is produced making better use oflocally-available processing and reducing latency penalties andtransfer costs The authors of [31] explore the concept of aserverless platform for real-time data analytics at the edge anddiscuss the benefits and challenges of the approach

CSPOT is an open source low-level distributed operat-ing system for IoT that executes a uniform FaaS modelacross devices ranging from servers in the cloud to resource-constrained sensors and microcontrollers [36] CSPOT exportsa distributed storage abstraction called a WooF for persistentstorage of FaaS function objects WooFs are append-onlylogs with built-in event tracking and repair capabilities [28]WooFs serve both the role of state storage and executionlog ndash appending data to WooFs optionally can trigger theexecution of an event handler that responds to the appendeddata Functions (ie event handlers) are implemented as Linuxexecutables allowing for flexibility of language choice andlibraries A pool of docker containers is maintained to runhandlers ndash by running only one handler in a container atany given time functions are isolated from one another Thisenables development of complex applications and provides thesecurity portability and scalability expected of FaaS systems

For Linux systems CSPOT provides an execution envi-ronment and AWS API compatibility (for S3 and Lambdaservices) for Python functions However this support does notextend to microcontrollers (which we address with the workherein) To execute C-language functions on microcontrollersCSPOT uses a threaded multitasking system with direct mem-ory addressing This support is available for ARM AVR and

Espressif microcontrollers FaaS functions in CSPOT must becompiled and installed on a per-device basis Our work enablesPython functions (including Python AWS Lambda functions)to be executed over CSPOT including on microcontrollersWe do so using a combination of automatic translation at theedge and interpretation on-device to enable both portabilityand efficiency

When functions can be made highly portable as is the casewith FaaS interesting possibilities for improving performanceemerge such as scheduled execution The authors of [7]propose a framework for scheduling execution among multipleFaaS providers The authors develop an extensible API forexecuting functions across multiple providers as well as ametrics database that tracks execution statistics from theseservices Once a model of function performance has been builtfor a given provider this information can be provided to a user-defined scheduler eg a Python function which selects theprovider with the minimum latency The authors find that byemploying a scheduler they can achieve a 200 faster averageround-trip execution time compared to executing on any oneservice provider This serves as a promising example of howthe portable nature of FaaS functions can provide applicationdevelopers with more flexible execution options and improvedperformance

III NanoLambda DESIGN AND IMPLEMENTATION

At present writing applications for IoT means programmingfor APIs and libraries specific to each device writing functionsthat are ldquolocked inrdquo to particular cloud provider APIs andusing only those devices that are supported by these platformsIt is our objective to support a common programming modeleverywhere extending from IoT devices to the edge to thecloud In service of this goal we opted to design our systemaround CSPOT a low-level framework for supporting IoTFaaS CSPOT provides the foundation for our high-level FaaSruntime with powerful cross-platform primitives for securefunction (event handler) execution and data persistence

However existing cloud-based FaaS offerings that have seenadoption in industry use high level languages such as Pythonfor their handlers These high-level languages are particularlyadvantageous for portability and programmer productivityThe CSPOT service while capable of running with minimaloverhead on a wide range of severely resource-constraineddevices defines its own API Thus while CSPOT is itselfportable it cannot execute FaaS functions written for existingcloud services such as AWS Lambda

It is our view that in order to provide the same conveniencecompatibility and security as FaaS handlers executing inthe cloud a FaaS system for resource-restricted non-Linuxdevices should have the following properties

1) Ease of development ndash we wish for NanoLambda tobe familiar to developers already using FaaS ideallyallowing existing FaaS functions to run with little tono code changes

3

CSPOT BackendObject Store

Python 36CSPOTHandler

CSPOT WooFInvocation

Log IoTPyCodeAPItranslation

and packaging

service

NanoLambda CloudEdge NanoLambda On Device

IoTPYon deviceand remoteexecution

capabilities

CSPOT WooF measurement

logs

Sensors amp drivers custom OS with monitoring C++ threads

Lambda APIEmulationService

S3 APIEmulationService

Fig 1 Service diagram outlining the interactions of the NanoLambda components that deliver FaaS for IoT Shaded components are those added to CSPOTby NanoLambda NanoLambda CloudEdge runs on Linux hosts in the cloud or at the edge NanoLambda On Device runs on microcontrollers and non-Linuxsystems

2) Portability ndash we want portable application code so thatthe same implementation can run in the cloud or on thedevice with little to no additional developer effort

3) Small code and memory footprint ndash NanoLambda shouldleave as much storage and memory space as possibleavailable for libraries and C extensions

4) Security ndash IoT devices can be particularly vulnerableto buffer overflow stack overflow and other classes ofexploits [2] [29] which we must protect against

Figure 1 shows the architecture of the system we havedeveloped to meet these goals We deliver on our goals withtwo core systems built on CSPOT

bull NanoLambda CloudEdge shown on the left in the figureprovides FaaS handler execution to Linux capable devicesas well as object storage It doubles as a centralizedrepository from which IoT devices can request functions

bull NanoLambda On Device shown on the right in the figureprovides on-device handler execution capabilities withIoTPy a Python interpreter tailor-made for FaaS handlerexecution

This separation of concerns is primarily driven by the needsto tailor our device implementation to best perform withinthe constraints of low power non-Linux IoT devices Thisallows us to offload complex and processing-intensive tasksto NanoLambda CloudEdge while keeping NanoLambda OnDevice space- and power-efficient

A NanoLambda CloudEdge ServiceFigure 1 illustrates the NanoLambda CloudEdge service

on the left The core of this service consists of two RESTAPI servers which export (i) an object store compatiblewith the Amazon Simple Storage Service (S3) [4] and (ii)a FaaS service that deploys FaaS functions written for AWSLambda [9] over CSPOTndash in the same way they are deployedover AWS

To enable this the servers leverage multiple CSPOT primi-tives This includes an object store and event model in whichhandlers are triggered in response to object updates CSPOTobjects are fixed-size records in append-only logs Handlersexecute inside Linux containers allowing for concurrent ex-ecution of handlers as isolated processes The NanoLambda

CloudEdge API servers build upon these primitives andCSPOT support for a subset of the S3 and AWS Lambda APIs(for Linux systems) to unify deployment and execution ofAWS Lambda Python functions across Linux and non-Linuxsystems

Specifically the S3 API Emulation Service provides awrapper for the CSPOT append-only object store It storesblobs to CSPOTrsquos fixed-size log records by first splitting theminto 16KB chunks and then linking these chunks together withmetadata records It then stores a mapping between the blobrsquoskey and the location of the first chunk separately in an indexlog Blobs are retrieved via a sequential scan of the indexlog to find their location Once located they are read back inchunks and streamed to the requester This service exports thisfunctionality via the Amazon S3 REST API [5]

The Lambda API Emulation Service manages functiondeployment and facilitates function execution in CSPOT Theservice stores the configuration and code package using theS3 API Emulation Service making it a single store for allpersistent data By offloading state management to a cen-tralized object store these services make it possible to runload-balanced instances across multiple physical machines toimprove performance and scalability

AWS Lambda compatibility is provided via a specialCSPOT handler binary which embeds a Python interpreterWhen an function invocation request is received NanoLambdaCloudEdge appends an invocation record to CSPOTrsquos objectstore This append creates an event which invokes this handlerin an isolated docker instance The handler reads the invocationrecord and checks for the presence of the Python code forthe handler If necessary the code is fetched from the objectstore Once ready the Python interpreter invokes the handlerfunction with the payload from the invocation record andwrites the result to a result log for return to the requester

To support handler execution on device NanoLambdaCloudEdge implements an IoTPy translator (cf Section III-C)which compiles and packages Python handlers for execu-tion on neighboring microcontroller devices This translationservice enables IoT devices to fetch handler code from thesame function repository used by the Lambda API Emulation

4

Service The service compiles handlers on-demand into acompact binary representation that can be sent to the devicefor execution Fetch requests check that the code size andmemory requirements (specified in the function configuration)fit within the resource constraints of the requesting device Ifrejected it is possible for the cloudedge device to execute thefunction on behalf of the IoT device

B NanoLambda On Device

NanoLambda On Device is runs Python handlers on non-Linux IoT devices As shown in Figure 1 (on right) the serviceleverages CSPOT WooF logs on-device for persistent storageand event invocation As for Linux systems FaaS handlers areinvoked in response to data appended to storage logs

As seen in the figure the typical model for NanoLambdaOn Device is that data is produced by threads monitoringsensors on the device and appending it to objects in the objectstore It is also possible for data to be remotely deliveredfor processing over CSPOTrsquos network API Each append tothe object store is processed by CSPOTrsquos event model whichruns a C-language handler function with the new data Similarto the NanoLambda CloudEdgersquos support for Python usinga special Linux handler binary with an embedded Pythoninterpreter we extend NanoLambda CloudEdge with Pythonsupport by registering a special C-language handler functionwhich interprets the handlerrsquos name to be a Python Lambdafunction deployed via the NanoLambda CloudEdge serviceWhen this function is invoked it triggers the handler withthe payload provided by the newly appended object in aninstance of the IoTPy VM By leveraging CSPOTrsquos existingfunctionality we are able to offer Python 3 Lambda supporton device

C IoTPy

To ease development and portability we choose Python asour FaaS programming language because its bytecode is easilyportable and its simple virtual machine implementation canbe easily ported to IoT devices Likewise we package FaaSfunctions using the package format defined for AWS Lambdafor compatibility purposes

To achieve a small code and memory footprint we optedto implement IoTPy our own extremely lightweight Pythoninterpreter built from the ground up with embedding in mindWe found existing solutions such as MicroPython [30] to beunsuitable because they left little additional space for featuressuch as our FaaS runtime and the drivers for our networkingand sensors On our most resource-constrained devices suchas the ESP8266 our binary must be able to fit in as little as512KB of flash memory

IoTPy is written to be as unintrusive as possible with afocus on allowing the Python interpreter to be treated like alibrary To keep IoTPy small we opted to omit the Pythonparserlexer on the device Instead these tasks are offloadedto the NanoLambda CloudEdge service This benefits us in anumber of ways

1) bytecode compilation as well as optimization steps canbe performed by a full Python 36 implementation onNanoLambda CloudEdge service

2) memory use is reduced since we are only shippingcompact bytecode representation to the device and

3) a simpler virtual machine implementation can be used ondevice ndash by delegating compilation to the NanoLambdaCloudEdge service our IoTPy is able to limit its func-tionality to implementing the core language features ofthe Python virtual machine (VM)

Together these capabilities mitigate the additional complexityof maintaining a simple Python bytecode VM To enablethese capabilities we make some tradeoffs with regard toprogram storage and architectural simplicity IoTPy provides aconvenient CC++ interface for extending Python with nativefunctions We currently provide implementations for only themost common Python libraries and emulation of only the AWSS3 and Lambda deployment services The libraries includecommon math functions JSON operations APIs for accessingdevice peripherals basic networking and interfaces for theAPIs provided by NanoLambda CloudEdge IoTPy cannothowever load standard Python packages

When comparing the size of the generated binary files forIoTPy and MicroPython we find that the code for the IoTPylibrary uses 290KB of flash storage whereas MicroPythonrequires 620KB of flash storage Thus the IoTPy imple-mentation leaves us with almost 300KB of additional storageavailable for programs drivers and the FaaS runtime

D Python VM Security and Isolation for FaaS Handlers

IoT devices are often so resource-constrained that theylack security features such as memory protection units toimplement separate address spaces for isolated execution ofuser code Additionally conventional FaaS isolation achievedby Linux features like container-based isolation is simply notpossible on devices like these The authors of [2] and [29] findvarious attacks many of which rely on out-of-range memoryaccess to exploit IoT devices and execute code or update theirfirmware with malicious payloads Yet it is essential that IoTdevelopers be able to reprogram and update their devices inthe field

By placing the majority of application logic in FaaS han-dlers we reduce potential attack surface area A secure Pythonvirtual machine implementation can ensure that the bytecoderunning within it cannot access memory out of range orotherwise escape the permissions allowed to the handler bythe built-in functions provided to it Python handlers can thenbe isolated from one another by running in separate instancesof the Python VM This isolation is similar to traditional Linuxcontainers

E Deploying and Running Functions

By leveraging NanoLambda CloudEdge and its compatibil-ity layer for Amazonrsquos AWS Lambda APIs we minimize theeffort to port functions to the IoTPy platform In many casesfunctions run directly on IoTPy without code changes

5

programmer deploys code toNanoLambda CloudEdge service

with aws cli

NanoLambdaCloudEdge

1 stores code in object store service2 compiles and caches compact

bytecode representation on-demand

when IoT devices find a handler is not cachedlocally they request bytecode representation

from NanoLambda service

Fig 2 Life cycle of a function deployment from the application developerrsquoscommand line to the NanoLambda CloudEdge service to the IoT devicerunning NanoLambda On Device

Function deployment is performed by developers using theaws-cli or the aws-sdk (using a NanoLambda CloudEdgedevice as the cloud target) meaning existing tools capable ofdeploying to AWS Lambda can also deploy to NanoLambdaCloudEdge with few changes Deploying a new version ofan application is done with the ldquoaws lambda create-functionrdquocommand which accepts the application code package as azip file as well as other configuration parameters includingthe function entry-point its memory limit and Python version(Python 36 is currently supported) The deployment processchecks code size and resource limits (as is done in AWSLambda) and also checks for use of unsupported AWS APIs inthe code (eg only AWS REST S3 API is currently supported)The validated function is stored in the S3 API EmulationService as described above

Function execution on device is performed by IoTPy Localinvocations are accelerated by an interpreter cache whichallows the IoTPy interpreter for a given handler to be reusedif it is still available from a previous invocation This isdone to avoid the relatively expensive process of reinitializingPython built-ins as well as functions and globals defined inthe bytecode

In Figure 2 we show the deployment and function downloadprocess mapped out for NanoLambda On Device When theinterpreter is still running (ie because it is already running afunction) execution involves simply setting up a new Pythonframe with the call stack for the function and invoking thefunction in the cached interpreter When no interpreter isrunning the NanoLambda service opens a TCP connectionto NanoLambda CloudEdge and serializes a request for thefunction using flatbuffers [18] a fast and memory efficientbinary protocol for network serialization When NanoLambdaCloudEdge receives this request the function is fetched fromour object store and compiled on the fly using the Linuximplementation of Python 36 and the Python dis package toextract generated bytecode

F Execution Offloading CapabilitiesOur NanoLambda On Device service provides direct han-

dler code compatibility with the NanoLambda CloudEdgeservice We achieve this by adopting the same AWS Lambda-compatible handler format

NanoLambda On Device supports local execution of han-dlers on microcontrollers like the ESP8266 and the CC3220SFNanoLambda CloudEdge leveraging CSPOTrsquos tiered-cloudapproach extends these execution options to include edgecloud nodes like the Raspberry Pi Zero as well as in-cloudsolutions like CSPOT running on AWS EC2 Lastly becauseboth services use a common function format which is code-compatible with AWS Lambda our handlers can be easilytransferred between the services or even executed on AWSLambda itself Selecting to develop application logic in com-partmentalized functions using a high-level language lends thedeveloper immediate portability benefits which grant her thepower to choose where code should be run to best utilizeresources or achieve desired performance

IV EVALUATION

In this section we evaluate NanoLambda on two exampleFaaS applications for sensor monitoring We provide a perfor-mance analysis of NanoLambda using micro-benchmarks toexpose a detailed analysis of its performance characteristicsunder its various configurations Additionally we show howNanoLambdarsquos portability can be leveraged to implement alatency-aware scheduler that migrates execution between thedevice and the edge cloud to improve performance across arange of problem sizes

A Automatic Light Control ApplicationWe show how NanoLambda might be used in an auto-

matic lighting control system ndash for example to automati-cally turn on walkway lighting at night based on ambientlight levels The hardware setup for this experiment is aphotoresistor connected to the analogue-to-digital converter(ADC) pin of an ESP8266 microcontroller This processoris extremely resource-constrained with 80KB of user dataRAM and 512KB of flash storage for code The only device-specific C code written for this experiment is a simple samplerthread which measures the voltage on the ADC pin of themicrocontroller and writes it out to an invocation log at a rateof 5 samples per second

In response to each log append (analogous to publishing toan MQTT queue) an IoTPy handler is invoked to analyze thenew data and send commands to the light system if necessaryThe implementation of the handler for this automatic lightcontrol experiment is shown in Figure 3

def new_light_lvl(payload ctx)if payload[0] gt 100

deviceset_pin_high(2)else

deviceset_pin_low(2)

Fig 3 Handler function for the light control application

6

In Table I we show invocation latencies for various config-urations of the NanoLambda runtime The ldquoNo Server Cacherdquoconfiguration represents the true ldquocold startrdquo cost of invokinga new AWS Lambda function for the first time This numberincludes both the time taken for the server to compile theLambda function and to deliver it to the device The LocalCache configuration shows us the typical amortized runtimeof the application where frequently invoked Lambda functionswill typically be already available on the device We cansee that the performance difference is substantial requiringalmost 32x as much time to invoke a function that is not incache This is largely due to the effects of both compilationtime and the cost of network latency Compilation and codetransfer each take roughly 100 milliseconds on the networktested We include ldquoNo Local Cacherdquo to show the startup costof a function shared by multiple devices In this scenario afunction typically has already been accessed and compiled bythe server but has not yet been delivered to a given deviceWe expect this to be representative of the startup cost for real-world deployments

We include a C implementation of the same handler functionas a baseline It is worth noting that this is on the order of20x faster than our configuration with local caching This is asubstantial overhead but it is representative of what we wouldexpect for the performance difference between an interpretedlanguage without JIT and a compiled language like C [32]Ultimately it is up to the application developer to determinewhen the flexibility and ease of deployment with Python andthe FaaS ecosystem outweigh its overhead

Configuration Latency in msNanoLambda Local Caching 67 msNanoLambda No Local Cache 1195 msNanoLambda No Server Cache 2204 msNanoLambda C Handler 03 ms

TABLE IAVERAGE LATENCY FOR 100 ITERATIONS OF THE HANDLER FUNCTION

B Predictive Maintenance Application

In this experiment we look at the detailed performance ofa NanoLambda application with more demanding processingrequirements compared to the previous example Predictivemaintenance techniques use sensors to detect part failures orother maintenance requirements

Our experiment considers motor maintenance by analyzingthe vibrations off of a motor as measured by an accelerometerIt is possible to detect if the motor or a connected part hasfailed by monitoring for changes in the distribution of themagnitude of vibrations picked up by the accelerometer Ourapplication detects these changes with a Python implementa-tion of the KolmogorovndashSmirnov (KS) test used to comparea reference distribution against real time measurements froman accelerometer The test provides the probability that theempirical distribution matches the reference distribution Apossible failure can be flagged if this probability drops belowa tuned threshold

list containing a reference distribution from accelerometerreference = []

def kstest(datalist1 datalist2) omitted see implementation from gistgithubcomdevries11405101

def new_accel_sample(payload ctx)global referencetransformed = []for record in payload

transformedappend(magnitude(record)))

prob = kstest(transformedreference)

return prob

Fig 4 The implementation of the handler providing the KS test results

In Figure 4 we show the implementation of our KS testhandler The handler is provided with a JSON payload con-taining a list of recent accelerometer data as tuples of (x y z)acceleration These records are normalized to a transformedarray of floating point magnitudes and then passed to anldquooff-the-shelfrdquo open-source implementation of the KS test byChristopher De Vries [27] This KS test implementation iscomputationally intensive and scales linearly with the inputproblem size

In Table II we show the performance of a real-worldinstallation The hardware configuration is a CC3220SF pro-cessor with 1MB of program flash and 256KB of RAM Thisprocessor is wired to an accelerometer monitored by a samplerthread (much like the photoresistor experiment) which collectsdata from the accelerometer 5 times per second and appends itto a data log Every 32 samples a handler is invoked to analyzerecent data The handler is always invoked on device andshows the performance of a more complex handler running onIoTPy The more complex function requires longer executiontime 147ms as well as a longer startup time of 436ms tocompile

We found this handler to be sufficiently computationallyintensive to be a good candidate for offloading to an edge clouddevice To this end we next present a scheduler for offloadingcomputationally-intensive tasks to remote FaaS providers

C Saving Power With Execution OffloadingIn Figure 5 we show the power used by the CC3220SF

microcontroller for KS problem sizes 20 and 80 over the

Configuration Latency (ms) Power Use (mJ) Memory Use KBNanoLambda Local Caching 2094 ms 2123 mJ 236KBNanoLambda No Local Cache 7290 ms 8500 mJ 217KB

TABLE IIAVERAGE TIMESPOWER CONSUMPTION OVER 100 ITERATIONS OF THE

HANDLER FUNCTION ON THE CC3220SF MICROCONTROLLER WITH A KSPROBLEM SIZE OF 32 IN EACH CONFIGURATION

7

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600Po

wer D

raw

in (m

W)

Power Use Over Local Request Lifecycle for Problem Size 20Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 20Remote Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w in

(mW

)

Power Use Over Local Request Lifecycle for Problem Size 80Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 80Remote Request

Fig 5 Comparison of power draw over request lifecycle for various KSproblem sizes using both remote and local strategies

life cycle of a request We observe here that with the ex-ception of some power spikes due to background activity onthe microcontroller the power used by the local request iscontinuous from the beginning of the invocation until the resultis available and so it grows with the size of the problem (plussome constant overhead from the FaaS runtime) With remoteexecution we see just over twice the peak power draw of thelocal execution strategy but we see a trough in the middle ofthe request life cycle where power use drops to near zeroThis is because the device is free to completely idle whilewaiting for results to return from the server Therefore wepay a large but short-lived power cost during transmissionbut when enough computation can be performed in the cloudwe can save power overall relative to local execution

D Predictive Maintenance Application with a Naive Offload-ing Scheduler

In this application we take advantage of our code compat-ibility between NanoLambda CloudEdge and NanoLambdaOn Device to extend our predictive maintenance applicationrunning on device with the capability to dynamically offloadcomputation to the edge

Our scheduler offloads computation when its algorithmestimates that better latency can be achieved by transferringexecution to the cloud We model execution latency by check-ing the runtime of the last execution on NanoLambda OnDevice and comparing it against the round trip latency of thelast invocation on NanoLambda CloudEdge The service withthe lowest total latency is selected to run the handler

However as conditions change execution-time measure-ments can become stale Much like the authors of [7] weperiodically retest each possible execution strategy to allow thescheduler to sense when the relative performance has changedeg as a result of evolving network conditions or changesin problem complexity In this experiment our scheduler isconfigured to retest the losing strategy 1 out of every 16invocations

To benchmark the scheduler we measure the average in-vocation latency for a range of synthetic problems from 5data points to 100 data points We examine problem sizes atmultiples of 5 with 1000 trials at each size

Figure 6 shows average invocation latency as a functionof problem size for the scheduler a remote-only invocationstrategy and a local-only invocation strategy Local invoca-tions provide very consistent runtime which scales linearlywith the problem size Remote invocation on the other handis highly subject to network latency and congestion as wecan see from occasional spikes The scheduled invocationsappear to offer the best of both worlds scaling linearly untilthe latency exceeds the average network latency at which pointthe strategy switches to remote invocation

Figure 7 shows average power per invocation as a functionof problem size for the scheduler the remote-only invocationstrategy and the local-only invocation strategy We see similartrends as the previous the latency graph ndash remote executionuses relatively constant power while local execution powerconsumption scales with execution time We see again thatthe scheduler offers the best of both strategies Interestinglyhowever even though the scheduler begins to switch to remoteinvocations at problem size 40 we do not observe powersavings from this until the problem size reaches 60

A more detailed breakdown of the choices made by thescheduler strategy are shown in Figure 8 The top of thischart shows the execution latency while the bottom shows thenumber of calculations run locally versus the number offloadedover the network Initially all calculations run on device forsmall problem sizes however once the problem size exceeds40 it becomes intermittently worthwhile to run over thenetwork when latency is low After the problem size exceeds60 almost all invocations run remote as the speed differencebetween on-device execution and remote execution outweighsthe latency cost of transferring data over the network Weobserve however that this naive scheduling algorithm leavessome room for improvement First with a fixed re-testinglatency we allocate more cycles to re-testing than may benecessary As we can see in this graph the scheduler isalways spending 601000 cycles re-testing the losing strategyWhen there is a large difference in performance this can be

8

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 6 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 7 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

a substantial overhead Second the scheduler has difficultyswitching to remote invocations when optimal as it is subjectto temporary network spikes

E An Improved Scheduler for Predictive Maintenance

Learning from the performance of our naive scheduler wenext construct a model adjusts to short-lived network spikesand that reduces the amount of time spent retesting To thisend we devise a new scoring function that incorporates anexponential term to adjust dynamically the re-testing intervalIn addition we use the median latency of recent requests as away of filtering outlier measurements

We define the new score function to be

score = median(H)minus (1 + r)d

Where H is the invocation latency history for the strategy(local or remote) being scored r is the decay rate which isset to r = 12 and d is the decay term ndash the number of

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)

Avg Invocation Latency vs Problem Size for Naive Scheduler

20 40 60 80 100ks problem size

20

40

60

80

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Naive Scheduler

remote invocationslocal invocations

Fig 8 Top average invocation latency as a function of KS problem size forthe offloading scheduler Bottom local invocation count vs remote invocationcount selected by the offloading scheduler for each problem size

invocations since this strategy was last selected We take themedian over a window of 3 samples which serves to filterout any one unusually fast or slow result allowing us to moreeasily recover from lost packets or other short-lived networkspikes

In Figure 9 we can see how this invocation strategy breaksdown in terms of request distribution When compared withthe naive schedulerrsquos strategy breakdown in Figure 8 weobserve that the improved scheduler begins to switch schedul-ing strategies slightly earlier than the naive scheduler It isinteresting to see that the improved scheduler with its noisefiltering transitions smoothly from local to remote invocationsas the problem size gradually increases When problem sizesare in the range from approximately 50 to 65 the schedulermixes local and remote invocations as a result of variations innetwork latency etc

With the improved scheduler the transition to remote exe-cution is much smoother and completes much earlier than theswitch-over seen in the naive strategy The naive strategy failsto fully commit to remote execution until a problem size of 75or 80 whereas the improved scheduler has mostly switchedover by the time the problem size reaches 65 Even when thelatency difference between remote execution is large as is thecase with problem sizes greater than 80 the naive schedulerappears to occasionally switch back to local execution This ismost likely a result of network timeouts which go unfiltered in

9

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

II RELATED WORK

Functions as a Service (FaaS) is a cloud-computing platformwith which developers write code without consideration forthe hardware on which it runs Applications are broken downinto distinct units of functionality and packaged as functionsthat can be distributed and updated on the fly Functions areexecuted in lightweight containers which provide portabilityscalability and security functions are automatically scaledacross multiple machines while execution is isolated fromother tenants of those instances A number of major cloudproviders such as AWS Lambda [9] Google Cloud Functions[22] and Microsoft Azure Functions [11] now offer hostedFaaS platforms

Because of its automatic scaling and convenience FaaS hasseen increased adoption in IoT in recent years AWS IoTfor example leverages FaaS functions extensively in order toallow users of the platform to attach functions that handle IoTevents [17] Serverless [34] a leading industry toolchain forFaaS programming estimates that IoT makes up 6 [1] ofthe use of FaaS in industry

An emerging development in IoT data processing is theprospect of FaaS at the edge for data processing Commercialofferings such as AWS GreenGrass [24] and open sourceApache OpenWhisk [6] make it possible for users to host theirown FaaS deployments on low-power devices By runningfunctions at the edge the hope is that handlers can berun closer to where data is produced making better use oflocally-available processing and reducing latency penalties andtransfer costs The authors of [31] explore the concept of aserverless platform for real-time data analytics at the edge anddiscuss the benefits and challenges of the approach

CSPOT is an open source low-level distributed operat-ing system for IoT that executes a uniform FaaS modelacross devices ranging from servers in the cloud to resource-constrained sensors and microcontrollers [36] CSPOT exportsa distributed storage abstraction called a WooF for persistentstorage of FaaS function objects WooFs are append-onlylogs with built-in event tracking and repair capabilities [28]WooFs serve both the role of state storage and executionlog ndash appending data to WooFs optionally can trigger theexecution of an event handler that responds to the appendeddata Functions (ie event handlers) are implemented as Linuxexecutables allowing for flexibility of language choice andlibraries A pool of docker containers is maintained to runhandlers ndash by running only one handler in a container atany given time functions are isolated from one another Thisenables development of complex applications and provides thesecurity portability and scalability expected of FaaS systems

For Linux systems CSPOT provides an execution envi-ronment and AWS API compatibility (for S3 and Lambdaservices) for Python functions However this support does notextend to microcontrollers (which we address with the workherein) To execute C-language functions on microcontrollersCSPOT uses a threaded multitasking system with direct mem-ory addressing This support is available for ARM AVR and

Espressif microcontrollers FaaS functions in CSPOT must becompiled and installed on a per-device basis Our work enablesPython functions (including Python AWS Lambda functions)to be executed over CSPOT including on microcontrollersWe do so using a combination of automatic translation at theedge and interpretation on-device to enable both portabilityand efficiency

When functions can be made highly portable as is the casewith FaaS interesting possibilities for improving performanceemerge such as scheduled execution The authors of [7]propose a framework for scheduling execution among multipleFaaS providers The authors develop an extensible API forexecuting functions across multiple providers as well as ametrics database that tracks execution statistics from theseservices Once a model of function performance has been builtfor a given provider this information can be provided to a user-defined scheduler eg a Python function which selects theprovider with the minimum latency The authors find that byemploying a scheduler they can achieve a 200 faster averageround-trip execution time compared to executing on any oneservice provider This serves as a promising example of howthe portable nature of FaaS functions can provide applicationdevelopers with more flexible execution options and improvedperformance

III NanoLambda DESIGN AND IMPLEMENTATION

At present writing applications for IoT means programmingfor APIs and libraries specific to each device writing functionsthat are ldquolocked inrdquo to particular cloud provider APIs andusing only those devices that are supported by these platformsIt is our objective to support a common programming modeleverywhere extending from IoT devices to the edge to thecloud In service of this goal we opted to design our systemaround CSPOT a low-level framework for supporting IoTFaaS CSPOT provides the foundation for our high-level FaaSruntime with powerful cross-platform primitives for securefunction (event handler) execution and data persistence

However existing cloud-based FaaS offerings that have seenadoption in industry use high level languages such as Pythonfor their handlers These high-level languages are particularlyadvantageous for portability and programmer productivityThe CSPOT service while capable of running with minimaloverhead on a wide range of severely resource-constraineddevices defines its own API Thus while CSPOT is itselfportable it cannot execute FaaS functions written for existingcloud services such as AWS Lambda

It is our view that in order to provide the same conveniencecompatibility and security as FaaS handlers executing inthe cloud a FaaS system for resource-restricted non-Linuxdevices should have the following properties

1) Ease of development ndash we wish for NanoLambda tobe familiar to developers already using FaaS ideallyallowing existing FaaS functions to run with little tono code changes

3

CSPOT BackendObject Store

Python 36CSPOTHandler

CSPOT WooFInvocation

Log IoTPyCodeAPItranslation

and packaging

service

NanoLambda CloudEdge NanoLambda On Device

IoTPYon deviceand remoteexecution

capabilities

CSPOT WooF measurement

logs

Sensors amp drivers custom OS with monitoring C++ threads

Lambda APIEmulationService

S3 APIEmulationService

Fig 1 Service diagram outlining the interactions of the NanoLambda components that deliver FaaS for IoT Shaded components are those added to CSPOTby NanoLambda NanoLambda CloudEdge runs on Linux hosts in the cloud or at the edge NanoLambda On Device runs on microcontrollers and non-Linuxsystems

2) Portability ndash we want portable application code so thatthe same implementation can run in the cloud or on thedevice with little to no additional developer effort

3) Small code and memory footprint ndash NanoLambda shouldleave as much storage and memory space as possibleavailable for libraries and C extensions

4) Security ndash IoT devices can be particularly vulnerableto buffer overflow stack overflow and other classes ofexploits [2] [29] which we must protect against

Figure 1 shows the architecture of the system we havedeveloped to meet these goals We deliver on our goals withtwo core systems built on CSPOT

bull NanoLambda CloudEdge shown on the left in the figureprovides FaaS handler execution to Linux capable devicesas well as object storage It doubles as a centralizedrepository from which IoT devices can request functions

bull NanoLambda On Device shown on the right in the figureprovides on-device handler execution capabilities withIoTPy a Python interpreter tailor-made for FaaS handlerexecution

This separation of concerns is primarily driven by the needsto tailor our device implementation to best perform withinthe constraints of low power non-Linux IoT devices Thisallows us to offload complex and processing-intensive tasksto NanoLambda CloudEdge while keeping NanoLambda OnDevice space- and power-efficient

A NanoLambda CloudEdge ServiceFigure 1 illustrates the NanoLambda CloudEdge service

on the left The core of this service consists of two RESTAPI servers which export (i) an object store compatiblewith the Amazon Simple Storage Service (S3) [4] and (ii)a FaaS service that deploys FaaS functions written for AWSLambda [9] over CSPOTndash in the same way they are deployedover AWS

To enable this the servers leverage multiple CSPOT primi-tives This includes an object store and event model in whichhandlers are triggered in response to object updates CSPOTobjects are fixed-size records in append-only logs Handlersexecute inside Linux containers allowing for concurrent ex-ecution of handlers as isolated processes The NanoLambda

CloudEdge API servers build upon these primitives andCSPOT support for a subset of the S3 and AWS Lambda APIs(for Linux systems) to unify deployment and execution ofAWS Lambda Python functions across Linux and non-Linuxsystems

Specifically the S3 API Emulation Service provides awrapper for the CSPOT append-only object store It storesblobs to CSPOTrsquos fixed-size log records by first splitting theminto 16KB chunks and then linking these chunks together withmetadata records It then stores a mapping between the blobrsquoskey and the location of the first chunk separately in an indexlog Blobs are retrieved via a sequential scan of the indexlog to find their location Once located they are read back inchunks and streamed to the requester This service exports thisfunctionality via the Amazon S3 REST API [5]

The Lambda API Emulation Service manages functiondeployment and facilitates function execution in CSPOT Theservice stores the configuration and code package using theS3 API Emulation Service making it a single store for allpersistent data By offloading state management to a cen-tralized object store these services make it possible to runload-balanced instances across multiple physical machines toimprove performance and scalability

AWS Lambda compatibility is provided via a specialCSPOT handler binary which embeds a Python interpreterWhen an function invocation request is received NanoLambdaCloudEdge appends an invocation record to CSPOTrsquos objectstore This append creates an event which invokes this handlerin an isolated docker instance The handler reads the invocationrecord and checks for the presence of the Python code forthe handler If necessary the code is fetched from the objectstore Once ready the Python interpreter invokes the handlerfunction with the payload from the invocation record andwrites the result to a result log for return to the requester

To support handler execution on device NanoLambdaCloudEdge implements an IoTPy translator (cf Section III-C)which compiles and packages Python handlers for execu-tion on neighboring microcontroller devices This translationservice enables IoT devices to fetch handler code from thesame function repository used by the Lambda API Emulation

4

Service The service compiles handlers on-demand into acompact binary representation that can be sent to the devicefor execution Fetch requests check that the code size andmemory requirements (specified in the function configuration)fit within the resource constraints of the requesting device Ifrejected it is possible for the cloudedge device to execute thefunction on behalf of the IoT device

B NanoLambda On Device

NanoLambda On Device is runs Python handlers on non-Linux IoT devices As shown in Figure 1 (on right) the serviceleverages CSPOT WooF logs on-device for persistent storageand event invocation As for Linux systems FaaS handlers areinvoked in response to data appended to storage logs

As seen in the figure the typical model for NanoLambdaOn Device is that data is produced by threads monitoringsensors on the device and appending it to objects in the objectstore It is also possible for data to be remotely deliveredfor processing over CSPOTrsquos network API Each append tothe object store is processed by CSPOTrsquos event model whichruns a C-language handler function with the new data Similarto the NanoLambda CloudEdgersquos support for Python usinga special Linux handler binary with an embedded Pythoninterpreter we extend NanoLambda CloudEdge with Pythonsupport by registering a special C-language handler functionwhich interprets the handlerrsquos name to be a Python Lambdafunction deployed via the NanoLambda CloudEdge serviceWhen this function is invoked it triggers the handler withthe payload provided by the newly appended object in aninstance of the IoTPy VM By leveraging CSPOTrsquos existingfunctionality we are able to offer Python 3 Lambda supporton device

C IoTPy

To ease development and portability we choose Python asour FaaS programming language because its bytecode is easilyportable and its simple virtual machine implementation canbe easily ported to IoT devices Likewise we package FaaSfunctions using the package format defined for AWS Lambdafor compatibility purposes

To achieve a small code and memory footprint we optedto implement IoTPy our own extremely lightweight Pythoninterpreter built from the ground up with embedding in mindWe found existing solutions such as MicroPython [30] to beunsuitable because they left little additional space for featuressuch as our FaaS runtime and the drivers for our networkingand sensors On our most resource-constrained devices suchas the ESP8266 our binary must be able to fit in as little as512KB of flash memory

IoTPy is written to be as unintrusive as possible with afocus on allowing the Python interpreter to be treated like alibrary To keep IoTPy small we opted to omit the Pythonparserlexer on the device Instead these tasks are offloadedto the NanoLambda CloudEdge service This benefits us in anumber of ways

1) bytecode compilation as well as optimization steps canbe performed by a full Python 36 implementation onNanoLambda CloudEdge service

2) memory use is reduced since we are only shippingcompact bytecode representation to the device and

3) a simpler virtual machine implementation can be used ondevice ndash by delegating compilation to the NanoLambdaCloudEdge service our IoTPy is able to limit its func-tionality to implementing the core language features ofthe Python virtual machine (VM)

Together these capabilities mitigate the additional complexityof maintaining a simple Python bytecode VM To enablethese capabilities we make some tradeoffs with regard toprogram storage and architectural simplicity IoTPy provides aconvenient CC++ interface for extending Python with nativefunctions We currently provide implementations for only themost common Python libraries and emulation of only the AWSS3 and Lambda deployment services The libraries includecommon math functions JSON operations APIs for accessingdevice peripherals basic networking and interfaces for theAPIs provided by NanoLambda CloudEdge IoTPy cannothowever load standard Python packages

When comparing the size of the generated binary files forIoTPy and MicroPython we find that the code for the IoTPylibrary uses 290KB of flash storage whereas MicroPythonrequires 620KB of flash storage Thus the IoTPy imple-mentation leaves us with almost 300KB of additional storageavailable for programs drivers and the FaaS runtime

D Python VM Security and Isolation for FaaS Handlers

IoT devices are often so resource-constrained that theylack security features such as memory protection units toimplement separate address spaces for isolated execution ofuser code Additionally conventional FaaS isolation achievedby Linux features like container-based isolation is simply notpossible on devices like these The authors of [2] and [29] findvarious attacks many of which rely on out-of-range memoryaccess to exploit IoT devices and execute code or update theirfirmware with malicious payloads Yet it is essential that IoTdevelopers be able to reprogram and update their devices inthe field

By placing the majority of application logic in FaaS han-dlers we reduce potential attack surface area A secure Pythonvirtual machine implementation can ensure that the bytecoderunning within it cannot access memory out of range orotherwise escape the permissions allowed to the handler bythe built-in functions provided to it Python handlers can thenbe isolated from one another by running in separate instancesof the Python VM This isolation is similar to traditional Linuxcontainers

E Deploying and Running Functions

By leveraging NanoLambda CloudEdge and its compatibil-ity layer for Amazonrsquos AWS Lambda APIs we minimize theeffort to port functions to the IoTPy platform In many casesfunctions run directly on IoTPy without code changes

5

programmer deploys code toNanoLambda CloudEdge service

with aws cli

NanoLambdaCloudEdge

1 stores code in object store service2 compiles and caches compact

bytecode representation on-demand

when IoT devices find a handler is not cachedlocally they request bytecode representation

from NanoLambda service

Fig 2 Life cycle of a function deployment from the application developerrsquoscommand line to the NanoLambda CloudEdge service to the IoT devicerunning NanoLambda On Device

Function deployment is performed by developers using theaws-cli or the aws-sdk (using a NanoLambda CloudEdgedevice as the cloud target) meaning existing tools capable ofdeploying to AWS Lambda can also deploy to NanoLambdaCloudEdge with few changes Deploying a new version ofan application is done with the ldquoaws lambda create-functionrdquocommand which accepts the application code package as azip file as well as other configuration parameters includingthe function entry-point its memory limit and Python version(Python 36 is currently supported) The deployment processchecks code size and resource limits (as is done in AWSLambda) and also checks for use of unsupported AWS APIs inthe code (eg only AWS REST S3 API is currently supported)The validated function is stored in the S3 API EmulationService as described above

Function execution on device is performed by IoTPy Localinvocations are accelerated by an interpreter cache whichallows the IoTPy interpreter for a given handler to be reusedif it is still available from a previous invocation This isdone to avoid the relatively expensive process of reinitializingPython built-ins as well as functions and globals defined inthe bytecode

In Figure 2 we show the deployment and function downloadprocess mapped out for NanoLambda On Device When theinterpreter is still running (ie because it is already running afunction) execution involves simply setting up a new Pythonframe with the call stack for the function and invoking thefunction in the cached interpreter When no interpreter isrunning the NanoLambda service opens a TCP connectionto NanoLambda CloudEdge and serializes a request for thefunction using flatbuffers [18] a fast and memory efficientbinary protocol for network serialization When NanoLambdaCloudEdge receives this request the function is fetched fromour object store and compiled on the fly using the Linuximplementation of Python 36 and the Python dis package toextract generated bytecode

F Execution Offloading CapabilitiesOur NanoLambda On Device service provides direct han-

dler code compatibility with the NanoLambda CloudEdgeservice We achieve this by adopting the same AWS Lambda-compatible handler format

NanoLambda On Device supports local execution of han-dlers on microcontrollers like the ESP8266 and the CC3220SFNanoLambda CloudEdge leveraging CSPOTrsquos tiered-cloudapproach extends these execution options to include edgecloud nodes like the Raspberry Pi Zero as well as in-cloudsolutions like CSPOT running on AWS EC2 Lastly becauseboth services use a common function format which is code-compatible with AWS Lambda our handlers can be easilytransferred between the services or even executed on AWSLambda itself Selecting to develop application logic in com-partmentalized functions using a high-level language lends thedeveloper immediate portability benefits which grant her thepower to choose where code should be run to best utilizeresources or achieve desired performance

IV EVALUATION

In this section we evaluate NanoLambda on two exampleFaaS applications for sensor monitoring We provide a perfor-mance analysis of NanoLambda using micro-benchmarks toexpose a detailed analysis of its performance characteristicsunder its various configurations Additionally we show howNanoLambdarsquos portability can be leveraged to implement alatency-aware scheduler that migrates execution between thedevice and the edge cloud to improve performance across arange of problem sizes

A Automatic Light Control ApplicationWe show how NanoLambda might be used in an auto-

matic lighting control system ndash for example to automati-cally turn on walkway lighting at night based on ambientlight levels The hardware setup for this experiment is aphotoresistor connected to the analogue-to-digital converter(ADC) pin of an ESP8266 microcontroller This processoris extremely resource-constrained with 80KB of user dataRAM and 512KB of flash storage for code The only device-specific C code written for this experiment is a simple samplerthread which measures the voltage on the ADC pin of themicrocontroller and writes it out to an invocation log at a rateof 5 samples per second

In response to each log append (analogous to publishing toan MQTT queue) an IoTPy handler is invoked to analyze thenew data and send commands to the light system if necessaryThe implementation of the handler for this automatic lightcontrol experiment is shown in Figure 3

def new_light_lvl(payload ctx)if payload[0] gt 100

deviceset_pin_high(2)else

deviceset_pin_low(2)

Fig 3 Handler function for the light control application

6

In Table I we show invocation latencies for various config-urations of the NanoLambda runtime The ldquoNo Server Cacherdquoconfiguration represents the true ldquocold startrdquo cost of invokinga new AWS Lambda function for the first time This numberincludes both the time taken for the server to compile theLambda function and to deliver it to the device The LocalCache configuration shows us the typical amortized runtimeof the application where frequently invoked Lambda functionswill typically be already available on the device We cansee that the performance difference is substantial requiringalmost 32x as much time to invoke a function that is not incache This is largely due to the effects of both compilationtime and the cost of network latency Compilation and codetransfer each take roughly 100 milliseconds on the networktested We include ldquoNo Local Cacherdquo to show the startup costof a function shared by multiple devices In this scenario afunction typically has already been accessed and compiled bythe server but has not yet been delivered to a given deviceWe expect this to be representative of the startup cost for real-world deployments

We include a C implementation of the same handler functionas a baseline It is worth noting that this is on the order of20x faster than our configuration with local caching This is asubstantial overhead but it is representative of what we wouldexpect for the performance difference between an interpretedlanguage without JIT and a compiled language like C [32]Ultimately it is up to the application developer to determinewhen the flexibility and ease of deployment with Python andthe FaaS ecosystem outweigh its overhead

Configuration Latency in msNanoLambda Local Caching 67 msNanoLambda No Local Cache 1195 msNanoLambda No Server Cache 2204 msNanoLambda C Handler 03 ms

TABLE IAVERAGE LATENCY FOR 100 ITERATIONS OF THE HANDLER FUNCTION

B Predictive Maintenance Application

In this experiment we look at the detailed performance ofa NanoLambda application with more demanding processingrequirements compared to the previous example Predictivemaintenance techniques use sensors to detect part failures orother maintenance requirements

Our experiment considers motor maintenance by analyzingthe vibrations off of a motor as measured by an accelerometerIt is possible to detect if the motor or a connected part hasfailed by monitoring for changes in the distribution of themagnitude of vibrations picked up by the accelerometer Ourapplication detects these changes with a Python implementa-tion of the KolmogorovndashSmirnov (KS) test used to comparea reference distribution against real time measurements froman accelerometer The test provides the probability that theempirical distribution matches the reference distribution Apossible failure can be flagged if this probability drops belowa tuned threshold

list containing a reference distribution from accelerometerreference = []

def kstest(datalist1 datalist2) omitted see implementation from gistgithubcomdevries11405101

def new_accel_sample(payload ctx)global referencetransformed = []for record in payload

transformedappend(magnitude(record)))

prob = kstest(transformedreference)

return prob

Fig 4 The implementation of the handler providing the KS test results

In Figure 4 we show the implementation of our KS testhandler The handler is provided with a JSON payload con-taining a list of recent accelerometer data as tuples of (x y z)acceleration These records are normalized to a transformedarray of floating point magnitudes and then passed to anldquooff-the-shelfrdquo open-source implementation of the KS test byChristopher De Vries [27] This KS test implementation iscomputationally intensive and scales linearly with the inputproblem size

In Table II we show the performance of a real-worldinstallation The hardware configuration is a CC3220SF pro-cessor with 1MB of program flash and 256KB of RAM Thisprocessor is wired to an accelerometer monitored by a samplerthread (much like the photoresistor experiment) which collectsdata from the accelerometer 5 times per second and appends itto a data log Every 32 samples a handler is invoked to analyzerecent data The handler is always invoked on device andshows the performance of a more complex handler running onIoTPy The more complex function requires longer executiontime 147ms as well as a longer startup time of 436ms tocompile

We found this handler to be sufficiently computationallyintensive to be a good candidate for offloading to an edge clouddevice To this end we next present a scheduler for offloadingcomputationally-intensive tasks to remote FaaS providers

C Saving Power With Execution OffloadingIn Figure 5 we show the power used by the CC3220SF

microcontroller for KS problem sizes 20 and 80 over the

Configuration Latency (ms) Power Use (mJ) Memory Use KBNanoLambda Local Caching 2094 ms 2123 mJ 236KBNanoLambda No Local Cache 7290 ms 8500 mJ 217KB

TABLE IIAVERAGE TIMESPOWER CONSUMPTION OVER 100 ITERATIONS OF THE

HANDLER FUNCTION ON THE CC3220SF MICROCONTROLLER WITH A KSPROBLEM SIZE OF 32 IN EACH CONFIGURATION

7

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600Po

wer D

raw

in (m

W)

Power Use Over Local Request Lifecycle for Problem Size 20Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 20Remote Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w in

(mW

)

Power Use Over Local Request Lifecycle for Problem Size 80Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 80Remote Request

Fig 5 Comparison of power draw over request lifecycle for various KSproblem sizes using both remote and local strategies

life cycle of a request We observe here that with the ex-ception of some power spikes due to background activity onthe microcontroller the power used by the local request iscontinuous from the beginning of the invocation until the resultis available and so it grows with the size of the problem (plussome constant overhead from the FaaS runtime) With remoteexecution we see just over twice the peak power draw of thelocal execution strategy but we see a trough in the middle ofthe request life cycle where power use drops to near zeroThis is because the device is free to completely idle whilewaiting for results to return from the server Therefore wepay a large but short-lived power cost during transmissionbut when enough computation can be performed in the cloudwe can save power overall relative to local execution

D Predictive Maintenance Application with a Naive Offload-ing Scheduler

In this application we take advantage of our code compat-ibility between NanoLambda CloudEdge and NanoLambdaOn Device to extend our predictive maintenance applicationrunning on device with the capability to dynamically offloadcomputation to the edge

Our scheduler offloads computation when its algorithmestimates that better latency can be achieved by transferringexecution to the cloud We model execution latency by check-ing the runtime of the last execution on NanoLambda OnDevice and comparing it against the round trip latency of thelast invocation on NanoLambda CloudEdge The service withthe lowest total latency is selected to run the handler

However as conditions change execution-time measure-ments can become stale Much like the authors of [7] weperiodically retest each possible execution strategy to allow thescheduler to sense when the relative performance has changedeg as a result of evolving network conditions or changesin problem complexity In this experiment our scheduler isconfigured to retest the losing strategy 1 out of every 16invocations

To benchmark the scheduler we measure the average in-vocation latency for a range of synthetic problems from 5data points to 100 data points We examine problem sizes atmultiples of 5 with 1000 trials at each size

Figure 6 shows average invocation latency as a functionof problem size for the scheduler a remote-only invocationstrategy and a local-only invocation strategy Local invoca-tions provide very consistent runtime which scales linearlywith the problem size Remote invocation on the other handis highly subject to network latency and congestion as wecan see from occasional spikes The scheduled invocationsappear to offer the best of both worlds scaling linearly untilthe latency exceeds the average network latency at which pointthe strategy switches to remote invocation

Figure 7 shows average power per invocation as a functionof problem size for the scheduler the remote-only invocationstrategy and the local-only invocation strategy We see similartrends as the previous the latency graph ndash remote executionuses relatively constant power while local execution powerconsumption scales with execution time We see again thatthe scheduler offers the best of both strategies Interestinglyhowever even though the scheduler begins to switch to remoteinvocations at problem size 40 we do not observe powersavings from this until the problem size reaches 60

A more detailed breakdown of the choices made by thescheduler strategy are shown in Figure 8 The top of thischart shows the execution latency while the bottom shows thenumber of calculations run locally versus the number offloadedover the network Initially all calculations run on device forsmall problem sizes however once the problem size exceeds40 it becomes intermittently worthwhile to run over thenetwork when latency is low After the problem size exceeds60 almost all invocations run remote as the speed differencebetween on-device execution and remote execution outweighsthe latency cost of transferring data over the network Weobserve however that this naive scheduling algorithm leavessome room for improvement First with a fixed re-testinglatency we allocate more cycles to re-testing than may benecessary As we can see in this graph the scheduler isalways spending 601000 cycles re-testing the losing strategyWhen there is a large difference in performance this can be

8

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 6 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 7 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

a substantial overhead Second the scheduler has difficultyswitching to remote invocations when optimal as it is subjectto temporary network spikes

E An Improved Scheduler for Predictive Maintenance

Learning from the performance of our naive scheduler wenext construct a model adjusts to short-lived network spikesand that reduces the amount of time spent retesting To thisend we devise a new scoring function that incorporates anexponential term to adjust dynamically the re-testing intervalIn addition we use the median latency of recent requests as away of filtering outlier measurements

We define the new score function to be

score = median(H)minus (1 + r)d

Where H is the invocation latency history for the strategy(local or remote) being scored r is the decay rate which isset to r = 12 and d is the decay term ndash the number of

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)

Avg Invocation Latency vs Problem Size for Naive Scheduler

20 40 60 80 100ks problem size

20

40

60

80

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Naive Scheduler

remote invocationslocal invocations

Fig 8 Top average invocation latency as a function of KS problem size forthe offloading scheduler Bottom local invocation count vs remote invocationcount selected by the offloading scheduler for each problem size

invocations since this strategy was last selected We take themedian over a window of 3 samples which serves to filterout any one unusually fast or slow result allowing us to moreeasily recover from lost packets or other short-lived networkspikes

In Figure 9 we can see how this invocation strategy breaksdown in terms of request distribution When compared withthe naive schedulerrsquos strategy breakdown in Figure 8 weobserve that the improved scheduler begins to switch schedul-ing strategies slightly earlier than the naive scheduler It isinteresting to see that the improved scheduler with its noisefiltering transitions smoothly from local to remote invocationsas the problem size gradually increases When problem sizesare in the range from approximately 50 to 65 the schedulermixes local and remote invocations as a result of variations innetwork latency etc

With the improved scheduler the transition to remote exe-cution is much smoother and completes much earlier than theswitch-over seen in the naive strategy The naive strategy failsto fully commit to remote execution until a problem size of 75or 80 whereas the improved scheduler has mostly switchedover by the time the problem size reaches 65 Even when thelatency difference between remote execution is large as is thecase with problem sizes greater than 80 the naive schedulerappears to occasionally switch back to local execution This ismost likely a result of network timeouts which go unfiltered in

9

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

CSPOT BackendObject Store

Python 36CSPOTHandler

CSPOT WooFInvocation

Log IoTPyCodeAPItranslation

and packaging

service

NanoLambda CloudEdge NanoLambda On Device

IoTPYon deviceand remoteexecution

capabilities

CSPOT WooF measurement

logs

Sensors amp drivers custom OS with monitoring C++ threads

Lambda APIEmulationService

S3 APIEmulationService

Fig 1 Service diagram outlining the interactions of the NanoLambda components that deliver FaaS for IoT Shaded components are those added to CSPOTby NanoLambda NanoLambda CloudEdge runs on Linux hosts in the cloud or at the edge NanoLambda On Device runs on microcontrollers and non-Linuxsystems

2) Portability ndash we want portable application code so thatthe same implementation can run in the cloud or on thedevice with little to no additional developer effort

3) Small code and memory footprint ndash NanoLambda shouldleave as much storage and memory space as possibleavailable for libraries and C extensions

4) Security ndash IoT devices can be particularly vulnerableto buffer overflow stack overflow and other classes ofexploits [2] [29] which we must protect against

Figure 1 shows the architecture of the system we havedeveloped to meet these goals We deliver on our goals withtwo core systems built on CSPOT

bull NanoLambda CloudEdge shown on the left in the figureprovides FaaS handler execution to Linux capable devicesas well as object storage It doubles as a centralizedrepository from which IoT devices can request functions

bull NanoLambda On Device shown on the right in the figureprovides on-device handler execution capabilities withIoTPy a Python interpreter tailor-made for FaaS handlerexecution

This separation of concerns is primarily driven by the needsto tailor our device implementation to best perform withinthe constraints of low power non-Linux IoT devices Thisallows us to offload complex and processing-intensive tasksto NanoLambda CloudEdge while keeping NanoLambda OnDevice space- and power-efficient

A NanoLambda CloudEdge ServiceFigure 1 illustrates the NanoLambda CloudEdge service

on the left The core of this service consists of two RESTAPI servers which export (i) an object store compatiblewith the Amazon Simple Storage Service (S3) [4] and (ii)a FaaS service that deploys FaaS functions written for AWSLambda [9] over CSPOTndash in the same way they are deployedover AWS

To enable this the servers leverage multiple CSPOT primi-tives This includes an object store and event model in whichhandlers are triggered in response to object updates CSPOTobjects are fixed-size records in append-only logs Handlersexecute inside Linux containers allowing for concurrent ex-ecution of handlers as isolated processes The NanoLambda

CloudEdge API servers build upon these primitives andCSPOT support for a subset of the S3 and AWS Lambda APIs(for Linux systems) to unify deployment and execution ofAWS Lambda Python functions across Linux and non-Linuxsystems

Specifically the S3 API Emulation Service provides awrapper for the CSPOT append-only object store It storesblobs to CSPOTrsquos fixed-size log records by first splitting theminto 16KB chunks and then linking these chunks together withmetadata records It then stores a mapping between the blobrsquoskey and the location of the first chunk separately in an indexlog Blobs are retrieved via a sequential scan of the indexlog to find their location Once located they are read back inchunks and streamed to the requester This service exports thisfunctionality via the Amazon S3 REST API [5]

The Lambda API Emulation Service manages functiondeployment and facilitates function execution in CSPOT Theservice stores the configuration and code package using theS3 API Emulation Service making it a single store for allpersistent data By offloading state management to a cen-tralized object store these services make it possible to runload-balanced instances across multiple physical machines toimprove performance and scalability

AWS Lambda compatibility is provided via a specialCSPOT handler binary which embeds a Python interpreterWhen an function invocation request is received NanoLambdaCloudEdge appends an invocation record to CSPOTrsquos objectstore This append creates an event which invokes this handlerin an isolated docker instance The handler reads the invocationrecord and checks for the presence of the Python code forthe handler If necessary the code is fetched from the objectstore Once ready the Python interpreter invokes the handlerfunction with the payload from the invocation record andwrites the result to a result log for return to the requester

To support handler execution on device NanoLambdaCloudEdge implements an IoTPy translator (cf Section III-C)which compiles and packages Python handlers for execu-tion on neighboring microcontroller devices This translationservice enables IoT devices to fetch handler code from thesame function repository used by the Lambda API Emulation

4

Service The service compiles handlers on-demand into acompact binary representation that can be sent to the devicefor execution Fetch requests check that the code size andmemory requirements (specified in the function configuration)fit within the resource constraints of the requesting device Ifrejected it is possible for the cloudedge device to execute thefunction on behalf of the IoT device

B NanoLambda On Device

NanoLambda On Device is runs Python handlers on non-Linux IoT devices As shown in Figure 1 (on right) the serviceleverages CSPOT WooF logs on-device for persistent storageand event invocation As for Linux systems FaaS handlers areinvoked in response to data appended to storage logs

As seen in the figure the typical model for NanoLambdaOn Device is that data is produced by threads monitoringsensors on the device and appending it to objects in the objectstore It is also possible for data to be remotely deliveredfor processing over CSPOTrsquos network API Each append tothe object store is processed by CSPOTrsquos event model whichruns a C-language handler function with the new data Similarto the NanoLambda CloudEdgersquos support for Python usinga special Linux handler binary with an embedded Pythoninterpreter we extend NanoLambda CloudEdge with Pythonsupport by registering a special C-language handler functionwhich interprets the handlerrsquos name to be a Python Lambdafunction deployed via the NanoLambda CloudEdge serviceWhen this function is invoked it triggers the handler withthe payload provided by the newly appended object in aninstance of the IoTPy VM By leveraging CSPOTrsquos existingfunctionality we are able to offer Python 3 Lambda supporton device

C IoTPy

To ease development and portability we choose Python asour FaaS programming language because its bytecode is easilyportable and its simple virtual machine implementation canbe easily ported to IoT devices Likewise we package FaaSfunctions using the package format defined for AWS Lambdafor compatibility purposes

To achieve a small code and memory footprint we optedto implement IoTPy our own extremely lightweight Pythoninterpreter built from the ground up with embedding in mindWe found existing solutions such as MicroPython [30] to beunsuitable because they left little additional space for featuressuch as our FaaS runtime and the drivers for our networkingand sensors On our most resource-constrained devices suchas the ESP8266 our binary must be able to fit in as little as512KB of flash memory

IoTPy is written to be as unintrusive as possible with afocus on allowing the Python interpreter to be treated like alibrary To keep IoTPy small we opted to omit the Pythonparserlexer on the device Instead these tasks are offloadedto the NanoLambda CloudEdge service This benefits us in anumber of ways

1) bytecode compilation as well as optimization steps canbe performed by a full Python 36 implementation onNanoLambda CloudEdge service

2) memory use is reduced since we are only shippingcompact bytecode representation to the device and

3) a simpler virtual machine implementation can be used ondevice ndash by delegating compilation to the NanoLambdaCloudEdge service our IoTPy is able to limit its func-tionality to implementing the core language features ofthe Python virtual machine (VM)

Together these capabilities mitigate the additional complexityof maintaining a simple Python bytecode VM To enablethese capabilities we make some tradeoffs with regard toprogram storage and architectural simplicity IoTPy provides aconvenient CC++ interface for extending Python with nativefunctions We currently provide implementations for only themost common Python libraries and emulation of only the AWSS3 and Lambda deployment services The libraries includecommon math functions JSON operations APIs for accessingdevice peripherals basic networking and interfaces for theAPIs provided by NanoLambda CloudEdge IoTPy cannothowever load standard Python packages

When comparing the size of the generated binary files forIoTPy and MicroPython we find that the code for the IoTPylibrary uses 290KB of flash storage whereas MicroPythonrequires 620KB of flash storage Thus the IoTPy imple-mentation leaves us with almost 300KB of additional storageavailable for programs drivers and the FaaS runtime

D Python VM Security and Isolation for FaaS Handlers

IoT devices are often so resource-constrained that theylack security features such as memory protection units toimplement separate address spaces for isolated execution ofuser code Additionally conventional FaaS isolation achievedby Linux features like container-based isolation is simply notpossible on devices like these The authors of [2] and [29] findvarious attacks many of which rely on out-of-range memoryaccess to exploit IoT devices and execute code or update theirfirmware with malicious payloads Yet it is essential that IoTdevelopers be able to reprogram and update their devices inthe field

By placing the majority of application logic in FaaS han-dlers we reduce potential attack surface area A secure Pythonvirtual machine implementation can ensure that the bytecoderunning within it cannot access memory out of range orotherwise escape the permissions allowed to the handler bythe built-in functions provided to it Python handlers can thenbe isolated from one another by running in separate instancesof the Python VM This isolation is similar to traditional Linuxcontainers

E Deploying and Running Functions

By leveraging NanoLambda CloudEdge and its compatibil-ity layer for Amazonrsquos AWS Lambda APIs we minimize theeffort to port functions to the IoTPy platform In many casesfunctions run directly on IoTPy without code changes

5

programmer deploys code toNanoLambda CloudEdge service

with aws cli

NanoLambdaCloudEdge

1 stores code in object store service2 compiles and caches compact

bytecode representation on-demand

when IoT devices find a handler is not cachedlocally they request bytecode representation

from NanoLambda service

Fig 2 Life cycle of a function deployment from the application developerrsquoscommand line to the NanoLambda CloudEdge service to the IoT devicerunning NanoLambda On Device

Function deployment is performed by developers using theaws-cli or the aws-sdk (using a NanoLambda CloudEdgedevice as the cloud target) meaning existing tools capable ofdeploying to AWS Lambda can also deploy to NanoLambdaCloudEdge with few changes Deploying a new version ofan application is done with the ldquoaws lambda create-functionrdquocommand which accepts the application code package as azip file as well as other configuration parameters includingthe function entry-point its memory limit and Python version(Python 36 is currently supported) The deployment processchecks code size and resource limits (as is done in AWSLambda) and also checks for use of unsupported AWS APIs inthe code (eg only AWS REST S3 API is currently supported)The validated function is stored in the S3 API EmulationService as described above

Function execution on device is performed by IoTPy Localinvocations are accelerated by an interpreter cache whichallows the IoTPy interpreter for a given handler to be reusedif it is still available from a previous invocation This isdone to avoid the relatively expensive process of reinitializingPython built-ins as well as functions and globals defined inthe bytecode

In Figure 2 we show the deployment and function downloadprocess mapped out for NanoLambda On Device When theinterpreter is still running (ie because it is already running afunction) execution involves simply setting up a new Pythonframe with the call stack for the function and invoking thefunction in the cached interpreter When no interpreter isrunning the NanoLambda service opens a TCP connectionto NanoLambda CloudEdge and serializes a request for thefunction using flatbuffers [18] a fast and memory efficientbinary protocol for network serialization When NanoLambdaCloudEdge receives this request the function is fetched fromour object store and compiled on the fly using the Linuximplementation of Python 36 and the Python dis package toextract generated bytecode

F Execution Offloading CapabilitiesOur NanoLambda On Device service provides direct han-

dler code compatibility with the NanoLambda CloudEdgeservice We achieve this by adopting the same AWS Lambda-compatible handler format

NanoLambda On Device supports local execution of han-dlers on microcontrollers like the ESP8266 and the CC3220SFNanoLambda CloudEdge leveraging CSPOTrsquos tiered-cloudapproach extends these execution options to include edgecloud nodes like the Raspberry Pi Zero as well as in-cloudsolutions like CSPOT running on AWS EC2 Lastly becauseboth services use a common function format which is code-compatible with AWS Lambda our handlers can be easilytransferred between the services or even executed on AWSLambda itself Selecting to develop application logic in com-partmentalized functions using a high-level language lends thedeveloper immediate portability benefits which grant her thepower to choose where code should be run to best utilizeresources or achieve desired performance

IV EVALUATION

In this section we evaluate NanoLambda on two exampleFaaS applications for sensor monitoring We provide a perfor-mance analysis of NanoLambda using micro-benchmarks toexpose a detailed analysis of its performance characteristicsunder its various configurations Additionally we show howNanoLambdarsquos portability can be leveraged to implement alatency-aware scheduler that migrates execution between thedevice and the edge cloud to improve performance across arange of problem sizes

A Automatic Light Control ApplicationWe show how NanoLambda might be used in an auto-

matic lighting control system ndash for example to automati-cally turn on walkway lighting at night based on ambientlight levels The hardware setup for this experiment is aphotoresistor connected to the analogue-to-digital converter(ADC) pin of an ESP8266 microcontroller This processoris extremely resource-constrained with 80KB of user dataRAM and 512KB of flash storage for code The only device-specific C code written for this experiment is a simple samplerthread which measures the voltage on the ADC pin of themicrocontroller and writes it out to an invocation log at a rateof 5 samples per second

In response to each log append (analogous to publishing toan MQTT queue) an IoTPy handler is invoked to analyze thenew data and send commands to the light system if necessaryThe implementation of the handler for this automatic lightcontrol experiment is shown in Figure 3

def new_light_lvl(payload ctx)if payload[0] gt 100

deviceset_pin_high(2)else

deviceset_pin_low(2)

Fig 3 Handler function for the light control application

6

In Table I we show invocation latencies for various config-urations of the NanoLambda runtime The ldquoNo Server Cacherdquoconfiguration represents the true ldquocold startrdquo cost of invokinga new AWS Lambda function for the first time This numberincludes both the time taken for the server to compile theLambda function and to deliver it to the device The LocalCache configuration shows us the typical amortized runtimeof the application where frequently invoked Lambda functionswill typically be already available on the device We cansee that the performance difference is substantial requiringalmost 32x as much time to invoke a function that is not incache This is largely due to the effects of both compilationtime and the cost of network latency Compilation and codetransfer each take roughly 100 milliseconds on the networktested We include ldquoNo Local Cacherdquo to show the startup costof a function shared by multiple devices In this scenario afunction typically has already been accessed and compiled bythe server but has not yet been delivered to a given deviceWe expect this to be representative of the startup cost for real-world deployments

We include a C implementation of the same handler functionas a baseline It is worth noting that this is on the order of20x faster than our configuration with local caching This is asubstantial overhead but it is representative of what we wouldexpect for the performance difference between an interpretedlanguage without JIT and a compiled language like C [32]Ultimately it is up to the application developer to determinewhen the flexibility and ease of deployment with Python andthe FaaS ecosystem outweigh its overhead

Configuration Latency in msNanoLambda Local Caching 67 msNanoLambda No Local Cache 1195 msNanoLambda No Server Cache 2204 msNanoLambda C Handler 03 ms

TABLE IAVERAGE LATENCY FOR 100 ITERATIONS OF THE HANDLER FUNCTION

B Predictive Maintenance Application

In this experiment we look at the detailed performance ofa NanoLambda application with more demanding processingrequirements compared to the previous example Predictivemaintenance techniques use sensors to detect part failures orother maintenance requirements

Our experiment considers motor maintenance by analyzingthe vibrations off of a motor as measured by an accelerometerIt is possible to detect if the motor or a connected part hasfailed by monitoring for changes in the distribution of themagnitude of vibrations picked up by the accelerometer Ourapplication detects these changes with a Python implementa-tion of the KolmogorovndashSmirnov (KS) test used to comparea reference distribution against real time measurements froman accelerometer The test provides the probability that theempirical distribution matches the reference distribution Apossible failure can be flagged if this probability drops belowa tuned threshold

list containing a reference distribution from accelerometerreference = []

def kstest(datalist1 datalist2) omitted see implementation from gistgithubcomdevries11405101

def new_accel_sample(payload ctx)global referencetransformed = []for record in payload

transformedappend(magnitude(record)))

prob = kstest(transformedreference)

return prob

Fig 4 The implementation of the handler providing the KS test results

In Figure 4 we show the implementation of our KS testhandler The handler is provided with a JSON payload con-taining a list of recent accelerometer data as tuples of (x y z)acceleration These records are normalized to a transformedarray of floating point magnitudes and then passed to anldquooff-the-shelfrdquo open-source implementation of the KS test byChristopher De Vries [27] This KS test implementation iscomputationally intensive and scales linearly with the inputproblem size

In Table II we show the performance of a real-worldinstallation The hardware configuration is a CC3220SF pro-cessor with 1MB of program flash and 256KB of RAM Thisprocessor is wired to an accelerometer monitored by a samplerthread (much like the photoresistor experiment) which collectsdata from the accelerometer 5 times per second and appends itto a data log Every 32 samples a handler is invoked to analyzerecent data The handler is always invoked on device andshows the performance of a more complex handler running onIoTPy The more complex function requires longer executiontime 147ms as well as a longer startup time of 436ms tocompile

We found this handler to be sufficiently computationallyintensive to be a good candidate for offloading to an edge clouddevice To this end we next present a scheduler for offloadingcomputationally-intensive tasks to remote FaaS providers

C Saving Power With Execution OffloadingIn Figure 5 we show the power used by the CC3220SF

microcontroller for KS problem sizes 20 and 80 over the

Configuration Latency (ms) Power Use (mJ) Memory Use KBNanoLambda Local Caching 2094 ms 2123 mJ 236KBNanoLambda No Local Cache 7290 ms 8500 mJ 217KB

TABLE IIAVERAGE TIMESPOWER CONSUMPTION OVER 100 ITERATIONS OF THE

HANDLER FUNCTION ON THE CC3220SF MICROCONTROLLER WITH A KSPROBLEM SIZE OF 32 IN EACH CONFIGURATION

7

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600Po

wer D

raw

in (m

W)

Power Use Over Local Request Lifecycle for Problem Size 20Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 20Remote Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w in

(mW

)

Power Use Over Local Request Lifecycle for Problem Size 80Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 80Remote Request

Fig 5 Comparison of power draw over request lifecycle for various KSproblem sizes using both remote and local strategies

life cycle of a request We observe here that with the ex-ception of some power spikes due to background activity onthe microcontroller the power used by the local request iscontinuous from the beginning of the invocation until the resultis available and so it grows with the size of the problem (plussome constant overhead from the FaaS runtime) With remoteexecution we see just over twice the peak power draw of thelocal execution strategy but we see a trough in the middle ofthe request life cycle where power use drops to near zeroThis is because the device is free to completely idle whilewaiting for results to return from the server Therefore wepay a large but short-lived power cost during transmissionbut when enough computation can be performed in the cloudwe can save power overall relative to local execution

D Predictive Maintenance Application with a Naive Offload-ing Scheduler

In this application we take advantage of our code compat-ibility between NanoLambda CloudEdge and NanoLambdaOn Device to extend our predictive maintenance applicationrunning on device with the capability to dynamically offloadcomputation to the edge

Our scheduler offloads computation when its algorithmestimates that better latency can be achieved by transferringexecution to the cloud We model execution latency by check-ing the runtime of the last execution on NanoLambda OnDevice and comparing it against the round trip latency of thelast invocation on NanoLambda CloudEdge The service withthe lowest total latency is selected to run the handler

However as conditions change execution-time measure-ments can become stale Much like the authors of [7] weperiodically retest each possible execution strategy to allow thescheduler to sense when the relative performance has changedeg as a result of evolving network conditions or changesin problem complexity In this experiment our scheduler isconfigured to retest the losing strategy 1 out of every 16invocations

To benchmark the scheduler we measure the average in-vocation latency for a range of synthetic problems from 5data points to 100 data points We examine problem sizes atmultiples of 5 with 1000 trials at each size

Figure 6 shows average invocation latency as a functionof problem size for the scheduler a remote-only invocationstrategy and a local-only invocation strategy Local invoca-tions provide very consistent runtime which scales linearlywith the problem size Remote invocation on the other handis highly subject to network latency and congestion as wecan see from occasional spikes The scheduled invocationsappear to offer the best of both worlds scaling linearly untilthe latency exceeds the average network latency at which pointthe strategy switches to remote invocation

Figure 7 shows average power per invocation as a functionof problem size for the scheduler the remote-only invocationstrategy and the local-only invocation strategy We see similartrends as the previous the latency graph ndash remote executionuses relatively constant power while local execution powerconsumption scales with execution time We see again thatthe scheduler offers the best of both strategies Interestinglyhowever even though the scheduler begins to switch to remoteinvocations at problem size 40 we do not observe powersavings from this until the problem size reaches 60

A more detailed breakdown of the choices made by thescheduler strategy are shown in Figure 8 The top of thischart shows the execution latency while the bottom shows thenumber of calculations run locally versus the number offloadedover the network Initially all calculations run on device forsmall problem sizes however once the problem size exceeds40 it becomes intermittently worthwhile to run over thenetwork when latency is low After the problem size exceeds60 almost all invocations run remote as the speed differencebetween on-device execution and remote execution outweighsthe latency cost of transferring data over the network Weobserve however that this naive scheduling algorithm leavessome room for improvement First with a fixed re-testinglatency we allocate more cycles to re-testing than may benecessary As we can see in this graph the scheduler isalways spending 601000 cycles re-testing the losing strategyWhen there is a large difference in performance this can be

8

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 6 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 7 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

a substantial overhead Second the scheduler has difficultyswitching to remote invocations when optimal as it is subjectto temporary network spikes

E An Improved Scheduler for Predictive Maintenance

Learning from the performance of our naive scheduler wenext construct a model adjusts to short-lived network spikesand that reduces the amount of time spent retesting To thisend we devise a new scoring function that incorporates anexponential term to adjust dynamically the re-testing intervalIn addition we use the median latency of recent requests as away of filtering outlier measurements

We define the new score function to be

score = median(H)minus (1 + r)d

Where H is the invocation latency history for the strategy(local or remote) being scored r is the decay rate which isset to r = 12 and d is the decay term ndash the number of

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)

Avg Invocation Latency vs Problem Size for Naive Scheduler

20 40 60 80 100ks problem size

20

40

60

80

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Naive Scheduler

remote invocationslocal invocations

Fig 8 Top average invocation latency as a function of KS problem size forthe offloading scheduler Bottom local invocation count vs remote invocationcount selected by the offloading scheduler for each problem size

invocations since this strategy was last selected We take themedian over a window of 3 samples which serves to filterout any one unusually fast or slow result allowing us to moreeasily recover from lost packets or other short-lived networkspikes

In Figure 9 we can see how this invocation strategy breaksdown in terms of request distribution When compared withthe naive schedulerrsquos strategy breakdown in Figure 8 weobserve that the improved scheduler begins to switch schedul-ing strategies slightly earlier than the naive scheduler It isinteresting to see that the improved scheduler with its noisefiltering transitions smoothly from local to remote invocationsas the problem size gradually increases When problem sizesare in the range from approximately 50 to 65 the schedulermixes local and remote invocations as a result of variations innetwork latency etc

With the improved scheduler the transition to remote exe-cution is much smoother and completes much earlier than theswitch-over seen in the naive strategy The naive strategy failsto fully commit to remote execution until a problem size of 75or 80 whereas the improved scheduler has mostly switchedover by the time the problem size reaches 65 Even when thelatency difference between remote execution is large as is thecase with problem sizes greater than 80 the naive schedulerappears to occasionally switch back to local execution This ismost likely a result of network timeouts which go unfiltered in

9

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

Service The service compiles handlers on-demand into acompact binary representation that can be sent to the devicefor execution Fetch requests check that the code size andmemory requirements (specified in the function configuration)fit within the resource constraints of the requesting device Ifrejected it is possible for the cloudedge device to execute thefunction on behalf of the IoT device

B NanoLambda On Device

NanoLambda On Device is runs Python handlers on non-Linux IoT devices As shown in Figure 1 (on right) the serviceleverages CSPOT WooF logs on-device for persistent storageand event invocation As for Linux systems FaaS handlers areinvoked in response to data appended to storage logs

As seen in the figure the typical model for NanoLambdaOn Device is that data is produced by threads monitoringsensors on the device and appending it to objects in the objectstore It is also possible for data to be remotely deliveredfor processing over CSPOTrsquos network API Each append tothe object store is processed by CSPOTrsquos event model whichruns a C-language handler function with the new data Similarto the NanoLambda CloudEdgersquos support for Python usinga special Linux handler binary with an embedded Pythoninterpreter we extend NanoLambda CloudEdge with Pythonsupport by registering a special C-language handler functionwhich interprets the handlerrsquos name to be a Python Lambdafunction deployed via the NanoLambda CloudEdge serviceWhen this function is invoked it triggers the handler withthe payload provided by the newly appended object in aninstance of the IoTPy VM By leveraging CSPOTrsquos existingfunctionality we are able to offer Python 3 Lambda supporton device

C IoTPy

To ease development and portability we choose Python asour FaaS programming language because its bytecode is easilyportable and its simple virtual machine implementation canbe easily ported to IoT devices Likewise we package FaaSfunctions using the package format defined for AWS Lambdafor compatibility purposes

To achieve a small code and memory footprint we optedto implement IoTPy our own extremely lightweight Pythoninterpreter built from the ground up with embedding in mindWe found existing solutions such as MicroPython [30] to beunsuitable because they left little additional space for featuressuch as our FaaS runtime and the drivers for our networkingand sensors On our most resource-constrained devices suchas the ESP8266 our binary must be able to fit in as little as512KB of flash memory

IoTPy is written to be as unintrusive as possible with afocus on allowing the Python interpreter to be treated like alibrary To keep IoTPy small we opted to omit the Pythonparserlexer on the device Instead these tasks are offloadedto the NanoLambda CloudEdge service This benefits us in anumber of ways

1) bytecode compilation as well as optimization steps canbe performed by a full Python 36 implementation onNanoLambda CloudEdge service

2) memory use is reduced since we are only shippingcompact bytecode representation to the device and

3) a simpler virtual machine implementation can be used ondevice ndash by delegating compilation to the NanoLambdaCloudEdge service our IoTPy is able to limit its func-tionality to implementing the core language features ofthe Python virtual machine (VM)

Together these capabilities mitigate the additional complexityof maintaining a simple Python bytecode VM To enablethese capabilities we make some tradeoffs with regard toprogram storage and architectural simplicity IoTPy provides aconvenient CC++ interface for extending Python with nativefunctions We currently provide implementations for only themost common Python libraries and emulation of only the AWSS3 and Lambda deployment services The libraries includecommon math functions JSON operations APIs for accessingdevice peripherals basic networking and interfaces for theAPIs provided by NanoLambda CloudEdge IoTPy cannothowever load standard Python packages

When comparing the size of the generated binary files forIoTPy and MicroPython we find that the code for the IoTPylibrary uses 290KB of flash storage whereas MicroPythonrequires 620KB of flash storage Thus the IoTPy imple-mentation leaves us with almost 300KB of additional storageavailable for programs drivers and the FaaS runtime

D Python VM Security and Isolation for FaaS Handlers

IoT devices are often so resource-constrained that theylack security features such as memory protection units toimplement separate address spaces for isolated execution ofuser code Additionally conventional FaaS isolation achievedby Linux features like container-based isolation is simply notpossible on devices like these The authors of [2] and [29] findvarious attacks many of which rely on out-of-range memoryaccess to exploit IoT devices and execute code or update theirfirmware with malicious payloads Yet it is essential that IoTdevelopers be able to reprogram and update their devices inthe field

By placing the majority of application logic in FaaS han-dlers we reduce potential attack surface area A secure Pythonvirtual machine implementation can ensure that the bytecoderunning within it cannot access memory out of range orotherwise escape the permissions allowed to the handler bythe built-in functions provided to it Python handlers can thenbe isolated from one another by running in separate instancesof the Python VM This isolation is similar to traditional Linuxcontainers

E Deploying and Running Functions

By leveraging NanoLambda CloudEdge and its compatibil-ity layer for Amazonrsquos AWS Lambda APIs we minimize theeffort to port functions to the IoTPy platform In many casesfunctions run directly on IoTPy without code changes

5

programmer deploys code toNanoLambda CloudEdge service

with aws cli

NanoLambdaCloudEdge

1 stores code in object store service2 compiles and caches compact

bytecode representation on-demand

when IoT devices find a handler is not cachedlocally they request bytecode representation

from NanoLambda service

Fig 2 Life cycle of a function deployment from the application developerrsquoscommand line to the NanoLambda CloudEdge service to the IoT devicerunning NanoLambda On Device

Function deployment is performed by developers using theaws-cli or the aws-sdk (using a NanoLambda CloudEdgedevice as the cloud target) meaning existing tools capable ofdeploying to AWS Lambda can also deploy to NanoLambdaCloudEdge with few changes Deploying a new version ofan application is done with the ldquoaws lambda create-functionrdquocommand which accepts the application code package as azip file as well as other configuration parameters includingthe function entry-point its memory limit and Python version(Python 36 is currently supported) The deployment processchecks code size and resource limits (as is done in AWSLambda) and also checks for use of unsupported AWS APIs inthe code (eg only AWS REST S3 API is currently supported)The validated function is stored in the S3 API EmulationService as described above

Function execution on device is performed by IoTPy Localinvocations are accelerated by an interpreter cache whichallows the IoTPy interpreter for a given handler to be reusedif it is still available from a previous invocation This isdone to avoid the relatively expensive process of reinitializingPython built-ins as well as functions and globals defined inthe bytecode

In Figure 2 we show the deployment and function downloadprocess mapped out for NanoLambda On Device When theinterpreter is still running (ie because it is already running afunction) execution involves simply setting up a new Pythonframe with the call stack for the function and invoking thefunction in the cached interpreter When no interpreter isrunning the NanoLambda service opens a TCP connectionto NanoLambda CloudEdge and serializes a request for thefunction using flatbuffers [18] a fast and memory efficientbinary protocol for network serialization When NanoLambdaCloudEdge receives this request the function is fetched fromour object store and compiled on the fly using the Linuximplementation of Python 36 and the Python dis package toextract generated bytecode

F Execution Offloading CapabilitiesOur NanoLambda On Device service provides direct han-

dler code compatibility with the NanoLambda CloudEdgeservice We achieve this by adopting the same AWS Lambda-compatible handler format

NanoLambda On Device supports local execution of han-dlers on microcontrollers like the ESP8266 and the CC3220SFNanoLambda CloudEdge leveraging CSPOTrsquos tiered-cloudapproach extends these execution options to include edgecloud nodes like the Raspberry Pi Zero as well as in-cloudsolutions like CSPOT running on AWS EC2 Lastly becauseboth services use a common function format which is code-compatible with AWS Lambda our handlers can be easilytransferred between the services or even executed on AWSLambda itself Selecting to develop application logic in com-partmentalized functions using a high-level language lends thedeveloper immediate portability benefits which grant her thepower to choose where code should be run to best utilizeresources or achieve desired performance

IV EVALUATION

In this section we evaluate NanoLambda on two exampleFaaS applications for sensor monitoring We provide a perfor-mance analysis of NanoLambda using micro-benchmarks toexpose a detailed analysis of its performance characteristicsunder its various configurations Additionally we show howNanoLambdarsquos portability can be leveraged to implement alatency-aware scheduler that migrates execution between thedevice and the edge cloud to improve performance across arange of problem sizes

A Automatic Light Control ApplicationWe show how NanoLambda might be used in an auto-

matic lighting control system ndash for example to automati-cally turn on walkway lighting at night based on ambientlight levels The hardware setup for this experiment is aphotoresistor connected to the analogue-to-digital converter(ADC) pin of an ESP8266 microcontroller This processoris extremely resource-constrained with 80KB of user dataRAM and 512KB of flash storage for code The only device-specific C code written for this experiment is a simple samplerthread which measures the voltage on the ADC pin of themicrocontroller and writes it out to an invocation log at a rateof 5 samples per second

In response to each log append (analogous to publishing toan MQTT queue) an IoTPy handler is invoked to analyze thenew data and send commands to the light system if necessaryThe implementation of the handler for this automatic lightcontrol experiment is shown in Figure 3

def new_light_lvl(payload ctx)if payload[0] gt 100

deviceset_pin_high(2)else

deviceset_pin_low(2)

Fig 3 Handler function for the light control application

6

In Table I we show invocation latencies for various config-urations of the NanoLambda runtime The ldquoNo Server Cacherdquoconfiguration represents the true ldquocold startrdquo cost of invokinga new AWS Lambda function for the first time This numberincludes both the time taken for the server to compile theLambda function and to deliver it to the device The LocalCache configuration shows us the typical amortized runtimeof the application where frequently invoked Lambda functionswill typically be already available on the device We cansee that the performance difference is substantial requiringalmost 32x as much time to invoke a function that is not incache This is largely due to the effects of both compilationtime and the cost of network latency Compilation and codetransfer each take roughly 100 milliseconds on the networktested We include ldquoNo Local Cacherdquo to show the startup costof a function shared by multiple devices In this scenario afunction typically has already been accessed and compiled bythe server but has not yet been delivered to a given deviceWe expect this to be representative of the startup cost for real-world deployments

We include a C implementation of the same handler functionas a baseline It is worth noting that this is on the order of20x faster than our configuration with local caching This is asubstantial overhead but it is representative of what we wouldexpect for the performance difference between an interpretedlanguage without JIT and a compiled language like C [32]Ultimately it is up to the application developer to determinewhen the flexibility and ease of deployment with Python andthe FaaS ecosystem outweigh its overhead

Configuration Latency in msNanoLambda Local Caching 67 msNanoLambda No Local Cache 1195 msNanoLambda No Server Cache 2204 msNanoLambda C Handler 03 ms

TABLE IAVERAGE LATENCY FOR 100 ITERATIONS OF THE HANDLER FUNCTION

B Predictive Maintenance Application

In this experiment we look at the detailed performance ofa NanoLambda application with more demanding processingrequirements compared to the previous example Predictivemaintenance techniques use sensors to detect part failures orother maintenance requirements

Our experiment considers motor maintenance by analyzingthe vibrations off of a motor as measured by an accelerometerIt is possible to detect if the motor or a connected part hasfailed by monitoring for changes in the distribution of themagnitude of vibrations picked up by the accelerometer Ourapplication detects these changes with a Python implementa-tion of the KolmogorovndashSmirnov (KS) test used to comparea reference distribution against real time measurements froman accelerometer The test provides the probability that theempirical distribution matches the reference distribution Apossible failure can be flagged if this probability drops belowa tuned threshold

list containing a reference distribution from accelerometerreference = []

def kstest(datalist1 datalist2) omitted see implementation from gistgithubcomdevries11405101

def new_accel_sample(payload ctx)global referencetransformed = []for record in payload

transformedappend(magnitude(record)))

prob = kstest(transformedreference)

return prob

Fig 4 The implementation of the handler providing the KS test results

In Figure 4 we show the implementation of our KS testhandler The handler is provided with a JSON payload con-taining a list of recent accelerometer data as tuples of (x y z)acceleration These records are normalized to a transformedarray of floating point magnitudes and then passed to anldquooff-the-shelfrdquo open-source implementation of the KS test byChristopher De Vries [27] This KS test implementation iscomputationally intensive and scales linearly with the inputproblem size

In Table II we show the performance of a real-worldinstallation The hardware configuration is a CC3220SF pro-cessor with 1MB of program flash and 256KB of RAM Thisprocessor is wired to an accelerometer monitored by a samplerthread (much like the photoresistor experiment) which collectsdata from the accelerometer 5 times per second and appends itto a data log Every 32 samples a handler is invoked to analyzerecent data The handler is always invoked on device andshows the performance of a more complex handler running onIoTPy The more complex function requires longer executiontime 147ms as well as a longer startup time of 436ms tocompile

We found this handler to be sufficiently computationallyintensive to be a good candidate for offloading to an edge clouddevice To this end we next present a scheduler for offloadingcomputationally-intensive tasks to remote FaaS providers

C Saving Power With Execution OffloadingIn Figure 5 we show the power used by the CC3220SF

microcontroller for KS problem sizes 20 and 80 over the

Configuration Latency (ms) Power Use (mJ) Memory Use KBNanoLambda Local Caching 2094 ms 2123 mJ 236KBNanoLambda No Local Cache 7290 ms 8500 mJ 217KB

TABLE IIAVERAGE TIMESPOWER CONSUMPTION OVER 100 ITERATIONS OF THE

HANDLER FUNCTION ON THE CC3220SF MICROCONTROLLER WITH A KSPROBLEM SIZE OF 32 IN EACH CONFIGURATION

7

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600Po

wer D

raw

in (m

W)

Power Use Over Local Request Lifecycle for Problem Size 20Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 20Remote Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w in

(mW

)

Power Use Over Local Request Lifecycle for Problem Size 80Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 80Remote Request

Fig 5 Comparison of power draw over request lifecycle for various KSproblem sizes using both remote and local strategies

life cycle of a request We observe here that with the ex-ception of some power spikes due to background activity onthe microcontroller the power used by the local request iscontinuous from the beginning of the invocation until the resultis available and so it grows with the size of the problem (plussome constant overhead from the FaaS runtime) With remoteexecution we see just over twice the peak power draw of thelocal execution strategy but we see a trough in the middle ofthe request life cycle where power use drops to near zeroThis is because the device is free to completely idle whilewaiting for results to return from the server Therefore wepay a large but short-lived power cost during transmissionbut when enough computation can be performed in the cloudwe can save power overall relative to local execution

D Predictive Maintenance Application with a Naive Offload-ing Scheduler

In this application we take advantage of our code compat-ibility between NanoLambda CloudEdge and NanoLambdaOn Device to extend our predictive maintenance applicationrunning on device with the capability to dynamically offloadcomputation to the edge

Our scheduler offloads computation when its algorithmestimates that better latency can be achieved by transferringexecution to the cloud We model execution latency by check-ing the runtime of the last execution on NanoLambda OnDevice and comparing it against the round trip latency of thelast invocation on NanoLambda CloudEdge The service withthe lowest total latency is selected to run the handler

However as conditions change execution-time measure-ments can become stale Much like the authors of [7] weperiodically retest each possible execution strategy to allow thescheduler to sense when the relative performance has changedeg as a result of evolving network conditions or changesin problem complexity In this experiment our scheduler isconfigured to retest the losing strategy 1 out of every 16invocations

To benchmark the scheduler we measure the average in-vocation latency for a range of synthetic problems from 5data points to 100 data points We examine problem sizes atmultiples of 5 with 1000 trials at each size

Figure 6 shows average invocation latency as a functionof problem size for the scheduler a remote-only invocationstrategy and a local-only invocation strategy Local invoca-tions provide very consistent runtime which scales linearlywith the problem size Remote invocation on the other handis highly subject to network latency and congestion as wecan see from occasional spikes The scheduled invocationsappear to offer the best of both worlds scaling linearly untilthe latency exceeds the average network latency at which pointthe strategy switches to remote invocation

Figure 7 shows average power per invocation as a functionof problem size for the scheduler the remote-only invocationstrategy and the local-only invocation strategy We see similartrends as the previous the latency graph ndash remote executionuses relatively constant power while local execution powerconsumption scales with execution time We see again thatthe scheduler offers the best of both strategies Interestinglyhowever even though the scheduler begins to switch to remoteinvocations at problem size 40 we do not observe powersavings from this until the problem size reaches 60

A more detailed breakdown of the choices made by thescheduler strategy are shown in Figure 8 The top of thischart shows the execution latency while the bottom shows thenumber of calculations run locally versus the number offloadedover the network Initially all calculations run on device forsmall problem sizes however once the problem size exceeds40 it becomes intermittently worthwhile to run over thenetwork when latency is low After the problem size exceeds60 almost all invocations run remote as the speed differencebetween on-device execution and remote execution outweighsthe latency cost of transferring data over the network Weobserve however that this naive scheduling algorithm leavessome room for improvement First with a fixed re-testinglatency we allocate more cycles to re-testing than may benecessary As we can see in this graph the scheduler isalways spending 601000 cycles re-testing the losing strategyWhen there is a large difference in performance this can be

8

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 6 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 7 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

a substantial overhead Second the scheduler has difficultyswitching to remote invocations when optimal as it is subjectto temporary network spikes

E An Improved Scheduler for Predictive Maintenance

Learning from the performance of our naive scheduler wenext construct a model adjusts to short-lived network spikesand that reduces the amount of time spent retesting To thisend we devise a new scoring function that incorporates anexponential term to adjust dynamically the re-testing intervalIn addition we use the median latency of recent requests as away of filtering outlier measurements

We define the new score function to be

score = median(H)minus (1 + r)d

Where H is the invocation latency history for the strategy(local or remote) being scored r is the decay rate which isset to r = 12 and d is the decay term ndash the number of

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)

Avg Invocation Latency vs Problem Size for Naive Scheduler

20 40 60 80 100ks problem size

20

40

60

80

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Naive Scheduler

remote invocationslocal invocations

Fig 8 Top average invocation latency as a function of KS problem size forthe offloading scheduler Bottom local invocation count vs remote invocationcount selected by the offloading scheduler for each problem size

invocations since this strategy was last selected We take themedian over a window of 3 samples which serves to filterout any one unusually fast or slow result allowing us to moreeasily recover from lost packets or other short-lived networkspikes

In Figure 9 we can see how this invocation strategy breaksdown in terms of request distribution When compared withthe naive schedulerrsquos strategy breakdown in Figure 8 weobserve that the improved scheduler begins to switch schedul-ing strategies slightly earlier than the naive scheduler It isinteresting to see that the improved scheduler with its noisefiltering transitions smoothly from local to remote invocationsas the problem size gradually increases When problem sizesare in the range from approximately 50 to 65 the schedulermixes local and remote invocations as a result of variations innetwork latency etc

With the improved scheduler the transition to remote exe-cution is much smoother and completes much earlier than theswitch-over seen in the naive strategy The naive strategy failsto fully commit to remote execution until a problem size of 75or 80 whereas the improved scheduler has mostly switchedover by the time the problem size reaches 65 Even when thelatency difference between remote execution is large as is thecase with problem sizes greater than 80 the naive schedulerappears to occasionally switch back to local execution This ismost likely a result of network timeouts which go unfiltered in

9

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

programmer deploys code toNanoLambda CloudEdge service

with aws cli

NanoLambdaCloudEdge

1 stores code in object store service2 compiles and caches compact

bytecode representation on-demand

when IoT devices find a handler is not cachedlocally they request bytecode representation

from NanoLambda service

Fig 2 Life cycle of a function deployment from the application developerrsquoscommand line to the NanoLambda CloudEdge service to the IoT devicerunning NanoLambda On Device

Function deployment is performed by developers using theaws-cli or the aws-sdk (using a NanoLambda CloudEdgedevice as the cloud target) meaning existing tools capable ofdeploying to AWS Lambda can also deploy to NanoLambdaCloudEdge with few changes Deploying a new version ofan application is done with the ldquoaws lambda create-functionrdquocommand which accepts the application code package as azip file as well as other configuration parameters includingthe function entry-point its memory limit and Python version(Python 36 is currently supported) The deployment processchecks code size and resource limits (as is done in AWSLambda) and also checks for use of unsupported AWS APIs inthe code (eg only AWS REST S3 API is currently supported)The validated function is stored in the S3 API EmulationService as described above

Function execution on device is performed by IoTPy Localinvocations are accelerated by an interpreter cache whichallows the IoTPy interpreter for a given handler to be reusedif it is still available from a previous invocation This isdone to avoid the relatively expensive process of reinitializingPython built-ins as well as functions and globals defined inthe bytecode

In Figure 2 we show the deployment and function downloadprocess mapped out for NanoLambda On Device When theinterpreter is still running (ie because it is already running afunction) execution involves simply setting up a new Pythonframe with the call stack for the function and invoking thefunction in the cached interpreter When no interpreter isrunning the NanoLambda service opens a TCP connectionto NanoLambda CloudEdge and serializes a request for thefunction using flatbuffers [18] a fast and memory efficientbinary protocol for network serialization When NanoLambdaCloudEdge receives this request the function is fetched fromour object store and compiled on the fly using the Linuximplementation of Python 36 and the Python dis package toextract generated bytecode

F Execution Offloading CapabilitiesOur NanoLambda On Device service provides direct han-

dler code compatibility with the NanoLambda CloudEdgeservice We achieve this by adopting the same AWS Lambda-compatible handler format

NanoLambda On Device supports local execution of han-dlers on microcontrollers like the ESP8266 and the CC3220SFNanoLambda CloudEdge leveraging CSPOTrsquos tiered-cloudapproach extends these execution options to include edgecloud nodes like the Raspberry Pi Zero as well as in-cloudsolutions like CSPOT running on AWS EC2 Lastly becauseboth services use a common function format which is code-compatible with AWS Lambda our handlers can be easilytransferred between the services or even executed on AWSLambda itself Selecting to develop application logic in com-partmentalized functions using a high-level language lends thedeveloper immediate portability benefits which grant her thepower to choose where code should be run to best utilizeresources or achieve desired performance

IV EVALUATION

In this section we evaluate NanoLambda on two exampleFaaS applications for sensor monitoring We provide a perfor-mance analysis of NanoLambda using micro-benchmarks toexpose a detailed analysis of its performance characteristicsunder its various configurations Additionally we show howNanoLambdarsquos portability can be leveraged to implement alatency-aware scheduler that migrates execution between thedevice and the edge cloud to improve performance across arange of problem sizes

A Automatic Light Control ApplicationWe show how NanoLambda might be used in an auto-

matic lighting control system ndash for example to automati-cally turn on walkway lighting at night based on ambientlight levels The hardware setup for this experiment is aphotoresistor connected to the analogue-to-digital converter(ADC) pin of an ESP8266 microcontroller This processoris extremely resource-constrained with 80KB of user dataRAM and 512KB of flash storage for code The only device-specific C code written for this experiment is a simple samplerthread which measures the voltage on the ADC pin of themicrocontroller and writes it out to an invocation log at a rateof 5 samples per second

In response to each log append (analogous to publishing toan MQTT queue) an IoTPy handler is invoked to analyze thenew data and send commands to the light system if necessaryThe implementation of the handler for this automatic lightcontrol experiment is shown in Figure 3

def new_light_lvl(payload ctx)if payload[0] gt 100

deviceset_pin_high(2)else

deviceset_pin_low(2)

Fig 3 Handler function for the light control application

6

In Table I we show invocation latencies for various config-urations of the NanoLambda runtime The ldquoNo Server Cacherdquoconfiguration represents the true ldquocold startrdquo cost of invokinga new AWS Lambda function for the first time This numberincludes both the time taken for the server to compile theLambda function and to deliver it to the device The LocalCache configuration shows us the typical amortized runtimeof the application where frequently invoked Lambda functionswill typically be already available on the device We cansee that the performance difference is substantial requiringalmost 32x as much time to invoke a function that is not incache This is largely due to the effects of both compilationtime and the cost of network latency Compilation and codetransfer each take roughly 100 milliseconds on the networktested We include ldquoNo Local Cacherdquo to show the startup costof a function shared by multiple devices In this scenario afunction typically has already been accessed and compiled bythe server but has not yet been delivered to a given deviceWe expect this to be representative of the startup cost for real-world deployments

We include a C implementation of the same handler functionas a baseline It is worth noting that this is on the order of20x faster than our configuration with local caching This is asubstantial overhead but it is representative of what we wouldexpect for the performance difference between an interpretedlanguage without JIT and a compiled language like C [32]Ultimately it is up to the application developer to determinewhen the flexibility and ease of deployment with Python andthe FaaS ecosystem outweigh its overhead

Configuration Latency in msNanoLambda Local Caching 67 msNanoLambda No Local Cache 1195 msNanoLambda No Server Cache 2204 msNanoLambda C Handler 03 ms

TABLE IAVERAGE LATENCY FOR 100 ITERATIONS OF THE HANDLER FUNCTION

B Predictive Maintenance Application

In this experiment we look at the detailed performance ofa NanoLambda application with more demanding processingrequirements compared to the previous example Predictivemaintenance techniques use sensors to detect part failures orother maintenance requirements

Our experiment considers motor maintenance by analyzingthe vibrations off of a motor as measured by an accelerometerIt is possible to detect if the motor or a connected part hasfailed by monitoring for changes in the distribution of themagnitude of vibrations picked up by the accelerometer Ourapplication detects these changes with a Python implementa-tion of the KolmogorovndashSmirnov (KS) test used to comparea reference distribution against real time measurements froman accelerometer The test provides the probability that theempirical distribution matches the reference distribution Apossible failure can be flagged if this probability drops belowa tuned threshold

list containing a reference distribution from accelerometerreference = []

def kstest(datalist1 datalist2) omitted see implementation from gistgithubcomdevries11405101

def new_accel_sample(payload ctx)global referencetransformed = []for record in payload

transformedappend(magnitude(record)))

prob = kstest(transformedreference)

return prob

Fig 4 The implementation of the handler providing the KS test results

In Figure 4 we show the implementation of our KS testhandler The handler is provided with a JSON payload con-taining a list of recent accelerometer data as tuples of (x y z)acceleration These records are normalized to a transformedarray of floating point magnitudes and then passed to anldquooff-the-shelfrdquo open-source implementation of the KS test byChristopher De Vries [27] This KS test implementation iscomputationally intensive and scales linearly with the inputproblem size

In Table II we show the performance of a real-worldinstallation The hardware configuration is a CC3220SF pro-cessor with 1MB of program flash and 256KB of RAM Thisprocessor is wired to an accelerometer monitored by a samplerthread (much like the photoresistor experiment) which collectsdata from the accelerometer 5 times per second and appends itto a data log Every 32 samples a handler is invoked to analyzerecent data The handler is always invoked on device andshows the performance of a more complex handler running onIoTPy The more complex function requires longer executiontime 147ms as well as a longer startup time of 436ms tocompile

We found this handler to be sufficiently computationallyintensive to be a good candidate for offloading to an edge clouddevice To this end we next present a scheduler for offloadingcomputationally-intensive tasks to remote FaaS providers

C Saving Power With Execution OffloadingIn Figure 5 we show the power used by the CC3220SF

microcontroller for KS problem sizes 20 and 80 over the

Configuration Latency (ms) Power Use (mJ) Memory Use KBNanoLambda Local Caching 2094 ms 2123 mJ 236KBNanoLambda No Local Cache 7290 ms 8500 mJ 217KB

TABLE IIAVERAGE TIMESPOWER CONSUMPTION OVER 100 ITERATIONS OF THE

HANDLER FUNCTION ON THE CC3220SF MICROCONTROLLER WITH A KSPROBLEM SIZE OF 32 IN EACH CONFIGURATION

7

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600Po

wer D

raw

in (m

W)

Power Use Over Local Request Lifecycle for Problem Size 20Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 20Remote Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w in

(mW

)

Power Use Over Local Request Lifecycle for Problem Size 80Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 80Remote Request

Fig 5 Comparison of power draw over request lifecycle for various KSproblem sizes using both remote and local strategies

life cycle of a request We observe here that with the ex-ception of some power spikes due to background activity onthe microcontroller the power used by the local request iscontinuous from the beginning of the invocation until the resultis available and so it grows with the size of the problem (plussome constant overhead from the FaaS runtime) With remoteexecution we see just over twice the peak power draw of thelocal execution strategy but we see a trough in the middle ofthe request life cycle where power use drops to near zeroThis is because the device is free to completely idle whilewaiting for results to return from the server Therefore wepay a large but short-lived power cost during transmissionbut when enough computation can be performed in the cloudwe can save power overall relative to local execution

D Predictive Maintenance Application with a Naive Offload-ing Scheduler

In this application we take advantage of our code compat-ibility between NanoLambda CloudEdge and NanoLambdaOn Device to extend our predictive maintenance applicationrunning on device with the capability to dynamically offloadcomputation to the edge

Our scheduler offloads computation when its algorithmestimates that better latency can be achieved by transferringexecution to the cloud We model execution latency by check-ing the runtime of the last execution on NanoLambda OnDevice and comparing it against the round trip latency of thelast invocation on NanoLambda CloudEdge The service withthe lowest total latency is selected to run the handler

However as conditions change execution-time measure-ments can become stale Much like the authors of [7] weperiodically retest each possible execution strategy to allow thescheduler to sense when the relative performance has changedeg as a result of evolving network conditions or changesin problem complexity In this experiment our scheduler isconfigured to retest the losing strategy 1 out of every 16invocations

To benchmark the scheduler we measure the average in-vocation latency for a range of synthetic problems from 5data points to 100 data points We examine problem sizes atmultiples of 5 with 1000 trials at each size

Figure 6 shows average invocation latency as a functionof problem size for the scheduler a remote-only invocationstrategy and a local-only invocation strategy Local invoca-tions provide very consistent runtime which scales linearlywith the problem size Remote invocation on the other handis highly subject to network latency and congestion as wecan see from occasional spikes The scheduled invocationsappear to offer the best of both worlds scaling linearly untilthe latency exceeds the average network latency at which pointthe strategy switches to remote invocation

Figure 7 shows average power per invocation as a functionof problem size for the scheduler the remote-only invocationstrategy and the local-only invocation strategy We see similartrends as the previous the latency graph ndash remote executionuses relatively constant power while local execution powerconsumption scales with execution time We see again thatthe scheduler offers the best of both strategies Interestinglyhowever even though the scheduler begins to switch to remoteinvocations at problem size 40 we do not observe powersavings from this until the problem size reaches 60

A more detailed breakdown of the choices made by thescheduler strategy are shown in Figure 8 The top of thischart shows the execution latency while the bottom shows thenumber of calculations run locally versus the number offloadedover the network Initially all calculations run on device forsmall problem sizes however once the problem size exceeds40 it becomes intermittently worthwhile to run over thenetwork when latency is low After the problem size exceeds60 almost all invocations run remote as the speed differencebetween on-device execution and remote execution outweighsthe latency cost of transferring data over the network Weobserve however that this naive scheduling algorithm leavessome room for improvement First with a fixed re-testinglatency we allocate more cycles to re-testing than may benecessary As we can see in this graph the scheduler isalways spending 601000 cycles re-testing the losing strategyWhen there is a large difference in performance this can be

8

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 6 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 7 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

a substantial overhead Second the scheduler has difficultyswitching to remote invocations when optimal as it is subjectto temporary network spikes

E An Improved Scheduler for Predictive Maintenance

Learning from the performance of our naive scheduler wenext construct a model adjusts to short-lived network spikesand that reduces the amount of time spent retesting To thisend we devise a new scoring function that incorporates anexponential term to adjust dynamically the re-testing intervalIn addition we use the median latency of recent requests as away of filtering outlier measurements

We define the new score function to be

score = median(H)minus (1 + r)d

Where H is the invocation latency history for the strategy(local or remote) being scored r is the decay rate which isset to r = 12 and d is the decay term ndash the number of

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)

Avg Invocation Latency vs Problem Size for Naive Scheduler

20 40 60 80 100ks problem size

20

40

60

80

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Naive Scheduler

remote invocationslocal invocations

Fig 8 Top average invocation latency as a function of KS problem size forthe offloading scheduler Bottom local invocation count vs remote invocationcount selected by the offloading scheduler for each problem size

invocations since this strategy was last selected We take themedian over a window of 3 samples which serves to filterout any one unusually fast or slow result allowing us to moreeasily recover from lost packets or other short-lived networkspikes

In Figure 9 we can see how this invocation strategy breaksdown in terms of request distribution When compared withthe naive schedulerrsquos strategy breakdown in Figure 8 weobserve that the improved scheduler begins to switch schedul-ing strategies slightly earlier than the naive scheduler It isinteresting to see that the improved scheduler with its noisefiltering transitions smoothly from local to remote invocationsas the problem size gradually increases When problem sizesare in the range from approximately 50 to 65 the schedulermixes local and remote invocations as a result of variations innetwork latency etc

With the improved scheduler the transition to remote exe-cution is much smoother and completes much earlier than theswitch-over seen in the naive strategy The naive strategy failsto fully commit to remote execution until a problem size of 75or 80 whereas the improved scheduler has mostly switchedover by the time the problem size reaches 65 Even when thelatency difference between remote execution is large as is thecase with problem sizes greater than 80 the naive schedulerappears to occasionally switch back to local execution This ismost likely a result of network timeouts which go unfiltered in

9

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

In Table I we show invocation latencies for various config-urations of the NanoLambda runtime The ldquoNo Server Cacherdquoconfiguration represents the true ldquocold startrdquo cost of invokinga new AWS Lambda function for the first time This numberincludes both the time taken for the server to compile theLambda function and to deliver it to the device The LocalCache configuration shows us the typical amortized runtimeof the application where frequently invoked Lambda functionswill typically be already available on the device We cansee that the performance difference is substantial requiringalmost 32x as much time to invoke a function that is not incache This is largely due to the effects of both compilationtime and the cost of network latency Compilation and codetransfer each take roughly 100 milliseconds on the networktested We include ldquoNo Local Cacherdquo to show the startup costof a function shared by multiple devices In this scenario afunction typically has already been accessed and compiled bythe server but has not yet been delivered to a given deviceWe expect this to be representative of the startup cost for real-world deployments

We include a C implementation of the same handler functionas a baseline It is worth noting that this is on the order of20x faster than our configuration with local caching This is asubstantial overhead but it is representative of what we wouldexpect for the performance difference between an interpretedlanguage without JIT and a compiled language like C [32]Ultimately it is up to the application developer to determinewhen the flexibility and ease of deployment with Python andthe FaaS ecosystem outweigh its overhead

Configuration Latency in msNanoLambda Local Caching 67 msNanoLambda No Local Cache 1195 msNanoLambda No Server Cache 2204 msNanoLambda C Handler 03 ms

TABLE IAVERAGE LATENCY FOR 100 ITERATIONS OF THE HANDLER FUNCTION

B Predictive Maintenance Application

In this experiment we look at the detailed performance ofa NanoLambda application with more demanding processingrequirements compared to the previous example Predictivemaintenance techniques use sensors to detect part failures orother maintenance requirements

Our experiment considers motor maintenance by analyzingthe vibrations off of a motor as measured by an accelerometerIt is possible to detect if the motor or a connected part hasfailed by monitoring for changes in the distribution of themagnitude of vibrations picked up by the accelerometer Ourapplication detects these changes with a Python implementa-tion of the KolmogorovndashSmirnov (KS) test used to comparea reference distribution against real time measurements froman accelerometer The test provides the probability that theempirical distribution matches the reference distribution Apossible failure can be flagged if this probability drops belowa tuned threshold

list containing a reference distribution from accelerometerreference = []

def kstest(datalist1 datalist2) omitted see implementation from gistgithubcomdevries11405101

def new_accel_sample(payload ctx)global referencetransformed = []for record in payload

transformedappend(magnitude(record)))

prob = kstest(transformedreference)

return prob

Fig 4 The implementation of the handler providing the KS test results

In Figure 4 we show the implementation of our KS testhandler The handler is provided with a JSON payload con-taining a list of recent accelerometer data as tuples of (x y z)acceleration These records are normalized to a transformedarray of floating point magnitudes and then passed to anldquooff-the-shelfrdquo open-source implementation of the KS test byChristopher De Vries [27] This KS test implementation iscomputationally intensive and scales linearly with the inputproblem size

In Table II we show the performance of a real-worldinstallation The hardware configuration is a CC3220SF pro-cessor with 1MB of program flash and 256KB of RAM Thisprocessor is wired to an accelerometer monitored by a samplerthread (much like the photoresistor experiment) which collectsdata from the accelerometer 5 times per second and appends itto a data log Every 32 samples a handler is invoked to analyzerecent data The handler is always invoked on device andshows the performance of a more complex handler running onIoTPy The more complex function requires longer executiontime 147ms as well as a longer startup time of 436ms tocompile

We found this handler to be sufficiently computationallyintensive to be a good candidate for offloading to an edge clouddevice To this end we next present a scheduler for offloadingcomputationally-intensive tasks to remote FaaS providers

C Saving Power With Execution OffloadingIn Figure 5 we show the power used by the CC3220SF

microcontroller for KS problem sizes 20 and 80 over the

Configuration Latency (ms) Power Use (mJ) Memory Use KBNanoLambda Local Caching 2094 ms 2123 mJ 236KBNanoLambda No Local Cache 7290 ms 8500 mJ 217KB

TABLE IIAVERAGE TIMESPOWER CONSUMPTION OVER 100 ITERATIONS OF THE

HANDLER FUNCTION ON THE CC3220SF MICROCONTROLLER WITH A KSPROBLEM SIZE OF 32 IN EACH CONFIGURATION

7

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600Po

wer D

raw

in (m

W)

Power Use Over Local Request Lifecycle for Problem Size 20Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 20Remote Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w in

(mW

)

Power Use Over Local Request Lifecycle for Problem Size 80Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 80Remote Request

Fig 5 Comparison of power draw over request lifecycle for various KSproblem sizes using both remote and local strategies

life cycle of a request We observe here that with the ex-ception of some power spikes due to background activity onthe microcontroller the power used by the local request iscontinuous from the beginning of the invocation until the resultis available and so it grows with the size of the problem (plussome constant overhead from the FaaS runtime) With remoteexecution we see just over twice the peak power draw of thelocal execution strategy but we see a trough in the middle ofthe request life cycle where power use drops to near zeroThis is because the device is free to completely idle whilewaiting for results to return from the server Therefore wepay a large but short-lived power cost during transmissionbut when enough computation can be performed in the cloudwe can save power overall relative to local execution

D Predictive Maintenance Application with a Naive Offload-ing Scheduler

In this application we take advantage of our code compat-ibility between NanoLambda CloudEdge and NanoLambdaOn Device to extend our predictive maintenance applicationrunning on device with the capability to dynamically offloadcomputation to the edge

Our scheduler offloads computation when its algorithmestimates that better latency can be achieved by transferringexecution to the cloud We model execution latency by check-ing the runtime of the last execution on NanoLambda OnDevice and comparing it against the round trip latency of thelast invocation on NanoLambda CloudEdge The service withthe lowest total latency is selected to run the handler

However as conditions change execution-time measure-ments can become stale Much like the authors of [7] weperiodically retest each possible execution strategy to allow thescheduler to sense when the relative performance has changedeg as a result of evolving network conditions or changesin problem complexity In this experiment our scheduler isconfigured to retest the losing strategy 1 out of every 16invocations

To benchmark the scheduler we measure the average in-vocation latency for a range of synthetic problems from 5data points to 100 data points We examine problem sizes atmultiples of 5 with 1000 trials at each size

Figure 6 shows average invocation latency as a functionof problem size for the scheduler a remote-only invocationstrategy and a local-only invocation strategy Local invoca-tions provide very consistent runtime which scales linearlywith the problem size Remote invocation on the other handis highly subject to network latency and congestion as wecan see from occasional spikes The scheduled invocationsappear to offer the best of both worlds scaling linearly untilthe latency exceeds the average network latency at which pointthe strategy switches to remote invocation

Figure 7 shows average power per invocation as a functionof problem size for the scheduler the remote-only invocationstrategy and the local-only invocation strategy We see similartrends as the previous the latency graph ndash remote executionuses relatively constant power while local execution powerconsumption scales with execution time We see again thatthe scheduler offers the best of both strategies Interestinglyhowever even though the scheduler begins to switch to remoteinvocations at problem size 40 we do not observe powersavings from this until the problem size reaches 60

A more detailed breakdown of the choices made by thescheduler strategy are shown in Figure 8 The top of thischart shows the execution latency while the bottom shows thenumber of calculations run locally versus the number offloadedover the network Initially all calculations run on device forsmall problem sizes however once the problem size exceeds40 it becomes intermittently worthwhile to run over thenetwork when latency is low After the problem size exceeds60 almost all invocations run remote as the speed differencebetween on-device execution and remote execution outweighsthe latency cost of transferring data over the network Weobserve however that this naive scheduling algorithm leavessome room for improvement First with a fixed re-testinglatency we allocate more cycles to re-testing than may benecessary As we can see in this graph the scheduler isalways spending 601000 cycles re-testing the losing strategyWhen there is a large difference in performance this can be

8

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 6 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 7 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

a substantial overhead Second the scheduler has difficultyswitching to remote invocations when optimal as it is subjectto temporary network spikes

E An Improved Scheduler for Predictive Maintenance

Learning from the performance of our naive scheduler wenext construct a model adjusts to short-lived network spikesand that reduces the amount of time spent retesting To thisend we devise a new scoring function that incorporates anexponential term to adjust dynamically the re-testing intervalIn addition we use the median latency of recent requests as away of filtering outlier measurements

We define the new score function to be

score = median(H)minus (1 + r)d

Where H is the invocation latency history for the strategy(local or remote) being scored r is the decay rate which isset to r = 12 and d is the decay term ndash the number of

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)

Avg Invocation Latency vs Problem Size for Naive Scheduler

20 40 60 80 100ks problem size

20

40

60

80

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Naive Scheduler

remote invocationslocal invocations

Fig 8 Top average invocation latency as a function of KS problem size forthe offloading scheduler Bottom local invocation count vs remote invocationcount selected by the offloading scheduler for each problem size

invocations since this strategy was last selected We take themedian over a window of 3 samples which serves to filterout any one unusually fast or slow result allowing us to moreeasily recover from lost packets or other short-lived networkspikes

In Figure 9 we can see how this invocation strategy breaksdown in terms of request distribution When compared withthe naive schedulerrsquos strategy breakdown in Figure 8 weobserve that the improved scheduler begins to switch schedul-ing strategies slightly earlier than the naive scheduler It isinteresting to see that the improved scheduler with its noisefiltering transitions smoothly from local to remote invocationsas the problem size gradually increases When problem sizesare in the range from approximately 50 to 65 the schedulermixes local and remote invocations as a result of variations innetwork latency etc

With the improved scheduler the transition to remote exe-cution is much smoother and completes much earlier than theswitch-over seen in the naive strategy The naive strategy failsto fully commit to remote execution until a problem size of 75or 80 whereas the improved scheduler has mostly switchedover by the time the problem size reaches 65 Even when thelatency difference between remote execution is large as is thecase with problem sizes greater than 80 the naive schedulerappears to occasionally switch back to local execution This ismost likely a result of network timeouts which go unfiltered in

9

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600Po

wer D

raw

in (m

W)

Power Use Over Local Request Lifecycle for Problem Size 20Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 20Remote Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w in

(mW

)

Power Use Over Local Request Lifecycle for Problem Size 80Local Request

0 100000 200000 300000 400000 500000 600000 700000 800000Time (us)

0

200

400

600

Powe

r Dra

w (m

W)

Power Use Over Remote Request Lifecycle for Problem Size 80Remote Request

Fig 5 Comparison of power draw over request lifecycle for various KSproblem sizes using both remote and local strategies

life cycle of a request We observe here that with the ex-ception of some power spikes due to background activity onthe microcontroller the power used by the local request iscontinuous from the beginning of the invocation until the resultis available and so it grows with the size of the problem (plussome constant overhead from the FaaS runtime) With remoteexecution we see just over twice the peak power draw of thelocal execution strategy but we see a trough in the middle ofthe request life cycle where power use drops to near zeroThis is because the device is free to completely idle whilewaiting for results to return from the server Therefore wepay a large but short-lived power cost during transmissionbut when enough computation can be performed in the cloudwe can save power overall relative to local execution

D Predictive Maintenance Application with a Naive Offload-ing Scheduler

In this application we take advantage of our code compat-ibility between NanoLambda CloudEdge and NanoLambdaOn Device to extend our predictive maintenance applicationrunning on device with the capability to dynamically offloadcomputation to the edge

Our scheduler offloads computation when its algorithmestimates that better latency can be achieved by transferringexecution to the cloud We model execution latency by check-ing the runtime of the last execution on NanoLambda OnDevice and comparing it against the round trip latency of thelast invocation on NanoLambda CloudEdge The service withthe lowest total latency is selected to run the handler

However as conditions change execution-time measure-ments can become stale Much like the authors of [7] weperiodically retest each possible execution strategy to allow thescheduler to sense when the relative performance has changedeg as a result of evolving network conditions or changesin problem complexity In this experiment our scheduler isconfigured to retest the losing strategy 1 out of every 16invocations

To benchmark the scheduler we measure the average in-vocation latency for a range of synthetic problems from 5data points to 100 data points We examine problem sizes atmultiples of 5 with 1000 trials at each size

Figure 6 shows average invocation latency as a functionof problem size for the scheduler a remote-only invocationstrategy and a local-only invocation strategy Local invoca-tions provide very consistent runtime which scales linearlywith the problem size Remote invocation on the other handis highly subject to network latency and congestion as wecan see from occasional spikes The scheduled invocationsappear to offer the best of both worlds scaling linearly untilthe latency exceeds the average network latency at which pointthe strategy switches to remote invocation

Figure 7 shows average power per invocation as a functionof problem size for the scheduler the remote-only invocationstrategy and the local-only invocation strategy We see similartrends as the previous the latency graph ndash remote executionuses relatively constant power while local execution powerconsumption scales with execution time We see again thatthe scheduler offers the best of both strategies Interestinglyhowever even though the scheduler begins to switch to remoteinvocations at problem size 40 we do not observe powersavings from this until the problem size reaches 60

A more detailed breakdown of the choices made by thescheduler strategy are shown in Figure 8 The top of thischart shows the execution latency while the bottom shows thenumber of calculations run locally versus the number offloadedover the network Initially all calculations run on device forsmall problem sizes however once the problem size exceeds40 it becomes intermittently worthwhile to run over thenetwork when latency is low After the problem size exceeds60 almost all invocations run remote as the speed differencebetween on-device execution and remote execution outweighsthe latency cost of transferring data over the network Weobserve however that this naive scheduling algorithm leavessome room for improvement First with a fixed re-testinglatency we allocate more cycles to re-testing than may benecessary As we can see in this graph the scheduler isalways spending 601000 cycles re-testing the losing strategyWhen there is a large difference in performance this can be

8

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 6 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 7 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

a substantial overhead Second the scheduler has difficultyswitching to remote invocations when optimal as it is subjectto temporary network spikes

E An Improved Scheduler for Predictive Maintenance

Learning from the performance of our naive scheduler wenext construct a model adjusts to short-lived network spikesand that reduces the amount of time spent retesting To thisend we devise a new scoring function that incorporates anexponential term to adjust dynamically the re-testing intervalIn addition we use the median latency of recent requests as away of filtering outlier measurements

We define the new score function to be

score = median(H)minus (1 + r)d

Where H is the invocation latency history for the strategy(local or remote) being scored r is the decay rate which isset to r = 12 and d is the decay term ndash the number of

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)

Avg Invocation Latency vs Problem Size for Naive Scheduler

20 40 60 80 100ks problem size

20

40

60

80

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Naive Scheduler

remote invocationslocal invocations

Fig 8 Top average invocation latency as a function of KS problem size forthe offloading scheduler Bottom local invocation count vs remote invocationcount selected by the offloading scheduler for each problem size

invocations since this strategy was last selected We take themedian over a window of 3 samples which serves to filterout any one unusually fast or slow result allowing us to moreeasily recover from lost packets or other short-lived networkspikes

In Figure 9 we can see how this invocation strategy breaksdown in terms of request distribution When compared withthe naive schedulerrsquos strategy breakdown in Figure 8 weobserve that the improved scheduler begins to switch schedul-ing strategies slightly earlier than the naive scheduler It isinteresting to see that the improved scheduler with its noisefiltering transitions smoothly from local to remote invocationsas the problem size gradually increases When problem sizesare in the range from approximately 50 to 65 the schedulermixes local and remote invocations as a result of variations innetwork latency etc

With the improved scheduler the transition to remote exe-cution is much smoother and completes much earlier than theswitch-over seen in the naive strategy The naive strategy failsto fully commit to remote execution until a problem size of 75or 80 whereas the improved scheduler has mostly switchedover by the time the problem size reaches 65 Even when thelatency difference between remote execution is large as is thecase with problem sizes greater than 80 the naive schedulerappears to occasionally switch back to local execution This ismost likely a result of network timeouts which go unfiltered in

9

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 6 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerNaive Scheduler

Fig 7 Comparison of average invocation latency for local invocation strategyremote invocation strategy and offloading scheduler invocation strategy

a substantial overhead Second the scheduler has difficultyswitching to remote invocations when optimal as it is subjectto temporary network spikes

E An Improved Scheduler for Predictive Maintenance

Learning from the performance of our naive scheduler wenext construct a model adjusts to short-lived network spikesand that reduces the amount of time spent retesting To thisend we devise a new scoring function that incorporates anexponential term to adjust dynamically the re-testing intervalIn addition we use the median latency of recent requests as away of filtering outlier measurements

We define the new score function to be

score = median(H)minus (1 + r)d

Where H is the invocation latency history for the strategy(local or remote) being scored r is the decay rate which isset to r = 12 and d is the decay term ndash the number of

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)

Avg Invocation Latency vs Problem Size for Naive Scheduler

20 40 60 80 100ks problem size

20

40

60

80

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Naive Scheduler

remote invocationslocal invocations

Fig 8 Top average invocation latency as a function of KS problem size forthe offloading scheduler Bottom local invocation count vs remote invocationcount selected by the offloading scheduler for each problem size

invocations since this strategy was last selected We take themedian over a window of 3 samples which serves to filterout any one unusually fast or slow result allowing us to moreeasily recover from lost packets or other short-lived networkspikes

In Figure 9 we can see how this invocation strategy breaksdown in terms of request distribution When compared withthe naive schedulerrsquos strategy breakdown in Figure 8 weobserve that the improved scheduler begins to switch schedul-ing strategies slightly earlier than the naive scheduler It isinteresting to see that the improved scheduler with its noisefiltering transitions smoothly from local to remote invocationsas the problem size gradually increases When problem sizesare in the range from approximately 50 to 65 the schedulermixes local and remote invocations as a result of variations innetwork latency etc

With the improved scheduler the transition to remote exe-cution is much smoother and completes much earlier than theswitch-over seen in the naive strategy The naive strategy failsto fully commit to remote execution until a problem size of 75or 80 whereas the improved scheduler has mostly switchedover by the time the problem size reaches 65 Even when thelatency difference between remote execution is large as is thecase with problem sizes greater than 80 the naive schedulerappears to occasionally switch back to local execution This ismost likely a result of network timeouts which go unfiltered in

9

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

20 40 60 80 100ks problem size

100

150

200

250

300

350

400

proc

essin

g tim

e (m

s)Avg Invocation Latency vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 9 Top average invocation latency as a function of KS problem size forthe improved scheduler Bottom local invocation count vs remote invocationcount selected by the scheduler for each problem size

the naive model causing an unnecessary switch in executionmode compounding the performance penalty of the timeout

In Figure 10 we show how the improved invocation strategybreaks down in terms of power consumption We can see thatfor the most part optimizing for latency also optimizes powerconsumption However this rule appears to not always holdtrue It is also interesting to see that around problem size 50-60 where the scheduler begins to select remote executionpower consumption increases non-linearly before reachingresting power levels consistent with what we would expectfrom the results in Figure 7 for remote execution power drawThis non-linearity could likely be smoothed out with a modelthat has a deeper understanding of the relationship betweenlatency and power draw for the two execution modes (localand remote)

In Figure 11 we show how this scheduling strategy com-pares to the strategies we have discussed so far We considernaive remote and local scheduling

In the local phase of execution (problem size less than 60)the advanced scheduler performs with the least overhead rela-tive to local-only execution though the difference is marginalThis is likely due to a slightly lower rate of re-testing withthe exponential back off which is supported by the the gapnarrowing as the problem size (and thus the difference inlatency) decreases and so the retesting frequency increases forthe advanced scheduler During the remote phase of execution

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

powe

r (uJ

)

Avg Invocation Power vs Problem Size for Improved Scheduler

20 40 60 80 100ks problem size

0

20

40

60

80

100

perc

enta

ge o

f req

uest

s

Local vs Remote Invocation Count for Improved Scheduler

remote invocationslocal invocations

Fig 10 Top average invocation power draw as a function of KS problemsize for the improved scheduler Bottom local invocation count vs remoteinvocation count selected by the scheduler for each problem size

20 40 60 80 100ks problem size

100

200

300

400

500

600

Late

ncy

(ms)

Average Time (MS) Per Invocation vs Problem SizeLocal Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 11 In this chart is a latency comparison for all four schedulers (localremote naive and adv)

the advanced schedulerrsquos latency is generally less volatile andslightly lower than the naive scheduler Both schedulers arehowever more consistent than the remote-only scheduler

Figure 12 shows the average power use for each schedulingstrategy In this graph the difference between the advancedscheduler and the naive scheduler is most pronounced duringthe local phase of execution ndash this is likely due to thehigher power penalty for activating the network Around a KS

10

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

20 40 60 80 100ks problem size

10000

20000

30000

40000

50000

60000

70000

Powe

r Micr

ojou

les

Average Power (uJ) Per Invocation vs Problem Size

Local Only SchedulerRemote Only SchedulerAdv SchedulerNaive Scheduler

Fig 12 In this chart is power comparison for all four schedulers discussedin this paper (local remote naive and adv)

problem size of 50 we see a pronounced spike in power drawfor both strategies as was discussed in Figure 10 Interestinglyduring this phase the advanced scheduler appears to drawsubstantially more power ndash although it appears to minimizelatency in Figure 11

These graphs show that leveraging NanoLambdarsquos portabil-ity to schedule execution of Python handler functions enablesoptimization of latency and power use to achieve betterperformance than either remote-only or local-only executionAdditionally these performance gains can be achieved withminimal overhead and no change to the handler implementa-tion

V LIMITATIONS

Our results show that NanoLambda is capable of runningPython AWS Lambda functions with demanding processingloads on-device as well as intelligently scheduling the ex-ecution of these function between the device and the edgeor cloud Despite providing a significant step toward raisingthe abstraction for and simplifying the programming of multi-scale IoT devices end-to-end multiple limitations remain thatwe hope to address in future work

Currently NanoLambda provides AWS Lambda compati-bility through our AWS Lambda compatible code packageformat and NanoLambda CloudEdge service which emulatesAWSrsquos APIs for AWS Lambda and AWS S3 This featuresimplifies code deployment and provides functions (runningin the cloud at the edge or on a sensor) with persistentobject storage capabilities Handlers running on NanoLambdaCloudEdge can directly access AWS Lambda and S3 orour emulated services running on top of CSPOT Unfortu-nately however functions running on-device are limited tousing wrapped interfaces that communicate only with ourNanoLambda CloudEdge emulated services Our goal is toextend NanoLambda CloudEdge with support for relayingrequests directly to AWS for services that lack emulation

Another limitation of NanoLambda is its support for Pythonpackages At the moment IoTPyrsquos approach to security lever-aging high-level Python VMrsquos for isolation means that onlyPython bytecode can be delivered from the cloud for executionon device As such functions that depend on Python 3 Cmodules must reimplement the functionality of those librariesin pure Python or extend IoTPy with trusted C code run-ning without memory isolation One area of future work forpossibly broadening the Python library base for NanoLambdamay be cross compiling Python modules to web assembly forisolated execution on-device

Finally NanoLambdarsquos execution scheduler is simple anddoes not account for device capability energy use battery lifeor other factors in its function placement decisions Similarlyon the security front NanoLambda does not authenticateedge devices or provide authorization or access control whentranslating or executing functions Therefore any device orclient may acquire every available function in the serverUnfortunately the mainstream access control mechanisms thatemploy public-key cryptography primitives are too resourceintensive for most resource-constrained devices To bring end-to-end security for functions deployed on device we planto leverage a promising new capability-based access controlmechanism for IoT that uses HMACs [14]

VI CONCLUSIONS

In this paper we introduce NanoLambda a new Pythonruntime system and tool chain designed for the least-capabledevices in IoT deployments The use of Python makes functionexecution on heterogeneous devices portable and raises thelevel of abstraction to facilitate programmer productivity Tobuild this functionality we adopt and mirror existing APIsas much as possible and thereby simplify the process ofporting applications to it To this end we present NanoLambdato leverage an S3-compatible REST API for its code andconfiguration storage as well as an AWS Lambda-compatiblefunction package format thus enabling many existing FaaSapplications to run on NanoLambda with little modificationWe detail the architecture of these services as well as ourapproach to on-device Python execution with IoTPy

IoTPy is a new Python interpreter designed for micro-controllers which defines its own bytecode language andcode packaging format NanoLambda offloads translation ofIoTPy binaries to a NanoLambda CloudEdge service IoTPyimplements FaaS- and device-aware optimizations that effi-ciently implement CSPOT capabilities while staying withinthe memory limitations of highly resource constrained IoTdevices

We evaluate NanoLambda with a range of example applica-tions for realistic use cases on the basis of invocation latency(important to control loop applications) as well as power drawAs part of our evaluation we present an approach to schedulededge cloud execution offloading as a way of winning backperformance by leveraging our platformrsquos ability to run thesame code on-device and at the edge We find that scheduled

11

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12

execution allows our platform to outperform either remote-only execution or on-device-only execution across a rangeof problem sizes by rescheduling execution in response to alatency model for each execution strategy

VII ACKNOWLEDGMENTS

This work has been supported in part by NSF (CNS-1703560 CCF-1539586 ACI-1541215) ONR NEEC(N00174-16-C-0020) and the AWS Cloud Credits forResearch program We thank the reviewers and our shepherdsfor their valuable feedback and suggestions

REFERENCES

[1] 2018 serverless community survey huge growth inserverless usage httpswwwserverlesscomblog2018-serverless-community-survey-huge-growth-usage [Onlineaccessed on 24-June-2020]

[2] T Alladi V Chamola B Sikdar and K R Choo Consumer iotSecurity vulnerability case studies and solutions IEEE ConsumerElectronics Magazine 9(2)17ndash25 2020

[3] Amazon DynamoDB httpsawsamazoncomdynamodb [Onlineaccessed 15-Nov-2016]

[4] Amazon s3 rdquowwwawsamazoncoms3rdquo [Online accessed on 24-June-2020]

[5] Amazon s3 rest api rdquohttpsdocsawsamazoncomAmazonS3latestAPIType API Referencehtmlrdquo [Online accessed on 24-June-2020]

[6] Apache OpenWhisk httpsopenwhiskapacheorg [Online accessed24-Jun-2020]

[7] A Aske and X Zhao Supporting multi-provider serverless computingon the edge In International Conference on Parallel Processing 2018

[8] AWS IoT Core rdquohttpsawsamazoncomiot-corerdquo [Online accessed12-Sep-2017]

[9] AWS Lambda httpsawsamazoncomlambda [Online accessed 24-Jun-2020]

[10] AWS Lambda IoT Reference Architecture httpdocsawsamazoncomlambdalatestdglambda-introductionhtml [Online accessed 12-Sep-2017]

[11] Azure Functions httpsazuremicrosoftcomen-usservicesfunctions[Online accessed 24-Jun-2020]

[12] Azure IoT Edge httpsazuremicrosoftcomen-usservicesiot-edge[Online accessed 22-Aug-2018]

[13] Azure IoT Hub httpsazuremicrosoftcomen-usservicesiot-hub[Online accessed 22-Aug-2018]

[14] F Bakir R Wolski C Krintz and G Sankar Ramachandran Devices-as-services Rethinking scalable service architectures for the internetof things In USENIX Workshop on Hot Topics in Edge Computing(HotEdge 19) July 2019

[15] A Banks and R Gupta Mqtt v311 protocol specification 2014[16] Nicole Berdy How to use Azure Functions with IoT Hub mes-

sage routing 2017 rdquohttpsazuremicrosoftcomen-usbloghow-to-use-azure-functions-with-iot-hub-message-routingrdquo

[17] Creating a rule with a aws lambda action httpsdocsawsamazoncomiotlatestdeveloperguideiot-lambda-rulehtml [Online accessed on 24-June-2020]

[18] Flatbuffers httpsgooglegithubioflatbuffers [Online accessed on 24-June-2020]

[19] D Floyer The Vital Role of Edge Computing inthe Internet ofThings rdquohttpwikiboncomthe-vital-role-of-edge-computing-in-the-internet-of-thingsrdquo [Online accessed 22-Aug-2016]

[20] Fog Data Services - Cisco httpwwwciscocomcenusproductscloud-systems-managementfog-data-servicesindexhtml [Online ac-cessed 22-Aug-2016]

[21] Function as a Service httpsenwikipediaorgwikiFunction as aService [Online accessed 12-Sep-2017]

[22] Google Cloud Functions httpscloudgooglecomfunctionsdocs [On-line accessed 24-Jun-2020]

[23] Google IoT Core httpscloudgooglecomiot-core [Online accessed12-Sep-2019]

[24] GreenGrass and IoT Core - Amazon Web Services httpsawsamazoncomiot-coregreengrass [Online accessed 2-Mar-2019]

[25] Internet of Things - Amazon Web Services httpsawsamazoncomiot[Online accessed 22-Aug-2016]

[26] E Jonas J Schleier-Smith V Sreekanti C Tsai A Khandelwal Q PuV Shankar J Carreira K Krauth N Yadwadkar J Gonzalez R PopaI Stoica and D Patterson Cloud Programming Simplified A BerkeleyView on Serverless Computing Feb 2019

[27] kstestpy httpsgistgithubcomdevries11405101 [Online accessed on24-June-2020]

[28] W-T Lin F Bakir C Krintz R Wolski and M Mock Data repairfor Distributed Event-based IoT Applications In ACM InternationalConference On Distributed and Event-Based Systems 2019

[29] Z Ling J Luo Y Xu C Gao K Wu and X Fu Securityvulnerabilities of internet of things A case study of the smart plugsystem IEEE Internet of Things Journal 4(6)1899ndash1909 2017

[30] Micropython httpsmicroPythonorg [Online accessed on 24-June-2020]

[31] S Nastic T Rausch O Scekic S Dustdar M Gusev B KoteskaM Kostoska B Jakimovski S Ristov and R Prodan A serverlessreal-time data analytics platform for edge computing IEEE InternetComputing 2164ndash71 01 2017

[32] Python 3 versus C gcc fastest programs httpsbenchmarksgame-teampagesdebiannetbenchmarksgamefastestPython3-gcchtml [Onlineaccessed 24-June-2020]

[33] M Satyanarayanan P Bahl R Caceres and N Davies The Case forVM-based Cloudlets in Mobile Computing IEEE Pervasive Computing8(4) 2009

[34] Serverless Platform [Online accessed 10-Feb-2019] wwwserverlesscom

[35] T Verbelen P Simoens F De Turck and B Dhoedt Cloudlets bringingthe cloud to the mobile user In ACM Workshop on Mobile CloudComputing and Services ACM 2012

[36] R Wolski C Krintz F Bakir G George and W-T Lin CSPOTPortable Multi-Scale Functions-as-a-Service for IoT In ACMIEEESymposium on Edge Computing 2019

12