Home Network Semantic Modeling and Reasoning - A Case Study

Topi Pulkkinen, Mikko Sallinen Networked Intelligence

VTT Technical Research Centre of Finland

Oulu, Finland topi.pulkkinen@vtt.fi

Jiyeon Son, Jun-Hee Park Green Computing Department

Electronics and Telecommunications Research Institute (ETRI)

Daejeon, Republic of Korea jyson@etri.re.kr

Yann-Hang Lee School of Computing, Informatics, and Decision Systems Engineering

Arizona State University (ASU) Tempe, Arizona, USA

yhlee@asu.edu

Abstract— To use smart home services efficiently, semantic modeling of different layers of home network resources as well as environment data and user's personal data is required. This suggests that a smart home system must be able to support data fusion and derive the context where services are operating in order to classify the new information provided by network agents correctly. Following the recent home network standardization efforts, the paper presents a feasible approach of smart home services and acts as a proof-of-concept for a dynamic home network management and diagnostics system.

Keywords-component; home network, service agent, semantic reasoning, home network standards

I. INTRODUCTION Home networks, emerging smart appliances and ever

increasing computing power of mobile devices and sensors have created a smart home ecosystem that has huge business potential [1]. A smart home aggregates various information sources intelligently to optimize and enhance available services according to user’s preference. The core function in this type of operation is to understand the context of different situations so the environment can be fully comprehended [2].

One method to define the context is to utilize semantic modeling and reasoning to map the relationships between the information sources and to select the suitable actions for the services. Because a great portion of the information relates to the life patterns of the user, fusion of time series sensor data is mandatory [3]. To understand how powerful context data can be, we can envision a smart washing machine which may be controlled to minimize the cost of electricity usage. Add-on services can consider user’s preferences, e.g. when the clothes should be dry, or safety issues, for example if the user doesn’t want to turn on the machine when no one is at home. These services can be realized once the machine can interact with the environment via a semantic modeling of user’s priorities, day and time, usage policies, and various dynamic factors including neighbors’ living patterns and optimal electricity usage in

apartment complex. Additionally, a semantic model of user and environment information enables us to create totally new type of services by integrating several services of individual appliances together. This aspect is critical when we wish to optimize home services beyond the limited functions of appliance. For instance, we may have two thermometers in different rooms monitoring home heating and cooling process. If the energy consumption from the heating and cooling device doesn’t match with the pattern of temperature measurements, there are some possible scenarios such as “a window is open in one of the rooms” or “the heating and cooling device is broken”. These abnormal cases can be detected by data clustering and reasoning after which intelligent actions can be taken.

The rest of the paper is organized as follows. Section 2 introduces the standardization goals and efforts done within ISO/IEC for home network interoperability. Section 3 presents the research work for context information modeling that is relevant for our work and describes the new model based on the standard proposal. Section 4 contains the proof-of-concept case, where a person’s heater is controlled based on the context information fused from many information sources. Section 4 also depicts the general software architecture that was implemented for the test-case and the results of the functional tests. Finally in Section 5 we present the conclusions of our results.

II. STANDARDIZATION ACTIVITIES OF HOME NETWORK RESOURCE MANAGEMENT

Authors are currently working on one of the home network standardization issues, which is HNRM (Home Network Resource Management). HNRM is being developed in ISO/IEC JTC1/SC25 which is responsible for HES (Home Electronic System) standards. The HES standard collection focuses on the interoperability of home network communication protocols and home network terminology, but also on system and application integration e.g. safety, security,

338

system management and building automation. Given the existing communication protocols, including IGRIS, Echonet, LonWorks, KNX, UPnP, etc., the interoperability in logical and semantic level can be seen as an enabler technology for different players, e.g. building constructors, device manufacturers, and service providers, to develop personalized and cost-effective home service infrastructure for the end-users.

Specifically HNRM tries to solve configuration problems between network entities as well as remote diagnostic and management problems [3]. It considers a conceptual resource model of a home network as illustrated in Fig. 1, which can be adapted into a semantic model discussed in introduction. Here the service-, device-, network- and floor plan information layers are coupled to provide simple management architecture for home environment and resources [4]. The example in the figure shows an Entertainment Management System (EMS) connected with devices that have relation on the network layer links. Two of the devices are located in the Living room.

Fig. 1 Conceptual smart home resource model for defining relationships

between the home network entities.

A. Home Resource Model The HNRM standard proposal specifies the Home

Resource Model as an abstract formal representation of objects in home that include their properties, relationships and the operations which can be performed on the objects [4]. The home resource model is divided into three hierarchy levels that includes resource object, resource class and resource domain. The hierarchies are illustrated in Fig. 3.

Resource object represents a managed object in home network and it has a one-to-one correspondence with a real-world component. Depending on the resource object domain, there is a variable amount of descriptive data associated with each resource object. For example an air-conditioner is a resource object of device domain and appliance class.

Resource class categorizes resource objects based on their functionality. This classification is useful for example when setting policies concerning a group of devices or managing services based on usage patterns.

Resource domain categorizes the resource objects by their resource type. Domain is important for remote fault

management because resource objects may be related to other objects of the same domain. This helps to detect problems such as power outage, network fault or software crash, which are normally difficult to analyze remotely.

The data structure of a resource object is presented in Fig. 2. A resource object is constructed of two parts: common information and domain specific information. The common information defines the object’s identifier, type and name. The domain specific object information contains the resource’s domain specific data.

Fig. 2 Resource object structure.

In TABLE I and TABLE II resource object information is listed in detail. The information structure in TABLE I is identical for all resource objects.

TABLE I. COMMON INFORMATION OF A RESOURCE OBJECT resource_id Constructs of a domain identifier and an object

identifier. Domain ID is the identifier of the domain the resource object belongs to. Object ID is the unique identifier for the object in the domain.

resource_type Resource type is combination of the class and the sub-class of a resource as shown in Fig. 3. It is hexadecimal number.

resource_name The resource object name. Character string.

Because the domain specific information varies, TABLE II shows the structure of a device domain resource object. The other domain specific resource objects that are specified in home resource model are: physical space domain resource object, network domain resource object and service domain resource object.

TABLE II. DOMAIN SPECIFIC INFORMATION OF A DEVICE DOMAIN OBJECT device_id Device identifier. Alpha-numeric string

device_name The name of device. Alpha-numeric string

physical address The physical address of the device

status The current status of device

version Device version

manufacturer Device manufacturer name

device_desription Device description with user-defined format

function_list The list of functions device supports

object_info Intra-device-domain relation, the list of intra-device-relation relation objects and related parameters

This work is supported by the IT R&D program MKE/KIAT. [2010-TD-300404-001, Home Information Remote Aggregation and Context Inference Prediction Technology Development] 339

Fig. 3 Hierarchy of the resource objects in home resource model.

III. MODELING OF THE CONTEXT INFORMATION The standardization work presented in Section 2 provides the

basis to interoperate various home network technologies for integrated services. It successfully introduces domain specific resource model which can be utilized for monitoring intra-domain relationships, such as to detect a network problem with remote fault-diagnosis [5]. However, it does not yet support techniques that a service integrator could use while creating intelligent home applications that handle multi-domain relationships effectively. Also, the standard work does not include models for context representation and awareness, e.g. user life patterns and preferences. Thus, to have a broader information model for smart home applications, the specified home resource model has to be extended.

A. Related work Here we describe the related research on context information

modeling. The description will focus only on the existing pervasive computing system information modeling, which are closely related to smart home environments. We also leave out application specific models because interoperability and reusability are key issues in home network environments. In addition, it is reasonable to suggest that there should be a centralized context information management component (for storing personal preferences, history data etc.) instead of a fully distributed system. Fully distributed systems could be better suited for public smart spaces where many services rely on the processing capacity of the smartphone.

CONON is one of the few existing context information models that are originally designed for smart home domain [6]. Its idea is simply to represent home conceptual entities with Ontology Web Language (OWL) format. The ontology constructs of entities that are either physical or conceptual objects (e.g. “person”, “location” or “activity” and “computational entity”). Each entity in CONON ontology

contains attributes (owl:DatatypeProperty) and relations (owl:ObjectProperty). Adding more entities is possible by using the built-in OWL property (owl:subClassOf), which enables ontology extension if additional service specific concepts are required. CONON ontology was later extended to contain sub-domain ontologies like “office domain” and “vehicle domain” and OWL-Time temporal ontology for more complex services.

CONON was utilized with OSGi (Open Services Gateway initiative) based Service Oriented Context Aware Middleware (SOCAM) architecture to produce rule-based reasoning results from context information and invoke smart home services [7].

Another very successful context information modeling is the one adapted by two software architectures namely Context Broker Architecture (CoBrA) and peer-to-peer data management architecture MoGATU [8]. CoBra ontology was originally designed for modeling knowledge in smart meeting rooms and MoGATU to model beliefs, desires and intentions (BDI) of software agents and human users. Both of them utilize OWL format for context modeling and reasoning purposes. Additionally, upper ontologies from Standard Ontology for Ubiquitous Pervasive Applications (SOUPA) are employed to enhance the application development process of the service integrator. This means that SOUPA contains various general vocabularies such as “time”, “space”, “location”, “action”, “events”, etc. which are imported to the to the domain ontology (e.g. Cobra-Ont) that is utilized by the architecture [9].

The most recent ontology related reasoning architecture for intelligent control of domotic devices is IntelliDomo [10,11]. IntelliDomo’s approach is to maintain a database that has the identical structure and data values with a device ontology stored in OWL format. The OWL device model also contains rules, written in semantic web rule language (SWRL), to derive intelligent services.

When a device changes its state, the data is first stored into the database and then the ontology instance is updated in the

340

OWL model. If a derived answer from the rule-engine updates the database, a software daemon will send out an update message using the home network bus. IntelliDomo also maintains OWL model for personal preferences so user’s requirements can be transferred into facts and considered in inference operation. It can be understood that, other than the home environment modeling, the most important aspect of IntelliDomo is to use the semantic web rule language to produce intelligent services in pre-meditated environment.

In addition to the context information models mentioned before there are other models with their own supporting frameworks such as Context Managing Framework [12] and Context Aware Sub-Structure (CASS) [13]. The typical feature for these context information models is simplicity so various applications can utilize the information structure easily. This is why it seems that their main focus was in their context framework operations rather than in context information modeling. For instance, Context Managing Framework utilizes data-centric blackboard architecture with simple abstraction model to share context data between the entities.

The main difference with our approach and the existing models is the layered smart home model where each domain (service, network, device and physical) will have their own individual sub-ontology instead of merging everything together. This layered structure provides an abstract separation among the components of smart home applications and allows the interoperability of these components. SOUPA is probably the closest related context information model as it can also be used to map concepts from different domains together.

B. Ontology structure based on Home Resource Model In general the different approaches of context information

modeling depend on the objectives of the modeling. The example objectives may consist of (i) the ease of capturing the real-world concepts to a digital form, (ii) the effective representation of context information model, (iii) the context reasoning support, and (iv) the easiness and scalability of context information management [14].

Based on the standard HES work, our approach is to construct a smart-home domain ontology utilizing several sub-ontologies e.g. the home domain information, the user-specific context data and the application specific data. This method supports context reasoning and provides means to expand the context information model easily.

The core ontology is shown in Fig. 4 where each class represents a mapping class to sub-ontology. The idea is to provide a common interface for different applications so they can utilize the functions provided by devices and also to be able to include additional ontologies, such as user, temporal, and location so context aware reasoning is possible. Each of the sub-ontologies can have their own versioning to support information

management if some totally new device type or concept emerges to home network markets.

Fig. 4 Core ontology model (left) where each class represent a mapping to a sub-

ontology and object properties of Device class (right).

The relationship structure of the core ontology is further described in in Fig. 5 where each element represents a sub-ontology and a mapping class in core ontology respectively. The Device class plays a central part of the model as it has many relationships but also User class is important in perspective of service integrator building context aware services. The sub-ontologies, which are marked with stripes, are based on the Home Resource Model domains: Service, Network, Device and Physical.

Fig. 5 The smart-home domain ontology structure including the ontologies based

on home resource model standard proposal.

The arrows in Fig. 5 illustrate object properties between the mapping classes. For example an instance in the User class hasService relationship with an instance in Service class. Also

341

instance in the Service class hasUser relationship to instance of the User class. Similarly there is one way relationship between User and Space (User isLocatedIn Space) and Physical and Location (Physical isLocatedAt Location). Practically this means Location ontology consists of different types of ways to measure a physical entity’s place such as Address, GPS or some other global coordinate system.

Time is also an important factor for many context aware applications so Temporal ontology that is related Space class by hasTime relationship. This is actually understandable because the space (location) has a relation with time in real world: e.g. time in USA is different than time in Korea.

To be able to compare the standard data description we present in Fig. 6 the same part (Device) that was presented in previous chapter when introducing Home Resource Model resource object. We imitated the resource model exactly and added only an object property hasFunction from Device class to Function class to be able to map Device with Function sub-ontology.

Fig. 6 Device sub-ontology main classes (left) with Device class properties

(right).

Finally we wish to highlight the Workticket class in the core ontology that is presented in Fig. 4. This class is designed to be used by the rule-engine when we are invoking services based on the reasoning result. Whenever a rule-engine fires a rule that requires an agent to take an action a new workticket instance will be generated and placed into Workticket class.

IV. TEST CASE: HEATING OPTIMIZATION To make a proof-of-concept for the HNRM standard proposal

as well as to demonstrate the semantic reasoning in an actual application area it was decided to run a test case with a heating optimization application. A person, who has a simple heater with no other control mechanism than on/off switch wants to control the device based on the available context data from her environment. The dynamic context data is presented in TABLE III.

The demonstration case expects the user’s home to reach the preference temperature when she arrives at home. During other times the energy consumption should be kept minimal. The basic scenario is as follows: the user will leave from her workplace at some time T and the reasoning engine will inference 1) when the

user will arrive at home, 2) when to turn on the heating device, 3) how to react to the context information changes.

TABLE III . ENVIRONMENT DATA USED IN DEMONSTRATION Information Service

Location of the user Mobile phone GPS

Weather Internet weather service

Traffic Dummy data from mobile phone

User’s schedule Internet calendar app

Home temperature measurement Home device interoperability system

Heater operation (on/off) Home device interoperability system

In addition to the dynamic data, the system already has some static information available like user’s temperature preferences, the average energy consumption of the heater, BIM (Building Information Model) of the house and a simplified models how different weather affects the traffic speed and heating need. The demonstration utilizes OWL-DL semantic model to handle the data relationships and SWRL rules to create axioms from the data model.

A. Software architecture The high-level software architecture is presented in Fig. 7

where the bottom shows the Inference Module (IM) and the top the plug-in services. These are connected together with a service framework which hosts services. The data is saved in an OWL format and the application rules are created with SWRL. The implementation utilizes a Jess rule-engine which invokes the service agents based on the inference results. When an external event is triggered, the service agents can place the event data into the ontology and invoke the inference module if necessary.

Fig. 7 Software architecture of the demonstration test case.

In the ontology, all services can register their underlying

functions, so the applications, which are using the individual services, can easily invoke correct service. As an example, let’s assume that a person has a thermometer at home whereas another

342

person has an A/C with a temperature sensor. Both devices provide the same functionality “temperature measurement” which can be used by other applications. This service enables the application developer to design an application that is usable in different environments. Of course this means the service agents and their functions need to be registered to the model. This process is shown in Fig. 8 sequence diagram. In the beginning the information model (OWL-file) is loaded by the operator to the inference module and then the related agents can register themselves. After registration, each agent will activate the reasoning procedures related to the services it can provide. When a registered service is identified and activated as a part of inference results, a “workticket” is generated that provides information to access the service.

Fig. 8 Registering the service and application agents into OWL model.

The worktickets are then retrieved by the different agents as presented in Fig. 9. When new data arrives to the model, other than appending the data into the ontology, the agents that have a permission to run reasoning (DoInference) can activate the rule engine to derive any new inference results. Because many parts of an application require data processing that is outside the capability of SWRL, the software agents can perform pre- or post-processing. An example would be calculating heating requirement based on the BIM (Building Information Model), measured temperature, weather and preference temperature, which is difficult to express with rule language. In the case that the agents notice no change on data values, they will not activate the rule engine.

Fig. 9 Agents invoking OSGi services and processing required data and

returning it back to the ontology model.

The activity diagram in

Fig. 10 describes the process of different agents working with gathering-, processing- and storing data. This process is stopped only if the application is terminated. The associated rules can then be deactivated in the ontology.

B. Testing The demonstration uses some existing software modules from

a Java service framework 1 to manage plug-in services, and a device interoperability system2, which can communicate with the different home networks. Also we utilized an existing Java based ontology modeling API3 for ontology management and a rule-based reasoning engine4 for context based reasoning.

Firstly, the real world concepts are captured by combining a cloud service and several individual home service agents that transfer the cloud service data to the context information model. As was mentioned before, we utilized three types of information sources: Android mobile phone, generic internet services, and home device interoperability system, which were presented in TABLE III.

1 OSGi (Open Service Gateway initiative) developed by OSGi alliance. 2 Device Interoperability architecture HeRA (Home Resource Architecture)

developed by ETRI. 3 Ontology editor - Protégé v. 3.4.4. developed by Stanford Center for

Biomedical Informatics Research. 4 Jess rule engine developed by Sandia National Laboratories

343

Fig. 10 Data gathering, processing and storing process.

In the demonstration scenario the Android application first sends its GPS location and dummy traffic data to a TCP server via JSON interchange format and the TCP server then provides an OSGi interface for the software agents to request data from the server. The identification to the server is done with a mobile phone number. The SW module setup is presented in Fig. 11.

Fig. 11 Demonstration architecture for collecting mobile phone data.

Android platform is also running a “result receiver manager” module which is used to show a simple timestamp when the user’s heater will be turned on after the inference module has received the necessary data and calculated the time. The GUI to this module is presented in Fig. 12. It shows the time in the same format that is used in Protégé dateTime property.

For the calendar and weather data we developed a simple Java application for interfacing Google weather and Google calendar internet services and abstracted the data for the demonstration. Basically we were interested in when the user may arrive at home and the need for heating/cooling operations according to weather information and user’s calendar data.

Fig. 12 GUI for presenting the result of the inference and sending dummy traffic

data to cloud server.

The final information source was device interoperability system called Home Resource Architecture (HeRA). Instead of a real test environment, we utilized a simulated temperature sensor and a virtual UPnP heating device whose state was controlled by HeRA. The heating device was a simple on/off device and the temperature sensor reading could be changed manually by web interface provided by HeRA. HeRA was also included in OSGi service framework as a plug-in service. The whole demonstration system therefore is constructed of four PCs and a mobile phone that is shown in

Fig. 13. Right-most computer was hosting virtual UPnP device. The laptop second from the right was running OSGi service framework and Jess rule-engine and was gathering data from mobile phone and internet services via software agents. The two computers on the left were hosting HeRA in VirtualBox and provided a temperature sensor web-access.

344

Fig. 13 Demonstration environment for testing. The test results show that, by extending the proposed data definition of the HES standard with user and environment information and utilizing ontology web language format, it is possible to create context-aware applications that fuse data intelligently from existing home network and internet services. The performance of our system is suitable for online control of the home devices as the delays from starting inference cycle to execution are in between 1-5 seconds, which is enough for most services.

V. CONCLUSIONS In this paper we described possible application to utilize the

data model that was defined by the ISO/IEC standard proposal of home network resource management (HNRM 30100-2). The data model supports different type of services: internet based services, mobile services, services provided by home appliances that are connected to a home network and the service integrator’s applications. In addition to this we extended the model to be able to support user’s life pattern data, which utilizes fused information such as home arrival time or the heating pattern of the user’s home.

We also described the design of home network agent architecture and verified the context information model with a semantic reasoning engine, where OSGi service framework provides data into ontology based reasoning engine. The demonstration scenario illustrated that it is possible to integrate various devices in different home networks and fuse information sources to create new useful services and to optimize the service efficiency by analyzing context data.

ACKNOWLEDGMENT This work was conducted using the Protégé resource, which

is supported by grant LM007885 from the United States National Library of Medicine.

REFERENCES [1] J. Lee, D. Jung, Y. Kim, Y. Lee, and Y. Kim, “Smart Grid solutions,

services, and business models focused on telco,” Network Operations and Management Symposium Workshops (NOMS Wksps), April 2010 IEEE/IFIP, pp. 323-326

[2] Y. Son, T. Pulkkinen, K. Moon, and C. Kim, “Home energy management system based on Power Line Communication,” IEEE Transactions on Consumer Electronincs, vol. 56, no. 3, pp. 1380-1386, Aug. 2010.

[3] Koolwaaij et al, “Context watcher: sharing context information in everyday life,” in Proceedings of the IASTED conference on Web Technologies, Applications and Services (WTAS), 2006

[4] Y.Son, T. Pulkkinen, and K. Lee “ISO/IEC CD 30100-1 INFORMATION TECHNOLOGY - Home Network Resource Management – Part 1: Requirements,” http://www.iso.org/iso/search.htm?qt=30100&searchSubmit=Search&sort=rel&type=simple&published=on

[5] Y.Son, T. Pulkkinen, and K. Lee “ISO/IEC CD 30100-2 INFORMATION TECHNOLOGY - Home Network Resource Management – Part 2: Architecture,” http://www.iso.org/iso/search.htm?qt=30100&searchSubmit=Search&sort=rel&type=simple&published=on

[6] Ji-Yeon Son, Jun-Hee Park, Kyeong-Deok Moon, Young-Hee Lee, “Resource-Aware Smart Home Management System by Constructing Resource Relation Graph,” IEEE Transactions on Consumer Electronics, Vol. 57, No. 3, Aug. 2011

[7] D. Zhang, T. Gu, X. Wang, “Enabling context-aware smart home with semantic technology,” International Journal of Human-friendly Welfare Robotic Systems Vol. 6 No. 4 pp. 12-20, 2005

[8] T. Gu1, H. Pung1, D. Q. Zhang, “A middleware for building context-aware mobile services,” IEEE 59th Vehicular Technology Conference. VTC 2004-Spring, 17-19 May 2004, Milan, Italy

[9] H. Chen, T. Finin, A. Joshi, “The SOUPA Ontology for Pervasive Computing,” Whitestein Series in Software Agent Technologies, Springer. 2005.

[10] H. Chen, et al., “Intelligent agents meet the semantic Web in smart spaces,” Internet Computing, IEEE Vol. 8 , No. 6, pp. 69-79, 2004

[11] P. Valiente-Rocha and A. Lozano-Tello, “Control model of domotic systems based on ontologies,” 2nd Int. Conference on Agents and Artificial Intelligence, pp. 470–473 Valencia,Spain, Jan. 22-24, 2010

[12] P. Valiente-Rocha and A. Lozano-Tello, “Ontology and SWRL-Based Learning Model for Home Automation Controlling,” in J.C. Augusto et al. (Eds.): Advances in Intelligent and Soft Computing Vol. 72 – Ambient Intelligence and Future Trends-International Symposium on Ambient Intelligence (ISAmI 2010, AISC 72), pp. 79–86, 2010

[13] P. Fahy and S. Clarke “CASS – a middleware for mobile context-aware applications,” Workshop on Context Awareness, MobiSys, 2004

[14] P. Korpipää, J. Mäntyjärvi, J. Kela, H. Keränen and E-J. Malm, “Managing context information in mobile devices,” IEEE Pervasive Computing, Vol. 2, No. 3, pp.42–51, July–September, 2003

[15] C. Bettini et al. “A survey of context modeling and reasoning techniques,” Pervasive Mobile Computing, Doi: 10.10 I6/j. pmcj.2009 .06.002, 2009

345