Haufe – Innovation Project

Summary A team from Microsoft, Haufe Group and Haufe Akademie set out to build a chatbot with

Microsoft LUIS for the website of Haufe Akademie.

Its main purpose is to interact with the customer and help him searching products on the website,

selecting training subject, date, location. With over 860 training subjects and more than 90.000

participants per year a technical solution is key. We focus on staying connected to our customer

24/7 - around the clock and globe.

Key technologies

• Microsoft Bot Framework

• Cognitive Services LUIS API

• Azure Search

• Azure Storage

• Azure Linux Virtual Machines

• Docker

• Jenkins

All Azure services and the Bot Framework have been programmed in Node.js.

Customer profile Haufe Akademie is one of the leading partners in the development of companies, professionals

and executives in German-speaking countries. We work with around 1,200 trainers and

consultants and provide more than 1,000 diverse topics across 23 fields with around 95,000

participants annually. All DAX 30 companies draw on Haufe Akademie’s expertise for their training

needs, thereby ensuring that they are prepared for the future. Haufe Academy is part of the Haufe

Group. The Group is widely regarded as one of the most innovative media and software

companies in Germany

Problem Statement Haufe Akademie offers their training portfolio through their website. As they offer various topics

of courses it is sometimes overwhelming for the customer to get the right one for the right date

and location. On the product landing page of a specific training they offer human chat support

but customers often don't reach those sites and the human support chat is limited to local

working hours. Here is where ACE - the chat bot is coming into play.

ACE will be available at any time as first contact chat for the customer. The customer can interact

with ACE and ACE will guide the customer to find the right trainings in the location and time

frame they're looking for. If the customer wants, ACE assists in connecting with a human support

buddy if they're logged in into the system.

Solution overview To enhance the user experience on the website we've decided to build a Bot based on the Bot

Framework and LUIS. In the first architecture session we declared the domain and determined the

necessary entities to interact with the Language Understanding Intelligence Service (a.k.a. LUIS).

This diagram shows the process of iterating through the domain

We've identified several entities but decided to go for the first sprint with those 3 entities

• Location

• Date

• Topic

Further we discussed how the bot will interact for the first sprint with the users and have defined

a basic workflow

Our architecture looks something like the diagram below. We have the bot framework app that

provides “channels” to talk to our bot through. One of these channels being the iframe that we

can embed in our website.

The framework app is configurable with the url (the endpoint has to be https!) of the actual bot

server. We used node.js and deployed the application as a docker container on an Azure VM. We

plan on migrating the container to the Azure Container Service.

The SDK allows for an easy setup of a connection to a luis.ai app. To make it work, one needs to

define the intents and entities that should be detected. An intent is basically the verb/action of

the sentence the user typed in, while an entity is a piece of information related to the intent. To

train Luis, we input as many “utterances” as we can think of - the more the better. This has to be

done for each supported intent. In our case, we only had to support “search” and “ask for a

human”. As entities, we have “course type”, “topic”, “datetime” and “location”.

Once the intent was detected as “search”, we simply used the entities as parameters for the Azure

Search Service. Since our data was a spreadsheed file, all we had to do in order to be able query it

was to export it as a csv file, upload it to an Azure Blob Storage and then use an Azure Search

Indexer to add it to our Azure Search Index.

Technical delivery Prerequisites Visual Studio Code (or any Editor of your choice)

We've chosen Visual Studio Code as Editor as it is cross-platform available and integrates great in

different programming languages. It has great Node.js support and debugging capabilities.

The programming langauge has been Node.js which we've used the latest stable version.

The dev machine should also install the Microsoft Bot Framework where you can find the

detailed documentation on how to install it and use the Node.js SDK for the Bot Framework.

Also install the Bot Framework Emulator as you want to test the bot reacting on user input on

your local dev box instead deploying it each time to test.

An active Azure subscription is needed create the Azure Search endpoint and the Azure Storage

endpoint. To create the Azure Search Service follow this instructions. To create the Azure Storage

endpoint follow this instructions How to create a storage account

Defining the LUIS Model The LUIS model we used for this project is pretty simple. It contains exactly two intents.

What was more of a challenge is that we're using LUIS in the German market and not each feature

of LUIS is already available. For instance we have no prebuilt domains we could use therefore we

had to define our one entities. This was especially challenging for the Date.

As this is just a matter of time we expect to use the prebuilt domains in future sprints. With that

set up we've began to enter utterances into the intents to define a good variety of training data

to LUIS. The following show the utterances for the search intent.

Implementing the Bot For the bot implementation we decided to go with LUIS Action Bindings. This makes it pretty

simple to react on certain intents and get easy resolving of the entities.

var SearchAction = { intentName: 'search', friendlyName: 'I want to search for a seminar', confirmOnContextSwitch: false, // allow to abandon this action without confirmation schema: { Location: { type: 'string', message: 'Wo soll das Seminar stattfinden?', optional: true }, StartDate: { type: 'string', message: 'Wann soll das Seminar stattfinden?', optional: true }, Topic: { type: 'string', message: 'Was ist das Thema des Kurses, den Sie suchen?', optional: false }, Type: { type: 'string', message: 'Wofur suchen sie nach?', optional: true }, }, fulfill: function (parameters, callback) { callback("") } }; module.exports = SearchAction;

This binding shows what happens if an entity is missing. For instance if the topic of the search is

missing, the binding automatically reacts and tries to ask the user for more information on this as

the optional parameter was set to false.

There was one caveat in using this bindings as they've been in the current state of the project.

You haven't been able to leverage any session related commands like Adaptive Cards as session

is not delegated into the binding. As of the time of this writing there is an open pull request

https://github.com/Microsoft/BotBuilder-Samples/pull/130 to bridge that gap. We've taken this

basic idea and implemented this into the core sdk in our project. That is the reason you just see

here the callback(""); trigger without parameters. It will be handled by the custom

implementation we did.

The custom implementation assumes that for each binding we'll have an handler. Handlers have

similarity to bindings.

var azureSearchHelper = require("../helpers/azureSearchHelper") var cardBuilderHelper = require("../helpers/cardBuilderHelper") var messageBuilderHelper = require("../helpers/messageBuilderHelper") var SearchHandler = { intentName: "search", handle: function (session, actionModel) { console.log("\nEntities found: " + JSON.stringify(actionModel.parameters) + "\n"); azureSearchHelper.azureSearchLuis(actionModel.parameters, function (result) { // console.log("\nFinal Result: \n" + JSON.stringify(result)); session.send(messageBuilderHelper.getSearchingMessage(actionModel.parameters, session)) if (!result || (result.length === 0)) { console.log("NO RESULT!") session.send('Ich konnte keine passenden Einträge für sie finden. Bitte versuchen sie es erneut.') } else { var reply = (cardBuilderHelper.getCarouselFromResult(result, session)) session.send(reply) session.send("Ich hoffe das Ergebnis ist für Ihnem hilfreich. Bitte fühlen Sie sich frei, eine neue Suchanfrage einzureichen.") }

session.endDialog(); }) } } module.exports = SearchHandler;

There we do the whole implementation of the result like in the above, searching for data and

creating an adaptive card carousel to output to the user in the webchat interface.

The binding and action will be triggered by the fulfillHandler based on the Pull Request of the bot

framework sample mentioned above.

var FulfillReplyHandler = function(session, actionModel) { let foundHandler = false; console.log('\nAction Binding "' + actionModel.intentName + '" completed:\n', actionModel, '\n'); AllHandlers.forEach(function(element) { if(actionModel.intentName === element.intentName) { foundHandler = true; element.handle(session, actionModel); } }, this); if(!foundHandler) { session.endDialog(actionModel.result.toString()); } }

Searching for data and displaying results The bot application is using helpers to retrieve data and display it. The data retrieval is a bit tricky

as the internal training data actually the catalog of trainings is in an internal ERP system which is

producing once a while a CSV list output. This output is stored on the blob storage endpoint

which is then used by the Azure Search service indexer which indexes all relevant CSV files.

This index is used by the helper to retrieve the data

require('dotenv-extended').load({ path: '../env' }); function azureSearch(searchParameters, callback) { // Setup azure-search var AzureSearch = require('azure-search'); var client = AzureSearch({ url: process.env.AZURE_SEARCH_URL, key: process.env.AZURE_SEARCH_KEY }); client.search(process.env.AZURE_SEARCH_INDEX, searchParameters, function (err, results) { if (err) { console.log("\nERROR:\n" + JSON.stringify(err) + "\n") } // console.log("\nAZURE SEARCH RESULTS:\n" + JSON.stringify(results) + "\n") callback(results) }); } function getAzureSearchEntity(parameters) { var Sugar = require("sugar-date"); Sugar.Date.setLocale('de'); var azureSearchEntity = { search: "", top: 200, queryType: "full" }

if (parameters.Topic) { azureSearchEntity.search += "((ThemenbereichUndThemenblock:" + filterKeywords(parameters.Topic) + ")" azureSearchEntity.search += "||(TitelUntertitel:" + filterKeywords(parameters.Topic) + "))" } if (parameters.Location) { azureSearchEntity.search += "&&(Ort:" + filterKeywords(parameters.Location) + ")" } if (parameters.StartDate) { var sugarDate = Sugar.Date.create(parameters.StartDate); var twoDigitDate = sugarDate.getDate(); if (twoDigitDate < 10) twoDigitDate = "0" + twoDigitDate; var twoDigitMonth = sugarDate.getMonth() + 1; if (twoDigitMonth < 10) twoDigitMonth = "0" + twoDigitMonth; azureSearchEntity.filter = "StartDatum ge " + sugarDate.getFullYear() + "-" + twoDigitMonth + "-" + twoDigitDate + "T00:00:00Z"; } console.log("AzureSearchEntity: " + JSON.stringify(azureSearchEntity) + "\n"); return azureSearchEntity } function combineResults(results) { var output = [] for (key in results) { var found = false; for (var key2 in output) { if (output[key2].Titel == results[key].Titel) { output[key2].SpaceTime.push(makeEntryLink(results[key])) found = true } } if (!found) { results[key].SpaceTime = [] results[key].SpaceTime.push(makeEntryLink(results[key])) output.push(results[key]) } } // console.log("\nOutput final: " + JSON.stringify(output)) return output } function makeEntryLink(entry) { var output = { "Location": entry.Ort, "StartDate": entry.Beginn, "AkaId": entry.Veranstaltungskennung, "webLink": "https://www.haufe-akademie.de/" + entry.Veranstaltungskennung } return output } function filterKeywords(topicString) { var temp = "" if (topicString.includes(" ")){ if(topicString.includes("und")){ temp+="/@" + topicString + "/@" temp+="&&/@" + topicString.replace(/ und /g, '/@&&/@') + "/@" } else{ temp += "/@" + topicString.replace(/ /g, '') + "/@" temp += "&&/@" +topicString.replace(/ /g, '/@&&/@') + "/@" } }

else{ temp = "/@" + topicString + "/@" } return temp } function azureSearchLuis(luisEntities, callback) { azureSearch(getAzureSearchEntity(luisEntities), function (azureSearchResults) { callback(combineResults(azureSearchResults)) }) } module.exports.azureSearchLuis = azureSearchLuis

Once the data is retrieved another helper is building the adaptive card outputs to build a nice

card carousel for the customer to iterate through. Here is the code of this helper.

var builder = require("botbuilder"); function getHeroCardFromEntity(entity, session) { var textMessage = "" // textMessage+="**Themenblock**: " + entity.Themenblock + "\n" // textMessage+="<b>Themenbereich\n: </b>" + entity.Themenbereich + "\n" textMessage += "Ort und Startdatum: " for (key in entity.SpaceTime) { if (key > 0) textMessage += ", " textMessage += entity.SpaceTime[key].Location + " " + entity.SpaceTime[key].StartDate } textMessage += ";\n" if (entity.Dauer1 && entity.Dauer2) { textMessage += " Dauer: " + entity.Dauer1 + " " + entity.Dauer2 + "\n" } return new builder.HeroCard(session) .title(entity.Titel) .subtitle(entity.Untertitel) .text(textMessage) .images([ builder.CardImage.create(session, 'https://www.haufe-akademie.de/images/header/logo_195x40.gif') ]) .buttons([ builder.CardAction.openUrl(session, 'https://www.haufe-akademie.de/' + entity.Veranstaltungskennung, 'Learn More') ]) } function getAddaptiveCardFromEntity(entity, session) { var temp = {} temp.contentType = "application/vnd.microsoft.card.adaptive" temp.content = {} temp.content.type = "AdaptiveCard" temp.content.body = [] temp.content.body.push({ "type": "Image", "size": "medium", "url": "https://www.haufe-akademie.de/images/header/logo_195x40.gif" }) temp.content.body.push({ "type": "TextBlock", "text": entity.Titel, "size": "extraLarge", "separation": "strong", "weight": "bolder" }) temp.content.body.push({ "type": "TextBlock", "text": entity.Untertitel, "size": "large",

"separation": "none", "weight": "lighter" }) var locdate = "" for (key in entity.SpaceTime) { if (key > 0) locdate += ", " var loc="" if (entity.SpaceTime[key].Location) loc = entity.SpaceTime[key].Location var date="" if (entity.SpaceTime[key].StartDate) date = entity.SpaceTime[key].StartDate locdate += loc + " " + date } temp.content.body.push({ "type": "TextBlock", "text": "**Ort und Startdatum:** " + locdate, }) if (entity.Dauer1 && entity.Dauer2) { temp.content.body.push({ "type": "TextBlock", "text": "**Dauer:** " + entity.Dauer1 + " " + entity.Dauer2, "separation": "none", }) } // temp.content.body.push({ // "type": "TextBlock", // "text": "**Suchergebnis score:** " + entity["@search.score"], // "separation": "none", // }) temp.content.actions = [] temp.content.actions.push({ "type": "Action.OpenUrl", "url": "https://www.haufe-akademie.de/" + entity.Veranstaltungskennung, "title": "Erfahren Sie mehr", }) return temp } function getCardsFromResult(result, session) { var cards = [] for (key in result) { cards.push(getAddaptiveCardFromEntity(result[key], session)) } return cards; } function getCarouselFromResult(result, session){ var cards = getCardsFromResult(result, session) // create reply with Carousel AttachmentLayout var reply if (!cards || (cards.length === 0)){ reply = "Ich konnte keine passenden Einträge für sie finden. Bitte versuchen sie es erneut." } else{ var cardText="" if (cards.length == 1) { cardText = "Ich habe einen eintrag gefunden, die Ihren Kriterien entsprechen:" } else { cardText = "Ich habe " + cards.length + " Einträge gefunden, die Ihren Kriterien entsprechen:" } reply = new builder.Message(session) .text(cardText)

.attachmentLayout(builder.AttachmentLayout.carousel) .attachments(cards); } return reply } module.exports.getCarouselFromResult=getCarouselFromResult

Deployment As already mentioned deployment is done through the Jenkins pipeline. In general the current

deployment strategy is to deploy the bot in an docker container and host that currently on a

Linux VM. Alternatives would be Azure Container Services or Azure Bot Service as hosting

environments. Due to the short time frame for the first sprint we leveraged this way as it was the

easiest for the project team.

Here is the dockerfile

FROM node:boron RUN mkdir -p /usr/src/app WORKDIR /usr/src/app COPY package.json /usr/src/app/ RUN npm install COPY app.js /usr/src/app ADD core /usr/src/app/core ADD handlers /usr/src/app/handlers ADD helpers /usr/src/app/helpers ADD actions /usr/src/app/actions EXPOSE 3978 CMD [ "npm", "start" ]

and the corresponding bash script

# !/bin/bash source .env set -e DOCKER_LOCAL=".docker/machine/" docker-machine -s ${DOCKER_LOCAL} ls #Checking if docker machine exists test_con=`docker-machine -s ${DOCKER_LOCAL} ls | grep ${VM_NAME}` #If docker machin doesn't exist create it if [ ${#test_con} -lt 2 ] ; then echo "----- Creating machine -----" docker-machine -s ${DOCKER_LOCAL} create --driver generic --generic-ip-address=${VM_IP} --generic-ssh-user=${VM_USER} --generic-ssh-key=${SSH_KEY_PATH} ${VM_NAME} fi export DOCKER_TLS_VERIFY="1" export DOCKER_HOST="tcp://${VM_IP}:2376" export DOCKER_CERT_PATH="${DOCKER_LOCAL}/machines/${VM_NAME}" export DOCKER_MACHINE_NAME=${VM_NAME} #Build image on remote host docker build -t haufe-lexware/akachatbot . set +e docker stop $(docker ps -aq --filter name="akachatbot") docker rm -vf $(docker ps -aq --filter name="akachatbot") set -e

docker run -p 3978:3978 -e "MICROSOFT_APP_PASSWORD=${MICROSOFT_APP_PASSWORD}" -e "MICROSOFT_APP_ID=${MICROSOFT_APP_ID}" -e "LUIS_MODEL_URL=${LUIS_MODEL_URL}" -e "AZURE_SEARCH_URL=${AZURE_SEARCH_URL}" -e "AZURE_SEARCH_KEY=${AZURE_SEARCH_KEY}" -e "AZURE_SEARCH_INDEX=${AZURE_SEARCH_INDEX}" -d --name akachatbot haufe-lexware/akachatbot docker system prune -f

Conclusions We solved the problem to have a chat interface enabled for customers around the clock in

assisting them to find trainings and seminars by implementing a Bot using the Bot Framework

leveraging Language Understanding Intelligence Service and accessing the training data through

Azure Search to provide the results in a conversation with the customer through the chat interface

via the Bot.

With the use of the Bot Framework and LUIS it was pretty straightforward to create a base

architecture and a first implementation to enhance this bot application easily within the next

sprints. The current implementation took a couple of days from zero to a working version.

Additional resources This video shows the basic functionality of the bot. The bot itself just supports german, but the

video has english narration.

<iframe width="560" height="315" src="https://www.youtube.com/embed/X8SVBXhvzr8"

frameborder="0" allowfullscreen></iframe>

Team To provide a proof-of-concept implementation, a team worked together in May 2017, each

focusing on a different part of the chatbot implementation:

• Ioan-Bogdan Cimpoesu, Haufe Gruppe

• George Ganea, Haufe Gruppe

• Dariusz Parys, Microsoft

• Larissa Gruner, Haufe Akademie