instructions - kore.ai · web viewalert tasks allow a bot to initiate a conversation with the user...

VENDOR DUE DILIGENCE: CRITERIA FOR

EVALUATING ENTERPRISE BOT SOLUTIONS

Must-Have Requirements for Bot Development

InstructionsThis document provides decision-makers with a comprehensive set of criteria for assessing enterprise bot solutions and objectively evaluating and ranking competing products or services.

Step 1: Choose two vendors to evaluate and rank.

Step 2: Review evaluation criteria and description (if applicable) and respond to each with Yes, No, Partial, or Roadmap (for upcoming features).

Step 3 (Optional): Add comments to Partial or Roadmap responses to further clarify differences between product offerings.

Step 4 (Optional): Use the comment field to mark features as must-haves, nice-to-haves, and will-not-use to help determine the importance of evaluation criteria for your organization.

Step 5: Review the responses given to each evaluation criteria and use the overall “scores” to help in your evaluation and purchasing decision.

Table of ContentsInstructions Custom vs. Out-of-the-Box Page 12

The Evaluation Process Page 2 Natural Language Configuration Page 13-14

Bot Capabilities Page 3-5 Developer Tools Page 15-16

Natural Language Processing (NLP) Page 6-7 Security Page 17

1

Omni-Channel Page 8-9 Enterprise Administration Page 18-19

Deployment & Architecture Page 10 Bot Store 0

Integration Page 11 Other Key Factors

The Evaluation ProcessWhy Build Bots?Bots help perform the routine functions essential to running your business, increasing organizational efficiency and worker productivity. They also help simplify the process of finding and consuming information, reduce app fatigue, and improve customer loyalty with right-time reminders, right-fit offers, and easy-to-access, simpler, anytime/anywhere, personalized service today’s consumers expect.

Why Choose a Bots Platform? Bots Platforms are designed to make it easy for enterprises and developers to build, host, and manage bots that can bring instant value to employees and customers. Top-ranking platforms provide a complete, customizable, secure, and cost-effective framework that ensures a consistent and scalable process for building bots without the need for expensive hosting, time-consuming coding, or complex custom software.

How to Choose the Perfect Bots Solution for Your Organization?Once you’ve reviewed your evaluation criteria, it’s time to select a shortlist of product candidates and proceed with your typical procurement process. You can use this document to review and rank the leading bots tools on the market.

In addition to this list of evaluation criteria for bot development solutions, you may want to consider these additional factors that will help in your technology search:

1. How easy and intuitive is it to learn? 2. How long does it take to deploy?3. What kind of support will you receive during and after implementation?4. How easy is it to collaborate on bot projects? 5. How long does it take to build bots using the solution?6. What is the pricing and pricing structure in relation to your must-have features?

7380 W. Sand Lake Rd., Suite 430 | Orlando, FL 32819 | +1 321.420.0990 | [email protected] | www.kore.ai 2

Bot CapabilitiesWhat should the solution provide? The solution should allow you to build bots for both your internal workforce and your customers - bots that can complete tasks, deliver detailed alerts, fetch reports or other long-form content, and simplify workflows between and across systems.

Why is this important? Using bots that deliver detailed information on-demand or on a schedule is essential to the

time to value users will gain so they don’t have to jump from screen-to-screen to get the information they need.

If a bot does not understand the user’s intent, it is imperative that it can analyze and execute long-form requests while providing guidance and collecting the information necessary to complete tasks.

Users must optimize productivity with bots that eliminate redundancies like automatically populating data fields of related tasks. Users need bots to provide frictionless system-to-system processes, alerts, and actionable data to get tasks done faster with more accuracy.

Ref #

Evaluation Criteria

Description Vendor A Vendor B Comments

1

Can the bots be targeted towards end consumers (B2C)?

1.1

Can the bots be targeted towards employees (B2E)?

1.2

Can the bots be targeted towards business users (B2B)?

1.3

Will the bot allow end users to authenticate and authorize using their own personal credentials?

1.4 Can the bot perform action tasks

Action tasks are user initiated interactions with the bot that usually


Ref #

Evaluation Criteria


for the users? update back-end systems or perform transactions.

1.5

Can the bot alert the user of specific events that maybe of relevance to the user?

Alert tasks allow a bot to initiate a conversation with the user by bringing relevant & personalized information to the users from systems of record.

1.6

Can users personalize the bot alerts?

Personalization can be in the form of individual filters and scheduling options that each user can specify per their preferences.

1.7

Can the user also request reports or information that the bot can fetch for the user?

1.8

Can the report requests be on demand and scheduled for periodic retrievals?

1.9

Does the bot enable you to respond to questions from users?

The question and answer (FAQs) content can be either within a content management system (accessed via APIs) or stored within the platform.

1.10

Does the platform provide the ability to make the alerts actionable?

Users should have the ability to take quick actions when alerts and notifications are delivered to the user.

1.11

Can enterprises execute workflows by mapping actions to actions?


Ref #

Evaluation Criteria


1.12

Can users automate and suggest subsequent actions?

1.13

Can bots be instantiated within a team context?

1.14

Do the bots support input in natural language so that users can utter sentences in natural language to the bot?

This is generally an unstructured input provided by the user.

1.15

Do the bots provide a forms based UI in addition to natural language based input?

Forms would need to be dynamically generated based on bot configuration when the user requests it. This feature also be used in combination with natural language input depending on use cases.


Natural Language Processing (NLP)

What should the solution provide? The solution should provide a powerful NLP engine. This engine should allow your bot to process text and speech and find and understand the semantics, syntax, and grammar of a sentence in order to process user requests and identify intent and parameters for executing tasks.

Why is this important? Without natural language understanding, bots are unable to decipher intent from human

utterance. Bots must state track accurately to understand what is being discussed at a point in time and in

the context of the full conversation. Dialogue management is also imperative for the bot, enabling it to reason over the course of

the conversation and decide what to say next with the help of machine learning. Bots must also be able to generate natural language, allowing it to respond in a way that makes

sense to users.

Ref #

Evaluation Criteria


2 Does the NLP engine support standard functions such as Language Parsing, Grammatical Correction/ Normalization, POS tagging and logging and tracing for all utterances?

2.1 Does the NLP engine support user prompting and dialog management to start and end a conversation with a user?

2.2 Can the bot have a multiple volley conversation with the


Ref #

Evaluation Criteria


human?2.3 Does the

platform have intent recognition?

2.4 Does the platform have entity extraction?

2.5 Does the platform use machine learning algorithms to train and tune the NLP accuracy?

2.6 Can enterprises only use NLP engine from the platform and develop their own custom logic to handle the task execution?

2.7 Does the bot have the ability to automatically remember context for the user?

User context can long term (across various sessions/interactions with the users) or short term (within a given session/interaction with the user).

2.8 Does the developer have the ability to manipulate the user context using custom code logic?

Developers may choose to further optimize the bot interaction with the user by promoting additional data to the user context or pulling data from the context to improve dialog with the user.

2.9 Does the platform provide its own universal bot that can interact with other individual bots?

A universal bot allows users to engage with a single bot that may perform tasks across various other bots that may be available and installed by the users or available within the Bot


Ref #

Evaluation Criteria


Store.

Omni-ChannelWhat should the solution provide? The solution should allow you to deploy your bots anywhere your employees and customers need - from social media platforms to popular messaging apps, websites, SMS, email, and more.

Why is this important? Sixty percent of customers interact through multiple channels, irrespective of device or

location, so a seamless experience between channels is critical to customer retention and satisfaction.

Building a bot once and deploying to multiple channels enables integration across those different channels, information sources, and applications that help maintain accuracy and personalization across users.

Bots built on a non-agnostic platform can leave bot builders needing another vendor for deployment across different channels. Use a bot builder that enables developers to leverage the native UX aspects of their desired communication channels, such as structured templates in Facebook Messenger, dialogue handling in Skype and more. That means bots get even more capability as they cross channels and function in same landscape the user will expect.

Ref #

Evaluation Criteria


3 Does the platform have omni-channel support?

Support should include the ability to support multiple channels using a single bot definition. Each channel should be easily enabled through the platform.

3.1 The channels supported by the platform are:

3.2 Email 3.3 SMS 3.4 Facebook

Messenger

3.5 Web apps (SDK for web applications)

3.6 Mobile apps (SDK for mobile applications)


Ref #

Evaluation Criteria


3.7 Slack 3.8 Amazon Echo 3.9 Other 3.10 Does the

platform allow the developer to take advantage of channel specific responses?

Each channels may provide UI widgets or elements and have specific experience that developers can take advantage of. It would be important to optimize the experience per channel in certain cases.

3.11 Does the platform provide its own messaging application & platform where bots can be instantiated?

Depending on the use cases and audience, enterprises can have a more controlled channel to offer the bot capabilities using a messaging application that they control with unique capabilities.


Deployment & ArchitectureWhat should the solution provide? The platform should support both on-premise and cloud deployments.

Why is this important? Understanding how a cloud or on-premise deployment affects operational, security, and

compliance risk management will help you determine the appropriate deployment option. Becoming knowledgeable about the data sources your bots project requires enables you to

understand the risk associated with your company’s data security requirements. Use this knowledge to make an informed choice for your organization based on facts, not misconceptions around data integration or security challenges.

Selecting a vendor that provides both options allows you to evaluate benefits and possible complexities like time to value, integration, payment options, and the maturity of your service delivery models.

Ref #

Evaluation Criteria


4

Is the bots platform available for hosting inside a cloud infrastructure?

Provide details on which cloud infrastructure providers are supported by your platform (for example, AWS, SAP Hana Cloud Platform, IBM Bluemix, etc.)?

4.1

Does the platform offer its own cloud connector to securely connect and access on-premise services from the cloud tenant?

4.2

Does the platform let you host and deploy the bot in an auto-scaling infrastructure environment?

4.3

Is the platform available for on-premise deployment?


Ref #

Evaluation Criteria


4.4

Does the bots platform also support a hybrid deployment architecture?

IntegrationWhat should the solution provide? The solution should allow you to easily and securely connect to systems, web applications, and web sites using APIs.

Why is this important? The breadth of your bot capabilities is only as good as the APIs the bot can consume. Multiple methods of communicating with the APIs are paramount if you intend on utilizing the

entire breadth of the platform. While SOAP and RESTful communication are obvious needs, you also need to support webhooks, webservices, and RSS connections.

Ref #

Evaluation Criteria Description Vendor A Vendor B Comments

5 Does the bot platform provide the ability to execute tasks by integrating and consuming APIs provided by various systems of record?

5.1 Does the platform support REST & ODATA APIs?

5.2 Does the platform support Web Services/ SOAP APIs?

5.3 Does the platform support other integration protocols? Please explain.

5.4 Does the platform have out-of-the-box adapters to support various message delivery channels?

5.5 Does the bots platform expose APIs that can be invoked


Ref #

Evaluation Criteria Description Vendor A Vendor B Comments

using another application?


Custom vs. OOTBWhat should the solution provide? The solution should offer extensive support for the building of new bots and the customization of existing bots. It should also provide a deep catalogue of ready-to-use bots.

Why is this important? Offering a variety of existing bots and solution bots allows developers to customize faster and

deploy with ease. Out-of-the-box bots enable enterprises to start with a predefined set of top tasks that can

immediately accomplish assignments without having to build bots from scratch. The appropriate platform should allow enterprises to design custom bots with specific

“personalities” and language by industry, while optimizing legacy systems and simplifying UIs without replacing systems all together.

Ref #

Evaluation Criteria


6 Does the platform enable any developer to build a bot for custom requirements?

6.1 Does the platform provide out-of-the-box bots that can be used?

6.2 Can enterprise customize and extend the out-of-the-box bots that are provided by the vendor?


Natural Language ConfigurationWhat should the solution provide? The solution should provide your developers auto-NLP and the tools they need to easily refine the way their bots communicate both in function and personality. It should allow your team to change, test, and extend the NLP capabilities of your bot.

Why is this important? The NLP technology should have a preset library of synonyms which allows tasks to have out-of-

the-box, rudimentary NLP capabilities when configured. Utterances should be validated while bot configuration is in flight. Synonyms and patterns must

be tested to ensure NLP accuracy before deploying the bot. The platform should maintain Chatlog history with an extensive success/failed scenario

database in order to “teach” the bot and fine tune NLP post deployment.

Ref #

Evaluation Criteria


7 Does the platform auto-NLP enable your bots?

Auto-NLP enablement refers to the ability to enable natural language input for your bot and tasks without any need for explicit NLP configuration.

7.1 Does the developer have to custom develop code and logic to define every user utterance and conversation?

7.2 Does platform provide its own library of synonyms and patterns that the enterprise can take advantage of?

7.3 Can the library of synonyms and patterns be extended and optimized by an enterprise for their own


Ref #

Evaluation Criteria


unique needs?7.4 Can the

enterprise share the NLP optimization across all bots within the enterprise?

7.5 Can enterprises perform manual or automated testing of NLP configuration to check accuracy or utterances?

7.6 Does the platform provide ability for enterprise to change and customize the bot personality (language, dialog and responses)?

7.7 Can enterprises also use supervised learning approach where developers or agents and view intent and entity successes and failure and refine the NLP accuracy?

7.8 Does the NLP engine support multiple languages?


Developer ToolsWhat should the solution provide? The solution should provide a complete framework and execution model for building custom bots. It should provide all the tools you need to manage, publish, tweak, test, and debug your bots before rolling them out to employees or customers.

Why is this important? Platforms with a “builder tool” should enable organizations to build single bots for universal

consumption and “solution bots”, multiple bots, that appear as one bot to a user. For example, a sales solution bot should integrate with CRM, calendar, and virtual meeting applications etc.

A robust developer tool is capable of testing during bot configuration and building which reduces development cycles and time to value.

Multiple developers must be able to collaborate and modify each bot. Capabilities such as adding developer rights or passing ownership to another developer are increasingly important as the bot matures, or multiple bots are authored.

An appropriate platform should provide existing bots that already have tasks built in, which can be extended to developers and customized as needed by organization.

Ref #

Evaluation Criteria


8 Does the platform provide a developer tool which allows them to build new bots or extend existing bots?

8.1 Are developers able to easily "configure" a task which consumes an API from a back-end system of record?

8.2 Are developers able to configure the response provided by the bot once the API calls are made?

Developers should be able to simply configure the responses as well as develop custom code logic to manipulate the API response before presenting to the user.


Ref #

Evaluation Criteria


8.3 Can developers test and run the bot to functionally test the bot?

8.4 Can developers follow a complete software developer lifecycle for bot development?

8.5 Can developers share bot and task development with other developers within the company?


SecurityWhat should the solution provide? The solution should provide enterprise-grade security to ensure data privacy and help your organization meet strict compliance requirements.

Why is this important? The appropriate platform will enable enterprises to authenticate users through their custom

methods. Encryption should be utilized where regulatory compliance is required. Data retention should be among the requirements for all organizations adopting bot

technology, both from a bot-to-platform and bot-to-user perspective to maintain system-to-user accuracy.

Ref # Evaluation Criteria


9 Does the bot allow enforcement of authentication and authorization rules as specified by the enterprise systems?

Enterprise systems may have specific user access privilege rules and other security configuration such as session timeouts that would need to be enforced and emulated by the bots platform.

9.1 Does the bot platform support standard authentication models (Basic HTTP Auth, Oauth, API key, etc.) to authenticate the user? Please explain.

9.2 Are bot messages encrypted while at rest within the database?

9.3 Are bot messages encrypted in transit?

9.4 Can enterprises audit the bot




conversations with users?

Enterprise AdministrationWhat should the solution provide? The solution should allow you to control who can use, manage, and deploy custom bots and access administrative consoles. It should allow you to easily track bot usage with comprehensive reporting and analytics.

Why is this important? Proper usage statistics logging and exportable reports are critical for bot adoption tracking. Enterprise Administrators should have the ability to control when and what changes are

published to the production bot as proper gate keeping allows enterprises to deliver better user experiences.

Directory sync methods should be utilized for employees, making enrollment as painless as possible.



10 Does the platform provide an administration console for bot admins to manage their bots and their usage?

10.1 Can enterprise admins control deployment requests for any bots built by developers within a company?

Enterprise admins would need to be the gate keepers to any and all deployment requests for bots developed within a company.

10.2 Can admins provision and assign access to a bot to users and groups?

10.3 Can admins manage bots deployments for both internal employee bots




as well as consumer facing bots?

10.4 Does the platform provide enterprises the ability to see analytics and dashboard of bot usage across their user population?

Analytics dashboard should allow enterprises to view their bot usage data using various time slices.

10.5 Can enterprises execute and view a custom analytics/usage report?

10.6 Can enterprises execute and view a custom audit report?

10.7 Can enterprise admins invite and enroll bot users from the admin console?

10.8 Can enterprise admins use directory sync (Active Directory sync) to import and enroll company users?

10.9 Can admins manage users and group privileges from the admin console?

User and Group management should include the ability to manage users and groups.

10.10 Can admins manage security and control policies from the admin console?

Security and control policy and configuration may include single-sign on, password policies, domain management, etc.

10.11 Can admins manage roles and permissions for other admins




within the company?

Bot StoreWhat should the solution provide? The solution should provide instant access to ready-to-use bots that connect directly to enterprise systems and personal apps through a secure virtual store. The platform should allow you to securely distribute custom-built bots to your workers and members of the extended enterprise through your very own enterprise bot store with full administrative control.

Why is this important? It is imperative that users know how and where to access out-of-the-box and custom built bots

for your organization to achieve maximum user adoption. Users should understand the difference between pubic and private bot marketplaces to ensure

they are using and authenticating through the appropriate custom bots. The bot store enables developers and admins to see availability of public bots and quickly

launch the builder tool to customize as needed.

Ref #

Evaluation Criteria


11 Does the platform provide an online store or marketplace for deploying and provisioning bots?

A Bot Store is important because it can make it easy for users to explore, search and find bots that meet their needs

11.1 Do enterprises have their own company specific bot store or marketplace?

Company controlled Bot Stores should only be accessible to company or domain users

11.2 Within the company Bot Store, can bots be assigned to specific users and groups?

This enables users to easily and quickly find bots that are relevant to their job functions

11.3 Can development partners also publish and

For example, can an ISV or system integrator develop and offer their own bots in the public


Ref #

Evaluation Criteria


deploy their own bots in the public ecosystem?

facing bot store?


instructions - kore.ai · web viewalert tasks allow a bot to initiate a conversation with the user...

Documents