TeraVM Architecture

TeraVM system level architecture Architecture Document

Introduction

© Shenick Network Systems - 2 -

TeraVM Architecture

1 Introduction ............................................................................................................................................... 3

2 Client Interfaces ........................................................................................................................................ 4

2.1 Java GUI .............................................................................................................................................................5

2.2 CLI ......................................................................................................................................................................6

2.3 TCL API ..............................................................................................................................................................8

2.4 iTest Plugin .........................................................................................................................................................8

2.5 Web Interface .....................................................................................................................................................9

3 TeraVM Controller .................................................................................................................................. 10

3.1 Management Network ....................................................................................................................................... 10

3.2 PostgreSQL Database ...................................................................................................................................... 11

3.3 Java Server....................................................................................................................................................... 11

3.4 Mongo DB ......................................................................................................................................................... 13

3.5 Statistics SRRD ................................................................................................................................................ 13

3.6 Apache httpd with PHP ..................................................................................................................................... 14

3.7 Configuration Files ............................................................................................................................................ 14

4 TeraVM Test Module ............................................................................................................................... 14

4.1 Communication or “Comms” Network................................................................................................................ 14

4.2 MLIPS (Test Agent) ......................................................................................................................................... 15

4.3 TeraVM-1 vs. TeraVM-2 .................................................................................................................................... 15

4.4 Fedora Guest OS .............................................................................................................................................. 16

5 License Server ........................................................................................................................................ 16

Introduction

© Shenick Network Systems - 3 -

TeraVM Architecture

1 Introduction

This document assumes a working knowledge of how a TeraVM system is used to perform L2 –

L7 testing and how a basic system is configured. For those that are not familiar or would

otherwise like a comprehensive review we suggest reading or at least scanning the product user

guide (also referred to as the “diversifEye User Guide”).

Because we will refer to these 3 items in the document we provide below a brief description of

each. For a more detailed understanding please refer to the product user guide.

1. Test Group: A test group is basically a definition of a test case and is the high level

construct that a TeraVM user begins with when defining test cases. A test group contains

multiple entities such as hosts and applications. Only one instance of a test group can be

run by a user at a given time.

2. Host: In a test group each host represents a network node (e.g. mobile phone, laptop,

set top box etc.) in the emulated network. Basically any network node that has L2 – L3

properties (i.e. MAC address, IP address) is represented in TeraVM as a host.

3. Application: Just as in a real world scenario applications are deployed on hosts (e.g.

mobile phone, laptop). Within a TeraVM system users define applications (e.g. HTTP,

VoIP) as per their test requirements and subsequently assign those defined applications

to one or more hosts which they have previously defined.

Just as many network orientated software applications can be running on a single

networked PC at any one time (e.g. web browser, email client), many applications can

run on the same TeraVM host at the same time.

This document provides a comprehensive architectural overview of the entire TeraVM system.

Every major building block in each subsystem is shown with enough description for a reasonably

technical person to grasp how the building blocks interwork and contribute to the overall system.

There are 4 major subsystems: 1) Client Interfaces, 2) TeraVM Controller 3) TeraVM Test

Modules and 4) TeraVM License Server as shown in Figure 1. Next we will discuss each

subsystem in detail.

Client Interfaces

© Shenick Network Systems - 4 -

Figure 1: TeraVM Overall Architecture

2 Client Interfaces

The client interfaces subsystem provides the various mechanisms by which a human or machine

(i.e. test automation) can interact with a TeraVM system to build and execute test cases. The

client interfaces subsystem only communicates with a TeraVM Controller and does so specifically

using a Java RMI (Remote Method Invocation) interface as shown in Figure 1. RMI is

a Java API that performs the object-oriented equivalent of remote procedure calls (RPC), with

support for direct transfer of serialized Java objects and distributed garbage collection.

Client Interfaces

© Shenick Network Systems - 5 -

Figure 2: Client Interface Options

Below we will describe each component listed in Figure 2.

2.1 Java GUI

The Java based graphical user interface (GUI) is the main vehicle used for human interaction with

TeraVM. The client provides an interface for configuring, executing, viewing and analyzing L2 –

L7 test scenarios and the statistics gathered during the execution of these test scenarios. The

client must be downloaded and installed on a local machine such as a laptop and is referred to as

the “diversifEye Client” as it was originally designed and deployed for interacting with a

diversifEye chassis or appliance.

There is a detailed description of the features and functions of the Java GUI client (“diversifEye

Client”) beginning with chapter 2 of the product user guide (also referred to as the “diversifyEye

User Guide”).

Client Interfaces

© Shenick Network Systems - 6 -

Figure 3: Java GUI Client

2.2 CLI

The command line interface (CLI) is the most often used mechanism to automate and script

interaction with a TeraVM Controller. The CLI is unique to TeraVM but uses a conventional verb-

noun style format. Some example CLI commands are as follows:

Log Out of Cli

$> exit

Configure Cli username and partition

$> cliConfigure

Add a test folder

$> cli addTestGroupFolder "//" <FolderName> <This_is_a_description>

List available tests

$> cli listTestGroups

Client Interfaces

© Shenick Network Systems - 7 -

Create a testgroup in a folder

$> cli addTestGroupFolder

Import a TestGroup

$> cli importTestGroup "//myfolder/ or //" <testname.xml>

Start a TestGroup

$> cli startTestGroup <testname>

Stop a TestGroup

$> cli stopTestGroup <testname>

Delete a TestGroup

$> cli deleteTestGroup <testname>

2.2.1 Perl XML Generator

Shenick does not provide a Perl API per se. When most people hear the term “Perl API” they

think we provide a Perl library of functions which they can call directly from a Perl script. The

expectation is that these library functions will mimic what we can do from the CLI, namely

functions such as starting and stopping tests, and capturing test output in real time.

Instead what Shenick provides is a PERL library which allows a user to define a test case with all

relevant test groups, hosts, applications etc. The defined test cases can then be output in XML

format and read in via the CLI and ultimately become part of the PostgreSQL database which

persistently stores all available test groups. This functionality is very helpful but requires multiple

manuals steps (i.e. export XML, load via CLI) and also does not lend itself to real time (i.e.

start/stop tests) scripting of a TeraVM system.

Client Interfaces

© Shenick Network Systems - 8 -

Figure 4: Perl XML Generation Example

2.3 TCL API

The TCL API is built on top of the Java API and is of course exposed through the TCL scripting

language. Most of the functionality supported by the Java API is exposed via TCL with similar

methods and parameters.

There is not a large customer user base for the TCL API as the CLI and Perl XML Generator are

more feature rich. As a result, minimal engineering effort has been put in to enhancing the TCL

API in recent years.

2.4 iTest Plugin

iTest is an integrated test authoring and execution solution from Spirent Communications

(acquisition of a startup called Fanfare) which provides an SDK to enable 3rd parties to develop

plug-ins for iTest allowing it to access and control their products. This interface is not used by

very many Shenick customers and details of configuration can be found in the “diversifEye User

Guide” chapter 16 entitled “Fanfare Integration”.

Client Interfaces

© Shenick Network Systems - 9 -

2.5 Web Interface

Figure 5: TeraVM Web Interface

The TeraVM web interface serves many different purposes, but it is in some sense the first place

to go after installing a TeraVM Controller. Access is very simple and just involves browsing to the

IP address of the TeraVM Controller. For our example in Figure 5 it would be simply

http://192.168.21.67.

As show in Figure 1 the TeraVM Controller has an Apache web server running in it which of

course hosts the web interface.

Following are some other functions of the web interface

Download Java GUI client

Backup and restore configurations

Upgrade to another version of TeraVM

Download system logs

Install CLI and/or scripting environments (e.g. TCL, PERL, Java API)

Global Settings

The last item, global settings, serves another very significant purpose for the web interface.

TeraVM Controller

© Shenick Network Systems - 10 -

Figure 6: Web Interface Global Settings Page

There are a wide variety of global settings that are made from this page. These include specific

settings for many of the supported applications such as TCP, HTTP, RTSP, and TWAMP as well

as cryptographic, IPv6 and general network settings to name but a few. Some of the settings are

temporary and just added in until the function can be formally introduced in to the Java GUI client.

Other settings are for customer specific implementations and hence are not to be exposed to a

wider audience. In general, adding a setting to the web interface is a quick way to add a capability

for proof of concept or time to market purposes. As shown in Figure 1 all settings that are made

via the web interface are committed to “Configuration Files” in the TeraVM Controller that are

ultimately read by the “Test Execution Server” when loading or executing a test.

3 TeraVM Controller

A significant part of the TeraVM Controller is a Java server which contains a variety of

components including the Test Execution Server which is the main vehicle for test execution and

communication with TeraVM Test Modules and the License Server.

3.1 Management Network

The TCP network between the client interfaces and TeraVM Controller is referred to as the

“management” network because it is the means by which a user “manages” their TeraVM

deployment. This network can have either a private (RFC 1918) or public IP addressing scheme

depending upon how the end user wants to access the TeraVM Controller. The IP address of the

TeraVM Controller

© Shenick Network Systems - 11 -

TeraVM Controller that is used by the Java GUI Client and Web Interface for access to the

Controller is clearly part of the management network.

Figure 7: TeraVM Controller

3.2 PostgreSQL Database

Client interface defined tests are persistently stored in this database. In other words if a user

defines test groups with various hosts and applications via the Java GUI client or CLI the entire

configuration is stored persistently in this database.

Whenever a test is run the “Test Execution Server” pulls the test configuration information (e.g.

hosts, applications etc.) from this database via the persistence layer.

3.3 Java Server

A significant portion of the TeraVM Controller is the Java Server which is often referred to as the

“back-end” because it serves as the back end for all of the client interfaces.

3.3.1 Test Execution Server

The Test Execution Server (TES) is in some sense the brains of the TeraVM Controller. When a

TeraVM user runs a test this triggers the TES to load, create and then execute the test. There are

6 key steps involved in this scenario as follows:

TeraVM Controller

© Shenick Network Systems - 12 -

Step 1: The defined test configuration is read from the PostgreSQL database and additional data

is taken in from relevant configuration files

Step 2: The test configuration is expanded into the run time datastore, which is backed by

MongoDB

Step 3: The licenses required to run the desired test are checked out from the license server

Step 4: Test agent(s) are instantiated on the appropriate test module(s) based on the test

configuration

Step 5: Statistics SRRD is informed that test agent(s) have been instantiated and will be sending

statistics updates directly to SRRD. This action triggers SRRD to establish a TCP connection with

each instantiated test agent (that it was just informed about) in order to receive statistics updates.

Step 6: If thresholds have been set, TES starts the thresholding engine

3.3.2 Thresholding Engine

Thresholding is a powerful TeraVM capability. Here is a brief description of the capability from the

product user guide.

“Thresholding functionality is available via the GUI and CLI and allows an operator to

automatically determine if something has gone wrong during the execution of the test. If an

operator is interested in knowing if, for example, the Mean Get Time of any HTTP Client

download is greater than 1s, a threshold containing this criteria can be applied to all HTTP Clients

and an event will be generated if the criteria is breached. This capability prevents the operator

from having to sift through lots of statistics to see if something has gone wrong.”

The Thresholding Engine receives a threshold filter (e.g. “show me every HTTP client with a

mean get time greater than 1s”) from a client interface (e.g. Java GUI or CLI) and passes it to the

“Statistics SRRD” system. This triggers SRRD to begin sifting through all the statistics returned

from the relevant test agents and filter out those that meet the filter criteria. This match data is

then sent to the Thresholding Engine and in turn forwarded to the client interface subsystem for

display or output.

3.3.3 Service Layer

The service layer is the private development API to the Java Server. It exposes all test

management functionality including test creation/editing, and test execution. It also exposes

secondary functionality around tests such as packet captures and statistics. The server layer is

exposed to the client interfaces via RMI.

TeraVM Controller

© Shenick Network Systems - 13 -

3.3.4 Manager Layer

The manager layer is an intermediate layer between the service layer and the persistence layer.

A manager exists for each test entity type. It is here that any business logic is implemented. It is

generally a light layer with very little code.

3.3.5 Persistence Layer

The persistence layer is an abstraction layer between the database and the rest of the server. It

takes care of the persistence of entities in either the static test database (PostgreSQL) and the

run time test database (MongoDb)

3.4 Mongo DB

This is a run time database that is utilized during test execution to persist the state of the running

entities (e.g. hosts, applications). It also persists run time assigned properties such as

dynamically assigned IPs (i.e. via DHCP), MTU, and gateway address. During test execution

certain information about the test is pulled from Mongo DB to update client interfaces.

When a test is stopped all related information is deleted from Mongo DB. This is therefore a

temporary, run time database.

Mongo DB is particularly useful for scaled entity testing because a large amount of information

must be manipulated to emulate tens of thousands of entities or more.

3.5 Statistics SRRD

This is the statistics engine which contains a database that is referred to as Simple Round Robin

Database or SRRD. Simply put this is where all test related statistics or metrics are stored and

retrieved. The entire thresholding feature depends upon SRRD to keep track of statistics which

are collected during test execution. For example if a user is interested in knowing if the “Mean

Get Time” of any HTTP Client download is greater than 1s, a threshold containing this criteria can

be applied to all HTTP Clients and an event will be generated if the criteria is breached.

SRRD establishes a separate TCP connection to each TeraVM Test Agent and this is the

mechanism by which it receives constant, real time statistics or metrics updates from each test

module. The Test Execution Server tells SRRD that a given test agent has been instantiated and

this triggers SRRD to establish a TCP connection with that test agent for statistics collection

purposes.

TeraVM Test Module

© Shenick Network Systems - 14 -

3.6 Apache httpd with PHP

This is an Apache web server daemon (httpd) with the PHP module loaded. PHP is used to

dynamically generate the web interface pages and to write to specific configuration files.

3.7 Configuration Files

These configuration files contain information that is set in the Web Interface for example global

settings.

Here are some examples of the configuration files that are updated:

/opt/shenick/diversifEye/master-server/MasterServerX.props

/etc/shenick/partitions.cfg

/etc/shenick/UI-defaults.props

/etc/mlips.conf

4 TeraVM Test Module

TeraVM Test Modules are the virtual machines that support the entities which actually generate

the traffic which is used to test a device or network under test. In some sense the test module is

the reason why a customer buys TeraVM and hence when we generically refer to “TeraVM” we

are referring to the actual test modules and not the controller or anything else. A “TeraVM

Controller” is named that way because it controls a “TeraVM” or in other words it controls one or

more TeraVM Test Modules.

Figure 7 shows three TeraVM Test Modules and how they interface with a TeraVM Controller. In

a typical deployment there are many more than three TeraVM Test Modules.

4.1 Communication or “Comms” Network

The TCP network between the TeraVM Controller and TeraVM test modules is often referred to

as the “communication” or “comms” network. Simply put the “comms” network is a private

network between all TeraVM test modules and the TeraVM Controller they are controlled by. A

given TeraVM test module can only be associated with one TeraVM Controller. The “comms”

network should have a private (RFC 1918) IP addressing scheme (e.g. 192.168.0.0/16) to avoid

any conflict with externally routable IPs. By default the subnet of 192.168.240.0/20 is used for

“comms” but this is user configurable. In the default set up IP address 192.168.240.1 is assigned

to the Controller, 192.168.240.2 is reserved (for potential future use) and the first TeraVM test

module will use IP 192.168.240.3 and so on until 192.168.255.254. This provides for a maximum

of 4,091 TeraVM test modules that can be controlled by a single TeraVM Controller.

TeraVM Test Module

© Shenick Network Systems - 15 -

4.2 MLIPS (Test Agent)

The traffic used to test a device or network under test is generated by one or more software

instances running inside a Test Module. These software instances are called the test agent. The

TeraVM test agent is also known as MLIPs (Multiple Lightweight TCP/IP Stack Simulator)

MLIPS is without a doubt Shenick’s “crown jewels” and the holder of much of Shenick’s

intellectual property.

Simply put, the MLIPS application is used by a TeraVM system to generate Layer 4-7 TCP and

UDP traffic over IP and provide other functionality to facilitate the running of TeraVM tests.

MLIPS is sometimes referred to as a “packet processor” or “PP”.

MLIPS has its own highly customized IP stack that is based upon New Reno.

We sometimes generically refer to MLIPS as the “test agent” or in other words the agent that is

generating the tests.

Although Figure 7 shows the test module with only one instance of MLIPS there is no reason why

a single test module cannot contain multiple instances of MLIPS or in other words multiple test

agents.

4.3 TeraVM-1 vs. TeraVM-2

We often refer to a TeraVM-2 or TVM-2 for short. This is a single test module or VM appliance

(TeraVM application + Guest OS) that utilizes a total of 2 CPU cores. MLIPS runs on one CPU

core and the other is used for interrupt processing. By contrast a TeraVM-1 (TVM-1) uses only

one core (running MLIPS) and has no interrupt core.

TVM-1 is useful for test scenarios where we are willing to sacrifice performance and deterministic

results in favor of testing with twice as many VMs and vNICs for the same amount of CPU

resources. An example would be vSwitch testing where the task is to send small amounts of data

across thousands of vNICs.

In the future Shenick might build a TVM-5 or TVM-8 which could contain multiple instances of

MLIPS each running on their own CPU core and some number of extra CPU cores reserved for

interrupt processing.

License Server

© Shenick Network Systems - 16 -

4.4 Fedora Guest OS

A TeraVM Test Module is in fact a virtual appliance by which we mean it is a package that

contains both the TeraVM software as well as a specific guest operating system. This package

can be deployed on supported hypervisors such as ESXi, Hyper-V and KVM.

In the case of a TeraVM Test Module the underlying guest OS is Fedora. Fedora is a well-known

open source Linux operating system that is supported by the Fedora project and Red Hat

Corporation. One of Fedora's main objectives is not only to contain software distributed under

a free and open source license, but also to be on the leading edge of such technologies. Fedora

developers prefer to make upstream changes instead of applying fixes specifically for Fedora—

this ensures that their updates are available to all Linux distributions. Red Hat Corporation

actually branches their Red Hat Enterprise Linux (RHEL) releases from versions of Fedora.

Fedora became the TeraVM Test Module guest OS in release 10.3. Prior to that the guest OS

was BusyBox which is a very stripped down and hence lightweight Linux OS. One of the main

reasons for switching to Fedora was so we could have a “mainstream” guest OS with support for

all common test and debugging tools. BusyBox apparently had limited tools support.

5 License Server

The TeraVM License Server (TMV-LS) is the only subsystem of a TeraVM system that is not

developed by Shenick. TVM-LS is developed by Reprise Software which includes the team at

Reprise that created the most widely-deployed license manager, FLEXlm. Shenick purchases the

product directly from Reprise and provides it to TeraVM customers.

TVM-LS cannot be administered from any of the available TeraVM client interfaces because it is

a third party product and has not been integrated from an administration perspective. All TVM-LS

administration is done from an independent web interface shown in Figure 8.

License Server

© Shenick Network Systems - 17 -

Figure 8: TVM-LS Administrative Web Interface (Reprise Software)

As shown in Figure 7 the only direct interaction from TVM-LS is via a TCP connection to the “Test

Execution Server”.

A TVM-LS licenses file is a human readable text file that contains signatures to prevent

unauthorized editing.

Following is a sample snippet from a license file:

-----------------------------------------------

HOST localhost 005056875de8

ISV shenick

LICENSE shenick rlm_server_enable_vm 1.0 permanent 1 _ck=87d6fc7a6a

sig="60P04504ARQFQ855M5C5HB9YN0Q5NSWH9C89NVR22GMAMRDQY8QG0VPYQP8J0V

6J0T3PYBYH9C"

LICENSE shenick tvmc 1.0 permanent 1000 timezone=ffffff _ck=f86670c509

sig="60Q04580RWCT94HKANRXHJ7WBMGA4NB2S6V1XAPE08AG16VW580D2EDGU0VMC3

WV9STFRQXCT94QA"

License Server

© Shenick Network Systems - 18 -

LICENSE shenick tvm_cores 1.0 permanent 1000 timezone=ffffff _ck=106670089c

sig="60P0451E358NPEASNYCHMRNU7AJE4K01F9YMVG822HDWVTD68V9PTT4D0CR3CW6

T9C6CX43PA4"

LICENSE shenick esxi_5.0_tvm_cores 1.0 permanent 1000 _ck=1a66fcec8f

sig"60PG4580MT502QG84UU909ST1YX860VD0FM6Y7PN08A30PW85ABDJX96FAGTW0G

G0MT709047BA0"

-----------------------------------------------------------------------

TeraVM Architecture

© Shenick Network Systems. All rights reserved. Shenick Network Systems, Shenick, the Shenick logo, TeraVM, and

diversifEye are registered trademarks of Shenick Network Systems, Inc. All other products and companies may be

trademarks of their respective owners.

www.shenick.com

info@shenick.com

NORTH AMERICA

1900 McCarthy Blvd

Suite 301

Milpitas, CA 95035

TEL : 408-385-7630

EUROPE

Brook House, Corrig Avenue

Dun Laoghaire, Dublin, Ireland

TEL : +353-1-236-7002