selenio next installation troubleshoot guide v1 edition b manual

Delivering the Moment

SelenioNext™ Dense Multiscreen Transcoding Broadcast Management System (BMS)

Software Version 1.0 July 2013 Edition B 175-100509-00

Installation and Troubleshooting Guide

© 2014 Imagine Communications Corp. Proprietary and Confidential

Publication Information © 2014 Imagine Communications Corp. Proprietary and Confidential.

Imagine Communications considers this document and its contents to be proprietary and confidential. Except for making a reasonable number of copies for your own internal use, you may not reproduce this publication, or any part thereof, in any form, by any method, for any purpose, or in any language other than English without the written consent of Imagine Communications. All others uses are illegal.

This publication is designed to assist in the use of the product as it exists on the date of publication of this manual, and may not reflect the product at the current time or an unknown time in the future. This publication does not in any way warrant description accuracy or guarantee the use for the product to which it refers.

Imagine Communications reserves the right, without notice to make such changes in equipment, design, specifications, components, or documentation as progress may warrant to improve the performance of the product.

Trademarks Selenio™ and SelenioNext™ are trademarks of Imagine Communications or its subsidiaries. Microsoft® and Windows® are registered trademarks of Microsoft Corporation. All other trademarks and trade names are the property of their respective companies.

Contact Information Imagine Communications has office locations around the world. For locations and contact information see: http://www.imaginecommunications.com/contact-us/

Support Contact Information For support contact information see:

Support Contacts: http://www.imaginecommunications.com/services/technical-support/

eCustomer Portal: http://support.imaginecommunications.com

http://www.imaginecommunications.com/services/technical-support/

http://support.imaginecommunications.com/

SelenioNext™

Dense Multiscreen Transcoding

Installation and Troubleshooting Guide

Software Version 1.0

July 2013 Edition B: June 2014

About this Document

SelenioNext Broadcast Management System V.1.0 2 Installation and Troubleshooting Guide

Publication Information

© 2014 Imagine Communications Corp. Proprietary and Confidential.

Imagine Communications considers this document and its contents to be proprietary and confidential. Except for making a reasonable number of copies for your own internal use, you may not reproduce this publication, or any part thereof, in any form, by any method, for any purpose, or in any language other than English without the written consent of Imagine Communications. All others uses are illegal.

This publication is designed to assist in the use of the product as it exists on the date of publication of this manual, and may not reflect the product at the current time or an unknown time in the future. This publication does not in any way warrant description accuracy or guarantee the use for the product to which it refers.

Imagine Communications reserves the right, without notice to make such changes in equipment, design, specifications, components, or documentation as progress may warrant to improve the performance of the product.

Trademarks

Selenio™ and SelenioNext™ are trademarks of Imagine Communications or its subsidiaries. Microsoft® and Windows® are registered trademarks of Microsoft Corporation. All other trademarks and trade names are the property of their respective companies.

Contact Information

Imagine Communications has office locations around the world. For locations and contact information see: http://www.imaginecommunications.com/contact-us/

Support Contact Information

For support contact information see:

Support Contacts: http://www.imaginecommunications.com/services/technical-support/

eCustomer Portal: http://support.imaginecommunications.com

http://www.imaginecommunications.com/services/technical-support/

http://support.imaginecommunications.com/

About this Document

SelenioNext 3 Installation and Troubleshooting Guide

About this Document This document contains instructions for installing the software components of SelenioNext (processor, packager, and BMS) as well as guidelines for troubleshooting. Chapters and Their Contents

1 Introduction Key components and acronyms; Contact information

Pg. 6

2 Prerequisites and General Guidelines

Summary of connectors; guidelines and constraints

Pg. 8

3 Hardware Setup Setup of the Dell R720 2 RU server Pg. 9

4 Software Installation Basic installation procedure Pg. 18

5 BMS Configuration Setting essential BMS parameters and running the configuration command

Pg. 20

6 SelenioNext Setup Initial and repeat installations of Static or HA servers

Pg. 26

7 Post-Installation Procedures Verifying NTP, SNMP Support, Time Zones; Database Replication

Pg. 32

8 Troubleshooting Tests, log files and high-availability status Pg. 45

9 Maintenance Procedures Logins, access, errors, resets, shutdown, and reboot

Pg. 48

A Appendix: Creation of a System Definition File

Creation of a System Definition File Pg. 50

Style Conventions

Convention Used for

Verdana Regular text

Arial Bold Names of commands and other elements of the user interface

Arial Italics Special terms, the first time they appear

Monospace Text entered by a user, values of parameters, and usage examples

Notes, which offer an additional explanation or a hint on how to overcome a common problem

Warnings of potential problems

Table of Contents


Table of Contents ABOUT THIS DOCUMENT ............................................................................. 3

TABLE OF CONTENTS .................................................................................. 4

1 INTRODUCTION .................................................................................... 6

1.1 ACRONYMS .........................................................................................6

2 PREREQUISITES AND GENERAL GUIDELINES........................................ 8

2.1 INSTALLATION FLOW ..............................................................................8

3 HARDWARE SETUP ................................................................................ 9

3.1 DELL R720 2 RU SERVER .......................................................................9

Installing the Next Cards in the Dell R720 Server ..................................9

Configuring the Dell R720 Server ...................................................... 16

3.2 EXTERNAL POWER FOR THE NEXT CARDS ..................................................... 16

4 SOFTWARE INSTALLATION ................................................................. 18

4.1 NEXT SOFTWARE INSTALLATION .............................................................. 18

4.2 BMS SOFTWARE INSTALLATION .............................................................. 19

5 BMS CONFIGURATION ........................................................................ 20

5.1 SETTING THE BMS IP AND GATEWAY ........................................................ 20

5.2 SETTING THE BMS HOSTNAME ................................................................ 20

5.3 RUNNING BMS-SETUP ......................................................................... 21

6 NEXT SETUP ........................................................................................ 26

6.1 INITIAL SETUP OF A STATIC (NON-HA) NEXT SERVER .................................... 26

Recovering Ethernet Interfaces ......................................................... 28

Initial Setup of IBS HA Cluster .......................................................... 28

6.2 REPEATED SETUP OF A SERVER (STATIC OR HA) ........................................... 29

7 POST-INSTALLATION PROCEDURES .................................................... 32

7.1 VERIFYING NTP ................................................................................. 32

7.2 SNMP SUPPORT ................................................................................ 33

7.3 MYSTRO MIB AND SELENIONEXT SNMP AGENT SUPPORT ............................... 34

7.4 TIME ZONE CHANGE ............................................................................ 35

7.5 ADDITIONAL BMS SETTINGS .................................................................. 36

7.6 SETTING UP DATABASE REPLICATION FOR BMS ............................................ 38

Master Configuration ....................................................................... 39

Table of Contents


Slave Configuration ......................................................................... 41

Recovery from a BMS Server Hardware Failure (Hot Standby) .............. 42

Recovery from a BMS Server Hardware Failure (Cold Standby) ............. 43

8 TROUBLESHOOTING ............................................................................ 45

8.1 VERIFYING NEXT OUTPUT STREAM ............................................................ 45

8.2 VERIFYING VIDEO SOURCES ................................................................... 45

8.3 TESTING SELENIONEXT EQUIPMENT OPERATION ........................................... 46

8.4 CHECKING LOG FILES .......................................................................... 46

8.5 VERIFYING STATUS OF HIGH AVAILABILITY .................................................. 46

9 MAINTENANCE PROCEDURES .............................................................. 48

9.1 LOGGING INTO A SERVER ...................................................................... 48

9.2 SHUTTING DOWN A SERVER ................................................................... 48

9.3 REBOOTING A SERVER .......................................................................... 48

APPENDIX A – CREATION OF A SYSTEM DEFINITION FILE ....................... 50

Introduction


1 Introduction

SelenioNext™ makes it possible for video consumer to access programming across an array of mobile and fixed devices and networks. For service providers, SelenioNext maximizes the quality, quantity, and footprint required in a multiscreen, TV Everywhere environment.

SelenioNext is an all-in-one TV Everywhere solution that replaces an entire headend of video processing in a single commercial-off-the-shelf (COTS) server platform. The SelenioNext solution is Enabling PersonalizedTV® by allowing service providers to ingest precompressed services, and then transcode, package, encrypt and stream multiscreen, multi-device video. Designed to provide live programming to mobile and connected screens, SelenioNext delivers a PersonalizedTV® experience, enabling video consumers to view and interact with content on the device of their choice at any time or location.

SelenioNext is available in 1RU, 2RU, or 10RU blade systems that fit into optimal form factors for all online video applications—from providing several IPTV streams, to thousands of multiscreen transcodes. Using advanced Adaptive Bit Rate (ABR) technology, the system supports up to 320 HD ABR or 320 SD ABR profiles per 2RU server, and scales up to any number of profiles per video program. SelenioNext also features an onboard broadcast management system for control and visibility into network resource optimization.

The SelenioNext software package holds several key components that are introduced in this document. A SelenioNext processor creates multiple output SPTS streams and multiple ABR groups. Additionally, it can package those streams in various formats.

The key components of SelenioNext can be installed in any sequence. However, several prerequisites must be met before installation: see Chapter 2.

1.1 Acronyms

You should be familiar with the following terms when installing the system.

Acronym Meaning

ICE Interchangeable Compressed Elements

IBS ICE Broadcast System

BMS Broadcast Management System

iLO Integrated Lights-Out Management (HP c7000)

AVP Advanced Video Processor

Introduction


MIB Management Information Base

SNMP Simple Network Management Protocol

NTP Network Time Protocol

Prerequisites and General Guidelines


2 Prerequisites and General Guidelines

As a prerequisite to the installation process, management network interfaces must be configured prior to installation of SelenioNext software.

SelenioNext version 1.0 works on Linux v3.0 (internal release number).

2.1 Installation Flow

The installation process includes the following major tasks:

1. Install hardware, including Linux, on the hardware platform:

Dell PowerEdge R720 (see Section 3.1 Dell R720 2 RU Server on page 9)

2. Install SelenioNext software (see Section 4.1 SelenioNext Software Installation on page 18).

3. Install BMS software (see Section 4.2 BMS Software Installation on page 19).

4. Configure BMS (see Chapter 5 BMS Configuration on page 20).

5. Set up SelenioNext (see Chapter 6 SelenioNext Setup on page 26).

Hardware Setup


3 Hardware Setup

3.1 Dell R720 2 RU Server

Installing the SelenioNext Cards in the Dell R720 Server

You can install up to two SelenioNext cards in each of the left and right chambers (orientation when looking from the back side) of the Dell R720 server, as described in the procedures in the following sections. If you want to install the maximum allowable four SelenioNext cards, you should start the installation from the left chamber.

Ensure that all power is removed from the server before removing or installing cards. You must disconnect the power cables before performing this procedure.

Perform electrostatic discharge and antistatic prevention measures prior to handling the server.

Installing SelenioNext cards in the left chamber

To install the SelenioNext cards in the left chamber of the Dell R720 server:

1. After removing the server cover, disconnect and remove riser 3.

Removing Riser 3

2. Make sure the two light blue card fasteners are pulled out.

Hardware Setup


Figure 1 – Card Fasteners Pulled Out

3. Connect the power cable to riser 2.

Figure 2 – Connecting Power Cable to Riser 2

4. Connect the power cable to the first card, and then insert the card into the bottom slot of riser 2.

Hardware Setup


Figure 3 – Connecting First Card

5. Insert the second card into the top slot of riser 2 without connecting the power cable to the card.

Figure 4 - Installing Second Card

6. Connect the power cable to the second card.

Hardware Setup


Figure 5 – Connecting Power Cable to Second Card

7. Lock the two light blue card fasteners that were pulled out in step 2.

Figure 6 – Locking Cards into Place

8. Reconnect riser 3 to the motherboard.

Hardware Setup


Figure 7 – Reconnecting Riser 3 to Motherboard

9. Lock the left chamber cards by clicking the blue button.

Figure 8 – Locking Cards in Left Chamber

Installing SelenioNext cards in the right chamber

To install the SelenioNext cards in the right chamber of the Dell R720 server:

1. Make sure the two light blue card fasteners are pulled out, and connect the power cable to riser 3.

Hardware Setup


Figure 9 – Connecting Power Cable to Riser 3

2. After connecting the power cable to the first card, insert the card into the bottom slot of riser 3.

Figure 10 – Connecting First Card

3. After connecting the power cable to the second card, insert the card into the top slot of riser 3.

Hardware Setup


Figure 11 - Connecting Second Card

4. Lock the two light blue card fasteners that were pulled out in step 1.

Figure 12 – Locking Cards into Place

5. Lock the left chamber cards by clicking the blue button.

Hardware Setup


Figure 13 – Locking Cards in Right Chamber

Configuring the Dell R720 Server

To configure the Dell R720 server:

1. Boot into BIOS (press DEL when prompted).

2. Make sure the settings are the same as listed below, and save and exit at the end of the process.

3. In iDRAC Settings, in Thermal menu, change Thermal Base Algorithm to Maximum performance, and change Fan Speed offset to High…

4. Under Miscellaneous Settings, disable F1/F2 Prompt on Error, and set Keyboard Error to Do Not Report.

3.2 External power for the SelenioNext cards

The SelenioNext cards need to receive external power through a 6-pin PCI Express power cable. The connector on the board can be seen in the following figure; only one of the two cards needs to be connected:

Hardware Setup


Figure 14 - Connectors

The Dell R720 server requires a Y cable for each pair of SelenioNext cards.

More information can be obtained in the following site:

http://www.playtool.com/pages/psuconnectors/connectors.html#pciexpress

http://www.playtool.com/pages/psuconnectors/connectors.html#pciexpress

Software Installation


4 Software Installation

Before starting this section please make sure you have the latest version of Linux installed, along with all the latest dependencies.

Software installation includes installation of SelenioNext (including co-located packager) and BMS.

4.1 SelenioNext Software Installation

The installation of Next version 1.0 is divided into several parts; the various software components are divided into three RPM packages in total (where XX is the most recent build number):

next-1.0.0-182.x86_64.rpm: contains major binaries and setup files

next-common-1.0.0-182.x86_64.rpm: contains common binaries and files

bms-next-1.0.0-182.x86_64.rpm: contains BMS application server files and scripts

In order to install Next v1.0 for the first time, please perform the server installation from scratch using the Tarball file. For further instructions refer to chapter 11 of the SelenioNext Configuration Manual.

In case of an upgrade, place all three packages in a single folder and issue the following command from that folder (using root user):

Rpm –Uvh *.rpm (installing all RPMs that are located in the folder)

Software Installation


4.2 BMS Software Installation

BMS installation is separated from SelenioNext SW installation and requires only one package.

Copy RPM into destination folder and issue:

rpm –Uvh bms-next-1.0.0-182.x86_64.rpm

During the installation process, you are provided with valuable information such as server vendor, product name and Bios version. These help avoid installation/racking problems.

BMS Configuration


5 BMS Configuration

Configuration of BMS involves:

Setting the BMS IP and Gateway

Setting the BMS Hostname

Running BMS-setup

5.1 Setting the BMS IP and Gateway

Open the file /etc/sysconfig/network-scripts/ifcfg-eth2, and verify the following settings:

IPADDR=a.b.c.d

NETMASK=255.255.x.x

GATEWAY=x.x.x.x

Onboot=yes

BOOTPROTO=static

If the server is not connected to the data network it is recommended to edit the file /etc/sysconfig/network-scripts/ifcfg-eth0, changing this line:

Onboot=yes

to

Onboot=no

5.2 Setting the BMS Hostname

Open the file /etc/sysconfig/network, and change the following setting:

HOSTNAME=<name>

HOSTNAME must start with an alphabetical letter. For example: Server-100 is legal, while 103_20 is illegal.

The hostname is valid only after a reboot command is issued to the server (recommended).

Hostnames containing the underscore character (“_”) are not supported.

BMS Configuration


5.3 Running BMS-setup

Use the following script for first-time installation as well as for upgrades.

All binaries and scripts are already in your PATH; there is no need to change the PATH to look for them.

To complete BMS configuration:

1. Start BMS setup by typing:

bms-setup.sh

You are prompted for the BMS hostname, BMS IP address, and NTP IP.

2. Press <Enter> for the default BMS hostname (206-50 in the example above) and press <Enter> once again for the default IP address of the BMS. Enter the NTP IP or press <Enter> to use the current default.

BMS hostname must be resolvable to assure proper operation of the BMS server.

The following screen appears.

BMS Configuration


3. To the question: "Is this an all-in-one configuration (currently no)? [y/N)", type the response y if the BMS and IBS are running on the same server. Otherwise, press <Enter> (for the default answer No).

During the bms-setup run, you are asked whether you want to wipe out the database. Do so only as part of a new installation on a machine. If you do so in any other case (for example, between upgrades) all saved information will be lost.

4. bms-setup.sh first resolves the server hostname. Afterwards, the user is prompted to specify whether the system should set up a SECTAMUS trap aggregator.

BMS Configuration


5. The BMS can aggregate all traps and send them to a user-defined IP. This IP must be resolvable. This adds an entry in the format “x.x.x.x compass” to /etc/hosts.

BMS Configuration


After adding trap-aggregator IP(s), the following question appears:

"Do you want to start the BMS services right now? (may take a while) [Y,n]:"

6. Press <Enter> to start the processes, and wait until the status “Finished” appears. (This takes a few minutes.)

When the process is complete, the following message appears:

"IBS Management setup is now complete"

BMS Configuration


SelenioNext Setup


6 SelenioNext Setup

Procedures for both initial and subsequent setup are described below.

6.1 Initial Setup of a Static (non-HA) Next Server

To set up a static SelenioNext server for the first time:

1. Run ibs-setup. (Note that ibs-setup remembers the last user entries).

Select the host type from the 4 options (see screenshot above).

2. Enter data as requested during the setup process:

SelenioNext Setup


SelenioNext Setup


The setup takes a few moments: Ethernet interfaces are listed, the route table is displayed, and the “finished running install script” notification appears. The user is returned to the command shell:

Defining a server’s role is done in the BMS, in the Auto Discovery dialog. Please refer to the SelenioNext BMS Configuration manual for more details.

Recovering Ethernet Interfaces

When running ibs-setup, the user is asked about restoring the interfaces with the user’s answers from previous runs of the command.

Initial Setup of IBS HA Cluster

The assumption is that the management stub IPs of all cluster machines are configured in advance. Please refer to Setting the BMS IP and Gateway on page 20.

SelenioNext Setup


To set up an HA cluster for the first time:

1. Create a system file describing the HA cluster, and place it on one of the machines in the cluster. For instructions on how to create the file, see Appendix A.

2. Run: ibs-setup-all --system <system file>

The /etc/hosts file on the machine running ibs-setup-all must be fully populated with data for all servers in the cluster.

3. When prompted, provide passwords for the qoduser and root accounts.

The script will configure the whole cluster, as defined by the settings file. During the process, you will be notified about each server being configured, and the status of the configuration.

6.2 Repeated Setup of a Server (Static or HA) During the setup process, a cache of the configuration is kept, and reconfiguring the server after a software upgrade is a straightforward procedure.

To repeat setup (e.g., after a software upgrade):

1. Run ibs-setup.

2. When asked whether to reuse (restore) the cached configuration of the server, press <Enter> to reply Yes.

SelenioNext Setup


SelenioNext Setup


If you do not wish to reuse the configuration, type anything other than yes or y, press <Enter>, and proceed to specify the various configuration parameters as described earlier.

When installation is complete, the “finished running install script” message appears:

If a network interface modification is needed, and reuse of the cache was specified, ibs-setup will fail. In order to force the network modifications, type the following: ibs-setup –r –-modify-net If a setup is needed without restarting network service, type the following: ibs-setup –rn

Post-Installation Procedures


7 Post-Installation Procedures

The following procedures can be done after IBS has been installed:

Verifying NTP

SNMP Support

Mystro MIB and SelenioNext SNMP Agent Support

Time Zone Change

Additional BMS Settings

Setting Up Database Replication for BMS

7.1 Verifying NTP

Issue the following command (after running ibsinstall) to check that the NTP is working properly.

root@host etc]# ntpq –p remote refid st t when poll reach delay offset jitter ============================================================================== *imagine-ntp.ima 192.114.62.249 3 u 3 64 377 0.093 1.044 0.234 LOCAL(0) .LOCL. 10 l 4 64 377 0.000 0.000 0.001

If imagine-ntp cannot be found, do the following:

1. Place the following line in /etc/hosts:

xxx.xxx.xxx.xxx imagine-ntp

2. Make sure that the IP is pingable.

To check that the NTP server is functional:

1. Stop NTPD by typing:

service ntpd stop

2. Issue the command:

ntpdate imagine-ntp

If this command is successful, a message such as the following appears:



28 Apr 07:36:36 ntpdate[17307]: adjust time server xxx.xxx.xxx.xxx offset -0.010892 sec

Otherwise, one of the following errors may appear:

28 Apr 07:37:18 ntpdate[17313]: no server suitable for synchronization found

(The server is pingable but not an NTP server.)

OR

Error: Name or service not known

28 Apr 07:37:04 ntpdate[17310]: can't find host imagine-ntp2

(The server is not pingable.)

3. Once done, type the following command to restart NTP:

service ntpd start

7.2 SNMP Support

SelenioNext software support sends SNMP Trap notifications to designated places of your choice. The SelenioNext processor looks for a special file that contains the IP addresses of all SNMP Trap listeners that you chose.



The filename is:

/etc/ibs/receivers

and the format is straightforward:

<ip-address>[:port]

For example:

172.16.0.140 192.168.10.165:1660 10.1.1.1:30000

Upon startup, the qod-mcs and qod-ivp read this file and send a trap to:

172.16.0.140 (using the default port: 162)

192.168.10.165 (using port 1660)

10.1.1.1 (using port 30000)

If the file does not exist, traps will not be configured, until configuration is done using either the qodcli utility (for a standby server in an HA cluster) or BMS (for all other servers).

You are advised to edit the receivers file of the standby server before installation. If you edit this file after the installation is complete, you must restart the relevant process. The BMS server’s IP is automatically added to the receivers list.

Please copy the SelenioNext MIB file that can be found at /usr/share/ibs/snmp/mibs/IMAGINE-MIB.my to your SNMP trap receiver.

7.3 Mystro MIB and SelenioNext SNMP Agent Support

To enable Mystro MIB support:

1. If this is the first time Mystro MIB support is installed, make sure this is a fresh install. This applies to both BMS and SelenioNext processor. The BMS DB is maintained; however, it is recommended that you export it as an extra precaution.

2. After installation, if necessary, re-configure qod.conf in the SelenioNext server (for VLANID tagging).

3. In /usr/share/ibs/PRM/conf/prm.properties on the SelenioNext processor, set “programTraps.twc=on”. This requires a PRM restart; therefore, do this before ibs-setup-host.



There is no need to configure the trap receiver in the BMS GUI or IP forwarding on the BMS.

4. Configure the Mystro server in BMS snmptrapd.conf – “forward .1.3.6.1.4.1.14373 SECTAMUS:161”

5. snmpd and snmptrapd need to be running on BMS; snmpd needs to be running on the SelenioNext processor.

Additional settings to support Mystro MIB are located in the section Additional BMS Settings.

To enable SelenioNext SNMP agent, refer to the section Additional BMS Settings. This will enable the operator to perform the following commands:

Failover next: processor #n to standby

Fail primary input – forcing secondary input

Mux server reconcile

Output TS reconcile

Output program reconcile

IBP server reconcile

7.4 Time Zone Change

The Linux running on SelenioNext (all versions) time zone is defaulted to the US/Eastern time zone (i.e., New York). If you wish to change this, do the following (for example):

[root@host root]# ln –s /usr/share/zoneinfo/US/Mountain /etc/localtime

This will set the time zone of the machine to Mountain time.

The U.S. spans 12 time zones:

Alaska

Aleutian

Arizona

Central

Eastern

East-Indiana



Hawaii

Indiana-Starke

Michigan

Mountain

Pacific

Samoa

7.5 Additional BMS Settings

The following properties should be added/updated in the bms.properties file on the BMS machine (located under $JBOSS_HOME/server/default/conf).

The values of the properties can be adjusted according to the system requirements; however, ranges should be configured properly.

Note: changing any of the parameters listed below requires the creation of a new database.

USE_IMP_SSM=true

This parameter defines the method of IP allocation for IMP messages.

If the flag is set to false, allocate a unique IP and port per output TS.

If the flag is set to true, allocate a unique IP per output program/output ghost, and a unique port per output TS.

IMP_IP_POOL_NUMBER=1

This parameter determines whether one or two IMP pools will be used. Default value is 1. If the value is set to 2, two IMP pools are used; one for main mux and one for protect mux (in this case, ALLOCATED_IP_PROTECT must be defined).

ALLOCATED_IP_MIN_RANGE=230.1.1.1 and

ALLOCATED_IP_MAX_RANGE=230.1.2.50

This is the range of multicast IP addresses defined for multicast IMP messages. If IMP_IP_POOL_NUMBER=2, this range will be used for the main mux. If IMP_IP_POOL_NUMBER=1 or not defined, this range will be used for all multicast IMP messages.

ALLOCATED_IP_PROTECT_MIN_RANGE=233.1.1.1 and



ALLOCATED_IP_PROTECT_MAX_RANGE=233.1.2.50

This is the second range of multicast IP addresses defined for multicast IMP messages. Only if IMP_IP_POOL_NUMBER=2 this range will be used for the protect mux.

ALLOCATED_PORT_MIN_RANGE=10000 and

ALLOCATED_PORT_MAX_RANGE=10300

This is the range of ports used for IMP and rate control messages.

ALLOCATED_LOCAL_AVP_MULTICAST_IP=230.100.1.0

This is the multicast IP defined for multicast IMP messages between the input and AVP.

SDI_SERVERS_IP=

This is a list (comma separated) of server IPs that have SDI inputs.

USE_FLAT_ORIGIN_HIERARCHY=false

This is a list (comma separated) of server IPs that have SDI inputs. If set to true, ICE streamer generates output files under the origin base URL, without folders.

Note: Running bms-setup –p allows the user to edit the following BMS properties: • IMP_IP_POOL_NUMBER • ALLOCATED_IP_MIN_RANGE • ALLOCATED_IP_MAX_RANGE • ALLOCATED_IP_PROTECT_MIN_RANGE • ALLOCATED_IP_PROTECT_MAX_RANGE • ALLOCATED_PORT_MIN_RANGE • ALLOCATED_PORT_MAX_RANGE

USER_NETWORK_ID=1

This is the original network ID field in IBS generated SDT.

PROGRAM_POLLING_TIME_DEFAULT=1000

PROGRAM_BITRATE_DIFF_THRESHOLD_DEFAULT=50

Input redundancy default parameters: polling time in milliseconds, threshold in %

EXPORT_FOLDER=/var/opt



The location on the BMS server where the exported BMS DB file is written

UPLOAD_FOLDER=/tmp

The location on the BMS server where the imported BMS DB file is written. This setting is used internally by BMS and should not be modified.

OUTPUT_TS_MAX_SCORE=16

Maximum resources on output TS

OUTPUT_TS_HD_SCORE = 4

The weight of an HD program in the output TS

OUTPUT_TS_SD_SCORE = 1

The weight of an SD program in the output TS

MYSTRO_SNMP_AGENT_SUPPORT = false

Change to true to support Mystro SNMP MIB.

IMAGINE_SNMP_AGENT_SUPPORT = false

Change to true to support SelenioNext SNMP MIB.

IMAGINE_SNMP_AGENT_READ_COMMUNITY = public

IMAGINE_SNMP_AGENT_WRITE_COMMUNITY = public

SNMP community

IMAGINE_SNMP_AGENT_SOURCE_FILTER =

List of requesting IPs that will be filtered; Use “,” as a delimiter

JBOSS must be restarted after changing parameters in the bms.properties file, and any existing sessions must be reconfigured. Also, a fresh copy of the bms.properties file is copied into the JBOSS on Fresh installation. In most cases, the properties are commented with a “#” to indicate that this property is already set in the code. For example: #OUTPUT_TS_MAX_SCORE=15 specifies a hard-coded value.

7.6 Setting Up Database Replication for BMS

It is important to continually replicate the database on the main BMS server, in order



to sustain the system in case of hardware problems, server malfunctions or other uncertain situations.

The following step-by-step procedure describes how to make sure the database is fully replicated to another BMS server (backup/protect).

A number of issues to note:

It is assumed that you have used bms-setup.sh to configure a BMS server, and have the server running.

The secondary server must go through the entire configuration process using bms-setup.sh, without starting the processes.

During this time, do not perform any transactions using the BMS GUI screens.

The local properties file must be copied manually from the master to the slave.

If there is no spare BMS server on site, please back up the BMS DB after installation and setup, and manually copy the database files to a different server.

Master Configuration

Copy the /usr/local/imagine/mgmtservices/jboss/server/default/conf/local.properties file from the master server to the slave server.

To set up the master server:

1. Edit /etc/my.cnf: Under the [mysqld] section, add the following section:

binlog_format=mixed log-bin = mysql-bin binlog-do-db=jbossdb binlog-do-db=bms_db server-id=1 innodb_flush_log_at_trx_commit=1 sync_binlog=1 expire_logs_days=2 max_binlog_size=100M

2. Restart MySQL:

/etc/init.d/mysqld restart

3. Log into the MySQL database as root and grant the user 'root' replication privileges:



mysql -u root -p Enter password:

This brings you to the MySQL shell.

GRANT REPLICATION SLAVE ON *.* TO 'root'@'%' IDENTIFIED BY 'root'; FLUSH PRIVILEGES;

Note that in this example the username is 'root' and password is 'root'.

SHOW MASTER STATUS;

The last command will result in a display similar to this:

+---------------+----------+--------------+------------------+ | File | Position | Binlog_do_db | Binlog_ignore_db | +---------------+----------+--------------+------------------+ | mysql-bin.004 | 98 | bms_db,jbossdb | |+--------------+----------+--------------+------------------+ 1 row in set (0.00 sec)

Make a note of this information, as you will need it later on for the slave.

4. Leave the MySQL shell:

quit;



5. Issue these commands:

mysqldump -u root -proot --opt jbossdb > jbossdb.sql mysqldump -u root -proot --opt bms_db > bms_db.sql

6. Copy these files into the slave server using your method of choice (for example, scp).

Slave Configuration

To set up the slave server:

1. If the mysqld daemon is not started, start it using the command:

service mysqld start

2. Use the following command to clean databases and ensure that the jboss database is ready to receive the table structure:

/usr/share/ibs/scripts/create_db.sh

3. Follow with these commands to copy the database structure from the master database into the slave:

mysql -u root -proot jbossdb </home/qoduser/jbossdb.sql mysql -u root -proot bms_db </home/qoduser/bms_db.sql

4. Make a copy of the /etc/my.cnf file by issuing the command:

cp /etc/my.cnf /etc/my.cnf.backup

5. Edit /etc/my.cnf to contain the following:

server-id=2 master-host=<master-ip-address> master-user=root master-password=root master-connect-retry=60 replicate-do-db=jbossdb replicate-do-db=bms_db slave-skip-errors=1062 expire_logs_days=2 max_binlog_size=100M

6. Restart MySQL:

/etc/init.d/mysqld restart



7. Enter the following:

mysql -u root -p Enter password: SLAVE STOP; Change MASTER to MASTER_HOST='<master-ip-address>', MASTER_USER='root', MASTER_PASSWORD='<root>', MASTER_LOG_FILE='mysql-bin.004', MASTER_LOG_POS=98;

Notice the master_log_file and master_log_pos. They appeared in the 'show master status;' command issued at the master server.

8. At the mysql prompt, enter:

START SLAVE; quit;

When jobss/jobss1 is updated on the master, all changes will be replicated to the same database on the slave.

Do not start the jboss daemon on the slave if the master is running. If you do so you will experience faults.

Recovery from a BMS Server Hardware Failure (Hot Standby)

If the Primary BMS server has malfunctioned, and the standby BMS server that has the replicated database is a BMS protect server, use the following switchover procedure.

To recover from a hardware failure (hot standby):

1. Shut down the Primary (main) BMS.

2. On the BMS protect server:

Edit /etc/sysconfig/network file, and change hostname to the main BMS name (for example CMC01-BMS_main)

Edit /etc/sysconfig/network-scripts/ifcfg-eth2 and change IP address to the main BMS IP

Type chkconfig jboss on.

Reboot.

At this point there will be a master BMS server and no ‘protect’ server.



3. When the former ‘main’ BMS server is brought back online or replaced by a new server, prepare a ‘protect’ server as follows:

Install the latest BMS software (if necessary)

Run bms-setup.sh, choosing to do a FRESH installation for mysql DB.

Repeat the slave configuration procedure on this server.

4. Follow the master configuration procedure on the NEW ‘main’ BMS server.

The two servers have now switched roles. The former protect is now main, and vice versa.

Recovery from a BMS Server Hardware Failure (Cold Standby)

If the Primary BMS server has malfunctioned, the replicated database ensures a quick recovery process.

To recover from a hardware failure (cold standby):

Follow this procedure carefully in order to maintain data integrity. Be sure to do all steps as the root user.

1. Prepare a new BMS server, loading Linux v3.0 for SelenioNext unto it.

2. Specify a hostname for the server, and the same IP address that it had before the failure. This is highly important; otherwise, you will not be able to control the MUXes. Make sure the server is ready to function as a BMS server.

3. Run bms-setup.sh on the new server. Do not start the BMS server at this point: MySQL needs to be running in order to load the primary data.

4. On the BACKUP server, make sure there is no instance of jboss running. (If there is, issue the service jboss stop and wait to see if jboss has stopped by issuing: ps –ef | grep java. If nothing appears, then the jboss service is stopped and you may proceed to the next step.)

5. Enter the following commands:

mysqldump -u root -proot --opt jbossdb > jbossdb.sql mysqldump -u root -proot --opt bms_db > bms_db.sql

6. Copy these files onto the NEW primary BMS server.

7. On the Backup server, copy /etc/my.cnf.backup (the backup file you created earlier when you set up the mysql slave server) to /etc/my.cnf).

8. Restart mysqld:



server mysqld restart

At this point, the backup server is standalone.

9. On the primary BMS server, import the raw data you have copied from the backup server:

mysql -u root -proot jbossdb <jbossdb.sql mysql -u root -proot bms_db <bms_db.sql

10. Start the jboss server on the Primary server. You should be able to log in and work as usual.

It is important to be aware that currently there is NO REPLICATION between master and slave servers. In order to duplicate the above, you must repeat the master replication procedure.

Troubleshooting


8 Troubleshooting

If you are experiencing trouble, the following procedures should be followed:

Verifying SelenioNext Output Stream

Verifying Video Sources

Testing SelenioNext Equipment Operation

Checking Log Files

Verifying Status of High Availability

8.1 Verifying SelenioNext Output Stream

To verify a SelenioNext output stream:

1. Use an h.264 analyzer to check the SelenioNext output.

2. Verify that the bit rate is in accordance with the setup.

3. Verify that the configured program exists within the SPTS.

4. Verify for the configured program video bit rate and audio bit rate. AC-3 audio bit rate should be approximately 384 kbps.

5. Verify an SCTE-35 PID (if applicable).

8.2 Verifying Video Sources

To verify video sources:

1. Use an MPEG-2/h.264 analyzer to check the sources.

2. Verify an IP bit rate for the source.

3. Verify that the source program is there.

4. Verify for each program a video bit rate and audio bit rate.

5. Verify an SCTE-35 PID (if applicable).

6. Right-click on a stream and select Media Info.

You should see video and audio PIDs with green (OK) status.

7. If there are no video sources, verify correct functionality of the IP routers, input ASI->IP conversion device, and/or satellite receiver.

Troubleshooting


8.3 Testing SelenioNext Equipment Operation

To test the SelenioNext equipment operation:

1. If the sources are good, and there is still no output from the SelenioNext (or programs are missing), check the SelenioNext servers.

2. If only one stream is missing from the output, check the IBP that is processing that stream:

Is the IBP powered on?

Are all Ethernet cables firmly connected? You can issue a ping to the server to check connectivity to the router.

Can you access the IBP via the SSH console?

3. If all streams are missing from the output of the ICE Mux, and the inputs are OK, check the ICE Mux.

Is the ICE Mux powered on?

Are the Ethernet cables firmly connected? You can issue a ping to the server to check connectivity to the router.

Can you access the ICE Mux via the SSH console?

(Note – the above procedure is using stat mux as an example, it is similar for CSP or any other SelenioNext application).

8.4 Checking Log Files

To check the log files:

1. SSH to the SelenioNext server.

2. Tail the log file with “tail –f /var/log/ibs/avp.log”.

3. Check if new events are being displayed in the log file. If no events are being logged, make sure that the Ethernet interface is up (by using the Linux utility ethtool eth0 (or eth2)), and that there are no problems with the Video source or router.

8.5 Verifying Status of High Availability

To check the status of High Availability:

1. SSH to one of the processors that is part of the cluster.

2. Issue the following command: sudo /usr/sbin/hare-mon -b (or –bx to include comments)

Troubleshooting


Verify that the username “qoduser” is located in the sudoers list before running this command. Otherwise, run as root.

3. Verify that all cluster hosts are present.

Maintenance Procedures


9 Maintenance Procedures

The following sections detail the maintenance procedures for each of the following scenarios:

Logging into a Server

Shutting Down a Server

Rebooting a Server

9.1 Logging into a Server

Putty and secure CRT are both popular SSH clients on the Microsoft Windows OS.

Using putty, log into the BMS_Main server as follows:

1. If you are at a KVM connected to the BMS, open a console window.

2. Type ‘ssh qoduser@servername’, where servername is the name of the server. The password is qoduser.

3. If you need root access, issue ‘su –‘. The password is XXX.

9.2 Shutting Down a Server

This will be service affecting.

To shut down a server:

1. Log in to the server you want to shut down by following the procedure in Section 9.1 Logging into a Server.

2. Switch to the root user with: 'su -'. The root password is XXX.

3. Shut down the server with: init 0 or with poweroff.

The server shuts down gracefully.

9.3 Rebooting a Server

Use this procedure as an alternative to rebooting from the BMS screen: selecting the server and clicking the Reboot tool button.

This will be service affecting.



To reboot a server:

1. Log in to the server you want to shut down by following the procedure in Section 9.1 Logging into a Server.

2. Switch to the root user with: 'su -'. The root password is XXX.

3. Reboot the server with the command reboot.

The server reboots. Boot time should be approximately three to four minutes.



Appendix A – Creation of a System Definition File

To generate a system definition file, create a copy of the sample system definition file (/usr/share/ibs/scripts/setup/xmls/<SAMPLE_TYPE>), preferably using a different name, and edit it to represent your system.

There are files for the following possible system types, in addition to the generic file:

Dell R720; <SAMPLE_TYPE> = system-drac.sample

Generic, without remote management cards; <SAMPLE_TYPE> = system-bmc.sample

This system definition file should be stored on a SelenioNext server and run as a “qoduser” user.

Sample Dell R720 Definition File #! /usr/bin/ruby /usr/bin/rvt .template /usr/share/ibs/scripts/setup/xmls/system.rxml

.output system-drac.xml

cluster-id Dell cluster-mcast-ip 255.255.255.255

cluster-port 16642 data-netmask 255.255.0.0

mgmt-netmask 255.255.0.0 mgmt-gw 172.16.0.1

ntp-ip 172.16.200.113 idp-ip 255.255.255.255

dynamic-routing yes

data0-if eth0 data1-if eth3

mgmt-if eth2

h: hostname cluster-ip data1-stub-ip mgmt-stub-ip stonith

h bms 10.200.1.56 10.200.2.56 172.16.203.56 ipmi:172.36.200.224:root:qodadmin

h next-1 10.200.1.124 10.200.2.124 172.16.200.124 ipmi:172.16.200.224:root:qodadmin

h next-2 10.200.1.125 10.200.2.125 172.16.200.125 ipmi:172.16.200.225:root:qodadmin

r: group-id mgmt-ip data0-ip data1-ip hosts

r dell-1 172.16.200.24 10.200.1.24 10.200.2.24 next-1, next-2



The above sample for Dell represents a 1+1 cluster, and therefore there is an entry for the BMS server.

Sample Generic System Definition File (2 data interfaces) #! /usr/bin/ruby /usr/sbin/rvt .template /usr/share/ibs/scripts/setup/xmls/system.rxml

.output system-bmc.xml

cluster-id ibs1 cluster-mcast-ip 255.255.255.255

cluster-port 16642

data-netmask 255.255.0.0

mgmt-netmask 255.255.0.0 mgmt-gw 172.16.0.1

ntp-ip 192.168.11.253 idp-ip 255.255.255.255

dynamic-routing no

data0-if eth0 data1-if eth3

mgmt-if eth2

h: hostname cluster-ip data1-stub-ip mgmt-stub-ip stonith

h ibp1 10.190.1.41 10.190.1.141 172.16.190.41 ipmi:172.16.190.11:root:qodadmin h ibp2 10.190.1.42 10.190.1.142 172.16.190.42 ipmi:172.16.190.12:root:qodadmin

h ibp3 10.190.1.43 10.190.1.143 172.16.190.43 ipmi:172.16.190.13:root:qodadmin h ibp-standby 10.190.1.44 10.190.1.144 172.16.190.44 ipmi:172.16.190.14:root:qodadmin

r: group-id mgmt-ip data0-ip data1-ip hosts

r ibp1 172.16.190.31 10.190.1.31 10.190.1.131 ibp1,ibp-standby r ibp2 172.16.190.32 10.190.1.32 10.190.1.132 ibp2,ibp-standby

r ibp3 172.16.190.33 10.190.1.33 10.190.1.133 ibp3,ibp-standby



Notes:

The system file needs to be executable: use “chmod a+rx <file>”

cluster-mcast-ip – Do not change this parameter; it must be set to 255.255.255.255.

cluster-port is the port being used by the cluster , default is 16642. The value of cluster-port is in the following range : 1024 < cluster-port<=65535

dynamic-routing is dynamic binding of multicast source to physical interface. When set to yes, the source can be joined from a different interface.

In order to alter interfaces used for primary and secondary data and mgmt-if, uncomment the lines such as # data0-if by removing the hash mark, and edit the interface names.

The h: table represents the physical hosts and their static details (hostname and IPs). These are the primary IPs.

Host Resources

Host resources include:

Hostname

Cluster-ip is the IP address that is assigned to data0-ip, and is used to transfer heartbeat messages between the servers

Data1-stub-ip is assigned to data1-if, and is used to keep the data1 interface active to save failover time

Mgmt-stub-ip is assigned to mgmt-if, and is used to keep the mgmt interface active to save failover time

Stonith are the parameters for Stonith (shutting down a potentially malfunctioning server): method (IPMI/Telnet/Shoot), IP, username, password (if required). In some cases, several methods may be supplied.

The r: table represents the resources managed by the cluster, which are moved between the hosts during a failover event. The last column in the table represents which hosts each resource can be assigned to, the first value representing the default. These are the “Virtual IPs”.

selenio next installation troubleshoot guide v1 edition b manual

Documents