seismic engineering research infrastructures for european … · 2012-05-22 · seismic engineering...

SEVENTH FRAMEWORK PROGRAMMECapacities Specific Programme

Research Infrastructures

Project No.: 227887

SERIESSEISMIC ENGINEERING RESEARCH INFRASTRUCTURES FOR

EUROPEAN SYNERGIES

Workpackage 2Deliverable 2.4 – Specifications for Distributed Testing (connectivity tests,

implementation)

Deliverable/Task Leader: UNIKARevision: Final

February, 2012

Specifications for Distributed Testing (connectivity tests, implementation)

i

ABSTRACT

The aim of SERIES/NA1.4 task is to provide the laboratories of the consortium with basicdistributed testing capabilities. Due to current hardware limitations on network connectivity, thefocus is on slow, non-synchronized distributed testing, mainly concerning Pseudo-dynamic testsand labs. The activity is also aimed at gaining practical experience in distributed testing and willpropose a European strategy towards fast and real-time distributed testing. The following actionswill be carried out:

Gather practical experience regarding distributed testing among the partners and reviewavailable tools.

Select, and adapt when needed, algorithms and software to connect partners; issuerelevant guidelines.

Implement the encapsulation software layer (possibly complying with NEEScommunication protocols) providing standard communication between theinfrastructures.

Organize the environment platform for distributed tests. Evaluate connectivity procedures (under normal and accidental operating conditions). Evaluate the performance of the distributed platform including time-scaled continuous

tests. Assess extensibility to non-linear physical structures, with or without rate-dependent

behaviour, and to complex numerical models. Propose and advance a European strategy towards fast and real-time distributed testing.

This deliverable includes the specifications and describes the necessary requirements toimplement the different communication protocols. Also, the new developments in Oxfordregarding the organisation of distributed distributed tests. An evaluation of the Craig-BamptonMethod and the use of Graphic Card Units (GPU) to include large numerical models are alsopresented.

Keywords: Distributed testing, Network protocols


ii


iii

ACKNOWLEDGMENTS

The research leading to these results has received funding from the European Community’s

Seventh Framework Programme [FP7/2007-2013] under grant agreement n° 227887.


iv


v

DELIVERABLE CONTRIBUTORS

[UNIKA] [Ferran Obón Santacana]

[UNITN] [Oreste S. Bursi]

[Giuseppe Abbiati]

[UPAT] [Stathis Bousias]

[UOXF.DF] [Tony Blakeborough]

[Ignacio Lamata]

[ITU] [Ercan Yuksel]

[METU] [Ozgur Kurc]

[Tunc Bahcecioglu]


vi


vii

CONTENTS

List of Figures ................................................................................................................................ xi

List of Tables ............................................................................................................................... xiv

1 Introduction .............................................................................................................................1

1.1 Objectives .......................................................................................................................1

1.2 Description......................................................................................................................1

1.3 Activities and connectivity tests .....................................................................................2

2 General Requirements and Specifications...............................................................................5

3 Networked Structural Experiment Protocol (NSEP) [UNIKA] ..............................................7

3.1 Description......................................................................................................................7

3.2 Additional requirements .................................................................................................8

3.3 Software specifications to communicate using NSEP....................................................9

3.3.1 Configuring the server – Configuration file.....................................................10

4 OpenFresco [UNIKA and UNITN] .......................................................................................13

4.1 Brief description ...........................................................................................................13

4.2 additional requirements ................................................................................................13

4.3 software specifications to communicate using OpenFresco [UNIKA] ........................13

4.3.1 The new Experimental Facility: GenericTCP ..................................................13

4.3.2 Implementing a generic client. .........................................................................15

4.4 PsD and real-time tests with substructuring using OpenFresco [UNITN] ...................17

4.4.1 Architecture of the numerical simulations of pseudo-dynamic tests with sub-

structuring ........................................................................................................19

4.4.1.1 Matlab – OpenFresco interface: TCPSocket use ............................................ 204.4.1.2 OpenFresco: implementation of the interface to the finite element model ..... 214.4.1.3 Numerical simulation of a split-mass SDoF system ....................................... 23


viii

4.4.1.4 Numerical simulation of a three-degrees-of-freedom system......................... 244.4.2 Architecture of the pseudo-dynamic test with sub-structuring ........................26

4.4.2.1 Matlab – OpenFresco interface: TCPSocket use ............................................ 284.4.2.2 OpenFresco: implementation of the interface to the xPCTarget..................... 284.4.2.3 XPCTarget and Predictor-Corrector ............................................................... 284.4.2.4 Pseudo-dynamic and fast time test on a split-mass SDoF system .................. 30

4.4.3 The PM method................................................................................................34

5 UI-SIMCOR [UPAT] ............................................................................................................37


5.2 Additional requirements ...............................................................................................37

5.3 software specifications to communicate using UI-SimCor ..........................................38

5.3.1 Using UI-SimCor .............................................................................................41

5.3.2 Test set-up ........................................................................................................44

6 UK-NEES [UOXF.DF] .........................................................................................................45


6.1.1 Substructure relationships ................................................................................45

6.1.2 PC/controller board communication ................................................................46

6.1.3 Network communications ................................................................................47

6.1.4 Signal messages and control ............................................................................48

6.1.5 Operator control tool ........................................................................................49

6.1.6 Signal extrapolator/data handler ......................................................................49

6.1.7 Other considerations.........................................................................................49

6.2 Additional requirements ...............................................................................................51

7 Introducing the prototype of the web based tool for organization of geographically

distributed tests [UOXF.DF] .................................................................................................53


7.2 Experiment phases ........................................................................................................53

7.3 General architecture......................................................................................................54

8 The Craig-Bampton method [ITU]........................................................................................57

9 Numerical models using graphic cards [METU]...................................................................63

9.1 Introduction ..................................................................................................................63

9.2 General Purpose Computation Using Graphic cards ....................................................64


ix

9.3 Computer Cluster system - CPU SOLUTION .............................................................65

9.3.1 Linear Dynamic Analysis with Implicit Time Integration ...............................65

9.3.1.1 Dense Matrix Factorization and Load Vector Solution .................................. 659.3.1.2 Speed of Matrix-Vector Multiplication........................................................... 66

9.3.2 Linear Dynamic Analysis with Explicit Time Integration ...............................67

9.3.2.1 Implementation ............................................................................................... 679.3.2.2 Tests ................................................................................................................ 68

9.4 Computation with GRaphic cards.................................................................................70

9.4.1 Linear Dynamic Analysis with Implicit Time Integration ...............................70

9.4.1.1 Dense Matrix Factorization and Back Substitution ........................................ 709.4.1.2 Speed of Matrix Vector Multiplication ........................................................... 70

9.4.2 Linear Dynamic Analysis with Explicit Time Integration ...............................71

9.4.2.1 Implementation ............................................................................................... 719.4.2.2 Results ............................................................................................................. 72

9.5 Conclusion ....................................................................................................................73

References......................................................................................................................................75

Appendix A: Example of a synchronous FCM and CGM.............................................................77


x


xi

List of Figures

Fig. 3.2 Example of the configuration file................................................................................ 10Fig. 3.3 Flowchart of the packages sent during a test using NSEP protocol (synchronous)11Fig. 4.1 Flowchart of the packages sent between GenericTCP and the controller (server) 16Fig. 4.2 Block diagram representing the architecture of an heterogeneous simulation withdynamic sub-structuring ............................................................................................................ 17Fig. 4.3 a) OpenFresco’s logo b) Tcl language’s logo.............................................................. 18Fig. 4.4 Shell of OpenFresco...................................................................................................... 19Fig. 4.5 Block diagram representing the numerical simulation of pseudo-dynamic test withsub-structuring ............................................................................................................................ 19Fig. 4.6 MatLAB – OpenFresco TCP/IP .................................................................................. 21Fig. 4.7 OpenFresco – NS interface b) OpenFresco – Simulated PS interface. .................... 21Fig. 4.8 Simulated split mass SDOF system............................................................................. 23Fig. 4.9 Response of the simulated split-mass SDoF system. ................................................. 24Fig. 4.10 Simulated three degrees of freedom system. ............................................................ 24Fig. 4.11 Response of the simulated three degrees of freedom system. ................................. 25Fig. 4.12 Block diagram representing the architecture of the pseudo-dynamic test withsub-structuring. ........................................................................................................................... 26Fig. 4.13 Arrangement of the experimental equipment.......................................................... 27Fig. 4.14 From left: Host PC, Target xPC, MTS Flex Test 60 Controller. ........................... 27Fig. 4.15 Predictor-corrector – OpenFresco interface............................................................ 28Fig. 4.16 Task sequence of the predictor corrector-event-driven finite state machine. ...... 29Fig. 4.17 a) Extrapolation procedure b) Interpolation procedure......................................... 29Fig. 4.18 Implementation of the Predictor-corrector algorithm as finite-state machine inStateFlow. .................................................................................................................................... 30Fig. 4.19 Split-mass single degree of freedom system. ............................................................ 30Fig. 4.20 a) Physical substructure HE140A clamped at the bottom b) Actuator test setup.31Fig. 4.21 Displacement response for the pseudo-dynamic test............................................... 32Fig. 4.22 Force response for the pseudo-dynamic test. ........................................................... 32Fig. 4.23 Displacement response for the fast-time test. .......................................................... 33Fig. 4.24 Force response for the fast-time test......................................................................... 33Fig. 4.25 Split mass single degree of freedom system. ............................................................ 34Fig. 4.26 Task sequence of the PM method. ............................................................................ 34Fig. 4.27 Matlab Parallel Computing Toolbox. ....................................................................... 35Fig. 5.1 Hybrid Simulation framework for UI-SimCor (UI-SimCor Manual, Febr 2007 ). 38Fig. 5.2 Simulation procedure and data flow (UI-SimCor Manual, Febr 2007). ................. 39


xii

Fig. 5.3 Test set-up at University of PATRAS......................................................................... 44Fig. 6.1 UK-NEES 3 site test distributed testing sytem- high level control layer view. ....... 46Fig. 6.2 dSpace board to host PC to network interface software environment.................... 47Fig. 6.3 Independent channel communication with IC-DHT.exe. ......................................... 48Fig. 6.4 Foreground and background operation of IC-DHT.exe for a two stie test ............ 50Fig. 7.1 Phases of an experiment............................................................................................... 54Fig. 7.2 General architecture .................................................................................................... 55Fig. 9.1 Nvidia Fermi Architecture [NVIDIA 2012] ............................................................... 64Fig. 9.2 2D Square Domain Model ........................................................................................... 69Fig. 9.3 Average Duration of a Solution for Single Time Step............................................... 69Fig. 9.4 Average Duration of a Solution Step: GPU System .................................................. 72


xiii


xiv

List of Tables

Table 3.1 Summary of NSEP Package types. ............................................................................... 10Table 4.1 Identity tags specific of GenericTCP experimental control.......................................... 15Table 4.2 Tags used in the generic client of OpenFresco. From (Schellenberg 2009c). .............. 15Table 9.1 Dense Matrix Factorization and Load Vector Solution Times. .................................... 66Table 9.2 Matrix-Vector Multiplication Times. ........................................................................... 67Table 9.3 Total Solution Times. ................................................................................................... 67Table 9.4 Fastest Average Solution Times for a Single Time Step: ............................................. 69Table 9.5 Dense Matrix Factorization and Load Vector Solution Times. .................................... 70Table 9.6 Matrix Vector Multiplication Speed; GPU System. ..................................................... 71Table 9.7 Total Timings for Matrix Vector Multiplication and Load Vector Solution Times; GPUSystem........................................................................................................................................... 71Table 9.8 Comparison of Average Solution Step Times. ............................................................. 72


xv


1

1 Introduction

1.1 OBJECTIVES

The main task of NA1.4 is to provide the laboratories of the consortium with basic distributedtesting capabilities. Due to current hardware limitations on network connectivity, the focus is onslow, non-synchronized distributed testing, mainly concerning Pseudo-dynamic tests and labs.The activity is also aimed at gaining practical experience in distributed testing and will propose aEuropean strategy towards fast and real-time distributed testing. This will establish an initialversion of the European Platform for Distributed Testing.

1.2 DESCRIPTION

With the increasing presence of high speed and more reliable networks the earthquakeengineering community has developed over the years, different communication protocols inorder to perform tests that are not limited to the own facility and therefore, take advantage ofequipment that would have been otherwise impossible to have the hands on.

This form of testing, named distributed testing, would allow performing collaborativeexperiments over internet, by sharing data and across the network using several standardcommunication protocols, such as the Transmission Control Protocol/Internet Protocol (TCP/IP)or User Datagram Protocol (UDP). Several projects, such as the United States’ Network forEarthquake Engineering Simulation (US NEES), the UK-NEES in the United Kingdom andISEE (Internet-based Simulation on Earthquake Engineering) in Taiwan, targeted at developingseveral software that would deal with the more specific problems that such tests pose building ontop of the aforementioned network protocols. The software that came out of these projects is:

NSEP (ISEE): Networked Structural Experiment Protocol (NSEP) is developed at theNational Research Center for Earthquake Engineering (NCREE), in Taiwan. It is an eventdriven protocol which features a star network topology, with one server and severalclients connected to it. NSEP protocol has been successfully used both locally andtranscontinental (Wang et. Al 2007).

OpenFresco (US NEES): Open-source Framework for Experimental Setup and Controlis a software actively developed in the University of Berkeley, California. It features amulti-tier client-server configuration. It is flexible, completely modular and easilyextensible, allowing also the integration of several software like Matlab/Simulink, LS-DYNA, Abacus and UI-SimCor 2.6 and controllers.


2

UI-Simcor (US-NEES): is an open-source software developed at the University of Illinoisto facilitate geographically distributed testing. It is based on Matlab and can interfacewith different analysis software ABACUS, FedeasLab, OpenSees, Vector2 and Zeus-NL.

Due to the different possibilities that are available for researchers when performing distributedtests, it has been decided that the European Platform for Geographically Distributed SeismicTests should be able to grant the European researchers complete freedom and non to restrict theoptions of cooperation with other facilities. The same applies in other areas, such as hardware orsoftware. Each facility within the platform should be able to integrate into it different equipment,controllers and integration algorithms or finite element programs, being the feature to send datathrough internet, in a specific format, the only limiting criteria.

The ability to perform distributed tests is not a complete solution by itself since it providesmeans to connect laboratory equipment, not to connect researchers and to share results. TheEuropean platform for distributed tests should rely on other ongoing tasks, such as Telepresence(Task NA1.2) and database (NA 1.1, 1.3 and 1.5) in order to provide a full set of tools to performdistributed hybrid testing, share results and communicate efficiently.

1.3 ACTIVITIES AND CONNECTIVITY TESTS

2010/08 – Connectivity tests with the Linux Cluster in the University of Kassel (UNIKA).2010/09 – Connectivity tests with NCREE using the first version of NSEP (UNIKA).2010/09 – Numerical Simulation with NCREE using the first version of NSEP (UNIKA).2010/09 – Connectivity tests with Oxford (UNIKA – UOXF.DF).2011/01 – 2011/04 – First simulations using Dorka’s substructure algorithm and the Linuxcluster in the University of Kassel (UNIKA).2011/05 – Numerical simulations using OpenFresco between UNIKA and UOXF.DF (UNIKA –UOXF.DF).2011/06 – Connectivity tests with University of California, Berkeley (UNIKA).2011/06 – Numerical simulations using OpenFresco between UNIKA and University ofCalifornia (UNIKA).2011/07 – Discussion with University of California in Berkeley regarding a new experimentalfacility for OpenFresco ECGeneric (UNIKA).2011/07 – 2011/09 – In-lab preparations for the application of UI-SimCor (UPATRAS).2011/08 – Discussion with NCREE regarding the new version of NSEP protocol (UNIKA).2011/10 – Numerical simulations using the new version of the NSEP protocol with NCREE(UNIKA).2011/10 – 2011/11 – Implementation of the new version of the NSEP protocol to work with thecontroller at UNIKA (UNIKA).2011/10 – 2011/11 – Design and construction of the test set-up (UPATRAS).2011/10 – 2011/12 – Implementation of the new experimental facility ECGeneric of OpenFrescoto work with the controller at UNIKA (UNIKA).2011/11 – Connectivity tests with Hybrid Simulation Testing Center (HYSTEC) in South Koreausing OpenFresco (UNIKA).2011/12 – Continuous numerical simulation with NCREE using Dorka’s substructure algorithmand the new version of NSEP protocol (UNIKA).


3

2012/01 – Distributed tests using OpenFresco and OpenSees with HYSTEC in South Koreausing Elastic and Inelastic specimens (UNIKA).2012/01 – Continuous numerical simulation between UOX.DF and UNIKA using Dorka’ssubstructure algorithm and the new version of NSEP protocol (UNIKA – UOXF.DF).2012/01 – Interaction between UPATRAS and UNITN.


4


5

2 General Requirements and Specifications

The European Platform for Distributed Testing requires:

Access to a research network like GÉANT. Be able to integrate the software and hardware to use at least one of the communication

protocols present in this document. Namely: NSEP, OpenFresco, UI-Simcor and UK-NEES.

Regarding the ports used during communication it would be of help to avoid using port numbers0 to 1023, known as well-known ports, since they are used by system processes in order toprovide common network services like HTTP (port 80). Therefore, and also because in Unix likesystems super user privileges must be granted to bind a socket to an IP address using well-knownports, they should be avoided in distributed hybrid simulation. Registered ports that do notconflict with a specific running service of the system (IANA 2012) or dynamic or ephemeralports (these are operating system dependant) should be used.


6


7

3 Networked Structural Experiment Protocol(NSEP) [UNIKA]

3.1 DESCRIPTION

The application protocol Networked Structural Experiment Protocol, or NSEP, has beendesigned to support the communication with the different clients of the Platform for NetworkedStructural Experiment (PNSE) and it is written in C++. This platform, aimed at performinggeographically distributed seismic tests, was developed during the ISEE (Internet-basedSimulation on Earthquake Engineering) project in the National Center for Research onEarthquake Engineering (NCREE).

The platform, redesigned during 2011, features a star topology network, with a single server andseveral clients connected to it as seen in Fig. 3.1. It uses the Transmission ControlProtocol/Internet Protocol (TCP/IP), a protocol designed to guarantee reliable data transferbetween different hosts, to communicate with the different components or modules.

...

Server

CGM

FCM1 … FCM

n

Fig. 3.1 Topology of the Platform for Networked Structural Experiment (PNSE)


8

The different elements in NSEP are:

Server: Receives, organises and sends all the data packages. Command Generation Module (CGM): Client where all the commands (e.g. a

displacement) are generated. It can be any finite element software or any integrationscheme.

Facility control module (FCM): Client located in the experimental facility where theactual specimen is installed. Imposes the commands generated by the CGM to thespecimen. There can be multiple FCMs.

Each module is in charge of generating and receiving certain data packets (or signals) in aspecific format (see section 3.3 for more details). For example, during a step or a commandcycle, the communication between the different modules would be:

1. After solving the numerical model for the current step, the CGM sends thedisplacement to the server.

2. The server receives the message from the CGM and forwards it to the FCM.3. The FCM receives the message from the server, applies the displacement to the

specimen and gives and sends the response of the specimen, in this case a force, to theserver.

4. The server receives the message from the FCM, forwards it to the CGM and theprocess starts again.

Due to its modular design and the fact that there are no assumptions of the software used duringthe experiments, the NSEP protocol allows treating each component/module in a completelyindependent way from each other. This translates into a great versatility when integratingdifferent facilities, equipment and algorithms into it.

3.2 ADDITIONAL REQUIREMENTS

For the Server implementation found in (Wang) and the FlexControl program:

Windows XP, Windows Vista or Windows 7.

Any software that is wanted to be used during an experiment that relies on the NSEP protocol tocommunicate between the different facilities must satisfy the following conditions:

Ability to send/receive messages using the TCP/IP protocol. The primitive types must have the same size in bytes.


9

3.3 SOFTWARE SPECIFICATIONS TO COMMUNICATE USING NSEP

The NSEP protocol uses the TCP/IP protocol to exchange data between the facilities, but itrequires a certain format in order to make the different modules understand each other. This isaccomplished by establishing different types of messages in a predefined format of the formLength + Type + List of Parameters and by limiting the primitive types used duringcommunication to integer (int in C/C++), single-precision float-point (float in C/C++) andto strings.

1. Length: An integer (size of 4 bytes in C/C++) containing the total length of thepacket/message to be sent.

2. Type: An integer containing the type of packet sent. This is used within NSEP to identifythe different packets and react accordingly (see table 3.1)

3. List of parameters: An optional list of parameters that contains useful information. Theprimitive type of the list of parameters varies (integer, float or string in C/C++) with thetype of message sent.

As of January 2012, there are ten different packages types implemented in the platform, coveringlogin procedures, testing data exchange between facilities and instant messaging (see table 3.1).It is worth to point out the two different running modes of the clients and the server:

Asynchronous: The server will act in “push mode”, which means that it will activelysend information, without any requests, the client that enables this mode, although theclient that runs asynchronously can still ask for specific information through theNSEP_QUERY package. This mode is deactivated by default in all the clients and mustbe enabled by sending a NSEP_SETPUSH packet. It is useful if the instant messagefeature is desired.

Synchronous: In this mode, the server will act in “pull mode”, or in other words, it willnot send any messages to the client unless it asks specifically for particular informationusing the NSEP_QUERY package. In this mode, the instant messaging feature cannot beimplemented since there is no way of knowing before hand when responsible of the otherfacilities would send an instant message.

Table 3.1 summarises the existing packages available in the NSEP protocol, but the explanationpresented is far from complete. The reader should refer to (Wang 2011) to have a complete andup to date description of each package and its options.

In order to understand better the roles of the Facility Control Module (FCM), the CommandGeneration Module (CGM) and the server, as well as a hint of what and when each moduleshould send, figure 3.3 summarises the process. Since this is intended to be an illustrativeexample, only the minimum number of components is considered: a FCM, a CGM and theServer. Also, to keep things simple, both modules, the FCM and the CGM, are consideredsynchronous and no reconnecting option has been considered nor any attempt of sending thelogin information again if it is wrong in the first attempt (see __AP_LOGIN_REQUEST or__AP_LOGIN_REQUEST in (Wang 2011) for more details). An implementation example ofsynchronous FCM and CGM programs, written in standard C, is also given in Appendix A.


10

Table 3.1 Summary of NSEP Package types.

Package Name Type Description__AP_LOGIN_REQUEST 1 Package containing the login information, username and password.__AP_LOGIN_REPLY 2 Reply from the server containing the result of the login procedure.NSEP_SETPUSH 3 Packet sent by the client in order to change its behaviour: synchronous or

asynchronous.NSEP_QUERY 4 Used by FCM and CGM to ask for information from the server.NSEP_EXPINFO 5 Packet containing information regarding the experiment configuration. It is

sent by the server to the client as an answer to NSEP_SETPUSH orNSEP_QUERY packets.

NSEP_EXPSTATE 6 Packet containing the current experiment state. It can have the values of:ready, running, interrupted and finished. The experiment state depends onthe client states.

NSEP_CLNSTATE 7 Packet sent by the client to the server to change the client running state:disconnected, not ready, running or finished.

NSEP_CMD 8 Packet sent by the CGM and to be received by the FCM. Contains thecommands (values obtained as a result of an integration process/FEMprogram) that must be applied in each FCM.

NSEP_CSIG 9 Packet sent by each FCM and to be received by the CGM containing theresponse of the specimen.

NSEP_IM 10 Packet used to enable the instant messaging feature in NSEP.

3.3.1 Configuring the server – Configuration file

The server must know, before hand how the topology looks like, how many FCMs there are,which machine is the CGM, which port is going to be used and what is required to besent/received in each moment and to whom. This is done through a configuration file similar tothe one shown in figure 3.2. Special attention must be paid to the numbers after the passwords. Inthe case of figure 3.2, the FCM must receive a NSEP_CGM packet containing 1 single precisionfloat point value, and send a NSEP_CSIG packet with the same length. The same applies to theCGM. In any case, the sum of NSEP_CSIG packets send by all the FCMs should match thenumber of NSEP_CSIG packets that the CGM should receive, and the sum of NSEP_CMDpackets sent by the CGM should match the sum of NSEP_CMD packets received by each FCM.

# Test name"Example Test in UNIKA"# Number of facility control modules1# Client 1:"FCMUNIKA" "UNIKAFCM" 1 1 # Client name, password, number ofNSEP_CGM packets to receive, number of NSEP_CSIG packets to send# Client 2:"CGMUNIKA" "UNIKACGM" 1 1 # Client name, password, number ofNSEP_CGM to send, number of NSEP_CSIG packets to receive# Listening port number3333

Fig. 3.2 Example of the configuration file


11

Fig. 3.3 Flowchart of the packages sent during a test using NSEP protocol (synchronous)


12


13

4 OpenFresco [UNIKA and UNITN]

4.1 BRIEF DESCRIPTION

The Open-source Framework for Experimental Setup and Control (OpenFresco) is a NEESsoftware actively developed in the University of Berkeley in California. It provides the necessarylayer in order to perform both local and distributed hybrid simulation using an object orientedprogramming approach (C++).

OpenFresco features a multi-tier client-server architecture as described in (Schellenberg 2009a)and it also integrates OpenSSL for secure communications.


The latest version of OpenFresco can run on Windows XP, Windows Vista and 7. Althougholder versions can run under Unix like systems this feature is broken as of January 2012. Thereader should refer to (Schellenberg 2009a and 2009b) in order to obtain more detailedinstructions concerning the installation and use of OpenFresco. It is has been reported that twodifferent versions of OpenFresco may be unable to communicate with each other. Therefore it ishighly recommended to use the same version in both local and remote site. Also, both sitesshould use the same primitive types (single or double precision) in order to avoid numericalproblems.

4.3 SOFTWARE SPECIFICATIONS TO COMMUNICATE USING OPENFRESCO[UNIKA]

4.3.1 The new Experimental Facility: GenericTCP

As of January 2012, the SVN branch of OpenFresco(svn://openfresco.berkeley.edu/usr/local/svn/OpenFresco) features a new experimental controlmodule that will interface with the different control and data acquisition systems in thelaboratory using the TCP/IP protocol. It has been designed in order to integrate any kind ofequipment that is not supported using the previous experimental control programs.

Due to its design, the GenericTCP experimental control requires another server-client connectionin order to work properly: the server, which receives/sends the data from/to the controller, andthe client, the GenericTCP component of OpenFresco. This does not mean that another computer


14

is required to implement this experimental control since the TCP/IP connection can be performedlocally.

The command line used in order to call the GenericTCP experimental control is similar to thosealready implemented in OpenFresco, and is as follows. The reader should refer to thedocumentation available in (Schellenberg 2009b) and the examples for a more detaileddescription of the common command options such as –ctrlModes, -daqModes, -ctrlFilters and –daqFilters and how to configure the TCL file for an experiment.

expControl GenericTCP $tag ipAddr $ipPort -ctrlModes (5 $mode) –daqModes (5 $mode) <-initFile fileName> <-ctrlFilters (5 $filterTag)><-daqFilters (5 $filterTag)>

Where:

ipAddr: contains the IP address of the computer that receives/sends information to thecontroller and/or data acquisition system (the server).

ipPort: the port that will be used in order to exchange information between the serverand the client (Generic TCP experimental control).

initFile fileName: is an optional argument that is used in order to send someinformation to the server, where fileName is the name of the configuration file locatedin the same directory where OpenFresco is executed. This initialisation file can containany kind of information (strings, integer, single precision float points ...) sinceGenericTCP treats each line as a string and sends it as a message to the server. Thus, theserver should be responsible to tokenise the string and convert it to the right primitivedata types required by the controller.

The command line for the GenericTCP experimental control can look like:

expControl GenericTCP 1 "127.0.0.1" 3333 -ctrlModes 1 0 0 0 0 -daqModes 1 0 0 2 0 0 -initFile "InitADwin.txt"

In this example, OpenFresco will know that the server will be located in the same machine (IPaddress 127.0.0.1) and port 3333 will be used in order to exchange communication. At each step,the client (a.k.a the experimental control program) must send to the server a vector of length one(ctrModes argument) that will be interpreted as a displacement. In exchange, the server shouldsend, as a response, a vector of length three (daqModes argument) where the first value will beinterpreted as a displacement while the next two will be treated as forces. The contents of the file“InitADwin” will be sent to the server as strings (initFile argument).

Some of the data must be sent using the format Identity + Contents, where identity is an integervariable used to differentiate the type message and contents is the actual message. Note, thatOpenFresco does not restrict the primitive type to use. It must be accorded beforehand what kindof floating point precision will be used. Table 4.1 contains the identity tag used for thecommunication.


15

Table 4.1 Identity tags specific of GenericTCP experimental control.

Message Tag DescriptionID message None This message, a vector of eleven integers, is sent with the sizeCtrl, sizeDaq

and dataSize information. It has to be the first message sent/received since itis necessary for the server and the client to know the length of the vector thatwill be send/receive. For the following configuration, the ID message wouldbe:

Command: ctrlModes 1 0 0 0 0 -daqModes 1 0 0 2 0 ID message: ID[1 0 0 0 0 1 0 0 2 0 dataSize]

Initialisation file message 2 The messages sent by the client to the server are of the form “2 + line of thefile as a string”. This message is sent/received as many times as lines in theinitialisation file. This is optional.

Control values 3 The message containing the control values is of the form “3 + vector ofcontrol values”. The size of the control values is specified by the –ctrlModes argument.

Request for DAQ ormeasured values message

6 Request from the client to the server asking for the measured values. Thefirst value is always ‘6’.

Send DAQ or measuredvalues

None Message sent from the server to the client with a vector of DAQ values asspecified in the –daqModes argument.

The data must be sent/received to/from the server in a specific sequence. Figure 4.1 summarisesthe process.

4.3.2 Implementing a generic client.

OpenFresco also has support for integrating any kind of clients into it. The user should refer tothe example using Matlab (Schellenberg 2009c) or the C example in the SVN repository ofOpenFresco. Summarised in Table 4.2 are the tags used to identify the different messages.

Table 4.2 Tags used in the generic client of OpenFresco. From (Schellenberg 2009c).

Tag Function Tag Function1 Start server process (optional) 9 Get acceleration vector2 Setup test 10 Get force vector3 Set trial response 11 Get time vector4 Execute (obsolete) 12 Get initial stiffness matrix5 Commit state 13 Get tangent stiffness matrix6 Get DAQ response vectors 14 Get damping matrix7 Get displacement vector 15 Get mass matrix8 Get velocity vector 99 End server process


16

SERVER GenericTCP (CLIENT)

Send ID vector

Receive ID vector

Send contents of the Init file (optional)to the server

Receive contents of the Init file(optional) from the client

Send control values

Receive control values from the client

Request DAQ values from the server

Receive the request for DAQ valuesfrom the client

Get the daq values from the DAQacqusition system

Send DAQ values to the Client

Receive DAQ values from the server

IsFinished?

Exit

Close connection

NoExit YesIs

Finished? Yes

TCP/IP communicationProgram sequenceLEGEND

Fig. 4.1 Flowchart of the packages sent between GenericTCP and the controller (server)


17

4.4 PSD AND REAL-TIME TESTS WITH SUBSTRUCTURING USINGOPENFRESCO [UNITN]

Hybrid simulation is an experimental testing technique where a test is executed based on a step-by-step numerical solution of the governing equations of motion for a hybrid model, formulatedconsidering both the numerical and physical portions of a structural system. Figure 4.2 shows theblock diagram of this testing paradigm.

Fig. 4.2 Block diagram representing the architecture of an heterogeneous simulation withdynamic sub-structuring

This report shows the implementation of the PM method time integration scheme (Pegon andMagonette, 2002) through the finite element (hence FE) open source code OpenSees(http://opensees.berkeley.edu/) and OpenFresco (https://nees.org/resources/858). Matlab is usedto implement the PM Method; the well-known OpenSees FE code is used to simulate thenumerical substructure, whilst the newly developed OpenFresco is used to transfer data amongMatlab, OpenSees and the experimental control system. Pseudo-dynamic and a fast-time test areperformed though actuator delay is not taken into account.

OpenFresco (the Open-source FRamework for Experimental Setup and COntrol) is anenvironment-independent software framework, based on modern object-oriented software designand programming methodologies (Reference). Object-oriented programming (OOP) isa programming paradigm using "objects" – data structures consisting of datafields and methods together with their interactions – to design applications and computerprograms. Programming techniques may include features such as dataabstraction, encapsulation, messaging, modularity, polymorphism, and inheritance. Object-oriented programming (OOP) represents an attempt to make programs more closely model theway people think about and deal with the world. In the older styles of programming, aprogrammer who is faced with some problem must identify a computing task that needs to beperformed in order to solve the problem. Programming then consists of finding a sequence of

http://opensees.berkeley.edu/


18

instructions that will accomplish that task. But at the heart of object-oriented programming,instead of tasks, we find objects – entities that have behaviours, which hold information and thatcan interact with one another. Programming consists of designing a set of objects that model theproblem at hand. Software objects in the program can represent real or abstract entities in theproblem domain. This is supposed to make the design of the program more natural and henceeasier to get right and easier to understand.

Basically, OpenFresco connects finite element models with control and data acquisition systemsin an extensible manner to facilitate conducting local and geographically distributed hybridsimulations of structural systems.

The huge number and variety of finite element codes and controllers makes challenging theinterfacing between numerical and physical substructures so, in order for the earthquakeengineering community to take full advantage of Hybrid simulation technique, OpenFrescostandardizes the deployment of hybrid simulation and extends its capabilities to applicationswhere advanced numerical techniques are utilized, boundary conditions are imposed in real-time,and dynamic loading conditions caused by wind, blast, impact, waves, fire, traffic, and, inparticular, seismic events are considered.

Accordingly, the architecture of the OpenFresco software package provides a great deal offlexibility, extensibility, and re-usability to the researcher or developer interested in hybridsimulation. In practice, OpenFresco acts as:

• Middleware for a wide variety of computational software packages, includingOpenSees, Matlab, Simulink, LS-DYNA, Abaqus and UI-SimCor.

• Interface with many popular experimental control and data acquisition systemsmanufactured by dSpace, MTS (STS and FlexTest systems), National Instruments,SCRAMNet, Shore Western and others.

The OpenFresco’s framework provides a set of class and operates by Tcl scripts. Tcl (originallyfrom "Tool Command Language") is a scripting language created by John Ousterhout. It iscommonly used for rapid prototyping scripted applications.

(a) b)

Fig. 4.3 a) OpenFresco’s logo b) Tcl language’s logo

First, the implementation of the OpenFresco’s framework for hybrid test with sub-structuring issimulated numerically: both the physical and the numerical substructure are represented by afinite element model within the same OpenFresco’s environment. At the end, the architectureproposed is extended in order to test physical specimen instead of pure numerically simulatedsubstructures. Figure 4.4 shows the OpenFresco’s shell.


19

Fig. 4.4 Shell of OpenFresco.

4.4.1 Architecture of the numerical simulations of pseudo-dynamic tests with sub-structuring

The block diagram of Figure 4.5 shows the architecture of the numerical simulation of a pseudo-dynamic test with sub-structuring using OpenFresco. The blue arrows represent the displacementcommands sent to the substructure; conversely the red arrows represent the reaction forcesreturned from the substructure to the integrator. MatLAB runs the time integrator, theabovementioned PM Method, whose description is provided at the end of the chapter. In thiscase, OpenFresco acts as interface between the time integrator and the finite element model.Both the OpenFresco interface and the FE model of each substructure are computed within theOpenFresco environment, therefore they are implemented in the same Tcl script. The mexroutine TCPSocket, supplied with OpenFresco, supports the data exchange between MatLABand the numerical substructures. It is necessary to emphasize that the whole OpenSees’ finiteelement framework is directly available in OpenFresco.

Fig. 4.5 Block diagram representing the numerical simulation of pseudo-dynamic test withsub-structuring


20

4.4.1.1 Matlab – OpenFresco interface: TCPSocket use

The MatLAB mex file TCPSocket.mexw32 provides the connection to OpenFresco. A MatLABimplementation example is reported below:

% Opening connections with Subdomain 2socketID2 = TCPSocket('openConnection','127.0.0.1',8090);

if (socketID2<0)errordlg('Unable to setup connection.')return;

end

dataSize2 = 2;sData2 = zeros(1,dataSize2);dataSizes2 = int32([1 0 0 0 0, 0 0 0 1 0, dataSize2]);TCPSocket('sendData',socketID2,dataSizes2,11);% Vector command initialisationsData2 = zeros(1,dataSize2);% Send displacement commandsData2(1) = 3;% Displacement valuesData2(2) = u;TCPSocket('sendData',socketID2,sData2,dataSize2);% Read force commandsData2(1) = 10;TCPSocket('sendData',socketID2,sData2,dataSize2);rData2 = TCPSocket('recvData',socketID2,dataSize2);% Commit state commandsData2(1) = 5;TCPSocket('sendData',socketID2,sData2,dataSize2);% Shutdown commandsData1 = zeros(1,dataSize1);sData1(1) = 99;TCPSocket('sendData',socketID1,sData1,dataSize1);TCPSocket('closeConnection',socketID1);

The main tasks performed by Matlab are summarized:

1. To establish the connection with OpenFresco;2. To send the displacement command to the substructure either physical or numerical;3. To read the reaction force from the substructure;4. To commit the state;5. Once, at the end of the procedure, shutdown the connection

The availability of a network is a fundamental requirement as shown in Figure 4.6. In our case,both MatLAB and OpenFresco run in the same machine, so a localhost ip-address is used(‘127.0.0.1’).


21

Fig. 4.6 MatLAB – OpenFresco TCP/IP

4.4.1.2 OpenFresco: implementation of the interface to the finite element model

In this case, both the numerical substructure and the simulated physical substructure aresimulated by OpenFresco:

(a) (b)

Fig. 4.7 OpenFresco – NS interface b) OpenFresco – Simulated PS interface.

In agreement to Figure 4.7, the same OpenFresco script implements the interface to MatLAB andthe relevant substructure FE model. The example below explains how to use the SimulationDomain to run a hybrid simulation. The Simulation Domain experimental control makes thewhole OpenSees finite element framework, including the material, section and element libraries,available in OpenFresco. This makes it possible to simulate a physical experimental specimenusing OpenSees. This feature provides a wide variety of applications. For example, theSimulation Domain can be used to test the computational model, experimental setup, andnetwork communication to ensure that all non-experimental aspects of a hybrid simulation arefunctioning properly before conducting an actual experiment.

# ELASTIC OPENSEES BEAM-COLUMN ELEMENT AS SIMULATED PHYSICAL /NUMERICAL SUBSTRUCTURE# element elasticBeamColumn $eleTag $iNode $jNode $A $E $Iz $transfTagelement elasticBeamColumn 1 1 2 $area $E $iner 1

# EXPERIMENTAL-CONTROL OBJECT: PHYSICAL DOF# expControlPoint $tag $nodeTag dir resp


22

expControlPoint 1 2 ux disp -fact 1.000expControlPoint 2 2 ux force -fact 1.000

# SIGNAL-FILTER OBJECT: TO SIMULATE THE PRESENCE OF NOISE IN CONTROL AND MEASURED SIGNALS# expSignalFilter ErrorSimRandomGauss $tag $avg $stdexpSignalFilter ErrorSimRandomGauss 1 0 0.005; # Displacement noiseexpSignalFilter ErrorSimRandomGauss 2 0 50.000; # Restoring force noise

# EXPERIMENTAL-CONTROL OBJECT: SIMULATION ENVIRONMENT# expControl SimDomain $tag -trialCP $control point Tag1 outCP $control point Tag2 -ctrlFilters -daqFiltersexpControl SimDomain 1 -trialCP 1 -outCP 2 -ctrlFilters 1 0 0 2 0 -daqFilters 1 0 0 2 0;

# EXPERIMENTAL-SETUP OBJECT# expSetup OneActuator $tag <-control $ctrlTag> $dir -sizeTrialOut $t $oexpSetup OneActuator 1 -control 1 1 -sizeTrialOut 1 1

# LOCALSITE OBJECT# expSite LocalSite $tag $setupTagexpSite LocalSite 1 1;

# OPEN TCP/IP CONNECTION# startSimAppSiteServer $SiteTag $ TCP/IP port <-ssl> (For TCP/IP Socket)startSimAppSiteServer 1 8090;

The class provided by OpenFresco and referred by the script are:

expControlPoint: a control point represents a logical container of one or more directions(DOF) for sending command signals to a controller or acquiring feedback signals from adata acquisition system. Control-points represent logical groupings of output controlchannels or input data acquisition channels. To define the control and data acquisitionaxes for a test configuration, the direction property of the expControlPoint object is used.Each direction (DOF) is associated with a control channel and response quantity. The setof control-points and the set of directions within these control-points define the controlaxes and their expected order for the expControl object, which is using the control-points.Similarly, the set of control-points and the set of directions within these control-pointsdefine the feedback axes and their expected order, as they are received from theexpControl object. Additionally, the expControlPoint command allows for the definitionof scaling factors as well as lower and upper limits for each response quantity. Theoptional scaling factors can be used to convert between different units and/or to accountfor similitude laws. Finally, the optional limits can be invoked if no safety limits can beset in the control system.

expSignalFilter: these objects are used to filter or modify the signals that are both beingsent and received from the control system.

expControl: these objects are used to interface the different control and data acquisitionsystems in the laboratories. In addition, these commands can be utilized to simulatephysical specimens.

expSetup: these objects transform between the basic experimental element degrees offreedom in OpenFresco and the actuator degrees of freedom in the laboratory. Linear andnon-linear transformations are available.

expLocalSite: this command is used to construct a LocalSite experimental site object. Itis used to run a local test where no client-server communication is necessary.

With regard to the aforementioned implementation, several simulations and tests can be found in(Bursi 2011). For the sake of brevity, main results are summarized below.


23

4.4.1.3 Numerical simulation of a split-mass SDoF system

A split-mass single-degree-of-freedom (SDoF) test problem is used in this subsection as exampleto show the effectiveness of the architecture proposed to simulated pseudo-dynamic test. Thesplit mass SDoF system is shown in figure 4.8.

Fig. 4.8 Simulated split mass SDOF system.

The parameters of the analyzed system are: mA=mB=18.49 kg kA=kB=730 N/m cA=cB=11.62 Ns/m

The PM method is used in combination with Newmark schemes whose parameters are reportedin the table below:

Substructure γ β Integration scheme Time step

A (numerical) ½ ¼ Implicit Newmark(Trapezoidal rule) 10 ms

B (physical) ½ 0 Explicit Newmark(Central difference) 2.50 ms

The system is subject to the NS, 1940 El Centro earthquake record with a PGA value of 0.15m/s2. The NS was simulated with uniaxialMaterial of OpenFresco in the simulation. The PS issimulated by the OpenFresco SimDomain experimental control. The PM method is implementedin Matlab.


24

Fig. 4.9 Response of the simulated split-mass SDoF system.

The results calculated with the Newmark-implicit method with a time step of 1 ms areconsidered as accurate. The displacement response of the system is shown in figure 4.8 and 4.9.The simulation results with the PM method match that considered as accurate.

4.4.1.4 Numerical simulation of a three-degrees-of-freedom system

The three-degrees-of-freedom (3DoFs) system shown in figure 4.10, with three degrees in sub-domain A (NS) and two degrees in sub-domain B (PS), is considered in this subsection.

Fig. 4.10 Simulated three degrees of freedom system.


25

The parameters of the system are: mA=mB=m1/2=m2/2=50 kg k1=k2=730 N/m c1=c2=0

The PM method is used in combination with Newmark schemes whose parameters are reportedin the table below:




The system is subject to the NS, 1940 El Centro earthquake record with a PGA value of 0.15m/s2. The NS was simulated with uniaxialMaterial of OpenFresco in the simulation. The PS issimulated by the OpenFresco SimDomain experimental control. The PM method is implementedin Matlab.

Fig. 4.11 Response of the simulated three degrees of freedom system.

The results calculated with the Newmark-implicit method with a time step of 1 ms areconsidered as accurate. The displacement response of the system is shown in Figure 4.11. Thesimulation results with the PM method match that considered as accurate.


26

4.4.2 Architecture of the pseudo-dynamic test with sub-structuring

Figure 4.12 shows the block diagram of the architecture of a typical pseudo-dynamic test withsub-structuring performed using OpenFresco. The PM method runs in MatLAB environment.The blue arrows represent the displacement commands sent to the substructures; conversely thered arrows represent the reaction forces returned from the substructures to the integrator over thesame degrees of freedom. In this case, OpenFresco acts as interface between the time integratorand both the numerical and the physical substructure. As for the previous case the OpenFrescointerfaces the substructures to MatLAB. The physical substructure is tested experimentally, so inthe Tcl script running within OpenFresco only the interface to the xPCTarget is defined. ThexPCTarget is the real-time operative system provided by Mathworks in which the predictor-corrector software runs. The predictor-corrector keeps the actuator moving continuously bysending a continuous command to the controller alternating extrapolation and interpolation tasksbased on the displacement solutions available from the previous time integration steps. TheShared Common RAM Network (Scramnet) GT200 is a real-time communications network andshares memory between xPCtarget and the controller:

• The control signal, the displacement command, is sent from the host PC to the xPCtargetwhere is copied instantaneously to the controller computer via Scramnet;

• The measured signal, the reaction force, is sent from sensors to the controller where iscopied instantaneously to the xPCtarget via Scramnet.

The data exchange between MatLAB and OpenFresco is still supported by the TCPSocket, thesame mex function supplied by OpenFresco that works over the TCP/IP protocol.

Fig. 4.12 Block diagram representing the architecture of the pseudo-dynamic test withsub-structuring.

The arrangement of the experimental equipment is shown in Figure 4.13. The physicalsubstructure is the specimen, represented by the cantilever beam clamped at the bottom:


27

Fig. 4.13 Arrangement of the experimental equipment.

The PM Method and OpenFresco codes are both implemented in the host PC; the Predictor-Corrector algorithm runs in the xPCtarget. The PID Controller represents the MTS Flex Test 60controller.

Fig. 4.14 From left: Host PC, Target xPC, MTS Flex Test 60 Controller.


28

4.4.2.1 Matlab – OpenFresco interface: TCPSocket use

The TCPSocket provides the connection between MatLAB and OpenFresco in the same way asthat proposed for the simulation of the pseudo-dynamic tests described before.

4.4.2.2 OpenFresco: implementation of the interface to the xPCTarget

By using OpenFresco, the trial displacements calculated by the PM Method are sent both to thephysical substructure and numerical substructure, and the respective reaction forces are measuredand sent back to MatLAB for calculating further step displacement commands. For the numericalsubstructure, the architecture proposed is the same as that adopted for the simulation of pseudo-dynamic tests so is not reported again. For the physical substructure the scheme is a little bitdifferent: the displacement command resulting from the PM method is sent to the xPCTargetwhere the Predictor-Corrector algorithm writes the extrapolated/interpolated command on theScramnet; the command becomes instantaneously available in the MTS Controller and is sent tothe actuator. The block diagram of the architecture described is reported in Figure 4.15

Fig. 4.15 Predictor-corrector – OpenFresco interface.

For both the real and the numerical simulation of the pseudo-dynamic test the Tcl script tointerface MatLAB and the numerical substructure is the same so isn’t reported again. The Tclscript adopted to interface MatLAB and the XPCTarget is reported below:

# expControl xPCtarget $tag $type ipAddr $ipPort appName appPathexpControl xPCtarget 1 1 "192.168.1.10" 22222 predictor_corrector_PM "C:\test\predictor_corrector"

# expSetup OneActuator $tag <-control $ctrlTag> $dir -sizeTrialOut $sizeTrial $sizeOutexpSetup OneActuator 1 -control 1 1 -sizeTrialOut 1 1

# expSite LocalSite $tag $setupTagexpSite LocalSite 1 1

startSimAppSiteServer 1 8080

The predictor-corrector Simulink model downloaded into the XPC must be specified in the firstrow.

4.4.2.3 XPCTarget and Predictor-Corrector

The Predictor-Corrector software provided by OpenFresco is an event-driven algorithm runningon the real-time system xPCTarget. It keeps the actuator moving during the MatLABcomputation – see (Campbell and Stojadinovic (1998), Mosqueda (2003)). Until the


29

displacement solution at the next time step is not available, the trajectory sent to the controller isextrapolated from the previous time steps.

Fig. 4.16 Task sequence of the predictor corrector-event-driven finite state machine.

When the displacement solution is available from MatLAB, the finite state machine switches tothe interpolation task and interpolates the trajectory over the last few commands and the solutionat the next time step. Figures 4.16 and 4.17 describe this procedure.

(a) (b)

Fig. 4.17 a) Extrapolation procedure b) Interpolation procedure.

A third AutoSlowDown state in which the actuator linearly approaches zero velocity is present; itis engaged if the displacement command is not still available when the 60% of the time step iselapsed. Figure 4.18 shows the implementation of the Predictor-Corrector as a StateFlow finite-state machine within Simulink. The Simulink model is downloaded into the XPC where isexecuted in real time.


30

Fig. 4.18 Implementation of the Predictor-corrector algorithm as finite-state machine inStateFlow.

So far, only the interface to one degree-of-freedom physical system has been developed. Amongthe most important outlook there is the extension of the interface to two-degrees-of-freedomphysical systems. The next subsection shows the experimental tests conducted to validate thearchitecture proposed and the hardware set-up.

4.4.2.4 Pseudo-dynamic and fast time test on a split-mass SDoF system

The simplified model that corresponds to the SDoF system is shown in Figure 4.19. The physicalsubstructure column is connected to the numerical column with hinge.

Fig. 4.19 Split-mass single degree of freedom system.


31

The properties of the analysed system are:

mA=mB= 5.3194104 kg; kA= 2.1106 N/m; cA=cB= 3.3423104 Ns/m corresponding to an equivalent damping of 5 %.

The PM method is combined with Newmark schemes whose parameters are reported in the tablebelow:




The physical substructure shown in figure 4.20 is a HE 140A steel profile with an effectiveheight of 1.35m. The system is subject to the NS, 1940 El Centro earthquake record with a PGAof 0.20 m/s2.

(a) (b)

Fig. 4.20 a) Physical substructure HE140A clamped at the bottom b) Actuator test setup.

The time scale is chosen to be 10 for the pseudo-dynamic test, which means that each laboratorytime step last 0.2s whist the integration time step is chosen to be 0.02s.

Time in the lab 10Time for the structure

No actuator delay compensation, estimated to be about 15 msec, is used. A comparison betweenthe displacement command from MatLAB and the corresponding displacement response fromthe controller is shown in Figure 4.21. It can be seen that the displacement response in the


32

controller agrees well with the corresponding desired command in MatLAB. Figure 4.22 showsthe corresponding measured force response of the system.

Fig. 4.21 Displacement response for the pseudo-dynamic test.

Fig. 4.22 Force response for the pseudo-dynamic test.

The time scale is chosen to be 1 for the fast-time test, which means that each laboratory timestep last 0.2s whist the integration time step is chosen to be 0.02s.

Time in the lab 1Time for the structure


33

No actuator delay compensation, estimated to be about 15 msec, is used. A comparison betweenthe displacement command from MatLAB and the corresponding displacement response fromthe controller is shown in Figure 4.23. It can be seen that the displacement response in thecontroller agrees well with the corresponding desired command in MatLAB. Figure 4.24 showsthe corresponding measured force response of the system.

Fig. 4.23 Displacement response for the fast-time test.

Fig. 4.24 Force response for the fast-time test

It is shown that the measured forces/displacements in MatLAB well match the correspondingforces/displacements in the Controller for both the pseudo-dynamic and the fast-time test. Thisshows the accuracy of the experimental result.


34

4.4.3 The PM method

The partitioned integration scheme adopted for the numerical and experimental tests is proposedby Pegon and Magonette, the so called PM method.

Fig. 4.25 Split mass single degree of freedom system.

For a typical split-mass single degree of freedom system, as that depicted in Figure 4.25, theequations of motion are reported below:

1 1 1 , 1 1( , )TA A A A A A A

n n n ext n nM u R u u F L

/ ( 1)/ ( 1)/ , ( 1)/ /( , )T

sub sub sub sub sub

B B B B B B Bn j n n j n n j n ext n j n n j nM u R u u F L

/ / 0sub sub

A A B Bn j n n j nL u L u

The third of the set imposes the velocity continuity over the interface between the two sub-domains by mean of Lagrange multiplier. The novel parallel procedure is sketched in Figure4.26. The time step exploited in sub-domain A is equal to 2 AT in order to anticipateinformation on the sub-domain B at the beginning of the new time step.

Fig. 4.26 Task sequence of the PM method.


35

The two processes highlighted in red and blue are executed simultaneously. At the beginning ofthe actual time step nt , the interpolated free-body solution till 1nt relevant to the sub-domain Ais available; in the sub-domain B the integration process advances and at the end of each sub-stepthe velocity continuity over the interface is restored by Lagrange multiplier. At the same time, aforward prediction of the numerical substructure solution to the time step 2nt is performed.

The parallel execution of the two highlighted processes is done by the MatLAB ParallelComputing Toolbox. The Parallel Computing Toolbox software allows offloading work fromone MatLAB session (the client) to other MATLAB sessions, called workers. It allows runningas many as eight MATLAB workers on the local machine in addition to the MATLAB clientsession. MATLAB Distributed Computing Server software allows running as many MATLABworkers on a remote cluster of computers as the licensing allows.

Fig. 4.27 Matlab Parallel Computing Toolbox.

It is proven that the PM method is stable and:

• Second-order accurate without sub-stepping ( 1ss );• First-order accurate with sub-stepping ( 1ss ) (Bonelli 2008).

One can use different implicit/explicit Newmark-based algorithms in different sub-domains;even use the -method implicitly or explicitly if needed (Bursi 2010). One can exchange sub-domains to accommodate different complexities:

• Complex Numerical Substructures (NS)• Complex Physical Substructures and so on, including delays.

In fact, sub-cycling in one of the two sub-domains can improve accuracy. Think, for instance, tothe Operator-splitting method: a small t allows for the use of the initial stiffness operatorinstead of other measured stiffnesses.


36


37

5 UI-SIMCOR [UPAT]


The platform has been developed at UIUC facility in NEES to facilitate geographicallydistributed PSD hybrid simulation and has been widely used for PsD hybrid simulation and alsofor multi-platform simulation. Due to the open source code (based on Matlab) and transparentarchitecture of the code, it has been also used to implement and verify new integrationalgorithms by researchers. To facilitate integration of a variety of analysis platforms and tosatisfy the diverse analytical environments dictated by user needs, interface programs for fiveanalysis platforms have been developed (ABACUS, FedeasLab, OpenSees, Vector2 and Zeus-NL). The latest release (v2.8) except for minor updates and bug-fixed includes new features,namely:

The architecture is modified such that new integration scheme can be easily implementedby a user.

Network Interface for Console Application (NICA.exe) is rewritten for performance ofthe code and replaces the former NEES-SAM.exe. NICA supports OpenSees, Zeus-NL,Abaqus, and other generic function written in C++ or Fortran. Up to 1,000 DOFs and10,000 time steps have been tested and confirmed that there is no issue in communicationbetween NICA.exe, UI-SimCor and other applications.

Tri-axial component of ground motions can be applied to the model. For concept development and verification of an integration scheme, resistance forces can

be directly calculated from predefined stiffness matrix. This functionality does notrequire substructure modules, and hence, is useful to check accuracy of integrationscheme and functionality of UI-SimCor before involving substructure modules.

Using SIMCOR as the Simulation Coordinator means that integration will take place withinSIMCOR engine, involving (only) the effective DOFs, where as the sub-structured staticmodules will provide the reaction forces for the applied displacement at each time step.


The platform operates under Windows environment and requires MATLAB (version not earlierthan v6.5) and the Instrument Control Toolbox for MATLAB.


38

5.3 SOFTWARE SPECIFICATIONS TO COMMUNICATE USING UI-SIMCOR

UI-SimCor can communicate with any experimental equipment through TCP/IP, LabView1,LabView2, OpenFresco and NHCP protocols. FEDEAS Lab and ABAQUS communicate withLabVIEW2 protocol. ZEUS-NL, OpenSees, and experimental modules communicate withTCP/IP, LabVIEW1/LabVIEW2 protocols (in which ASCII code is used to send and receiveinformation). OpenFresco communicate with OpenFresco1D protocol. NHCP Simulation Server,MiniMOST 1 and 2 at UIUC or SDSC communicate with NHCP. If the experiment site usesdifferent protocol to communicate with actuator then the experiment site provides a method ofcommunication. Any program language that may impose displacements and retrieve measureddisplacements/forces from experiment, can be used. The Application Program Interface (API) isalso need for communication between UI-SIMCOR and experiment site.

It is noted that NICA application is an interface program between TCP/IP network and a consoleapplication (SimCor) which does not have networking capability. The NICA runs the consoleapplication as a child process and exchange data with it through inter-process communication.

While OpenFresco is a set of middleware communication services (similar to NHCP) that isintended to facilitate hybrid simulation and needs to be supplemented with OpenSees, the pseudodynamic simulation capability is an intrinsic feature of UI-SimCor with the modules supplyingonly restoring forces (see figure below).

Fig. 5.1 Hybrid Simulation framework for UI-SimCor (UI-SimCor Manual, Febr 2007 ).


39

Fig. 5.2 Simulation procedure and data flow (UI-SimCor Manual, Febr 2007).


40

TCP connection with binary data format is used for TCP/IP protocol. For the TCP connectionwith a remote site, Instrument Control Toolbox of MATLAB is used. This toolbox is acollection of M-file functions built on the MATLAB environment. The toolbox provides aframework for communicating with instruments that support, among others, the TCP/IP andUDP protocols. The data transferred can be binary (numerical) or text. The transfer can besynchronous (blocking the MATLAB command line), or asynchronous (allowing access to theMATLAB command line).

A typical sequence of commands from TCP/IP protocol to the control system looks like this:

Initialize: create TCP/IP communication object set parameters

Syntax:Comm_obj = tcpip(IP, PORT); : to create TCP/IP objectset(Comm_obj, ‘PropertyName’, PropertyValue); : to set parameterswhere Comm_obj is TCP/IP communication object returned from TCP connection.IP and PORT are IP address and port number of remote site, respectively.PropertyName and PropertyValue are a property name for Comm_obj and a propertyvalue supported by PropertyName, respectively. For UI-SIMCOR, PropertyNameand PropertyValue for initialization process are set to ‘InputBufferSize’, and1024×100, respectively.

Example:Comm_obj = tcpip(127.0.0.1, 11997);Set(Comm_obj, ‘InputBufferSize’, 1024×100);

Open: connect interface to remote site send initialize data to remote site receive acknowledgement from remote site

Syntax:fopen(Comm_obj); to connect interface object to remote sitefwrite(Comm_obj, iniData); to send initialize data to remote sitefread(Comm_obj, size); to receive acknowledgement data from remote siteExample:fopen(Comm_obj);fwrite(Comm_obj, 500); set total number of simulation stepsfread(Comm_obj, 4); read 4 bytes of data

Loop N times: propose

Syntax:fwrite(Comm_obj, T_DISP); to write binary data to remote sitewhere T_DISP is target displacement calculated from UI-SIMCOR.

querySyntax:fread(Comm_obj, size); to read size bytes binary data from remote site.


41

Close: send closing data to remote site disconnect interface object from remote site

Syntaxfwrite(Comm_obj, closeData); to send closing data to remote site.fclose(Comm_obj); to disconnect interface object from remote site.

5.3.1 Using UI-SimCor

For each substructured problem, one folder per substructure must be created plus one for the sitecoordinating the activity:

1. Coordinator folder includes :a. a file (SimConfig.sys) with the configuration parameters for hybrid simulation

(Number of modules involved, Number of Control P\oints, Stiffness evaluation,Ground Motion input file in two-column (time, accel.) format, Analysis scheme,Masses at each DOF of each Control Point, File names, Module ports, Moduleprotocols, etc.), and,

b. as many ASCII files necessary to describe the input (accelerogram_file_name).


42

2. Sub-structure_{i} module folder contains :a. an input file (NICA.cfg) for the Network Interface for Console Application

(NICA.exe) which will provide the communication of the relevant softwareanalyzing the respective sub-structure. The input parameters to be defined in thefile are: Connection port to NICA (The connection port, e.g. 127.0.0.1: 1990-11999 are local ports, i.e. for running everything on one machine), Applicationused in analysis at remote site {i}, Control node numbers, Effective DOFs,Model file name, local node which will act as a Control Point, e.t.c.


43

b. the NICA.BATc. the input file for the software employed at remote site {i}, e.g. OpenSees, ZEUS-

NL, e.t.c.


44

3. The same is repeated for all the remote sites.

After preparing the necessary input files, (i+1) processes must be started, with i being thenumber of modules (sub-structures) of the problem plus an additional process for who will act asthe coordinator. It is noted that all processes may be on the same machine or on multiple ones. Ineach local or remote module the communication is first initialized (NICA.bat). The UI-SimCor isthen executed in MATLAB at the coordinating site.

5.3.2 Test set-up

The tests regarding connectivity and other issues employing UI-SimCor will be realized througha test set-up specially developed for the above purpose. The set-up consists of a back-to-backconfiguration of two rubber isolation devices subjected to constant axial force and horizontalexcitation time-histories. It is conceived as one of the substructures in which an actual bridgemay be partitioned into and it will be used to test communication among remote sites and thecontroller of this physical part of the structure under test.

Fig. 5.3 Test set-up at University of PATRAS.


45

6 UK-NEES [UOXF.DF]


UK-NEES was a joint research project, funded by the UK’s EPSRC, between the universities ofOxford, Bristol and Cambridge to develop facilities, programs and techniques for internetcollaboration. One aspect of this was the extension of the real-time hybrid testing systemdeveloped at Oxford to distributed testing with geographically separated physical specimens andnumerical models. The Oxford real-time testing system was developed to perform ‘fast’ tests tosimulate physical structures with natural frequencies of up to 15Hz. This required the use ofcompact numerical models and tight coupling between the numerical and physical substructures.This philosophy was maintained in the application to distributed testing with the intention that afast system could always be made to run at a slower pace, but the reverse is not true.

Development was centred in three principal areas:- i) network communication, ii) testmanagement and iii) datahandling and control. The majority of the distributed tests wereconducted between Bristol and Oxford. Both of these labs use dSpace boards installed inWindows PCs running ControlDesk, and the programs written are specific to this configuration.The dSpace programs are prepared using Simulink with the critical blocks written in C, so themodel is capable of adaption to other control board/systems with appropriate changes to theboard/system communication interface. A smaller number of tests was performed betweenOxford and Cambridge and also between all three universities. Cambridge used LabView tocontrol their actuator on the centrifuge.

6.1.1 Substructure relationships

The UK-NEES distributed test system is predicated on a radial control structure, with a centralnumerical model acting as a client to a series of directly coupled servers each running a physicalor numerical model. The control loop is based on displacement commands from the numericalsubstructure and force feedback from the physical models. With the possibility of interruptablecommunication this is the safer strategy. Figure 6.1 shows such an arrangement with theaddition that the client site is also running a physical substructure locally.


46

Fig. 6.1 UK-NEES 3 site test distributed testing sytem- high level control layer view.

The diagram shows the separate physical components of the system and the interconnections.There is a division of labour of the computational tasks. The PCs are connected to the internetand deal with the inter-site communications and the controller boards execute the local controlalgorithms and command the actuators. Under UK-NEES system the numerical model is alwayssmall and located on a controller board, but there is no requirement for this, the numerical modelcould quite easily be on another PC. In contrast to other systems which are event drivencontrollers the UK-NEES system is a streaming process. That is the system does not aim to seeka transition from one ‘equilibrium’ state (an event point) to the next but instead aims to runcontinuously carrying forward errors which are minimized by the control system over time. Thisis necessary to achieve the real-time frequency requirements.

The stability of the system requires the feedback time delay (sometimes called latency) to bekept to a minimum. Each link in Figure 6.1 introduces some delay. Some of these are present inall testing, and whilst all delay should be reduced as much as possible the ones which tend todominate in distributed testing are to do with the communication of data between the sites. Theimportant time is the delay between the numerical model commanding a displacement and thearrival of the consequential change of force in the physical substructures. The path for thisinformation involves transmission of the command from the client control board to the client PC,communication across the internet, and then from the server PC to the server control board, andthen the feedback of the force along the reverse of the command route.

6.1.2 PC/controller board communication

There are two channels of communication between the PC and the controller board: the passageof the control variables during the test and the control and monitoring of the board itself. Figure6.2 shows how these separate channels are effected in the UKNEES system. The control andmonitoring of the programs on the dSpace board are controlled using ControlDesk (dSpace


47

proprietary software), and the critical internet communications are routed through a separate datachannel via a C++ program (IC-DHT.exe) using dSpace CLib and the Windows API libraries.

Fig. 6.2 dSpace board to host PC to network interface software environment.

The program shows the separate links and processes that are running on the PC and controllerboard during the execution of a test.

6.1.3 Network communications

In the early tests the communications used TCP/IP, but the protocol is to slow for real time tests.There are two reasons for this. The first is generic to TCP/IP and the second particular, thoughnot exclusively so, to the Windows operating system. The first reason is because TCP/IP isdesigned to provide a guaranteed transmission (provided the link is maintained). It does this bytaking detailed control of the transmission, splitting the message up into blocks and rebuildingthe message at the receiving site. In the process packages are grouped and acknowledged.Transmission is also controlled so as not to overload the network by repeated sends. All thistakes time, and is not under the control of the programmer. In particular, the software will heldpackages for a time to wait to see if there is a group that it can send together. It is a very goodprotocol for guaranteed transmission provided low latency is not required. There are alsoparticular problems with the way that Windows and some other operating systems implementTCP/IP. There are two modes: blocking – in which transmission of a package to one site blocksall other transmissions until receipt is acknowledged, and non-blocking – which allows paralleltransmission. Windows Winsock2 implements only the blocking mode. When there are two


48

server connections communication to one has to complete before the second is contacted, i.e.they are contacted serially, not in parallel.

UDP/IP is an alternative protocol that sends small packages quickly, but without the guarantee ofdelivery. If traffic is high UDP packages can be dropped by internet routers. It is a fast lowlatency service but programmers must be prepared to cope with dropped messages.

6.1.4 Signal messages and control

For the speed required a simple and concise messaging system was needed. The systemdeveloped in UK-NEES cuts down the messaging to the bare minimum. In essence once the testsystem is armed, which is done by the local operators, all the server system needs to do isrespond to three types of message: 1) is the site ready, 2) move to this position, and 3) stop. Theserver must also be able to cope locally with a break in transmission, or a sequence of delayed ordropped commands, which in effect amounts to a break.

In UKNEES the overall test is initiated by the operator on the Client PC who sends a commandto the ICDHT(client) program to confirm the readiness of the system. It does this by sendingqueries to each of the ICDHT(server) programs, which respond when ready. When all the serversites are ready ICDHT(client) informs the PC operator. The operator then sends the signal to thecontroller board to start the numerical model, which runs with a fixed timestep. The system thenstreams the displacement commands, which comprise a data pair of the simulation time anddesired displacement, to the servers. The servers respond with a copy of the transmitted timestamp and the achieved force and displacement. This process repeats itself until a stop command(a negative time stamp) is sent or the server detects a break in transmission, in which case itstops itself. This whole procedure is outlined in figure 6.3.

Fig. 6.3 Independent channel communication with IC-DHT.exe.


49

6.1.5 Operator control tool

Controlling a large distributed test requires the coordination of actions at a set of sites.Communication can be difficult and the operator is isolated from the testing environment at theremote sites. Under these circumstances there is a high probability for error. In addition, thecost of any mistake is also likely to be high. It is therefore advisable that the complicated andhigh risk components of the testing sequence are controlled automatically, and the operator hastools available for controlling the important stages of the test.

The UKNEES system contains an operator tool that runs alongside ICDHT to supervise thecontrol of the joint test and facilitate and record the handing over of control to the distributed sitesoftware. It is important to record this in case something goes wrong. The tool is a softwareinterface written using the ControlDesk package. It allows monitoring the local and incomingdata during the test, but also coordinates and synchronizes the start of the test and provides visualfeedback on the state of the systems and the progress of the test.

Figure 6.4 gives a graphic description of the steps and actions during the test.

6.1.6 Signal extrapolator/data handler

The command and feedback signals are transmitted using UDP, which does not guaranteedelivery. The local system must then be able to cope when some packages are dropped, and ifthere is too great a delay amounting to a break in transmission, the controller must stop the testsafely and perform a graceful shutdown and save the data. Various strategies have been tried,but the best appears to be i) if a package is delayed extrapolate the missing point to the nexttimestep using a quadratic method, ii) if a second point is missed extrapolate linearly and iii) if athird point is missing stop the test by decelerating to zero velocity. This system must run on thecontrol board, since delays can occur in the PC/control board communication, and it must workfor both extrapolating the command displacements at the server site and dong the same for forcefeedback for the numerical model.

6.1.7 Other considerations

For fast testing it is also necessary to reduce the possibility of interruptions from other processesin the PC. It is essential to switch off any anti-virus software since this will be monitoring theinternet sockets, and will want to check the traffic. Another key process to close is WindowsExplorer. This should be started again immediately after the test, but it can be safely disabledduring the short time necessary for a fast test. Other system processes, many of whichautomatically enabled but are unnecessary, can also cause unwanted delays. Modern processorchips have several computational cores, and it is possible to allocate key processes to specificcores, and block other processes from them. Using these techniques it is possible to cut downthe likelihood of unwanted interruption of the test to a minimum. These measures are not asessential for slower tests, but are advisable for faster tests.


50

Fig. 6.4 Foreground and background operation of IC-DHT.exe for a two stie test


51


PC running Window XP (not tested on later operating systems) Matlab, Simulink, Windows API allowing access to Winsock2. Windows VC++

compiler dSpace board, ControlDesk, CLIB


52


53

7 Introducing the prototype of the web basedtool for organization of geographicallydistributed tests [UOXF.DF]


Distributed testing enables the usage of the resources of different locations to achieve a moreaccurate experiment. Having all the resources in a single laboratory requires a big amount ofspace and, normally, a big economical investment. On the other hand, sharing the facilities ofseveral laboratories makes experiments affordable, creating the illusion of using a singlelaboratory when several laboratories are involved.

At the University of Oxford, an information system to support the operation and interconnectionof resources of (distributed) facilities is being developed. This information system is created tosupport carrying out distributed experiments between multiple nodes. This will define theexperiment workflow engine, agreements between participants, the transmission of data and thepreparation of results for their storage.

7.2 EXPERIMENT PHASES

An experiment normally goes through a clearly separated number of phases, as depicted inFigure 7.1. In this figure, the participation of researchers, computers and laboratory facilities isshown. In most of the stages, human participation is required.The description of the phases is given below:

• During the test organization, participants discuss about the experiment objective and theresources that will participate in the experiment. They also have to agree on the datastructures that would be exchanged during the test and have to exchange informationsuch as network addresses among other details.

• Once the resources of the experiment are clear, a resources booking phase takes place.This guarantees that the resources will be available when the experiment is conducted.

• When all participant laboratories are ready, an experiment preparation phase might takeplace. During this phase, the resources are prepared by the participants and a formal orinformal workflow or protocol might be followed.


54

• The experiment is run in the experiment execution phase. Sometimes, no userinteraction is required or desired at this stage. The devices and machines execute theorders given until the test is finished.

• The results are collected by the researchers and a result interpretation is made. This cantake an arbitrary long time.

• When the results and conclusions are ready, they can go through the result storage /sharing phase. This allows re-use of work by others, enables machines to operate thedata and establishes a working methodology.

Fig. 7.1 Phases of an experiment

7.3 GENERAL ARCHITECTURE

A reduced main architecture is depicted in Figure 7.2. The testing platform is composed offacility software (run on each laboratory) and coordinator software. Facility and coordinatorapplications work in a special hybrid peer-to-peer (P2P) network. The benefits of using a P2Pnetwork the testing platform are given below:

• Robustness. There is no a unique coordinator, so many nodes can coordinate adistributed test between facilities.

• Decentralization. Both resources and information about resources are decentralised.• Several facilities can simultaneously share experimental and other hardware

resources to conduct the experiment.


55

Fig. 7.2 General architecture

In figure 7.2, two types of nodes are shown:

Coordinator or Manager, also called Sky nodes, which facilitate a distributed test. Theytest that the participant nodes are available, prepare the data to be exchanged andorganize the experiment as a whole.

Participants or Ground nodes, which actually conduct the test. They test the link speedbetween the participants, conduct the distributed test and collect the results for theCoordinator.

If a new participant wants to join in the testing network, the participant should know the contactdetails of a Sky node (basically an IP address or hostname and a port number).

For the test execution, two different operation modes must be considered: Fast mode. This is the only suitable mode to conduct fast experiments in real-time

basis. The exchanged data barely have overhead. It executes code normallyprogrammed with C or assembler and uses connectionless network protocols. TheCoordinator is not contacted when the experiment is taking place.

Standard mode. This is a general option to exchange data in a distributed test. Theprogramming language is not an issue and it is normally Java. The Coordinatormust be informed of every step for supervision, and reliable network protocols areused. The data might have overhead and deliver additional experimentalinformation.


56


57

8 The Craig-Bampton method [ITU]

Modelling of complex structures requires fine meshed, large size Finite Element (FE) models forcertain analysis. On the other hand, dynamic analysis of these structures requires only a fewdeformation modes which can be calculated with coarse meshed, time efficient FE models.Instead of creating a new coarse mesh for these models, one can keep the fine meshed model butdecrease the computation time by employing a suitable reduction method in the analysis. Inlarge problems, reducing the complete model is still a cumbersome task. That is why thetendency is to divide the complete structure into several sub-structures and perform the analysiscomponent wise which is called sub-structuring. The so-called Component Mode SynthesisTechnique is both a sub-structuring and a reduction method.

Craig-Bampton methodology is one of the Component Mode Synthesis techniques. It determinesthe dynamic behavior of a coupled system on the basis of a normal mode description of theuncoupled systems. The method is easy to implement whenever mass and stiffness matrices ofthe substructures are known theoretically, such as FE model. It is used especially in theaerospace industry to re-characterize large finite element models into a set of relatively smallmatrices containing mass, stiffness and mode shape information that capture the fundamentallow frequency response modes of the structure. The mode shape information consists of allboundary modes expressed in physical coordinates and a truncated set of elastic modes expressedin modal coordinates. These matrices are easily manipulated for a wide range of dynamicanalyses.

However one may have problems when dealing with experimental studies in which someidentification techniques have to be used in order to be able to determine the mass, damping andstiffness matrices of the subsystems. The possible drawback of Craig-Bampton method may beas follows:

• Applicability of the methodology is usually limited to linear and stationary systems withconstant parameters.

• For experimental sub-structuring, most measurements are limited to translational degreesof freedom because rotational degrees of freedom are difficult to measure.

• Producing of the dynamic sub-structuring code is a tedious work.• Noise is also measured with the physical quantity. The matrix inversions that are needed

in the algorithm will propagate measurement noise, resulting in an inaccurate solution forthe complete system.

The Craig-Bampton method is based on a re-formulation of the equations of motion for astructure from the set of physical coordinated to a set of coordinated consisting of physical


58

coordinated at some subset of boundary points and modal or generalized coordinates at the non-boundary points. Once transformed to modal coordinates, mode shapes representing higherfrequency responses may be truncated without loss of information.

To apply the method, transfom the coordinate system for the equation of motion for a linear,damped elastic structure

(8.1)

Using the Craig-Bampton transformation

(8.2)

This yields equations of motion in terms of truncated modal coordinates:

(8.3)

Multiply this equation by the transpose of the Craig-Bampton transformation matrix TB to yield:

(8.4)

This equation is the Craig-Bampton equation of motion. These equations may be readily solvedfor a large number of practical problems. The transformation is successful because the modesbecome uncoupled from each other, greatly reducing the manipulation required to solve theequations. Typically, modal displacement and accelerations are computed by numericalintegration for a given set of initial conditions and forcing function time histories. Physicalaccelerations and displacements follow from the Craig-Bampton transformation matrix.


59

The last equation may be re-written as follows;

(8.5)

Where:

(8.6)

MBB is the structural mass matrix reduced to the boundary nodes in the same way a Guyanreduction would be done.

(8.7)

(8.8)

Mmm is the generalized mass and is almost always set equal to the identity matrix.


60

(8.9)

KBB is the Guyan reduced stiffness matrix and is zero if the boundary is statically determinate.

(8.10)

ωo is the natural frequency of the fixed base modes.

(8.11)

ζ is the equivalent viscous damping defined as the ratio of damping, c to critical damping. Theamplitude of the response of a structure to a steady state excitation is inversely proportional tothe damping. Thus, if the damping is doubled, the response is halved. The amplitude of theresponse of a structure to a transient excitation is far less depends on damping and quite often,the difference in response between 1 and 2% damping is negligible.

As a practical matter, TB CB , TB C and T CB are nearly always chosen equal to [0].Damping of the boundary modes are non-standard and cannot be verified by test; only the sub-matrix 02 has significance.


61

The generalized mass and stiffness matrices are defined to be [Mmm] and [Kmm], respectively.

All forces applied to the structure come through the boundary points and there are no appliedloads to the non-boundary points, i.e. only FR is of concern and FL is null. However, in the full-coupled loads analysis of payload and booster, applied loads do act at non-boundary points in thebooster and they must be considered.

For most practical problems, the generalized mass matrix is normalized, damping is ignored andonly boundary forces are considered. For these conditions the dynamic equation of motion forthe Craig-Bampton method is re-stated as follows;

(8.12)

where KBB is zero for a statically determinate interface.


62


63

9 Numerical models using graphic cards[METU]

9.1 INTRODUCTION

Using graphic cards for scientific computation have been an interesting alternative to usingCPUs for the last few years. Vendors came up with different approaches to enable generalpurpose computation with graphic cards in order to secure a place in supercomputing business.Today both ATI and NVIDIA graphic cards can be used for scientific computation with differentsoftware frameworks such as CUDA (CUDA webpage), OpenCL (OpenCL webpage),DirectCompute (DirectComputer webpage) and C++ AMP (AMP webpage). As of November2011, 39 of clusters of the Top 500 super-computers are using graphic card technologies(Top500 webpage).

In this section, the possibility of utilizing general purpose graphic cards for the analysis of finiteelement models excited by earthquake loads was investigated. This way, larger structural modelcan be solved and tested within a hybrid simulation environment. Several numerical simulationswith different model sizes were performed to illustrate the performance of graphical cards duringa linear dynamic analysis with implicit and explicit integration techniques. Two different highperformance computing environment were utilized; first one is a computer cluster with 14 IntelXeon CPU with 56 cores (2.5 GHz clock speed) and 168 GB of RAM connected by InfiniBandnetwork architecture with a speed of 40Mbits. The second computing environment is a singlecomputer with NVIDIA’s Tesla C2050 graphic card having 448 CUDA cores with 1.15 GHzcore speed. Tesla C2050 has 3 GB global memory. Tests were composed of two parts. In the firstpart, the speed of load factorization and dense matrix-vector multiplication was measured. Itshould be noted that in case of linear dynamic analysis with implicit integration technique, thelinear, the system matrix is factorized once and the governing operation at each time step ismainly the forward and back substitution. If the structural matrices are condensed to somespecific nodes, the effective system matrices will be a dense matrix. Because of these reasons,the first series of tests mainly examined the speed of load factorization of dense matrices. In thesecond part, a dynamic analysis with explicit integration technique was implemented and itsspeed was tested on models with different sizes. The main advantages of explicit integration arethe parallel scalability and memory utilization because such algorithms do not require theassembly and factorization of the system matrices, all the computations can be performed at theelement level.


64

9.2 GENERAL PURPOSE COMPUTATION USING GRAPHIC CARDS

Computer graphics and animations require heavy usage of floating point operations. With theboom of game industry the prices of graphical processing units (GPU) fell down, becamegradually more advanced and thus can be used for general purpose applications.

The hardware structure GPUs differs vastly from CPUs (Central Processing Units) in design.Figure 9.1 presents Nvidia Fermi architecture that is used in the newest graphic cards of thecompany. Main execution unit of Fermi architecture is the CUDA core (dark green rectangles inFigure 9.1). Every 32 CUDA cores are grouped as streaming multiprocessors (SMs). In Figure9.1, 512 CUDA cores combine for 32 SMs. ATI GPU design is somewhat similar to the Nvidiadesign; however, the main processing unit is called streaming processor units (SPUs).

All CUDA cores host a small amount of local memory that is comparable to registers of CPUs.In addition, all CUDA cores in an SM share a slower memory called shared memory. Moreover,all SMs share the global memory, which is the largest and slowest memory type. Shared memoryof GPUs is comparable to the cache memory of CPUs and global memory is comparable toRAM. Cache memory and registers are usually controlled automatically by the CPUs. This is notthe case for registers and shared memory of GPUs. Programmers have to control data occupationand flow between global memory, shared memory and registers. Fermi architecture supports anL2 Cache used while transferring data from global memory.

Fig. 9.1 Nvidia Fermi Architecture [NVIDIA 2012]

Number of threads initiated in the GPU, is determined by the programmer. GPU assigns a uniqueid to each thread, thus programmer can divide the workload along different threads. Threads areexecuted in groups called warps. Threads in a warp (composed of 32 threads in Fermiarchitecture) can execute only a single instruction together at a given time. For efficientparallelism, programmer must assign the same instruction to all the threads in a warp. Anotherimportant rule for efficient parallelism is coalesced memory access. Threads in a warp can access


65

neighboring memory position at the same time. Otherwise, they have to wait for each other.Since both shared memory and global memory can only be written sequentially, race conditionsare a big issue in GPU architecture.

Partitioned design of GPUs allow for incredibly fast floating point operations. For exampleNvidia GTX 460 GPU can compute 900 GFLOP (Giga Floating Point Operations) a secondwhereas a similarly priced INTEL CPU, the Intel Core i5 760 can only compute around 50GFLOP a second. On the other hand, using all the processing power of a GPU is harder than aCPU because of the partitioned processor and memory design of GPUs. Another downside ofGPU is they are very fast at single precision floating point arithmetic but they are much slower atperforming double precision floating point operations.

Different software platforms are available for programming GPUs. CUDA language has similarsyntax and features with C and C++ languages and can only be used to program Nvidia GPUs.OpenCL is based on C99 (ISO/IEC 9899:1999 Standard of C language) and provides a generalframework which can be used for any type of processor not limited to GPUs. DirectCompute isanother language similar to C that can be used to program GPUs. DirectCompute is a part ofDirectX 11. C++ AMP is based on DirectCompute. Rather than implementing a new language,C++ AMP brings additions to C++ language that enables programming GPUs and CPUs togetherin C++ source code. PGI FORTRAN (Portland 2012) compilers have additional directivessimilar to OpenMP (OpenMP webpage) that are used to accelerate FORTRAN code using GPUs.

9.3 COMPUTER CLUSTER SYSTEM - CPU SOLUTION

9.3.1 Linear Dynamic Analysis with Implicit Time Integration

9.3.1.1 Dense Matrix Factorization and Load Vector Solution

Matrix factorization and load vector factorization steps are important steps for a vast majority ofalgorithms in finite element method. For example, linear dynamic analysis with implicit timeintegration depends on the factorization of matrix and solution of the load vector as presentedin Equation 9.1 (Newark 1959).

(9.1)

where

(9.2)


66

When the structure behaves linear, the effective stiffness matrix ( , Equation 9.2) is factorizedonce and the load vectors, right hand side of Equation 1, is solved repetitively for each time step.Performances of these two operations directly affect solution time of the whole analysis.

In order to test the performance of an implicit time integration algorithm, matrix factorizationand load vector solution times of the computer cluster system was measured. Factorization andload vector solution of dense matrices was performed utilizing the optimized Intel MKL libraries(Intel MKL webpage) in C++ language. In the implementation, a single machine with two IntelXeon processors each having four cores was utilized. These tests resemble operations required tocalculate the displacements for the following time step for a condensed structural model.

Table 9.1 Dense Matrix Factorization and Load Vector Solution Times.

Matrix Size Dense MatrixFactorization Times (ms)

Load VectorSolution Times (ms)

64 0.17 0.0109506128 0.48 0.0443713256 1.97 0.129416512 5.52 0.41143

1024 33.16 1.470082048 230.25 5.90484096 1558.10 23.3178

According to the results presented in Table 9.1 dense matrix factorization consumes significantamount of time when compared with the load factorization times. A single computer with dualCPU and eight cores can solve a load vector having a size of 1024 in 1.47 ms. The load vectorsolution time increases in O(n2) where n is the number of equations.

9.3.1.2 Speed of Matrix-Vector Multiplication

For structures modeled with equivalent viscous damping, the damping matrix must be multipliedwith effective velocities at each time step in order to compute the damping forces. For acondensed system, these operations are mainly dense matrix-vector multiplications. Thus, to testthe time consumed during these operations, a matrix-vector multiplication test was implementedin C++ language using Eigen matrix library. Eigen library has support for the latest Intelinstructions (SSE, SSE2, SSE3 and SSE4) thus can take advantage of the latest CPU technology.Tests were done using a single core, meaning no parallelism is used during computations in orderto get a picture of the raw power of the system.

Table 9.2 presents matrix-vector multiplication performance of a single CPU. The executiontimes are not significant up to a matrix size of 1024. After 1024 equations, matrix-vectormultiplication considerably slowed down due to inefficient cache memory utilization.


67

Table 9.2 Matrix-Vector Multiplication Times.

Matrix Size Time (ms)64 0.0032874

128 0.0131765256 0.0053218512 0.0240761

1024 0.05802782048 0.49931904096 1.9687600

Table 9.3 presents total timings for matrix vector multiplication and load vector solution timesfor the computer cluster system. This total case is a representation of the dynamic analysis of adamped system for a single time step. According to the results of Table 9.3, a condensedstructural model with 1024 equations can be solved in less than 2 ms with a single dual CPUcomputer.

Table 9.3 Total Solution Times.

Matrix Size Matrix-VectorMultiplication (ms)

Load Vector Solution(ms) Total (ms)

64 0.0032874 0.0109506 0.014238128 0.0131765 0.0443713 0.057548256 0.0053218 0.129416 0.134738512 0.0240761 0.41143 0.435506

1024 0.0580278 1.47008 1.5281082048 0.4993190 5.9048 6.4041194096 1.9687600 23.3178 25.28656

9.3.2 Linear Dynamic Analysis with Explicit Time Integration

9.3.2.1 Implementation

Explicit linear dynamic analysis was implemented using an explicit implementation of Newmarkdynamic algorithm. This explicit implementation was first implemented by (Hughes 1979).Explicit Newmark algorithm is based on approximation of the fundamental dynamic Equation(9.4) by central difference formulas (9.3).

(9.3)


68

(9.4)

Equation 2, known as the fundamental dynamic equation, relates dynamic external forces ( )to displacement ( ), velocity ( ) and acceleration ( ) of the system. In Equation 2, M is themass, C is the damping and K is the stiffness matrices of the system. Discretization of thisequation by the central difference equation (9.1) results in Equation 9.3.

(9.5)

For linear systems, internal forces can be computed as:

(9.6)

Using Equations 9.5 and 9.6, displacements at the following time step can be computed fromdisplacements at current step and displacements of the previous time step. Notice that left handside of the discretized equation does not involve the stiffness matrix of the model. If massmatrices of a structural are computed as lumped mass matrices, Equation 9.4 can be solvedelement by element without the need of assembling and solving the structural matrices. Thisimplementation detail makes this analysis approach very suitable for systems that are limited inmemory such as graphic cards and simplifies parallel implementations.

The linear dynamic analysis of a finite element model is performed using the above approach intwo stages. The first stage is the initialization stage. In this stage stiffness, mass, and dampingmatrices of each finite element are computed and stored in the memory for use in the secondstage. The initialization step is executed only once for the whole analysis. The second stage is thesolution stage. The solution stage uses element matrices computed in the first stage to computethe displacements utilizing Equation 5 for each time step.

Parallelism in the implementation is achieved using partitioning. Finite element model ispartitioned into n sub-domains using Metis library (Metis webpage) where n is taken equal to thenumber of CPU cores used in the solution. Every CPU core carries the computations belongingto their assigned sub-domain and at the end of every time step solutions from different CPUcores are combined using MPI functions.

9.3.2.2 Tests

For testing the implemented algorithm, a simple 2D square domain was modeled using differentmesh sizes. Quadrilateral plane stress elements were used in models (Figure 9.2). Models wereanalyzed and the time required for the initialization and solution stages were measured. Table 9.4presents the solution times for a single time step for models of different sizes. The presentedalgorithm finalizes each times step for a model with 40,000 elements in 9.64 ms using 16 cores.For models smaller than 800 elements fastest results are achieved using 4 cores and for modelsthan have between 800 and 5000 elements fastest results are achieved using 8 cores. Solutionstep durations for analyses using different number of cores are presented in Figure 4.


69

Table 9.4 Fastest Average Solution Times for a Single Time Step:

# Elements # Core Solution Time (ms)400 4 0.33800 4 0.55

1600 8 0.912500 8 1.125000 8 1.84

10000 16 3.4020000 16 5.2440000 16 9.64

Fig. 9.2 2D Square Domain Model

Fig. 9.3 Average Duration of a Solution for Single Time Step


70

9.4 COMPUTATION WITH GRAPHIC CARDS

9.4.1 Linear Dynamic Analysis with Implicit Time Integration

9.4.1.1 Dense Matrix Factorization and Back Substitution

In this case, dense matrices were solved utilizing NVIDIA's Tesla C2050 graphic cards. Table 3presents dense matrix factorization and load vector solution times for different number ofequations. In the GPU implementation of the matrix factorization, GPU optimized linear algebralibrary, CULA (CULA webpage), was used. All the computations were performed with doubleprecision floating point numbers.

Table 9.5 Dense Matrix Factorization and Load Vector Solution Times.

Matrix Size Dense MatrixFactorization Times (ms)

Load VectorSolution Times (ms)

64 0.91914 1.16062128 1.43922 1.60092256 4.26334 2.75443512 8.99159 5.15902

1024 22.5388 12.1372048 81.5738 34.06014096 397.029 109.881

For small matrices, not enough parallelization can be achieved thus, for matrices whose size aresmaller than 1024, the factorization speeds are slower than the CPU solution with 16 cores. Asthe size of the dense matrix gets larger, the large number of CUDA cores is more efficientlyutilized and the factorization times of larger matrices are faster than the CPU solution. On theother hand, parallelization during load vector solution is not very efficient. Even though havinglarger matrix sizes improves the parallelization of load vector solution, the solution times are stillmuch slower than the CPU solution. This situation is mainly due to the fact that memoryoperations take more time than the computations.

9.4.1.2 Speed of Matrix Vector Multiplication

Table 6 presents results of matrix-vector multiplication tests performed using Nvidia TeslaC2050 graphic card. As necessary parallelism cannot be achieved for small size matrices,graphic card performs poorly compared to CPU results until a matrix size of 2048. For a matrixsize of 4096, graphic cards outperform the CPU executing matrix vector multiplication in 1.657ms compared to 1.969 ms achieved with the CPU.


71

Table 9.6 Matrix Vector Multiplication Speed; GPU System.

Matrix Size Time (ms)64 0.070

128 0.081256 0.101512 0.194

1024 0.3352048 0.6574096 1.657

Table 9.7 presents total timings for matrix vector multiplication and load vector solution timesfor the GPU system. In the total timings load vector solution times consumes significant amountof time hence for all matrix sizes utilizing CPUs are much more efficient that graphic cards incase of an implicit integration algorithm. Tesla C2050 graphic card can only solve a condensedstructure with 128 equations less than 2 ms.

Table 9.7 Total Timings for Matrix Vector Multiplication and Load Vector Solution Times;GPU System.

Matrix Size Matrix-VectorMultiplication (ms) Load Vector Solution (ms) Total (ms)

64 0.070 1.16062 1.23062128 0.081 1.60092 1.68192256 0.101 2.75443 2.85543512 0.194 5.15902 5.35302

1024 0.335 12.137 12.4722048 0.657 34.0601 34.71714096 1.657 109.881 111.538

9.4.2 Linear Dynamic Analysis with Explicit Time Integration

9.4.2.1 Implementation

Implementation of the explicit linear dynamic analysis for graphic cards is similar to the onedesigned for CPUs. Same algorithm was utilized but the implementation details were differentfor compliance with the GPU architecture. Computational parts of the algorithm were rewrittenin CUDA language. CUDA implementation was designed to use a single thread for computationsrelated to each finite element of the model. Two staged analysis, consisting of the initializationand solution stages was retained. At the initialization phase, model information is first copiedfrom computer's memory (RAM) to the global memory of the GPU. Then, the element matricesare computed and saved in the same global memory. At the solution stage, results of the analysis


72

are computed using the GPU and then copied to computer's memory (RAM) at each time step.For all computations, double precision floating point numbers are used.

9.4.2.2 Results

GPU implementation was tested using the same models used that were used while testing thecomputer cluster system. Average durations of each time step for different model sizes arepresented in Figure 9.5. Since parallelism is achieved without any significant data transferamong CUDA cores, the computation time varied almost linearly with the size of the model.

Fig. 9.4 Average Duration of a Solution Step: GPU System

Table 9.8 compares the fastest average solution times achieved with the computer cluster systemto average solution step times achieved with the GPU implementation. GPU implementationperforms similarly to the computer cluster system. For models smaller than 1600 elements, thegap between the solution times of computer cluster and graphic card is larger since enoughparallelism cannot be achieved in the GPU for small size models. For larger models, theperformance of a single graphic card is comparable to performance of a 16-core solution.

Table 9.8 Comparison of Average Solution Step Times.

# elements Solution Step GPU (ms) Solution Step CPU (ms)400 1.02 0.33800 1.14 0.55

1600 1.26 0.912500 1.63 1.125000 2.52 1.84

10000 4.00 3.4020000 6.98 5.2440000 12.8 9.64


73

9.5 CONCLUSION

Graphic cards provide a fast and rather cheap method for performing scientific computations.The type of algorithm used and details of the implementation determine performance of graphiccards. For an algorithm that is mostly memory bounded and high data dependency, graphic cardsperform rather poorly which was the case for implicit time integration algorithms. On the otherhand, if GPU is saturated with computation as in the explicit linear dynamic analysis algorithm,they can perform comparable to expensive high performance computing systems. Thus, GPUbased explicit integration algorithms are good candidates for hybrid simulation environments.


74


75

References

Agrawal, B.N., Mode Synthesis Techniques for Dynamic Analysis of Structures, Journal of Acoust. Soc. Am., Vol59, No. 6, June 1976.

Bursi O.S., He L., Bonelli A., Pegon P., 2010, Novel generalized-α methods for interfield parallel integration ofheterogeneous structural dynamic systems, Journal of Computational and Applied Mathematics,doi:10.1016/j.cam.2009.08.082.

Bursi O.S., Xu, G., Abbiati G. 2011, Numerical and Experimental Validation of the PM Method coupled toOpenSEES Research Report, University of Trento.

Bonelli A., Bursi O.S., He L., Magonette G. and Pegon P., 2008, Convergence analysis of a parallel interfieldmethod for heterogeneous simulations with dynamic substructuring, International Journal for NumericalMethods in Engineering, DOI: 10.1002/nme.2285.

C++ AMP (C++ Accelerated Massive Parallelism) webpage http://msdn.microsoft.com/en-us/library/hh265136(v=vs.110).aspx, accessed January 2012. Microsoft, Visual Studio 11 Developer Preview

CUDA (Compute Unified Device Architecture) http://www.nvidia.com/object/cuda_home_new.html, accessedJanuary 2012. NVIDIA, CUDA 4.0

CULA library webpage http://www.culatools.com/, accessed January 2012. EM Photonics, CULA R12DirectCompute webpage, http://blogs.msdn.com/b/chuckw/archive/2010/07/14/directcompute.aspx, accessed

January 2012. Microsoft, DirectX 11Intel 2012 Intel MKL (Math Kernel Libraries) webpage 2012, www.intel.com/software/products/mkl/, accessedJanuary 2012. Intel, MKL 10.3IANA 2012 http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xml,

accessed January 2012.METIS Serial Graph Partitioning and Fill-reducing Matrix Ordering webpage (2012)

http://glaros.dtc.umn.edu/gkhome/views/metis, accessed January 2012. Karypis Lab, METIS 4.0.3Newmark N.M 1959, A Method of Computation for Structural Dynamics, Newmark, N. M. ASCE Journal of

the Engineering Mechanics Division, Vol. 85, No. EM3.NVIDIA 2012, Nvidia CUDA Programming Guide NVIDIA, Version 4.0OpenCL (Open Computing Language) webpage http://www.khronos.org/opencl/, accessed January 2012. Khronos

Group, OpenCL 1.1OpenPM (Open Multi-Processing) webpage http://openmp.org, accessed January 2012. OpenMP Architecture

Review Board, OpenMP 3.1OpenFresco webpage, https://nees.org/resources/858, accessed January 2012OpenSees webpage, http://opensees.berkeley.edu/index.php, accessed January 2012.Pegon P., Magonette G. 2002, Continuous PSD testing with non-linear substructuring: Presentation of a stable

parallel inter-field procedure, Technical Report 24I.02.167, E.C., JRC, ELSA, Italy.Perdahcioglu, D.A., Ellenbroek, M.H.M., Boer, A. , A Hybrid Design Optimization Method Using Enriched Craig-

Bampton Approach, The Sixteenth International Congress on Sound and Vibration, Krakow, 5-9 July, 2009.Portland 2012, Portland Group http://www.pgroup.com/ The Portland Group, PGI Visual Fortran 2010Rixen, D., A Dual Craig-Bampton Method for Dynamic Sub-structuring, Journal of Computational and Applied

Mathematics, 168, pp. 383-391, 2004.

http://msdn.microsoft.com/en-

http://www.nvidia.com/object/cuda_home_new.html

http://www.culatools.com/

http://blogs.msdn.com/b/chuckw/archive/2010/07/14/directcompute.aspx

www.intel.com/software/products/mkl/

http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xml

http://glaros.dtc.umn.edu/gkhome/views/metis

http://www.khronos.org/opencl/

http://openmp.org

http://opensees.berkeley.edu/index.php

http://www.pgroup.com/


76

Schellenber, A. Kim, H.K, Takahashi, Y. Fenves, G.L and Mahin, S.A 2009a, OpenFresco Framework for HybridSimulation: Installation and Getting Started Manual, Department of Civil and Environmental Engineering,University of California, Berkeley.

Schellenber, A. Kim, H.K, Takahashi, Y. Fenves, G.L and Mahin, S.A 2009b, OpenFresco Command LanguageManual, Department of Civil and Environmental Engineering, University of California, Berkeley.

Schellenber, A. Kim, H.K, Takahashi, Y. Fenves, G.L and Mahin, S.A 2009c, Framework for Hybrid Simulation:MATLAB Example, Department of Civil and Environmental Engineering, University of California, Berkeley.

T.J.R Hughes and W.-K.Liu 1979, Implicit-Explicit Finite Elements in Transient Analysis: Implementation andNumerical Examples, ASME Journal of Applied Mechanics and Engineering.

Top500 webpage http://top500.org/, accessed January 2012.Young, J., Primer on the Craig-Bampton Method, October 4, 2000.Wang, K.J., Keh-Chyuan, T., Wang, Wang, S.J., Cheng, W.C. and Yang, Y.S. 2007, ISEE: Internet-based

Simulation for Earthquake Engineering – Part II: The application protocol approach. Earthquake Engineeringand Structural Dynamics, Wiley InterScience

Wang, K..J. 2011, PNSE Platform for Networked Structural Experiment,http://ws2.ncree.org/~kjwang/NSEP/NSEP.html, accessed January 2012.

http://top500.org/


77

Appendix A: Example of a synchronous FCM and CGM

The programs listed below are an example for of synchronous FCM and CGM programs writtenin standard C for Unix like operating systems. Window users, should include “winsock.h”instead of “sys/socket.h”, “arpa/inet.h” and “unistd.h”, substitute “close( Socket)” by “closesocket( Socket )” and modify the OpenSocket function to match theWindows operating specifications.

#include <stdio.h> /* For printf() and fprintf() */#include <stdlib.h> /* For atoi() and exit( ) */#include <sys/socket.h> /* For socket(), bind() and connect() */#include <arpa/inet.h> /* For sockaddr_in and inet_ntoa() */#include <string.h> /* For memset() */#include <unistd.h> /* For close() */

/* login information related constants */#define __AP_ERROR 0#define __AP_LOGIN_REQUEST 1#define __AP_LOGIN_REPLY 2#define __AP_LOGIN_REPLY_NOTLOGINPACKET 0#define __AP_LOGIN_REPLY_UNRECOGNIZEDACCOUNT 1#define __AP_LOGIN_REPLY_WRONGPASSWORD 2#define __AP_LOGIN_REPLY_EXCEEDTRIALTIME 3#define __AP_LOGIN_REPLY_ALREADYLOGGEDIN 4#define __AP_LOGIN_REPLY_OK 5

#define __AP_END 2

/* NSEP related constants */#define NSEP_SETPUSH (__AP_END+1)#define NSEP_QUERY (__AP_END+2)#define NSEP_EXPINFO (__AP_END+3)#define NSEP_EXPSTATE (__AP_END+4)#define NSEP_CLNSTATE (__AP_END+5)#define NSEP_CMD (__AP_END+6)#define NSEP_CSIG (__AP_END+7)#define NSEP_IM (__AP_END+8)

/* NSEP client state constants */#define NSEP_CS_DISCONNECTED 0#define NSEP_CS_NOTREADY 1#define NSEP_CS_RUNNING 2#define NSEP_CS_FINISHED 3

/* Experimental State */#define NSEP_ES_NOTREADY 0#define NSEP_ES_RUNNING 1

http://ws2.ncree.org/~kjwang/NSEP/NSEP.html


78

#define NSEP_ES_INTERRUPTED 2#define NSEP_ES_FINISHED 3

typedef struct {char IP[20]; /*!< IP address (IPv4) of the server. */unsigned short int Port; /*!< Port that will be used for TCP/IP communication. */char Type[11]; /* Type of Server to connect: OpenFresco, PNSE or Custom */char Account_Name[10]; /*!< Account information */char Account_Password[10]; /*!< Account password */

} Remote_Machine_Info;

void Communicate_With_PNSE( const int WhatToDo, float Time,const float *const Data_To_Send,float *Data_To_Receive, const int Order );

void PrintErrorAndExit ( const char *ErrorMessage );void OpenSocket( const Remote_Machine_Info Server, int *sock );

int FACILITY_CONTROL_MODULE ( void ){

int i, j; /* Counters */int Is_Not_Finished;

int OrderSend, OrderRecv;float *Send, *Recv;

/* Initialise variables the controllers here* or do any other stuff */

...

/* Open the Socket, send the login information and change to running state */* WhatToDo = 0 */

printf( "Establishing connection with PNSE server.\n" );Communicate_With_PNSE( 0, 0.0, Send, Recv, 0 );

Is_Not_Finished = 1;while ( Is_Not_Finished ){

/* Send the NSEP_QUERY message asking for NSEP_CGM package. Store the* displacement in Recv. WhatToDo = 3 */Communicate_With_PNSE( 3, 0.0, Send, Recv, OrderRecv );

/* Send the displacement to the controller, get the data back and*/.. and store it in “Send” */

...

/* Send the NSEP_CSIG package. WhatToDo = 4 */Communicate_With_PNSE( 4, 0.0, Send, Recv, OrderSend );/* Send the NSEP_QUERY message asking for NSEP_EXPSTATE package. Store the* state in Recv. WhatToDo = 5 */Communicate_With_PNSE( 5, 0.0, Send, Recv, 0);if ( Recv[0] == -9999.0 ){

Is_Not_Finished = 0;}

}

/* Send the NSEP_CLNSTATE message saying that the process has finished.* WhatToDo = 6 */


79

Communicate_With_PNSE( 6, 0.0, Send, Recv, 0 );

/* Free dynamically allocated memory or do similar stuff */...

return 0;}

int COMMAND_GENERATION_MODULE ( void ){

int istep; /* Counter */int Nstep; /* Number of steps */float Time; /* Variable to store the time. Time = NStep*DeltaT, where

*DeltaT is the time increment between two consecutive steps */

/* TCP socket connection Variables */int Socket;int OrderSend, OrderRecv;float *Send, *Recv;

/* Initialise variables the controllers here* or do any other stuff */

.../* Open the Socket, send the login information and change to running state */* WhatToDo = 0 */

printf( "Establishing connection with PNSE server.\n" );Communicate_With_PNSE( 0, 0.0, Send, Recv, 0 );

/* Begin the integration process */istep = 1;while ( istep <= Nstep ){

/* Compute the new Displacement and store it in “Send” */...

/* Send the NSEP_CGM message containing the new displacement/* WhatToDo = 1*/Communicate_With_PNSE( 1, Time, Send, Recv, OrderSend );

/* Send the NSEP_QUERY message asking for NSEP_CSIG package. Store the* displacement in Recv. WhatToDo = 2 */Communicate_With_PNSE( 2, 0.0, Send, Recv, OrderRecv );

...

istep = istep + 1;}

/* Send the NSEP_CLNSTATE message saying that the process has finished.* WhatToDo = 6 */

Communicate_With_PNSE( 6, 0.0, Send, Recv, 0 );

/* Free dynamically allocated memory or do similar stuff */...

return 0;}


80

void Communicate_With_PNSE( const int WhatToDo, float Time,const float *const Data_To_Send,float *Data_To_Receive, const int Order )

{static Remote_Machine_Info Server;static int Length;static int NSEP_Type;static int Socket;static int Error_Code; /* To help identifying the error source */static int What;static int ExpState;

char Send_Buffer[MAXBUFLEN], Receive_Buffer[MAXBUFLEN], *pos;

/* Connect and login to the server and set the state to running */if ( WhatToDo == 0 ){

/* Get the information regarding the PNSE server: Account Name, Password* address and port and store it in the variable “Server” */

...

/* Connect to the PNSE server using the prev*/OpenSocket( Server, &Socket );

/* Login to the server. Send a __AP_LOGIN_REQUEST package*/Length = 8 + strlen( Server.Account_Name ) + 1 +

strlen( Server.Account_Password ) + 1;NSEP_Type = __AP_LOGIN_REQUEST;

pos = Send_Buffer;memcpy( pos, &Length, sizeof (int) );

pos = pos + sizeof (int);memcpy( pos, &NSEP_Type, sizeof (int) );

pos = pos + sizeof (int);memcpy( pos, Server.Account_Name,

(strlen( Server.Account_Name ) + 1)*sizeof (char) );

pos = pos + (strlen(Server.Account_Name ) + 1)*sizeof (char);memcpy( pos, Server.Account_Password,

(strlen(Server.Account_Password ) + 1)*sizeof (char) );

/* Send the __AP_LOGIN_REQUEST package through TCP/IP and check if there were* any problems with the communication */*/

if ( send( Socket, Send_Buffer, Length, 0) != (int) sizeof (char)*Length ){close( Socket );PrintErrorAndExit( " send(): sent a different number of bytes than

expected. Exiting." );}

/* Receive the __AP_LOGIN_REPLY message from the server and check if there were* any problems with the communication */if ( recv( Socket, Receive_Buffer, MAXBUFLEN, 0 ) <= 0 ){

close( Socket );PrintErrorAndExit( "Recv() failed or connection closed prematurely.

Exiting.\n" );} else {


81

/* Put the contents of the received message into the correct variables */memcpy( &Length, Receive_Buffer, sizeof (int) );memcpy( &NSEP_Type, Receive_Buffer + sizeof (int), sizeof (int) );memcpy( &Error_Code, Receive_Buffer + 2*sizeof (int), sizeof (int) );

/* Check if the login was not successful. If so, exit */if( NSEP_Type == __AP_LOGIN_REPLY && Error_Code != __AP_LOGIN_REPLY_OK ){

close( Socket );PrintErrorAndExit( "Login process not successful. Exiting." );

}}

/* Tell the PNSE Server to change the state to Running.* Run State = 2*/Send_Client_State( Socket, Send_Buffer, &Error, NSEP_CS_RUNNING );if ( Error == 1 ){

close( Socket );PrintErrorAndExit( "CGM: send() sent a different number of bytes than


} else if ( WhatToDo == 1 ) { /* Send the displacement to the server */

/* Compose the NSEP_CMD packet and send it to the server */Length = 12 + 4*Order;NSEP_Type = NSEP_CMD;


pos = pos + sizeof (int);memcpy( pos , &NSEP_Type, sizeof (int) );

pos = pos + sizeof (int);memcpy( pos, &Time, sizeof (float) );

pos = pos + sizeof (float);memcpy( pos, Data_To_Send, sizeof(float)*Order );if ( send( Socket, Send_Buffer, Length, 0 ) != (int) sizeof (char)*Length ){



} else if ( WhatToDo == 2 ) { /* Request a message from the server *//* Send the NSEP_QUERY packet to query for the NSEP_CSIG package */Length = 12;NSEP_Type = NSEP_QUERY;What = 9;


pos = pos + sizeof (int);memcpy( pos,& NSEP_Type, sizeof (int) );

pos = pos + sizeof (int);memcpy( pos ,&What, sizeof (int) );if ( send( Socket, Send_Buffer, Length, 0 ) != (int) sizeof (char)*Length ){


82



/* Wait for the NSEP_CSIG to arrive */printf("CGM: Waiting for NSEP_CSIG...\n" );

if ( recv( Socket, Receive_Buffer, MAXBUFLEN, 0 ) <= 0 ){close( Socket );PrintErrorAndExit( "CGM: recv() failed or connection closed prematurely.


/* Get the restoring force */memcpy( Data_To_Receive, Receive_Buffer + 2*sizeof(int),

sizeof (float)*Order );}

} else if ( WhatToDo == 3 ) { /* Request a message from the server (FCM) *//* Send the NSEP_QUERY packet to query for the critical signal */Length = 12;NSEP_Type = NSEP_QUERY;What = NSEP_CMD; /* NSEP_CMD = 8 */




close( Socket );PrintErrorAndExit( "FCM: send() sent a different number of bytes than


/* Wait for the NSEP_CMD to arrive */printf("FCM: Waiting for NSEP_CMD...\n" );

if ( recv( Socket, Receive_Buffer, MAXBUFLEN, 0 ) <= 0 ){close( Socket );PrintErrorAndExit( "FCM: recv() failed or connection closed prematurely.


/* Get the displacement */pos = Receive_Buffer + 2*sizeof( int );memcpy( &Time, pos, sizeof (float) );pos = pos + sizeof( float );memcpy( Data_To_Receive, pos, sizeof (float)*Order );printf("Time %e\t Data Received:%e\n", Time, Data_To_Receive[0]);

}} else if ( WhatToDo == 4 ) { /* Send the force to the server */

/* Compose the NSEP_CSIG packet and send it to the server */Length = 8 + 4*Order;NSEP_Type = NSEP_CSIG; /*NSEP_CSIG;*/


83


pos = pos + sizeof (int);memcpy( pos , &NSEP_Type, sizeof (int) );

pos = pos + sizeof (float);memcpy( pos, Data_To_Send, sizeof(float)*Order );if ( send( Socket, Send_Buffer, Length, 0 ) != (int) sizeof (char)*Length ){

close( Socket );PrintErrorAndExit( "FCM: send() sent a different number of bytes thanexpected. Exiting." );

}printf("%i\t%e\n", Send_Buffer[0], Data_To_Send[0] );

} else if ( WhatToDo == 5 ) {/* Ask the PNSE server for the experiment state */Length = 12;NSEP_Type = NSEP_QUERY;What = NSEP_EXPSTATE; /* NSEP_EXPSTATE */




close( Socket );PrintErrorAndExit( "FCM: send() sent a different number of bytes than


if ( recv( Socket, Receive_Buffer, MAXBUFLEN, 0 ) <= 0 ){close( Socket );PrintErrorAndExit( "FCM: recv() failed or connection closed prematurely.


/* Get the Experimental state */memcpy( &ExpState, Receive_Buffer + 2*sizeof(int), sizeof (int) );

}

if ( ExpState == NSEP_ES_FINISHED ){Data_To_Receive[0] = -9999.0;

}

} else if ( WhatToDo == 6 ) { /* Terminate Connection with the server */

/* Tell the PNSE Server to change the state to finished* Run State = 3*/Send_Client_State( Socket, Send_Buffer, &Error, NSEP_CS_FINISHED );if ( Error == 1 ){




84

close( Socket );

} else {close( Socket );fprintf( stderr, "CGM: Invalid WhatToDo value %i. WhatToDo should be between 0

and 3.\n", WhatToDo );}

}

void OpenSocket( const Remote_Machine_Info Server, int *Socket ){

struct sockaddr_in Server_Addr; /* Type for handling the connections (IP, port) */

if ( (*Socket = socket( AF_INET, SOCK_STREAM, 0 )) < 0 ){PrintErrorAndExit( "socket() failed" );

}

memset( &Server_Addr, 0, sizeof( Server_Addr ) );/* Specify the address family as internet */Server_Addr.sin_family = AF_INET;/* Set the server address */Server_Addr.sin_addr.s_addr = inet_addr( Server.IP );/* Set the server port */Server_Addr.sin_port = htons( Server.Port );

if ( connect( *Socket, (struct sockaddr *) &Server_Addr,sizeof( Server_Addr ) ) < 0 ){

close( *Socket );PrintErrorAndExit( "connect() failed." );

} else {printf("Successfully connected to the %s Server: %s on port %hi.\n",

Server.Type, Server.IP, Server.Port );}

}

void Send_Client_State( const int Socket, char *const Send_Buffer,int *Error_TCP, const int Run_State )

{int Length; /* Length of the message */int NSEP_Type; /* NSEP type of message */char *pos; /* Pointer to identify the position within the message */

Length = 12;NSEP_Type = NSEP_CLNSTATE;


pos = pos + sizeof (int);memcpy( pos, &NSEP_Type, sizeof (int) );

pos = pos + sizeof (int);memcpy( pos, &Run_State, sizeof (int) );if( send( Socket, Send_Buffer, Length, 0 ) != (int) sizeof (char)*Length ){

(*Error_TCP) = 1;}

}


85

void PrintErrorAndExit ( const char *ErrorMessage ){

fprintf( stderr, "%s.\n", ErrorMessage );fprintf( stderr, "Exiting program." );exit( EXIT_FAILURE );

}

seismic engineering research infrastructures for european … · 2012-05-22 · seismic engineering...

Documents