the legion project

8/7/2019 The Legion Project

1/23

The Legion Project

Key words: parallel processing, high performance, object-oriented, distributedsystems, metasystems, wide area, gigabit networks

Legion is an object-based, meta-systems software project at the University of Virginia.From the project's beginning in late 1993, the Legion Research Group`s goal has been ahighly useable, efficient, and scalable system founded on solid principles. We have beenguided by our own work in object-oriented parallel processing, distributed computing,and security, as well as by decades of research in distributed computing systems. Oursystem addresses key issues such as scalability, programming ease, fault tolerance,security, site autonomy, etc. Legion is designed to support large degrees of parallelism inapplication code and manage the complexities of the physical system for the user. Thefirst public release was made at Supercomputing '97, San Jose, California, on November17, 1997.

Legion is a work in progress: ourteam will not finish Legion but will create an "open"system that allows and actively encourages third-party development of applications, run-time library implementations, and core system components.

1.8 Release Notes -- 6/20/01The notable changes are:

We've changed the software's directory structure. This change is relevant todevelopers and system administrators, but not to users. It does not affect any tools
http://www.virginia.edu/http://legion.virginia.edu/team.htmlhttp://www.virginia.edu/http://legion.virginia.edu/team.html


2/23

or context space. Please note that you may need to update paths in makefiles orchange library paths. The complete Legion package now consists of fivepackages:

1. Core: This is the basic Legion package and the minimum for running aLegion system. It lets you start up and shut down Legion, work in context

space, run Legion security, etc.2. Software development kit(SDK): This contains development-orientedtools and libraries, such as the stub generator, Legion Grid library,LegionArray library, etc.

3. High-performance computing(HPC): The HPC module lets you run yourprograms in Legion. It contains PVM and MPI tools, the two-dimensionalFileObject interfaces, JobProxy and JobQueue objects, batch queue classand host object, and legion_run and legion_run_multi.

4. Extra: This adds functionality to the basic Legion package. It contains theround robin scheduler, simple k-copy class (SKCC), process controldaemon host objects, etc. It is not necessary, but it gives you more control

over your objects.5. Applications: The Apps package also extends the basic Legion package.

When first starting a new system, you will need to initialize the HPC, Extra, andApplications packages with the legion_init_HPC, legion_init_Extra, and

legion_init_Apps command-line tools.

This restructuring has meant that you now need to download and install OpenSSL on your own. Legion uses public key cryptography based on the RSA 2.0algorithm, as implemented by OpenSSL. You will need to download OpenSSL0.9.5 or higher from http://www.openssl.org. You'll need to untar, configure, and

compile it. Be sure that you set your $OPENSSL_INC and $OPENSSL_LIBvariables to the correct directory. Suggested values are:

(ksh or sh users)

export OPENSSL_INC=/includeexport OPENSSL_LIB=/lib

(csh users)

setenv OPENSSL_INC /includesetenv OPENSSL_LIB /lib

You can use the JobQueue, with the legion_nq, legion_manage_job, and

legion_manage_queue command-line tools, to start and monitor remote jobs.

You can edit information about your user profile and security settings withlegion_configure_profile. You can modify the implicit parameter set for your

current session with legion_modify_parameters.

Two new command-line tools, legion_skcc_set_class_vaults and

legion_skcc_set_defaults, let you set defaults for SKCC classes.
http://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_runhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_run_multihttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_run_multihttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_init_HPChttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_init_Extrahttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_init_Appshttp://www.openssl.org/http://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#jobqueuehttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_nqhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_manage_jobhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_manage_queuehttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_configure_profilehttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_modify_parametershttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_skcc_set_class_vaultshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_skcc_set_defaultshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_runhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_run_multihttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_init_HPChttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_init_Extrahttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_init_Appshttp://www.openssl.org/http://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#jobqueuehttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_nqhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_manage_jobhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_manage_queuehttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_configure_profilehttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_modify_parametershttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_skcc_set_class_vaultshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.8.html#legion_skcc_set_defaults


3/23

The list of supported platforms has changed. We don't have a working binary forthe SGI Workstations/IRIX 6.5 n64 build although we're working on it. We aredropping support for the x86/FreeBSD 4.2 platform, although we will consideradding it back in if someone needs it. We may be adding a T3E platform in thefuture. We are also not currently supporting Windows platforms. If you need any

of these platforms, please contact us [email protected].

1.7 Release Notes -- 10/27/00The notable changes are:

We've added simple K-copy classes (SKCC). This allows certain Legion objectsto use backup vaults to replicate their persistent state, in case their primary vaultcrashes or is unavailable when an object needs to reactivate. This makes it easierto tolerate host failures. There are four new commands associated with SKCC:legion_set_backup_vaults, legion_synch_vaults, legion_set_worm, and

legion_unset_worm.

We are now using OpenSSL to implement the RSA algorithm. Since the RSAREFpatent has expired, we can now export Legion abroad with full encryption.

The 1.7 release now includes a set of GUIs for Windows 2000 machines. TheseGUIs are collectively known as the Worldwide File Server (WWFS). The WWFSis a discrete set of applications that you download and install on your Windowsmachine. It connects your machine to an existing Legion net (such as NPACI-net)and lets you work in your context space. The WWFS binary package includesfour GUIs to let you work in Legion context space and an FTP daemon, whichuses standard ftp protocols to transfer files between context space and any ftpclient (Legin credentials and full security are always managed by the daemon).The binary package is available from Applied MetaComputing.

For NPACI-net users, we've added a web-portal for running Amber on our Legionweb browser. The portal works on both IE 4 and Netscape Communicator, but forbest results we'd suggest you use IE.

We've improved legion_run and legion_run_multi. We have added a probeobjects, which allows you to check your runs while they are executing and movefiles to and from the executing remote host(s). You can also start your jobs inblocking or nonblocking mode. For more information, please see the updatedlegion_run andlegion_run_multi FAQs.

We've added a new MPI tool, legion_mpi_probe. This tool allows you to check

your MPI runs. You can now use wildcards with legion_ls, legion_cp, and legion_rm. For

example, you could ask to remove all context names beginning with "Foo" byentering:

$ legion_rm Foo\*

Note that you need to escape the "*" character.
mailto:[email protected]:[email protected]:[email protected]://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_set_backup_vaultshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_synch_vaultshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_set_wormhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_unset_wormmailto:[email protected]://sirius.cs.virginia.edu/browserhttp://legion.virginia.edu/documentation/FAQ_run.2.htmlhttp://legion.virginia.edu/documentation/FAQ_run_multi.2.htmlhttp://legion.virginia.edu/documentation/FAQ_run_multi.2.htmlhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_mpi_probemailto:[email protected]://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_set_backup_vaultshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_synch_vaultshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_set_wormhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_unset_wormmailto:[email protected]://sirius.cs.virginia.edu/browserhttp://legion.virginia.edu/documentation/FAQ_run.2.htmlhttp://legion.virginia.edu/documentation/FAQ_run_multi.2.htmlhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_mpi_probe


4/23

We've added the ability to temporarily lock down individual objects or all of aclass's objects. This makes it easier to shut down a Legion system or performclass and system maintenance or upgrades. Thelegion_deactivate_object and

legion_deactivate_instances commands have a new -stay_down flag, which

causes the object or instances to inactive after it being successfully deactivated. It

can only be reactivated by legion_allow_activation. We've reworked the binding agents, to improve system caching. We've also

chagned the default configuration so that each host now has its own local bindingagent (either on or nearby the host). Objects that are started on a host with its ownbinding agent will automatically use that binding agent. You can also choose touse a specific binding agent during a login session. Once you've logged in, run thelegion_set_binding_agent tool to set or unset a binding agent for the session.

Binding agents and the Legion library have been improved to cache moreinformation. Caching now includes object interfaces, context names, and contents.Context information caching is only allowed for objects that export a

"context_contents_cacheble('YES')"attribute.

Upon login, we now cache some high-use objects' bindings and high-use contexts'LOIDs. These binding may become stale, so we have added a new tool,legion_refresh_local_cache, to refresh them on request. We advise refreshing

your cache if you notice a consistent delay of around thirty seconds before andafter commands respond.

Finally, we've improved the I/O library and updated the communication system(with a UDP communication-layer sliding window protocol) so that version 1.7 isremarkable faster, more scalable, and more flexible.

1.6.6 Release Notes -- 8/4/00The notable changes are:

You can use wildcards with legion_mpi_run's -in/-IN and -out/-OUT flags to

name groups of files to be used as input and output files. The following wildcardscan be used with -in/-out and -IN/-OUT:* match 0 or more characters? match any one character[-] match any character listed between the brackets (use these to specify a range

of characters)

\ treat the character as a literal For example, if you wanted to identify done.1, done.2, done.3 ... done.9 as

your inputs, you could use square brackets to identify them as a group: $ legion_mpi_run -n 2 -IN done.[0-9] /mpi/programs/mpiFoo

You can use wildcards on the command line or in an option file. They can only beused with file names, however, not with directories.

The legion_native_mpi_run command now has a -debug flag.
http://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_deactivate_objecthttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_deactivate_objecthttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_deactivate_http://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_allow%20activationhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_set_binding_agenthttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_update_attributeshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_refresh_local_cachehttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_deactivate_objecthttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_deactivate_http://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_allow%20activationhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_set_binding_agenthttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_update_attributeshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.7.html#legion_refresh_local_cache


5/23

A new command, legion_make_hostlist, lets you create a host list for

legion_mpi_run.

1.6.5 Release Notes -- 6/13/00This release contains bug fixes and updates for 1.6.4, most notably:

We have added three new commands for subcollections. A subcollection is acollection that is attached to a parent collection. The parent collection can query asubcollection for resource data. These commands arelegion_add_sub_collection, legion_remove_sub_collection, and

legion_list_sub_collections.

You can adjust a collection's polling frequency by setting thecollection_update_frequency_secs attribute on the collection object (use

legion_update_attributes). The default is currently 300 seconds.

There are several changes to the legion_run_multi command. The specification

file now takes -in/out/constant flags as well as -IN/OUT/CONSTANT. It also

uses pattern specification holders. Please see the man page for furtherinformation. One change that is not mentioned in the man page: the format forspecifying files for the CONSTANT variable in legion_run_multi has changed.

The old format was:

CONSTANT

The new format is:

CONSTANT

For example,

CONSTANT foo /home/my_files/foo_file

Note that the does not need to match the : in this case,

the program will copy the contents of/home/my_files/foo_file to a local file

and assign it the name foo.

The legion_link command has a -FC flag. This flag allows you to specify a

Fortran compiler.

1.6.4 Release Notes -- 3/21/00This release contains several bug fixes and improvements. Primary points are:

The legion_create_user command has new flags that allow you to specify a

new user id's password from the command line and to specify the new user'shome context space. The parameter is also now a full path, which can

be given as a relative or absolute path.
http://legion.virginia.edu/documentation/release_more.html#legion_make_hostlisthttp://legion.virginia.edu/documentation/release_more.html#legion_add_sub_collectionhttp://legion.virginia.edu/documentation/release_more.html#legion_remove_sub_collectionhttp://legion.virginia.edu/documentation/release_more.html#legion_list_sub_collectionshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_create_userhttp://legion.virginia.edu/documentation/release_more.html#legion_make_hostlisthttp://legion.virginia.edu/documentation/release_more.html#legion_add_sub_collectionhttp://legion.virginia.edu/documentation/release_more.html#legion_remove_sub_collectionhttp://legion.virginia.edu/documentation/release_more.html#legion_list_sub_collectionshttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_create_user


6/23

We have added new flags to the legion_mpi_runcommand. The new flags,

-in/-out/-stdin/-stdout/-stderr, -IN/-OUT/-STDIN/-STDOUT/-STDERR, and

-a/-A, give you more control over input and output data for your mpi program.

They resemble the legion_run flags.

There is a new -f flag forlegion_add_host_account. This allows you to set up

a mapping file that lists all of your Unix-Legion account mappings for that PCDhost.

There are new keywords available forlegion_run_multi: you can now specify

stdout/stderr/stdin for local file space.

When starting a PCD host object, once you have started the PCD host object and(if necessary) the accompanying vault, you must change the following filepermissions on the node that is actually running the PCD host.

o $LEGION_OPR should be set to 755

o $LEGION_OPR/LegionClass.config* should be set to 644

o $LEGION_OPR/BootstrapVaultOPR should be set to 777 (If your

bootstrap host is a PCD host)

o $LEGION_OPR/.OPA should be set to 777 (If the bootstraphost is not a PCD host)

These changes should be made by the Legion administrator.

A fully implemented object migration. The TCP version of Legion communication layer. Added intelligent switching between using UDP and TCP communication based

on message size and destination. Improved robustness of HostObject being making it more tolerant to

implementation cache failures.

1.6.3 Release Notes -- 1/13/00This is an upgrade of 1.6, and contains several bug fixes. The primary fixes are listedbelow.

There are several MPI-related changes:o In a secure net, the legion_mpi_register command now puts a new MPI

application's context name into the /home//mpi context

instead of/mpi EXCEPT when the command is run by admin or a guest

user (i.e., a user who isn't logged in). In an insecure net the newapplication's context name will continue to be placed in /mpi.

o The legion_mpi_run command's -p flag (which names a context to holdPIDs) is more flexible. Previously, only currently existing contexts couldbe used. The flag will now create a new context to hold PIDs.

o The MPI libraries have been renamed: previously they used the form

libmpi.a or -lmpi and now they use the form libLegionMPI.a or-lLegionMPI.
http://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_mpi_runhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_mpi_runhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_add_host_accounthttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_run_multihttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_run_multihttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_mpi_runhttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_add_host_accounthttp://legion.virginia.edu/documentation/tutorials/1.6.3/commands_all_1.6.4.html#legion_run_multi


7/23

There are also several small changes to the command-line utilities. All commandsnow have a -debug and -help flag. A new command, legion_set_vault,

migrates a Legion object to a specific vault.

1.6.2 Release Notes -- 11/1/99

This is an upgrade of 1.6, and contains several bug fixes. The primary fixes are listedbelow.

MPI is now integrated with vector create. This involved added two new options tothe legion_mpi_run option, -hf and -HF.

We've added two new command-line tools: legion_mkdir and legion_cd. These

two commands perform exactly the same functions as legion_context_create

and legion_set_context, respectively.

The legion_ping tool now has a -timeout flag, which allows you to set a

timeout period for pinging a Legion object. The Legion libraries (libLegion1 and libLegion2) now use version numbers.

The performance of passing messages in secure Legion systems has been greatlyimproved.

1.6 Release Notes -- 8/27/99

We have added two new platforms to the 1.6 release: a beta release of WindowsNT 4.0 and a FreeBSD 3.0 for x86 machines.

We have also added tools for buildingvirtual hosts. This allows you to runprograms on unsupported machines, such as a Cray T3E.

The 1.6 release has tools to help debug and analyze Legion applications(legion_recordandlegion_replay).

We've added the support for operating in environments that require Kerberosauthentication.

We have added thelegion_export_dir tool, which lets you link a local

directoryto your context space. We have also added acheckpointing library for SPMD-style (Single Program

Multiple Data) applications to deal with MPI application failure. The legion_check_system tool has two new flags which will report errors in

command-line or context objects and then destroy the erring objects. We have updated the legion_run_multi tool, so that you can use keywords to

specify an input/output file's location.

1.5.15 Release Notes -- 5/26/99This is an upgrade of 1.5, and contains several bug fixes. The primary fixes are listedbelow.

Host objects: We've put in a bug fix in the host object restart code, so that hostobjects will restart more reliably. This is especially relevant for multi-hostsystems, although it will apply to all Legion systems.
http://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.2.html#legion_mpi_runhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.2.html#legion_mkdirhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.2.html#legion_cdhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.2.html#legion_pinghttp://legion.virginia.edu/documentation/tutorials/1.6/manuals/sysadmin_1_6.10.htmlhttp://legion.virginia.edu/documentation/tutorials/1.6/manuals/sysadmin_1_6.10.htmlhttp://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.10.htmlhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_recordhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_replayhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_replayhttp://legion.virginia.edu/documentation/tutorials/1.6/manuals/sysadmin_1_6.8.html#pgfId=527554http://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_export_dirhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_export_dirhttp://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.9.html#pgfId=528096http://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.9.html#pgfId=528096http://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.9.html#pgfId=528096http://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.e.html#pgfId=439097http://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.e.html#pgfId=439097http://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_check_systemhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_run_multihttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.2.html#legion_mpi_runhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.2.html#legion_mkdirhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.2.html#legion_cdhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.2.html#legion_pinghttp://legion.virginia.edu/documentation/tutorials/1.6/manuals/sysadmin_1_6.10.htmlhttp://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.10.htmlhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_recordhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_replayhttp://legion.virginia.edu/documentation/tutorials/1.6/manuals/sysadmin_1_6.8.html#pgfId=527554http://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_export_dirhttp://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.9.html#pgfId=528096http://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.9.html#pgfId=528096http://legion.virginia.edu/documentation/tutorials/1.6/manuals/BasicUser_1_6.e.html#pgfId=439097http://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_check_systemhttp://legion.virginia.edu/documentation/tutorials/1.6/commands_all_1.6.html#legion_run_multi


8/23

MPI: 1.5.15 has MPI-2 conversion functions for C and Fortran interoperability. Italso has a bug fix forMPI_pack().

Security: We've put in a bug fix for thelegion_init_security command-line

utility. It should now run much more reliably.

1.5 Release Notes -- 4/30/99Version 1.5 includes an updated GUI, several new command-line tools, improvedresource management, and the ability to connect Legion systems together to form larger,multi-domained systems. We have also added a batch queue host object for runningLegion on local queueing systems and a process control daemon (PCD) host object forbetter control over process ownership. To improve parallel application performancewe've added two-dimensional file interfaces.

1.2 Release Notes -- 7/7/98Version 1.2 offers support for remote execution of arbitrary programs, from either thecommand line or from the GUI, using the new legion_runcommand. This includes the

ability to run serial programs with multiple input files and mulitple executions, with thelegion_flogger command. We have also improved the GUI, which can be run from the

command line or from Windows95.

1.0 Release Notes -- 2/9/98The February 9, 1998 release contains bug fixes of the previous release (December 16,1997). No new features have been added.
http://legion.virginia.edu/documentation/tutorials/1.5/commands_all_1.5.html#legion_init_securityhttp://legion.virginia.edu/documentation/tutorials/1.5/commands_all_1.5.html#legion_init_securityhttp://legion.virginia.edu/documentation/tutorials/commands_all_1.2.html#legion_runhttp://legion.virginia.edu/documentation/tutorials/commands_all_1.2.html#legion_runhttp://legion.virginia.edu/documentation/tutorials/commands_all_1.2.html#legion_floggerhttp://legion.virginia.edu/documentation/tutorials/1.5/commands_all_1.5.html#legion_init_securityhttp://legion.virginia.edu/documentation/tutorials/commands_all_1.2.html#legion_runhttp://legion.virginia.edu/documentation/tutorials/commands_all_1.2.html#legion_flogger


9/23

Legion OverviewThe wide-area virtual environment of the future

As computer networks are get larger, faster, and more powerful, they offer newopportunities. Gigabit networks, connecting powerful high-performance machines and

workstations, have enormously powerful infrastructures that can solve complex problemsand distribute huge amounts of information. Linked together, these connected resourcesmake up a single, worldwide, virtual computer. We now need easy-to-use software thatcan manage a complex physical system and support large degrees of parallelism so that avirtual computer becomes a reliable, efficient, and real opportunity for a wide variety ofusers.

Legion, an object-based metasystems software project at the University of Virginia, isdesigned for a system of millions of hosts and trillions of objects tied together with high-speed links. Users working on their home machines see the illusion of a single computer,with access to all kinds of data and physical resources, such as digital libraries, physical

simulations, cameras, linear accelerators, and video streams. Groups of users canconstruct shared virtual work spaces, to collaborate research and exchange information.This abstraction springs from Legion's transparent scheduling, data management, faulttolerance, site autonomy, and a wide range of security options.

As new requirements and new opportunities for distributed computing emerge and futureusers make unforeseen demands on resources and software, the demands placed on avirtual computer will evolve and grow. What works today or even tomorrow will soon beworse than useless, and we strongly believe that Legion should be a flexible tool that canadapt to new needs. Legion is therefore an open system, designed to encourage thirdparty development of new or updated applications, run-time library implementations, and

core components.

Legion sits on top of the user's operating system, acting as liaison between its own host(s)and whatever other resources are required. The user isn't bogged down with time-consuming negotiations with outside systems and system administrators, since Legion'sscheduling and security policies act on his or her behalf. Conversely, it can protect its


10/23

own resources against other Legion users, so that administrators can choose appropriatepolicies for who uses which resources under what circumstances. To allow users to takeadvantage of a wide range of possible resources, Legion offers a user-controlled namingsystem called context space, so that users can easily create and use objects in farflungsystems. Users can also run applications written in multiple languages, since Legion

supports interoperability between objects written in multiple languages.

Legion Objectives and Constraints

There are ten design objectives listed here, and three constraints. They were laid outbefore any Legion code was written and have been carefully considered at each stage.

Objectives

Site autonomy

Legion is not a monolithic system, but is composed of resources owned andcontrolled by a variety of organizations. Since these organizations require controlover their own resources, Legion can not dictate how much of a particular

resource can be used, when it can be used, or who can use it. Extensible core

We cannot predict all of the needs of current and future users. We must buildLegion with extensible and replaceable components that permit Legion to evolveover time and allow users to construct their own mechanisms and policies.

Scalable architecture

Legion cannot rely on a centralized structure. If the system is to eventuallyencompass millions of hosts, it must use a scalable architecture.

Easy-to-use, seamless, computational environment

We must mask the complexity of the hardware environment and thecommunications synchronization of parallel processing. Users should not be

aware of machine boundaries. Compilers, in cooperation with run-time facilities,should manage the environment.

High-performance via parallelism

We must support easy-to-use parallel processing by means of large degrees ofparallelism (this includes task and data parallelism and their arbitrarycombinations).

Single, persistent, name space

One of the most significant obstacles to wide-area parallel processing is the lack


11/23

of a single name space for file and data access. Existing multiple disjoint namespaces makes writing applications for multiple sites very difficult. Legiontherefore uses a single, persistent, name space.

Security for users and resource owners

We cannot replace existing host operating systems (see below), but we can ensure

that existing mechanisms are not weakened by Legion. Legion does not define asecurity policy or requires a "trusted" Legion, but offers mechanisms for users tomanage their own security needs.

Management and exploitation of resource heterogeneity

Legion must support interoperability between heterogeneous hardware andsoftware components, as well as exploit architectural strengths as possible whenmaking scheduling decisions and policy.

Multiple language support and interoperability

Legion applications will be written in a variety of languages, and heterogeneoussource language application components must be integrated. We must alsosupport legacy codes.

Fault toleranceAt any given moment in a large system, several hosts, communication links, anddisks will fail. Legion must be able to handle their failures and dynamicreconfiguration.

Constraints

We cannot replace host operating systems

Organizations cannot allow their operating systems to be replaced. That wouldrequire rewriting applications and retraining users, as well as raising compatibilityproblems with other machines in the organization.

We cannot legislate changes to the interconnection networkWe must assume that network resources and protocols currently in use will notchange. While this means accommodating operating system heterogeneity, wemust accept the available resources.

We cannot require the Legion run as "root"

To protect their resources, most users will want to run Legion with the fewestpossible privileges.

Legion ApplicationsAdapting and Running Applications


12/23

Legion aims to provide an easy-to-use environment in which users have access to all theresources of the worldwide metacomputing environment for their applications. The fourapplication examples illustrated here show how the advanced features of Legion, such asflexible security and transparent file access, can be used to extend today's applications onto larger sets of resources.

Transparent Remote Execution

Legion allows programs to execute transparently and securely on remote hosts, takingadvantage of Legion's distributed resources. Users can quickly and easily adapt theirprograms to run in Legion, although modification may not even be necessary. Legionalso includes a remote "make" tool, which compiles binaries for other machinearchitectures without requiring the user to log in to another machine.

Parameter Space Studies

Not all applications are best suited to simple remote execution, so we have extended thiscapability to attack the class of problems known as parameter space searches. Forexample, the NASA NAS effort demands thousands of combinations of CFDcomputations, using a variety of wing designs, wing angles, and air speeds. A singlecomputation requires running five programs in sequence, each reading the output filesgenerated by the previous program. No individual program consumes significant CPUtime, but the total CPU time consumed by all runs adds up to tens of thousands of hours.

We have written a simple tool to facilitate running parameter space studies under Legion.Users can specify input and output files, the maximum number of jobs to run on aparticular host, and the total number of jobs to be run at a time. Once the initial jobs have

started on each host, future jobs are sent to the hosts that have finished previous runs.This dynamic scheduling allows for load balancing and faster processing time.

Wide-Area Parallel Applications

Some existing parallel applications run more efficiently on more CPUs than are availableon a single machine. For example, a DSMC (Direct Simulation Monte-Carlo) code in useat the University of Virginia to study vapor deposition onto surfaces has a low ratio ofcommunications to computation, so that it can run on a large number of widely separatedmachines. This approach allows the solution of much larger problems, but can causedifficulties with some conventional parallel tools. Most vendor-supplied versions of MPI,

a popular communications library, cannot be parallelized on multiple supercomputers.Successfully running this type of problem requires overcoming MPI's limitations as wellas transparent access to files and appropriate scheduling support. Legion provides thesecapabilities today.

Another example of a wide-area parallel application that can benefit from Legion'scollection of metacomputing resources is neural network modeling. Some types ofmodels use relatively low amounts of communications and are limited only by the


13/23


14/23

Users can provide their own classes: Legion allows users to define and buildtheir own class objects; therefore, Legion programmers can determine and evenchange the system-level mechanisms that support their objects. Legion 1.4 (andfuture Legion systems) contains default implementations of several useful typesof classes and metaclasses. Users will not be forced to use these implementations,

however, particularly if they do not meet the users' performance, security, orfunctionality requirements. Core objects implement common services: Legion defines the interface and basic

functionality of a set of core object types that support basic system services, suchas naming and binding, and object creation, activation, deactivation, and deletion.Core Legion objects provide the mechanisms that classes use to implementpolicies appropriate for their instances. Examples of core objects include hosts,vaults, contexts, binding agents, and implementations.

The Model

Legion objects are independent, logically address-space-disjoint active objects thatcommunicate with one another via non-blocking method calls that may be accepted inany order by the called object. Each method has a signature that describes the parametersand return value, if any, of the method. The complete set of method signatures for anobject fully describes that object's interface, which is determined by its class. Legionclass interfaces can be described in an interface description language (IDL), several ofwhich will be supported by Legion.

Legion implements a three-level naming system. At the highest level, users refer toobjects using human-readable strings, called context names. Context objects map contextnames toLOIDs (Legion object identifiers), which are location-independent identifiers

that include an RSA public key. Since they are location independent, LOIDs bythemselves are insufficient for communication; therefore, a LOID is mapped to anLOA(Legion object address) for communication. An LOA is a physical address (or set ofaddresses in the case of a replicated object) that contains sufficient information to allowother objects to communicate with the object (e.g., an pair).

Legion will contain too many objects to simultaneously represent all of them as activeprocesses. Therefore, Legion requires a strategy for maintaining and managing therepresentations of these objects on persistent storage. A Legion object can be in one oftwo different states, active orinert. An inert object is represented by an OPR (objectpersistent representation), which is a set of associated bytes that exists in stable storage

somewhere in the Legion system. The OPR contains state information that enables theobject to move to an active state. An active object runs as a process that is ready to acceptmember function invocations; an active object's state is typically maintained in theaddress space of the process (although this is not strictly necessary).

Core objects


15/23

Several core object types implement the basic system-level mechanisms required by allLegion objects. Like classes and metaclasses, core objects are replaceable systemcomponents; users (and in some cases resource controllers) can select or implementappropriate core objects.

Host objects: Host objects represent processors in Legion. One or more hostobjects run on each computing resource that is included in Legion. Host objectscreate and manage processes for active Legion objects. Classes invoke themember functions on host objects in order to activate instances on the computingresources that the hosts represent. Representing computing resources with Legionobjects abstracts the heterogeneity that results from different operating systemshaving different mechanisms for creating processes. Further, it provides resourceowners with the ability to manage and control their resources as they see fit.

Vault objects: Just as a host object represents computing resources and maintainsactive Legion objects, a vault object represents persistent storage, but only for thepurpose of maintaining the state, in OPRs, of the inert Legion objects that the

vault object supports. Context objects: Context objects map context names toLOIDs, allowing users to name objects with arbitrary high-level string names, andenabling multiple disjoint name spaces to exist within Legion. All objects have acurrent context and a root context, which define parts of the name space in whichcontext names are evaluated.

Binding agents: Binding agents are Legion objects that map LOIDs to LOAs. A pair is called a binding. Binding agents can cache bindings andorganize themselves in hierarchies and software combining trees, in order toimplement the binding mechanism in a scalable and efficient manner.

Implementation objects: Implementation objects allow other Legion objects torun as processes in the system. An implementation object typically contains

machine code that is executed when a request to create or activate an object ismade; more specifically, an implementation object is generally maintained as anexecutable file that a host object can execute when it receives a request to activateor create an object. An implementation object (or the name of an implementationobject) is transferred from a class object to a host object to enable the host tocreate processes with the appropriate characteristics.


16/23

Summary

Legion specifies functionality and interfaces, not implementations. Legion 1.4 providesuseful default implementations of class objects and of all the core system objects, butusers are never required to use our implementations. In particular, users can select (orbuild their own) class objects, which are empowered by the object model to select orimplement system-level services. This feature of the system enables object services (e.g.creation, scheduling, security) to be appropriate for the object types on which theyoperate, and eliminates Legion's dependence on a single implementation for its success.

Legion High PerformanceParallelism and resource selection

Legion achieves high-performance computing by selecting resources based on load andjob affinity and through parallel processing.

High performance via resource selectionEven single task jobs can get better performance when presented with a range ofpossible execution sites. The user can, for example, choose the host with thelowest load or the greatest power. Power, in this context, might be determined byfactors such as performance on the Spec benchmakrs adjusted for load or with theapplication itself as a benchmark. Either way, Legion's flexible resourcemanagement scheme lets user-level scheduling agents choose the right resource.


17/23

High performance via parallelismParallel processing has been around for some time, on both tightly cupled MPPsand on workstation and PC clusters. Legion supports a distributed mememoryparallel computing model, but since Legion's objects are often on different hosts,perhaps thousands of miles apart, communication overhead can run from single

digit milliseconds to tens of milliseconds. The result is that Legion is notappropriate for fine-grain parallel programs.

Legion can be used for parallel processing in a variety of application styles. It canexecute a single application across geographically separate hosts or support meta-applications (e.g., schedule the components of a single meta-application on thenodes of an MPP).

Legion supports parallel processing in four ways:

1. Supporting popular parallel libraries, such as MPI

2. Supporting parallel languages, such as MPL3. Offering wrap parallel components4. exporting the run-time library interface to library, toolkit, and compiler writers

Support of parallel libraries

The vast majority of parallel applications today are written in MPI and PVM. Legion

supports both libraries, via emulation libraries that use the underlying Legion run-timelibrary. Existing applications only need to be recompiled and relinked in order to run onLegion. MPI and PVM users can thus reap the benefits of Legion with existingapplications. In the future, libraries such as Scalapak will also be supported.

Parallel language support


18/23

Legion supports MPL (Mentat Programming Language) and BFS (Basic FortranSupport). MPL is a parallel C++ language in which the user specifies those classes thatare computationally complex enough to warrent parallel execution. Class instances arethen used like C++ class instances: the compiler and run-time system take over andconstruct parallel computation graphs of the program and then execute the methods in

parallel on different processors. Legion is written in MPL: BFS is a set of pseudo-comments for Fortran and a preprocessor that gives the Fortran programmer access toLegion objects. It also allows parallel execution via remote asynchronous procedure callsand the construction of program graphs. HPF may also be supported in the future.

Wrap parallel components

Object wrapping is a time-honored traditiion in the object-oriented world. We haveextended the notion of encapsulating existing legacy codes into objects by encapsulatingparallel components into objects. To other Legion objects the encapsulated object appearssequential but it executes faster. PVM, HPF, and shared memory threaded applications

can thus be encapsulated into a Legion object.

Export the run-time library

We do not expect to provide the full range of languages and tools that users require:instead of developing everything here at the University of Virginia, we anticipate Legionbecoming an open, community, artifact, to which other tools and languages are ported.To support these third party developments, the complete run-time library is available.User libraries can directly manipulate the run-time library.

The library is completely reconfigurable. It supports basic communication,encryption/decryption, authentication, and exception detection and propagation, as wellas parallel program graphs. Program graphs represent functions and are first class andrecursive. Graph nodes are member function invocations on Legino objects or sub-


19/23

graphs. Arcs model data dependencies. Graphs may be annotated with arbitraryinformation, such as resource requirements, architecture affinities, etc. The annotationsmay be used by schedulers, fault-tolerance protocols, and other user-defined services.

Legion SchedulingApplication-level scheduling and total site autonomy

Philosophy

The Legion scheduling philosophy is one of reservation through a negotiation process

between resource providers and resource consumers. We view autonomy as the singlemost crucial aspect of this process.

Site autonomy is crucial in attracting resource providers. In particular,participating sites must be assured that their local policies will be respected by thesystem at large. Therefore, final authority over the use of a resource is placed withthe resource itself.

User autonomy is crucial to achieving maximum performance. A singlescheduling policy will not be the best answer for all problems and programs:rather, users should be able to choose between scheduling policies, and select theone which best fits the problem at hand or, in the extreme, provide their own

schedulers. A special, and vitally important, case of user-provided schedulers isthat of application-level scheduling. This allows users to provide per-applicationschedulers that are specially tailored to match the needs of the application.Application-level schedulers will be commonplace in high-performancecomputing domains.

To paraphrase the 1996 Presidential election campaign, "It's the autonomy, stupid!"

Model

Legion presently provides two types of resources: hosts (computational resources) and

vaults (storage resources). We will incorporate network resources in the future. As seenbelow, the Legion scheduling module consists of three major components: a resourcestate information database, a module which computes request (object) mapping toresources (hosts and vaults), and an activation agent responsible for implementing thecomputed schedule. We call these items the Collection, Scheduler, and Enactor,respectively.


20/23

The Collection interacts with resource objects to collect state information describing thesystem (step 1). The Scheduler queries the Collection to determine a set of availableresources that match the Scheduler's requirements (step 2). After computing a schedule,or set of desired schedules, the Scheduler passes a list of schedules to the Enactor forimplementation (step 3). The Enactor then makes reservations with the individual

resources (step 4), and reports the results to the Scheduler (step 5). Upon approval by theScheduler, the Enactor places objects on the hosts, and monitors their status (step 6).

If the user does not wish to select or provide an external scheduler, the Legion system(via the class mechanism) provides default scheduling behavior supplying general-purpose support. Through the use of class defaults, sample schedulers, and application-level schedulers, the user can balance the effort put into scheduling against the resultingapplication performance gain.

Features in 1.4

Resource reservations for Host and Vaults. Collection objects providing resource information for schedulers, using data

collection agents that push information. Enactor objects to implement schedules, by obtaining resource reservations and

starting objects. Support for application-level, per-object schedulers. Per-class default external schedulers and placements (these may be overridden at

user's behest). Intelligent scheduling for stateless objects, which balances the workload across

available hosts. A pull model for Collection data gathering will be added in future releases, as

well as additional monitoring support and sample schedulers.


21/23

Legion SecuritySecurity is built into the system from the beginning

Philosophy

Legion is a software infrastructure that unites large collections of heterogeneouscomputing resources into single, coherent systems. With Legion, users can accessscattered resources easily, share data and computing power, and build new meta-applications running across the network. While these possibilities are attractive, userswill only adopt Legion if they feel confident that it will protect the privacy and integrityof their existing resources as well as the new resources they create within Legion.Without security, Legion systems can offer some limited uses. But for the full Legionvision of large-scale metacomputers to become a reality, security is essential.

Recognizing this fact, we have made security a part of the Legion design from thebeginning. There are five main requirements that must be satisfied:

Do no harm. The installation of Legion at a site should not compromise that site'ssecurity policies and goals. In general, Legion must not allow unauthorized accessto system resources, where resources can be broadly defined to range from userfiles to root privileges.

Adapt to local policies. In concert with the first requirement, Legion must beconfigurable to the security needs of different organizations. Of course, morestringent security constraints generally exact a price in performance and ease ofuse.

Provide an access control framework. All local resources are represented inLegion as objects, and the fundamental Legion resource is the ability to call a

method on an object. Objects must have flexible access control mechanisms forauthorizing and denying method calls.

Maintain and protect identities. Objects and users have identities that can be usedto independently authenticate and authorize one another. These identities arerepresented through private keys and signed credentials of various types. In adistributed object system, it is often necessary to delegate authority to otherobjects. Legion should not only protect identities from theft and spoofing, but alsominimize the dispersion of authority that delegation causes.

Protect communication. A Legion system may span public or semi-publicnetworks. Objects must be able to communicate with guaranteed integrity andprivacy as needed. Message replay must be detected and prevented.

Model

The security model for Legion differs significantly from that of conventional systems. ALegion "system" is really a federation of resources from multiple administrative domains,each with its own separately evaluated and enforced security policies. As such, there is nocentral kernel or trusted code base that can monitor and control all interactions between


22/23

users and resources. Nor is there the concept of a superuser--no one person or entitycontrols all of the resources in a Legion system.

Legion programs and objects run on top of host operating systems, in user space. Theyare thus subject to the policies and administrative control of the local OS, and the Legion

objects running on a particular host must trust that host. However, there is no requirementfor Legion objects to trust other Legion objects. A critical aspect of Legion security isthat the security of the overall system does rely on every host being trustworthy. A largeLegion system will include multiple trust domains, and even within one trust domain,some of the hosts may be compromised or may even be malicious. For example, twoorganizations might use Legion to share certain resources in specifically constrainedways. Such sharing would clearly not be acceptable if one organization could subvert theother's objects through its ownership of some part of a Legion system.

These aspects of Legion allow for considerable flexibility in the security policiesassociated with various Legion objects, which in turn provides the foundation for

satisfying the first two security requirements. For example, local policy may requireKerberos authentication, audit logs, resource usage accounting, encapsulation of criticalsecurity functionality in small, easily vetted programs, etc. Any of these features can beimplemented without departing from the overall model and the minimal assumptions thatare made between Legion objects.

Access control for Legion objects first requires that the user determine the security policyfor an object by defining the object's rights and the method calls they allow. Accesscontrol is then supported via a special member function called MayIpresent in everyobject. MayI is Legion's traffic cop: All method calls to an object must first pass throughMayI before the target member function is invoked. Only if the caller has the appropriate

rights for the target method will MayI allow that method invocation to proceed. Thefigure below shows a call from object A to object B.

To make rights available to a potential caller, the owner of an object gives it anunforgeable credential that lists the rights granted. When the caller invokes a method onthe object, it presents the appropriate credential to MayI, which then checks the scope andauthenticity of the credential. Alternatively, the owner of an object can semipermanently


23/23

assign a set of rights to a particular caller or group. MayI's responsibility is then toconfirm the identity of the caller and its membership in one of the allowed groups,followed by comparing the rights authorized with the rights required for the method call.

The means for establishing identity in Legion also address the requirement for protecting

communications between objects. Every Legion object has a public key pair; the publickey is part of the object's name. Objects can use the public key of a target object toencrypt their communications to it. Likewise, an object's private key can be used to signmessages, providing authentication and nonrepudiation. The integration of public keysinto object names allows Legion to avoid the need for a certification authority (althoughsuch an authority is still useful for establishing user identities). If an intruder tries totamper with the public key of a known object, it will create a new name that is unknown.

The combined components of the security model encourage the creation of a large-scaleLegion system with multiple overlapping trust domains. Each domain can be separatelydefined and controlled by the users it affects. When difficult problems arise such as

merging two trust domains, Legion provides a common and flexible context in whichthey can be resolved.

Security Features

Public-key cryptography based on RSAREF 2.0. Three message-layer security modes: private (encrypted communication),

protected (fast digested communication with unforgeable secrets to ensureauthentic replies to message calls), and no security.

Caching secret-keys for faster encryption of multiple messages betweencommunicating parties.

Auto-encrypted bearer credentials with free-form rights. Propagation of securitymodes and certificates through calling trees (e.g., if a caller demands encryption,all downstream calls will use it automatically).

Drop-in addition of MayI functionality to existing objects. Persistent authentication objects that serve as the representation for users in a

trust domain. Secure legion shell to allow users to login to their authentication objects and

obtain associated credentials and environment information. Isolation and protection of objects using local OS accounts. Easily checked Process Control Daemon for granting limited OS privileges to

Legion Host Objects.

Context space configured with access control for multiple users.

the legion project

Documents