condor gotchas iii

22nd Oct 2008 Euro Condor Week 2008Barcelona

1

Condor Gotchas III

John KewleySTFC Daresbury [email protected]


2

OutlineIn this talk we will look at a few aspects of

condor that frequently cause problems for new users.

Custom attributes / ClassAdsArguments and Environment variablesConnectivity (especially Firewalls)


3

What is a "gotcha"

From Wikipedia: "A 'gotcha' is a feature of a system, a program or a programming language that works in the way it is documented, but is counter-intuitive and almost invites mistakes because it is both enticingly easy to invoke and completely unexpected and/or unreasonable in its outcome."


4

Custom attributesHow do I declare my own attributes so my jobs can distinguish between Red Hat 9 and other Linuxes?

On Red Hat 9 machines add the following to its configuration:OPSYS_FLAVOUR = RH9

OPSYS_FLAVOUR = "RH9"

STARTD_EXPRS = OPSYS_FLAVOUR

STARTD_ATTRS = OPSYS_FLAVOUR, $(STARTD_ATTRS)

It can then be matched with a REQUIREMENTS statement:REQUIREMENTS = (OPSYS_FLAVOUR = RH9)

REQUIREMENTS = (OPSYS_FLAVOUR == RH9)

REQUIREMENTS = (OPSYS_FLAVOUR == "RH9")

REQUIREMENTS = (OPSYS_FLAVOUR =?= "RH9")

√


5

Custom Attributes

1. Declare the attribute and match against it2. Always quote non-numerics3. "Export" the attribute by adding to

STARTD_ATTRS, rather than STARTD_EXPRS4. Remember the use of the meta operators: =?=

and =!=


6

How do I pass ROOT=C:\ to Windows?

ENVIRONMENT = ROOT=C:\ARGUMENTS = first_param

ENVIRONMENT = ROOT=C:\\\ARGUMENTS = first_param

ENVIRONMENT = ROOT=C:\\ARGUMENTS = first_param

ENVIRONMENT = ROOT=C:\□ARGUMENTS = first_param

ENVIRONMENT = ROOT=C:\\

ARGUMENTS = first_param

ENVIRONMENT = ROOT=C:\|DUMMY=""ARGUMENTS = first_param

ROOT=C:ARGUMENTS = … ROOT=C:\ARGUMENTS = …

ROOT=C:\\ARGUMENTS = … ROOT=C:ARGUMENTS = …

√ √ENVIRONMENT = ROOT="C:\"ARGUMENTS = first_param ?

Environment variables


7

Environment variablesHow about passing more than one environment variable on Windows and/or Linux?

E1=AB E1=AB

□E2=DE □E2=DE

E1=AB E1=AB

E2=DE E2=DE

ENVIRONMENT = E1=AB|□E2=DE ENVIRONMENT = E1=AB□|□E2=DE

ENVIRONMENT = E1=AB□|E2=DE ENVIRONMENT = E1=AB|E2=DE

√ √


8

Environment VariablesWhat about using newlines for better formatting when passing lots of environment variables?

E1=AB

□E2=DE

E1=AB

E2=DE

ENVIRONMENT = E1=AB;□\ E2=DE

ENVIRONMENT = E1=AB□;\ E2=DE

ENVIRONMENT = E1=AB\ ;E2=DE

ENVIRONMENT = E1=AB;\ E2=DE√


9

Environment variables

Remember a trailing backslash will escape the newline, even if you add trailing whitespace or escape it with another backslash, or both.

Be careful about blanks and tabs in the middle of the environment line; they may render your declarations impotent. Probably better to avoid whitespace.

OR, even better …


10

The "new" syntax [Paraphrased from condor_submit man pages]

• Put double quotes around the entire string. • To insert a double quote, repeat it.• Use whitespace to separate arguments. To

include whitespace, surround by single quotes.

• To insert a single quote, repeat it inside a single-quoted section.


11

How are previous gotchas handled?

The following all work as expected for passing environment variables:

ENVIRONMENT = "ROOT=C:\"

ENVIRONMENT = "E1=AB E2=AC"

ENVIRONMENT = "E1=AB \

E2=AC"


12

What? No Gotchas?How do I setup an environment variables for a single quote or

a double quote?

DQUOTE="\"" DQUOTE='"'

SQUOTE="'"

# in old Syntax

ENVIRONMENT = DQUOTE="\"" ENVIRONMENT = DQUOTE='"'

ENVIRONMENT = SQUOTE="'"


13

DQUOTE

# New Syntax

# Trying to produce DQUOTE='"'

ENVIRONMENT = "DQUOTE=''''""''''"

# Trying to produce DQUOTE="\""

ENVIRONMENT = "DQUOTE=""\"""""


14

SQUOTE

# New Syntax

# Trying to produce SQUOTE="'"

ENVIRONMENT = "SQUOTE=""''''"""


15

So …

Use the "new syntax"


16

Condor + Connectivity

Firewalls are becoming more commonplace; as well as site firewalls, many machines will have their own firewalls.

Many grid resources or local clusters will have their worker nodes on private networks which are not directly addressable.

Is this a problem?


17

Execute MachineSubmit Machine

Submit

Schedd

Starter JobShadow

Startd

Central Manager

CollectorNegotiator

Slide based on one from the University of Wisconsin-Madison


18

Condor + Firewalls

All nodes in the pool must be able to "see" the central node(s)

Every submit node must be able to "see" all execute nodes on which its jobs are expected to run and vice versa

["see" means accessible for the relevant condor fixed ports as well as the ephemeral ports (defined using HIGHPORT and LOWPORT) for TCP and UDP ]


19

Condor + GridsHow is Flocking affected?

Your jobs will not run on remote execute nodes which you cannot "see" or can't "see" you.

How about Glidein?Your jobs will not run on remote execute nodes which you cannot "see" or can't "see" you.

Grid Universe (incl. Condor-C)Your job is effectively handed over to that grid so firewalls are less of a problem.


20

Firewall solutions1. Run the whole pool inside a single (site?) firewall

so no problems.2. Use a restricted set of submit nodes (possibly

with portal front-ends) and open resources for these.

3. Mirror the firewall settings for each machine in its ClassAds (exercise for the user – but see me for hints)

4. Use the Grid universe5. Use the Generic Connection Broker – see Todd!


21

In Summary

If you declare your own attributes – avoid the above GotchasUse the "new" syntax for environment and argument values in submit filesBe aware/wary of Firewalls and design your pool accodingly

For further info on Firewall mirroring:http://tardis.dl.ac.uk/Condor

condor gotchas iii

Documents