PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
NAME
psub - Submits batch job to DPCS
SYNOPSIS
psub [-A date-time] [-a account] [-b bank]
[-c constraints] [-cpn cpus_per_node] [-d jobid]
[-dm] [-e file] [-eo] [-expedite] [-exempt [exlist]]
[-g [cpus][@option]] [-i input] [-ke] [-ko] [-lc limit]
[-ld limit] [-lf limit] [-lF limit]
[-lM jsize] [-ln nodecount] [-lt limit] [-mb] [-me]
[-mu user-name] [-nc] [-net protocol]
[-nettype type] [-np] [-nr] [-ns resource]
[-o file] [-p priority] [-r job-name] [-re] [-ro]
[-s shell-name] [-sp] [-standby] [-tM time] [-tm time]
[-tW time] [-x] [-noDFS]
[file [session-args]]
DESCRIPTION
The utility psub is used to submit batch jobs to the DPCS. On suc-
cessful submission, DPCS spools a job for execution. So, after you
submit a script, you may then change the contents of the script file
without affecting the job. Attributes of jobs are entered into the
DPCS job database on submission. See pstat, palter, prm, and prel for
information about how to modify attributes or otherwise manage jobs
in DPCS. DPCS currently utilizes one of two batch systems, LoadLev-
eler on IBM SP machines and NQS everywhere else, to actually run
batch jobs. When DPCS schedules a job to run, it "commits" the job to
the native batch system running on the targeted host. If a job
attribute or job related action is specified via an option to psub
and that attribute or action is not supported by either DPCS or by
the native batch system, the corresponding option is ignored. (Note:
on paired options, there must be a space between the option and the
option value.)
Options:
-A Specifies the earliest time at which the job may be
scheduled.
-a Account that is to be charged.
-b Bank from which permitted resources are to be drawn.
-c Constraint used to determine the set of hosts on which the
job must run.
-cpn On a multi-node machine such as an IBM SP, this option indi-
cates the number of cpus the job requires on each node. On
machines that are scheduled as a cluster, this option indi-
cates the number of cpus the job requires on the node to
which it is assigned. This option option is ignored on SMP
machines that are scheduled by memory load. Use 'pcsmgr' to
DOE/LLNL/LCC June 6, 2002 1
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
determine the scheduling type of a machine. If the scheduling
type is either "cluster" or "node", this option may be used.
On cluster scheduled machines, the value is advisory only.
-d Job which must complete before this job is run.
-dm Job is to be deleted once max time has been delivered to the
job.
-e Directs standard error output to the stated destination.
-eo Directs standard error output to the standard output destina-
tion.
-exempt Exempt the job from listed exemptions.
-expedite
Job is to be expedited upon submission to DPCS.
-g The geometry of node usage.
-i A shortcut method to indicate a simple command directive to
DPCS.
-ke Leaves the stderr file on the execution machine.
-ko Leaves the stdout file on the execution machine.
-lc Rstablishes per-process corefile size limits.
-ld Establishes per-process data-segment size limits
-lf Establishes per-process file-size limits.
-lF Establishes per-job permanent file space limits.
-ln This establishes the number of dedicated nodes, or hosts,
necessary for a parallel job. -lM Advisory as to expected
maximum resident set size to be attained by the job.
-lt Establishes per-process CPU time limits.
-mb Sends mail when the job begins execution.
-me Sends mail when the job ends execution.
-mu Sends mail about the job to the stated user.
-nc Specifies that the batch job may not be checkpointed.
-net Specifies the protocol type.
DOE/LLNL/LCC June 6, 2002 2
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
-nettype
Specifies the communications network to be used.
-np Specifies that the batch job may not be preempted.
-nr Specifies that the batch job may not be rerun (from the
beginning) after system reboot.
-ns Non-shared resource requirements of the job.
-o Directs standard output to the stated destination.
-p Assigns a priority to the job.
-r Assigns stated name to the job.
-re Writes the standard error file of the job directly to its
destination.
-ro Writes the standard output file of the job directly to its
destination.
-s Specifies shell to interpret the batch shell script. This
must contain the fully qualified path to the shell (ie. -s
/bin/csh) (Warning: there have been problems reported with
using this option. You are encouraged to allow your default
login shell to interpret the script.)
-sp Submits the job as a short production job.
-standby
Sumbits the job as a standby production job.
-tM Maximum job cpu time.
-tm Maximum process cpu time.
-tW Maximum elapsed run time. (Also, called "wall clock" time.)
-x Exports all environment variables with job.
-noDFS Specifies that the users job will not access DFS and that
there is no need to manage DCE credentials for this job.
session-args
Arguments to be passed to script via environment variable
SESSARGS.
On successful submission of the job, psub will respond with the job
identifier associated with the job. This identifier uniquely identi-
fies the job to DPCS throughout the DPCS domain.
DOE/LLNL/LCC June 6, 2002 3
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
By default, DPCS returns the files produced by your job to the host
and directory from which the job was submitted, under the following
names:
stderr: .e
stdout: .o
is the string specified by the -r option or is the base
name of the script file if the -r option is not used or is your user
name if you don't use the -r option and use the -i option or type in
the script directly to psub. The job-name is truncated to seven char-
acters if a longer name is specified or implied. The names of the
standard output and standard error files can be set to whatever you
wish by use of the -e and -o options, respectively.
Options can be embedded within a script file prior to the first exe-
cutable command of the shell script by preceding them with the string
"# PSUB". The characters "PSUB" must be upper-case and must not be
separated by white spaces.
For example, to specify that the batch job should send mail at job
termination would be expressed as:
# PSUB -me
NOTE: If the same option appears both in the script file and on the
command line, the command-line option takes precedence. Multiple
occurrences of an option on the command-line will cause an error.
However, multiple occurrences of an option within a job script will
result in only the last occurence of the option being recognized.
Comments may be included within embedded options in the script file
by preceding them with a # character. Comments can be appended to an
embedded option by preceding them with the # character thus:
# PSUB -A "01/23/93 18:00:00" # Do not run job before
# 6 p.m. on Jan 23, 1993
A typical extended example of the use of embedded options within the
script file would be as follows:
#
# Batch job shell script example:
#
# PSUB -r myjob # Set job-name to myjob
# PSUB -me # Send mail at job termination.
# PSUB # No more embedded options.
#
nmake all
DOE/LLNL/LCC June 6, 2002 4
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
For other examples of typical DPCS batch jobs, refer to the EXAMPLES
section.
If you are bringing a script from a machine that runs NQS (without
DPCS), you may leave the embedded # QSUB options intact, psub will
ignore them as will qsub ignore the # PSUB statements.
-A date-time
Holds the batch job until the specified date and time. If
both a date and a time are specified, then both must be
enclosed in quotes so that the shell does not treat them as
separate tokens (e.g., -A "12/15/93 3:30PM"). The date-time
is specified as follows:
[date] time [time-zone]
Permissible forms for date are:
YYYY-MM-DD DD-Month-YY Month DD YYYY
YYYY DD-Month DD-Month-YYYY MM/DD YYYY
YYYY Month DD DD-Month YYYY MM/DD/YY
YYYY MM/DD
If date is not specified, the current day is assumed.
The month is specified in full or as any acceptable prefix of
the calendar month that is at least three characters. month
is case insensitive.
The day is specified in full or as any acceptable prefix of
the day of the week that is at least three characters. day is
case insensitive. An optional "." may follow the specifica-
tion. Alternative specifications are as follows:
today tomorrow
Permissible forms for time are as follows:
HH HH:MM HH:MM:SS
noon midnight
Permissible forms for meridian are as follows:
am pm m (indicating noon)
In the absence of meridian, a 24-hour clock interpretation
prevails.
The time-zone is specified as any North American time zone or
"GMT". If time-zone is not specified, the local time zone
prevails.
DOE/LLNL/LCC June 6, 2002 5
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
Valid date-time examples:
"01-Jan-1988 12am, PDT"
"Tuesday, 23:00:00"
"11pm tues."
"tomorrow 23 MST"
-a account
Account that is to be associated with all resource usage
reports for the job.
-b bank Bank from which allocated resources is to be drawn. If you do
not use this option, resources used by the job will be drawn
from your default bank. (See defbank.) If you do not have a
default bank, or your default bank does not permit batch mode
usage, you must use this option to specify a bank that does
permit batch mode usage. (See bac.) You must have been
granted permission to use the resources allocated to the
selected bank.
-c constraints
Job constraints. This argument specifies which features must
be assigned to hosts that are permitted to run the job. The
job will run only on one of the hosts that satisfy the speci-
fied constraints. If there is no host that satisfies the
given constraint, the job submission fails and an error mes-
sage is printed. (See A WORD ABOUT CONSTRAINTS in the example
section.)
-d jobid
Job which must complete before this job is run. Default is no
dependency. A dependency on jobid 0 is the same as no depen-
dency on any job.
-dm Delete the job once it has received its maximum time. On a
checkpointing machine a job is held in the USED>MAX state
once it has received the maximum requested time. This gives a
user a chance to increase the maximum time limit in the event
that the job needs to run longer. The advantage to using this
option on a checkpointing machine, is that it frees up an
active job slot that would otherwise limit the number of jobs
that could be executing on the machine. On a non-
checkpointing machine this is the default action.
-e file Directs the standard error file (stderr) produced by the job
to the specified destination. This string may not contain a
':', this character has a special meaning to some of the
underlying batch systems.
-eo Directs all output that would normally be sent to the stderr
file to the stdout file for the job. This option is not valid
DOE/LLNL/LCC June 6, 2002 6
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
when the -e file or -re options are used.
-exempt [exlist]
Makes the job exempt from specified administrative con-
straints. If exlist is not specified, the job is made exempt
from all constraints the owner of the job is permitted to
exempt her/his jobs from. If used, the job is made exempt
from the statuses listed in addition to statuses it may
already be exempted from. exlist is a comma separated list of
administratively imposed constraints. An administratively
imposed constraint is specified by entering the status the
job might have if not exempt from the constraint. For
instance, specifying -exempt QTOTLIM would exempt a job from
the administrative constraint regarding the total number of
jobs that are permitted to run on a machine. (Note: this
option may not be used in conjunction with the standby
option, standby jobs are not exemptable.)
The administratively imposed constraints from which a job may
possibly be exempted are:
WMEML
machine load within target parameters
QTOTLIM
machine is running as many jobs as is permitted
TOOLONG
job is requesting more time than is permitted
CPUS>MAX
job is requesting more nodes than is permitted
CPU&TIME
job is requesting more node time than is permitted
QTOTLIMU
machine is running as many jobs for user as is per-
mitted (not permitted on a running job.)
JRESLIM
maximum allowable jobs for the user or bank are
already running
NRESLIM
maximum allowable nodes for the user or bank are
already in use
NHRESLIM
maximum node-time limit for the user or bank has been
reached
DOE/LLNL/LCC June 6, 2002 7
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
-expedite Causes the job's expedite flag to be set. Doing so raises
the job's priority so that it is higher than any other
job except other expedited jobs. To expedite your job,
you must have been granted the expedite permission to the
bank the job will use for its resources. (Note, expedit-
ing a job does not actually change its priority. Rather
it, in effect, places the job in a "class" which is
priority scheduled ahead of the normal "class". Also
note, this option may not be used in conjuction with the
standby option.)
-g [tasks][switch][@layout]
Geometry is used in conjunction with the -ln option to
determine how tasks will be assigned to nodes. The tasks
value is the total number of tasks that will be assigned
to the job. The switch value is the communications switch
type and may be either "ip" or "us". layout may be either
"tpn" where is the number of tasks per
node or "dist" which specifies that tasks should be
spread as evenly as possible among nodes. The default
switch is "us" and the default layout is "dist". If tasks
is not specified, then if the layout is "dist", the
number of tasks is equal to the number of nodes; but if
the layout is "tpn", then the number of tasks is
the multiple of the number of nodes requested and the
of tasks per node specified. If both tasks is
specified and the layout is specified as "tpn",
then the number of tasks per node multiplied by the
number of nodes requested must equal the number of tasks
specified.
-i input Eliminates the need to have a script or use stdin when a
job requires only a single line of input.
-ke Leaves the stderr file on the execution machine; nor-
mally, the stderr file is returned to the originating
machine.
This option is invalid if the -eo option is specified or
if an explicit machine destination is given for the
stderr parameter of the -e option.
-ko Leaves the stdout file on the execution machine; nor-
mally, the batch system returns the stdout file to the
machine that originated the job.
This option should not be used if an explicit machine
destination is given for the stdout parameter of the -o
option.
-lc limit Set a per-process maximum core file size limit for the
batch job. See the limits subsection for more
DOE/LLNL/LCC June 6, 2002 8
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
information.
-ld limit Set a per-process maximum size limit for the batch job.
See the limits subsection for more information.
-lf limit Sets a per-process maximum file size limit for the batch
job. See the Limits subsection for more information.
-lF limit Sets the maximum per-job permanent file space limit for
the batch job. See the Limits subsection for more infor-
mation.
-ln nodecount
Sets the number of nodes required for a parallel job. If
this option is not specified, the job will be executed as
a single node serial job. If "-ln 1" is specified, the
job will be executed as a single node parallel job. The
value given for the -ln option must be greater than 0.
The -ln option is appropriate when the job is to be run
on a multi-node machine such as an IBM SP or a LINUX
cluster running parallel clustering software.
-lM jsize Advises the scheduler as to the expected maximum resident
set size to be attained by the job during its execution.
The maximum resident set size of jobs that have ter-
minated as well as the maximum resident set size reported
so far by currently executing jobs may be obtained
through the pstat utility. (See pstat.) By providing
accurate estimates, the scheduler can more appropriately
schedule jobs to avoid memory contention by jobs on a
machine. The scheduler will not use memory advisories
for users who consistently underestimate their jobs'
sizes.
-lt limit Specifies the CPU time limit for each process of a batch
job. If the CPU time limit is exceeded by any process, it
is terminated. The default units when using the
nnnn[dhms] format will be minutes.
See the Limits subsection for more information on the
implementation of batch job limits and for a description
of the syntax of a per-process CPU time limit.
-mb Sends mail to the user on the originating machine when
the job begins execution. If the -mu option is also
present, the batch system sends mail to the user speci-
fied by -mu instead of to the invoking user.
-me Sends mail to the user on the originating machine when
the job has ended execution. If the -mu option is also
present, the batch system sends mail to the user speci-
fied by -mu instead of to the invoking user.
DOE/LLNL/LCC June 6, 2002 9
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
-mu user-name
Specifies that any mail concerning the job is delivered
to the user user-name. User-name must be the user name of
a user with an account on the execution host. In the
absence of this option, the batch system sends mail con-
cerning the job to the job owner.
-nc Specifies that the job is not checkpointable. Job will be
killed if DPCS determines its execution must be stopped
before it receives the maximum resources requested.
When a batch system that supports checkpointing is shut-
down gracefully, then a checkpoint of all running, check-
pointable jobs is taken at the time of shutdown. When a
batch system that does not support checkpointing is shut-
down, then obviously no checkpoint of any job is taken.
When a batch system that supports checkpointing is res-
tarted, it will attempt to resume processing of each job
running at the time of the last shutdown at the point it
was when it was last checkpointed. This is called a batch
job restart and makes sense only when the job runs under
a batch system that supports checkpointing. If a batch
job restart fails, the job is treated as if it had not
been checkpointed.
When a batch system first initializes, typically after
rebooting (whether or not it supports checkpointing), all
the jobs running at the time of shutdown that cannot be
restarted are rerun from the beginning. (Except see the
-nr option.)
The -nc option directs the batch system that runs the job
to treat the job as not checkpointable. This option is
obviously implied on batch systems that do not support
checkpointing. In the absence of -nc, batch systems that
support checkpointing treat the job as checkpointable.
Jobs that are not running at the time of a batch system
interruption or shutdown are not effected by the -nc or
-nr options.
See the NOTES section for more discussion concerning the
checkpoint/restart of batch jobs.
-net Specifies that the network protocol type that is to be
used for this job. (Only valid on machines that support
this option.)
-nettype type
Specifies that the network communications type that is to
be used for this job. (Only valid on machines that sup-
port this option.)
DOE/LLNL/LCC June 6, 2002 10
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
-np Specifies that the job is not preemptable. This option is
only valid for machines that do job preemption.
-nr Specifies that the batch job is nonrerunnable; the job
will not be rerun at a batch system initialization if the
job is not rerunnable.
In the absence of -nr, the batch system attempts to rerun
a job on batch system initialization.
A job that is not running at the time of a batch system
interruption or shutdown are not effected by the -nc or
-nr options.
-ns resources
Non-shared resources. This argument specifies which non-
shared resources must be available on hosts that are per-
mitted to run the job. The job will run only on one of
the hosts that provides the specified resources. If there
is no host that satisfies the specified resource require-
ments, the job submission fails and an error message is
printed. The syntax for specifying non-shared resources
for a job is similar to that of is printed. (See A WORD
ABOUT CONSTRAINTS in the example section.)
-o file Directs the standard output file (stdout) produced by the
batch job to the specified destination. This string may
not contain a ':', this character has a special meaning
to some of the underlying batch systems.
-p priority Set the scheduling priority for the selected job to prio.
The value specified with this option must be a number
between 0.0 and 1.0 inclusive or the special value float.
After setting the priority of a job to a numerical value,
DPCS will not reevaluate its priority. If you set the
priority to float, DPCS will evaluate the job's priority
again.
-r job-name Assigns the specified job-name to the job. If the
job-name is not specified, the default is the name of the
script file with the leading path name removed, as given
on the command line. If a script file is not used, the
default job-name is the owner's user name. The job-name
will be truncated to 15 characters by psub before being
passed on to DPCS. Note that the job-name cannot begin
with a digit, and cannot contain a '%' sign. If the
job-name does start with a digit, an uppercase r will be
inserted as the first character of the name.
-re Specifies that the stderr output produced by the job is
written to the final destination file as output is gen-
erated. This option is ignored if the job's error output
DOE/LLNL/LCC June 6, 2002 11
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
is to be routed to a host other than the host where the
job executes.
By default, all standard error output generated by a job
is spooled to a temporary file in a protected directory
which the batch system maintains on the executing
machine. When the batch job completes execution, this
file is routed to its final destination.
-ro Specifies that the standard output produced by the job is
written to the destination file as output is generated.
This option is ignored if the job's output is to be
routed to a host other than the host where the job exe-
cutes.
By default, all standard output generated by a job is
spooled to a temporary file in a protected directory
which the batch system maintains on the executing
machine. When the batch job completes execution, this
file is routed to its final destination.
-s shell-name
Specifies the absolute path name of the shell used to
interpret the batch job shell script. (Warning: there
have been problems reported with using this option. You
are encouraged to allow your default login shell to
interpret the script.)
In the absence of this option, your login shell will be
used to run this job.
-sp This specifies that the job is being submitted as a short
production job. If submitted successfully, the job is
scheduled at the earliest possible time. There are some
downsides to its usage, they are:
The job will be non-checkpointable.
A submittal will be rejected and a suitable error message
sent when a user attempts to submit a short production
job while he/she already has a configuration parameter
number of submitted short production jobs active.
The -tM option may not exceed a system wide configuration
parameter value (the plim utility will give you this
information).
The -A, -d, and options will be ignored and set to zero
before being passed to DPCS.
You must have short production permission enabled by your
coordinator before you can use the -sp option. If you submit
DOE/LLNL/LCC June 6, 2002 12
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
a job without this permission, it will not run until you do
get permission.
-standby The job will be submitted as a stanby job. This job will
be exempt from the user and bank resource limits that are
assigned to each partition. However, the job will be
removed (or otherwise stopped) when a non-standby job is
eligible to run on the machine and is in need of the
resources being used by the job. If the job is
registered to receive a signal, it will be signalled
prior to removal within the configured grace time
limit.(Note: The standby option and exempt option can not
be used together. A standby job is not exemptable by the
user.)
-tM time Maximum cpu time. The job will be killed or checkpointed
and not allowed to run any more after it accumulates the
specified (or defaulted) amount of cpu time. The default
is a configuration parameter. (See pcsmgr or plim). The
default units when using the nnnn[dhms] format is
minutes. The resulting interpretation of this value must
be greater than or equal to 1 minute.
-tm time Maximum process cpu time. If the batch system supports
this option (only NQS does), then the job will be killed
when any process in the job accumulates the specified
amount of cpu time. If this option is not used or is not
supported by the native batch system (i.e., Loadleveler),
the process cpu time limit is unlimited.
-tW time Maximum elapsed run time. The job will be killed or
checkpointed and not allowed to run any more after its
elapsed run time exceeds the specified or defaulted time.
The default is a configuration parameter. The default
units when using the nnn[dhms] format is minutes. The
resulting interpretation of this value must be greater
than or equal to 1 minute.
-x Exports all environment variables (except ENVIRONMENT).
DPCS always defines and initializes several environment
variables before committing your job to the batch system.
These environment variables are available to your job
during execution. They are:
PCS_REQID is set to the DPCS job id (for backward
compatibility)
PCS_TMPDIR Unlike the other environment variables,
this one will only be set if DPCS has
created a temporary directory for the
DOE/LLNL/LCC June 6, 2002 13
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
job. In this case, it will be set to the
name of the directory. This directory
will be exempt from any file system
purges during the course of the job, but
will still be subject to file system quo-
tas. When the job terminates, this direc-
tory, and all files within it, will be
removed.
PSUB_HOME is set to your home directory on the sub-
mission host.
PSUB_HOST is set to the submission host name
PSUB_JOBID is set to the DPCS job identifier
PSUB_LOGNAME is set to your UNIX login name on the
submission host
PSUB_MAIL is set to your mail repository on the
submission host
PSUB_PATH is set to the value of the environment
variable PATH
PSUB_REQNAME is set to the job's specified or implied
request name
PSUB_SHELL is set to your login shell on the submis-
sion host.
PSUB_TZ is set to the time zone in effect on the
submission host
PSUB_USER is set to your UNIX login name on the
submission host
PSUB_WORKDIR is set to your current directory on the
submission host
HOME is set to your home directory on the exe-
cuting host
SHELL is set to your login shell on the execut-
ing host.
PATH is set to "/bin:/usr/bin"
-noDFS Specifies that the users job will not access DFS and that
there is no need to manage DCE credentials for this job.
DOE/LLNL/LCC June 6, 2002 14
PSUB(l) Livermore Computing Local UNIX Manual PSUB(l)
session-args
Any characters following the script name will be stored in
the environment variable SESSARGS. SESSARGS will be exported
with the script. SESSARGS can be retrieved at run-time. It
will be returned as a simple string. It is up to the user to
process the string.
For example, if the following script is submitted,
#!/bin/csh
#PSUB
#
/usr/bin/date
#
echo $SESSARGS
with the execute line
psub