本地便携式批处理系统

LPBS的Python项目详细描述


http://github.com/goerz/LPBS

作者:Michael Goerz

本地便携式批处理系统:在本地工作站上模拟PBS

此代码是根据 GPL

PBS/TORQUE 是一个用于许多高性能的作业调度系统 计算集群。lpbs包提供了运行pbs作业的工具 本地工作站上的脚本。具体来说,lpbs提供 lqsub命令,它接受与pbs相同的选项qsub 命令并在本地环境中运行作业脚本 与PBS/转矩提供的相同。作业将运行 背景并被分配一个作业ID。与PBS系统不同, lpbs不会执行任何调度,而只是运行作业 提交给它。lpbs提供了进一步的工具来管理运行 乔布斯。

安装

lpb可以从pypi安装,使用

pip install LPBS

或者,可以使用

python setup.py install

配置

lpbs将其所有配置和运行时数据存储在文件夹中 在环境变量$LPBS_HOME中给定。这种环境 必须定义变量。配置在文件中 lpbs.cfg$LPBS_HOME内。如果此文件不存在 当运行任何lpbs脚本时,一个配置文件 将创建以下默认值:

[Server]

# Full hostname of submission server (hostname.domain). Will be made available
# to running job through the environment variable PBS_SERVER. Job IDs will end
# in the server hostname

hostname: localhost
domain: local


[Node]

# Full hostname of the execution node (hostname.domain). Will be made available
# to running job through the environment variable PBS_O_HOST. Since LPBS is
# designed to execute jobs locally, the settings here should in general be
# identical to those in the [Server] section

hostname: localhost
domain: local


[LPBS]

# Setting for job execution.
# If 'username_in_jobid' is enabled, the job IDs will have the form
# 'seqnr.user.hostname.domain' where 'user' is the username of the user
# submitting the job.
# The file given in 'sequence_file' is used for keeping track of the 'seqnr'
# appearing in the job ID.
# The file given in 'logfile' is used for logging all LPBS events. Both
# 'sequence_file' and 'logfile' are relative to $LPBS_HOME.

username_in_jobid: 0
sequence_file: sequence
logfile: lpbs.log


[Scratch]

# Settings for the scratch space provided to jobs. 'scratch_root' defines a
# location where jobs should write temporary data. If given as a relative path,
# it is relative to $LPBS_HOME. Environment variables will be expanded at the
# time of the job submission.
# If the value of # 'create_jobid_folder' is set to 1, a folder with the name of
# the full job ID is created inside scratch_root. This folder is automatically
# deleted when the job ends, unless 'keep_scratch' is set to 1. If the job
# failed, the scratch will not be deleted, unless 'delete_failed_scratch' is set
# to 1.

scratch_root: $SCRATCH_ROOT
create_jobid_folder: 0
keep_scratch: 0
delete_failed_scratch: 0


[Notification]

# Settings on how the user should be be notified about events such as the start
# and end of a job. If sent_mail is set to 1, emails will be sent for
# notifications depending on the value of the '-m' option to lqsub. If
# 'send_growl' is set to 1, Growl (http://growl.info) is used for notification
# on MacOS X. Notifications via Growl do not take into account the '-m' options
# during job submission.

send_mail: 0
send_growl: 0


[Mail]

# SMTP settings for email notifications. Notification emails will be sent from
# the address given by the 'from' option. The SMTP server given in 'smtp' is
# used for sending the emails, if 'authenticate' is set to 1, authentication is
# done with the given 'username' and 'password'. If 'tls' is 1, TLS encryption
# will be used.

from: nobody@example.org
smtp: smtp.example.com:587
username: user
password: secret
authenticate: 0
tls: 1


[Growl]

# Settings for Growl notifications. Notifications are sent to either
# localhost or a remote host via the GNTP protocol. The 'hostname' setting
# gives the address and port of the Growl server, the given 'password' is
# used for authentication (note that if sending to localhost, no
# authentication is necessary, and the password should be empty). If
# 'sticky' is set to 1, the Growl notifications will be sticky.  It is
# possible to send notifications to more than one host. In this case, both
# 'hostname' and 'password' should be a comma-separated list of values, with
# each item corresponding to one host. E.g.
# hostname: localhost, remotehost
# password: , secret

hostname: localhost:23053
password:
sticky: 0


[Log]

# 'logfile' gives the name of the central log file, relative to $LPBS_HOME.

logfile: lpbs.log

注意,如果使用默认配置文件,则环境 必须定义变量$SCRATCH_ROOT,并将其目录 点应该存在并可写入所有可能的用户。 提交作业。

对于系统范围的安装,/var/lpbs是一个合适的 $LPBS_HOME的位置。环境变量可以在 /etc/bash.bashrc适用于所有用户。对于用户中的安装 空间,virtualenv是 推荐。

在主配置文件之后,文件$HOME/.lpbs.cfg也将 被阅读。用户可以使用此选项覆盖系统中的设置 配置。最后,qsub命令还有一个--config 选项,该选项允许指定 阅读。

用法

lpbs包含脚本lqsublqstatlqdel, 它模拟pbs/torque命令qsubqstat,以及 ^分别为{tt15}$。lqsub命令用于提交 作业,lqstat用于显示有关运行作业的信息, 并且lqdel用于中止正在运行的作业。例如:

goerz@localhost:~> lqsub job.pbs
3.localhost.local

goerz@localhost:~> lqstat
Job id               Name            User            Walltime
-------------------- --------------- --------------- ---------------
3.localhost.local    pbstest         goerz           0:00:22

goerz@localhost:~> lqstat -f 3.localhost.local
Job Id: 3.localhost.local
    Job_Name = pbstest
    Job_Owner = goerz
    server = localhost.local
    exec_host = localhost.local
    PID = 14649
    Error_Path = STDERR
    Join_Path = True
    Mail_Points = n
    Output_Path = pbstest.out
    resources_used.walltime = 0:00:43

goerz@localhost:~> lqdel 3.localhost.local

qsub命令旨在理解所有命令行 torque版本2.18中qsub命令的选项,除了 与调度相关的所有选项都将被忽略。因此,所有 PBS作业脚本应无更改地提交。关于细节, 使用^{tt27}运行lqsublqstatlqdel。$ 选项,和/或查看 TORQUE manual

作业脚本示例

下面是一个简单的pbs作业脚本的示例 打印出作业看到的完整环境。你可能想 提交此作业时,请同时提交lpbs和pbs/torque,以验证 在lpbs.cfg中的适当设置,lpbs提供相同的 作为lpbs的环境。

#!/bin/bash
#PBS -N pbstest
#PBS -j oe
#PBS -l nodes=1:ppn=1
#PBS -l walltime=00:00:10
#PBS -l mem=10mb
#PBS -o pbstest.out

echo "####################################################"
echo "User: $PBS_O_LOGNAME"
echo "Batch job started on $PBS_O_HOST"
echo "PBS job id: $PBS_JOBID"
echo "PBS job name: $PBS_JOBNAME"
echo "PBS working directory: $PBS_O_WORKDIR"
echo "Job started on" `hostname` `date`
echo "Current directory:" `pwd`
echo "PBS environment: $PBS_ENVIRONMENT"
echo "####################################################"

echo "####################################################"
echo "Full Environment:"
printenv
echo "####################################################"

echo "The Job is being executed on the following node:"
cat ${PBS_NODEFILE}
echo "##########################################################"

echo "Job Finished: " `date`
exit 0

欢迎加入QQ群-->: 979659372 Python中文网_新手群

推荐PyPI第三方库


热门话题
Java游戏socket网络路由器问题   使用Java构造函数初始化。newInstance(args),为什么会出现“args数错误”错误?   javascript当处理Java模式转换为Java脚本模式时,我的3D对象不会显示   java Memcached与Hibernate的集成   servlets如何在java中作为对象访问HttpServletResponse   由于NoClassDefFoundError,java XStream无法反序列化XML   性能远程Neo4j在JavaTinkerpop3Bolt设置中随时间变慢   不能使用java。util。正则表达式。模式,即使我在eclipse中使用jre8   带局部变量的selenium java流比较列表   java能否获得Android或iOS移动设备屏幕的实际尺寸?   java如何解决org。阿帕奇。hadoop。木卫一。无法将LongWritable强制转换为组织。阿帕奇。hadoop。木卫一。文本   java无法用字符串(变量)验证文本,不管它如何在静态文本“测试”中正常工作   如果使用java RandomAccessFile,如何更新数据   Java聚合中面临空指针异常   java查找类中定义的接口   java如何在Spring AMQP中侦听现有队列?   java有可能与PDO建立JDBC连接吗?   java使用ajax填充servelet的下拉列表   使用Azure java sdk更新用户和组属性   java找不到符号:LoggingEvent将log4j 1.2升级到2.8