本地便携式批处理系统
LPBS的Python项目详细描述
本地便携式批处理系统:在本地工作站上模拟PBS
此代码是根据 GPL
PBS/TORQUE 是一个用于许多高性能的作业调度系统 计算集群。lpbs包提供了运行pbs作业的工具 本地工作站上的脚本。具体来说,lpbs提供 lqsub命令,它接受与pbs相同的选项qsub 命令并在本地环境中运行作业脚本 与PBS/转矩提供的相同。作业将运行 背景并被分配一个作业ID。与PBS系统不同, lpbs不会执行任何调度,而只是运行作业 提交给它。lpbs提供了进一步的工具来管理运行 乔布斯。
安装
lpb可以从pypi安装,使用
pip install LPBS
或者,可以使用
python setup.py install
配置
lpbs将其所有配置和运行时数据存储在文件夹中 在环境变量$LPBS_HOME中给定。这种环境 必须定义变量。配置在文件中 lpbs.cfg在$LPBS_HOME内。如果此文件不存在 当运行任何lpbs脚本时,一个配置文件 将创建以下默认值:
[Server] # Full hostname of submission server (hostname.domain). Will be made available # to running job through the environment variable PBS_SERVER. Job IDs will end # in the server hostname hostname: localhost domain: local [Node] # Full hostname of the execution node (hostname.domain). Will be made available # to running job through the environment variable PBS_O_HOST. Since LPBS is # designed to execute jobs locally, the settings here should in general be # identical to those in the [Server] section hostname: localhost domain: local [LPBS] # Setting for job execution. # If 'username_in_jobid' is enabled, the job IDs will have the form # 'seqnr.user.hostname.domain' where 'user' is the username of the user # submitting the job. # The file given in 'sequence_file' is used for keeping track of the 'seqnr' # appearing in the job ID. # The file given in 'logfile' is used for logging all LPBS events. Both # 'sequence_file' and 'logfile' are relative to $LPBS_HOME. username_in_jobid: 0 sequence_file: sequence logfile: lpbs.log [Scratch] # Settings for the scratch space provided to jobs. 'scratch_root' defines a # location where jobs should write temporary data. If given as a relative path, # it is relative to $LPBS_HOME. Environment variables will be expanded at the # time of the job submission. # If the value of # 'create_jobid_folder' is set to 1, a folder with the name of # the full job ID is created inside scratch_root. This folder is automatically # deleted when the job ends, unless 'keep_scratch' is set to 1. If the job # failed, the scratch will not be deleted, unless 'delete_failed_scratch' is set # to 1. scratch_root: $SCRATCH_ROOT create_jobid_folder: 0 keep_scratch: 0 delete_failed_scratch: 0 [Notification] # Settings on how the user should be be notified about events such as the start # and end of a job. If sent_mail is set to 1, emails will be sent for # notifications depending on the value of the '-m' option to lqsub. If # 'send_growl' is set to 1, Growl (http://growl.info) is used for notification # on MacOS X. Notifications via Growl do not take into account the '-m' options # during job submission. send_mail: 0 send_growl: 0 [Mail] # SMTP settings for email notifications. Notification emails will be sent from # the address given by the 'from' option. The SMTP server given in 'smtp' is # used for sending the emails, if 'authenticate' is set to 1, authentication is # done with the given 'username' and 'password'. If 'tls' is 1, TLS encryption # will be used. from: nobody@example.org smtp: smtp.example.com:587 username: user password: secret authenticate: 0 tls: 1 [Growl] # Settings for Growl notifications. Notifications are sent to either # localhost or a remote host via the GNTP protocol. The 'hostname' setting # gives the address and port of the Growl server, the given 'password' is # used for authentication (note that if sending to localhost, no # authentication is necessary, and the password should be empty). If # 'sticky' is set to 1, the Growl notifications will be sticky. It is # possible to send notifications to more than one host. In this case, both # 'hostname' and 'password' should be a comma-separated list of values, with # each item corresponding to one host. E.g. # hostname: localhost, remotehost # password: , secret hostname: localhost:23053 password: sticky: 0 [Log] # 'logfile' gives the name of the central log file, relative to $LPBS_HOME. logfile: lpbs.log
注意,如果使用默认配置文件,则环境 必须定义变量$SCRATCH_ROOT,并将其目录 点应该存在并可写入所有可能的用户。 提交作业。
对于系统范围的安装,/var/lpbs是一个合适的 $LPBS_HOME的位置。环境变量可以在 /etc/bash.bashrc适用于所有用户。对于用户中的安装 空间,virtualenv是 推荐。
在主配置文件之后,文件$HOME/.lpbs.cfg也将 被阅读。用户可以使用此选项覆盖系统中的设置 配置。最后,qsub命令还有一个--config 选项,该选项允许指定 阅读。
用法
lpbs包含脚本lqsub、lqstat和lqdel, 它模拟pbs/torque命令qsub,qstat,以及 ^分别为{tt15}$。lqsub命令用于提交 作业,lqstat用于显示有关运行作业的信息, 并且lqdel用于中止正在运行的作业。例如:
goerz@localhost:~> lqsub job.pbs 3.localhost.local goerz@localhost:~> lqstat Job id Name User Walltime -------------------- --------------- --------------- --------------- 3.localhost.local pbstest goerz 0:00:22 goerz@localhost:~> lqstat -f 3.localhost.local Job Id: 3.localhost.local Job_Name = pbstest Job_Owner = goerz server = localhost.local exec_host = localhost.local PID = 14649 Error_Path = STDERR Join_Path = True Mail_Points = n Output_Path = pbstest.out resources_used.walltime = 0:00:43 goerz@localhost:~> lqdel 3.localhost.local
qsub命令旨在理解所有命令行 torque版本2.18中qsub命令的选项,除了 与调度相关的所有选项都将被忽略。因此,所有 PBS作业脚本应无更改地提交。关于细节, 使用^{tt27}运行lqsub、lqstat和lqdel。$ 选项,和/或查看 TORQUE manual。
作业脚本示例
下面是一个简单的pbs作业脚本的示例 打印出作业看到的完整环境。你可能想 提交此作业时,请同时提交lpbs和pbs/torque,以验证 在lpbs.cfg中的适当设置,lpbs提供相同的 作为lpbs的环境。
#!/bin/bash #PBS -N pbstest #PBS -j oe #PBS -l nodes=1:ppn=1 #PBS -l walltime=00:00:10 #PBS -l mem=10mb #PBS -o pbstest.out echo "####################################################" echo "User: $PBS_O_LOGNAME" echo "Batch job started on $PBS_O_HOST" echo "PBS job id: $PBS_JOBID" echo "PBS job name: $PBS_JOBNAME" echo "PBS working directory: $PBS_O_WORKDIR" echo "Job started on" `hostname` `date` echo "Current directory:" `pwd` echo "PBS environment: $PBS_ENVIRONMENT" echo "####################################################" echo "####################################################" echo "Full Environment:" printenv echo "####################################################" echo "The Job is being executed on the following node:" cat ${PBS_NODEFILE} echo "##########################################################" echo "Job Finished: " `date` exit 0