jug-schedule 0.4.2

要求

这个项目依赖于drmaa，显然依赖于jug。

安装

安装jug_schedule时使用：

pip install jug-schedule

或来自水蟒的：

conda install -c unode jug-schedule

然后简单地将以下内容添加到您的~/.config/jug/jug_user_commands.py：

try:
    from jug_schedule.schedule import ScheduleCommand
    schedule = ScheduleCommand()
except Exception as e:
    import sys
    sys.stderr.write("Couldn't import schedule, error was {0}\n".format(e))

如果直接从git运行，则可以使用：

import sys

sys.path.insert(0, "/path/to/clone/of/jug_schedule/")

try:
    from jug_schedule.schedule import ScheduleCommand
    schedule = ScheduleCommand()
except Exception as e:
    sys.stderr.write("Couldn't import schedule, error was {0}\n".format(e))

用法

如果安装正确，运行jug现在应该包含一个schedule子命令。

运行它将尝试检测排队系统并向其提交作业。 jug schedule只会产生警告和错误。使用--verbose debug查看详细消息。

jug status将照常工作，是检查执行进度的推荐方法。

#!/usr/bin/env bash

# Enabling modules (http://modules.sourceforge.net/) in the current shell
if [ -f /etc/profile.d/modules.sh ]; then
    source /etc/profile.d/modules.sh
fi

# Loading (ana)conda
module add conda
# and an environment called py3 where jug was installed
source activate py3

# Then calling jug with all given arguments. Make sure to keep the quotes on $@
jug "$@"

jug schedule --script script.helper

$ ls
myjugfile.helper myjugfile.py
$ jug schedule myjugfile.py  # <- will use myjugfile.helper

配置

jug_schedule依赖于drmaa与排队系统的交互。

为了使用jug_schedule，您的环境需要定义DRMAA_LIBRARY_PATH。 如果运行env | grep DRMAA_LIBRARY_PATH未返回匹配项，请向系统管理员询问此库的位置。

然后使用：

export DRMAA_LIBRARY_PATH=/path/to/libdrmaa.so

您只需要在运行jug schedule的环境中设置此选项。

资源

jug_schedule的另一个特性是能够定义作业资源。

如果您已经知道jug的TaskGeneratordecorator，您可以在适用的情况下用以下内容替换它：

from jug_schedule.resources import ResourcesTaskGenerator

@ResourcesTaskGenerator(cpu=10, mem=100, queue="default")
def func(...):
    ...

对于任意选项，支持的参数包括：cpu、mem（以MB为单位）、queue和custom。

命令行选项

提供以下选项：

--script            - command used to run jug on the cluster. Point this to a shell script if you need to setup jug's environment prior to execution
--max-jobs          - how big is the pool of jug jobs (max number of simultaneous jobs)
--max-array         - when submitting jobs to the queue system, limit the maximum number of jobs per submission
--logs              - where to write job logs. Defaults to a directory 'jug_logs' in the current directory.
--cycle-time        - how many seconds to wait between every interaction with the queue system. Defaults to 60
--stop-on-error     - jug_schedule will continue until all jobs fail. Default is to continue queueing jobs when a job fails.
--recycle           - when a job fails, instead of removing one job from the pool, recycle it and keep the pool size constant.