minionize 0.3.3

最简单的用例

在这种情况下，接收到的参数将附加到 迷你程序。如果你需要更多的参数控制 写你自己的回叫。在

• 使用Execo引擎：

  # Install the execo minionizer
  pip install minionize[execo]# Create the queue of params
  # You'll have to run this prior to launching your minions (adapt to
  # your need / make a regular script)
  $) python -c "from execo_engine.sweep import ParamSweeper; ParamSweeper('sweeps', sweeps=range(10), save_sweeps=True)"# start your minions
  $)MINION_ENGINE=execo minionize echo hello
  hello 0
  hello 1
  hello 2
  hello 3
  hello 4
  hello 5
  hello 6
  hello 7
  hello 8
  hello 9

  Note

  In other words the ^{tt17}$ wrapper lets you populate the queue with strings representing the parameter of your command line

  在

• 记录一些统计信息：您需要设置一个Reporter来报告您的统计信息。在

  # Install the execo minionizer
  pip install minionize[execo]# Create the queue of params
  # You'll have to run this prior to launching your minions (adapt to
  # your need / make a regular script)
  $) python -c "from execo_engine.sweep import ParamSweeper; ParamSweeper('sweeps', sweeps=range(10), save_sweeps=True)"# start your minions
  MINION_ENGINE=execo MINION_REPORTER=json minionize sleep
  
  # read the stats (while running or no)
  MINION_REPORTER=json minion-status
  

  在

• 在OAR集群（Igrida/Grid5000）上：

  • 例如使用Execo生成队列

    python -c "from execo_engine.sweep import ParamSweeper; ParamSweeper('sweeps', sweeps=range(1000), save_sweeps=True)"

    • 创建oar扫描脚本：
    ^{pr2}$
    • 启动你的仆从

    echo"MINION_ENGINE=execo" > .env
    oarsub --array 10 -S ./oar.sh
    

    注意

    .env文件在缩小开始时被读取，以便扫描脚本可以 无论使用什么引擎，都要保持不变。在

    • 输出示例：

    $) cat OAR.1287856.stdout
    [...]
    hello from 1287856135
    hello from 1287856139
    hello from 1287856143
    hello from 1287856147
    hello from 1287856151
    hello from 1287856155
    hello from 1287856159
    hello from 1287856163
    hello from 1287856167[...]

    注意

    正如预期的那样，参数已经分配给了不同的仆从

    在
  在

自定义回调

发送到程序的参数可以是任何东西（例如python dict）。在 在某些情况下（实际上很多情况下），您需要将这些参数转换为 你的程序可以理解的东西。所以你需要告诉 minionie如何最小化。这是通过使用特定的回调来实现的。在

编写自定义回调的最简单方法是继承 ProcessCallback或FuncCallback。有了这些回电你就不知道了'吨 不得不担心确认逻辑。在

#   a.out is invoked like this: a.out --arg1 varg1 varg2#   but the queue holds json like object:#   {"arg1": varg11, "arg2": varg21}, {"arg1": varg12, "arg2": varg22} ...# we can write a custom ProcessCallback which overrides the to_cmd methodclassMyProcessCallBack(ProcessCallback):defto_cmd(param:Param):returnf"a.out --arg1 {param['arg1']}{param['arg2']}"m=minionizer(MyProcessCallback())m.run()

#   you want to minionize a python function `my_function`#   but the queue holds json like object:#   {"arg1": varg11, "arg2": varg21}, {"arg1": varg12, "arg2": varg22} ...# we can use the FuncCallback for this purposedefmyfunc(...)# this is your functiondef_myfunc(param:Param)# this is the wrapper which invokes myfunc based on the paramsreturnmyfunc(param["arg1"],param["arg2"])m=minionizer(FuncCallback(_myfunc))m.run()

默认值

--------------------------------------------

# which engine (queue implementation) to use
MINION_ENGINE=execo # google, pulsar
# Execo
EXECO_PERSISTENCE_DIR=sweeps

# Google
GOOGLE_PROJECT_ID=/mandatory/
GOOGLE_TOPIC_ID=/mandatory/
GOOGLE_SUBSCRIPTIOn=/mandatory/
GOOGLE_APPLICATION_CREDENTIALS=/mandatory/
GOOGLE_DECODER=identity


# Pulsar
PULSAR_CONNECTION=pulsar://localhost:6650
PULSAR_TOPIC=/mandatory/
PULSAR_DECODER=identity

---------------------------------------------

# Stat reporter
MINION_REPORTER=null # json, stdout
# Json
REPORTER_JSON_DIRECTORY=minion-report