2.2.4. RADICAL-Cybertools Getting Started Tutorial
RADICAL-Cybertools support the execution of ensemble applications at scale on high performance computing (HPC) platforms. Ensemble applications enable using diverse algorithms to coordinate the execution of up to \(10^6\) tasks on all the processors (CPU/GPU) of an HPC machine. This type of applications are common in biophysical systems, climate science, seismology, and polar science domains. RADICAL-Cybertools address challenges of scale, diversity and reliability.
Adaptive ensemble are a particularly interesting type of ensemble applications in which adaptivity is used to determine the behavior of the application at runtime. For example, many biomolecular sampling algorithms are formulated as adaptive: replica-exchange, Expanded Ensemble, etc. Introducing adaptivity, improved simulation efficiency of up to a factor three but implementing adaptive ensemble applications is challenging due to the complexity of the required algorithms.
2.2.4.1. RADICAL-EnsembleToolkit (EnTK)
RADICAL-Cybertools offers RADICAL-EnsembleToolkit (EnTK), a workflow engine specifically designed to support the execution of (adaptive) ensemble applications at scale on HPC platforms. EnTK allows users to separate adaptive logic and simulation/analysis code, while abstracting away from the users issues of resource management and resource management and runtime execution coordination.
EnTK exposes a simple application programming interface (API), implemented in Python and with two (Pythonic) collections of objects and three classes: * Set: contains objects that have no relative order with each other * Sequence/List: contains objects that have a linear order, i.e. object ‘i’ depends on object ‘i-1’ * Task: description of executing kernel * Stage: set of Tasks, i.e. all tasks of a stage may execute concurrently * Pipeline: sequence of Stages, i.e., Stage 2 may only commence after Stage 1 completes
Thus, in EnTK an ensemble application is described as a set of pipelines, in which each pipiline has a sequence/list of stages, and each stage has a set of tasks. The following figure shows an example of ensemble application in which tasks are represented by arrows:
2.2.4.2. Preparing the Execution Environment
As we will be executing this tutorial within a Jupyter notebook, we install EnTK directly into the notebook kernel via pip, but we could also equally use conda.
Note: We “mute” the output of the cell with %%capture capt to not pollute the notebook output.
Depending on the execution environment, you may want to use the Spack package or the container provided by Exaworks SDK, or load the module provided by the administrators of the high performance computing (HPC) platform on which you are executing this tutorial.
[1]:
%%capture capt
%pip install radical.entk
Currently, EnTK and its runtime system RADICAL-Pilot require a RabbitMQ and MongoDB server. Those serves need to be deployed and made available before using EnTK. Here we set the access parameters for the servers.
Note:The following assumes that: 1. you have a shell; 2. you export the relevant environment variables; 3. you execute the command jupyter notebook from that shell. In that way, the relevant env variables will be read here via os.environ.get(‘NAME_VARIABLE’).
[2]:
%%capture capt
import os
rmq_host = os.environ.get('RMQ_SERVER', 'rabbitmq')
rmq_port = os.environ.get('RMQ_PORT', '5672')
rmq_name = os.environ.get('RMQ_NAME', 'guest')
rmq_pswd = os.environ.get('RMQ_PSWD', 'guest')
mdb_host = os.environ.get('MDB_SERVER', 'mongodb')
mdb_port = os.environ.get('MDB_PORT', '27017')
mdb_name = os.environ.get('MDB_NAME', 'guest')
mdb_pswd = os.environ.get('MDB_PSWD', 'guest')
mdb_dtbs = os.environ.get('MDB_DTBS', 'default')
%env RADICAL_PILOT_DBURL=mongodb://$mdb_name:$mdb_pswd@$mdb_host:$mdb_port/$mdb_dtbs
2.2.4.3. Ensemble of Simulation Pipelines
The following example application shows the execution of a simple ensemble of simulations. Each ensemble member is in itself a pipeline of three different stages:
generate a random seed as input data
evolve a model based on that input data via a set of ensembles
derive a common metric across the model results
Similar patterns are frequently found in molecular dynamics simulation workflows. For the purpose of this tutorial, the stages are:
random seed : create a random number
evolve model : N tasks computing n’th power of the input
common metric: sum over all ‘model’ outputs
The final results are then staged back and printed on STDOUT.
The following image offers a representation of the application we are going to code and then run for Example 1.
The two pipelines execute concurrently and, as per EnTK API definitions, each stage inside each pipeline executes sequentially. Importantly, when a stage contains multiple tasks, all those tasks can execute concurrently, assuming that enough resources are available. Given a set of resources, EnTK always executes the ensemble application with the highest possible degree of concurrency but, when not enough resources are available, the tasks of a stage may be executed sequentially. All this is transparent to the user that is left free to focus on the ensemble algorithm without having to deal with parallelism and resource management.
First we import EnTK Python module in our application so to be able to use its API.
[3]:
import radical.entk as re
The following function generates a single simulation pipeline, i.e., a new ensemble member. The pipeline structure consisting of three steps as described above.
[4]:
def generate_pipeline(uid):
# all tasks in this pipeline share the same sandbox
sandbox = uid
# first stage: create 1 task to generate a random seed number
t1 = re.Task()
t1.executable = '/bin/sh'
t1.arguments = ['-c', 'od -An -N1 -i /dev/random']
t1.stdout = 'random.txt'
t1.sandbox = sandbox
s1 = re.Stage()
s1.add_tasks(t1)
# second stage: create 10 tasks to compute the n'th power of that number
s2 = re.Stage()
n_simulations = 10
for i in range(n_simulations):
t2 = re.Task()
t2.executable = '/bin/sh'
t2.arguments = ['-c', "echo '$(cat random.txt) ^ %d' | bc" % i]
t2.stdout = 'power.%03d.txt' % i
t2.sandbox = sandbox
s2.add_tasks(t2)
# third stage: compute sum over all powers
t3 = re.Task()
t3.executable = '/bin/sh'
t3.arguments = ['-c', 'cat power.*.txt | paste -sd+ | bc']
t3.stdout = 'sum.txt'
t3.sandbox = sandbox
# download the result while renaming to get unique files per pipeline
t3.download_output_data = ['sum.txt > %s.sum.txt' % uid]
s3 = re.Stage()
s3.add_tasks(t3)
# assemble the three stages into a pipeline and return it
p = re.Pipeline()
p.add_stages(s1)
p.add_stages(s2)
p.add_stages(s3)
return p
[5]:
%set_env RADICAL_LOG_TGT = stdout
env: RADICAL_LOG_TGT=stdout
Now we write the ensemble application. We create an EnTK’s application manager which executes our ensemble.
[6]:
appman = re.AppManager(hostname=rmq_host,
port=rmq_port,
username=rmq_name,
password=rmq_pswd)
EnTK session: re.session.radical.3.mturilli.019318.0000
Creating AppManagerSetting up RabbitMQ system ok
ok
We assign resource request description to the application manager using three mandatory keys: target resource, walltime, and number of cpus:
[7]:
appman.resource_desc = {
'resource': 'local.localhost',
# 'resource': 'local.localhost_flux',
'walltime': 10,
'cpus' : 2
}
Validating and assigning resource manager ok
We create an ensemble of n simulation pipelines:
[8]:
n_pipelines = 10
ensemble = set()
for cnt in range(n_pipelines):
ensemble.add(generate_pipeline(uid='pipe.%03d' % cnt))
We assign the workflow to the application manager, then run the ensemble and wait for completion:
[9]:
appman.workflow = ensemble
appman.run()
Setting up RabbitMQ system n/a
new session: [re.session.radical.3.mturilli.019318.0000] \
database : [mongodb://rct-tutorial:****@95.217.193.116:27017/rct-tutorial] ok
create pilot manager ok
submit 1 pilot(s)
pilot.0000 local.localhost 2 cores 0 gpus ok
All components created
create task managerUpdate: pipeline.0003 state: SCHEDULING
Update: pipeline.0003.stage.0009 state: SCHEDULING
Update: pipeline.0003.stage.0009.task.0036 state: SCHEDULING
Update: pipeline.0004 state: SCHEDULING
Update: pipeline.0004.stage.0012 state: SCHEDULING
Update: pipeline.0004.stage.0012.task.0048 state: SCHEDULING
Update: pipeline.0001 state: SCHEDULING
Update: pipeline.0001.stage.0003 state: SCHEDULING
Update: pipeline.0001.stage.0003.task.0012 state: SCHEDULING
Update: pipeline.0007 state: SCHEDULING
Update: pipeline.0007.stage.0021 state: SCHEDULING
Update: pipeline.0007.stage.0021.task.0084 state: SCHEDULING
Update: pipeline.0006 state: SCHEDULING
Update: pipeline.0006.stage.0018 state: SCHEDULING
Update: pipeline.0006.stage.0018.task.0072 state: SCHEDULING
Update: pipeline.0000 state: SCHEDULING
Update: pipeline.0000.stage.0000 state: SCHEDULING
Update: pipeline.0000.stage.0000.task.0000 state: SCHEDULING
Update: pipeline.0005 state: SCHEDULING
Update: pipeline.0005.stage.0015 state: SCHEDULING
Update: pipeline.0005.stage.0015.task.0060 state: SCHEDULING
Update: pipeline.0002 state: SCHEDULING
Update: pipeline.0002.stage.0006 state: SCHEDULING
Update: pipeline.0002.stage.0006.task.0024 state: SCHEDULING
Update: pipeline.0009 state: SCHEDULING
Update: pipeline.0009.stage.0027 state: SCHEDULING
Update: pipeline.0009.stage.0027.task.0108 state: SCHEDULING
Update: pipeline.0008 state: SCHEDULING
Update: pipeline.0008.stage.0024 state: SCHEDULING
Update: pipeline.0008.stage.0024.task.0096 state: SCHEDULING
Update: pipeline.0003.stage.0009.task.0036 state: SCHEDULED
Update: pipeline.0004.stage.0012.task.0048 state: SCHEDULED
Update: pipeline.0001.stage.0003.task.0012 state: SCHEDULED
Update: pipeline.0007.stage.0021.task.0084 state: SCHEDULED
Update: pipeline.0006.stage.0018.task.0072 state: SCHEDULED
Update: pipeline.0000.stage.0000.task.0000 state: SCHEDULED
Update: pipeline.0005.stage.0015.task.0060 state: SCHEDULED
Update: pipeline.0002.stage.0006.task.0024 state: SCHEDULED
Update: pipeline.0009.stage.0027.task.0108 state: SCHEDULED
Update: pipeline.0008.stage.0024.task.0096 state: SCHEDULED
Update: pipeline.0003.stage.0009 state: SCHEDULED
Update: pipeline.0004.stage.0012 state: SCHEDULED
Update: pipeline.0001.stage.0003 state: SCHEDULED
Update: pipeline.0007.stage.0021 state: SCHEDULED
Update: pipeline.0006.stage.0018 state: SCHEDULED
Update: pipeline.0000.stage.0000 state: SCHEDULED
Update: pipeline.0005.stage.0015 state: SCHEDULED
Update: pipeline.0002.stage.0006 state: SCHEDULED
Update: pipeline.0009.stage.0027 state: SCHEDULED
Update: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: https://pymongo.readthedocs.io/en/stable/faq.html#is-pymongo-fork-safe
pipeline.0008.stage.0024 state: SCHEDULED
ok
Update: submit: pipeline.0003.stage.0009.task.0036 state: SUBMITTING
Update: #pipeline.0004.stage.0012.task.0048 state: SUBMITTING
#Update: pipeline.0001.stage.0003.task.0012 state: SUBMITTING
#Update: #pipeline.0007.stage.0021.task.0084 state: SUBMITTING
#Update: #pipeline.0006.stage.0018.task.0072 state: SUBMITTING
Update: #pipeline.0000.stage.0000.task.0000 state: SUBMITTING
#Update: pipeline.0005.stage.0015.task.0060 state: SUBMITTING
#Update: #pipeline.0002.stage.0006.task.0024 state: SUBMITTING
Update: #pipeline.0009.stage.0027.task.0108 state: SUBMITTING
#Update: pipeline.0008.stage.0024.task.0096 state: SUBMITTING
###########################Update: #pipeline.0003.stage.0009.task.0036 state: DONE
Update: #pipeline.0003.stage.0009 state: DONE
#Update: pipeline.0004.stage.0012.task.0048 state: DONE
#Update: #pipeline.0004.stage.0012 state: DONE
Update: #pipeline.0001.stage.0003.task.0012 state: DONE
#Update: pipeline.0001.stage.0003 state: DONE
#Update: #pipeline.0007.stage.0021.task.0084 state: DONE
Update: #pipeline.0007.stage.0021 state: DONE
#Update: pipeline.0006.stage.0018.task.0072 state: DONE
#Update: #pipeline.0006.stage.0018 state: DONE
Update: #pipeline.0000.stage.0000.task.0000 state: DONE
#Update: pipeline.0000.stage.0000 state: DONE
#Update: #pipeline.0005.stage.0015.task.0060 state: DONE
Update: #pipeline.0005.stage.0015 state: DONE
#Update: #pipeline.0002.stage.0006.task.0024 state: DONE
#Update: pipeline.0002.stage.0006 state: DONE
#Update: #pipeline.0009.stage.0027.task.0108 state: EXECUTED
Update: #pipeline.0009.stage.0027.task.0108 state: DONE
#Update: pipeline.0009.stage.0027 state: DONE
#Update: #pipeline.0008.stage.0024.task.0096 state: EXECUTED
Update: #pipeline.0008.stage.0024.task.0096 state: DONE
#Update: pipeline.0008.stage.0024 state: DONE
####Update:
pipeline.0003.stage.0010 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0042 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0041 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0045 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0046 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0044 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0038 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0043 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0040 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0039 state: SCHEDULING
Update: pipeline.0003.stage.0010.task.0037 state: SCHEDULING
Update: pipeline.0004.stage.0013 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0053 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0054 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0058 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0052 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0051 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0050 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0055 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0056 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0057 state: SCHEDULING
Update: pipeline.0004.stage.0013.task.0049 state: SCHEDULING
Update: pipeline.0001.stage.0004 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0016 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0015 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0022 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0017 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0018 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0013 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0019 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0021 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0020 state: SCHEDULING
Update: pipeline.0001.stage.0004.task.0014 state: SCHEDULING
Update: pipeline.0007.stage.0022 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0094 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0085 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0091 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0093 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0092 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0086 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0088 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0089 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0090 state: SCHEDULING
Update: pipeline.0007.stage.0022.task.0087 state: SCHEDULING
Update: pipeline.0006.stage.0019 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0078 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0077 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0079 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0082 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0074 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0080 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0081 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0076 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0073 state: SCHEDULING
Update: pipeline.0006.stage.0019.task.0075 state: SCHEDULING
Update: pipeline.0000.stage.0001 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0001 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0008 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0009 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0003 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0005 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0004 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0002 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0007 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0006 state: SCHEDULING
Update: pipeline.0000.stage.0001.task.0010 state: SCHEDULING
Update: pipeline.0005.stage.0016 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0066 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0068 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0063 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0061 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0064 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0065 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0070 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0067 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0062 state: SCHEDULING
Update: pipeline.0005.stage.0016.task.0069 state: SCHEDULING
Update: pipeline.0002.stage.0007 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0026 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0028 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0027 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0031 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0025 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0029 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0030 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0034 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0033 state: SCHEDULING
Update: pipeline.0002.stage.0007.task.0032 state: SCHEDULING
Update: pipeline.0009.stage.0028 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0113 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0114 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0116 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0110 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0115 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0111 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0117 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0112 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0118 state: SCHEDULING
Update: pipeline.0009.stage.0028.task.0109 state: SCHEDULING
Update: pipeline.0008.stage.0025 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0101 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0099 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0097 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0103 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0106 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0105 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0100 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0104 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0102 state: SCHEDULING
Update: pipeline.0008.stage.0025.task.0098 state: SCHEDULING
Update: Update: pipeline.0003.stage.0010.task.0042 state: SUBMITTING
pipeline.0003.stage.0010.task.0042 state: SUBMITTING
Update: Update: pipeline.0003.stage.0010.task.0041 state: SUBMITTING
pipeline.0003.stage.0010.task.0041 state: SUBMITTING
Update: Update: pipeline.0003.stage.0010.task.0045 state: SUBMITTING
pipeline.0003.stage.0010.task.0045 state: SUBMITTING
Update: Update: pipeline.0003.stage.0010.task.0046 state: SUBMITTING
pipeline.0003.stage.0010.task.0046 state: SUBMITTING
Update: pipeline.0003.stage.0010.task.0044 state: SCHEDULED
Update: pipeline.0003.stage.0010.task.0038 state: SCHEDULED
Update: pipeline.0003.stage.0010.task.0044 state: SUBMITTING
Update: submit: pipeline.0003.stage.0010.task.0043 state: SCHEDULED
Update: Update: pipeline.0003.stage.0010.task.0038 state: SUBMITTING
pipeline.0003.stage.0010.task.0040 state: SCHEDULED
#Update: Update: pipeline.0003.stage.0010.task.0043 state: SUBMITTING
pipeline.0003.stage.0010.task.0039 state: SCHEDULED
Update: #Update: pipeline.0003.stage.0010.task.0037 state: SCHEDULED
pipeline.0003.stage.0010.task.0040 state: SUBMITTING
Update: Update: pipeline.0004.stage.0013.task.0053 state: SCHEDULED
#pipeline.0003.stage.0010.task.0039 state: SUBMITTING
Update: pipeline.0004.stage.0013.task.0054 state: SCHEDULED
Update: #Update: pipeline.0003.stage.0010.task.0037 state: SUBMITTING
#pipeline.0004.stage.0013.task.0058 state: SCHEDULED
Update: Update: #pipeline.0004.stage.0013.task.0053 state: SUBMITTING
pipeline.0004.stage.0013.task.0052 state: SCHEDULED
#Update: Update: pipeline.0004.stage.0013.task.0051 state: SCHEDULED
pipeline.0004.stage.0013.task.0054 state: SUBMITTING
#Update: #Update: pipeline.0004.stage.0013.task.0050 state: SCHEDULED
pipeline.0004.stage.0013.task.0058 state: SUBMITTING
Update: #pipeline.0004.stage.0013.task.0055 state: SCHEDULED
Update: #Update: pipeline.0004.stage.0013.task.0052 state: SUBMITTING
pipeline.0004.stage.0013.task.0056 state: SCHEDULED
Update: #Update: pipeline.0004.stage.0013.task.0051 state: SUBMITTING
#pipeline.0004.stage.0013.task.0057 state: SCHEDULED
Update: Update: pipeline.0004.stage.0013.task.0050 state: SUBMITTING
#pipeline.0004.stage.0013.task.0049 state: SCHEDULED
#Update: Update: pipeline.0004.stage.0013.task.0055 state: SUBMITTING
pipeline.0001.stage.0004.task.0016 state: SCHEDULED
#Update: Update: #pipeline.0001.stage.0004.task.0015 state: SCHEDULED
pipeline.0004.stage.0013.task.0056 state: SUBMITTING
Update: Update: #pipeline.0001.stage.0004.task.0022 state: SCHEDULED
pipeline.0004.stage.0013.task.0057 state: SUBMITTING
#Update: Update: pipeline.0001.stage.0004.task.0017 state: SCHEDULED
pipeline.0004.stage.0013.task.0049 state: SUBMITTING
#Update: #Update: pipeline.0001.stage.0004.task.0018 state: SCHEDULED
pipeline.0001.stage.0004.task.0016 state: SUBMITTING
Update: #pipeline.0001.stage.0004.task.0013 state: SCHEDULED
Update: #pipeline.0001.stage.0004.task.0015 state: SUBMITTING
Update: pipeline.0001.stage.0004.task.0019 state: SCHEDULED
Update: #Update: pipeline.0001.stage.0004.task.0022 state: SUBMITTING
#pipeline.0001.stage.0004.task.0021 state: SCHEDULED
Update: Update: pipeline.0001.stage.0004.task.0017 state: SUBMITTING
#pipeline.0001.stage.0004.task.0020 state: SCHEDULED
#Update: Update: pipeline.0001.stage.0004.task.0018 state: SUBMITTING
pipeline.0001.stage.0004.task.0014 state: SCHEDULED
#Update: Update: #pipeline.0007.stage.0022.task.0094 state: SCHEDULED
pipeline.0001.stage.0004.task.0013 state: SUBMITTING
Update: Update: #pipeline.0007.stage.0022.task.0085 state: SCHEDULED
pipeline.0001.stage.0004.task.0019 state: SUBMITTING
#Update: Update: pipeline.0007.stage.0022.task.0091 state: SCHEDULED
pipeline.0001.stage.0004.task.0021 state: SUBMITTING
#Update: Update: #pipeline.0007.stage.0022.task.0093 state: SCHEDULED
pipeline.0001.stage.0004.task.0020 state: SUBMITTING
Update: Update: #pipeline.0001.stage.0004.task.0014 state: SUBMITTING
pipeline.0007.stage.0022.task.0092 state: SCHEDULED
#Update: Update: pipeline.0007.stage.0022.task.0086 state: SCHEDULED
pipeline.0007.stage.0022.task.0094 state: SUBMITTING
#Update: Update: #pipeline.0007.stage.0022.task.0088 state: SCHEDULED
pipeline.0007.stage.0022.task.0085 state: SUBMITTING
Update: Update: #pipeline.0007.stage.0022.task.0089 state: SCHEDULED
pipeline.0007.stage.0022.task.0091 state: SUBMITTING
#Update: Update: pipeline.0007.stage.0022.task.0090 state: SCHEDULED
pipeline.0007.stage.0022.task.0093 state: SUBMITTING
#Update: Update: #pipeline.0007.stage.0022.task.0087 state: SCHEDULED
pipeline.0007.stage.0022.task.0092 state: SUBMITTING
Update: Update: #pipeline.0006.stage.0019.task.0078 state: SCHEDULED
pipeline.0007.stage.0022.task.0086 state: SUBMITTING
#Update: pipeline.0006.stage.0019.task.0077 state: SCHEDULED
Update: #Update: pipeline.0007.stage.0022.task.0088 state: SUBMITTING
#pipeline.0006.stage.0019.task.0079 state: SCHEDULED
Update: Update: pipeline.0007.stage.0022.task.0089 state: SUBMITTING
#pipeline.0006.stage.0019.task.0082 state: SCHEDULED
Update: #Update: pipeline.0007.stage.0022.task.0090 state: SUBMITTING
pipeline.0006.stage.0019.task.0074 state: SCHEDULED
#Update: Update: #pipeline.0007.stage.0022.task.0087 state: SUBMITTING
pipeline.0006.stage.0019.task.0080 state: SCHEDULED
Update: Update: #pipeline.0006.stage.0019.task.0081 state: SCHEDULED
pipeline.0006.stage.0019.task.0078 state: SUBMITTING
#Update: Update: pipeline.0006.stage.0019.task.0077 state: SUBMITTING
pipeline.0006.stage.0019.task.0076 state: SCHEDULED
#Update: Update: #pipeline.0006.stage.0019.task.0073 state: SCHEDULED
pipeline.0006.stage.0019.task.0079 state: SUBMITTING
Update: Update: #pipeline.0006.stage.0019.task.0075 state: SCHEDULED
pipeline.0006.stage.0019.task.0082 state: SUBMITTING
#Update: Update: pipeline.0000.stage.0001.task.0001 state: SCHEDULED
pipeline.0006.stage.0019.task.0074 state: SUBMITTING
#Update: #Update: pipeline.0000.stage.0001.task.0008 state: SCHEDULED
pipeline.0006.stage.0019.task.0080 state: SUBMITTING
Update: #pipeline.0000.stage.0001.task.0009 state: SCHEDULED
Update: #Update: pipeline.0006.stage.0019.task.0081 state: SUBMITTING
pipeline.0000.stage.0001.task.0003 state: SCHEDULED
Update: #Update: pipeline.0006.stage.0019.task.0076 state: SUBMITTING
#pipeline.0000.stage.0001.task.0005 state: SCHEDULED
Update: Update: pipeline.0006.stage.0019.task.0073 state: SUBMITTING
#pipeline.0000.stage.0001.task.0004 state: SCHEDULED
#Update: Update: pipeline.0006.stage.0019.task.0075 state: SUBMITTING
pipeline.0000.stage.0001.task.0002 state: SCHEDULED
#Update: Update: #pipeline.0000.stage.0001.task.0007 state: SCHEDULED
pipeline.0000.stage.0001.task.0001 state: SUBMITTING
Update: Update: #pipeline.0000.stage.0001.task.0006 state: SCHEDULED
pipeline.0000.stage.0001.task.0008 state: SUBMITTING
#Update: Update: pipeline.0000.stage.0001.task.0010 state: SCHEDULED
pipeline.0000.stage.0001.task.0009 state: SUBMITTING
#Update: Update: #pipeline.0005.stage.0016.task.0066 state: SCHEDULED
pipeline.0000.stage.0001.task.0003 state: SUBMITTING
Update: #Update: pipeline.0005.stage.0016.task.0068 state: SCHEDULED
#pipeline.0000.stage.0001.task.0005 state: SUBMITTING
Update: pipeline.0005.stage.0016.task.0063 state: SCHEDULED
Update: #pipeline.0000.stage.0001.task.0004 state: SUBMITTING
Update:
pipeline.0005.stage.0016.task.0061 state: SCHEDULED
Update: Update: pipeline.0000.stage.0001.task.0002 state: SUBMITTING
pipeline.0005.stage.0016.task.0064 state: SCHEDULED
Update: Update: pipeline.0000.stage.0001.task.0007 state: SUBMITTING
pipeline.0005.stage.0016.task.0065 state: SCHEDULED
Update: pipeline.0005.stage.0016.task.0070 state: SCHEDULED
Update: Update: pipeline.0000.stage.0001.task.0006 state: SUBMITTING
pipeline.0005.stage.0016.task.0067 state: SCHEDULED
Update: Update: pipeline.0005.stage.0016.task.0062 state: SCHEDULED
pipeline.0000.stage.0001.task.0010 state: SUBMITTING
Update: pipeline.0005.stage.0016.task.0069 state: SCHEDULED
Update: Update: pipeline.0002.stage.0007.task.0026 state: SCHEDULED
pipeline.0005.stage.0016.task.0066 state: SUBMITTING
Update: pipeline.0002.stage.0007.task.0028 state: SCHEDULED
Update: Update: pipeline.0002.stage.0007.task.0027 state: SCHEDULED
pipeline.0005.stage.0016.task.0068 state: SUBMITTING
Update: pipeline.0002.stage.0007.task.0031 state: SCHEDULED
Update: pipeline.0002.stage.0007.task.0025 state: SCHEDULED
Update: Update: pipeline.0002.stage.0007.task.0029 state: SCHEDULED
pipeline.0005.stage.0016.task.0063 state: SUBMITTING
Update: pipeline.0002.stage.0007.task.0030 state: SCHEDULED
Update: Update: pipeline.0002.stage.0007.task.0034 state: SCHEDULED
pipeline.0005.stage.0016.task.0061 state: SUBMITTING
Update: Update: pipeline.0002.stage.0007.task.0033 state: SCHEDULED
pipeline.0005.stage.0016.task.0064 state: SUBMITTING
Update: Update: pipeline.0002.stage.0007.task.0032 state: SCHEDULED
pipeline.0005.stage.0016.task.0065 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0113 state: SCHEDULED
Update: Update: pipeline.0009.stage.0028.task.0114 state: SCHEDULED
pipeline.0005.stage.0016.task.0070 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0116 state: SCHEDULED
Update: Update: pipeline.0005.stage.0016.task.0067 state: SUBMITTING
pipeline.0009.stage.0028.task.0110 state: SCHEDULED
Update: Update: pipeline.0009.stage.0028.task.0115 state: SCHEDULED
pipeline.0005.stage.0016.task.0062 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0111 state: SCHEDULED
Update: Update: pipeline.0009.stage.0028.task.0117 state: SCHEDULED
pipeline.0005.stage.0016.task.0069 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0112 state: SCHEDULED
Update: Update: pipeline.0009.stage.0028.task.0118 state: SCHEDULED
pipeline.0002.stage.0007.task.0026 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0109 state: SCHEDULED
Update: Update: pipeline.0008.stage.0025.task.0101 state: SCHEDULED
pipeline.0002.stage.0007.task.0028 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0099 state: SCHEDULED
Update: Update: pipeline.0002.stage.0007.task.0027 state: SUBMITTING
pipeline.0008.stage.0025.task.0097 state: SCHEDULED
Update: pipeline.0008.stage.0025.task.0103 state: SCHEDULED
Update: Update: pipeline.0008.stage.0025.task.0106 state: SCHEDULED
pipeline.0002.stage.0007.task.0031 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0105 state: SCHEDULED
Update: Update: pipeline.0008.stage.0025.task.0100 state: SCHEDULED
pipeline.0002.stage.0007.task.0025 state: SUBMITTING
Update: Update: pipeline.0008.stage.0025.task.0104 state: SCHEDULED
pipeline.0002.stage.0007.task.0029 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0102 state: SCHEDULED
Update: pipeline.0008.stage.0025.task.0098 state: SCHEDULED
Update: pipeline.0003.stage.0010 state: SCHEDULED
Update: pipeline.0002.stage.0007.task.0030 state: SUBMITTING
Update: pipeline.0004.stage.0013 state: SCHEDULED
Update: Update: pipeline.0001.stage.0004 state: SCHEDULED
pipeline.0002.stage.0007.task.0034 state: SUBMITTING
Update: pipeline.0007.stage.0022 state: SCHEDULED
Update: Update: pipeline.0006.stage.0019 state: SCHEDULED
pipeline.0002.stage.0007.task.0033 state: SUBMITTING
Update: pipeline.0000.stage.0001 state: SCHEDULED
Update: Update: pipeline.0005.stage.0016 state: SCHEDULED
pipeline.0002.stage.0007.task.0032 state: SUBMITTING
Update: pipeline.0002.stage.0007 state: SCHEDULED
Update: pipeline.0009.stage.0028.task.0113 state: SUBMITTING
Update: pipeline.0009.stage.0028 state: SCHEDULED
Update: Update: pipeline.0009.stage.0028.task.0114 state: SUBMITTING
pipeline.0008.stage.0025 state: SCHEDULED
Update: pipeline.0009.stage.0028.task.0116 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0110 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0115 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0111 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0117 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0112 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0118 state: SUBMITTING
Update: pipeline.0009.stage.0028.task.0109 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0101 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0099 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0097 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0103 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0106 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0105 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0100 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0104 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0102 state: SUBMITTING
Update: pipeline.0008.stage.0025.task.0098 state: SUBMITTING
Update: pipeline.0003.stage.0010.task.0042 state: DONE
Update: pipeline.0003.stage.0010.task.0041 state: DONE
Update: pipeline.0003.stage.0010.task.0045 state: DONE
Update: pipeline.0003.stage.0010.task.0046 state: DONE
Update: pipeline.0003.stage.0010.task.0044 state: DONE
Update: pipeline.0003.stage.0010.task.0038 state: DONE
Update: pipeline.0003.stage.0010.task.0043 state: DONE
Update: pipeline.0003.stage.0010.task.0040 state: EXECUTED
Update: pipeline.0003.stage.0010.task.0040 state: DONE
Update: pipeline.0003.stage.0010.task.0039 state: EXECUTED
Update: pipeline.0003.stage.0010.task.0039 state: DONE
Update: pipeline.0003.stage.0010.task.0037 state: EXECUTED
Update: pipeline.0003.stage.0010.task.0037 state: DONE
Update: pipeline.0003.stage.0010 state: DONE
Update: pipeline.0004.stage.0013.task.0053 state: EXECUTED
Update: pipeline.0004.stage.0013.task.0053 state: DONE
Update: pipeline.0004.stage.0013.task.0054 state: EXECUTED
Update: pipeline.0004.stage.0013.task.0054 state: DONE
Update: pipeline.0004.stage.0013.task.0052 state: EXECUTED
Update: pipeline.0004.stage.0013.task.0052 state: DONE
Update: pipeline.0004.stage.0013.task.0055 state: EXECUTED
Update: pipeline.0004.stage.0013.task.0055 state: DONE
Update: pipeline.0004.stage.0013.task.0056 state: EXECUTED
Update: pipeline.0004.stage.0013.task.0056 state: DONE
Update: pipeline.0001.stage.0004.task.0016 state: EXECUTED
Update: pipeline.0001.stage.0004.task.0016 state: DONE
Update: pipeline.0001.stage.0004.task.0017 state: EXECUTED
Update: pipeline.0001.stage.0004.task.0017 state: DONE
Update: pipeline.0001.stage.0004.task.0018 state: EXECUTED
Update: pipeline.0001.stage.0004.task.0018 state: DONE
Update: pipeline.0001.stage.0004.task.0020 state: EXECUTED
Update: pipeline.0001.stage.0004.task.0020 state: DONE
Update: pipeline.0007.stage.0022.task.0094 state: EXECUTED
Update: pipeline.0007.stage.0022.task.0094 state: DONE
Update: pipeline.0007.stage.0022.task.0091 state: EXECUTED
Update: pipeline.0007.stage.0022.task.0091 state: DONE
Update: pipeline.0007.stage.0022.task.0092 state: EXECUTED
Update: pipeline.0007.stage.0022.task.0092 state: DONE
Update: pipeline.0006.stage.0019.task.0081 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0081 state: DONE
Update: pipeline.0006.stage.0019.task.0076 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0076 state: DONE
Update: pipeline.0000.stage.0001.task.0001 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0001 state: DONE
Update: pipeline.0000.stage.0001.task.0005 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0005 state: DONE
Update: pipeline.0005.stage.0016.task.0064 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0064 state: DONE
Update: pipeline.0005.stage.0016.task.0070 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0070 state: DONE
Update: pipeline.0002.stage.0007.task.0026 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0026 state: DONE
Update: pipeline.0002.stage.0007.task.0032 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0032 state: DONE
Update: pipeline.0009.stage.0028.task.0113 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0113 state: DONE
Update: pipeline.0009.stage.0028.task.0110 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0110 state: DONE
Update: pipeline.0009.stage.0028.task.0115 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0115 state: DONE
Update: pipeline.0009.stage.0028.task.0117 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0117 state: DONE
Update: pipeline.0009.stage.0028.task.0109 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0109 state: DONE
Update: pipeline.0008.stage.0025.task.0101 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0101 state: DONE
Update: pipeline.0008.stage.0025.task.0099 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0099 state: DONE
Update: pipeline.0008.stage.0025.task.0097 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0097 state: DONE
Update: pipeline.0008.stage.0025.task.0103 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0103 state: DONE
Update: pipeline.0008.stage.0025.task.0106 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0106 state: DONE
Update: pipeline.0008.stage.0025.task.0105 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0105 state: DONE
Update: pipeline.0008.stage.0025.task.0100 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0100 state: DONE
Update: pipeline.0008.stage.0025.task.0104 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0104 state: DONE
Update: pipeline.0008.stage.0025.task.0102 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0102 state: DONE
Update: pipeline.0008.stage.0025.task.0098 state: EXECUTED
Update: pipeline.0008.stage.0025.task.0098 state: DONE
Update: pipeline.0008.stage.0025 state: DONE
Update: pipeline.0003.stage.0011 state: SCHEDULING
Update: pipeline.0003.stage.0011.task.0047 state: SCHEDULING
Update: pipeline.0008.stage.0026 state: SCHEDULING
Update: pipeline.0008.stage.0026.task.0107 state: SCHEDULING
submit: Update: Update: pipeline.0003.stage.0011.task.0047 state: SUBMITTING
pipeline.0003.stage.0011.task.0047 state: SUBMITTING
#Update: Update: #pipeline.0008.stage.0026.task.0107 state: SUBMITTING
pipeline.0008.stage.0026.task.0107 state: SUBMITTING
#Update: Update: #pipeline.0004.stage.0013.task.0058 state: DONE
pipeline.0003.stage.0011 state: SCHEDULED
#Update: Update: #pipeline.0004.stage.0013.task.0051 state: DONE
pipeline.0008.stage.0026 state: SCHEDULED
#Update: #pipeline.0004.stage.0013.task.0050 state: DONE
#Update: #pipeline.0004.stage.0013.task.0057 state: EXECUTED
#Update: #pipeline.0004.stage.0013.task.0057 state: DONE
#Update: #pipeline.0004.stage.0013.task.0049 state: EXECUTED
#Update: #pipeline.0004.stage.0013.task.0049 state: DONE
#Update: #pipeline.0004.stage.0013 state: DONE
##Update: #pipeline.0001.stage.0004.task.0015 state: EXECUTED
#Update: ##pipeline.0001.stage.0004.task.0015 state: DONE
#Update: #pipeline.0001.stage.0004.task.0022 state: EXECUTED
##Update: #pipeline.0001.stage.0004.task.0022 state: DONE
#Update: ##pipeline.0001.stage.0004.task.0013 state: EXECUTED
#Update: #pipeline.0001.stage.0004.task.0013 state: DONE
##Update: #pipeline.0001.stage.0004.task.0019 state: EXECUTED
#Update: ##pipeline.0001.stage.0004.task.0019 state: DONE
#Update: #pipeline.0001.stage.0004.task.0021 state: EXECUTED
##Update: #pipeline.0001.stage.0004.task.0021 state: DONE
#Update: ##pipeline.0001.stage.0004.task.0014 state: EXECUTED
#Update: #pipeline.0001.stage.0004.task.0014 state: DONE
##Update: #pipeline.0001.stage.0004 state: DONE
#Update: ##pipeline.0007.stage.0022.task.0085 state: EXECUTED
#Update: ##pipeline.0007.stage.0022.task.0085 state: DONE
#Update: ##pipeline.0007.stage.0022.task.0093 state: EXECUTED
##Update: #pipeline.0007.stage.0022.task.0093 state: DONE
##Update: ##pipeline.0007.stage.0022.task.0086 state: EXECUTED
#Update: ##pipeline.0007.stage.0022.task.0086 state: DONE
Update: pipeline.0007.stage.0022.task.0088 state: EXECUTED
Update: pipeline.0007.stage.0022.task.0088 state: DONE
Update: pipeline.0007.stage.0022.task.0089 state: EXECUTED
Update: pipeline.0007.stage.0022.task.0089 state: DONE
Update: pipeline.0007.stage.0022.task.0090 state: EXECUTED
Update: pipeline.0007.stage.0022.task.0090 state: DONE
Update: pipeline.0007.stage.0022.task.0087 state: EXECUTED
Update: pipeline.0007.stage.0022.task.0087 state: DONE
Update: pipeline.0007.stage.0022 state: DONE
Update: pipeline.0006.stage.0019.task.0077 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0077 state: DONE
Update: pipeline.0006.stage.0019.task.0074 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0074 state: DONE
Update: pipeline.0006.stage.0019.task.0080 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0080 state: DONE
Update: pipeline.0006.stage.0019.task.0073 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0073 state: DONE
Update: pipeline.0006.stage.0019.task.0075 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0075 state: DONE
Update: pipeline.0000.stage.0001.task.0008 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0008 state: DONE
Update: pipeline.0000.stage.0001.task.0009 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0009 state: DONE
Update: pipeline.0000.stage.0001.task.0003 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0003 state: DONE
Update: pipeline.0000.stage.0001.task.0002 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0002 state: DONE
Update: pipeline.0000.stage.0001.task.0007 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0007 state: DONE
Update: pipeline.0000.stage.0001.task.0006 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0006 state: DONE
Update: pipeline.0000.stage.0001.task.0010 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0010 state: DONE
Update: pipeline.0005.stage.0016.task.0066 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0066 state: DONE
Update: pipeline.0005.stage.0016.task.0068 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0068 state: DONE
Update: pipeline.0005.stage.0016.task.0063 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0063 state: DONE
Update: pipeline.0005.stage.0016.task.0061 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0061 state: DONE
Update: pipeline.0005.stage.0016.task.0065 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0065 state: DONE
Update: pipeline.0005.stage.0016.task.0067 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0067 state: DONE
Update: pipeline.0005.stage.0016.task.0069 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0069 state: DONE
Update: pipeline.0002.stage.0007.task.0028 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0028 state: DONE
Update: pipeline.0002.stage.0007.task.0027 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0027 state: DONE
Update: pipeline.0002.stage.0007.task.0031 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0031 state: DONE
Update: pipeline.0002.stage.0007.task.0025 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0025 state: DONE
Update: pipeline.0002.stage.0007.task.0029 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0029 state: DONE
Update: pipeline.0002.stage.0007.task.0030 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0030 state: DONE
Update: pipeline.0002.stage.0007.task.0034 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0034 state: DONE
Update: pipeline.0002.stage.0007.task.0033 state: EXECUTED
Update: pipeline.0002.stage.0007.task.0033 state: DONE
Update: pipeline.0002.stage.0007 state: DONE
Update: pipeline.0009.stage.0028.task.0114 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0114 state: DONE
Update: pipeline.0009.stage.0028.task.0116 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0116 state: DONE
Update: pipeline.0009.stage.0028.task.0111 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0111 state: DONE
Update: pipeline.0009.stage.0028.task.0112 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0112 state: DONE
Update: pipeline.0009.stage.0028.task.0118 state: EXECUTED
Update: pipeline.0009.stage.0028.task.0118 state: DONE
Update: pipeline.0009.stage.0028 state: DONE
Update: pipeline.0006.stage.0019.task.0078 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0078 state: DONE
Update: pipeline.0006.stage.0019.task.0079 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0079 state: DONE
Update: pipeline.0006.stage.0019.task.0082 state: EXECUTED
Update: pipeline.0006.stage.0019.task.0082 state: DONE
Update: pipeline.0006.stage.0019 state: DONE
Update: pipeline.0000.stage.0001.task.0004 state: EXECUTED
Update: pipeline.0000.stage.0001.task.0004 state: DONE
Update: pipeline.0000.stage.0001 state: DONE
Update: pipeline.0005.stage.0016.task.0062 state: EXECUTED
Update: pipeline.0005.stage.0016.task.0062 state: DONE
Update: pipeline.0005.stage.0016 state: DONE
Update: pipeline.0003.stage.0011.task.0047 state: DONE
Update: pipeline.0003.stage.0011 state: DONE
Update: pipeline.0003 state: DONE
Update: pipeline.0008.stage.0026.task.0107 state: DONE
Update: pipeline.0008.stage.0026 state: DONE
Update: pipeline.0008 state: DONE
Update: pipeline.0004.stage.0014 state: SCHEDULING
Update: pipeline.0004.stage.0014.task.0059 state: SCHEDULING
Update: pipeline.0001.stage.0005 state: SCHEDULING
Update: pipeline.0001.stage.0005.task.0023 state: SCHEDULING
Update: pipeline.0007.stage.0023 state: SCHEDULING
Update: pipeline.0007.stage.0023.task.0095 state: SCHEDULING
Update: pipeline.0006.stage.0020 state: SCHEDULING
Update: pipeline.0006.stage.0020.task.0083 state: SCHEDULING
Update: pipeline.0000.stage.0002 state: SCHEDULING
Update: pipeline.0000.stage.0002.task.0011 state: SCHEDULING
Update: pipeline.0005.stage.0017 state: SCHEDULING
Update: pipeline.0005.stage.0017.task.0071 state: SCHEDULING
Update: pipeline.0002.stage.0008 state: SCHEDULING
Update: pipeline.0002.stage.0008.task.0035 state: SCHEDULING
Update: pipeline.0009.stage.0029 state: SCHEDULING
Update: pipeline.0009.stage.0029.task.0119 state: SCHEDULING
submit: Update: Update: pipeline.0004.stage.0014.task.0059 state: SUBMITTING
pipeline.0004.stage.0014.task.0059 state: SUBMITTING
#Update: Update: #pipeline.0001.stage.0005.task.0023 state: SUBMITTING
pipeline.0001.stage.0005.task.0023 state: SUBMITTING
Update: Update: #pipeline.0007.stage.0023.task.0095 state: SUBMITTING
pipeline.0007.stage.0023.task.0095 state: SUBMITTING
##Update: Update: #pipeline.0006.stage.0020.task.0083 state: SUBMITTING
pipeline.0006.stage.0020.task.0083 state: SUBMITTING
#Update: Update: ##pipeline.0000.stage.0002.task.0011 state: SUBMITTING
pipeline.0000.stage.0002.task.0011 state: SUBMITTING
#Update: Update: #pipeline.0005.stage.0017.task.0071 state: SUBMITTING
pipeline.0005.stage.0017.task.0071 state: SUBMITTING
##Update: Update: #pipeline.0002.stage.0008.task.0035 state: SUBMITTING
pipeline.0002.stage.0008.task.0035 state: SUBMITTING
#Update: Update: ##pipeline.0009.stage.0029.task.0119 state: SUBMITTING
pipeline.0009.stage.0029.task.0119 state: SUBMITTING
#Update: #pipeline.0004.stage.0014 state: SCHEDULED
##Update: #pipeline.0001.stage.0005 state: SCHEDULED
#Update: ##pipeline.0007.stage.0023 state: SCHEDULED
#Update: #pipeline.0006.stage.0020 state: SCHEDULED
##Update: #pipeline.0000.stage.0002 state: SCHEDULED
#Update: ##pipeline.0005.stage.0017 state: SCHEDULED
#Update: #pipeline.0002.stage.0008 state: SCHEDULED
Update: ##pipeline.0004.stage.0014.task.0059 state: DONE
Update: #pipeline.0009.stage.0029 state: SCHEDULED
Update: #pipeline.0004.stage.0014 state: DONE
##Update: #pipeline.0004 state: DONE
#Update: ##pipeline.0001.stage.0005.task.0023 state: DONE
#Update: #pipeline.0001.stage.0005 state: DONE
##Update: #pipeline.0001 state: DONE
#Update: ##pipeline.0007.stage.0023.task.0095 state: EXECUTED
#Update: #pipeline.0007.stage.0023.task.0095 state: DONE
##Update: #pipeline.0007.stage.0023 state: DONE
#Update: ##pipeline.0007 state: DONE
#Update: #pipeline.0006.stage.0020.task.0083 state: EXECUTED
##Update: #pipeline.0006.stage.0020.task.0083 state: DONE
#Update: ##pipeline.0006.stage.0020 state: DONE
#Update: #pipeline.0006 state: DONE
#
Update: pipeline.0000.stage.0002.task.0011 state: EXECUTED
Update: pipeline.0000.stage.0002.task.0011 state: DONE
Update: pipeline.0000.stage.0002 state: DONE
Update: pipeline.0000 state: DONE
Update: pipeline.0005.stage.0017.task.0071 state: DONE
Update: pipeline.0005.stage.0017 state: DONE
Update: pipeline.0005 state: DONE
Update: pipeline.0002.stage.0008.task.0035 state: DONE
Update: pipeline.0002.stage.0008 state: DONE
Update: pipeline.0002 state: DONE
Update: pipeline.0009.stage.0029.task.0119 state: DONE
Update: pipeline.0009.stage.0029 state: DONE
Update: pipeline.0009 state: DONE
close task manager ok
1669119599.495 : radical.entk.task_manager.0000 : 2110368 : 139860056708864 : ERROR : Heartbeat response no body
closing session re.session.radical.3.mturilli.019318.0000 \
close pilot manager \
wait for 1 pilot(s)
O39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m39m timeout
ok
session lifetime: 120.3s ok
All components terminated
We check results which were staged back
[10]:
for cnt in range(n_pipelines):
data = open('pipe.%03d.sum.txt' % cnt).read()
result = int(data)
print('%3d -- %25d' % (cnt, result))
0 -- 3970061241222038761260
1 -- 842343099884068959
2 -- 1544905336569047335135
3 -- 614901618755976010525
4 -- 2032690349846648892114
5 -- 46503479300141160
6 -- 3970061241222038761260
7 -- 3694381919610155001250
8 -- 832923773978133231460
9 -- 1855803167254306908