I'm using IPython from current Git and created some parallel tasks which are queued one after the other to one engine (HWM = 1).
After that I abort one of them:
In [12]: client.abort("2e8e2c96-1767-4c7e-b26c-498682fa4c85")
Now the task looks as follows:
In [13]: client.db_query({'msg_id' : "2e8e2c96-1767-4c7e-b26c-498682fa4c85" })
Out[13]:
[{u'client_uuid': u'0b138971-90a2-48c6-8f03-77a1a3434667',
u'completed': None,
u'content': {},
u'engine_uuid': u'7040bc13-0809-4fc6-bb68-d4344d6e5e61',
u'header': {u'after': [],
u'date': datetime.datetime(2012, 3, 1, 22, 41, 44, 342000),
u'follow': [],
u'msg_id': u'2e8e2c96-1767-4c7e-b26c-498682fa4c85',
u'msg_type': u'apply_request',
u'retries': 1000,
u'session': u'4f4fed1853c27b709f000000',
u'targets': [],
u'timeout': 0.0,
u'username': u'4f44271053c27b135e000004'},
u'msg_id': u'2e8e2c96-1767-4c7e-b26c-498682fa4c85',
u'pyerr': None,
u'pyin': None,
u'pyout': None,
u'queue': u'task',
u'received': None,
u'resubmitted': datetime.datetime(2012, 4, 23, 14, 54, 36, 86000),
u'result_content': None,
u'result_header': None,
u'started': None,
u'stderr': u'',
u'stdout': u'',
u'submitted': datetime.datetime(2012, 3, 1, 22, 41, 44, 342000)}]
When I now try to resubmit that aborted task I get the following exception:
In [14]: client.resubmit("2e8e2c96-1767-4c7e-b26c-498682fa4c85")
---------------------------------------------------------------------------
RemoteError Traceback (most recent call last)
/Users/kaazoo/<ipython-input-15-bd12698bd336> in <module>()
----> 1 client.resubmit("2e8e2c96-1767-4c7e-b26c-498682fa4c85")
/Users/kaazoo/<string> in resubmit(self, indices_or_msg_ids, subheader, block)
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/parallel/client/client.pyc in spin_first(f, self, *args, **kwargs)
65 """Call spin() to sync state prior to calling the method."""
66 self.spin()
---> 67 return f(self, *args, **kwargs)
68
69
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/parallel/client/client.pyc in resubmit(self, indices_or_msg_ids, subheader, block)
1238 content = msg['content']
1239 if content['status'] != 'ok':
-> 1240 raise self._unwrap_exception(content)
1241
1242 ar = AsyncHubResult(self, msg_ids=theids)
RemoteError: ValueError(Task u'2e8e2c96-1767-4c7e-b26c-498682fa4c85' appears to be inflight)
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/IPython/parallel/controller/hub.py", line 1149, in resubmit_task
raise ValueError("Task %r appears to be inflight" % msg_id)
ValueError: Task u'2e8e2c96-1767-4c7e-b26c-498682fa4c85' appears to be inflight
Why is that task still inflight? It shouldn't be sitting on the engine anymore.
But it seems as if the engine still has that task:
In [27]: client.queue_status("All", True)
Out[27]:
{2: {u'completed': [u'3a09be07-00ff-4f5e-b1b9-6909c73a005e',
u'9f5ce31a-e97f-4aff-831a-c4f0b2ff2de4',
u'db9ea4d0-e861-4e2f-952d-1b53f75ffd54',
u'2368e382-f0d6-4677-b3a7-67194834a6a4',
u'0cc2593b-6189-44f0-b8c5-f9594d037121',
u'357b7c65-ac62-4b72-81d9-7ccf110c663c',
u'f8adf60a-73f0-4634-9dc0-5ea620bbbcae',
u'cf2e30b1-1a53-4d3f-83bf-56b182207bb5',
u'8843d70a-008f-4cb4-9836-01836bc0666a',
u'd78a02df-3e63-4cc3-86f5-32b52ff75cfb',
u'cad9d610-04a4-4cfc-90a2-9cf1e9c4819c',
u'65abf0fa-57ce-4615-9f40-9ba672650b26',
u'a41de711-8ad3-4876-979b-94627291b14c',
u'287d6068-585e-4d46-82f6-542e28e8abde',
u'e776bac6-90cd-4593-8fb0-a05eb4ecf2f8',
u'4e125e84-3cab-4e8f-a4b7-ac84fbdd46cc',
u'86753b71-c232-4a66-93ce-c71461937a32',
u'82b21f2a-7ce5-4dcb-b690-f570d96a546d',
u'7456404f-0f6a-4a67-85e9-04f4fe45ead5',
u'8545ec24-24be-4d3f-bf07-264c5b2a20aa',
u'4a418dc3-47bf-4554-b3b6-8f645580760e'],
u'queue': [],
u'tasks': [u'2e8e2c96-1767-4c7e-b26c-498682fa4c85']},
u'unassigned': []}
Am I missing something?
Is this correct behavior?
How can I get rid of inflight tasks without restarting the engine?
I'm using IPython from current Git and created some parallel tasks which are queued one after the other to one engine (HWM = 1).
After that I abort one of them:
Now the task looks as follows:
When I now try to resubmit that aborted task I get the following exception:
Why is that task still inflight? It shouldn't be sitting on the engine anymore.
But it seems as if the engine still has that task:
Am I missing something?
Is this correct behavior?
How can I get rid of inflight tasks without restarting the engine?