Skip to content

Merge divergent Kernel implementations #1630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ellisonbg merged 69 commits into from
May 25, 2012
Merged

Merge divergent Kernel implementations #1630

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
ff8a701
drop in apply_request handler in zmq.ipkernel
minrk Mar 23, 2012
09cd42a
lazy-formatting logs in ipkernel
minrk Mar 23, 2012
1901b79
allow multiple shell_sockets in ipkernel
minrk Mar 23, 2012
1587a7f
allow aborting tasks
minrk Mar 23, 2012
fbf9e2e
respect copy flag in Session.recv
minrk Mar 24, 2012
d362026
Don't copy shell requests
minrk Mar 24, 2012
ed57daf
move apply serialization into zmq.serialize
minrk Mar 24, 2012
6174cb0
add ids to ipkernel
minrk Mar 24, 2012
9440f9e
use IOLoop in ipkernel
minrk Mar 24, 2012
24fdfb1
use IPython.zmq.kernel in parallel Engines
minrk Mar 24, 2012
5c5118d
allow exceptions in handlers without crashing
minrk Mar 24, 2012
7f6086f
use execute_request for parallel execute, instead of apply
minrk Mar 24, 2012
08b95f8
scrub: ident on iopub
minrk Mar 24, 2012
ccd2ca4
enable topics on IOPub messages
minrk Mar 24, 2012
02449c1
use shell namespaces in apply_request
minrk Mar 24, 2012
c6717a5
don't abort queue on execution failure
minrk Mar 24, 2012
716157f
avoid passing code objects to execute, now that execute uses code-as-…
minrk Mar 24, 2012
f9d7005
fixup shutdown/exit now that we use IOLoop
minrk Mar 24, 2012
3724f43
remove now-obsolete kernel fork used in engine
minrk Mar 24, 2012
cfdfd64
don't use history files in Engines
minrk Mar 25, 2012
99269af
add silent kwarg to run_cell
minrk Mar 28, 2012
de14aa1
use new run_cell(silent=True) instead of run_code in ipkernel
minrk Mar 28, 2012
c5c7935
do not use ZMQStream for IOPub
minrk Apr 19, 2012
2f3f0c8
remove disused pykernel
minrk Apr 19, 2012
4048621
remove references to removed pykernel and associated pure flag
minrk Apr 23, 2012
5cc602b
fix subheaders for execute_reply and aborted messages
minrk Apr 24, 2012
3531213
test resubmit of aborted tasks (#1647)
minrk Apr 24, 2012
4cda2dc
fix Kernel import in engineapp
minrk May 8, 2012
d8f08af
fix kernel_argv in qtconsole
minrk May 8, 2012
788c738
expose silent in view.execute
minrk May 8, 2012
60c884a
track display data in the parallel Client
minrk May 8, 2012
a27adad
discard parallel.util.asbytes in favor of py3compat.cast_bytes
minrk May 9, 2012
e0899e5
fix topic on displayhook in ZMQShell
minrk May 12, 2012
e7f402a
fix message when trying new table due to mismatch in SQLiteDB
minrk May 12, 2012
4e5eb7d
soften messages for monitor messages with bad topics
minrk May 12, 2012
fa181ff
resubmitted tasks are now wholly separate (new msg_ids)
minrk May 12, 2012
3c9c367
remove old _really_apply call
minrk May 13, 2012
6423e9d
store execute_replies in a nice wrapper
minrk May 13, 2012
d34fb7b
minor fixes to allow kernel to be re-entrant
minrk May 15, 2012
f56f5a1
upstream change preventing kernel exit on SIGINT during eventloop int…
minrk May 15, 2012
13736b9
InteractiveShell.restore_sys_module_state actually restores main
minrk May 15, 2012
ea917e3
test re-entrant embed_kernel
minrk May 15, 2012
0d0437b
undo initial clobbering done by init_sys_modules in embed_kernel
minrk May 15, 2012
176eff9
add @interactive decorator to IPython.parallel namespace
minrk May 15, 2012
05f0cd8
add len(view)
minrk May 15, 2012
cf52d06
add topic to display publisher, and fix set_parent for apply_requests
minrk May 15, 2012
1805e53
test new execute/output behaviors
minrk May 15, 2012
efa86d9
clearer error when trying to use closed Client
minrk May 22, 2012
04351cd
Don't use asserts when checking for connection files
minrk May 22, 2012
4cb7712
fix date objects in _reserialize_reply
minrk May 22, 2012
4afa366
disable save-thread when using in-memory history db.
minrk May 22, 2012
ea84367
fix run_cell signature in parallelmagic
minrk May 22, 2012
e4f5c29
expose kernel at app-level, so EngineApp looks like KernelApp
minrk May 22, 2012
e9a0b8e
check for any Application with Kernel in zmq.eventloop
minrk May 22, 2012
7576fb6
test %pylab on engines
minrk May 22, 2012
85e05e1
demote enter/exit eventloop messages to INFO
minrk May 22, 2012
d2f71ea
move default log setup to _log_default from init_logging
minrk May 22, 2012
9fc6597
Application.log_format is a configurable
minrk May 22, 2012
f24fe97
split init_sockets into a few steps, so they can be better reused
minrk May 22, 2012
3b1700a
reduce parentless IOPub message to warning level
minrk May 22, 2012
23d0ff7
add listen_kernel method to IPEngineApp
minrk May 22, 2012
83b4b12
allow embed_kernel to be called from engine
minrk May 22, 2012
3fab994
kernel app cleans up connection files that it wrote
minrk May 22, 2012
100a5e3
relax some timing constraints in parallel tests
minrk May 22, 2012
46db02d
store whole content, instead of just display data in metadata.outputs
minrk May 23, 2012
e0b0d5e
revert embed_kernel changes that implied bind_kernel in engine
minrk May 23, 2012
64f2f8d
add IPython.parallel.bind_kernel
minrk May 23, 2012
45a4d04
use [sys.exe, "-c", "…launch_new_instance()"] in launchers
minrk May 23, 2012
87af741
switch order of failed-to-start errors in test_embed_kernel
minrk May 25, 2012
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
store whole content, instead of just display data in metadata.outputs
  • Loading branch information
@minrk
minrk committed May 23, 2012
commit 46db02de8172b63a623a8822da896285133d2083
34 changes: 17 additions & 17 deletions IPython/parallel/client/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,39 +91,39 @@ def __getattr__(self, key):

@ellisonbg ellisonbg May 21, 2012

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you are processing the pyout message type, but what about the other message types associated with output (stream, display_data). Shouldn't we also forward those in some way? I suppose we could take those other message types and put them into the _repr_* method output as well. The resulting ExecuteReply object would end up flattening these different message types, but they would still be there.

@minrk minrk May 22, 2012

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the _repr_* mechanism assumes singletons, we cannot do this. What is the _repr_png_ of a dozen PNGs?

@minrk minrk May 22, 2012

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I have been quite frustrated on numerous occasions by this general shortcoming of the display system. I think we should revisit the whole _repr_foo machinery and consider allowing it to return containers of a given mimetype, rather than a single entry. Once that is done, then it might make sense to include display_data outputs in these reprs.

@minrk minrk May 22, 2012

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option would be to special case repr_json to include everything - outputs, pyout/err, stdout/err, execution_count, etc.

@ellisonbg ellisonbg May 22, 2012

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of allowing the _repr_foo methods to return a sequence of things to display. There really isn't any reason to not do that. And we should be able to support that without breaking existing code because all of the existinng methods should return strings.

def __repr__(self):
pyout = self.metadata['pyout'] or {}
text_out = pyout.get('text/plain', '')
text_out = pyout.get('data', {}).get('text/plain', '')
if len(text_out) > 32:
text_out = text_out[:29] + '...'

return "<ExecuteReply[%i]: %s>" % (self.execution_count, text_out)

def _repr_html_(self):
pyout = self.metadata['pyout'] or {}
return pyout.get("text/html")
pyout = self.metadata['pyout'] or {'data':{}}
return pyout['data'].get("text/html")

def _repr_latex_(self):
pyout = self.metadata['pyout'] or {}
return pyout.get("text/latex")
pyout = self.metadata['pyout'] or {'data':{}}
return pyout['data'].get("text/latex")

def _repr_json_(self):
pyout = self.metadata['pyout'] or {}
return pyout.get("application/json")
pyout = self.metadata['pyout'] or {'data':{}}
return pyout['data'].get("application/json")

def _repr_javascript_(self):
pyout = self.metadata['pyout'] or {}
return pyout.get("application/javascript")
pyout = self.metadata['pyout'] or {'data':{}}
return pyout['data'].get("application/javascript")

def _repr_png_(self):
pyout = self.metadata['pyout'] or {}
return pyout.get("image/png")
pyout = self.metadata['pyout'] or {'data':{}}
return pyout['data'].get("image/png")

def _repr_jpeg_(self):
pyout = self.metadata['pyout'] or {}
return pyout.get("image/jpeg")
pyout = self.metadata['pyout'] or {'data':{}}
return pyout['data'].get("image/jpeg")

def _repr_svg_(self):
pyout = self.metadata['pyout'] or {}
return pyout.get("image/svg+xml")
pyout = self.metadata['pyout'] or {'data':{}}
return pyout['data'].get("image/svg+xml")


class Metadata(dict):
Expand Down Expand Up @@ -834,9 +834,9 @@ def _flush_iopub(self, sock):
elif msg_type == 'pyin':
md.update({'pyin' : content['code']})
elif msg_type == 'display_data':
md['outputs'].append(content.get('data'))
md['outputs'].append(content)
elif msg_type == 'pyout':
md['pyout'] = content.get('data')
md['pyout'] = content
else:
# unhandled msg_type (status, etc.)
pass
Expand Down
25 changes: 15 additions & 10 deletions IPython/parallel/tests/test_view.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -593,7 +593,7 @@ def test_execute_reply(self):
er = ar.get()
self._wait_for(lambda : bool(er.pyout))
self.assertEquals(str(er), "<ExecuteReply[%i]: 5>" % er.execution_count)
self.assertEquals(er.pyout['text/plain'], '5')
self.assertEquals(er.pyout['data']['text/plain'], '5')

def test_execute_reply_stdout(self):
e0 = self.client[self.client.ids[0]]
Expand All @@ -610,7 +610,8 @@ def test_execute_pyout(self):
self._wait_for(lambda : all(ar.pyout))

expected = [{'text/plain' : '5'}] * len(view)
self.assertEquals(ar.pyout, expected)
mimes = [ out['data'] for out in ar.pyout ]
self.assertEquals(mimes, expected)

def test_execute_silent(self):
"""execute does not trigger pyout with silent=True"""
Expand Down Expand Up @@ -645,9 +646,10 @@ def test_execute_displaypub(self):
ar = view.execute("[ display(i) for i in range(5) ]", block=True)

self._wait_for(lambda : all(len(er.outputs) >= 5 for er in ar))
outs = [ {u'text/plain' : unicode(i)} for i in range(5) ]
expected = [outs] * len(view)
self.assertEquals(ar.outputs, expected)
expected = [ {u'text/plain' : unicode(j)} for j in range(5) ]
for outputs in ar.outputs:
mimes = [ out['data'] for out in outputs ]
self.assertEquals(mimes, expected)

def test_apply_displaypub(self):
"""apply tracks display_pub output"""
Expand All @@ -661,9 +663,10 @@ def publish():
ar = view.apply_async(publish)
ar.get(5)
self._wait_for(lambda : all(len(out) >= 5 for out in ar.outputs))
outs = [ {u'text/plain' : unicode(j)} for j in range(5) ]
expected = [outs] * len(view)
self.assertEquals(ar.outputs, expected)
expected = [ {u'text/plain' : unicode(j)} for j in range(5) ]
for outputs in ar.outputs:
mimes = [ out['data'] for out in outputs ]
self.assertEquals(mimes, expected)

def test_execute_raises(self):
"""exceptions in execute requests raise appropriately"""
Expand All @@ -672,7 +675,7 @@ def test_execute_raises(self):
self.assertRaisesRemote(ZeroDivisionError, ar.get, 2)

@dec.skipif_not_matplotlib
def test_amagic_pylab(self):
def test_magic_pylab(self):
"""%pylab works on engines"""
view = self.client[-1]
ar = view.execute("%pylab inline")
Expand All @@ -684,6 +687,8 @@ def test_amagic_pylab(self):
self._wait_for(lambda : all(ar.outputs))
self.assertEquals(len(reply.outputs), 1)
output = reply.outputs[0]
self.assertTrue("image/png" in output)
self.assertTrue("data" in output)
data = output['data']
self.assertTrue("image/png" in data)