Skip to content

[TOREE-526] Enable jupyter_kernel_test - #202

Draft
pan3793 wants to merge 3 commits into
apache:masterfrom
pan3793:TOREE-526
Draft

pan3793 wants to merge 3 commits into
apache:masterfrom
pan3793:TOREE-526

Conversation

@pan3793

@pan3793 pan3793 commented Aug 5, 2023

Copy link
Copy Markdown
Member

jupyter_kernel_test seems revived, try first.

Some conclusions:

Currently, Toree implementation is based on Jupyter 5.0 protocol, and the latest version is 5.4.

The jupyter_kernel_test seems to become stricter now. e.g.

the docs say

All reply messages have a 'status' field ...

But Toree's implementation does not satisfy such restriction. e.g. the KernelInfoReply does not contain "status", so that it will be failed with

======================================================================
ERROR: test_kernel_info (__main__.ToreeScalaKernelTests.test_kernel_info)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/jupyter_kernel_test/__init__.py", line 61, in test_kernel_info
    validate_message(reply, "kernel_info_reply", msg_id)
  File "/opt/conda/lib/python3.11/site-packages/jupyter_kernel_test/msgspec_v5.py", line 149, in validate_message
    raise ValidationError(str(e)) from None
jsonschema.exceptions.ValidationError: 'status'

So we need to revise the protocol definition and our implementation before enabling the system-test.

@pan3793

pan3793 commented Aug 8, 2023

Copy link
Copy Markdown
Member Author

#207 is opened to address "status" issue.

etc/bin/run.sh resolves the kernel assembly by name from the lib
directory next to it, and the jar is now named
apache-toree-assembly-<version>.jar, so the lookup matched nothing and
spark-submit failed with "Failed to load class org.apache.toree.Main".
Toree declared protocol 5.0 in message headers and in kernel_info_reply,
so jupyter_kernel_test applies its legacy envelope checks to every
message it receives and rejects the msg_id/msg_type keys that
jupyter_client 7+ adds when deserializing messages. 5.2 is the lowest
revision not subject to those checks, which lets the kernel info,
completion, is_complete and execution tests reach the reply contents.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant