Replies: 1 comment 1 reply
-
|
@sergiimk Thanks for creating an issue. I hadn't seen Spark Connect before, looks interesting! Livy alternatives are much needed. There aren't plans to support it, but at a glance, it seems very feasible. Would you be open to contributing? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
@devstein I would be interested. Any open discussions for reference? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I've been a user of Sparkmagic <-> Livy combo for a long time and while upgrading spark I stumbled upon Spark Connect.
Since Livy is essentially unmaintained, I was wondering if there are plans to support this interface instead.
It seems to be based on Apache Arrow, which should provide an top-notch performance in combination with Pandas.
Our company is also heavily invested in DataFusion and Arrow FlightSQL (SQL API over Arrow) so I would really like to get to a state where Sparkmagic works with both Spark and our DataFusion+FlightSQL engines.
Beta Was this translation helpful? Give feedback.
All reactions