- Description of the Py4J plugin for Warp 10
- Configuration parameters
- Short example with WarpScript
- Usage in Jupyter notebooks
- Read more
Python and Jupyter
You can access a WarpScript execution environment (a stack) on a Warp 10 platform directly from Python.
The principled way to do that is via a gateway implemented by the Py4J library (the same that is used by PySpark for example).
To transfer data between WarpScript and Python, Py4J makes automatic conversions for common types. For larger objects, we recommend using Python's pickle representation that is also supported by WarpScript (see ->PICKLE and PICKLE->).
Description of the Py4J plugin for Warp 10
The Py4J plugin launches a gateway in same JVM running Warp 10. Instances of Java objects are accessible from Python through this gateway. By default, the gateway address is 127.0.0.1 and its port is 25333.
Using the Py4J plugin has multiple advantages:
- you don't need to use POST requests to execute your code
- you can access WarpScript objects from Python without them being serialized in JSON format
- you can go back and forth between Python and the WarpScript stack that is in the JVM
The Py4j plugin depends on these parameters (default value is given if parameter is not set):
# Address and port of the gateway py4j.host = '127.0.0.1' py4j.port = 25333 # Address and port used for Python callbacks py4j.python.host = '127.0.0.1' py4j.python.port = 25334 # Timeout specifications py4j.timeout.read = 0 py4j.timeout.connect = 0 # Whether to use imposed limits on the stack or not py4j.stack.nolimits = false # If set, GatewayParameters must set the argument auth_token to this value to authorize the connection #py4j.authtoken = # Allowing to fetch data from Python (or other external access method) with FETCH, FIND or FINDSTATS. egress.clients.expose = false
Short example with WarpScript
Once the plugin is installed, restart Warp 10. Then, you can set a connection with the gateway from a Python interpreter:
# pip install py4j from py4j.java_gateway import (JavaGateway, GatewayParameters) # if you need to launch a gateway manually, you can use py4j.launch_gateway() function # Connect to the gateway params = GatewayParameters('127.0.0.1', 25333, auto_convert=True) gateway = JavaGateway(gateway_parameters=params)
Then, you can create a WarpScript stack and start querying the Warp 10 platform.
# Instanciates a WarpScript stack stack = gateway.entry_point.newStack() stack = stack.execMulti('<My_WarpScript_code_here>') # Extract top of the stack and store it in a Python variable # @see https://www.py4j.org/advanced_topics.html#collections-conversion my_var = stack.pop()
The following methods are useful:
- execMulti(str): executes a multiline WarpScript code on the stack
- pop(): extract the top of the stack
- get(int): get the object at a certain level of the stack
- clear(): reset the stack
- load(str): get the object that was stored under a certain variable in WarpScript
- getStackAttribute(key): get an attribute
- setStackAttribute(key,value): set an attribute
Note that only basic conversions are supported automatically (primitive types, lists, sets, maps, iterators, byte arrays). Therefore, data of GTS need to be stored beforehand in a list, a map, or byte array using WarpScript functions. For that purpose, the functions VALUES, TICKLIST and ->PICKLE (since pickled representation are byte arrays) come in handy.
Usage in Jupyter notebooks
The python package
warp10-jupyter contains the cell magic called
%%warscript that executes WarpScript code through the Py4J plugin. You can install it with
pip install warp10-jupyter, or from the source code available here.
For example, you can load it in a Jupyter notebook with
Then you can start executing WarpScript code in following cells:
%%warpscript --stack s <My_WarpScript_code_here>
The newly created WarpScript stack is stored under the Python variable
s. Therefore its objects can be accessed with the aforementioned methods.
After the cell has been executed, the stack is still in the JVM so subsequent cells starting with
%%warpscript --stack s will execute WarpScript code on the same stack.
Recall that if needed, the docstring of this cell magic can be invoked within a notebook with
We wrote a blog post on the Py4J plugin, where we also explained how to use WarpScript in Python without a running Warp 10 platform.
In another blog post on the warp10-jupyter package, we show examples of notebooks.
In a blog post on WarpScript for Pythonists, we explain why and when Python users benefit from using WarpScript.