What is Spark?

Spark is a popular data analytics engine offering high-level operators which make it easy to build parallel applications.

Is Spark suited for analyzing Time Series data?

Given its popularity, Spark is chosen by many data scientists as the framework of choice to manipulate and analyze their data, regardless of their type. Unfortunately the operators that Spark provides remain high-level and are not tailored to specific data types such as graphs, images, or time series.

This lack of specialized functions means that rather often Spark users have to craft User Defined Functions to manipulate their data of interest. This leads to a lack of focus on the initial business goals and an important waste of time and resources to build the required low level tooling.

For these reasons, Spark out of the box is not suited for analyzing Time Series data.

Those are also the reasons why we integrated WarpScript into Spark so you can benefit from the rich and specialized set of functions while remaining within the popular Spark framework.

What does WarpScript bring to Spark?

The integration of WarpScript within Spark allows to express data transformations in WarpScript and apply those transformations on Spark datasets in either Java, Scala or Python in SparkSQL.

The immediate benefit is that the WarpScript corpus you have already written is fully usable from within Spark, thus leading to reduced development times and increased focus on your business objectives.

Learn more working with Time Series on Spark on the blog.