Kotlin script running experiment

In this blog post I'll go over building a prototype of Java application that can load and run Kotlin scripts at runtime. Extra requirement is to support scripts written in various mutually incompatible Kotlin language versions. This experiment is an isolated, self-contained project, but it does explore a real-world use-case that I've encountered.

Use-case

Kotlin has language features that make it well suited for scripting and using DSLs. As such it is a good candidate for writing more involved configuration files. A good example of this is Gradle's Kotlin DSL. The configuration file use-case can be generalized into a plugin system. In this case the base application can be extended at runtime with any number of Kotlin scripts.

Such plug-able application can be paired with an HTTP server. In this case the server can execute a plugin script in response to an HTTP request, and use its output as an HTTP response. At this point the application is begging to resemble a simplified Function as a service system. It can read plugin scripts from a public source (e.g. a git repository) and have users submit their own scripts.

With time, as the service grows in popularity, users submit many different scripts. On the other hand development of Kotlin progresses and there are new language version releases. Script developers, enticed by new language features, start requesting support for newer Kotlin versions.

Unfortunately, some of the code in existing scripts is using experimental Kotlin 1.1 features that have suffered breaking changes in subsequent 1.2 and 1.3 releases. This makes upgrading the Kotlin version in the base application problematic. Migrating to 1.3 would make enthusiastic script developers happy, but would also break some existing scripts. On the other hand, scripts cannot be upgraded to newer Kotlin versions and still run on the existing 1.1 version used by the server application. This chicken and the egg problem causes a stand still and Kotlin version is locked to 1.1.

The solution

In this experiment I've built a simplified plugin engine that can be used to load and execute Kotlin scripts with various language versions. I've skipped fetching scripts form public git repositories, running HTTP server and matching requests to scripts. The solution is lacking in security and performance as well.

What it does achieve is illustrate a solution of the stated use-case in an isolated, self contained project. Complete source code can be found on GitHub.

Running scripts

My first step was to build Java app that can load and run a Kotlin script at runtime. Java itself provides a scripting API (defined by JSR 223) for abstracting the mechanics of running scripts in compatible languages. Form its version 1.1 Kotlin provides an implementation of this Java API.

API documentation suggests registering scripting engine implementations in a special file inside META-INF directory and using dedicated manager to obtain engine instances. However, since I was planning on doing some jar juggling later on, I decided to instantiate Kotlin script engine programmatically and keep all of it in my source code.

At this stage the application was a straightforward maven project with few lines of Java code.

<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>io.github.antolius</groupId>
    <artifactId>kotlin-engine-experiment</artifactId>
    <version>0.1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <properties>
        <java.version>8</java.version>
        <kotlin.version>1.1.61</kotlin.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-script-util</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-script-runtime</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-compiler-embeddable</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-reflect</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

public class Main {
    public static void main(String[] args) throws Exception {
        String kotlinScript = "\"Using Kotlin version: "
            + "${KotlinVersion.CURRENT}\"";
        ScriptEngine engine = new KotlinJsr223JvmLocalScriptEngineFactory()
            .getScriptEngine();
        Object result = engine.eval(kotlinScript);
        System.out.println(result); // Using Kotlin version: 1.1.60
    }
}

ScriptEngine provides a method for evaluating source code from a java.io.Reader, so executing a script from a file was easy too.

Multi module

Now that the app could load and run Kotlin scripts it was time to support multiple language versions. The maven project described in the previous section declares a compile-time dependency on a specific Kotlin version. Since I needed to support multiple versions I decided to load Kotlin dependencies into separate class loaders at runtime. That way I could work with different versions of Kotlin libraries at the same time.

Next step was to turn the application into a multi module maven project. That way each module could declare a dependency on a distinct Kotlin version. I created 4 modules:

Each of the three Kotlin modules included a single class:

package io.github.antolius.engine.kotlin1;
// imports...
 
public class ScriptEngineSupplier implements Supplier<ScriptEngine> {
    @Override
    public ScriptEngine get() {
        return new KotlinJsr223JvmLocalScriptEngineFactory().getScriptEngine();
    }
}

The only difference was the package name. Module with 1.1 Kotlin dependency had package kotlin1, module with 1.2 package kotlin2 etc.

The engine module created a dedicated class loader for each language version and loaded individual Kotlin module jar in runtime. It instantiated the Supplier from that jar:

public class ScriptEngineFactory {
    public ScriptEngine newEngine(
        URL kotlinModuleJar,
        String fullyQuelifiedSupplierClassName
    ) {
        ClassLoader classLoader = newClassLoaderWith(kotlinModuleJar);
        Supplier<ScriptEngine> supplier = instantiateSupplierFrom(
            fullyQuelifiedSupplierClassName,
            classLoader
        );
        return supplier.get();
    }

    private ClassLoader newClassLoaderWith(URL kotlinModuleJar) {
        try {
            return URLClassLoader.newInstance(
                new URL[] {kotlinModuleJar},
                getClass().getClassLoader()
            );
        } catch (Exception e) {
            throw new RuntimeException("Couldn't create class loader", e);
        }
    }

    private Supplier<ScriptEngine> instantiateSupplierFrom(
        String className,
        ClassLoader classLoader
    ) {
        try {
            Class<?> loadedClass = Class
                .forName(className, true, classLoader);
            Class<Supplier<ScriptEngine>> supplierClass = cast(loadedClass);
            Constructor<Supplier<ScriptEngine>> constructor = supplierClass
                .getConstructor();
            return constructor.newInstance();
        } catch (Exception e) {
            throw new RuntimeException("Couldn't load " + className, e);
        }
    }

    @SuppressWarnings("unchecked")
    private Class<Supplier<ScriptEngine>> cast(Class<?> loadedClass) {
        return (Class<Supplier<ScriptEngine>>) loadedClass;
    }
}

This ScriptEngineFactory could be used to obtain an instance of JSR 223 script engine capable of running Kotlin scripts with different versions. The problem was in knowing where to find those Kotlin module jars.

The application technically worked, however it was difficult to build manually. At this point I had to compile each Kotlin module into a fat jar that contains both the Supplier implementation and its Kotlin dependencies. Then I had to somehow provide paths to those fat jars to the engine application at runtime.

Maven tricks

Building jars

First thing that maven could help with was building fat jars for the three Kotlin modules. For this I used maven assembly plugin. Specifically I configured the plugin for Kotlin 1 module with:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-assembly-plugin</artifactId>
    <version>3.1.1</version>
    <configuration>
        <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
        <finalName>kotlin-1-module</finalName>
        <appendAssemblyId>false</appendAssemblyId>
    </configuration>
    <executions>
        <execution>
            <id>assemble-all</id>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
        </execution>
    </executions>
</plugin>

I defined the final jar's name as kotlin-1-moule. This is important, since by default maven would have included the project version in the name and that would have complicated things for me later on. Config for other modules was nearly identical, just with different jar names.

Packaging jars

The next problem was finding those jars from within the engine module's code. I solved this by packaging fat jars as resources of the engine module. For this I used maven resources plugin configured in engine module's pom with:

<plugin>
    <artifactId>maven-resources-plugin</artifactId>
    <version>3.1.0</version>
    <executions>
        <execution>
            <id>copy-resources</id>
            <phase>validate</phase>
            <goals>
                <goal>copy-resources</goal>
            </goals>
            <configuration>
                <outputDirectory>
                    ${project.build.outputDirectory}/kotlin-jars
                </outputDirectory>
                <resources>
                    <resource>
                        <directory>
                            ${rootdir}/kotlin-1-module/target
                        </directory>
                        <filtering>false</filtering>
                        <includes>
                            <include>kotlin-1-module.jar</include>
                        </includes>
                    </resource>
                    <resource>
                        <directory>
                            ${rootdir}/kotlin-2-module/target
                        </directory>
                        <filtering>false</filtering>
                        <includes>
                            <include>kotlin-2-module.jar</include>
                        </includes>
                    </resource>
                    <resource>
                        <directory>
                            ${rootdir}/kotlin-3-module/target
                        </directory>
                        <filtering>false</filtering>
                        <includes>
                            <include>kotlin-3-module.jar</include>
                        </includes>
                    </resource>
                </resources>
            </configuration>
        </execution>
    </executions>
</plugin>

where ${rootdir} was defined as:

<properties>
    <rootdir>${project.parent.basedir}</rootdir>
</properties>

For this to work properly I needed maven to build Kotlin modules first. This would generate the fat jars in their individual target directories. Then maven could build the engine module, and include fat jars as resources. Since engine module didn't declare direct dependencies on Kotlin modules, maven built modules in the order they appeared in the parent's pom:

<modules>
    <module>kotlin-1-module</module>
    <module>kotlin-2-module</module>
    <module>kotlin-3-module</module>
    <module>engine</module>
</modules>

Cleanup

With this setup engine module was packaged with all of its runtime dependencies. Since individual jars didn't include project version in their names, the jar resource names were constant. This allowed me to define the jar URL and the fully qualified name of the Supplier implementation in an enum:

public enum Language {

    KOTLIN_1_1(
            "io.github.antolius.engine.kotlin1.ScriptEngineSupplier",
            "/kotlin-jars/kotlin-1-module.jar"
    ),
    KOTLIN_1_2(
            "io.github.antolius.engine.kotlin2.ScriptEngineSupplier",
            "/kotlin-jars/kotlin-2-module.jar"
    ),
    KOTLIN_1_3(
            "io.github.antolius.engine.kotlin3.ScriptEngineSupplier",
            "/kotlin-jars/kotlin-3-module.jar"
    );

    private final String supplierClass;
    private final String resourceName;

    Language(String supplierClass, String resourceName) {
        this.supplierClass = supplierClass;
        this.resourceName = resourceName;
    }

    public String getSupplierClass() {
        return supplierClass;
    }

    public URL getJarURL() {
        return this.getClass().getResource(resourceName);
    }
}

The ScriptEngineFactory#newEngine method could now be simplified into:

public class ScriptEngineFactory {
    public ScriptEngine newEngine(Language version) {
        // rest of the code is more-less the same...
    }
}

Plugin API

At this point maven could build the project automatically, and the ScriptEngineFactory could be used to obtain JSR 223 script engine for any Kotlin version. The last usability problem left was defining an interface between scripts and the engine.

For this purpose I introduced a dedicated API maven module, and added it as a dependency for the rest of the modules. Within it I defined the Plugin interface as:

public interface Plugin {
    @NotNull
    Response process(@NotNull Request req);
}

Request and Response were just POJOs encapsulating all input and output parameters of the process method. All that Kotlin scripts need to do in order to be compatible with the engine is return an instance of a Plugin.

I also wanted to demonstrate how dependencies might be injected into scripts from Java code, so I defined two more interfaces:

public interface Printer {
    void print(@NotNull String line);
}

and

public interface PrinterAware {
    void set(@NotNull Printer printer);
}

Java code should implement the Printer, and scripts interested in using it should implement PrinterAware interface. To pull it all together I created a PluginLoader class in the engine module:

public class PluginLoader {
    private final ScriptEngineFactory factory;
    private final Printer printer;

    public PluginLoader(ScriptEngineFactory factory, Printer printer) {
        this.factory = factory;
        this.printer = printer;
    }

    public Plugin load(File sourceFile, Language kotlinVersion) {
        Reader sourceFileReader = readerFrom(sourceFile);
        ScriptEngine engine = factory.newEngine(kotlinVersion);
        Plugin plugin = runScript(sourceFileReader, engine);
        autowirePrinter(plugin);
        return plugin;
    }

    private Reader readerFrom(File source) {
        try {
            FileInputStream stream = new FileInputStream(source);
            return new InputStreamReader(stream, StandardCharsets.UTF_8);
        } catch (FileNotFoundException e) {
            String message = "Couldn't create a reader for " + source;
            throw new RuntimeException(message, e);
        }
    }

    private Plugin runScript(Reader sourceReader, ScriptEngine engine) {
        try {
            Object result = engine.eval(sourceReader);
            return (Plugin) result;
        } catch (Exception e) {
            String message = "Couldn't evaluate script to a Plugin instance";
            throw new RuntimeException(message, e);
        }
    }

    private void autowirePrinter(Plugin plugin) {
        if (PrinterAware.class.isAssignableFrom(plugin.getClass())) {
            PrinterAware aware = (PrinterAware) plugin;
            aware.set(printer);
        }
    }
}

PluginLoader could be used to load Kotlin scripts from files and provide the rest of the application with implementations of the Plugin interface.

In conclusion

The final Java application uses a few tricks to enable execution of mutually incompatible Kotlin scripts:

  1. Kotlin's implementation of JSR 223 script engine is used to evaluate scripts.
  2. Script engines themselves are loaded into separated class loaders at runtme.
  3. Various maven plugins are used to package all the different Kotlin dependencies as resources at build time.
  4. Simple interfaces define a contract between Kotlin scripts and the rest of the application.

As mentioned before, the complete project can be found on GitHub. In addition to the code discussed in this post, the full project contains a few extra bits:

Building this prototype was a fun exercise for me, but now I'd be interested in hearing from you as well. Do you see this kind of dynamic plugin system fitting into some of your projects? Have you encountered similar use-cases before, and if so, how did you solve them? Let me know: