How others fail

I'll start by exploring languages I've used in the past and how their interactive shells have failed to be as transformative as Clojure's REPL.

JavaScript

JavaScript may be the most widely used language which touts instant feedback. I can open the console in my browser and start playing around with the language. For my server-side needs I have Node.js which comes with a REPL. While all this is undeniably true, I've never felt it integral to my JS coding experience. I think there are a couple of reasons for this:

1. Web apps these days are far removed from basic console interactions. We use build tools such as Webpack to bundle and transpile our code, and frameworks like React to render the app. Not to mention using language features not yet supported by the browser, or coding in TypeScript.
2. Even if I try to write a snippet of code in my console, I'm fighting against the language's imperative syntax. As soon as my snippet surpasses a few lines of code it's easier to put it in a script and runt it from there. JavaScript shares this with languages from the C family.
3. Web apps are heavily intertwined with DOM, or some equivalent internal state. This makes poking at my code from browser console more difficult as it is often tied to DOM events or depends on a complex state.

This is not to say that JavaScript coding experience is objectively bad, just that I prefer to interact with my web apps through their UI instead of interactive console.

Java

Back with Java 9 in 2017 Oracle introduced JShell:

The Java Shell tool (JShell) is an interactive tool for learning the Java programming language and prototyping Java code. JShell is a Read-Evaluate-Print Loop (REPL) ...

Oracle docs

Is it cool that Java has an official REPL? Yeah, I guess. Has it in any way changed my approach to writing Java code? No. It's notable that I felt the need to link to the docs, as if to prove this thing actually exists. I use Java daily and most of the time I simply forget about JShell.

The greatest impediment to good REPL experience in Java is its object orientation along with all design patterns used with that paradigm. For example, I rely on Spring's dependency injection to bootstrap my Java apps. In unit tests I need Mockito library to help me mock all dependencies of a class before I can start working with it. This makes it near impossible to interact with my code from REPL. Imperative code structure and verbosity don't help either.

Go

Other languages I've used in the past usually have some combination of these flaws. For example Go has a fast compiler. In fact people go as far as using it as “scripting language” in place of shell scripting. However, the speed of compiler itself is not enough. Being highly imperative Go would make for an unpractical REPL experience.

How Clojure succeeds

Previous examples illustrate a few key points. For a good experience it's not enough that a REPL, or a similar interactive shell, simply exists. It's also not a necessity for the language to be interpreted (Clojure is not, mind you). It might be a good idea to have a relatively fast compiler, but that in itself is not enough.

So, why is Clojure REPL so transformative? Here are a few language characteristics that I found critical to my usage:

1. Expression based syntax
2. Functional paradigm
3. Dynamically typed
4. Simple at scale

Expression based syntax

This is where imperative languages fall short. In Clojure the code is structured as a series of expressions. This lends itself extremely well to REPL interaction, which is essentially a loop of reading and evaluating expressions. My “normal” Clojure code, which I store in files, doesn't differ too much from a series of expressions I input into a REPL session. This makes interacting with REPL feel as natural as coding. There's no urge so quit REPL and just dump it all in a script file.

Functional paradigm

Being functional helps the language be expression based. Beyond that it favors composition. In Clojure complex behaviors are implemented by combining simpler functions. This is in contrast with Object Oriented languages where primary tool is inheritance. With OO complex features are implemented by building type hierarchies, which breaks the REPL experience. If I want to poke around at different segments of a composed function I invoke its component functions. With type hierarchy on the other hand I'd probably need to implement abstract classes to try to isolate and invoke some of their methods.

In addition to composition, another benefit is working with mostly pure functions. The fact that things I'm playing with in REPL don't have side-effects is hugely helpful. REPL shines in exploratory interactions and being able to repeat function calls without messing up some internal state is critical.

Dynamically typed

One benefit of dynamic typing is capability of functions to care less about their parameters. As long as parameters contain particular data that function needs it doesn't matter what else is in there. In Clojure there might be some state carried over between functions, but they themselves don't care about entirety of it. All that's needed to invoke a function in REPL is it's own minimal set of data.

Of course, extra benefit is not having to write a class for every single piece of data.

Simple at scale

In some of the other languages REPL starts promising, but there's a breaking point which production applications cross and it becomes useless. Think of Java hello world example vs. a minimum Spring Boot app. Amount of code might not be all that different, but effectiveness of JShell tanks.

I lack personal experience to speak on this point. As I mentioned I'm just starting with Clojure, and I haven't worked on any production Clojure apps. However, simplicity is famously one of the core objectives of Clojure language. There's plenty of testimony from experience Clojurians to attest to this. So I'm optimistic about my REPL being able to keep up with me as my programs grow in scale.

The end

In the end it took a particular language built around a set of conscious design decisions to ignite my love for REPL. In hindsight it seems obvious that design of a language would influence programming techniques that work well with it. However, that gets increasingly overlooked in current programming landscape where more and more languages strive for feature parity.

If you'd like to get in touch with me, either to share in my enthusiasm for Clojure, or to tell me that I should have learned Python and saved myself all this overthinking, you can find me on Mastodon @antolius.

If you liked the post and are interested in more similar content you can subscribe to this blog's RSS feed or follow @antolius@qua.name in fediverse.

Use-case

At work I've been running a Spring Cloud Gateway app in production for over a year and a half. By now we are exposing most of our 500+ API endpoints through it. With so many endpoints we encountered some tricky situations. One of them is overlapping path definitions. Here's an example:

Imagine a Foo product implemented by a foo-service application. Since all foo related endpoints are handled by the same app all it needs is a single Cloud Gateway route:

builder.routes()
  .route(r -> r.path("/foo/**").uri(fooServiceUri))
  .build();

Spring Cloud Gateway uses Ant-style matching here, so ** matches zero or more path segments.

As time goes by a new version of the product is designed. new-foo-serice app is built to implement it. The new app exposes /foo/2/** endpoints. Finally, in order to enable independent scaling, /foo/2/bars endpoint is extracted into a separate bars-service app.

In this case a request to GET /foo/2/bars matches all 3 routes.

The problem

This by itself is not a problem, as Cloud Gateway has a mechanism for handling such situations. The Route class implements the Ordered interface. Gateway picks a route with the highest precedence to handle an incoming request. This seems flexible and is easy to configure using the route builder. However, it does not solve my problem.

In my production application I don't know all the routes in advance. They are discovered and updated at runtime, one downstream app at a time. This means that my gateway loads routes in batches and only updates the ones that have actually changed. This is a problem, because order of each individual route needs to be defined at load time and depends on all the other routes.

Imagine routes loading happens like this:

1. foo-service routes load first, and /foo/** route is given order 0.
2. bars-service loads next. Since its path /foo/2/bars is more exact its route is given order -1, which has higher precedence.
3. new-foo-service loads last. At this point there's no order that could be given to its /foo/2/** route that would correctly position it between the two previously loaded routes.

In this case a “solution” might be to load downstream apps in a particular order. However, it's easy to construct a counter example in which an app could have both more exact route in one hierarchy and a more general one in another. This makes the app ordering solution insufficient.

How Cloud Gateway works

Cloud Gateway itself is built on top of Spring WebFlux. As such, there are two key classes that implement handing of all incoming HTTP requests:

• RoutePredicateHandlerMapping which implements WebFlux's HandlerMapping. This is the class that assigns a route to an incoming request.
• FilteringWebHandler which implements WebHandler by delegating request processing to the chain of Cloud Gateway filters defined by the route.

RoutePredicateHandlerMapping is of interest here. More specifically, its lookupRoute method. Here's what the method looks like with logging, error and empty state handling removed:

protected Mono<Route> lookupRoute(ServerWebExchange exchange) {
    // start with an ordered list of all registered routes:
    return this.routeLocator.getRoutes()
        .concatMap(route -> Mono.just(route)
                // check if a route matches the request:
                .filterWhen(r -> r.getPredicate().apply(exchange))
        )
        // take the first route that matches:
        .next()
        .map(route -> {
            // by default all routes are valid
            validateRoute(route, exchange);
            return route;
        });
}

The trick here is in the .next() operator. Because of it Cloud Gateway always picks the first route that matches the request. Also of note is that the lookupRoute method is protected!

Solution

This allowed me to implement my own handler mapping class by extending the RoutePredicateHandlerMapping and overriding the lookupRoute method. The change was simple: I sorted the matching routes so that “more exact” ones come first. The rest of the implementation is the same:

protected Mono<Route> lookupRoute(ServerWebExchange exchange) {
    return this.routeLocator.getRoutes()
        .concatMap(/* filter routes */)
        // sort routes so that more exact ones come first:
        .sort(pathBasedRouteComparator)
        .next() // pick the first route, like before
        .map(/* validate picked route */);
}

What's left to determine is what precisely “more exact” route means. For routes R1 with path /foo/** and R2 with path /foo/2/** I defined R2 to be more exact because pattern /foo/** matches the literal path /foo/2/** , while the opposite is not true. This is how I implemented it in a comparator:

@Component
public class PathBasedRouteComparator implements Comparator<Route> {
    @Override
    public int compare(Route r1, Route r2) {
        var info1 = (PathInfo) r1.getMetadata().get(PathInfo.KEY);
        var info2 = (PathInfo) r2.getMetadata().get(PathInfo.KEY);
        if (info1 == null || info2 == null) {
            // fall back to native route ordering:
            return compareOrders(r1, r2);
        }
        var r1MatchesR2 = info1.getPattern().matches(info2.getPath());
        var r2MatchesR1 = info2.getPattern().matches(info1.getPath());
        if (r1MatchesR2 && !r2MatchesR1) {
            return 1;
        }
        if (r2MatchesR1 && !r1MatchesR2) {
            return -1;
        }
        // fall back to native route ordering:
        return compareOrders(r1, r2);
    }
    private int compareOrders(Route r1, Route r2) {
        return Integer.compare(r1.getOrder(), r2.getOrder());
    }
}

For this comparator to work routes need to have a PathInfo added to their metadata. This can be done at route loading time because, unlike the order, path info depends only on the individual route that's currently loading.

Alternative solution

This customization allowed Cloud Gateway application to handle any convoluted path overriding that API developers could came up with. It did, however, come with a runtime performance cost. With the new routing logic all loaded routs are matched against each incoming request. As expected, the lower latency percentiles were impacted by this. They represent situations in which request matched one of the high precedence routes, so a lot of matching was skipped in the original implementation.

For use-cases where routes are either known in advanced or are updated all at once I'd recommend using the original, order based, routing logic. In this case route order can be determined upfront at route loading time. The comparator logic can be reused to calculate the route order. This solution would have no impact on the runtime performance.

Example

I've created a demo project on GitHub that illustrates the solution using the custom handler mapping. Feel free to check out the project and play around with it. If you find a better solution, please let me know about it! You can reach me at:

• josip.antolis@protonmail.com
• @antolius on Mastodon

Derived classes must be substitutable for their base classes.

Liskov substitution principle

Considering each of the SOLID princpiles Liskov substitution principle (LSP) often gets least attention in both theoretical analysis and practical application. Single responsibility invites discussion with its definition of The One reason to change. As a result it has been extensively talked and blogged about. Open-closed principle is well covered by the object-oriented design patterns as many of them are essentially clever tricks for applying the principle. Similarly, there are many popular dependency injection frameworks that implement the dependency inversion. This empowers developers to more easily adhere to those two principles.

That leaves interface segregation and Liskov substitution principle. Of those two I find the former to be easier to abide by. In its definition interface segregation invites the code author to make their interfaces more concise. It's easier to follow because it relates directly to the activity that I, as a code author, am involved in; i.e. writing my interface.

In contrast, Liskov substitution asks me as author of the derived class to make sure it behaves same as the base class (written by someone else) from the perspective of the user code (written by yet another author). It's no just about me and my code, I need to consider two more perspectives. It requires a sort of developer empathy.

This isn't helped by examples that are often used to illustrate the violation of the principle. Classic example is the problem of modeling a square class as a derivative of a more generalized rectangle class. It's a valid example in theory, but is obviously contrived and as such fails to resonate. That's why I was excited when one of my real world bugs boiled down to a violation of Liskov substitution principle.

The bug

I've recently been learning Vala programming language and GTK framework. Vala is a higher-order language, with syntax similar to C# and Java, that compiles down to C. It promises to make targeting the GNOME stack simple. Vala is strongly typed and object-oriented, however this example does not rely on the language itself and can be reproduced in C code as well. GTK is a feature-packed UI toolkit with a rich hierarchy of widgets. It is in the thick of its widgets' inheritance tree that the root of my problem lies.

In my app I needed a container widget that could expand dynamically row by row as I added more child elements to it. Luckily GTK covers just such use-case with its FlowBox component. The FlowBox extends the abstract Container widget which defines the interface for adding and removing child widgets:

This was convenient as I needed to occasionally remove child widgets from the container. Here's a simplified example of what I wanted to achieve:

public class ExampleApp : Gtk.Application {
	protected override void activate () {
		var window = new Gtk.ApplicationWindow (this);
		window.set_default_size (300, 200);

		var hello_world = new Gtk.Label ("Hello, World!");

		var box = new Gtk.FlowBox ();
		box.add (hello_world);
		// removing the child immediately to simplify the example
		box.remove (hello_world);

		window.add (box);
		window.show_all ();
	}

	public static int main (string[] args) {
		var app = new ExampleApp ();
		return app.run (args);
	}
}

Try guessing what this program does:

1. It opens an empty window.
2. It opens a window with “Hello, World!” text.
3. It segfaults.

Spoiler alert: it's option 3:

$ valac --pkg gtk+-3.0 app.vala && ./app
(app:9912): GLib-CRITICAL **: 23:54:22.670: g_sequence_remove: assertion 'iter != NULL' failed
(app:9912): Gtk-CRITICAL **: 23:54:22.670: gtk_widget_show_all: assertion 'GTK_IS_WIDGET (widget)' failed
(app:9912): Gtk-CRITICAL **: 23:54:22.697: gtk_widget_get_visible: assertion 'GTK_IS_WIDGET (widget)' failed
(app:9912): Gtk-CRITICAL **: 23:54:22.701: gtk_widget_get_visible: assertion 'GTK_IS_WIDGET (widget)' failed
(app:9912): Gtk-CRITICAL **: 23:54:22.701: gtk_widget_get_visible: assertion 'GTK_IS_WIDGET (widget)' failed
(app:9912): Gtk-CRITICAL **: 23:54:22.701: gtk_widget_get_visible: assertion 'GTK_IS_WIDGET (widget)' failed
(app:9912): Gtk-CRITICAL **: 23:54:22.701: gtk_widget_get_visible: assertion 'GTK_IS_WIDGET (widget)' failed
(app:9912): Gtk-CRITICAL **: 23:54:22.701: gtk_widget_is_visible: assertion 'GTK_IS_WIDGET (widget)' failed
[1]    9912 segmentation fault  ./app

Flawed box

What went wrong? The first thing I checked was documentation of add and remove methods on the Container class. The latter one was more interesting (full doc available here):

Removes widget from this. widget must be inside this...

This seemed like a reasonable constraint; I can't remove what's not there. But I just put my hello_world label into the box, how could it not be inside? On to the FlowBox docs (available in full here):

A GtkFlowBox positions child widgets in sequence according to its orientation. ... Although a GtkFlowBox must have only FlowBoxChild children, you can add any kind of widget to it via add, and a GtkFlowBoxChild widget will automatically be inserted between the box and the widget.

There it was: FlowBox wrapped my Label, and so it really wasn't hello_world inside the box, although I thought I had added it. That's why program segfaulted when I tried to remove the label.

Liskov principle violation

In this case it's the FlowBox that breaks the substitution principle. Instance of FlowBox cannot be treated as a Container by the user. This one was easy to spot as I was instantiating the box myself so I knew its exact type. The problem would have been worse had add and remove methods been invoked farther away from the instantiation code. It might have been written by another developer who doesn't even know of the FlowBox class at all.

Understanding what caused the issue was satisfying, but I wasn't ready to let go of it just yet. I wanted to understand what the fundamental issue here was. Container in this case could be considered as a generic container, and FlowBox a derived class with a narrower element type. In this case the GTK classes themselves are not parametrized, but my Java trained brain was happy to recognize a familiar pattern!

Java type system is infamous for having invariant generics, meaning List<String> is not a subtype of List<Object>, even thought String is a subtype of Object. This restriction limits developer's options when writing code but it provides greater compile time safety. (Just as a sidenote; Vala is more loose in this case, allowing more assignments at compile time and failing at runtime.) Kotlin language offers more fine-tuned control while maintaining strict compile time checks. It does this at the expanse of language complexity.

Thinking of Kotlin in this context made me suspect that I was still missing something crucial. I generally find Kotlin interesting but overly complex at times. Its handling of generics is one example of that complexity. So perhaps I was too focused on the type system and compiler in this case? And in GTK widgets hierarchy there was no generic type information to begin with.

Reframing the problem

The issue with FlowBox and Container wasn't that type system was too weak to properly define relationship between them. It's simpler than that. The Container defined a special relationship between its add and remove methods. This contract is described in the documentation. It's easily understandable by a human reader, in fact that's how I managed to find the issue. However, it's completely transparent to the type system and the compiler.

Liskov substitution principle is helpful with precisely this type of problems. When working with strongly typed languages it's easy to forget how many expectations and rules for using a piece of code there are. This is to be expected since the compiler tries to check those rules for us. In a perfect world compiled would stop us from creating invalid derived classes in the first place. However, we are not there yet, and perhaps never will be.

There are a lot of situations when contracts are not enforceable by the compiler. Kotlin's experimental contracts feature is an interesting attempt to expand the scope of the compile time checks. But even that covers only a simple form of contract that is related to a single method. It's hard to declaratively express arbitrary contracts involving multiple methods. The Container's add / remove is one example of such contract. Another popular one is a relationship between equals and hashcode methods.

With this in mind the Liskov substitution principle could be reworded as:

Derived class must not break contracts of its base class.

I find this to be an interesting wording from the perspective of an author of a derived class. Specifically:

1. It drops the user of the code from my consideration. I no longer have to think about how someone else might attempt to use my derived class. There's no more thinking about substitutions.
2. It points me directly at what I need to pay attention to: it's the contracts defined by the base class. Those contracts might be expressible in the type system, in which case I can rely on the compiler. Or they might be implicit, in which case I need to pay more attention. In this way it's more actionable that the original wording.

Admittedly, this new wording has a problem of defining a contract. In this aspect the original form is far more elegant and precise. It's easier to define what a substitution is.

Another problem is that there's no clear standard for expressing contracts outside a type system. Putting them into code comments or documentation is one way to go about it. Comments are cheap to add initially and human readers can usually understand them relatively easily. On the other hand they are hard to maintain and reading them requires context switching. There are also annotations that can be used for expressing contracts. For example Java's @Nullable and @NotNull placed on methods to describe return values and parameters. Some times the code of a base class is available so contracts might be inferred from reading the source itself.

Wrap up

Having written all of this down I was finally happy with my conclusions. Of course, none of this helped me fix my code. For that I found that if I manually wrapped my child widget into FlowBoxChild then add method will add it as-is, making the remove work.

On the GTK side of things, the next major framework version will no longer include the Container class with add and remove methods. The migration guide suggests using specialized methods on the derived classes to add child widgets.

Latest Kotlin version 1.4 does not include any improvements to the experimental contracts feature introduced in 1.3. And speaking of programing languages, I've intentionally refrained from bringing up the Go contracts proposal in this discussion. It had only dealt with generics and was abandoned in favor of reusing existing language concepts.

Lastly, I've created a git repository with the example app from above. I've also included versions in C and Python. Unsurprisingly, all three versions behave the same. You can find them on GitHub.

If you liked this post and are interested in more similar content you can subscribe to this blog's RSS feed or follow @antolius@qua.name in fediverse. Alternatively, you can tell me all the things I'm wrong about over at @antolius on Mastodon.

However, I've found that Cobra's documentation and its generation tool for scaffolding CLI projects encourage writing code with coupled responsibilities that's hard to test. In this post I'll explore an alternative approach to structuring Cobra based CLI apps. My goal is to decouple business logic from the command line interface, and to cover it with unit tests. I've put together a simple project to showcase the idea: Passgen.

Exploring the problem

Let's start by creating a new project using Cobra Generator:

$ cobra init passgen --pkg-name github.com/antolis/passgen
Your Cobra applicaton is ready at ~/passgen
$ tree passgen
passgen
├── cmd
│   └── root.go
├── LICENSE
└── main.go

There's nothing wrong with this file structure per se, but I'd prefer my main.go placed into a properly named subdirectory under cmd/. This will allow me to add other executables in the future, and it works nicely with the go tool. In fact, if you're a fan of meta references, you can check the source of Go tools themselves and see how they implements this pattern.

If we take a look inside the generated main.go file, it looks innocent enough:

// cmd/main.go
func main() {
	cmd.Execute()
}

The cmd.Execute() function call hints at trouble. And here's what the generated root.go file looks like:

// cmd/root.go
var cfgFile string // global state! 😧️

var rootCmd = &cobra.Command{
	/* command initialization */
} // more global state! 😖️

func Execute() {
	if err := rootCmd.Execute(); err != nil {
		fmt.Println(err)
		os.Exit(1) // I'd like main func to define exit code
	}
} // "static" func, uses all that global state 😓️

func init() {
	cobra.OnInitialize(initConfig)

	rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.passgen.yaml)")
	rootCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
} // global state manipulation 😭️

func initConfig() { /* handling of the config file */ }

As you may have noticed, I don't like global state. Here are a few blogs that helped influence my opinions and explain the issue better than I could:

• A theory of modern Go, by Peter Bourgon
• Go, without package scoped variables, by Dave Cheney

In addition to objections to global state in general, there's one issue I'd like to emphasize here: readability. This code implements the registration pattern based on the init func. That makes it hard to unravel the entire CLI app's command structure by starting from the main function.

When adding new sub-commands using the Generator:

$ cobra add generate

newly generated sub-commands use the init func to register themselves with the root command. This is convenient as it allows the Generator to add new sub-commands without having to update any of the preexisting code. However it obscures the sub-command structure, as there's no single place where this strucutre is encoded in the source code.

Cobra's documentation offers an example of an alternative approach. In it all sub-commands are instantiated within programs main function. This is convenient for the documentation example that needs to fit on one page, but doesn't scale well for a larger app.

Better project structure

I structured the Passgen project a bit differently:

passgen
├── cmd
│   ├── passgen
│   │   └── main.go         # package main
│   ├── generate.go         # package cmd
│   ├── params.go           # package cmd
│   └── root.go             # package cmd
└── internal
    └── app
        ├── app.go          # package app
        ├── app_test.go     # package app_test
        └── words.go        # package app

With package dependencies:

+------+   +-----+   +-----+   +----------+
| main +-->+ cmd +-->+ app +<--+ app_test |
+------+   +-+-+-+   +-----+   +----------+
             | |
        +----+ +----+
        |           |
        v           v
    +-------+   +-------+
    | Cobra |   | Viper |
    +-------+   +-------+

This approach matches the hexagonal architecture. As such it's nothing fundamentally new. What I try to do here is apply those tried and true ideas to the particulars of the Cobra library and CLI app development in Go.

Package main

My main func is simple, but introduces a slight variation on the generated one:

// cmd/passgen/main.go
func main() {
	root := cmd.RootCmd() // creating new instance of command
	if err := root.Execute(); err != nil { // Execute is a method
		log.Fatal(err) // exit code is defined here in main
	}
}

This enables me to define the root command with:

// cmd/root.go
func RootCmd() *cobra.Command {
	cmd := &cobra.Command{ /* command initialization */ }
	cmd.AddCommand(
		generateCmd(), // sub-commands are listed explicitly
	)
	return cmd
}

I admit that listing sub-commands explicitly violates the open-closed principle because I have to update my root command whenever I add another top level sub-command. However, I find the simplicity of directly invoking methods preferable to the opaque self-registration based on the init function. This way you can follow method calls from the main func right down to the business logic.

Business logic

Speaking of the business logic, it's the interface between that logic housed in the app package and the cli package that's more interesting. Here I have two goals:

1. Keep Cobra and Viper dependencies confined to the cli package.
2. Inject other dependencies into the app package to enable easy testing.

With that in mind, here's what the app itself looks like:

// internal/app/app.go
type App struct {
	Params *Params
	Out io.Writer
	Random io.Reader
}

type Params struct {
	Min      int
	// other externally configurable parameters
}

func New() *App {
	return &App{ &Params{}, os.Stdout, rand.Reader }
}

func (a *App) Generate() error {
	// implementation that first validates a.Params
	// then it uses them and the provided a.Random
	// to generate a password that it writes to a.Out
}

The App struct encapsulates all application dependencies.

The custom Params struct defines all user-configurable input parameters. Note that the App neither knows nor cares how user provided those parameters. App does, however, handle parameter validation, making that part of the testable business logic.

App sends all its output to the provided io.Writer. This makes writing unit tests easier because they don't need to sniff out the standard output, and they can still check the output formatting. This makes output formatting another part of the business logic.

Lastly, the injected Random io.Reader is used as a source of randomness. The crypto/rand package from the standard library dictates this type. This is just one example of a dependency; in other applications you might want to inject HTTP client, database connection, etc.

With all this in place I can write classic table driven unit tests for all my business logic. It also means that I can write a CLI command like this:

// cmd/generate.go
func generateCmd() *cobra.Command {
	a := app.New() // App instance with production grade dependencies
	cmd := &cobra.Command{
		Use:     "generate",
		// documentation related fields...
		PreRunE: func(cmd *cobra.Command, args []string) error {
			// this reads parameters from command line
			// arguments, flags and config files:
			return initParams(cmd, a.Params)
		},
		RunE: func(cmd *cobra.Command, args []string) error {
			return a.Generate()
		},
	}
	cmd.Flags().IntVarP(&a.Params.Min, "min", "m", 16, "Min length")
	// the rest of the mapping from app.Props to flags
	return cmd
}

One good thing about this approach is that no matter how complex the app logic gets its Cobra command will always be more-less the same. The only code that I allow in the cmd package is used to set up the Cobra command.

Some extra benefits

In addition to easy testing, there are a few nice consequences of this design. Not all of them are immediately useful, and some of them may indeed never be used. I still find them interesting!

First of all, since I removed all the static init functions from the cli package, and since all commands are explicitly registered with their parent commands, it's now easy to create separate executables for a subset of those commands. All that's needed is to define a separate main function that executes not the root command, but one of the other higher level sub-commands.

Second, I use the same app.Props struct for all sub-commands. This helps me keep various input parameters consistent and unique on the application level. I can then read them from a config file, where a user can define defaults for any flag without specifying the individual command that the flag is used with.

Lastly, the App and it's logic is completely decoupled from the command line. That means it can easily be exposed behind other interfaces, like an HTTP server, or wrapped into a long running daemon process. This is usually not the first consideration when building CLI apps, but can become a handy benefit as project grows and evolves.

Conclusion

In conclusion, I find this project architecture allows me to test and scale my CLI apps and to get most out of libraries like Cobra and Viper. As previously mentioned, you can find the complete Passgen project on GitHub.

If all this command line talk has piqued your interest, I invite you to check my more general blog series on designing a CLI application:

1. CLI Development; Part 1 – Tower of Babel
2. CLI Development; Part 2 – Command line citizen
3. CLI Development; Part 3- Advanced use cases

And in case you are interested in seeing this kind of content from me in the future, you can:

• Subscribe to this blog's RSS feed.
• Follow @antolius@qua.name in fediverse.

Or just come chat with me @antolius on Mastodon

Use-case

Like a lot of stories, this one begins with a feature request. A few days ago I received a request from a client who wished to send large amounts of data through query parameters. Servers generally have a default limit on the maximum HTTP request size and will reject offending requests with HTTP status code 413 or 414. This behavior is configurable, and Spring Boot, which Cloud Gateway builds upon, provides a nice configuration abstraction that works with different servers.

This should be easy; 1 story point.

famous last words

1st attempt

Spring Boot's common application properties include a property for just this situation: server.max-http-header-size. At first, I thought I'd solve the tasks with a single config line in my app's application.yaml. Luckily I also added a test to cover this case. In my tests I use WireMock to simulate the backend service and then send requests through the gateway application. Test was rather straightforward:

@Test
void shouldSupportLongQueryParameters() {
    // given
    var givenQueryParamLength = 10_000;
    var givenQueryParam = new String(new char[givenQueryParamLength]).replaceAll("\0", "a");
    stubFor(get(urlEqualTo("/foo/bars?q=" + givenQueryParam))
            .willReturn(aResponse().withStatus(200))
    );

    // when
    var actual = client.get().uri("/foo/bars?q=" + givenQueryParam).exchange();

    // then
    actual.expectStatus().isOk();
}

To my surprise, running it resulted in an assertion failure:

Status expected:<200 OK> but was:<413 PAYLOAD_TOO_LARGE>

2nd attempt

Ok, I guess the common application property doesn't work. No problem. Spring Cloud Gateway is using Reactor Netty web server under the hood. So the next thing I looked at was that project's documentation. And indeed, as per documentation, Netty has 2 configuration parameters of note:

1. The maximum length of the initial HTTP request line
2. The maximum length of all headers

That explained why setting the max-http-header-size config property did not solve the issue: Netty requires a separate parameter for the initial line length, which query parameters are a part of. At this point I registered a slightly customized instance of NettyReactiveWebServerFactory as a bean in my context. In it I set both max header size and max initial line length:

@Bean
public NettyReactiveWebServerFactory nettyFactory(ServerProperties serverProperties) {
    var maxInBytes = (int) serverProperties.getMaxHttpHeaderSize().toBytes();
    var factory = new NettyReactiveWebServerFactory();
    factory.addServerCustomizers(
            server -> server.httpRequestDecoder(
                    reqDecoder -> reqDecoder
                            .maxInitialLineLength(maxInBytes)
                            .maxHeaderSize(maxInBytes)
            )
    );
    return factory;
}

That got me absolutely nowhere:

Status expected:<200 OK> but was:<413 PAYLOAD_TOO_LARGE>

3rd attempt

From this failure I realized I'm still missing something crucial about the way in which Spring Boot sets up the Netty server. So I sprinkled a few breakpoints around the code and started debugging. The first thing I noticed was that my server factory's addServerCustomizers method was getting invoked a second time from within Spring code, specifically from NettyWebServerFactoryCustomizer.

Turns out I didn't have to instantiate the entire server factory bean. There's a factory customization mechanism and all I needed to do was provide my own implementation of the customizer interface. The one implementation that Spring provides sets the Netty's max header size to the value of max-http-header-size application property. However, this did not explain why the value of max initial line length I was setting in my factory bean was getting ignored.

To understand this I had to look into Reactor's HttpServer, specifically the code that factory customizers use to set max header and initial line values. According to the implementation of the HttpServer::httpRequestDecoder method each attempt to customize those values causes server to create a brand new spec object. This explained how Spring's factory customizer, which only sets max header size, managed to override my initial line config. What I needed to do was provide a factory customizer with lower order of execution, so that my values don't get overridden. I created my own WebServerFactoryCustomizer, and set its order to Ordered.LOWEST_PRECEDENCE. Implementation was the same as before: I set both max header size and initial line length to the same value.

I rerun the test and got:

Status expected:<200 OK> but was:<414 URI_TOO_LONG>

That was... different?

4th attempt

I've already established that Netty will return status code 413 if incoming request goes over max initial line setting. So this latest issue must have been somewhere else. Going through my test output log I found this mystery line:

w.org.eclipse.jetty.http.HttpParser      : URI is too large >8192

Where did Jetty come from? Well, there were actually 2 servers in the test: Spring Cloud Gateway app running on Netty and WireMock running on Jetty. Which meant my last attempt worked, I just needed to fix the test! Up until then I relied on Spring to auto-configure WireMock for me. However, I now needed to customize the max URL size for it. Luckily, this can be done easily by providing an instance of an Options bean:

@Bean
public Options wireMockOptions(@Value("${wiremock.server.port}") int port) {
    return WireMockConfiguration.options()
            .port(port)
            .jettyHeaderBufferSize(16_384);
}

Jetty header buffer size was the only property that WireMock needed in order to configure larger max URL size. With this change the test finally passed, and the feature was complete!

Example

I've created a demo project on GitHub that illustrates this implementation. I marked each of my attempts with a git tag, so you can follow them step by step. Feel free to check out the project and play around with it. If you find a better solution, please let me know about it! You can reach me at:

• josip.antolis@protonmail.com
• @antolius on Mastodon

I will examine some coding best practices and see how they can be applied to code review itself. I'll borrow inspiration from Uncle Bob Martin. I agree with many of his ideas and they make for some good quotes. I have experience working in a product oriented, mid-sized development team. Ideas I'll be discussing relate to a 5 – 10 person team reviewing each others code that's mostly constrained to a handful of products they own.

It's not about catching bugs

The secondary value of software is its behavior. ... The ability of software to tolerate and facilitate such ongoing change is the primary value of software.

Uncle Bob

I really like this idea of Uncle Bob's: that fulfilling immediate requirements is secondary to facilitating the future changes. I believe similar can be said of code reviews. The secondary value of code review is to improve the quality of a given code update. Its primary value is to improve all the future code updates!

This commitment to long term improvement means that code reviews should be a learning opportunity for everyone involved. The goal is no longer just to identify bugs, but to teach each other how to produce better code.

In order to achieve this, reviewer should use their comments to educate. It's not enough just to point out deficiencies. They should propose improvements along with the reasoning for them. The reviewer can be helped in this by a shared set of team values and a common language.

The set of shared values can be as simple as everyone on a team agreeing that they should write clean code. This is where common language kicks in by insuring everyone's idea of clean code is the same. Common team language is also valuable outside code reviews and there are techniques for building it. One technique I like is a book club in which team members read the same technical book and meet every few weeks to discuss the latest set of chapters. Another technique is the architecture drawing exercise I wrote about before. All this allows reviewer to, for example, simply reference one of the SOLID principles as a motivation for their comment.

Education goes both ways! One way for code author to educate the reviewer is by following good coding practices. They can, however, do more. While comments in code itself are undesirable, I find them useful in a code review. Author of the code can leave comments in the pull request that can be educational for the reviewer.

I find this a good method for teaching design patterns. They can often be hard for junior developers to grasp. While books on the subject do contain examples, those can be hard to relate to. When a code author introduces a particular design pattern in a pull request they can leave a comment with a link to a theoretical explanation of the pattern. This way a junior reviewer can learn the pattern by seeing it applied to a code base they're familiar with, and solving a domain problem they understand.

Encourage good behavior

Leave the campground cleaner than the way you found it.

Uncle Bob

There are some good behaviors you may wish to encourage in your team. For me a good example of that is the famous boy scout rule: always leave the code just a little bit better than it was when you found it. This rule is easy to express and can be easily recognized by a human. Yet, it's somewhat hard for automatic checks such as liters to detect and enforce it.

This makes it a perfect consideration during code review. Reviewer can call for boy scouting where author may have missed it. However, they have an even greater opportunity: to praise the improvements already submitted by the author! This is a handy way to give immediate positive feedback on a desired behavior. And it's a damn effective one. If you've ever received a comment praising your pull request you'll know how good it can feel when someone else recognizes your hard work and commitment to good practices.

Review for readability

The ratio of time spent reading (code) versus writing is well over 10 to 1 ... (therefore) making it easy to read makes it easier to write.

Uncle Bob

Code review is often the first time someone besides the author reads the code. Simultaneously, this is also the time when it's easiest to change the code. It's still fresh in author's mind, it's not even in the master branch yet, no one is using it. This combination can be a powerful one.

Junior reviewers can have an advantage here. Intermediate level developers can have a tendency to over-complicate their code. This is a simple byproduct of their expanding skill set. Having a junior reviewer on a such pull request can be helpful as they can call it out. Sometimes a complex chain of callbacks can be rewritten as a simple for loop that will be easier to understand for everyone.

Code review is also a good opportunity to get buy in from another member of the team. If maintenance and future development is shared between all team members it's important that everyone accept and understand the author's code as if it was their own. Code review is the first time this happens. We should encourage reviewers to express their confusion with the code. If they don't understand the code today, when they are seeing it in isolated pull request and they have time to examine it, then they won't be able to troubleshoot it tomorrow, when it's integrated with the code base and they are under pressure to fix an issue.

Don't automate away the humanity

Programming is a social activity.

Uncle Bob

There are aspects of code review that can and should be automated. Things like code formatting, compiling and linting. By all means add git hooks to ensure uniform code style. Hook up CI pipeline to build the project and run tests on each commit. In my team we agree to start the review only after Jenkins job has passed.

All these tools help automate repetitive steps and make reviewer's life easier. However, I'd caution against relying on them exclusively. I still need a fellow developer to check the readability of my code. Static code analysis can gamify development, but it can't replace the recognition and encouragement of my peers. There's no substitute for discussion of coding practices and learning how to apply them that might happen during a review.

Quality code review requires time, commitment and skill. Reviewer needs to approach it with dedication and seriousness. Code author should take the feedback in with an open mind. It's hard to receive critique of ones own code. It's hard to distinguish personal preference from best practice. This can be solved with mutual respect, trust and good intentions from both parties. With that, benefits of a quality code review span far beyond a single pull request.

If you have any feedback on the subject of code reviews, you can reach me:

• via email: josip.antolis@protonmail.com
• or on Mastodon @antolius

Use-case

Kotlin has language features that make it well suited for scripting and using DSLs. As such it is a good candidate for writing more involved configuration files. A good example of this is Gradle's Kotlin DSL. The configuration file use-case can be generalized into a plugin system. In this case the base application can be extended at runtime with any number of Kotlin scripts.

Such plug-able application can be paired with an HTTP server. In this case the server can execute a plugin script in response to an HTTP request, and use its output as an HTTP response. At this point the application is begging to resemble a simplified Function as a service system. It can read plugin scripts from a public source (e.g. a git repository) and have users submit their own scripts.

With time, as the service grows in popularity, users submit many different scripts. On the other hand development of Kotlin progresses and there are new language version releases. Script developers, enticed by new language features, start requesting support for newer Kotlin versions.

Unfortunately, some of the code in existing scripts is using experimental Kotlin 1.1 features that have suffered breaking changes in subsequent 1.2 and 1.3 releases. This makes upgrading the Kotlin version in the base application problematic. Migrating to 1.3 would make enthusiastic script developers happy, but would also break some existing scripts. On the other hand, scripts cannot be upgraded to newer Kotlin versions and still run on the existing 1.1 version used by the server application. This chicken and the egg problem causes a stand still and Kotlin version is locked to 1.1.

The solution

In this experiment I've built a simplified plugin engine that can be used to load and execute Kotlin scripts with various language versions. I've skipped fetching scripts form public git repositories, running HTTP server and matching requests to scripts. The solution is lacking in security and performance as well.

What it does achieve is illustrate a solution of the stated use-case in an isolated, self contained project. Complete source code can be found on GitHub.

Running scripts

My first step was to build Java app that can load and run a Kotlin script at runtime. Java itself provides a scripting API (defined by JSR 223) for abstracting the mechanics of running scripts in compatible languages. Form its version 1.1 Kotlin provides an implementation of this Java API.

API documentation suggests registering scripting engine implementations in a special file inside META-INF directory and using dedicated manager to obtain engine instances. However, since I was planning on doing some jar juggling later on, I decided to instantiate Kotlin script engine programmatically and keep all of it in my source code.

At this stage the application was a straightforward maven project with few lines of Java code.

<project>
    <modelVersion>4.0.0</modelVersion>
    <groupId>io.github.antolius</groupId>
    <artifactId>kotlin-engine-experiment</artifactId>
    <version>0.1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <properties>
        <java.version>8</java.version>
        <kotlin.version>1.1.61</kotlin.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-script-util</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-script-runtime</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-compiler-embeddable</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jetbrains.kotlin</groupId>
            <artifactId>kotlin-reflect</artifactId>
            <version>${kotlin.version}</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

public class Main {
    public static void main(String[] args) throws Exception {
        String kotlinScript = "\"Using Kotlin version: "
            + "${KotlinVersion.CURRENT}\"";
        ScriptEngine engine = new KotlinJsr223JvmLocalScriptEngineFactory()
            .getScriptEngine();
        Object result = engine.eval(kotlinScript);
        System.out.println(result); // Using Kotlin version: 1.1.60
    }
}

ScriptEngine provides a method for evaluating source code from a java.io.Reader, so executing a script from a file was easy too.

Multi module

Now that the app could load and run Kotlin scripts it was time to support multiple language versions. The maven project described in the previous section declares a compile-time dependency on a specific Kotlin version. Since I needed to support multiple versions I decided to load Kotlin dependencies into separate class loaders at runtime. That way I could work with different versions of Kotlin libraries at the same time.

Next step was to turn the application into a multi module maven project. That way each module could declare a dependency on a distinct Kotlin version. I created 4 modules:

• Three Kotlin modules, each with a dependency on a different Kotlin version and code for instantiating the Kotlin JSR 223 script engine.
• Engine module, that contained script loading logic, and didn't have any Kotlin dependency. Instead it loaded the three Kotlin modules' jars in runtime.

Each of the three Kotlin modules included a single class:

package io.github.antolius.engine.kotlin1;
// imports...
 
public class ScriptEngineSupplier implements Supplier<ScriptEngine> {
    @Override
    public ScriptEngine get() {
        return new KotlinJsr223JvmLocalScriptEngineFactory().getScriptEngine();
    }
}

The only difference was the package name. Module with 1.1 Kotlin dependency had package kotlin1, module with 1.2 package kotlin2 etc.

The engine module created a dedicated class loader for each language version and loaded individual Kotlin module jar in runtime. It instantiated the Supplier from that jar:

public class ScriptEngineFactory {
    public ScriptEngine newEngine(
        URL kotlinModuleJar,
        String fullyQuelifiedSupplierClassName
    ) {
        ClassLoader classLoader = newClassLoaderWith(kotlinModuleJar);
        Supplier<ScriptEngine> supplier = instantiateSupplierFrom(
            fullyQuelifiedSupplierClassName,
            classLoader
        );
        return supplier.get();
    }

    private ClassLoader newClassLoaderWith(URL kotlinModuleJar) {
        try {
            return URLClassLoader.newInstance(
                new URL[] {kotlinModuleJar},
                getClass().getClassLoader()
            );
        } catch (Exception e) {
            throw new RuntimeException("Couldn't create class loader", e);
        }
    }

    private Supplier<ScriptEngine> instantiateSupplierFrom(
        String className,
        ClassLoader classLoader
    ) {
        try {
            Class<?> loadedClass = Class
                .forName(className, true, classLoader);
            Class<Supplier<ScriptEngine>> supplierClass = cast(loadedClass);
            Constructor<Supplier<ScriptEngine>> constructor = supplierClass
                .getConstructor();
            return constructor.newInstance();
        } catch (Exception e) {
            throw new RuntimeException("Couldn't load " + className, e);
        }
    }

    @SuppressWarnings("unchecked")
    private Class<Supplier<ScriptEngine>> cast(Class<?> loadedClass) {
        return (Class<Supplier<ScriptEngine>>) loadedClass;
    }
}

This ScriptEngineFactory could be used to obtain an instance of JSR 223 script engine capable of running Kotlin scripts with different versions. The problem was in knowing where to find those Kotlin module jars.

The application technically worked, however it was difficult to build manually. At this point I had to compile each Kotlin module into a fat jar that contains both the Supplier implementation and its Kotlin dependencies. Then I had to somehow provide paths to those fat jars to the engine application at runtime.

Maven tricks

Building jars

First thing that maven could help with was building fat jars for the three Kotlin modules. For this I used maven assembly plugin. Specifically I configured the plugin for Kotlin 1 module with:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-assembly-plugin</artifactId>
    <version>3.1.1</version>
    <configuration>
        <descriptorRefs>
            <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
        <finalName>kotlin-1-module</finalName>
        <appendAssemblyId>false</appendAssemblyId>
    </configuration>
    <executions>
        <execution>
            <id>assemble-all</id>
            <phase>package</phase>
            <goals>
                <goal>single</goal>
            </goals>
        </execution>
    </executions>
</plugin>

I defined the final jar's name as kotlin-1-moule. This is important, since by default maven would have included the project version in the name and that would have complicated things for me later on. Config for other modules was nearly identical, just with different jar names.

Packaging jars

The next problem was finding those jars from within the engine module's code. I solved this by packaging fat jars as resources of the engine module. For this I used maven resources plugin configured in engine module's pom with:

<plugin>
    <artifactId>maven-resources-plugin</artifactId>
    <version>3.1.0</version>
    <executions>
        <execution>
            <id>copy-resources</id>
            <phase>validate</phase>
            <goals>
                <goal>copy-resources</goal>
            </goals>
            <configuration>
                <outputDirectory>
                    ${project.build.outputDirectory}/kotlin-jars
                </outputDirectory>
                <resources>
                    <resource>
                        <directory>
                            ${rootdir}/kotlin-1-module/target
                        </directory>
                        <filtering>false</filtering>
                        <includes>
                            <include>kotlin-1-module.jar</include>
                        </includes>
                    </resource>
                    <resource>
                        <directory>
                            ${rootdir}/kotlin-2-module/target
                        </directory>
                        <filtering>false</filtering>
                        <includes>
                            <include>kotlin-2-module.jar</include>
                        </includes>
                    </resource>
                    <resource>
                        <directory>
                            ${rootdir}/kotlin-3-module/target
                        </directory>
                        <filtering>false</filtering>
                        <includes>
                            <include>kotlin-3-module.jar</include>
                        </includes>
                    </resource>
                </resources>
            </configuration>
        </execution>
    </executions>
</plugin>

where ${rootdir} was defined as:

<properties>
    <rootdir>${project.parent.basedir}</rootdir>
</properties>

For this to work properly I needed maven to build Kotlin modules first. This would generate the fat jars in their individual target directories. Then maven could build the engine module, and include fat jars as resources. Since engine module didn't declare direct dependencies on Kotlin modules, maven built modules in the order they appeared in the parent's pom:

<modules>
    <module>kotlin-1-module</module>
    <module>kotlin-2-module</module>
    <module>kotlin-3-module</module>
    <module>engine</module>
</modules>

Cleanup

With this setup engine module was packaged with all of its runtime dependencies. Since individual jars didn't include project version in their names, the jar resource names were constant. This allowed me to define the jar URL and the fully qualified name of the Supplier implementation in an enum:

public enum Language {

    KOTLIN_1_1(
            "io.github.antolius.engine.kotlin1.ScriptEngineSupplier",
            "/kotlin-jars/kotlin-1-module.jar"
    ),
    KOTLIN_1_2(
            "io.github.antolius.engine.kotlin2.ScriptEngineSupplier",
            "/kotlin-jars/kotlin-2-module.jar"
    ),
    KOTLIN_1_3(
            "io.github.antolius.engine.kotlin3.ScriptEngineSupplier",
            "/kotlin-jars/kotlin-3-module.jar"
    );

    private final String supplierClass;
    private final String resourceName;

    Language(String supplierClass, String resourceName) {
        this.supplierClass = supplierClass;
        this.resourceName = resourceName;
    }

    public String getSupplierClass() {
        return supplierClass;
    }

    public URL getJarURL() {
        return this.getClass().getResource(resourceName);
    }
}

The ScriptEngineFactory#newEngine method could now be simplified into:

public class ScriptEngineFactory {
    public ScriptEngine newEngine(Language version) {
        // rest of the code is more-less the same...
    }
}

Plugin API

At this point maven could build the project automatically, and the ScriptEngineFactory could be used to obtain JSR 223 script engine for any Kotlin version. The last usability problem left was defining an interface between scripts and the engine.

For this purpose I introduced a dedicated API maven module, and added it as a dependency for the rest of the modules. Within it I defined the Plugin interface as:

public interface Plugin {
    @NotNull
    Response process(@NotNull Request req);
}

Request and Response were just POJOs encapsulating all input and output parameters of the process method. All that Kotlin scripts need to do in order to be compatible with the engine is return an instance of a Plugin.

I also wanted to demonstrate how dependencies might be injected into scripts from Java code, so I defined two more interfaces:

public interface Printer {
    void print(@NotNull String line);
}

and

public interface PrinterAware {
    void set(@NotNull Printer printer);
}

Java code should implement the Printer, and scripts interested in using it should implement PrinterAware interface. To pull it all together I created a PluginLoader class in the engine module:

public class PluginLoader {
    private final ScriptEngineFactory factory;
    private final Printer printer;

    public PluginLoader(ScriptEngineFactory factory, Printer printer) {
        this.factory = factory;
        this.printer = printer;
    }

    public Plugin load(File sourceFile, Language kotlinVersion) {
        Reader sourceFileReader = readerFrom(sourceFile);
        ScriptEngine engine = factory.newEngine(kotlinVersion);
        Plugin plugin = runScript(sourceFileReader, engine);
        autowirePrinter(plugin);
        return plugin;
    }

    private Reader readerFrom(File source) {
        try {
            FileInputStream stream = new FileInputStream(source);
            return new InputStreamReader(stream, StandardCharsets.UTF_8);
        } catch (FileNotFoundException e) {
            String message = "Couldn't create a reader for " + source;
            throw new RuntimeException(message, e);
        }
    }

    private Plugin runScript(Reader sourceReader, ScriptEngine engine) {
        try {
            Object result = engine.eval(sourceReader);
            return (Plugin) result;
        } catch (Exception e) {
            String message = "Couldn't evaluate script to a Plugin instance";
            throw new RuntimeException(message, e);
        }
    }

    private void autowirePrinter(Plugin plugin) {
        if (PrinterAware.class.isAssignableFrom(plugin.getClass())) {
            PrinterAware aware = (PrinterAware) plugin;
            aware.set(printer);
        }
    }
}

PluginLoader could be used to load Kotlin scripts from files and provide the rest of the application with implementations of the Plugin interface.

In conclusion

The final Java application uses a few tricks to enable execution of mutually incompatible Kotlin scripts:

1. Kotlin's implementation of JSR 223 script engine is used to evaluate scripts.
2. Script engines themselves are loaded into separated class loaders at runtme.
3. Various maven plugins are used to package all the different Kotlin dependencies as resources at build time.
4. Simple interfaces define a contract between Kotlin scripts and the rest of the application.

As mentioned before, the complete project can be found on GitHub. In addition to the code discussed in this post, the full project contains a few extra bits:

• Tests that verify this whole thing actually works.
• An example main method that can be executed to run a script.
• A few Kotlin scripts implementing the Plugin interface.
• Ability to load .kt classes in addition to .kts scripts.
• Caching and reuse of Kotlin version specific class loaders.

Building this prototype was a fun exercise for me, but now I'd be interested in hearing from you as well. Do you see this kind of dynamic plugin system fitting into some of your projects? Have you encountered similar use-cases before, and if so, how did you solve them? Let me know:

• via email: josip.antolis@protonmail.com
• or on Mastodon @antolius

In the team I lead there are members with different skill levels, and we are developing and supporting multiple production services. I wanted to simultaneously:

• Improve the architecture-level overview juniors had of our services.
• Bring our shared visual language to a next level.

Here's an exercise I came up for achieving this.

Rules of the game

Players sit in a circle, for example around a conference table. They should have some space for themselves on the table, and they should have a clear idea of who the next person around the table is. At the beginning of the game, each player is given a peace of paper with a name of one of the team's services written on the top.

In the first round, which takes 7 minutes, each player must draw the architecture of the service named on their paper. They must not use any letters or numbers in their diagrams. Other symbols, arrows, etc are permitted. After they're done drawing they should fold the paper so that only the diagram is visible and pass it to the next person around the table.

In this way papers are shifted by one place, and each player is now presented with a drawing of an architecture diagram. In the second turn each player must guess the service that the diagram is depicting and write its name on the paper. This turn is a bit shorter, lasting at most 5 minutes. After the time expires, or everyone is done, each player should fold the paper again so that only the service name is visible and pass it on to the next player.

The third turn is the same as the first one, with all players drawing the service named on their papers, folding them and passing them on. The game continues with a succession of odd drawing turns and even service naming turns until papers have made a full circle around the table. At the end all papers are unfolded and the group compares and comments on their diagrams and guesses. Each player picks the best ideas for drawing the service they started with and presents the definitive diagram for it. Those diagrams can later on be documented and refereed back to when needed. Additionally, expressed ideas, iconography, patterns etc. become part of the common visual language of the team and can be used in future diagrams to convey ideas more quickly and precisely.

Takeaways from running the exercise

It was immediately apparent how fun the exercise was for all participants. We've had many laughs through the drawing and guessing parts of the game. It was equally fun to guess the service from others' diagrams and to despair over ones own predicament when having to draw them. The fun helped more recluse team members to open up, and encouraged experimentation while reducing the fear of failure for less experienced teammates.

It was also interesting to see the difference in approach between team members with more and less experience. The more senior people tended to use established diagram types and shapes corresponding to certain architecture component. For example using a cilindar for a database or storage component, rectangle for an application, etc. On the other hand juniors tended to freestyle more, with some of them drawing business processes in almost cartoon-like style.

The rule prohibiting the use of letters and numbers meant that team members had to come up with their own ways of labeling certain architecture components. In this aspect seniority did not appear to play a major role, as all participants approached the challenge with different levels of creativity. Common solutions included labeling third party technologies with their respective logos, and making up icons for internal applications that lack such branding. (As a side note, this got us thinking about developing branding for our in-house applications!)

In the end we agreed on a common set of symbols for representing general architecture components and iconography for particular services. We also created a canonical diagram for each of our team's services.

Review

Later on I collected direct feedback on the exercise from all team members in our by-weekly one-on-one meetings. They all reacted well to it, and reported having fun (that much was always obvious). What was left as an open question is if the gained knowledge warranted the time investment. In the end an equivalent effect might have been achieved if a senior developer on the team took the time to render the architecture diagrams for all services and had juniors examine it and read up on different types of technical drawings.

One benefit of the drawing exercise over this top-down approach is in its team building potential. Since all team members participate in defining the canonical diagrams and common iconography the sense of ownership over the final drawings is also shared. Additionally, those team members who benefit more from affirmation and positive feedback will find the inclusive nature of the exercise rewarding, since everyone's ideas are incorporated in the final design.

If spending some extra time on this type of exercise sounds like a worthwhile trade-off to you, then feel free to run it with your own team! If you do so, I'd love to hear your own results, reaction of your team members and any additional observations you might have. You can get in touch with me:

• Via email: josip.antolis@protonmail.com
• Or on Mastodon @antolius

In this blog post I'll describe my personal experience with some challenges of establishing the incident management process and why I built a tabletop role-playing game Deployments and Disasters to deal with them.

My perspective

I work at Infobip. We are a tech company with over 300 developers and additional 200 customer support engineers. Our developers are divided into teams of around 5 to 10 people. Teams own their services and are responsible for both building and maintaining them in production. Support engineers monitor key business metrics and maintain contact with clients. In total that's over 500 people and each one of them can be called upon to participate in resolving an ongoing incident.

One benefit of this approach is that people most familiar with a given code-base will directly work on resolving the incident. Additionally, trained support personnel are in contact with clients thus allowing developers to focus on fixing the issue.

On the other hand, there are drawbacks to this approach. Not everyone will work equally well under pressure, different people have different levels of experience, each team might develop their own procedures, use different tools, etc. Most of these can be addressed with a fine-tuned incident management procedure and a set of common tools to back it up. Some good ideas include chatops, centralized logging and metrics, premade dashboards and alerting. I will not go into details on these things here.

Challenges to tackle

What I'd like to focus on instead are the following three challenges that emerge after management procedure and all of the tools are in place:

1. All involved with incident management should familiarize themselves with the procedure and available tools. The more people the bigger this issue is. At one point even awareness of the process can become a problem.
2. Incident management process involves different roles: customer support, programmers, sysadmins, database administrators, devops engineers, etc. They all need to work together, despite having different objectives at any given time during the incident.
3. Many roles involved in incident management are technical. They view resolving the incident as their objective and are focused on detecting and removing the immediate cause of the issue. As a result they may not think of affected customers and thus miss out on opportunities to notify them of the impact, or even alleviate parts of it sooner.

Awareness of the procedure

Educating people about procedures and tools can generally be achieved with incident management training. Roughly speaking, there are 2 approaches to it:

1. Simulating the incident realistically, with participants using actual tools, interacting with high fidelity data and directly applying their real life skills.
2. Keeping the training abstract and basing it on gaming techniques. I've found success with adopting the mechanics of tabletop role-playing games.

Picking a game based approach has several advantages. For one, it reduces the prohibitive cost of recreating the data required to realistically simulate the incident. It allows for addressing the other two challenges, namely the empathy towards other roles and customer centric mentality.

However, the killer feature of game based training is that it's fun. Especially when compared to reading procedure documentation and how-to guides, or attending seminars. The benefits of this are twofold. First it makes the exercise more memorable for the attendants. Secondly, it helps with organizing future sessions, as people are more interested in attending.

There's one additional benefit of role-playing based approach. It turns entire exercise into a structured storytelling experience. This structure provides a safe environment for all attendants to share their insights and knowledge with each other. The benefits are most noticeable with introverted players.

Empathy for other roles

At any one time during the incident, each different role might have a different objective. For example, support engineer needs to inform the clients of exact impact of the incident. On the other hand programmers need to identify the cause of the issue. In this situation support needs information on client facing API from the development team that is focused on debugging the backend. This can create tension between those two roles.

One thing that games excel at is placing players into other people's shoes. In Deployments and Disasters I facilitate this by defining specific roles with unique mechanical characteristics. When starting the game session I make sure that players shuffle the roles so that they don't play the same one they have in real world. For example, I encourage developers to play the role of customer support.

This has two benefits:

1. Players get to experience what incident looks like from the perspective of other roles. This builds empathy by making players go through the tough choices and strive for hard to reach objectives that their colleagues usually experience.
2. It also encourages players to share their knowledge and practices. It reverses the real world dependencies between the roles. For example, if developers usually depend on database administrators for optimizing their databases then inverting the roles will make admins more sympathetic towards the other role's needs.

Customer centric mindset

I'd like my developers to approach incident management with more of a customer centric mindset. Other teams, companies or situations may require some other adjustments. Fortunately, game mechanics are well suited for this.

In games, players regularly receive and accomplish arbitrary objectives. By carefully picking stated objectives and mechanical incentives game designers can impact player mindset.

In Deployments and Disasters I achieve this with a few rules:

1. The main objective of the game session is to resolve the incident within a set number of turns, represented by an incident clock. At the beginning of the game players have 6 turns to resolve the issue. However, If they devote time to communicate the issues to the clients their time doubles to a total of 12 turns.
2. Clients are active actors in the game (controlled by the DM) and they can impact the state of the system. For example, they can escalate the problem by attempting to fix it themselves. Alternatively they can be used to reveal valuable information and hints.
3. During the course of the game some important (gold / platinum) clients can contact the players and ask for status updates or request special attention. This can be used to illustrate different types of clients.
4. Incident scenario starts with only some clients impacted. Players can still escalate the situation and spread the impact to other clients. Or, they can proceed with caution and reduce the impact as they go along.

Work so far

So far I've set up basic set of rules and game mechanics for Deployments and Disasters which you can find on GitHub. The game presented there is early sample of a work in progress. One significant ommition is the lack of incident scenarios. I've created one of them for test runs I've played at work, however it is tightly coupled with our internal procedures and custom tools we use. My plan is to create an example scenario with open-source tooling that anyone can use as a base for their exercise.

I've held two test training sessions at work and feedback was generally good. Players found the game entertaining, but also reported learning about new tools and procedures. I'm yet to create additional scenarios, but there's interest in replaying the existing one with teams that haven't seen it yet. I'm also exploring ways of connecting the exercise with employee evaluation and professional development programs that we have.

Feel free to use the Deployments and Disasters to build your own incident scenarios on top of. Or stay tuned for future developments, as I will strive to publish example scenarios myself. You can watch the GitHub repo for updates, or follow this blog by:

• Subscribing to its RSS feed.
• Following @antolius@qua.name in fediverse.

If you have any feedback, comments or improvement ideas you can send me a pull request, or just contact me at:

• josip.antolis@protonmail.com
• @antolius on Mastodon

In this part of CLI development series I'll go over some of the more advanced use cases. I've previously discussed general tips for making command line apps nice to use. If you missed that blog post you can find it here.

As use cases grow more complex it makes more and more sense to look for existing solutions and reuse / incorporate them into our own applications. That's why I'll devote more space in this post to highlighting existing projects, as opposed to talking about my own experiences.

Source code generation

One use case where CLIs excel is project scaffolding and code generation. For example of such apps you can take a look at Yeoman and its list of generators. Yeoman itself is oriented towards web development, although its most popular generator JHipster outputs Spring Boot applications. As a side note, if you ever find yourself with some spare time, checking out JHipster is a wonderful way to spend few hours. One more old school example of project scaffolding in Java world are Maven archetypes.

If you look at those examples you will quickly find that they provide rich set of features and customizations. One thing that most dedicated code generation tools have in common is a plugin system that allows users to define their own templates. If code generation is your primary use case, developing a plugin for an existing tool is a good idea. That approach will save you a lot of time and you'll end up with a more polished product.

On the other hand, there are CLIs that offer code generation as an addition to their core features. For example think of init commands in tools like npm or git. Extracting scaffolding features out of these tools and delegating it to dedicated code generations apps would be detrimental to user experience. If you find yourself in a similar situation you should implement code generation within your CLI instead.

Most popular approach to code generation is to treat source files as plain text. In order to generate them you will need a good templating library. I've had to use some clumsy and cumbersome templating libs in legacy projects so I appreciate the importance of picking a library that works for you. One experiment I like to do when evaluating a templating library is to try to make a template for serializing some data structure into a pretty formatted json. Json format has a few tricky rules, like requiring comma after all but the last property, escaping quotes in string values, proper indentation for pretty format etc. If a templating library makes writing json templates enjoyable you'll probably have no problems with source code.

One last trick that can simplify source code generation is running output of your CLI through standard formatters for the language you are generating. This doesn't work if there are competing formatting standards, or if community uses widely different formats. Example of this would be Java world where no two code bases look the same. On the other hand, Go programming language comes with prescribed gofmt formatter that code generation tools use. Having properly formatted source code becomes important when it comes to pull requests and similar situation that require diffing 2 files / versions.

Introspection

Another advanced use case for CLIs is source code analysis. This one is more complicated that source code generation. While generation can be implemented using text manipulation, in order to analyze source code you will generally need to tokenize it and build a syntax tree out of it. This is getting away from templating and into compiler theory.

Fortunately, most modern languages provide tools for introspection of their own source code. So, if you know you'll need to analyze Java code it might be a good idea to write your CLI in Java. You can probably find a library for parsing your language of choice, which you can reuse in your tool.

Problems with this approach arise if you need your tool to analyze multiple languages. One example of this is code editors and IDEs. A common solution for that type of apps is to use Language Server protocol to communicate to dedicated language server implementations. That way the application code is decoupled from language servers which can be implemented for each language. A more advanced example is source{d} engine, an application for source code analysis. Under the hood it is using Babelfish, a universal code parser that can produce syntax trees for various languages. Like LSP, it too has dedicated language drivers executing in separate Docker containers that communicate with a single app server over gRPC.

If your CLI requires analysis of source code from multiple languages you should probably use one of existing solutions. If features of LSP are enough for you that seems like the most widely adopted solution. If you need full flexibility of abstract syntax trees, then Babelfish might be a bit more complex, but more powerful solution.

Text-based UI

You might find that a simple loop of reading the arguments, processing them and outputting results is no longer enough for your use case. Processing can be complex, so you may need to update users on it's progress. You may need extra input from your users. Workflow you are implementing might be complex and require continuous interaction from users. At this point you may need to build a Text-based User Interface (TUI).

Input prompts are the most basic use case. Example of this is how npm init (and may other scaffolding tools) guide users through process of setting up a project. When designing such interactions you should allow for all prompts to be customized with CLI flags, so that command can execute without any prompts. Common pattern for doing this is to add a -y / --yes flag that will automatically accept default options. In doing this you will make your command usable by larger scripts for which user interaction is impractical.

Next up, there's the use case of informing users of the progress of CLI command execution. Modern dependency management tools (again npm is a good example) will display a live progress bar while downloading dependencies. Another example is HTTP load testing tool vegeta. It can dynamically output the progress of a stress test while its running. It also does something interesting: it allows you to pipe its output through a formatter tool to a dedicated plotting terminal application jplot. Jplot then renders live charts in the terminal. This is a good pattern to follow if you need live plotting and don't feel like re-implementing it yourself.

Lastly, there are full-blown TUI applications, starting with “simple” ones like htop and on to the likes of vim and emacs. If you are thinking of building similar TUI apps you should be able to find a framework in your language of choice that can help with laying out your application's UI elements. However, if you are expecting other developers' contribution to your application, it might be a better idea to go with something like a web app UI. That way you will have a larger pool of contributors to attract to your project.

What I did

In the CLI I built for work I implemented some code generation features. For project scaffolding I actually reused an existing parent project that all subsequent ones fork off of. Because of this my init command basically does a shallow git clone of the parent repository.

I also implemented an add commands for generating config and code required to expose a new API endpoints. Since I picked Go for my language I went with standard library's template package. I found it expressive enough to write all my templates and generate properly formatted json, Groovy and Kotlin code. It has just enough features for all my use cases, and not too many as to make it complicated to use. Much like Go language itself, using it was a zen-like experience.

I did not have any use for code analysis or TUI in that particular project. However, I've recently been playing around with termui, a terminal dashboard library also written in Go. It's easy enough to work with, but my use cases are not all that advanced either.

In conclusion

This blog post concludes the series on CLI development. You can find first post that deals with picking the technology and distribution stack here, and general CLI usability tips here. While the series is done, I might have some more thought on CLI development in the future. In case you are interested in this type of content, you can:

• Subscribe to this blog's RSS feed.
• Follow @antolius@qua.name in fediverse.