CLI Development; Part 3

Advanced use cases

In this part of CLI development series I'll go over some of the more advanced use cases. I've previously discussed general tips for making command line apps nice to use. If you missed that blog post you can find it here.

As use cases grow more complex it makes more and more sense to look for existing solutions and reuse / incorporate them into our own applications. That's why I'll devote more space in this post to highlighting existing projects, as opposed to talking about my own experiences.

Source code generation

One use case where CLIs excel is project scaffolding and code generation. For example of such apps you can take a look at Yeoman and its list of generators. Yeoman itself is oriented towards web development, although its most popular generator JHipster outputs Spring Boot applications. As a side note, if you ever find yourself with some spare time, checking out JHipster is a wonderful way to spend few hours. One more old school example of project scaffolding in Java world are Maven archetypes.

If you look at those examples you will quickly find that they provide rich set of features and customizations. One thing that most dedicated code generation tools have in common is a plugin system that allows users to define their own templates. If code generation is your primary use case, developing a plugin for an existing tool is a good idea. That approach will save you a lot of time and you'll end up with a more polished product.

On the other hand, there are CLIs that offer code generation as an addition to their core features. For example think of init commands in tools like npm or git. Extracting scaffolding features out of these tools and delegating it to dedicated code generations apps would be detrimental to user experience. If you find yourself in a similar situation you should implement code generation within your CLI instead.

Most popular approach to code generation is to treat source files as plain text. In order to generate them you will need a good templating library. I've had to use some clumsy and cumbersome templating libs in legacy projects so I appreciate the importance of picking a library that works for you. One experiment I like to do when evaluating a templating library is to try to make a template for serializing some data structure into a pretty formatted json. Json format has a few tricky rules, like requiring comma after all but the last property, escaping quotes in string values, proper indentation for pretty format etc. If a templating library makes writing json templates enjoyable you'll probably have no problems with source code.

One last trick that can simplify source code generation is running output of your CLI through standard formatters for the language you are generating. This doesn't work if there are competing formatting standards, or if community uses widely different formats. Example of this would be Java world where no two code bases look the same. On the other hand, Go programming language comes with prescribed gofmt formatter that code generation tools use. Having properly formatted source code becomes important when it comes to pull requests and similar situation that require diffing 2 files / versions.

Introspection

Another advanced use case for CLIs is source code analysis. This one is more complicated that source code generation. While generation can be implemented using text manipulation, in order to analyze source code you will generally need to tokenize it and build a syntax tree out of it. This is getting away from templating and into compiler theory.

Fortunately, most modern languages provide tools for introspection of their own source code. So, if you know you'll need to analyze Java code it might be a good idea to write your CLI in Java. You can probably find a library for parsing your language of choice, which you can reuse in your tool.

Problems with this approach arise if you need your tool to analyze multiple languages. One example of this is code editors and IDEs. A common solution for that type of apps is to use Language Server protocol to communicate to dedicated language server implementations. That way the application code is decoupled from language servers which can be implemented for each language. A more advanced example is source{d} engine, an application for source code analysis. Under the hood it is using Babelfish, a universal code parser that can produce syntax trees for various languages. Like LSP, it too has dedicated language drivers executing in separate Docker containers that communicate with a single app server over gRPC.

If your CLI requires analysis of source code from multiple languages you should probably use one of existing solutions. If features of LSP are enough for you that seems like the most widely adopted solution. If you need full flexibility of abstract syntax trees, then Babelfish might be a bit more complex, but more powerful solution.

Text-based UI

You might find that a simple loop of reading the arguments, processing them and outputting results is no longer enough for your use case. Processing can be complex, so you may need to update users on it's progress. You may need extra input from your users. Workflow you are implementing might be complex and require continuous interaction from users. At this point you may need to build a Text-based User Interface (TUI).

Input prompts are the most basic use case. Example of this is how npm init (and may other scaffolding tools) guide users through process of setting up a project. When designing such interactions you should allow for all prompts to be customized with CLI flags, so that command can execute without any prompts. Common pattern for doing this is to add a -y / --yes flag that will automatically accept default options. In doing this you will make your command usable by larger scripts for which user interaction is impractical.

Next up, there's the use case of informing users of the progress of CLI command execution. Modern dependency management tools (again npm is a good example) will display a live progress bar while downloading dependencies. Another example is HTTP load testing tool vegeta. It can dynamically output the progress of a stress test while its running. It also does something interesting: it allows you to pipe its output through a formatter tool to a dedicated plotting terminal application jplot. Jplot then renders live charts in the terminal. This is a good pattern to follow if you need live plotting and don't feel like re-implementing it yourself.

Lastly, there are full-blown TUI applications, starting with “simple” ones like htop and on to the likes of vim and emacs. If you are thinking of building similar TUI apps you should be able to find a framework in your language of choice that can help with laying out your application's UI elements. However, if you are expecting other developers' contribution to your application, it might be a better idea to go with something like a web app UI. That way you will have a larger pool of contributors to attract to your project.

What I did

In the CLI I built for work I implemented some code generation features. For project scaffolding I actually reused an existing parent project that all subsequent ones fork off of. Because of this my init command basically does a shallow git clone of the parent repository.

I also implemented an add commands for generating config and code required to expose a new API endpoints. Since I picked Go for my language I went with standard library's template package. I found it expressive enough to write all my templates and generate properly formatted json, Groovy and Kotlin code. It has just enough features for all my use cases, and not too many as to make it complicated to use. Much like Go language itself, using it was a zen-like experience.

I did not have any use for code analysis or TUI in that particular project. However, I've recently been playing around with termui, a terminal dashboard library also written in Go. It's easy enough to work with, but my use cases are not all that advanced either.

In conclusion

This blog post concludes the series on CLI development. You can find first post that deals with picking the technology and distribution stack here, and general CLI usability tips here. While the series is done, I might have some more thought on CLI development in the future. In case you are interested in this type of content, you can: