veer66

veer66

(Posted on dev.to on 2025-01-19)

There are many ways to manipulate JSON. I reviewed a few rapid ways today, which are using a command line tool called jq and libraries that support JSONPath query language.

jq

Awk is a powerful domain-specific language for text processing, but it lacks built-in support for manipulating hierarchical data structures like trees. jq fills this gap by providing a tool for transforming JSON data. However, one potential drawback is that jq requires a separate installation, which may add complexity to my workflow.

So I write a shell script using jq to extract user's names and user's urls from a cURL response, and then output it in TSV format.

curl -s 'https://mstdn.in.th/api/v1/timelines/public?limit=10' | jq '.[].account | [.username, .url] | @tsv' -r

The result looks like this:

kuketzblog      https://social.tchncs.de/@kuketzblog
cats    https://social.goose.rodeo/@cats
AlJazeera       https://flipboard.com/@AlJazeera
TheHindu        https://flipboard.com/@TheHindu
GossiTheDog     https://cyberplace.social/@GossiTheDog
kuketzblog      https://social.tchncs.de/@kuketzblog
weeklyOSM       https://en.osm.town/@weeklyOSM
juanbellas      https://masto.es/@juanbellas
noborusudou     https://misskey.io/@noborusudou
jerryd  https://mastodon.social/@jerryd

With a TSV file, we can use Awk, sed, etc. to manipulate them as usual.

JSONPath

JSONPath, which was explained in RFC 9535, is supported by many libraries and applications, e.g. PostgreSQL. Still, I try to it in Python by the jsonpath_nq library.

from jsonpath_ng import parse
import requests


res = requests.get("https://mstdn.in.th/api/v1/timelines/public?limit=10")

compiled_path = parse("$[*].account")


for matched_node in compiled_path.find(res.json()):
    print(matched_node.value["username"] + "\t" + matched_node.value["url"])

It gave the same result to the shell script above. The code is a bit longer than the shell script. However, it can be integrated with many Python libraries.

Python is a fantastic language, but in specific situations, Awk can offer significant advantages, particularly in terms of portability, longevity, conciseness, and interoperability.

While Python scripts are generally portable, they may not always run seamlessly on popular Docker base images like Debian and Alpine. In contrast, Awk scripts are often readily available and executable within these environments.

Although Python syntax is relatively stable, its lifespan is shorter compared to Awk. For example, the print 10 syntax from the early 2000s is no longer valid in modern Python. However, Awk scripts from the 1980s can still be executed in current environments.

Python is known for its conciseness, especially when compared to languages like Java. However, when it comes to text processing and working within shell pipelines, Awk often provides more concise solutions. For instance, extracting text blocks between “REPORT” and “END” can be achieved with a single line in Awk: /REPORT/,/END/ { print }. Achieving the same result in Python typically involves more lines of code, including handling file input and pattern matching.

While Python can be embedded within shell scripts like Bash, aligning the indentation of multiline Python code with the surrounding shell script can often break the Python syntax. Awk, on the other hand, is less sensitive to indentation, making it easier to integrate into shell scripts.

Although different Awk implementations (such as Busybox Awk and GNU Awk) may have minor variations, Awk generally offers advantages over Python in the situations mentioned above.

I usually run this shell script after installing SBCL because to develop a program in Common Lisp practically, I usually need libraries. Thus, this script install Quicklisp and Ultralisp as a package manager and a package repository, respectively. Moreover, I set working directory for my Common Lisp projects to Develop in my home directory because when I put them in quicklisp/local-projects, I usually forget to backup or even forget where that the projects exist.

#!/bin/bash

# My working directory is $HOME/Develop. You probably want to change it.

rm -rf ~/quicklisp
rm quicklisp.lisp
wget https://beta.quicklisp.org/quicklisp.lisp
sbcl --load quicklisp.lisp \
        --eval '(quicklisp-quickstart:install)' \
        --eval '(ql-util:without-prompting (ql:add-to-init-file))' \
        --quit
rm quicklisp.lisp

sbcl --eval '(ql-dist:install-dist "http://dist.ultralisp.org/" :prompt nil)' --quit
if [ -e ~/.config/common-lisp ]; then
    cp -rp ~/.config/common-lisp ~/.config/common-lisp.bak-$(date -I)-$$
fi
mkdir -p ~/.config/common-lisp

cat <<EOF > ~/.config/common-lisp/source-registry.conf
(:source-registry
     (:tree (:home "Develop"))
     :inherit-configuration)
EOF

I wonder if I can use pyenv and pipenv on Fedora Workstation 40 although I don't use these utilities in my personal projects. And the answer is yes.

The steps are as follow:

Install dependencies

sudo dnf builddep python3

Install pyenv

curl https://pyenv.run | bash

I know that you don't like running Bash script immediately from cURL.

Modify .bashrc

Pyenv normally told you to append these lines to .bashrc, and the restart your terminal.

export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

# Restart your shell for the changes to take effect.

# Load pyenv-virtualenv automatically by adding
# the following to ~/.bashrc:

eval "$(pyenv virtualenv-init -)"

Install Python via Pyenv

pyenv install 3.10 # You can choose other versions

I know that we can install Python using configure and make like many other packages, but you can use pyenv as well.

Set default python

pyenv global 3.10 # or other version

And then restart your terminal.

And finally, install pipenv

pip install pipenv

My frequent mistake in Go is overusing pointers, like this unrealistic example below:

type BBox struct {
	X1 float64
	Y1 float64
	X2 float64
	Y2 float64
}

func ShowWidth(b *BBox) {
	w := math.Abs(b.X2 - b.X1)
	fmt.Println(w)
}

func main() {
	b1 := BBox{X1: 10.1, Y1: 100.2, X2: 1024.4, Y2: 4096.188888}
	b2 := BBox{X1: 10.1, Y1: 100.2, X2: 2024.4, Y2: 4096.188888}
	b3 := BBox{X1: 10.1, Y1: 100.2, X2: 3024.4, Y2: 4096.188888}
	ShowWidth(&b1)
	ShowWidth(&b2)
	ShowWidth(&b3)
}

I pass a pointer of BBox to ShowWidth, which according to @meeusdylan's post, it slows down my program because the garbage collector has to determine if a BBox must be in stack or heap.

In the alternative code below, I don't use pointer.

func ShowWidth(b BBox) {
	w := math.Abs(b.X2 - b.X1)
	fmt.Println(w)
}

func main() {
	b1 := BBox{X1: 10.1, Y1: 100.2, X2: 1024.4, Y2: 4096.188888}
	b2 := BBox{X1: 10.1, Y1: 100.2, X2: 2024.4, Y2: 4096.188888}
	b3 := BBox{X1: 10.1, Y1: 100.2, X2: 3024.4, Y2: 4096.188888}
	ShowWidth(b1)
	ShowWidth(b2)
	ShowWidth(b3)
}

I worried that my program will copy the entire BBox every time ShowWidth is called. So, I checked the generated asssembly code. It looks like this:

	ShowWidth(b1)
  0x48098e		f20f10059ab60300	MOVSD_XMM $f64.4024333333333333(SB), X0	
  0x480996		f20f100d9ab60300	MOVSD_XMM $f64.40590ccccccccccd(SB), X1	
  0x48099e		f20f10159ab60300	MOVSD_XMM $f64.409001999999999a(SB), X2	
  0x4809a6		f20f101daab60300	MOVSD_XMM $f64.40b000305af6c69b(SB), X3	
  0x4809ae		e82dffffff		CALL main.ShowWidth(SB)			

So, what I worried was true. MOVSD_XMM is for copying value from a member of a BBox in memory to a register one-by-one. You may see MOVSD_XMM was called 4 times per each ShowWidth call.

I didn't measure which one is faster or slower. I've heard that Skymont support loads per cycle. And, I wish they meant loading float64 using MOVSD_XMM as well. So, copying entire BBox is hopefully fast. And, at least, as far as I have been told, a BBox will definitely remain in stack without a need of checking by the GC.

Moreover, passing by value seems to comply to Go community better than pointer. So it will look familiar, and everyone will be happy to see passing by value.

My plan is avoiding pointer by default, and I will use it only when I have to. About performance, I think I may have to benchmark before using a pointer. Or if the speed is acceptable, I won't optimize.

I worked on a TODO code assignment for showing off my skills, and more importantly, showing my weak points. I coded in Golang and Masterminds/squirrel. Later, I ported only the SQL generation part to Clojure to compare and discuss why I prefer Clojure, which I have usually been asked about or even met with opposition for. I will discuss function by function and type by type. The first function is makeStatement.

func (repo *TodoRepoPg) makeStatement(orders []entity.Order, filters []entity.Filter) (string, []any, error) {
	builder := repo.Builder.Select("id, title, description, created, image, status")
	if err := buildOrders(&builder, orders); err != nil {
		return "", nil, err
	}
	if err := buildFilters(&builder, filters); err != nil {
		return "", nil, err
	}
	return builder.From("task").ToSql()
}

The makeStatement function's name clearly indicates it utilizes the builder pattern. However, to improve readability and avoid cluttering the function with too many details, it delegates order and filter information building to separate functions: buildOrders and buildFilters. Next one is the make-statement function in Clojure with HoneySQL.

(defn make-statement [orders filters]
  (sql/format (merge {:select [:id :description :status]
                      :from [:task]}
                     (filters->map filters)
                     (orders->map orders))))

In Clojure version, the main difference is that filters->map and orders->map are pure functions, which won't mutate or change their inputs like buildOrders and buildFilters do with the builder in Golang. The next one I will show contract or type or spec.

const (
	ID = iota
	Title
	Description
	Date
	Status
)

const (
	ASC = iota
	DESC
)

type Order struct {
	Field        int
	SortingOrder int
}

type Filter struct {
	Field int
	Value string
}

In Golang, to complement function definitions, I define custom types for conveying order and filter information. While using strings for this purpose is also acceptable, I prefer using types to leverage Go's static analysis and prevent typos.

(s/def :db1/orders (s/coll-of (s/tuple #{:title :created :status} #{:+ :-})))
(s/def :db1/filters (s/coll-of (s/tuple #{:title :description} any?)))

On the other hand, in Clojure, I defined similar contracts using Clojure Spec. Here, the information about orders and filters being collections of tuples resides within the Spec definition itself, unlike the separate function definitions in Golang.

func buildOrders(builder *squirrel.SelectBuilder, orders []entity.Order) error {
	for _, order := range orders {
		var fieldName string
		switch order.Field {
		case entity.Title:
			fieldName = "title"
		case entity.Date:
			fieldName = "created"
		case entity.Status:
			fieldName = "status"
		default:
			return fmt.Errorf("invalid field: %d", order.Field)
		}
		var sortOrder string
		switch order.SortingOrder {
		case entity.ASC:
			sortOrder = "ASC"
		case entity.DESC:
			sortOrder = "DESC"
		default:
			return fmt.Errorf("invalid sorting order: %d", order.SortingOrder)
		}
		orderExpr := fieldName + " " + sortOrder
		*builder = builder.OrderBy(orderExpr)
	}
	return nil
}

buildOrders looks very familiar. It reminds me of Pascal, which I learned 30 years ago. This suggests that the code utilizes a well-established approach, making it understandable to most programmers even without prior Go experience. However, I've identified potential code duplication between the type definition and the switch-case within this function.

(defn orders->map [orders] 
  (when-not (s/valid? :db1/orders orders)
    (throw (ex-info "Invalid input orders" (s/explain-data :db1/orders orders))))

  (->> orders
       (mapv #(let [[field order-dir] %] 
                [field (case order-dir
                         :+ :asc
                         :- :desc)]))
       (array-map :order-by)))

The Clojure function orders->map might have surprised my younger self from 30 years ago. However, it leverages Clojure Spec to its full potential. Spec validates the input to the function, and provide clear explanations when validation fails. Furthermore, orders->map is a pure function, meaning it doesn't modify its input data. Both the input and output data leverage Clojure's persistent maps, a fundamental data structure known for immutability. Therefore, unit testing for the orders->map function is relatively straightforward. I have no idea how to write a unit test for buildOrders in Go.

(deftest generate-orders-maps
  (is (= {:order-by []}
         (orders->map [])))
  (is (= {:order-by [[:title :desc]]}
         (orders->map [[:title :-]])))
  (is (= {:order-by [[:status :asc]]}
         (orders->map [[:status :+]])))
  (is (thrown-with-msg? Exception 
                        #"Invalid input orders"
                        (orders->map [[:id :+]]))))

In conclusion, Go's main advantage lies in its familiarity for programmers from various languages like Pascal, Java, JavaScript, Python, and C. This familiarity extends to the builder pattern, which offers the additional benefit of auto-completion in IDEs and smart editors. On the other hand, Clojure and HoneySQL emphasize using data structures, especially persistent maps, for building queries.

While auto-completion is less important for Clojure programmers who are comfortable manipulating basic data structures, Clojure Spec offers significant advantages in data validation.

Spec can explain what happens when data fails to meet the requirements, promoting better error handling and adherence to the open-closed principle (where code can be extended without modifying existing functionality). Additionally, Clojure Spec is not part of the function definition itself, allowing for greater flexibility and potential separation of concerns.

More importantly, writing unit tests in Clojure with HoneySQL is significantly more efficient. Because orders->map is based on persistent data structures, it avoids modifying the input data. This immutability, along with the ease of comparing maps, makes them ideal for testing.

I struggled to fix bugs in my custom bilingual alignment tree. Unlike standard trees from libraries like React or Tcl/Tk, this tree has unique data manipulation requirements. Most of the bugs stemmed from these data handling complexities, not from the visualization itself. This is just one example of the challenges I faced. I encountered similar issues repeatedly, for instance, when building directed acyclic graphs (DAGs) for word breakers.

I've applied various programming techniques I've learned, but unfortunately, none of them seem to solve my problem. These techniques included tracking my bathroom habits (yes, you read that right!), coding in Ruby and Python with OOP, using static checking in Java, C++, and Rust, and applying a functional programming approach like Haskell's function coloring. Surprisingly, the answer came when I started working with Lisp.

Non-destructive data transformation

I initially coded in LOGO and Racket (DrScheme), both members of the Lisp family, but these experiences were limited to toy problems. As a result, I didn't grasp the true benefit of Lisp until I revisited it by re-implementing a part of my word breaker in Racket. Notably, Racket is another Lisp dialect that heavily relies on singly-linked lists, the default data structure in Lisp. Unlike array-based lists common in other languages, singly-linked lists manipulated with the CONS function in Lisp avoid destructing existing data. This means constructing a new list based on an existing one doesn't require copying the entire original list, making it ideal for frequent operations involving large datasets. CONS itself operates in constant time, further enhancing efficiency. With Lisp, I can focus on using CONS for data manipulation without concerning on less important things like class design and function coloring. Additionally, since Lisp avoids data destruction, I can leverage the interactive environment conveniently without reloading data from files. While I acknowledge some past frustrations, using Lisp has finally allowed me to achieve what I've been striving for for years. Furthermore, adopting this functional programming approach can be beneficial in other languages, although limitations exist. These limitations will be discussed in the section on building collective team value through programming language choices.

Sustainability

I've been coding for a long time, starting with Python 1.6. As Python evolved through versions 2.X and now 3.X, I've seen the benefits of new features alongside the challenges of breaking changes. While Python's rapid development is great for startups, it can make long-term maintenance difficult for projects with strict stability requirements, such as those for governments or foundations.

Similar challenges exist in other languages. While Java boasts strong backward compatibility, the vast developer landscape can lead to codebases with varying levels of modernity (pre-generics, pre-Java 8, etc.), necessitating additional maintenance efforts. Rust, while powerful, shares Python's characteristic of rapid evolution.

The Stability of Common Lisp

Like other languages, Lisp has evolved over time. However, Common Lisp, the standardized dialect established in 1994 (with Lisp's roots dating back to 1959), offers a key advantage: stability. Code written in Common Lisp is less susceptible to breaking changes due to language updates. Additionally, powerful macros allow for the creation of new syntactic constructs at the library level, without modifying the core language specification that impacts everyone.

Furthermore, my observations of Common Lisp projects on GitHub and elsewhere reveal a remarkable consistency in coding style, even when developers aren't actively collaborating. This suggests a strong community culture that prioritizes code clarity and maintainability.

Building collective team value through programming language choices

Since most of the problems I encounter seem universal, I believe adopting a Lisp-style coding approach could benefit the entire team. However, a complete language switch might not be realistic for many developers and organizations. As a compromise, I attempted to implement Lisp-style coding in Java, but this proved unsuccessful. While elements like parentheses, CAR, CDR, and macros are easy to identify and often the target of complaints, they're not the main obstacles.

However, an experienced programmer once mentioned the difficulty of building complex programs in LOGO despite its easy start. This suggests that the core challenge might lie in the Lisp programming paradigm itself. Introducing Lisp concepts into Java often leads to endless debates rather than progress, even with extensive explanations. However, for team members who grasp the Lisp paradigm, overcoming the hurdle of parentheses becomes a less significant issue.

Searching for how to accomplish something in Java won't yield results written in Lisp style, which can lead to frustration and confusion for developers wondering if their approach is incorrect. In contrast, searching for the same task in Lisp will provide solutions that leverage proper Lisp idioms. This continuous cycle of coding and searching in Lisp builds the common value of the team.

Clojure

While Common Lisp offers a rich ecosystem for web development (front-end, back-end), and mobile apps, some developers may find its toolset less extensive compared to popular languages like JavaScript, Go, or Java. Clojure, a dialect of Lisp, addresses this by seamlessly integrating with Java/Scala/Kotlin libraries. ClojureScript provides a similar bridge to the JavaScript world, and ClojureDart is under development for Dart environments. Beyond its powerful concurrency features and data structures, Clojure and its sub-dialects make Lisp a viable option for building modern applications. A prime example is Nubank, a major online bank with over 100 million customers, which demonstrates the power of Clojure for large-scale applications.

Static type analysis excels in error checking, autocomplete, and native code generation. Conversely, dynamic typing shines in interactive environments like Jupyter Notebook, where type inference can struggle due to the unpredictable nature of future code. Languages can leverage both.

It seems managing singly-linked lists with shared sub-lists using RAII can be tricky. While I experimented with Rust's reference counting, it led to stack overflows for lists above 300,000 elements. Memory arenas appear more suitable, but effectively handling arena lifetimes in nested lists with varying types is challenging.

You can check my library at the repository at Codeberg. Perhaps, you want to fork it, or even take over the project.

Sometimes I don't aware that I use different data structures.

  • [1,2,3] in Rust is an array.
  • [1,2,3] in Elixir is a singly-link list.
  • [1,2,3] in PHP is an ordered hash map.
  • [1,2,3] in Clojure is a hash array mapped trie.