Mastering JQ: Part 1

Jun 30, 2020

This is the first part of an ongoing series on mastering jq. This series does not assume prior experience with jq, but it does assume basic fluency in shell programming.

jq is a valuable tool that every fast coder has in their tool chest. It contains depths of immense power. In part 1, we'll start off with the basics.

For each application of jq, we’ll lead off with an example that you can copy and paste into your shell to see how it works. The rest of the section discusses the application in more detail.

Pretty print json

echo '{"k1": [{"k2": [9]}]}' | jq '.'

One of the most valuable applications of jq is also the easiest to use: pretty printing json.

Pass the desired json on stdin to jq and it will print pretty print json on stdout. jq's pretty printing adds both shape and color to the data. Shape and color make the data much faster to read and debug as discussed in Debug With Your Eyeballs. One caveat is that jq will not render colors if jq's output is piped into another command or a file. For example:

echo '"string"' | jq '.'

Or if the json data is large, we can pipe it into less (with -R to render colors properly).

echo '"string"' | jq '.' | less -R

However, if we do this, we’ll notice there are no colors! To get colors back, we can use jq’s -C flag.

echo '"string"' | jq -C '.' | less -R

Select json data

echo '{"k1": [{"k2": [9]}]}' | jq '.k1 | .[0] | .k2 | .[0]'

We pretty print json data to make it easier to read. We’d like to go one step further and only see the relevant data. Can jq do this? Yes. jq can select json data (and pretty print it). To understand how to select the relevant data, we need to understand how jq works a little bit.
jq runs the following algorithm.

parse a json value from stdin and set it as the initial result
for each function, apply the function to the result, and set the output as the result for the next function.
The final result is pretty printed on stdout.

Let’s run through the example to see how jq performs its selection.

echo '{"k1": [{"k2": [9]}]}' | jq '.k1 | .[0] | .k2 | .[0]'

The initial result is:

{"k1": [{"k2": [9]}]}

Then jq applies the .k1 selector, after which the result is:

[{"k2": [9]}]

Then jq applies the .[0] selector, after which the result is:

{"k2": [9]}

Then jq applies the .k2 selector, after which the result is:

[9]

Then jq applies the .[0] selector, after which the result is:

which is the final result and therefore pretty printed on stdout as

Here are the selectors I find most useful:

is the identity selector, it returns the whole value. This is why jq '.' pretty prints the whole value passed in on stdin.

.key

(where key is any string)

If the result is

a json object and the key is present, this returns, for the object at "key", the value.
a json object and the key is not present, this returns null.
not an object, jq returns an error.

."key-with-special-characters"

Like .key, but it works even if the string contains special characters.

.[n]

(where n is a number)
If the result is:

a json array and the nth element is present, this returns the nth element.
a json array and the nth element is not present, this returns null.
not an array, jq returns an error.

.[n:m]

where n and m are numbers
Let l be the length of the result array (if applicable)
If the result is:

a json array and l >= m, the subarray is returned
a json array and n < l < m, the subarray .[n:l] is returned
a json array and l <= n, an empty array is returned
If the result is not an array, jq returns an error.

.[:m]

This is equivalent to .[0:m]

.[n:]

This is equivalent to .[n:l] where l is the length of the array.

Transform json data

echo '[{"n": "bob", "v": 100}, {"n": "jim", "v": 101}]' | jq 'map({name: .n, value: .v})'

Sometimes the data we want to see itself has suboptimal structure for viewing. We want to do more than select, we want to transform the structure. “Selectors” are special cases of “filters” and jq has a rich, powerful collection of filters to transform data. Most of the jq manual specifies different filters that can transform json data. I’ve highlighted a few filters I find most useful along with an example you can run in a shell. You can read about them and more in the jq manual.

Array filters

length

echo '[1, 2]' | jq 'length'

Gets the length of the array.

reverse

echo '[1, 2]' | jq 'reverse'

reverses an array

add

echo '[1, 2]' | jq 'add'

sums up elements of an array

map

echo '[{"k1": "v1"}, {"k1": "v2"}]' | jq 'map(.k1)'

Map is map from functional programming

map select

echo '[{"k1": "v1"}, {"k1": "v2"}]' | jq 'map(select(.k1 == "v1"))'

“map select” is filter from functional programming

flatten

echo '[[1], [2]]' | jq 'flatten'

Flatten is flatten from functional programming. It “flattens” a list of lists into a single list.

Array constructor

echo '[1, 2, 3]' | jq '[.[0], .[2]]'

By using [ ], we can make our own arrays manually.

Object filters

keys

echo '{"k1": "v1", "k2": "v2"}' | jq 'keys'

Returns an array of keys from an object. Especially useful when the object is very big and so the keys cannot be determined by just looking at the data.

to_entries

echo '{"k1": "v1", "k2": "v2"}' | jq 'to_entries'

to_entries turns a object into an array of key value pairs.

from_entries

echo '[{"key":"k1","value":"v1"},{"key":"k2","value":"v2"}]' | jq 'from_entries'

from_entries is the inverse of to_entries, it turns an array of key value pairs into a object.

Object Constructor

echo '[1, 2, 3]' | jq '{k1: .[0], k2: .[2]}'

By using { }, we can make our own objects.

Transform newline delimited data

seq 10 | jq -s 'add'

So far we’ve used jq for json data. What about other data formats? Can we use jq with them? Well, jq has built in support for newline delimited data, so let’s start with there.

To use jq with newline data, we need to understand how jq handles multiple results.

Multiple results

I lied. Earlier I said jq applies each filter to a single json result. Actually, there can be multiple json results. jq applies the filter to each result independently and passes the results to the next filter.

Reading multiple values

Multiple values can be passed into jq by delimiting the values by 1 or more whitespace, including newline. Technically, there are some cases where no delimiter is needed between values, but it doesn’t impact our discussion.

Pretty printing multiple results

For every final result, jq prints them to stdout, newline delimited.

To use jq with newline data, we also need to know about 3 jq flags and one jq filter.

jq -s aka "Slurp"

echo '[][]' | jq -s '.'

jq -s reads from stdin, multiple values combines them into a single array. In practice, we use it to combine newline delimited results into a single result.

jq -r aka "Raw string output"

Compare

echo '0' | jq '"f"'

which returns

"f"

echo '0' | jq -r '"f"'

which returns

The quoted f is a json string and the unquoted f is a “raw string”. jq -r pretty prints results of type string not as a quoted json string but as a raw string.

jq -R aka "Raw string input"

echo 'string' | jq -R '.'

jq -R interprets stdin as json strings delimited by newlines.

jq '.[]' aka "Antislurp"

echo '[1, 2, 3]' | jq '.[]'

The .[] filter takes an array result and splits it into multiple results.

With these, we're ready to handle newline delimited data.

Reading newline delimited data

If our input is newline delimited numbers, as we saw in the example, we use slurp:

seq 10 | jq -s '.'

If the data is newline delimited strings, we also need to use "raw string input". However, we cannot use slurp and raw string input together. If we do, jq parses the entire input as a single raw string. As such, we need to call jq twice, once to quote the strings and a second time to slurp them into a single array.

echo -e "string\nstring" | jq -R '.' | jq -s '.'

Writing newline delimited data

To output an array of numbers as newline delimited numbers, we use antislurp:

echo '0' | jq '[1, 2, 3] | .[]'

To output an array of strings as newline delimited strings, we also need to use "raw string output":

echo '0' | jq -r '["a", "b", "c"] | .[]'

Conclusion

This concludes the first step in mastering jq. So far we’ve seen how jq can:

pretty print json
select json data
transform json
transform newline data

All of these ways of using jq are discussed in the jq manual. They’re readily available and easy to find. In future installments, we’ll dig deeper and uncover the hidden powers of jq. Powers that are not discussed in the manual. If you don’t want to miss it, just click the subscribe button