jq - JSON Power For The Terminal

Do you find yourself sometimes writing small scripts just to introspect some JSON? Say we want to get some image from the Mapillary API, the URL for getting GeoJSON from an image search is http://api.mapillary.com/v1/im/search?min-lat=55.611&max-lat=55.613&min-lon=12.994&max-lon=12.995&max-results=3&geojson=true:

curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true
{"features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[12.99455273,55.61112242]},"properties":{"location":"Beijerskajen, Malmö","marker-color":"#707070","marker-size":"medium","image":"https://d1cuyjsrcm0gby.cloudfront.net/BjoCtrQEPlLbj-qM-CKt4Q/thumb-320.jpg","key":"BjoCtrQEPlLbj-qM-CKt4Q","compass_direction":"2","ca":90.8944244384766}},{"type":"Feature","geometry":{"type":"Point","coordinates":[12.994548972049,55.6111892340047]},"properties":{"location":"Beijerskajen, Malmö","marker-color":"#707070","marker-size":"medium","image":"https://d1cuyjsrcm0gby.cloudfront.net/XSPgbXcjLkIgOIfn-9C-ow/thumb-320.jpg","key":"XSPgbXcjLkIgOIfn-9C-ow","compass_direction":"7","ca":334.037878787879}},{"type":"Feature","geometry":{"type":"Point","coordinates":[12.994737,55.611425]},"properties":{"location":"Beijerskajen, Malmö","marker-color":"#707070","marker-size":"medium","image":"https://d1cuyjsrcm0gby.cloudfront.net/UsOg_TQgsuPfNJTOjwHmuA/thumb-320.jpg","key":"UsOg_TQgsuPfNJTOjwHmuA","compass_direction":"6","ca":275.6486}}],"type":"FeatureCollection"}

Pretty awkward to find anything here right?

I recently discovered jq, the most awesome tool to look at JSON fast. To install, just do the familiar

brew install jq

And we are ready to look at the JSON we pipe into jq, just pretty-formatting the output (for the prompt coloring and git branch magic - I'm using oh-my-zsh):

curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true | jq '.'

alt image with links

Nice colors 'n' all!

Nice, the GeoJSON is a map with a features array with maps in it. Let's count the length by piping the features array into a length filter, much along the lines of data flow frameworks, giving 3 as the result:

➜  mapillary_api (master) ✗) curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true | jq '.features | length' 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1076    0  1076    0     0   3368      0 --:--:-- --:--:-- --:--:--  3373
3

Now, let's drill down into the first images longitude with:

  mapillary_api (master) ✗) curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true | jq '.features[0].geometry.coordinates[0]'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1076    0  1076    0     0   3187      0 --:--:-- --:--:-- --:--:--  3183
12.99455273

However, jq can do almost arbitrary transformations on JSON, much like awk and sed for textfiles. Say we want to just list certain attributes like the key and ca of every image in a line-style fashion, we just apply some pipes and output literal strings with something like jq '.features[] | "key: \(.properties.key), ca: \(.properties.ca)"':

➜  mapillary_api (master) ✗) curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true | jq '.features[] | "key: \(.properties.key), ca: \(.properties.ca)"'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1076    0  1076    0     0   5853      0 --:--:-- --:--:-- --:--:--  5879
"key: BjoCtrQEPlLbj-qM-CKt4Q, ca: 90.8944244384766"
"key: XSPgbXcjLkIgOIfn-9C-ow, ca: 334.037878787879"
"key: UsOg_TQgsuPfNJTOjwHmuA, ca: 275.6486"

This certainly makes my days with the terminal more enjoyable when dealing with REST APIs like Mapillary, ElasticSearch, Neo4j, AWS and many others. How about you?

/peter

Tags for this post:
comments powered by Disqus