jq - JSON Power For The Terminal
Do you find yourself sometimes writing small scripts just to introspect some JSON? Say we want to get some image from the Mapillary API, the URL for getting GeoJSON from an image search
is http://api.mapillary.com/v1/im/search?min-lat=55.611&max-lat=55.613&min-lon=12.994&max-lon=12.995&max-results=3&geojson=true:
curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true
{"features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[12.99455273,55.61112242]},"properties":{"location":"Beijerskajen, Malmö","marker-color":"#707070","marker-size":"medium","image":"https://images.mapillary.com/BjoCtrQEPlLbj-qM-CKt4Q/thumb-320.jpg","key":"BjoCtrQEPlLbj-qM-CKt4Q","compass_direction":"2","ca":90.8944244384766}},{"type":"Feature","geometry":{"type":"Point","coordinates":[12.994548972049,55.6111892340047]},"properties":{"location":"Beijerskajen, Malmö","marker-color":"#707070","marker-size":"medium","image":"https://images.mapillary.com/XSPgbXcjLkIgOIfn-9C-ow/thumb-320.jpg","key":"XSPgbXcjLkIgOIfn-9C-ow","compass_direction":"7","ca":334.037878787879}},{"type":"Feature","geometry":{"type":"Point","coordinates":[12.994737,55.611425]},"properties":{"location":"Beijerskajen, Malmö","marker-color":"#707070","marker-size":"medium","image":"https://images.mapillary.com/UsOg_TQgsuPfNJTOjwHmuA/thumb-320.jpg","key":"UsOg_TQgsuPfNJTOjwHmuA","compass_direction":"6","ca":275.6486}}],"type":"FeatureCollection"}
Pretty awkward to find anything here right?
I recently discovered jq, the most awesome tool to look at JSON fast. To install, just do the familiar
brew install jq
And we are ready to look at the JSON we pipe into jq, just pretty-formatting the output (for the prompt coloring and git branch magic - I'm using oh-my-zsh):
curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true | jq '.'
Nice colors 'n' all!
Nice, the GeoJSON is a map with a features array with maps in it. Let's count the length by piping the features array into a length filter, much along the lines of
data flow frameworks, giving 3 as the result:
➜ mapillary_api (master) ✗) curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true | jq '.features | length'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1076 0 1076 0 0 3368 0 --:--:-- --:--:-- --:--:-- 3373
3
Now, let's drill down into the first images longitude with:
➜ mapillary_api (master) ✗) curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true | jq '.features[0].geometry.coordinates[0]'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1076 0 1076 0 0 3187 0 --:--:-- --:--:-- --:--:-- 3183
12.99455273
However, jq can do almost arbitrary transformations on JSON, much like awk and sed for textfiles. Say we want to just list certain attributes like the key and ca of every image in a line-style
fashion, we just apply some pipes and output literal strings with something like jq '.features[] | "key: \(.properties.key), ca: \(.properties.ca)"':
➜ mapillary_api (master) ✗) curl http://api.mapillary.com/v1/im/search\?min-lat\=55.611\&max-lat\=55.613\&min-lon\=12.994\&max-lon\=12.995\&max-results\=3\&geojson\=true | jq '.features[] | "key: \(.properties.key), ca: \(.properties.ca)"'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1076 0 1076 0 0 5853 0 --:--:-- --:--:-- --:--:-- 5879
"key: BjoCtrQEPlLbj-qM-CKt4Q, ca: 90.8944244384766"
"key: XSPgbXcjLkIgOIfn-9C-ow, ca: 334.037878787879"
"key: UsOg_TQgsuPfNJTOjwHmuA, ca: 275.6486"
This certainly makes my days with the terminal more enjoyable when dealing with REST APIs like Mapillary, ElasticSearch, Neo4j, AWS and many others. How about you?
/peter