On: Parsing JSON at the CLI: A Practical Introduction ...

[heleninboodler]

I needed this tutorial 6 months ago (and 6 months before that, and 6 months before that). :D Highly recommend looking at and maybe including "gron"[1] in this as a very nice complement to jq. It fills in some use cases in a very straightforward way that are pretty cumbersome in jq, such as finding a field deeply nested in an optional parent.

[1] https://github.com/tomnomnom/gron

[donut → heleninboodler]

Agreed, gron is a fantastic tool.

Earler today I was looking at some deeply nested structures which had leaf nodes with a field named "fileURL", and values that were mostly "https://" but some were "http://". I needed to see how many, etc.

cat file.json | gron | grep fileURL | grep http | grep -v https

... and presto, I had only four such nodes.

Would've been a ton more work to get there with just jq.

[Arnavion → donut]

Based on your description, it sounds like:

    <file.json jq '.. | .fileURL? | select(startswith("http://"))' -r

... would've done the job?

Or, if you can't remember `startswith`:

    <file.json jq '.. | .fileURL?' -r | grep '^http://'

[donut → Arnavion]

Nice and terse :-)

Right, the ".." part was hard to remember because it's something like:

  {
    "entries": [
      {
        "fields": {
          "fileURL": {
            "en-US": "https://..."
          },
          ...
        },
        ...
      },
      ...
    ]
  }

and as you can see the field fileURL is actually an object with another field en-US (with a hyphen) so the jq becomes something like this:

<file.json jq -r '.entries[] | .fields | select(.fileURL) | .fileURL["en-US"] | select(startswith("http://"))'

And later I had to do the same for other fields that ended in URL (websiteURL, etc.)

Anyway, gron made it simpler to get a quick summary of what I needed because it represents every node in the tree as a separate line that has the path in it, which is perfect for grep.

I still use jq more than gron :-)

[Arnavion → donut]

>Right, the ".." part was hard to remember because it's something like:

You misunderstand. The command I wrote is meant to be used as I wrote it. `..` is not a placeholder for you to replace.

https://stedolan.github.io/jq/manual/#RecursiveDescent%3a%2e...

>and as you can see the field fileURL is actually an object with another field en-US (with a hyphen) so the jq becomes something like this:

Sure, so then it's:

    <file.json jq '.. | select(.fileURL?).fileURL["en-US"] | select(startswith("http://"))' -r

[donut → Arnavion]

Oh, I had no idea ".." was an operator. That's a pretty cool feature, combined with the "?" to match optional objects.

Thanks for explaining!

[tejtm → heleninboodler]

not a jq expert but do a) get to explore random huge files sometimes as json. b) may care when the structure (but not the values) change.

so a jq utility[] that spits out all the jq paths for a json files helps in both cases.

[] https://github.com/TomConlin/json_to_paths

[sequoia → heleninboodler]

oh wow cool I didn't know about gron!! Is it alright if I add this comment to the bottom of my post?

[busfahrer → sequoia]

gron is great! And because the 80% use case of "gron -> grep -> gron -u to turn it back into JSON" is so common, I use these two convenience wrappers that make it easier. I called them grop because its' gron + grep:

https://github.com/busfahrer/grop

[heleninboodler → sequoia]

Of course, although I wouldn't be surprised if a jq expert comes along and gives an easy way to deal with deeply nested children of an optional parent, making my comment moot. :D

[sequoia → heleninboodler]

Ha, I never let that stop me! You can do some of the stuff in my post other ways, like using the `--jsonpath` switch in kubectl, but 1. I prefer learning jq rather than the idiosyncratic syntax/flag of each tool & 2. Let 1000 flowers bloom. I figure its OK if there's more than one way to do something!