Knowledge Base (Posts about json)

Python3 run command and convert stdout from csv to json

bgstack15 — Sun, 14 Mar 2021 13:17:18 GMT

Because apparently this is a non-trivial process to find on the Internet.

import os, sys, subprocess, json, csv

def get_json(username):
   result = []
   # https://stackoverflow.com/a/19706994
   # https://stackoverflow.com/a/33927495
   # https://stackoverflow.com/a/8880539
   cmd = "/path/to/csv-generator.sh"
   a = subprocess.Popen([cmd,username], stdout=subprocess.PIPE) # output is a CSV
   b = a.communicate()[0].decode('utf-8')
   c = csv.DictReader(b.splitlines())
   # fieldnames must be left undefined, to read first row as column titles
   for row in c:
      try:
         row.pop("") # try removing blank key, because CSV is proper format and ends with trailing comma
      except:
         pass
      result.append(row)
   return json.dumps(result)

if __name__ == "__main__":
   print(get_json("bgstack15"))

And in the future, use AD groups for access control instead of a custom database, devs...

Python json to csv

bgstack15 — Mon, 25 Jan 2021 14:18:45 GMT

I'm sure there are many tutorials for how to convert json arrays to csv. I wrote one that finds all the possible keys and then generates a csv. Check it out at my gitlab. The logic is broken up into several parts.

   # Learn keys at this level
   keys=[]
   for i in json_obj:
      for j in list(i.keys()):
         if j not in keys:
            keys.append(j)
            debugprint("Found key {0}".format(j),debug=debug)

And then I found the concatenation to a large string, and then a single write operation to a file, to be much more performant than writing to the file for each entry of each row (duh).

   x=0
   fullstring=""
   for i in json_obj:
      x += 1
      # only if there are actually contents of "all" do we print the headers
      if x == 1:
         for k in keys:
            fullstring += str(k) + ","
         fullstring += "\n"
      for k in keys:
         p = ""
         try:
            p = str(i[k]).replace(",",":")
         except:
            # no value for this key for this entry
            pass
         pp = "{0},".format(p)
         fullstring += pp
      fullstring += "\n"

   if fullstring != "":
      if csvfile == "stdout":
         print(fullstring)
      else:
         with open(csvfile,"a") as of:
            of.write(fullstring)

So you'll see that I also replace some commas with colons. I should probably have parameterized those, but I can leave that as an exercise to the reader. I even threw in a cli front-end to my library.

$ ./json-to-csv.py --help
usage: json-to-csv.py [-h] [-d [{0,1,2,3,4,5,6,7,8,9,10}]] [-i INFILE] [-o OUTFILE]

optional arguments:
  -h, --help            show this help message and exit
  -d [{0,1,2,3,4,5,6,7,8,9,10}], --debug [{0,1,2,3,4,5,6,7,8,9,10}]
                        Set debug level.
  -i INFILE, --infile INFILE
                        Json file to convert
  -o OUTFILE, --outfile OUTFILE
                        Csv file to output

Backstory

Some hospitals present their machine-readable data for pricing transparency as json. And some devices don't know how to read json, but a csv can be opened like a spreadsheet.

Pretty print json in python

bgstack15 — Fri, 10 Nov 2017 14:16:15 GMT

For python2 I wanted to show what variables are in use in a function, and I wanted to see it in a nicer format than a really long, single line.

import inspect, json
def function():
print json.dumps(locals(),indent=3,separators=(',',': '))

Bonus

To view what parameters were passed in to a function, add these.