Knowledge Base (Posts about csv)

TheMovieDb: Generate my Custom CSV for a show

bgstack15 — Wed, 26 Apr 2023 13:02:47 GMT

I've discussed before how I use a CSV to help apply metadata to media files. One of my problems with just manually munging the Wikipedia list of TV show episodes is that TheMovieDB organizes episodes a little differently. Sometimes a two-parter is listed as a single episode, and so on. All minor things, but my episode number can get off which affects my filenames and metadata.

So I decided to investigate using TheMovieDb to generate my CSV/spreadsheet. I found a great python library for tmdbv3api. I think there's multiple python libraries for this, but they all get to the same thing: the contents of the show details.

Here's my little wrapper script: files/2023/04/listings/eplib.py (Source)

	`#!/usr/bin/env python3`
	`# File: eplib.py`
	`# Location: /mnt/public/Support/Programs/themoviedb`
	`# Author: bgstack15`
	`# Startdate: 2023-04-19-4 13:51`
	`# Title: Episode CSV Generator`
	`# Purpose: facilitate my csv list of episodes of a TV show`
	`# History:`
	`# Usage:`
	`# source ~/venv1/bin/activate ; cd /mnt/public/Support/Programs/themoviedb/`
	`# python3`
	`# >>> import importlib, eplib`
	`# References:`
	`# 1. https://pypi.org/project/tmdbv3api/`
	`# 2. https://stackoverflow.com/questions/517923/what-is-the-best-way-to-remove-accents-normalize-in-a-python-unicode-string`
	`# Improve:`
	`# Dependencies:`
	`# One time:`
	`# python3 -m venv ~/venv1`
	`# source ~/venv1/bin/activate`
	`# pip3 install tmdbv3api`
	`from tmdbv3api import TMDb, TV, Season`
	`import os`
	`tmdb = TMDb()`
	`tmdb.language = "en"`
	`tmdb.debug = True`
	`tv = TV()`

	`def set_api_key(filename = None):`
	`""" Given a filename (or use hardcoded default file if None), load api key from file. """`
	`if filename is None:`
	`filename = "/mnt/public/Support/Programs/themoviedb/apikey"`
	`try:`
	`with open(filename,"r") as o:`
	`tmdb.api_key = o.read().rstrip('\n')`
	`except:`
	`return -1`
	`return 0`

	`### Ripped from ref 2`
	`from unicodedata import combining, normalize`
	`LATIN = "ä æ ǽ đ ð ƒ ħ ı ł ø ǿ ö œ ß ŧ ü "`
	`ASCII = "ae ae ae d d f h i l o o oe oe ss t ue"`
	`def remove_diacritics(s, outliers=str.maketrans(dict(zip(LATIN.split(), ASCII.split())))):`
	`return "".join(c for c in normalize("NFD", s.translate(outliers)) if not combining(c))`
	`### end ref 2`

	`# bgstack15 function`
	`def filename_safifier(s):`
	`""" Cruddy detox for myself. Should consider using my customized detoxrc """`
	`return remove_diacritics(s.replace("'","").replace('"',"").replace("...","").replace(",","").replace(":","_").replace("!","").replace(".","").replace("?","").replace("&","and").replace("/","_"))`

	`def get_tv_details(show_id):`
	`""" Uses web connection to get details of the item, either id number or show name """`
	`if show_id.isdigit():`
	`details = tv.details(show_id)`
	`else:`
	`show_temp = get_tv_search(show_id)`
	`try:`
	`details = tv.details(show_temp[0]["id"])`
	`except:`
	`print("Invalid TV show name. Found these names though:")`
	`for i in show_temp:`
	`print(f"id {i['id']}: {i['name']}")`
	`return -1`
	`return details`

	`def get_tv_search(show_id):`
	`""" Uses web connection to search for item, probably the name of the show """`
	`return tv.search(show_id)`

	`def get_episodes_for_show(show_id = "", include_specials = False):`
	`"""`
	`Given a show id (can be show_id integer, show name string, or a TV().details object), return useful list of episodes. If given show_id is a TV().details object, use that to avoid all the web calls. Example:`
	`a = TV().details(1855)`
	`get_episodes_for_show(a)`
	`OR`
	`a = get_tv_details("Star Trek Deep Space Nine")`
	`get_episodes_for_show(a)`
	`If include_specials, then also list episode titles and airdates for these special features but they still are not counted as absolute episode numbers.`
	`"""`
	`if type(show_id).__name__ == "AsObj":`
	`#if show_details is not None:`
	`show = show_id`
	`else:`
	`if show_id.isdigit():`
	`# IMPROVE: accept show name?`
	`show = get_tv_details(show_id)`
	`else:`
	`show_temp = get_tv_search(show_id)`
	`try:`
	`show = get_tv_details(show_temp[0]["id"])`
	`except:`
	`print("Invalid TV show name. Found these names though:")`
	`for i in show_temp:`
	`print(f"id {i['id']}: {i['name']}")`
	`return -1`
	`#print(show)`
	`name = f"{show['name']} ({show['first_air_date'][0:4]})"`
	`season = Season()`
	`seasons = show["seasons"]`
	`print(f"Name: {name}")`
	`show_id = show["id"]`
	`abs_epnum = 0`
	`abs_epnum_including_specials = 0`
	`for s in show["seasons"]:`
	`snum = s["season_number"]`
	`season_eps = season.details(show_id, snum).episodes`
	`for e in season_eps:`
	`abs_epnum_including_specials += 1`
	`if snum != 0:`
	`abs_epnum += 1`
	`if include_specials or snum != 0:`
	`sep = e["episode_number"]`
	`epname = e["name"].replace("(1)","Part 1").replace("(2)","Part 2").replace("(3)","Part 3")`
	`if '"' in epname:`
	`epname = "'" + epname + "'"`
	`if ',' in epname and not '"' in epname:`
	`epname = '"' + epname + '"'`
	`airdate = e["air_date"]`
	`filename = filename_safifier(f"s{snum:02d}e{sep:02d} - {epname}")`
	`#print(f"{snum},{abs_epnum},{sep},{epname},{airdate},")`
	`if abs_epnum_including_specials == 1:`
	`print("s,ep,sep,epname,airdate,filename,")`
	`print(f"{snum:02d},{abs_epnum:03d},{sep:02d},{epname},{airdate},{filename},")`
	`#print("%02d,%03d,%02d,%s,%s," % (snum, abs_epnum, sep, epname, airdate))`

	`# Run this regardless of __main__`
	`set_api_key(os.environ.get("TMDB_APIKEY"))`

So now I can get the output I want!

Import passwords to Chrome browser with csv file

bgstack15 — Tue, 24 Aug 2021 12:32:22 GMT

If your Chome web browser is locked down and unable to offer to save passwords, you can probably still load in passwords. You can use csv import and export features to examine how Chrome exports them, and then write your csv entry manually and import it. You might need to enable password import: Visit chrome://flags#PasswordImport and enable that flag. Export any current passwords you have to a csv, so you can examine the columns. For me, the columns were:

name,url,username,password
site.internal.example.com,https://site.internal.example.com/full/url/to/login/page,daUserName,Th3p@s5w*rd

The passwords are saved in file "Login Data" which in Windows is stored as %localappdata%\Google\Chrome\User Data\Default\Login Data. If Chrome is wiping your passwords when you close the browser, then you can mark the "Login Data" file as read-only before running Chrome. But to import a new password from csv, you need to run the program with the file as writeable, and then back up the file after importing the password. Then, close the browser and it will wipe the passwords in the main file. Then restore your backup, and mark it as read-only, and then you can run Chrome and close it at will, with the new passwords.

Python3 run command and convert stdout from csv to json

bgstack15 — Sun, 14 Mar 2021 13:17:18 GMT

Because apparently this is a non-trivial process to find on the Internet.

import os, sys, subprocess, json, csv

def get_json(username):
   result = []
   # https://stackoverflow.com/a/19706994
   # https://stackoverflow.com/a/33927495
   # https://stackoverflow.com/a/8880539
   cmd = "/path/to/csv-generator.sh"
   a = subprocess.Popen([cmd,username], stdout=subprocess.PIPE) # output is a CSV
   b = a.communicate()[0].decode('utf-8')
   c = csv.DictReader(b.splitlines())
   # fieldnames must be left undefined, to read first row as column titles
   for row in c:
      try:
         row.pop("") # try removing blank key, because CSV is proper format and ends with trailing comma
      except:
         pass
      result.append(row)
   return json.dumps(result)

if __name__ == "__main__":
   print(get_json("bgstack15"))

And in the future, use AD groups for access control instead of a custom database, devs...

Python json to csv

bgstack15 — Mon, 25 Jan 2021 14:18:45 GMT

I'm sure there are many tutorials for how to convert json arrays to csv. I wrote one that finds all the possible keys and then generates a csv. Check it out at my gitlab. The logic is broken up into several parts.

   # Learn keys at this level
   keys=[]
   for i in json_obj:
      for j in list(i.keys()):
         if j not in keys:
            keys.append(j)
            debugprint("Found key {0}".format(j),debug=debug)

And then I found the concatenation to a large string, and then a single write operation to a file, to be much more performant than writing to the file for each entry of each row (duh).

   x=0
   fullstring=""
   for i in json_obj:
      x += 1
      # only if there are actually contents of "all" do we print the headers
      if x == 1:
         for k in keys:
            fullstring += str(k) + ","
         fullstring += "\n"
      for k in keys:
         p = ""
         try:
            p = str(i[k]).replace(",",":")
         except:
            # no value for this key for this entry
            pass
         pp = "{0},".format(p)
         fullstring += pp
      fullstring += "\n"

   if fullstring != "":
      if csvfile == "stdout":
         print(fullstring)
      else:
         with open(csvfile,"a") as of:
            of.write(fullstring)

So you'll see that I also replace some commas with colons. I should probably have parameterized those, but I can leave that as an exercise to the reader. I even threw in a cli front-end to my library.

$ ./json-to-csv.py --help
usage: json-to-csv.py [-h] [-d [{0,1,2,3,4,5,6,7,8,9,10}]] [-i INFILE] [-o OUTFILE]

optional arguments:
  -h, --help            show this help message and exit
  -d [{0,1,2,3,4,5,6,7,8,9,10}], --debug [{0,1,2,3,4,5,6,7,8,9,10}]
                        Set debug level.
  -i INFILE, --infile INFILE
                        Json file to convert
  -o OUTFILE, --outfile OUTFILE
                        Csv file to output

Backstory

Some hospitals present their machine-readable data for pricing transparency as json. And some devices don't know how to read json, but a csv can be opened like a spreadsheet.

Windows: Map network drives from csv

bgstack15 — Tue, 15 Oct 2019 13:22:09 GMT

Even back in my olden days of using a non-free operating system, I was automating my tasks. The non-persistent sessions at the computer science lab required that I script my tasks. This is what I came up with for mapping network drives. You can set a csv file with the following contents:

driveletter,sharename,username,password,vollabel
SERVER,192.168.1.2
w,vol1\ben,,,W_svol1_ben
x,vol1,,,X_svol1
y,vol2,,,Y_svol2
z,vol5,,,Z_svol5
SERVER,hulk
s,hulk_ben,,,S_hulk_ben
t,smash,,,T_smash
SERVER,dax
u,dax_ben,,,U_dax_ben
v,science,,,V_science

So it's not a real csv file; it's got headers delineated by "SERVER" and a server name. The script will operate on the following lines, until the next "SERVER" line. It will mount the drive letter, from the following CIFS share, using a username and password or prompting if blank, and a pretty name. It will only mount those entries, if the SERVER actually is reachable. So it fails safely if you are on a different network than the requested server at the time. The script is available on my gitlab. And that example csv is also there. I eventually scrapped this script when I deployed an Active Directory domain and I used group policy to define a number of mounts to define.