Monday 22 July 2013

Quick and dirty multiply raw file conversion

Multi-raw2dng-converter

I asume that you know all about installiation and usage of Magic Lanters raw video functionality. 

If you like me find your self with many raw files, using raw2dng manually can be pain in a**.  Here you have python 3.x script working and tested on Windows 8. Given the argument -d "file directory" it will look for any subfolders there and for each raw file in subfolder will create new folder and will run raw2dng on that file. The resulting dng will end up in folder named "ra_file_name" without RAW. The script utilizes as many new processes as there cores in CPU.

Note that this script is just quick and  dirty way to fix my problem. I hope it will be useful for other as well. Feel free to modify and extend.

Usage:
D:\dev\python multiConverter.py -d "F:\Images\2013\0722"

Output:
Successfully processed: F:\Images\2013\0722\1\M17-2243.RAW

Script:
#!/usr/bin/python
#searches for raw files and converts then into a directory with the 
#same name as file
import glob
import sys, getopt
import os, multiprocessing, queue, subprocess
from threading import Thread, Lock

FILE = 0
DIR = 1
RAW2DNG = r"C:path\to\raw2dng.exe"
def optionsGet(argv):
 inputdir = ""
 try:
  opts, args = getopt.getopt(argv,"hd:",["ddir="])
 except getopt.GetoptError:
  print('multiconverter.py -d ')
  sys.exit(2)
 for opt, arg in opts:
  if opt == "-h":
   print("multiconverter.py -d ")
   sys.exit()
  elif opt in ("-d", "--ddirectory"):
   inputdir = arg
 return inputdir

def getCPUs():
 # Windows
    try:
        res = int(os.environ['NUMBER_OF_PROCESSORS'])
        if res > 0:
            return res
    except (KeyError, ValueError):
        pass

def createFolder(name):
 pass
 

def getFileNames(name):
 return glob.glob(name + "\*.RAW")

def getsubdirs(inputdir):
 file_list = []
 dirs=os.listdir(r'%s' %inputdir)
 for d in dirs:
  tmp = r'%s\%s' %(inputdir,d)
  file_list.append(tmp)
 return file_list
  
def createJobs(dir_list, q):
 for d in dir_list:
  files = getFileNames(d)
  for f in files:
   q.put((f,d))
   
  
   
def deployWorkers(file_list, l, num_workers, func):
 jobs = multiprocessing.Queue()
 createJobs(file_list, jobs)
 workers = []
 for i in range(num_workers):
  jobs.put(None)
  tmp = multiprocessing.Process(target=func, args=(jobs,))
  tmp.start()
  workers.append(tmp)
 for worker in workers:
  worker.join()
  
def raw2dngWorker(queue):
 while True:
  job = queue.get()
  if job == None:
   break
  else:
   #enter right dir
   os.chdir(job[DIR])
   #create folder
   folder = job[FILE].split(".")[0]
   os.makedirs(folder)
   #change dir
   os.chdir(folder)
   proc = subprocess.Popen([RAW2DNG, job[FILE]] , 
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT, )
   stdout_value, stderr_value = proc.communicate()
   if not stderr_value:
    print("Successfully processed: %s" %job[FILE])
   else:
    print("ERROR %s " %stderr_value)
   
   
 
   
if __name__ == "__main__":
 inputdir = optionsGet(sys.argv[1:])
 cores = getCPUs()
 l = Lock()
 file_list = getsubdirs(inputdir)

 #start cores = workes
 deployWorkers(file_list, l, cores, raw2dngWorker) 


Cheers

3 comments:

  1. I wish this would work for me.. after deleting the 2 tags I tried to run the script, but it justs quits silently without doing anything.

    ReplyDelete
    Replies
    1. hum what OS you run on? I have been using it on weekly bases without any issues. As shown in example the "F:\Images\2013\0722" is a top directory where you have your files, it will look for sub-directories where files lives. If you have all files in just one directory not modified script will not work. See description "it will look for any subfolders there and for each raw file in subfolder will ..."
      I will modify script to work with files in current dir too.
      The reason it is made this way is because I usually need to move files from the card to the hard drive many times during the shooting. So I end up with directory structure
      0722/1/file1.raw
      0722/1/file2.raw
      0722/1/file3.raw
      0722/2/file1.raw
      0722/2/file2.raw
      0722/2/file3.raw
      0722/3/file1.raw
      0722/3/file2.raw
      0722/3/file3.raw

      Delete
  2. (i meant the 2 inputdirectory tags that got at the end of your script, which are obviously wrong)

    ReplyDelete