stib's guff

cute Processing easter egg:

Now it's easy enough with the command line to get a whole lot of processes running in parallel, that's one of the great advantages of CLI software like ffmbc and its older sibling ffmpeg. With GUI applications however you're generally limited to whatever multi-processor support the software maker built in. So here's a trick that can help you when you have CPUs and RAM to spare, but not time.

Command Line, Meet the GUI

There's an OS X - only command-line programme called open that, you guessed it, opens stuff. Generally it works as if you'd double clicked things with your mouse, so if you type open foo.png it will open foo.png in whatever your default application for looking at pngs happens to be, or if you point it at /Applications/someApp.app/ it will recognise it as an application and run it (In case you didn't know, applications in OS X are actually folders).

Where the magic happens is one of open's command line options. Here's the list of options from the man page:

Now the important one for us is the -n option. This lets us override the usual mac behaviour and lets you open more than one instance of a programme. This is great if you want to run something like MPEG Streamclip which has a built in limit of 4 simultaneous encodes. Here's a screensot of it running 3 instances, doing 4 encodes each.

The CPU and RAM on this machine had power to spare - I could have even run another instance if I'd needed it. And my job took a third of the time. 12 cores. Love it.

So the script to get this going is simple. Open Terminal and type

open -n /Applications/MPEG\ Streamclip.app

rinse, repeat. Of course this works for other programmes as well, you just substitute /Applications/MPEG\ Streamclip with whatever the path to the application is.

I synch in FCP by matching the audio recorded by the 5D with the audio off the Zoom. It's a pain, but it's alot easier than synching on a Steenbeck. When I've got a match I modify the timecode of the video to match the audio, as the Zoom records with proper timecode, the 5D doesn't.

Now since editing h.264 footage in FCP is about as fun as falling off a bicycle into a barbed wire fence I make a proRes-422 production copy of all my 5D footage before I start editing. I keep my original camera footage in my archives, and if I want to be able to resurrect the project later without having to keep all my modified proRes movies I have to make sure that the origianal footage from the 5D matches the timecode of my production copies.

Luckily ffmbc lets me write timecode to files without any transcoding. So here's the script I use:

cameraFolder=/path/to/the/5D/footage/

productionFolder=/path/to/my/proRes/production/copies/

I was given a portable hard drive with a whole lot of video shot on a Sony AVCHD consumer camera, in the form of m2ts files. And now I'm faced with the task of turning them into something I can persuade FCP (7) to open. Log and transfer doesn't work (I'm assuming that whoever copied the camera files didn't do the right voodoo with the file structure) leaving a whole lot of orphaned files that FCP just refuses to open. Just to frustrate me, they play fine with VLC, so I can see them but I can't use them.

M2TS is the format used by blu-ray and some Sony cameras. It's an MPEG2 Transport Stream container, with h.264 encoded video inside. If you don't know what that means, it's a bit like getting a Betacam SP cassette case and putting an SD Card inside it. My first response is to break out MPEG Streamclip and batch them into ProRes. Gah! For once MPEG Streamclip lets me down. Episode, Compressor all fail too. Since VLC can play them I give the streaming / transcoding wizard from VLC a bash, but as usual it doesn't work. I've never yet had that thing work for transcoding - it looks like it's doing something, and takes a whole lot of CPU time to produce video files that won't play. Great media player though.

Turns out I shouldn't have mucked around with these GUI apps, I should have just gone straight to the command line. Using ffmbc (the 'broadcast' version of ffmpeg) I can convert them to quicktimes using an uncompressed codec (v210 in this case, I could have used qtrle - AKA Apple Animation codec - and saved some disk space, but v210 was four times faster). I want ProRes for my edit so I have to then recompress, but that's a simple job for MPEG Streamclip.

I had a bit of trouble working out what audio format Quicktime likes, turns out it's 16 bit signed, Big Endian. So just to remind my future self what CLI juju I need here's the magick for converting a folder of m2ts files:

for i in *.m2ts;
do ffmbc -i "$i" -vf yadif=0:-1 -vcodec v210 -acodec pcm_s16be "${i/m2ts/mov}" ;
done

The for loop is so it batches a whole folder of m2ts files. If you wanted to do just one you'd get rid of the first and third line and the do on the second line. and you'd replace "$i" with your input file and "${i/m2ts/mov}" with your output file. The  -vf yadif=0:-1 bit is for deinterlacing them with Yadif (the numbers mean 1 frame per two fileds and automatic field order), so if you don't want the output to be deinterlaced you could remove that bit.

I've only recently discovered the usefulness of Apple's Spotlight search tool. For some reason I'd never really clicked that it could be a very handy little launcher for applications and documents, until I noticed the Mac techo guy who came to plug all the wires in and adjust the tappets on my new 12 core machine at work <smirks>. But now I'm addicted - I've even mapped the Spotlight keyboard shortcut on to one of my mouse buttons and got rid of Butler.

So of course I couldn't be satisfied with just launching apps and dox, I want all my webs to appear in Spotlight too. After some googling I find that this isn't as easy as it would appear, especially if you don't want to use Safari.

Well here's what I came up with. A modification of this modification of a Perl Script that produces URL files. Why URLs? Well they're simple than webbokmarks, and since FireFox doesn't read the content of .webbookmark or .webloc filesbut relies on data in the resource fork, weblocs and webbokmarks didn't work anyway. Bad Firefox.

So I need a little helper app to read the url file and open FF at the right page. It's simple enought to do with a shell script or a bit of perl etc, but the OS won't let you assign these as the default app for the file type. So I used an AppleScript as a wrapper.

So if you want your Bookmarks to appear in Spotlight here's what you have to do.

first run this perl script:

#!/usr/bin/perl
use strict;
use warnings;
my $meta = "$ENV{HOME}/Library/Caches/Metadata/Firefox";
system("mkdir -p $meta");
system("rm $meta/*.url");
my $bookmarks = `find \~/Library/Application\\ Support/Firefox/Profiles/ -name 'bookmarks.html'`;
open (IN, $bookmarks);
while (<IN>){
    my $line = $_;
    if (($line =~ m|HREF="(.*?)".*>(.*?)</A>|) && ($1 !~ /^javascript\:/)){
   
        my $url = $1;
        $url =~ s/&/&amp;/g;
        my $name = $2;
        $name =~ s{[/:;\*\\\"\'\.\|]+}{-}g;
#         my $uuid = `uuidgen`;
#         $uuid =~ s/\n$//;
        print $line;
        open (OUT, ">$meta/$name.url");
        print OUT "[InternetShortcut]\nURL=$url\n";
        close OUT;   
    }
}

You can copy and paste it or download it here. To run, just open a terminal cd tot he directory you downloaded the script (usually cd ~/Downloads will get you there) and type perl ffbm.pl

If you want to open the links with Firefox or Chrome, read on:

FF on the mac doesn't handle .url files or .webloc or .webbokmark files properly. We need a little helper to read the files and kick the contents to Firefox / Chrome.

So I've created this applescript. Plonk it in your Applications folder (or wherever). This basically munges the .URL file to get the URL out of it, and uses your default web browser to open it. Here's the code:

on open theURLfile --called when the OS sends a URL file to the script
    openURLfile(theURLfile)
end open

openURLfile(choose file) --called when the user just runs this script by double clicking

on openURLfile(theURLfile) --the actual Hoo-Hah
    set URLfilePpath to POSIX path of theURLfile
    set theURL to (do shell script "cat  \"" & URLfilePpath & "\"|grep \"URL=\"|sed \"s/.*=//\"")
    do shell script "open " & theURL --the OSX open command opens whatever you give it according to your default application choice
end openURLfile

I'm applying to do the Peer 2 Peer University introduction to Javascript course, and one of the signup tasks is a blog post about this video, where Douglas Crockford gives us the background on Javascript, and describes the key concepts. So what have I learned from it?

Well I've found out about a whole lot of corporate infighting, between Sun and Netscape and Microsoft, which explains (I think) why Java is called Java (they thought up the name in a coffee shop) and why Java != HotJava (the browser) and Javascript. I've listened to it several times with and without coffee and my eyes still glaze over during this bit.

Doing some research of my own I discover that ECMA, sorry Ecma (the letters no longer stand for anything) is some sort of geek United Nations, and that the Ecma General Assembly approved ECMA-262 in June 1997. Sounds very momentous (Who are the General Assembly of Ecma? Do they have a complaints line I can call?). Anyway after all the dust has settled what's important is that JavaScript is a warts-and-all implementation of the ECMAScript language standard.

Fair enough, that's all kinda fascinating. But what about loose typing and lambda functions and prototypal inheritance? Now we're getting somewhere.

So Javascript is loosely typed. That means I believe that the type of a variable is determined internally, and also that any variable can be assigned any type of value. Its going to take some getting used to.

So as far as types go we have numbers, strings, booleans, Null and Undefined, and everything else which are Objects. Numbers and strings seem fairly straight forward, Null and Undefined are weird hippies. So on to Objects. There follows a bit of discussion about hashtables, which caused my eyes to start glazing over again. Let's ask Wikipedia:

In computer science, a hash table or hash map is a data structure that uses a hash function to map identifying values, known as keys (e.g., a person's name), to their associated values (e.g., their telephone number). Thus, a hash table implements an associative array. The hash function is used to transform the key into the index (the hash) of an array element (the slot or bucket) where the corresponding value is to be sought.

Oh, ok.. So objects are like phone books. Or buckets with slots. Hmm.

Let's move on to the syntax. I'm going to have to watch this a few more times to internalise the details, but basically it turns out that Javascript is syntactically similar to C. This is reassuring, somehow, even though I don't program in C. It's like finding that your new house is built out of bricks instead of Carbon nanotubes.

More to come on this I hope

After investigating several render managers, I decided the best approach for my setup (small render farm with 8 nodes running windows) was to write a script to produce shell scripts that could be run on each machine. The advantage of this is zero CPU, RAM or network overhead while rendering, no waiting for commands between jobs, and no need for a central machine to control the render process. The disadvantage is that it is completely unaware of the progress of the jobs, it doesn't do any load ballancing, and it can't cope with a problem in any of the nodes.

Basically what it does is to add the commands needed to start a render to the end of a shell script, which can then be executed by a render node. You can add jobs to the end of a render that is in progress, but you can't stop or edit it.

There are two LScripts, one adds the current scene to the render queue, and the other adds any saved scene to the queue. The script that it writes does keep a log, so the progress of the render can be monitored by humans.

The scripts may need tinkering with for your setup, and I can't guarantee that they are fit for any purpose whatsoever. Caveat Downloador. Free as in beer & speech yada yada. Have fun.

Download FarmHand here.

Interesting to compare how browsers deal with colour profiles. Using the test image on this page you can see how various browsers fare. If they render the colour profiles correctly the image should appear normal, if not you'll see differences between each quadrant.

This screenshot was taken on an intel mac running OS 10.6.4. The internet explorer windows are running under Parallels. 

Something's wrong wi' the new computator. More disgruntled folks here

(download)

Click here to download:

ethernet.mov (183 KB)

One of the positive side effects of Finder's lameness is that it can help you learn shell scripting.

Here's a program to batch rename things on a mac or linux (or any other *nix) machine. Run it in Terminal, obviously.

cd /directory/where/the/files/are

for file in *

do mv "$file" "${file/patternToReplace/ReplacementString}"

done

Ok, let's break it down

cd /directory/where/the/files/are

The first line changes the working directory to /directory/where/the/files/are. Change this to the path to your source directory. the path is the list of folders that contain it, all the way back to the root folder (which on mac is your system drive). You don't need to know this - see the red text below for how to avoid typing it all in.

As well as being case sensitive, this path has to have escape characters for spaces and any naughty characters in the file name - a path like
/Volumes/spud/my folder would be treated as two different things,
/Volumes/spud/my and folder.

To stop this from happening you need to "escape" the special characters and spaces. To escape them you put a back slash in front of them, so the folder above would be written as /Volumes/spud/my\ folder. The backslash before the space tells the script that it's still part of the folder name, not the start of a new word. You can also enclose the path in quotes like "/Volumes/spud/my folder/", but you can't use both - "/Volumes/spud/my\ folder" won't work. Other special characters include brackets, quotes and of course backslashes, which you escape just the same as spaces by putting a backslash in front of them like \\.

But don't be a'feared, there's an easier way: just type cd then drag the folder on to the terminal window and it will generate the path. Too easy. Bonus tip: if you want to look like one of the hacker dudes you see in the movies, there's another easy way. Type the first couple of letters in the directory and hit tab. The shell will automagicaly fill in the rest (provided there isn't more than one file that starts with those letters). So to type in /Volumes/spud/my\ folder you might actually type /V[tab]sp[tab]m[tab] and every time you hit [tab] the missing olumes/, ud and y\ folder will magickally appear. Woot! Now you're a l337 h4xx0r!

Ok, on with the show.

for file in *

The next line is the start of a loop - this is where it loops through every file in the directory. This is a for loop, the command for which takes the form for arg in list, where every time it goes through the loop arg takes on the value of the next variable in the list. To get it to loop through all the files in a directory we can either manually type in every file we want it to loop through, which would be dull, or we can use the magic asterisk (that's its technical name). The magic asterisk AKA * is a wildcard, it matches any file name. So writing for file in * in a directory that contains the files foo, bar, and spud is just the same as writing for file in foo bar spud.

You can make it more specific eg. for file in *.mov will only match files with a .mov extension, or *.* will match match any file with a dot "." in the name (a good way to skip folders), or foo_* would match every file starting with foo_, like foo_01, foo_02 foo_bar, etc.

file is the name of the argument or variable, and when we refer to it again we put the $ sign in front of it - the dollar sign is the character grumpy ol' Mr Bash (he's your shell for today) uses to denote a variable. Inside the loop $file will equate to the name of each file as it works through the list. So putting echo "$file" inside the loop would print out the list of files. you don't have to use the word "file", but i find it's helpful to name variables with what they represent.

do mv "$file" "${file/patternToReplace/ReplacementString}"

The next line uses the mv command, which is where the renaming takes place. It does parameter expansion on the name to do a find - and - replace on each file name that was selected in the previous line. PatternToReplace is the bit of the name you want to replace and ReplacementString is what you want to replace it with, eg. "${file/foo/bar}" will rename foo.mov to bar.mov or spudfoo_99.mov to spudbar_99.mov. If you want to replace something more than once in each file name (eg change foospudfoo to barspudbar put another forward slash before patternToReplace like this "${file//foo/bar}" The quotes are there for file names that have spaces in them). If you're timid you can replace the mv command on this line with echo, and it will print out the before and after names without actually doing the renaming. I do that if I'm doing anything tricky. The do at the front is part of the syntax for the for loop. I always want to type for x in y do on one line but that doesn't work, I don't understand why it is that way, it just is. You can use for x in y; do on one line, the semicolon is a synonym for the end of a line.

done

The last line is the marker for the end of the loop. from here the program goes back to the for command.

If you want fries with that

..you can add -v after the mv command as in mv -v etc This makes it verbose, so it will keep you informed about what it's doing.

More options include -i to make it interactive and ask you before overwriting files, or -n to skip any files that already exist in the target directory. This can be used to merge two directories like so:

cd /directory/where/the/files/are

mv -n * /path/to/the/target/folder/

One, er, four words of warning: There Is No Undo. Use cp to copy instead of move and then delete the originals afterwards if you're unsure. You can delete them using the rm command

Read more about such trickery here.