Friday, October 9, 2009

math, graphs, and process crap

This is process stuff that maybe ought to be in my .org journal, but I can use pictures here, and not there.

Free Image Hosting at

This is a graph of the production time officially noted for every SH video so far. Let's see what's of interest. (open full size if you actually want to follow along, don't just squint at the preview.)

The first and most striking thing is that there is a sine wave rather clearly apparent in the results, with a period/wavelength of approximately 9 videos. This is not perfectly consistent, but it's remarkable that a pattern like this is detectable at all. Even from someone (perhaps one of the few) who keeps stats on their process, it's really weird to see an emergent mathematical pattern in the data.

The second is that most SH videos finish within 20 hours of official start of work, and nearly everything is done within 40 hours of work. This helps explain the frantic pace of completion in the fall of 2002 and winter/spring of 2003, when I was job-hunting and GRE-prepping about 25 hours a week and spending about 25 hours on AMVage. The third is that there are quite a few videos, usually bunched together, running well over 60 hours, but this behavior disappears entirely after SH053. What happened after SH052 that resulted in this sort of spike disappearing?

Looking at where the spikes are, and what videos are represented, there is a simple formula at stake: videos that spike out were produced from digisubs, largely or entirely, and were all done over at least 10 episodes. That the spikes exist, and that those on the demos aren't even higher, is an artifact of changes in the clipping process and how records were kept.

SH videos have always been assembled out of clips rather than from regions of episodes selected in the editing environment. However, the process by which these clips were created has changed several times, and until SH093 when DVD rips came in at long last, some of the stages were dependent on whether I'd capped the video or was using someone else's encode. Up until SH052, the digisub process went like this:
- render source file to MPEG2 (usually; some stuff for demo 1 was done to MPEG1 because that was what I was using at the time) with TMPEG
- clip in MovieStar, producing full-quality MPEG2 clips and low-quality MPEG1 clips for drafting (videos from SH020 to SH092 were drafted with MPEG1 clips, due to problems synching SH019, then reassembled from the appropriate MPEG2 source for the final build)
After SH052, the process went like this:
- clip in Virtual Dub, processing the .avi to 640x480
- render clips to MPEG2 with TMPEG
- produce drafting versions in MovieStar

From the journal at the time of the change:
"I'm trying out a new clip-collection process on the current video as kind of an experiment. With my current hardware, I need to put in 17-18 hours of processing time per ep before I can even start thinking about getting clips. I'm trying now to make clips in VDub, run 'em through TMPEG to make the HQs, then make the LQ draft versions in my main editor. This is all attempting to save time, on the proposition that even if I get 3 minutes of usable source from an ep, that's 19 or so minutes that I'm not able to use in the video and shouldn't waste time working with."

This held up, and the precipitous drop after SH052 is notable; only two later videos -- one involving a lot of building from manga cuts, and the other thanks solely to some encoding mistakes -- even go above 50 hours, while before this it was virtually guaranteed that a "serious" video would require more than 60 hours. The sole reason for this was the extended render time on Keystone, which has been noted here, below, as a 500MHz K6-II with less than 200MB RAM. As noted, each episode represented between 10 and 18 hours of effectively dead time running the render, which all went into the production stats, since this was effectively the same step as capturing the source, which was what the process tracking was initially built on. When this is taken out by clipping in VDub, the resulting stats are a hell of a lot closer to the actual attention time given to the video; speed was improved along with tracking accuracy.

The demos (demo 2 and 3), incidentally, are where they are, despite each using 20-30 episodes' worth of source, because common production time was reckoned proportionately on each demo for the videos included. Each video has about four hours of unique production work, something that you can also pick up by looking back at demo 1; the rest of those 70-95 hours are its share of the rendering and clipping that produced the source pool. If that production time was counted in full on each video, the demos would look like each video was the end product of 250-350 hours of work, which is even less accurate as far as effort goes.

Thankfully, all this is in the past. The limitations of drafting, NTSC-TV framerate, and cuts no shorter than 15 frames are no longer a part of how I make AMVs. A good school in many ways, but one I'm glad to be out of.

No comments:

Post a Comment