Digital is Over There! It’s Only a Matter of Sampling!

Bruce Lawton made me aware of an article in the New York Times that I found highly annoying.  It was highly annoying because it was inaccurate.  It reflects the complete misunderstanding of what “digital” means in the media and public.  In short, the public and media seem to believe this:

“Digital imaging processes are a modern miracle and are a complete replacement and upgrade from older technologies.  All digital images are perfect by their nature and will never degrade or become outdated.”

This is simply not true.  I hate to burst your bubble.  A closer summation would be this:

“Digital imaging is a miraculous tool that allows us to do things that were previously impossible to accomplish.  They can produce very high quality, not perfect, reproductions of their source images.  Their biggest drawback is that they become outdated quickly and most digital storage devices have short shelf lives.”

Now, once again, I’ll draw criticism from the masses: “You hate anything digital!  You’re a luddite!  You’re clinging to an outdated technology like film!  Get with the modern program!”

Once again, this is not true.  I use digital imaging all the time.  I think it’s great.  I did digital restorations for the Buster Keaton picture Seven Chances.  I am doing a digital restoration on King of the Kongo.  But I still believe in film.  Film doesn’t get computer viruses, hard drive crashes, or incompatible software upgrades.

I have film, actual film stock, manufactured in 1926 that is still projectable in modern projectors and plays fine.  I have digital images from 1991, carefully saved and copied,  that are incompatible with any modern program.

What would you think of a library that had a book from 1991 that you couldn’t read anymore?  Not because it was damaged in some way, but rather because they couldn’t figure out how to open it. You’d say they were crazy.  You’d be right.

I’m going to refute the New York Times article point by point, but first I have to lay out some ground work.  Fear not, technophobes. I’ll try to make it as clear as possible and minimize all the math.  It really is pretty simple, but for some reason, people want to believe in the miracle part of it instead of the truth.

In the early 1980s, Disney made the first real computer feature.  It took years to complete, but it was called Tron, released in 1982.  Tron was made with a bank of computers each with less computing power than your iPhone.  Your old iPhone.  Yeah, that slow one.

Tron is not notable for many dramatic triumphs (after all, it’s basically The Wizard of Oz set inside a computer), but for cinema, it was a real breakthrough.  Disney experimented with various resolutions.  Now, before you get all paranoid about a scary word like resolutions, let me explain.  It simply means how many pixels (little squares, like the ones you see in the image above) are used in the image.

Higher resolution = more pixels = smaller squares = sharper image.  In television, this is also measured in lines, which is the number of horizontal lines in the TV picture.  You know how people keep trying to sell you 1080p HDTV?  Well, standard definition was 525 lines, and HDTV is 1080.  Again, more lines = more pixels = sharper image.  See?  Simple!

Disney knew that they would have to output their computer graphics to 35mm film in some way.  There was no digital projection at the time.  They were very concerned about “stair-stepping.”  This is an effect also called aliasing.  Don’t be scared.  Look at the picture above.  You notice that it’s made of little squares?  Omar Sharif’s collar isn’t a collar, but it’s a jagged set of white lines.  You went to plot something that was supposed to be a line and you ended up with a jagged representation instead.  It’s aliased because the thing you tried to plot isn’t what you got!

Disney’s people discovered that they could see aliasing on most images until they put the resolution at 4000 lines.  This has been the “gold standard” of digital imaging for years.  Well, almost.  Tron had a limited color palette because of the software and hardware of the time.  This made jagged lines easier to spot.  As we were able to represent more colors and shades, we discovered that we could drop the resolution to 2000 lines, and it still looked pretty good… just a little blurry to some people. Remember, this is for material generated by the computer, not something scanned from an outside source.

In engineering parlance, 4000 lines = 4K, 2000 lines = 2K, and HDTV at 1080 lines makes almost exactly 1K.

I have to introduce one last concept.  It’s called the Nyquist Sampling Theorem.  I know, it’s an engineer thing.  Nyquist is a law of digital sampling.  It says that if you are scanning an analog signal (like a piece of film), the minimum rate you can use, so that you get no significant loss of data, is twice the number of the highest frequency in the source.

Oh, no.  The mathophobes are dying now.  Please don’t.  That simply means if you’re scanning a 4K image, you need to scan it at 8K or else you’re get a picture blurrier than it should be.  For a 2K image, you scan at 4K.

Now, we can tackle this article.  Take a deep breath.

Error 1:

“(Lawrence of Arabia was shot in 65 millimeter — nearly twice the width of a 35-millimeter frame — so its negative had to be scanned in 8K, creating 8,192 pixels across each line. But it is still referred to as a 4K scan because it has the same density of pixels, the same resolution across 65 millimeters that 4K has across 35 millimeters.)”

This is a very poor way of explaining the concept.  They’re saying that this means they’re scanning more lines because the negative is bigger, not because they’re scanning more lines per inch of film.

And, guess what?  What we’re seeing here, by Nyquist, through Disney’s research, shows that they’re undersampling (blurring) the negative.  Now, I don’t blame them, and it’s probably “good enough,” and very expensive to do more, but let’s start on the right playing field.

Errors 2-3:

“When Lawrence was last restored, in 1988, some of these flaws could be disguised by ‘wetgate printing,’ a process of dousing the print in a special solution. But the new restoration has no prints. The film’s digital data are stored on a hard drive, about the size of an old videocassette, which is inserted into a 4K digital projector. In short, the problems would now have to be fixed.”

Wetgate printing is still used.  It’s simple enough.  You take the negative (not the print), and soak it gently in a fluid (some archives use dry cleaning fluid), that fills in the scratches on the clear film base.  That fluid evaporates by the time the film hits the takeup reel.  Similar processes can be used in scanning.  If it wasn’t done that way in this case, then it means more work for the people retouching the images.

The new restoration has no prints.  SO WHAT?  That has nothing to do with what you’re talking about and is a diversion from the point.  Wetgate has to do with the scanning or printing the negative, not projection. Note to the sticklers out there: yes, we can use wetgate transfers on prints, if that’s all we have, but that is not what is happening here.

Error 4:

“Luckily, there have been dramatic advances in digital-restoration technology in just the last few years. New software can erase scratches, clean dirt and modify contrast and colors not just frame by frame but pixel by pixel. In the old days (circa 2006), if you wanted to brighten the desert sand in one scene because it was too dark, you’d have to brighten the sky too. Now you can brighten the sand — or even a few grains of the sand — while leaving everything else alone. And in those days there was a limited palette for restoring faded colors. Today’s digital palettes are much vaster.

“In one sense, this restored Lawrence might look better than the original. Because of the film stock’s exposure to the desert’s heat, some of its photochemical emulsion dried and cracked, resulting in vertical fissures. ‘Some were just a few pixels wide,’ Mr. Crisp said, ‘but some scenes had hundreds of them, filling as much as one-eighth of the frame.’”

The way this is written implies that there were shooting errors that caused exposure problems with things being too dark or too bright.  It further implied that Grover Crisp and his co-workers are going in and haplessly changing things to suit their own artistic eye, not that of director of photography Freddie Young or director David Lean.

I have a lot of respect for Grover Crisp, and I know he’s not doing that.

Lawrence of Arabia was shot on Eastman color stock that was very unstable (it was especially bad from 1958-63.)  The colors fade unevenly, and brightness fades unevenly.  What they are actually doing, despite the way the article is written, is to match the colors with the way some of the old Technicolor reference prints look (Technicolor prints don’t fade, but they are 35mm and 2-3 generations down from the negative).  This is restoration, not willy-nilly artistry.  There are certain colors that will be almost entirely gone (especially blues and greens).

Error 5:

“Sony went to so much trouble to create not just this release but also a new archive for the ages. Film degrades; digital files of 0’s and 1’s do not. In the coming years, new software might allow still better restorations. But the technicians making them can work from the 4K scan. They won’t have to go back to the negative.”

This is just crazy on a lot of levels:

1. Robert Harris made a nice duplicate negative in 65mm, on color-stable stock, for the 1980s restoration.   At the time he made it, there were already a number of unrecoverable scenes and missing bits.  This article makes it seem that Harris’ work is now outdated and rather trivial.  Nothing could be further from the truth.  Harris and director David Lean worked together to save Lawrence of Arabia, and without them, Lawrence would be less than it is today.
2. Ones and zeroes don’t degrade.  Hard drives do.  These are spinning media that are subject to magnetic fields, ball bearing problems, heat, cold, and probably the most fatal problem, sticktion.  A hard drive with sticktion has had the spinning magnetic rotor stick to the read head (much like a sticky record album sticking to the needle).  If it sticks too hard, then the drive can’t spin, and the disk is ruined.
3. Ones and zeroes don’t degrade, but file formats aren’t forever.  Neither are disk drive formats.  Had Lawrence of Arabia been restored digitally in 1989, the results could have been saved on 5.25” floppy disks, and no one could read them today.
4. Scanners are wonderful and they get better every day.  I’d bet that if the film is stored well, it will hang together well enough to survive until better scanners come along so that it can be scanned and improved again.

This same thing happens often with other “restorations.”  Gone With the Wind and The Wizard of Oz were shot in 3-strip Technicolor, which produces three extremely stable black-and-white negatives.  These are a pain to reproduce, so they got “restored” in the 60s to “modern” Eastman color stock.

Whoops, the restoration faded in a few years.  No trouble.  They reprinted it again, with better technology, in the 1970s.  They went back to the black-and-white negatives, which were still around.

Whoops, that restoration faded too.  No trouble.  Another restoration was done in the 1980s.  Guess how?  From the black-and-white negatives.

Oh, wait, they got a better way to reproduce the film and make the alignment sharper?  Back to the negatives.

And they needed to re-scan to make a Blu-ray (well, this time, they did an 8K transfer, which is what the Nyquist sampling theorem says we should do for such a film).  Gee, they went back to the negatives.

The moral of the story: save the negatives for as long as you can because they seem to get used a lot for restorations.

Error 6:

“Between the detective work and lots of video improvement (before the days of digital), it took Mr. Harris 26 months to restore the movie — 10 months longer than it took David Lean to make it.”

The preservation work Harris did on Lawrence of Arabia was on film.  He didn’t use video improvement.  There was no video that would do the work.

Error 7-8:

“Its life in home video has been spotty as well. The first DVD, in 2001, was made from a badly done HD transfer: colors were way off, contrasts too bright or dim. A redo, two years later, was much better, but the dirt and scratches were cleaned up by a ham-fisted process called ‘digital noise resolution’ — the easiest and, for some problems, the only technique available at the time, but it softened the focus and dulled detail.”

I am not sure, and it’s not really worth looking up, but I doubt that the DVD was made from an HD (High Definition) transfer in 2001.  It’s technically possible, but it’s unlikely.  It was probably done from a standard definition transfer, which would also account for the color drift, since the color gamut on standard definition television is pretty limited.

I have no idea what “digital noise resolution” is.  I suspect that what he means is “digital video noise reduction” (also DVNR), which is an automated process to remove scratches and other imperfections from films.  Cartoon aficionados have been bemoaning this for years.  DVNR is still used, fairly often in fact, but it can be done gently or in a ham-fisted way that the author describes.

“A forthcoming Blu-ray Disc of the film, out Nov. 13, fixes all those problems, in part because it’s Blu-ray but more because it’s mastered from the same 4K restoration as the theatrical release.”

Is the mere fact that something is Blu-ray some way of saying it’s anointed with a perfection not yet seen?  Blu-rays, DVDs, films, and videos can all look great or terrible depending on how they are handled technically.

The overarching thing that the author misses (and that others are not missing) is that this digital restoration is not archival no matter how much we would like it to be.  I’m on mailing list after mailing list from archives in a panic about how to store things so that they will last.

I was at the Library of Congress recently seeing the process of the entire run of Laugh-In being copied from 2” tape, a format now long obsolete, to something now (we hope) more permanent.

At the same visit, I saw a roll of film made in 1893 by the Edison people.

Which of these is archival?

The Library of Congress still uses, and intends to use, 35mm film for archival storage.  They haven’t found anything to beat it yet.  They are keeping Kodak and Fuji from shutting down the manufacturing lines.  Other archives demand film, too.  It just holds up better.

That doesn’t mean digital doesn’t have its place.  It’s just that digital isn’t the magic panacea that cured the world’s ills.

It’s a tool, just like anything else.