The de-noising and super-resolution looks quite good to me. It's the colorization that's super unstable and looks ugly. If they'd just left it B&W it would IMHO be more impressive.
I kept waiting for it to get better than a light brown haze with flashes of green... but it never did. The fire looked okay, but even that looked weird at the beginning of the sequence.
What really made me laugh was the blue smoke followed by a headline implying that was all at night. The AI filled in pristine blue skies and inverted the wreckage colors. It actually made up history. I'm calling it: Unintentional automated disinformation.
The original video was interlaced. The deinterlacing part of the algorithm is not very good, and I think they would have gotten better results with a special-purpose pass before handing off to the neural network.
I'm now imagining a low-paid 1930s film job called "interlacer" which consisted of taking every frame and drawing tiny, perfectly straight black lines on it, with a tiny fineliner and a tiny ruler, on odd and even rows, every next frame.
I know of a company that did the opposite to handle 2:3 pulldown for rotoscoping. They had a photoshop action that would select every other line, copy&paste to a new doc, collapse the empty space, and then paint the frame. Then, do that a second time for the other half. Finally, more actions to recombine.
I couldn't make this up. My jaw just hit the floor when it was explained to me the first time. I still shake my head typing it up to post here.
They solved the problem by increasing the work unnecessarily. So to explain the issue with more detail and not requiring peeps to jump to wikipedia, 2:3 pulldown is how 24fps film was converted to 29.97 fps interlaced video. This increases the number of frames roughly 20%, but it does this by repeating fields of the original progressive data. When you step through this 29.97 video frame-by-frame, you will see a repeating pattern of 3 progressive frames followed by 2 interlaced frames. To do this asinine made up workflow, they then increased the number of frames again. Instead, they should have done an inverse telecine (IVTC), which reverses the 2:3 pulldown returning 24 fps progressive frames. When rotoscoping, you definitely do not want to be creating additional frames "just because". Applying a small amount of logic should have suggested that these interlaced frames are odd since the orgininal source had nothing but clean progressive frames. If this was introduced in the conversion to video, surely it can be undone.
They, as you, decided it was acceptable to do whatever needed to be done at whatever expense rather than taking 10 minutes to find the proper workflow which would have saved them money. A simple phone call to the company they used to transfer the film to video would have been able to explain to them how to do this in less than 5 minutes. I know because I worked for the company doing the film transfer and had helped several other clients with the video for film post workflows. Today, it's even easier because there's like a bazillion write ups on how to do this posted on the web.
Edit: I didn't answer the question directly after pontificating. IVTC is the process that needed to be applied. Many many tools exist(ed) for this. The tools at this company's disposal would have been After Effects, Avid, etc. Later more tools became available like AVISynth, FFMPEG, and other dedicated tools were created by people to tackle this directly.
Analogue video (e.g. television) interlacing did actually begin in the 1930s, a few years before the Hindenburg disaster. But the original disaster footage would have been shot on film (not interlaced of course), cut into newsreels and then converted to a television signal via telecine.
What is missing is object continuity with respect to color. That would quiet things down tremendously. Right now it is as if every object gets re-painted from one frame to the next in a completely new (and often garish or wildly incorrect) color.
I believe there already is quite some continuity, otherwise the colors would flicker much stronger from frame to frame. In the video it varies smoothly from frame to frame.
There are many instances of frame-to-frame discontinuity that I can't explain other than by a lack of object detection and labeling. It would be less wrong to use the color from the previous frame even if the lighting changes than to use an entirely different hue for the same object.
Only things like TV screens and other displays (and some interesting objects covered with micro surfaces that can cause light interference) can change color that rapidly given the same color incident light.
I think they get that not because of putting in that constraint, but only because subsequent frames are similar. That makes the coloring algorithm pick similar coloring.
Looks like the AI can handle objects it "knows" (people, grass, sky etc.) okish, but is completely confounded by the Zeppelin - which is sad, as the Hindenburg is of course present in most of the shots. Maybe they should have trained it with some color(ized) photos of the Hindenburg (like this one: https://www.alamy.de/das-deutsche-zeppelin-luftschiff-die-hi...) first?
This is where a human videographer could trivially do better or add value. The obsession with "automating everything" is a real disease in ML and CS generally. Sometimes it makes things worse! This is one of those situations.
I had to stop 60% of the way through. It was giving me a headache.
This video proves to me that you can’t do denoise, upscale and then color without stabilizing first. The result is too jarring once image stab is a known transform.
Considering it is entirely automated, this is extremely good.
I'm not a bit against ML restoration. We've been paying artist to color BW in the early days, this is just replacing the artist by machines.
Judging from the video, it looks like no inter-frame relation is considered, so color varies wildly from on frame to another. The video still lacks some form of stabilization. Frames still have defects inherited from the film.
Also, it clearly was recorded in different speeds, so people have an uncanny walk in the last scenes.
I hope, in the not too distant future, these flaws will be taken care of and we'll see restorations that are very hard to differentiate from original footage.
I couldn't possibly disagree more. This is nothing more than the digital equivalent of the monkey Jesus restoration: https://en.wikipedia.org/wiki/Ecce_Homo_(Mart%C3%ADnez_and_G... It's extremely disrespectful. The colours are shifting all over the place. At times it's no better than overlaying a random gradient over the original.
> We've been paying artist to color BW in the early days, this is just replacing the artist by machines.
And some people are very upset over those. (Have you ever actually seen classic movies like "It’s a Wonderful Life" in colour?) But this ML nonsense doesn't even hold a candle to those. In the manual process, for every object, they pick a colour and stick with it.
The original medium contains information collected at the time, with the technology available at the time. As such, it represents a time capsule that includes much more context than we can possibly realize; especially analog filmstock and audio. "Upscaling" and "autocolorizing" is injecting inferred (and often wrong) context from today using today's technology and context, tainting, corrupting (IMHO), and again, IMHO, ruining what the original artifact represents.
In our current technological, digital zeitgeist, everything that seems like higher and better resolution is universally better and must absolutely replace what was there before. But this is just a shifting fad, unrecognizable to the people who were alive then, and probably unrecognizable to people in the future.
Preserve, conserve, protect and pass on. Don't corrupt and "better" these things.
Sure, you can train all you want, but showing off something that is inferior to other things that can already be done is just embarassing. At least it should be. There's a difference of showing off progress to your parents or investors or whatever, but making a public post of something clearly inferior in a way that screams "look what I did" is sad really.
Interesting. This is a common consumer view. Since pure consumers aren't acquainted with anything but the best in the field, they demand very high quality of anything they witness. That makes sense.
At least part time content creators know that creation is harder - because that first step to actually doing something is mentally hard (pure consumers find it impossible). People will make crappy things before they make good things and they'll share them with each other.
Perhaps the problem is when pure consumers, seeking further stimulus, enter creator spaces. Or where part-creators accidentally expand the audience for their fellow creators into pure consumer spaces or more-consumer-than-creator spaces like HN.
If you are attempting make something from scratch that has a competitor on the markent now, why would you evern think someone would use your product when it is so clearly inferior to other offerings? Hope to attract business by being the cheaper option? That just means you're going to get the clients nobody wants.
There's a difference between showing your friends and family something, but opening up the entire public with posting to YouTube and asking "tell me what you think" means you must be pretty proud of it. If you were proud of the results from this, then just wow.
In full disclosure, I've spent many an hour in the chair of restoring film/video. I've written many a tool to help with this endeavor for internal use as well as used professional tools. I have beta tested software that later went to production release. I have sent content out to outsource the work and critically evaluated the results. If this was the result someone sent me after suggesting they could do the work, I would never send them the work as well as vocally tell others not to waster their time.
This isn't even good enough to send someone's 8mm footage to for viewing on modern devices let alone professional.
You’re writing this comment of yours and posting it publicly. Are you so incredibly proud of it? It’s riddled with typos and wouldn’t get you a passing grade in middle school.
That’s what I would say if I were being overly critical of someone sharing their opinion. It really doesn’t need this degree of hyperbolic value judgment. If it’s such rubbish it will easily be outcompeted for attention.
Exactly, the upscaling is good but any human intern would have fixed the color for free, even by choosing a random one, and it would have produced a result closer to expectations...
> Update (7/17/2021): The article below has been superseded by the results discussed in “Far Beyond the Stars” and the accompanying tutorial, “How to Upscale Star Trek: Deep Space Nine.” These articles are the latest that I’ve published and the best showcase for my latest work.
>> The video still lacks some form of stabilization.
Yeah, I wonder why. Once you are able to create colors, I'd think stabilizing the picture must be really easy. And since we're doing fancy interpretation (yeah, I say "interpretation" 'cos ML can't know the exact colors, just see the craft turning red and blue all of the time), then it'd be nice to go to the end of it : color, stabilization, speed fixing, the whole thing.
Besides, how do we know how such "interpretation" is close or far from reality ? What sort of validity testing is done ?
I'd sooner call it vandalism. Restoration is what's shown in "They Shall Not Grow Old" where they paid attention to detail and approached it with respect.
Sharp edges and shifting red-blue colours is not what I would call restoration.
It’s not vandalism unless the originals have been damaged. If someone makes a digital copy and goofs around with it I really don’t see any harm being done.
It's getting sadder and sadder to see people enjoying these fake restorations. Eventually we'll have so much of a mess that the original source will be forgotten or lost.
In this particular case I think there is the extra complication that the ship skin was made of a special material that likely does not burn with the usual wood fire colors. Combined with the hydrogen on the inside I actually have no idea what color the fire would have been - the most accurate way to restore this would probably have been with an actual eyewitness in the loop, or by restoring a section of the ship including the flammable paint.
Apparently, most of the passengers were along the window bay in preparation for landing. They were able to hop out of the windows as the guest compartment reached a safe distance to the ground, and before hot wreckage could fall on them.
The three things that saved them was A) an airship, even one leaking and on fire, had a slow enough fall to safely evacuate B) the majority of fuel sources were above the crew and guests and C) most of the guests had an easy exit.
About half of the guests had an easy exit, the other half were on the other side of the airship where the door had become jammed, most of the people on that side perished in the flames.
Indeed, a little over half a minute from the start of the fire to total destruction of the airframe, it is amazing that there were that many survivors. Important to note that the crew and the airframe itself were two separate components, but that the passenger area was embedded in the fuselage.
More miraculously when you have seen that footage is to realize that some of the passengers and crew walked away without major injury.
It is mostly hydrogen, so most of the energy was dissipated in the first few seconds. Most of the other materials or do not burn well or have very low thermal capacity. It was not on its highest altitude when burning started and drag dumped the falling to much slower speeds than a free-falling person.
I'd estimate that most people who died were trapped or unable to move. Those who were free to run and had enough "air" were very likely to survive.
Hydrogen does not burn energetically, especially before it has been mixed with oxygen. The tragedy was caused by the coating used on the exterior which was essentially solid rocket fuel.
That theory has been largely discredited. There wasn't nearly enough reagents in the fabric to create a large thermite reaction. An oxyhydrogen explosion is the most likely explanation: there was probably a leak prior to the explosion, which allowed air to mix with hydrogen inside and surrounding some of the gas bladders. Also, a walkway ran through the bladders, which acted as an oxygen source once the bladders burst, as evidenced by flames being directed through the axial walkway.
> Hydrogen does not burn energetically, especially before it has been mixed with oxygen.
Drop the word "especially" and I might agree, but a mere party balloon of hydrogen mixed with oxygen in a stoichiometric ratio going off will sound like a rifle and rattle windows.
When I was young, I volunteered at a science museum who would, as part of their chemistry show, fill a balloon with hydrogen and ignite it off a tesla coil.
One time, we (~15yo kids) convinced the person giving the show (maybe...college age kid?) to also mix O2 in the balloon, instead of only H2.
The boom was so loud that management folks on the other floors sent people down to see WTF was happening. I don't think he got into (much) trouble, but he sure never did that again.
I don't know why this theory is repeated as though it's fact. The explanation that a lifting gas bag ruptured, mixing it's contents with the air seems a better fit to the witness accounts.
New footage shows the tail-fins catch fire before the gas-bladders erupted. Those tail-fins must, therefore, have been quite flammable. The video suggests that the trigger for the fire was the release of a large build up of static electricity on approaching proximity to the ground. An airship is essentially, an oversized Leyden jar, and such a build up would cause problems, even for ones using helium for lift.
The Hindenberg was also reportedly quite tail-heavy coming in for landing, which suggests a leak. If it was leaking at the tail and this was triggered by static discharge it would fit most of the facts.
This happens quite often with accidental explosions I've noticed. Explosions are not as lethal as you might expect, unless someone deliberately ensured that it caused a lot of damage.
It was a design flaw. The flames were able to travel quickly through the central passage which passed through the center of the gas-bladders. The footage shows flames venting from the nose of the airship.
This gives a much better view into what everyday life must have been like at that time, it almost reminds me in some ways of movies such as Koyaanisqatsi.
In the early 90's I was working in early streamed digital video. One of the projects was a history of aviation documentary. The original negatives of early flight, including the original Wright Brothers flights are lost. But in many cases paper print outs of them remain. Before copyright was extended to film, to get a copyright a film had to be printed to paper. These paper print outs are all that remain. But this being the early 90's, there was no machine learning nor a formal field of computer vision as we know it now. We painstakingly scanned the thousands of feet of printed film, and reconstructed them as CLUT-127 (color look up table) animations. I seem to remember that project got streaming media awards of some sort. It was a long time ago.
Me too, really interesting to see and easier to connect somehow when the quality is better, compared to the black and white versions with strange frame-speeds.
I agree. Correcting the frame rate jitter and also motion stabilization (in addition to upscaling the resolution), really does result in a much more engaging view into the past.
I'm perfectly happy without colorization, but also willing to overlook it.
The temporal color jitter is pretty bad, but the other processing seemed to crisp it up. Based on my watching two minute papers I have to guess they can do better on the temporal consistency of the color.
Károly Zsolnai-Fehér is just so darn charismatic. That has to be one of the reasons Two Minute Papers[0] has >1M subs. Is there anyone in the space even close to that level of engagement?
I did notice them as well. I wonder if the original video had already been edited so that the swastikas were (in my opinion) mostly clipped out, or if this is a more recent edit...
I think it would be cool if you could have some kind of combo human+ML system, where you use ML to actually detect the different objects and track them between frames, and a human can choose what color to use for each one. In a couple shots at least the Hindenburg showed as fully red instead of silver, and I don't think it is always possible for an algorithm to actually know the correct color without external data. If the algorithm could just say "I've identified this object between frames", and then the human can choose the correct color, could be best of both worlds.
Or maybe you could feed in some reference photos with the same objects, but already colorized. And then the algorithm could match objects from the reference photos to those in the black and white ones to get the colors.
It’s interesting that the color of the Hindenburg itself is always shifting. The object is so out of sample that the ML colorizer has no idea what to do with it.
Although visually it appears to be high resolution, it feels like an illusion, like my brain still receives no additional information Vs the original. Which is odd because I can look at individual elements and see more precision.
This sent me down a zeppelin Wikihole and led me to a much greater appreciation of the engineering and operational complexity of these airships. Thanks!
And I've been unaware that Manhattan had so many skyscrapers at the time. Or maybe not. But one can glimpse that for a few seconds in the video. Awesome.
Another way of looking at that is how little change a typical city will undergo once it is laid out. This goes for Rome and Amsterdam just as it does for NYC.
I was away from Amsterdam for about a decade before coming back there after having lived there before for a stretch of nearly 28 years (with a few longer absences) and what surprised me is how little had changed, and yet, here and there there were buildings that I was pretty sure weren't there before or some familiar landmarks that had gone. Over many decades or even centuries that kind of change will add up, and even though for Amsterdam in particular at some point there was a plan to 'overhaul' it and make it more modern (which fortunately got arrested in the earliest stage, but it did do a lot of damage to the east side of the river Amstel) the vast bulk of the city has been unchanged since decades.
What real change there is is expansion wherever there was undeveloped land, but that isn't change insomuch as it is simply growth, what was there before remains.
Just zoom in on the city, click on any building outline on the map and on the right hand side you will find all kinds of interesting information, including the year that ground was broken or the building was formally entered into the registry. This information is not always available but it still contains a wealth of information.
I think I could handle unrealistic colors but the way they flicker so much frame to frame is really jarring.
It's interesting that the algorithm seems to generate chromatic aberration at hard edges? Most clearly around the letters on the title cards.