I am the author of the referenced article. The problem is matching surveillance camera images (like from Lord & Taylor's) is impossible because of the steep down tilt angle of the cameras plus their low resolution. Smartphone images could be a little better as the angle is more head on and the resolution is higher. However, even those images tended to be in a wide field of view which means low pixel density, hurting matching performance. Last, but not least, the caps and sunglasses hurt recognition as well.
Matching surveillance camera images is always possible no matter how bad the image quality is. The only matter is how many false positives show up.
Facial recognition algorithms always give their best matching candidates, with thresholds to skip those not quite matching ones. Your article makes it sound like no such results were produced at all, which is unlikely how recognition is supposed to work.
First of all, almost any facial surveillance system would simply reject images like those from Lord & Taylor's. The pixel density is simply too low to even attempt plus the downtilt would also likely trigger a rejection. That's how facial recognition does work in practice.
Even if the system would allow it, not only would false positives be enormous, you'd probably miss the match as well (false negative).
Not impossible though. Given that such software is used to steer the future efforts of the investigators, it is possible that the software does say "no match found" if the probability of a real match is too low.
Why is surveillance video not HD? Shouldn't this quality be almost a commodity by now? How different are these cameras than what you get in, say the latest Samsung Galaxy S4, for example:
Probably because the cost of surveillance cameras is more than just the cost of the resolution or storage. the ones that hang outside need to be weather proof and vandalism proof. they were probably bought as part of a large and expensive system, intended to be paid off over 10-15 years[0], to replace all those cameras whenever more resolution becomes available, is probably quite expensive, regardless of how cheap the actual camera is. It's a lot of cameras, in a lot of hard-to-reach places, with expensive reinforced casings, and would the output format of the cameras be compatible with the old system?
OTOH, I do actually see more and more HD (or relatively high def) surveillance footage. It all depends on how when the security system was installed, I guess. Also smaller businesses can afford a shorter turnaround for these things, as well as not having the handicap of a head-start[1].
[0] actually I'm guessing those lifetime spans, it seems a bit much? but I also can't see anyone signing off on a security system that's expected to be replaced in 5 years. good thing my job isn't facility management ... :P
Its not only about the cameras. Its currently expensive to store all that HD video. ( and now, one cant just upload it to the cloud. Necessary bandwidth isn't yet available locally ).
One are also dealing with a lot of legacy systems her. But the next generation surveillance system will probably be HD, and then maybe 3D, 4k, 16k etc.
Surveillance cameras are typically bought and then forgotten. So the new ones are probably HD, but you get a collection of cameras between HD and Betamax if you look at the cameras that are actually in use.
And for outdoor cameras, how often do the lenses get cleaned? I'd think that in a city center, with lots of car exhaust, etc. they'd be pretty dusty/greasy in just a few months. HD won't help much if there's a film of diesel exhaust soot on the lens.
Interesting to see articles promoting adding more surveillance cameras for all their flaws (bad angle, resolution trails behind phone cameras, etc). I would focus on acquiring publicly available photo data in real time at dense events and scan them. I.e. crowdsourced surveillance. I'm sure someone is working on that.
In a real life scenario, surveillance camera images are all you will get to work with, most likely. So maybe, the focus should be on figuring a way out to recognize individuals based on images taken at those angles. Is there any progress in this regard?
The problem is that any non-perfect automated system will wrongly convict so many civilians as being terrorists that nobody will trust it anymore.
Let's say that you want to catch terrorists creeping around in your airport. To do this, you have a magic camera that, when it takes a picture of a civilian, compares it to a picture of a known terrorist. If they're in fact the same person, it will flag security over. Let's say that it is indeed a magic camera and makes the correct decision 99.99% of the time, which is far better than the current state of the art[1].
Do you have any idea how much DHS or DoD would pay for such a magic camera?
But it won't be enough.
Consider a watch list of 100 terrorists that we're looking for in our airport. If our camera compares each civilian's picture to all 100 terrorists in succession, and if we model each comparison as a Bernoulli trial, then each civilian has a 0.9999^100 = 99.004% chance of getting through security unfazed, which means we flag innocent people as terrorists about 1% of the time.
This doesn't work in an airport setting. Assuming 5,000 people pass through your small airport per hour and you have a 14-hour day, that means your security people have to go through 0.01 * 5000 * 14 = 700 false alarms each day. That's about 1.2 false alarms per minute. Your security people will have to go through all their checks each time this happens. Before long, they're going to stop trusting your magic camera.
As you scale your watch list up, it gets worse. How well would it work for 1,000 people on the watch list? 10,000? Note that the current US watch list has 50,000 people on it.
Even if it were a perfect magic camera, it coudln't catch terrorists on their maiden crime who aren't on the watch list at all.
Do you know how unreliably people recognize strangers? Many people are already wrongly convicted on such evidence.
Even perfect automated facial recognition won't convict people. It will merely point out people that should be looked at. If a jury looking at the image and the guy's face think the system is wrong, they can disregard it.
As far as prevention goes, what we need are systems that can detect deception reliably, which will help if we ever decide to ask people meaningful questions. Not to convict, but to focus attention on a possible threat.
Sure, I should have picked a better word than "Convict", especially since I didn't mean it in a legal sense. Given the context, "flag" or "decide" or "indicate" would have worked better. I only wanted to convey that the system thinks the probe and gallery pictures are the same person
This is a completely different situation; there are 2 people we have pictures of that we know are terrorists; we have a database of suspected terrorists; had we found 100 matches in that database, with one of them being the right one, it would have narrowed the investigation considerably. It was the perfect situation for AFR, had AFR either been more advanced, or the pictures been more amenable to current AFR technology.
How hard would it be to throw off facial recognition by simply wearing very high quality makeup and prosthetics to alter your facial appearance (ala hollywood movies)? If I were the boston bombers, I would have first learned how to make myself look completely different. It's unbelievable to me that these guys didn't realize they'd be captured on tons of cameras that day.
If you wore Mission Impossible style masks, there's no reason why the facial recognition would be able to pick you up from underneath. You might alert the suspicion of people around you more though, as they don't ever seem to be perfect.
When the FBI released the photos of the two suspects, I kept wondering, "Why do they need the publics help when I'm sure there is facial recognition software to help match the suspects' faces?" Well I guess this article somewhat answers this, but I am still in disbelief that our government does not have better software.
For how long it took to discover their names, the two suspects had TOO much time to escape. They had about 48 hours to book flights out of the country, flee to another part of the country, do more damage, etc. The lack of facial recognition software could have ultimately led to this case being unnecessarily longer. This is an issue that CAN and NEEDS to be fixed quickly--in fact, I hope the government is working on a more sophisticated facial recognition software right now. Ideally, the facial recognition software should be able to crawl through the web to search for faces.
Now that I think about it, a good hacker could build a search engine that allows people to post photos and the program will run an automated search across the web (Social networks, Google, etc.) to see which pictures match it. Hell, Facebook even has facial recognition to some extent. A more sophisticated one cannot be that difficult for the government to build and put to use.
There is a lot of money being put into facial recognition software by law enforcement and the military (including DARPA), so it's certainly the case that it's being worked on. But it's a hard problem, especially to do robustly at a large scale. What they do have is almost certainly the state of the art— in fact much of the current commercial state-of-the-art is itself based on stuff developed in older DARPA projects.
Facebook only has to distinguish among friends of the uploader. And it's not perfect. This man was wearing sunglasses and a baseball cap. He would not have been identifiable out of the tens of thousands of people at the event, matched across the (probably) millions of people in the database.
What's wrong with needing the public's help? The public did a great job in providing critical evidence.
Maybe instead of more intrusive surveillance we find more ways for the public to participate. Maybe we uphold the public's right to take pictures and video in "sensitive areas". Etc.
I get what you mean when you say these guys had way too much time to make some kind of an escape and I agree then that there is an absolute need for a much better software. Having said that, I do not think it would be wise for government to build and use an "image crawler" to search and store images of citizens...it would provoke widespread criticism concerning privacy of millions of citizens. On top of that in this particular instance, the police commissioner points out that "both Tsarnaevs’ images exist in official databases"..the software failed to make a match.
I watched this lecture recently http://youtu.be/-IUe3ce_29I which talked about how hard it is to do image recognition at low resolution. One solution is a "fovea" system that scans an area for interesting items and zooms in on them with a high resolution camera automatically to get better recognition.
I wonder if this would help in similar instances. A system that uses facial detection to isolate faces and then zoom in on them very quickly seems feasible. I'm assuming it would be overwhelmed in a scenario like a marathon of course, but perhaps it would help in other situations.
Not just facial recognition, the "terror watch list" failed (after a warning from Russia about the older brother) lack of background checks on the AR15s they owned failed, the secured perimeter failed (the teenager was outside of it) the city lockdown failed (the guy smoking the cigarette outside when he saw the blood was technically violating the lockdown).
In the end it was mostly dumb luck, dumb criminals, and the fact the poor victim who lost their legs managed to survive. In fact that was the success, that more wounded people didn't die thanks to fantastic work by the hospital and doctors.
Instead of patting law enforcement on their backs, how about thanking the doctors, where is their applause?
It seems disingenuous to say that facial recognition failed. As far as I can tell, all the images we had of these suspects were either very low-resolution, taken from angles, or both. Seems plausible that law enforcement didn't have any good candidate images from which to run a facial recognition search.
> In the end it was mostly dumb luck
What? Painstaking police work went in to getting images of these suspects, and the release of those photos are almost certainly what led to Thursday's events and, ultimately, the killing and capture of the suspects.
> the city lockdown failed (the guy smoking the cigarette outside when he saw the blood was technically violating the lockdown)
Firstly, the lockdown is what forced suspect #2 to hide out in a boat instead of fleeing the area. Secondly, and less importantly, the guy smoking a cigarette wasn't violating the lockdown. It had ended prior to him going outside.
Was there luck involved? Of course. But there was also tremendous work done by a number of law enforcement agencies - without which, the suspects may never have been killed and captured.
> It seems disingenuous to say that facial recognition failed. As far as I can tell, all the images we had of these suspects were either very low-resolution, taken from angles, or both. Seems plausible that law enforcement didn't have any good candidate images from which to run a facial recognition search.
"Failed" might be too strong, but I'd definitely say it's not yet ready for deployment.
My impression is that the Boston marathon was extremely well-photographed from a variety of angles, by everything from nearby security cameras to folks in the crowd. This seems like the best kind of data set you'd reasonably expect outside a controlled environment (i.e., a lab or security checkpoint). If it's not yet good enough to at least flag possible matches based on this data, I'm not sure I believe it's ready for widespread use.
The brothers didn't leave the area, didn't go into hiding and were able to kill a campus officer and hijack a car, all without police finding them for a week.
It was mostly dumb luck. It's not like they were hiding!
They didn't go into hiding or leave because they didn't think they needed to. It seems pretty clear they they didn't want to be caught.
From what I can tell, numerous people called in after the photos were released. Even if Thursday's events hadn't taken place, the police would have known the identities of both suspects shortly thereafter.
It just seems crazy to me not to acknowledge the role of investigators in all of this and chock it up to luck.
As for the edit you just made about the bombs in Hanover: are you implying there's a link to the Marathon bombers? Seems like conspiracy theory reaching - unless there's some sort of evidence you'd like to share.
The brothers didn't leave the area, didn't go into hiding and were able to kill a campus officer and hijack a car, all without police finding them for a week.
I know you dislike law enforcement in general, but such casual exaggeration isn't helping your case. As of today, it's only 6 days since the bombings took place. The bombings happened late on a Monday morning. By Thursday evening, there were photos and videos of the suspects all over the media. There's a good chance that that's what flushed them out of cover.
You are wrong. We know that the younger brother returned to his school 60 miles away from the bombing, and was seen at his school partying it up the day before the MIT officer was killed. Please get your facts straight.
that more wounded people didn't die thanks to fantastic work by the hospital and doctors
The EMTs and hospitals did marvelous work, but one nonobvious fact that helped a great deal was that the bombing was within 2 miles of FIVE DIFFERENT level 1 trauma centers. All five are world class teaching hospitals and one of them, MGH, had been recognized as the best hospital in the US at one point.
There are several states that don't have even a single level 1 trauma center....
Actually, the lockdown was voluntary, though with strong community pressure to comply.
But yes, multiple subsystems failed in the police response. If you listened to police scanners, you also heard that they had some trouble identifying streets correctly, or keeping their mics off when not talking. But police response protocols are one human system that definitely needs to be able to work in the face of multiple subsystem failure. I say judge the police on how the response worked as a whole.
I have to say though, thank god that Dunkin' Doughnuts' Tactical Ration Deployment Vehicle ran smoothly.
to be fair if you live in Massachusetts you would know, all of the streets are not marked as you would expect in other areas. If there are street signs at intersection often times only one of the streets is marked.
The order to close businesses was pretty broadly heeded, and with no workplaces, shops, daycares or school programs to go to the voluntary 'shelter in place' situation became pretty mandatory, even for people outside the lockdown but working in the Boston area.
Talking about the scanners and technology (and failures) didn't "homeland security" give millions to metro police departments around the country to get their communication encrypted? I vaguely remember reading about that a couple years ago. I wonder where that money went if that is accurate.
Apparently the police prefer interoperability over secrecy - if all the departments had encrypted comms during this manhunt, nobody would have been able to talk to each other because there were so many different police forces out there.
Interoperability is very highly valued. I just went through some government training courses on incident response, and interoperability was stressed many times.
I've wanted to ask about this since seeing the Boston police channel available on the internet. In the UK all the channels are encrypted making it virtually impossible for a civilian to listen in. Is it simply financial reasons that the USA doesn't do this?
Speculating from the perspective of a Brit in Boston, it may be to do with the fact that police organization in the US is much more fragmented and federated - rolling out compatible radio encryption equipment, handling key management etc. across multiple police forces would be much more complicated, financially, logistically and politically. On the operation in Boston on Friday, you will doubtless have heard of the involvement of the FBI, ATF, Massachusetts state police, city of Boston police department, Watertown police, MIT campus police, transit police... cops from all the surrounding towns were also involved in the lockdown. In the UK this would all have been handled by one constabulary, possibly with support from specialists from the Metropolitan police, and with fundamentally common funding and control structures from the UK Home Office, whereas all those different police organizations in the US are responsible to different funding and governing authorities.
Is it simply financial reasons that the USA doesn't do this?
Surely not only financial reasons. In the United States, consumer purchase of inexpensive radios that pick up police communication frequencies has been commonplace since the early 1970s (when I first owned one). I remember a Hacker News thread a year or two ago in which a European was surprised that a news report including the name of a person arrested (not tried and convicted, but just arrested) was permitted in the United States. Our historical sense in the United States is that police operations and arrests should be largely in public view, reportable by the news media and observable by the public. Different societies come up with different trade-offs on this issue, in law and in custom, but in the United States arrest records are public (and routinely reported, although often with some time lag, in local weekly newspapers) and police radio activity can be monitored by many private citizens. It was noteworthy this last week that the Boston police make more use of Twitter than many urban police departments, but the general degree of openness observed in the police response to the marathon bombings is fairly typical of the United States.
What you fail to realize is that the "terror watch list" you speak of doesn't hold people on it from just any hearsay. The articles said that the FBI did its due diligence and found no reason to continue an investigation on them.
Think about it this way: would you want just anyone to be watchlisted based on tips that could be politically motivated, intentionally misleading, or just completely factually inaccurate?
What you can never account for in terrorism screening is the lone-wolf element. There will always be some fringe guys in parts of the world who hatch their plots outside of the view of what we're able to observe. The only options you have are to limit their access to harmful weapons/devices, or contain them as fast as possible when they do go on the offensive.
Yes, doctors and other emergency responders did a great job, but don't be so quick to take away a victory from law enforcement when there are very few positive stories circulated about them ever. The boston marathon runners/crowd didn't know they were putting their lives on the line that day. Law enforcement, on the other hand, made a conscious decision to do so.
We don't yet know if the "terror watch list" failed. Perhaps he was checked and they found nothing, because at that time, they weren't terrorists and weren't planning a terrorist attact at that point? We're not living in a minority-report style pre-crime detection world yet.
You can't state that as fact either. It's possible the lock down and house to house search left the suspect with no other choice but to find a hiding place and stay there. If residents remained on the streets it's possible he could have slipped away.
Well, considering "the second suspect" was possibly armed to the teeth, and maybe wearing a suicide vest, the hope was that, yeah, a regular person would not stumble on him and get killed. The guy who found him got lucky, it sounds like.
this is absurd. i'm in Boston. there is absolutely no shortage of support and praise for the doctors and hospitals here.
lots of things went wrong for this to happen. thanks for pointing out the painfully obvious. our police deserve the "pats on the back" (which, by the way, is an extremely condescending way to put it) they're getting, and more.
Massachusetts does in fact have an assault weapon ban in place (similar to the 1994 Federal AWB), although you can get modified/MA-compliant versions of many "assault weapons" such as the AR-15/AK that are legal, however you are limited to a 10 round magazine. They would have needed a Class-A LTC to even contemplate owning such a weapon -- this involves fingerprints, a full background check, references and a "letter of purpose" along with permission from your local police chief to get.
However, the younger brother was too young to get a permit, the older brother was in fact not a US citizen and was here on a green card and thus is ineligible for a gun permit. Having said that, even if he was a US citizen his previous arrest for domestic assault would've rendered him ineligible for/and or revoked an existing LTC-A had he had one.
I have not seen anything yet reported as to what kind of long gun(s) they had or where they got them. The FBI warned agents responding to Watertown on Thursday night that they had "fully automatic AK47s" [along with grenades/dynamite and IEDs]. It is my understanding that they only recovered two handguns at the scene of the firefight in Watertown and one handgun and a long gun were found on the boat after the final surrender [aside from the IEDs scattered over Watertown].
It is interesting that despite the incredible firepower the brothers had access to (handguns, long gun(s), multiple IEDs), that when there's someone shooting back, the brothers plans were pretty much shut down and the loss of life/injury was very minimal considering the circumstances.
My big questions is why were the suspects faces in the database? Why would a 19 year old, never done anything face be in the facial recognition database?
I'm going to venture to guess that either a) the officer was talking out of his ass to make a quick point or b) someone, somewhere massively scraped every social network they could think of for anyone with ties to the area or the event. That being said, I don't really like the sound of it either, regardless of how true it is.
--pre-submit thought: perhaps it was the older brother? Our background is rather limited but it would not be surprising if he had had prior run-ins with the law. Even a minor offense will get your mugshot in the file.
Funnily enough, if you download your information using Facebook's tools, you get a copy of your facial recognition data too. Presumably US law enforcement has access to that for their own uses.
Facial recognition is still treated like some magical technology, like the "zoom in, enhance" feature much mocked about on TV crime shows. I bet Facebook's technology is ahead of law enforcement's in terms of data, algorithm, and refinement, and yet it still stumbles on pretty obvious faces (if you have the default auto-tag feature left on)
I can understand that recording live footage with very high quality is a problem, but how about capturing quality stills to complement the video with lower quality.
Someone with knowledge of cameras please either correct or endorse this, but I think the largest price/quality tradeoff in any camera happens at the lens choice, and given a cheap lens, the stills can't be much better than the video.
A lot of it has to do with the fact that most security cameras are not setup for capture of facial details (or license plates for that matter).
Most security cameras are setup for general overview shots, with around 10-20ppf (pixels per foot), whereas good detail starts at around 80ppf (meaning that you'd need 4x as many cameras in most areas to start to get something approaching an image quality that you could use for reliable facial recognition.
On top of that there is the storage factor. A "common" camera in a new deployment is a 1080p (2.1 Megapixel) camera. If this is set to record good quality images 24/7, it consumes about 60GB of storage per day. Various tricks like "record on motion", etc. can reduce this to maybe 10GB/day, at the risk of potentially missing key evidence or having gaps in the recording.
Lenses are important, but it's not that hard to find a lens with sufficient resolution for these cameras. A good lens rated at 2+ megapixels is around $70-$120 (depending on various factors), and a good outdoor camera (including lens) can be had for $500-$1200. Big price ranges, yes, but not to the point that the lenses are the limiting factor budget-wise.
I think that is right (I can't claim much specific knowledge on camera lenses though), it used to be especially bad with the whole "megapixel wars" a few years back. Pixel resolution increased, but image clarity didn't, and you would get much better images on old low-megapixel cameras with relatively high-end lenses than with a cheap but new camera of approx 4-8x the resolution in pixels.
However, nowadays I am quite amazed by the picture quality of the average smartphone camera. So maybe something changed? Still, we're talking about better lenses/quality than smartphone photos. They may be sharp, but that comes from the advantage of being up close and coming in huge numbers.
And in addition to the lens, there's a lot of other expensive parts in a security system, that may need to be replaced or reconfigured because it is all integrated with the current low-res cameras. The bigger such a system is, the more expensive it becomes to change it all up when better tech has come around, even if the new parts are relatively cheap by themselves.
certainly the lens and sensor are the biggest trade offs with price these would have the biggest impact with image clarity and the amount of light taken in to take the image.
Dazzle camouflage isn't meant to hide you in your surroundings, but rather make it more difficult to assess the geometry -- how large you are, where you are heading, etc.
It's more for confusing the lion running after the zebra, than an attempt hide from the lion.
Bottom line is that it's nearly impossible to exactly match facial features to anyone beyond a set distance when there's too much ambiguity between distances and rubbish quality CCTV video.
Matching gait may be a more effective means of identification at a distance (and, in this and many other cases, at odd angles).
There are a few, though less than you might think.
Lots of general progress has been made in terms of facial detection algorithms, but there are almost no "general" security camera deployments that are setup to capture sufficient detail to get a good face match.
There are not enough events like this in open places that drive requirements for facial recognition use. Limited cases (airports, casinos, etc.) more frequently have cameras setup to get good straight-on high-res facial shots, but almost no typical outdoor area is covered that way.
