Revealed: how facial recognition has invaded shops – and your privacy
by Chris Frey for the Guardian
On a Thursday morning in February, there were more gawkers than shoppers swelling the aisles for the debut of the new Saks Fifth Avenue in Toronto’s Eaton Centre. Given crowd reactions to the opulent surroundings – hanging clouds of glass sculptures, silk area rugs and theatrically posed mannequin displays – it might have been the opening of a museum’s latest blockbuster exhibition.
An influx of high-end US retailers are betting large on what they feel is an underserved luxury market in Canada’s biggest city. Saks has just opened two stores here; Nordstrom will launch in the fall. Both hope to inject some American-style flair to the staid world of Canadian retail. “Saks is more than just things,” company president Marc Metrick told the Toronto Star. “Saks is a dream. Saks is a feeling.”
Specifically, Saks is the feeling of being watched. Its new stores, which have been built to the standards of an impregnable, hi-tech fortress, employ some of the world’s most aggressive, cutting edge approaches to in-store security and customer surveillance. If anonymity is a traditional virtue of cities, it potentially goes out the window any time you enter a Saks shop – and a host of others.
“None of the other retail work we’ve done has anywhere near this kind of security,” says a source with direct knowledge of Saks’s building requirements, and who has worked extensively with other big Canadian and international retail chains.
The windows are fitted with hurricane-proof glass, the interior walls internally reinforced with wire cage. At Saks’ Sherway Gardens location, steel bollards capable of stopping a vehicle – and the so-called “smash and grab” – have been installed outside. So have roller security grilles, and measures to curtail employee theft: the staff lockers have clear plexiglass doors to ensure the contents inside are visible. The company’s “asset protection” book, revealed to the Guardian, is meticulously detailed, down to the make and model of almost every screw, bracket and hinge.
“Some of the doors you couldn’t get through with a police battering ram,” says the source. “It’s like there’s an Oceans Eleven-type scenario going on somewhere, planning this break-in into the Saks to steal all the luxury handbags.”
Of course, it shouldn’t necessarily surprise that a high-end retailer would go to such lengths to protect their merchandise. Organised retail crime accounts for about half of the $4.6bn a year in “shrinkage” reported by Canadian companies. Premium prices – say, $20,700 for a pair of diamond Kwiat earrings – can pay for premium security.
Saks, though, has also been an early adopter of a more controversial and potentially intrusive tactic in retail loss prevention – facial recognition software, a technology that until recently was, in the physical world, largely employed by airport security, border agents, casinos and cops.
In its commercial retail application, faces of individuals caught on camera are converted into a biometric template and cross-referenced with a database for a possible match with past shoplifters or known criminals. Some stores in the US give shoplifting suspects the option of allowing themselves to be photographed, rather than arrested. All this had been made possible by the arrival of networked, high-resolution security cameras and rapidly advancing analytical capabilities.
In a presentation at a loss prevention conference in Fort Lauderdale in 2014, Patrick McEvoy, the senior manager of asset protection systems for Saks’ parent company, Hudson’s Bay Company, shared the retailer’s initial experience with facial recognition software. Though he identified some limitations with the emerging technology, McEvoy found that the “desired results were definitely obtainable”. The cameras in all Saks stores are networked so as to be viewable at company headquarters in New York.
The same technologies used to identify thieves, however, can also be used to track customers.
Indeed, video surveillance is increasingly doing double duty to help stores improve sales. Many of the companies providing facial recognition capabilities, such as Axis Communications (which supplies Saks with IP cameras), Japanese tech giant NEC and Face First of California, either directly or through partners, also offer sophisticated analytics applications that capture your in-store “dwell times”, responses to product displays and traffic flow.
“For the management of a store there are two issues: minimise your losses and increase sales,” says Al Shipp, CEO of San Francisco-based 3VR, a provider of video intelligence software. “Our platform will help you do both.”
In other words, Shipp says, “the same platform that investigators use to identify shoplifters [also] helps retailers figure out which displays are working better, and where the customer traffic is.”
For traditional brick and mortar retailers, this sort of intel has helped level the playing field with online sellers, who routinely track and profile their customers through cookies. But in-store metrics – like dwell times – typically preserve your anonymity. Much of the excitement about facial recognition comes with putting a face to all those aggregated bits of data collected through loyalty programmes, point of sale records and other sources. Make a purchase at a Saks store and the clerk will likely ask for a driver’s license or credit card, presumably to create a customer profile.
In Europe, a number of high-end hotels and retailers are reportedly using facial recognition to help identify VIPs and celebrities for preferred treatment when they enter the front door. In the UK, a 2015 survey of 150 retail executives by the IT services firm Computer Services Corporation suggested that a quarter of all British shops use facial recognition. Fashion retailers have been especially keen: the survey reported that 59% use some form of facial recognition.
If retail brands are already embracing mobile marketing – sending us tailored messages, push notifications and ads to enhance the “customer experience” – then pull on this thread long enough and you arrive at the discomforting scenario depicted in the sci-fi film Minority Report, in which shoppers are identified with optical sensors and bombarded with personalised advertisements. In this creepy version of the future we’ll shop and be shopped, both at the same time.
Worried about how all this looks, big retailers have been careful to avoid any suggestion that they’re matching faces with data – or at least to skip the question. Saks declined to comment on whether they use facial recognition software.
For Geoff White, a privacy specialist with the Public Interest Advocacy Centre in Ottawa, that’s a problem. Companies using the technologies have an obligation to be more transparent .
“What happens once you’ve stored someone’s data with a profile based on their face?” he says. “How safe is it from hackers? Who has access to that information, and what other types of information is it being correlated with?”
Most valuable to retailers, he adds, “are the predictive analytics that come from tying that one piece of information, your face, to your whereabouts, to your patterns, to the way I move through time and space.”
Discerning our patterns in time and space might be the easiest part of retailers’ profiling efforts – thanks to our smartphones. White points to Apple’s popular iBeacon technology that’s able to connect with a customer’s device through a Bluetooth signal, or the now common practice of stores offering free WiFi. (Saks has been offering free WiFi in all its stores since 2012, making it among the first major US retailers to do so.) In either case, the location-based data on our phones could be accessible over these networks. If you own an iPhone and leave your location services on, dig deep within the settings submenus and you’ll find something called “Frequent Locations” – a highly personal accounting of your movements: where you go, when and for how long.
“People tend to think their location is a random thing,” says White, “but tons of research shows that your location is unlike anyone else’s. My phone knows where my home is even though I never told it. It’s really intimate, and from it you can infer a lot of things.”
“There are a lot of good reasons for a store to offer free WiFi, but it’s a question of what data the customer is giving up to gain that access. In a lot of big shopping malls you log [into WiFi] using your Facebook or Twitter account. A tremendous exchange of data in such a simple log-in.”
Online, both Facebook and Google have been on the forefront of developing facial recognition algorithms. The first time you uploaded and tagged a party selfie you signed up for the programme. For many consumer and privacy advocates, however, the use of facial recognition and biometric profiling in the physical world is more troubling. You can delete your cookies and your web browser’s cache, and control your social media or phone’s privacy settings. But there are no privacy settings for walking down a city street, or browsing in a mall. And you can’t delete your face.
What’s more, current laws do little to constrain what the private owners of publicly accessible spaces in cities can do with the information they gather from watching you.
Talks convened by the US Commerce Department last summer to devise voluntary guidelines over the commercial use of facial recognition came to an abrupt halt when consumer and privacy advocates walked out.
One of the walkouts was Alvaro Bedoya, a Georgetown law professor and privacy advocate. He says they became convinced that industry simply wasn’t interested in getting a customer’s consent when enrolling them in an facial recognition programme.
“This software is being deployed in a very opaque manner,” he says, “and it could be rife with problems. But we have no legal ability in the US (outside of Texas and Illinois) to look under the hood and figure out what’s going on.”
“The irony in the US is that we think we’re living in this dark era of government surveillance, but in reality there are structures to combat government overreach. To this day you can get a lot of information out of law enforcement on their use of facial recognition by using the Freedom of Information Act. There is no FOIA for tech or facial recognition companies. We don’t have a way to keep tabs on these companies and rely, frankly, on journalists and inside leaks.”
Bedoya even questions the use of facial recognition as a tool for store security and identifying “bad actors”.
“There are various anecdotal cases where the [facial recognition] software has not performed in commercial settings as well on African Americans as it has on white people,” he says.
Citing a 2014 case in which the luxury retailer Barneys agreed to pay $525,000 to settle a racial profiling complaint, he adds: “Already African Americans suffer from profiling in a lot of these stores. So when you’re deploying a software that anecdotally at least has proven not to function as well on precisely the people that are already targeted, that’s a problem.”
In cities anonymity is becoming an increasingly antiquated idea, as more of the environments we spend time in – streets, shops, malls – marry biometrics with big data for the purposes of tracking our habits and tastes. When strolling the local high street or financial district, you can assume there are thousands of security cameras looking at you. Walk into a shop, and you can also assume there are cameras to prevent theft.
According to White, your assumptions should stop there.
“Do you know if you’re also being photographed so that a database can be used to profile you? Do you know the end uses that the information is being put to? I don’t think people are fully aware the extent to which their personal information is being collected.”