New York City’s subway system, a 24/7 behemoth that logs a billion and a half trips per year, is synonymous with archaic technology, from a signals system that dates to the Great Depression era to rail cars in service for more than four decades. The introduction last year of OMNY, a $574 million new contactless payment system for city buses and subways, bucks that trend. 

However, experts say the OMNY payment scheme is rife with problems, based on the limited information about the system made public in its terms of service and privacy policy. The collection of significant amounts of information from users, including smartphone device identifiers and location, which, coupled with payment and transportation data, could be used to map out riders’ patterns of life in minute detail and create a privacy nightmare.

Created for the MTA by Cubic Corporation, OMNY uses near-field communication (NFC) technology to enable tap payment at turnstiles via debit cards, smartphone payment apps, and eventually a loadable card such as those used by transit riders in London, Sidney, San Francisco, and Washington, DC. Cubic has created NFC card payment systems for transit systems in San Diego, Sydney, Vancouver, and the Bay Area in recent years, and is also expected to debut mobile payment apps on the Chicago Transit Authority later this year.

This replacement for the venerable MetroCard (the magnetic stripe swipe card introduced in 1992 to replace the subway token) will supposedly speed up bus service and entry to the subway system – and spare countless out-of-towners the embarrassment of not knowing how to correctly swipe in at a turnstile. 

In addition to privacy concerns, there are also questions related to the security of such data, whether OMNY could be used by the MTA to unilaterally exclude people from New York City’s transit system, and language in the payment system’s terms of service that indemnifies the MTA from liability for customers being double-charged for rides.

The problems have been exacerbated by the MTA’s refusal to engage with questions about different aspects of OMNY, even as the payment system is being rapidly introduced throughout the city. By the end of 2020, OMNY validators will be in every subway station and bus in the city. Early next year, the new payment system will be introduced to the Metro North and Long Island Rail Road commuter lines. 

Riders of New York City subways and buses are well accustomed to the reality that they are being tracked on public transportation. Surveillance cameras proliferated as anti-terrorism and crime measures in the years after 9/11, while MetroCard data is routinely used by police to recreate a criminal suspect’s movements.

NYPD officers and district attorney investigators have for years used judicial subpoenas to retrieve MetroCard information stored by the MTA to track criminal suspects. One such instance involved the murder of a baby boy by his father, who worked as a subway cleaner. Detectives used the man’s MTA-issued MetroCard to track his movements from Co-Op City in the Bronx to lower Manhattan, where he allegedly threw his son’s body into the East River. 

However, the introduction of a payment system that ties a rider’s movements not only to their bank card, but potentially their smartphone via payment apps creates a raft of privacy and data security issues. In OMNY’s privacy policy, the MTA states that information including, but not limited to, payment information, billing address, and the point of entry to the transit system will be logged in Cubic Corporation’s servers. 

Steve Brunner, Cubic’s general manager for the tri-state area, said the firm had multiple local data centers to safeguard against losing information in a catastrophic event. “If there is an outage or failure of a component at one data center, it will automatically either partially or fully roll over to the other data center,” Brunner said in an interview with The Verge last year.

In addition, the privacy policy authorizes the MTA and Cubic to retain the data for an indefinite period — the MTA claims that it stores transaction information for six months, but keeps other portions of the transaction information for up to seven years. Riders can log in to their OMNY account and review their movement history for the 90 days prior.

Privacy advocates say that OMNY’s retention of individual rider data and smartphone device identifiers for an indeterminate period that could run over half a decade warrants greater disclosure and public discussion. 

“If you’re using OMNY on your phone – there’s no card yet – it’s not clear to me what other information they’re taking from your phone or how that can identify you,” said Jerome Greco, a staff attorney at the Legal Aid Society’s digital forensics unit who specializes in surveillance technology.

OMNY’s privacy policy also includes a carve-out for the collection of additional information “that is not specifically listed” in the document, allowing the transit authority broad leeway to harvest additional data from riders. According to the MTA, such information includes IP addresses and device numbers from phones used to pay for rides, creating a whole new category of sensitive information that could be used either to push advertisements toward riders or track their movements outside of the transit system via Bluetooth, Wi-Fi, or their device’s MAC address.

The MTA maintains that it retains all information securely with triple DES encryption and that such data is never decrypted. 

“Our transactions are encrypted from the moment you touch the validator,” said Al Putre, the MTA’s program director for OMNY. “We keep them in an encrypted state even when we store it in our account-based processor. We use state of the art encryption methods and security module hardware. We do absolutely everything we can do to maintain the integrity of the transaction to ensure it’s secure. If we have just one little glitch, our credibility goes out the door.”

Indeed, the OMNY terms of service contain specific language that admits riders run the risk of incursions to their privacy by using the payment system. “Security risk is inherent in all internet and information technologies, and we cannot guarantee the security of your Personal Information,” the policy reads. While the MTA maintains this is standard contractual language for information technology products, it is telling straphangers they will be sacrificing privacy for convenience.

Cubic, the company in charge of designing and implementing the OMNY system, has run into problems around data security before: in San Francisco, information from its contactless payment system for the Muni light rail system was hacked and held ransom for $73,000 in Bitcoin, forcing the system to let riders use it for free. Last year, London’s Oyster payment system was taken offline after a credential stuffing spree compromised the accounts of an untold number of riders.

“We’re definitely concerned about issues on privacy and how the MTA is using data,” said Jaqi Cohen, the campaign direction for the New York Public Interest Research Group’s Straphangers Campaign. “Any way the MTA is planning on using and protecting data should be known to riders and the public – the terms of service should not be hidden from the riders and the way the MTA plans to use these data should be made very explicit.”

The MTA is already facing an open records lawsuit in New York regarding its unannounced deployment of facial recognition technology in the Times Square station last Spring. In London, where former New York City Transit President Andy Byford drew much inspiration for his projects, Cubic has already tested facial recognition options for payment.,The MTA denies that facial recognition is being considered for any integration into the OMNY payment system. 

The retention of cellphone device identifiers by OMNY was singled out by advocates as a significant matter for concern. Law enforcement makes particular use of cellphone location data to identify and track persons of interest. In New York City, US Immigration and Customs Enforcement agents use cell-site simulators to track down undocumented immigrants by their cellphones. ICE has shown an appetite for both criminal justice and transportation data to locate deportation targets nationwide, and recently subpoenaed New York City authorities for data on four people slated for deportation.

“If they’re able to single out your individual phone, then can they get more data from your phone company or iCloud backup, and those would require warrants,” said Greco of the Legal Aid Society. However, he pointed out that OMNY’s privacy policy does not require a warrant to turn information over to law enforcement.

Much like how cashless payments have come under fire for discriminating against people without bank accounts or mobile payment apps or debit cards, OMNY’s implementation is running into questions over access and equity. Last summer, in the early stages of the new payment system’s rollout, riders who pay with a MasterCard debit were reimbursed $5.50 every Friday. In other transit systems run by Cubic, customers can earn fare discounts by watching ads on their cellphone.

At a moment when the MTA and Governor Andrew Cuomo are taking a hard line with fare evasion, the idea that OMNY’s promotions are effectively subsidizing wealthier riders’ trips has proven galling for some.

“We’re creating a system where wealthy riders pay less while Cuomo is deploying an army to crack down on black and brown riders,” said Albert Fox Cahn, the director of the Surveillance Technology Oversight Project, which issued a critical report on OMNY last year and filed the open records suit over the MTA’s use of facial recognition.

Transit advocates say many low-income riders often pay more per ride because they cannot afford to purchase the weekly or monthly passes that cost riders less money per rides. The MTA has said it will continue providing discounted rides for students and seniors, as well as the discounted weekly and monthly cards, in the coming months. 

With regard to fares, the MTA has also included language in its terms of service that indemnify the transit agency for accidental double payments, several of which have recently taken place when riders swiped in to the subway system with a MetroCard, only for their cellphone’s Apple Pay app to accidentally deduct a $2.75 ride from their account after coming into contact with the display. 

The relevant passage from OMNY’s terms of service essentially blames riders for failing to properly use their devices, and states that the “MTA is not responsible if your fare is charged to a card or through a smart device that you did not intend to use.” The MTA maintains such language is necessary to indemnify the transit authority against fraud, the faulty double payment was caused by Apple’s unannounced update last November to the Express Transit mode payment option, and it has fully refunded fares for all the roughly 500 instances of double payment. However, transit advocates are not satisfied with the response.  

“It’s particularly outrageous that there’s explicit language in the Terms of Service saying that if you don’t pay, it’s your fault, while the MTA is claiming fare evasion is a huge issue and using it to hire 500 new cops,” said Cohen from NYPIRG’s Straphangers’ Campaign. “How much [money] has been collected in error?” 

Aside from concerns over surveillance and functionality, OMNY’s terms of service also hint that the MTA is looking to use the tap payment system as a new method to unilaterally exclude people from city subway stations, buses, and commuter railways.

Access to the transit system, according to OMNY’s terms of service, can be blocked for “suspicion of other illegal activity, in MTA’s sole discretion.” What’s more, the MTA claims the right to suspend access to OMNY “if you engage in activity that we conclude, in our sole and absolute discretion, breach our code of conduct.” Behaviors deemed illegal by the MTA in recent years include putting your feet up on a seat, sleeping on the train, or passing between subway cars.

In response to queries by The Verge, the MTA’s Putre said the terms of service language would be amended to remove prohibitions on people accessing the transit system. 

“The purpose of OMNY is to provide our customers with an easy and convenient way to pay the fare and we are committed to protecting NYC Transit riders’ privacy and preventing fraud,” Putre said in a statement. “For clarity and effective immediately, the OMNY Terms of Service have been amended to remove references to actions that might summarily prohibit access to OMNY services — a provision that has never been used. The Terms, as they did previously, will continue to protect customers from fraudulent use of their accounts by allowing interruption of OMNY charges in that situation.”

“By putting your feet up or falling asleep, you could get your OMNY account suspended,” said Cohen. “That’s why the MTA needs to be transparent and explain to the public how this will work.”

“Public transit is public space, it’s part of the public sphere. The idea of banning anyone from public transit raises prominent constitutional issues for us,” said Daniel Pearlstein, the policy and communications director for the New York Riders Alliance, a transit advocacy organization.

Pearlstein said that the possibility of the MTA issuing unilateral bans to individuals for perceived offenses outside the criminal justice systems could amount to a de facto form of segregation.

“We are skeptics around the MTA’s narrative on fare evasion. Their rhetoric is about blaming low income riders of color for the ills of a transit system that are overwhelmingly the fault of powerful people going back a generation.”

To date, individual exclusion from public transit is not something that has taken place outside of specific criminal cases. NYPD Commissioner Dermot Shea has pushed legislators in Albany to pass a law banning repeat sex offenders from using city subways. However, the MTA’s codification of unilateral authority to ban people for incidents that may not even rise to the level of criminality may also run into problems around due process.

“Here, they’re just talking about suspicion. They’re not talking about people who’ve been convicted: this is suspicion by the MTA. The MTA becomes the judge, jury and executioner,” said Greco of the Legal Aid Society.

“It seems to be even more egregious if it is in the MTA’s sole discretion. How do I appeal that? How do they make that determination? Who makes that determination? What standards are they using? Is this going to become like the no-fly list?”

In May and June 2013, when New Orleans’ murder rate was the sixth-highest in the United States, the Orleans Parish district attorney handed down two landmark racketeering indictments against dozens of men accused of membership in two violent Central City drug trafficking gangs, 3NG and the 110ers. Members of both gangs stood accused of committing 25 murders as well as several attempted killings and armed robberies.

Subsequent investigations by the Bureau of Alcohol, Tobacco, Firearms and Explosives, the Federal Bureau of Investigation, and local agencies produced further RICO indictments, including that of a 22-year-old man named Evans “Easy” Lewis, a member of a gang called the 39ers who was accused of participating in a drug distribution ring and several murders.

 

This article was reported in partnership with The Investigative Fund at The Nation Institute.

According to Ronal Serpas, the department’s chief at the time, one of the tools used by the New Orleans Police Department to identify members of gangs like 3NG and the 39ers came from the Silicon Valley company Palantir. The company provided software to a secretive NOPD program that traced people’s ties to other gang members, outlined criminal histories, analyzed social media, and predicted the likelihood that individuals would commit violence or become a victim. As part of the discovery process in Lewis’ trial, the government turned over more than 60,000 pages of documents detailing evidence gathered against him from confidential informants, ballistics, and other sources — but they made no mention of the NOPD’s partnership with Palantir, according to a source familiar with the 39ers trial.

The program began in 2012 as a partnership between New Orleans Police and Palantir Technologies, a data-mining firm founded with seed money from the CIA’s venture capital firm. According to interviews and documents obtained by The Verge, the initiative was essentially a predictive policing program, similar to the “heat list” in Chicago that purports to predict which people are likely drivers or victims of violence.

The partnership has been extended three times, with the third extension scheduled to expire on February 21st, 2018. The city of New Orleans and Palantir have not responded to questions about the program’s current status.

Predictive policing technology has proven highly controversial wherever it is implemented, but in New Orleans, the program escaped public notice, partly because Palantir established it as a philanthropic relationship with the city through Mayor Mitch Landrieu’s signature NOLA For Life program. Thanks to its philanthropic status, as well as New Orleans’ “strong mayor” model of government, the agreement never passed through a public procurement process.  

In fact, key city council members and attorneys contacted by The Verge had no idea that the city had any sort of relationship with Palantir, nor were they aware that Palantir used its program in New Orleans to market its services to another law enforcement agency for a multimillion-dollar contract.

Even James Carville, the political operative instrumental in bringing about Palantir’s collaboration with NOPD, said that the program was not public knowledge. “No one in New Orleans even knows about this, to my knowledge,” Carville said.

More than half a decade after the partnership with New Orleans began, Palantir has patented at least one crime-forecasting system and has sold similar software to foreign intelligence services for predicting the likelihood of individuals to commit terrorism.

Even within the law enforcement community, there are concerns about the potential civil liberties implications of the sort of individualized prediction Palantir developed in New Orleans, and whether it’s appropriate for the American criminal justice system.

“They’re creating a target list, but we’re not going after Al Qaeda in Syria,” said a former law enforcement official who has observed Palantir’s work first-hand as well as the company’s sales pitches for predictive policing. The former official spoke on condition of anonymity to freely discuss their concerns with data mining and predictive policing. “Palantir is a great example of an absolutely ridiculous amount of money spent on a tech tool that may have some application,” the former official said. “However, it’s not the right tool for local and state law enforcement.”

Six years ago, one of the world’s most secretive and powerful tech firms developed a contentious intelligence product in a city that has served as a neoliberal laboratory for everything from charter schools to radical housing reform since Hurricane Katrina. Because the program was never public, important questions about its basic functioning, risk for bias, and overall propriety were never answered.

Co-founded in 2004 by Alexander Karp and Peter Thiel (the company’s single largest shareholder), Palantir Technologies’ rapid ascent to becoming one of the highest-valued private Silicon Valley companies has been driven by lucrative contracts with the Pentagon and United States intelligence services, as well as foreign security services. In recent years, Palantir has sought to expand its data fusion and analysis business to the private sector, with mixed success.

Prediction is not new territory for Palantir. Since at least 2009, Palantir was used by the Pentagon to predict the location of improvised explosive devices in Afghanistan and Iraq — a wartime risk-assessment program absent the civil liberties concerns that come with individualized predictive policing. Its commercial software platform, Metropolis, also reportedly uses predictive analytics to help businesses develop consumer markets and streamline investments. But before 2012 with the New Orleans program, there is no publicly available record that Palantir had ventured into predictive policing.

Interest and investment in predictive policing technology accelerated after 2009 when the National Institute of Justice began issuing grants for pilot projects in crime forecasting. Those grants underpin some of the best-known — and most scrutinized — predictive policing efforts in Chicago and Los Angeles. Programs vary, and the algorithms are often proprietary, but they all aim to ingest vast stores of data — geography, criminal records, the weather, social media histories — and make predictions about individuals or places likely to be involved in a crime. In the following years, many startup firms have struggled to monetize the crime-fighting method — most notably PredPol, a California startup whose contract awards have foundered after an initial blitz of publicity in the early 2010s.

As more departments and companies began experimenting with predictive policing, government-funded research cast doubts on its efficacy, and independent academics found it can have a disparate impact on poor communities of color. A 2016 study reverse-engineered PredPol’s algorithm and found that it replicated “systemic bias” against over-policed communities of color and that historical crime data did not accurately predict future criminal activity. One of the researchers, a Michigan State PhD candidate named William Isaac, had not previously heard of New Orleans’ partnership with Palantir, but he recognized the data-mapping model at the heart of the program. “I think the data they’re using, there are serious questions about its predictive power. We’ve seen very little about its ability to forecast violent crime,” Isaac said.

According to interviews and documents obtained by The Verge, Palantir first approached New Orleans in 2012 through a well-known intermediary: James Carville, the Democratic Party power broker and architect of Bill Clinton’s successful 1992 presidential campaign. Carville is a paid adviser of Palantir whose involvement with the data-mining company dates back at least to 2011.  

In an interview, Carville told The Verge that he was the impetus for the collaboration between Palantir and New Orleans. “I am the sole driver of that project. It was entirely my idea,” said Carville, adding that he and Palantir CEO Alex Karp flew down to New Orleans to meet with Mayor Landrieu. “To me, it was a case of morality. Young people were shooting each other, and the public wasn’t as involved as they should have been.”

The documents outlining Palantir’s relationship with New Orleans describe the company’s role as “pro bono” and philanthropic. In 2015, Palantir mentioned its work in New Orleans in its annual philanthropic report, characterizing the effort as collaborative “network analysis” for law enforcement and other city stakeholders.  

Carville’s remarks on a Bay Area public radio station four years ago elucidate how Palantir’s relationship with the city came about. In a January 2014 appearance on KQED’s Forum talk show, Carville and his wife Mary Matalin touted Palantir’s work in New Orleans as a major driver in the city’s two-year decline in the murder rate.

“The CEO of a company called Palantir – the CEO, a guy named Alex Karp — said that they wanted to do some charitable work, and what’d I think? I said, we have a really horrific crime rate in New Orleans,” Carville told KQED Forum’s host Michael Krasny, without mentioning his professional relationship to Palantir. “And so he came down and met with our mayor… they both had the same reaction as to the utter immorality of young people killing other young people and society not doing anything about it. And we were able to, at no cost to the city, start integrating data and predict and intervene as to where these conflicts were going to be arising. We’ve seen probably a third of a reduction in our murder rate since this project started.”

Matalin, who is also a political consultant, made it clear to Krasny that the prediction work being done with NOPD by the Palo Alto firm was both a prototype and potentially could sweep up innocent people.

“We’re kind of a prototype,” said Matalin. “Unless you’re the cousin of some drug dealer that went bad, you’re going to be okay.”

Ronal Serpas, the New Orleans chief of Police from 2010 through August 2014, recalled his initial contact with Palantir’s staff during a meeting initiated by Mayor Landrieu’s office. “They came over and discussed the kind of work they do in theaters of war, the kind of work they do in other parts of the world,” Serpas said during an interview in his office at Loyola University. “My impression was Palantir was also interested in trying to develop products that could do some predicting of crime.”

The relationship between New Orleans and Palantir was finalized on February 23rd, 2012, when Mayor Landrieu signed an agreement granting New Orleans free access to the firm’s public sector data integration platform. Licenses and tech support for Palantir’s law enforcement platform can run to millions of dollars annually, according to an audit of the Los Angeles County Sheriff’s Department.

In January 2013, New Orleans would also allow Palantir to use its law enforcement account for LexisNexis’ Accurint product, which is comprised of millions of searchable public records, court filings, licenses, addresses, phone numbers, and social media data. The firm also got free access to city criminal and non-criminal data in order to train its software for crime forecasting. Neither the residents of New Orleans nor key city council members whose job it is to oversee the use of municipal data were aware of Palantir’s access to reams of their data.

Palantir has a history of secrecy, and New Orleans is not the only instance of the company conducting business with government agencies through associated nonprofits, avoiding the public procurement process.

Palantir provides data analysis and integration for the Los Angeles Police Department, but the arrangement was made through the LA Police Foundation rather than the LAPD itself. In New York, the firm’s contract was not disclosed by the city comptroller for security reasons (NYPD does this with surveillance equipment contracts), and it was never brought to the city council for approval. Palantir’s work with NYPD only became public when documents about its tumultuous relationship with the country’s largest police force were leaked to BuzzFeed reporter William Alden.   

In New Orleans, according to extensive reporting by The Verge, Mayor Landrieu’s office, the city attorney, and the NOPD appear to be the only entities aware of the firm’s work in the city. Key members of the city council were not aware of Palantir’s work in New Orleans until approached by The Verge.

The Palantir partnership would have likely received more scrutiny from the city council had it been itemized in a budget, but the council’s approval isn’t required for such a program. The structure of city government in New Orleans is predicated on a “strong mayor” model where the council does not have approval authority over contracts or policies for the city police department.

Cities around the country have recently begun to grapple with the question of if and how municipalities should regulate data sharing and privacy. Some cities like Seattle and Oakland have passed legislation establishing committees to craft guidelines and conduct oversight, while others like New York are discussing what role city councils should play regarding privacy in the digital age.

Several civil and criminal attorneys who are heavily involved with the New Orleans’ criminal justice system were also unaware of any predictive policing efforts by the NOPD. Multiple criminal attorneys had never seen Palantir analytic products as part of any discovery materials turned over to them in the course of trial cases, although such analysis would typically be required to be given to defense counsel if it had been used as part of an NOPD investigation.  

Jason Williams, the president of the New Orleans city council and a former defense attorney, reviewed documentation of Palantir’s collaboration with NOPD at the request of The Verge. Williams said he had never heard of the company’s involvement with NOPD.

“I don’t think there’s anyone in the council that would say they were aware that this had even occurred because this was not part and parcel to any of our budget allocations or our oversight,” Williams said in an interview during a council meeting.

Williams, who also served as a criminal court judge before his election to the city council in 2014, said that he wasn’t necessarily opposed to using data-driven methods to help at-risk New Orleanians.

“My primary concern would be how this was used in my city. If it was used to identify marginalized people that are at risk of being harmed, to stop them from being harmed, I’m going to have a whole different appreciation of that than I’m gonna have if this system was used nefariously.”

Councilwoman Susan Guidry, who chairs the council’s criminal justice committee and has been in office since 2010, was also unaware of New Orleans’ partnership with Palantir and NOPD’s crime-forecasting work. When shown NOPD documentation of the program, Guidry told The Verge she had never encountered it before.

The Verge shared documentation of the program with a group of New Orleans civil rights attorneys. None were previously aware of NOPD’s prediction work — though one had heard rumors that Palantir was collaborating with NOPD — and they were troubled by the secrecy that surrounded the program.

“It’s especially disturbing that this level of intrusive research into the lives of ordinary residents is kept virtually a secret,” said Jim Craig, the director of the Louisiana office of the Roderick and Solange MacArthur Justice Center. Craig, who reviewed documentation of the program at The Verge’s request, compared the predictive policing effort to signals intelligence work. “It’s almost as if New Orleans were contracting its own version of the NSA to conduct 24/7 surveillance of the lives of its people,” Craig said. Authorities, he believes, have kept the program under wraps because it would elicit widespread outrage. “Right now, people are outraged about traffic cameras and have no idea this data-mining project is going on,” Craig said. “The South is still a place where people very much value their privacy.”

Nicholas Corsaro and Robin Engel are two University of Cincinnati professors who conducted a recent evaluation of the New Orleans’ violence reduction strategy that Palantir was used for, and helped design an NOPD gang database that the Palantir forecasting model draws on. Both Engel and Corsaro were unaware of New Orleans’ predictive policing efforts, its involvement with Palantir, or even the fact that the database they designed was feeding into the program. “Trying to predict who is going to do what based on last year’s data is just horseshit,” Corsaro said in an interview.

Palantir did sometimes publicly refer to its work in New Orleans. However, none of Palantir’s public presentations about the program that The Verge was able to identify went into detail about individualized crime forecasting, scraping of social media data, or the use of social network analysis for crime prediction. Instead, the company represented its work in New Orleans as “developing a better understanding of violent crime propensity and designing targeted interventions to protect the city’s most vulnerable populations.”

In a public speaking appearance where he touted the efficacy of their work in New Orleans, Courtney Bowman, a Palantir civil liberties engineer heavily involved with the company’s work with NOPD, acknowledged that excessive secrecy could deepen the rift between law enforcement and over-policed communities. During a May 6th, 2016 presentation at the UC Berkeley School of Information’s DataEdge conference, Bowman said, “These sorts of programs only work if the community is comfortable with the degree to which this type of information is being applied and if they’re aware of how the information is being used.”

The city of New Orleans and Palantir both declined requests for comment about how their partnership was formed, and what sort of input other elected officials and the public had into the data-mining firm’s predictive policing efforts.

Ronal Serpas, who ran NOPD when the partnership with Palantir began, said that he believed the city council and public at large should have been informed about the police department’s decision to engage in predictive policing with Palantir. The role of local legislatures and governing bodies in overseeing the sharing of government data is far from settled, but Serpas believes that agreements with firms like Palantir warrant greater scrutiny.

“It is, to me, something that certainly requires a view, requires a look,” Serpas said.

Though neither Palantir staff nor current New Orleans officials would talk about the day-to-day functioning of the crime-forecasting initiative, documents obtained by The Verge, external studies, and the recollections of former Chief Serpas offer a portrait of how the predictive policing beta test has functioned over the past six years.

Palantir’s prediction model in New Orleans used an intelligence technique called social network analysis (or SNA) to draw connections between people, places, cars, weapons, addresses, social media posts, and other indicia in previously siloed databases. Think of the analysis as a practical version of a Mark Lombardi painting that highlights connections between people, places, and events. After entering a query term — like a partial license plate, nickname, address, phone number, or social media handle or post — NOPD’s analyst would review the information scraped by Palantir’s software and determine which individuals are at the greatest risk of either committing violence or becoming a victim, based on their connection to known victims or assailants.

The data on individuals came from information scraped from social media as well as NOPD criminal databases for ballistics, gangs, probation and parole information, jailhouse phone calls, calls for service, the central case management system (i.e., every case NOPD had on record), and the department’s repository of field interview cards. The latter database represents every documented encounter NOPD has with citizens, even those that don’t result in arrests. In 2010, The Times-Picayune revealed that Chief Serpas had mandated that the collection of field interview cards be used as a measure of officer and district performance, resulting in over 70,000 field interview cards filled out in 2011 and 2012. The practice resembled NYPD’s “stop and frisk” program and was instituted with the express purpose of gathering as much intelligence on New Orleanians as possible, regardless of whether or not they committed a crime.

NOPD then used the list of potential victims and perpetrators of violence generated by Palantir to target individuals for the city’s CeaseFire program. CeaseFire is a form of the decades-old carrot-and-stick strategy developed by David Kennedy, a professor at John Jay College in New York. In the program, law enforcement informs potential offenders with criminal records that they know of their past actions and will prosecute them to the fullest extent if they re-offend. If the subjects choose to cooperate, they are “called in” to a required meeting as part of their conditions of probation and parole and are offered job training, education, potential job placement, and health services. In New Orleans, the CeaseFire program is run under the broader umbrella of NOLA For Life, which is Mayor Landrieu’s pet project that he has funded through millions of dollars from private donors.

According to Serpas, the person who initially ran New Orleans’ social network analysis from 2013 through 2015 was Jeff Asher, a former intelligence agent who joined NOPD from the CIA. If someone had been shot, Serpas explained, Asher would use Palantir’s software to find people associated with them through field interviews or social media data. “This data analysis brings up names and connections between people on FIs [field interview cards], on traffic stops, on victims of reports, reporting victims of crimes together, whatever the case may be. That kind of information is valuable for anybody who’s doing an investigation,” Serpas said.

According to Palantir’s own documentation, Asher and his colleagues ran social network analyses of every victim of a fatal or non-fatal shooting in New Orleans from 2011 through 2013. Through this technique, which Asher dubbed “The NOLA Model,” the city devised a list of roughly 3,900 people who were at the highest risk of being involved in gun violence because of their connection to a previous shooter or victim. “We can identify 30-40% of shooting victims,” Asher claimed at Palantir’s 2014 internal conference. Asher declined repeated requests for an interview.

Theoretically, Asher’s approach is substantially influenced by the research of Andrew Papachristos, a Yale professor who tracked violence as if it were a communicable disease spreading through networks of association. However, since his work was cited as the academic underpinning for crime-forecasting models employed by PredPol and the Chicago Police Department, Papachristos has sought to distance his research from those methods.

Once NOPD generated its list of likely shooters and victims, the police department and social service providers — for the “carrot” side of NOLA For Life — would select people who were either incarcerated or on court supervision for a “call-in meeting.”

Mayor Landrieu’s office touted the program frequently, referring to it as an essential part of New Orleans’ criminal justice policy. Palantir also claimed credit: “we’re helping to break the cycle of violence” in New Orleans, read a passage in the company’s 2015 Philanthropy Engineering report. But its actual impact is unclear.

Of the 308 people who participated in call-ins from October 2012 through March 2017, seven completed vocational training, nine completed “paid work experience,” none finished a high school diploma or GED course, and 32 were employed at one time or another through referrals. Fifty participants were detained following their call-in, and two have since died.

By contrast, law enforcement vigorously pursued its end of the program. From November 2012, when the new Multi-Agency Gang Unit was founded, through March 2014, racketeering indictments escalated: 83 alleged gang members in eight gangs were indicted in the 16-month period, according to an internal Palantir presentation.

Call-ins declined precipitously after the first few years. According to city records, eight group call-ins took place from 2012 to 2014, but only three took place in the following three years. Robert Goodman, a New Orleans native who became a community activist after completing a prison sentence for murder, worked as a “responder” for the city’s CeaseFire program until August 2016, discouraging people from engaging in retaliatory violence. Over time, Goodman noticed more of an emphasis on the “stick” component of the program and more control over the non-punitive aspects of the program by city hall that he believes undermined the intervention work. “It’s supposed to be ran by people like us instead of the city trying to dictate to us how this thing should look,” he said. “As long as they’re not putting resources into the hoods, nothing will change. You’re just putting on Band-Aids.”

After the first two years of Palantir’s involvement with NOPD, the city saw a marked drop in murders and gun violence, but it was short-lived. Even former NOPD Chief Serpas believes that the preventative effect of calling in dozens of at-risk individuals — and indicting dozens of them — began to diminish.

“When we ended up with nearly nine or 10 indictments with close to 100 defendants for federal or state RICO violations of killing people in the community, I think we got a lot of people’s attention in that criminal environment,” Serpas said, referring to the racketeering indictments. “But over time, it must’ve wore off because before I left in August of ‘14, we could see that things were starting to slide”

Nick Corsaro, the University of Cincinnati professor who helped build NOPD’s gang database, also worked on an evaluation of New Orleans’ CeaseFire strategy. He found that New Orleans’ overall decline in homicides coincided with the city’s implementation of CeaseFire program, but the Central City neighborhoods targeted by the program “did not have statistically significant declines that corresponded with November 2012 onset date.”

Put plainly, the study did not confirm claims by Palantir and city officials that data-driven interventions were behind the temporary drop-off in violent crime.  

Though the call-ins dropped off, emails obtained by The Verge indicate that the NOPD continued to use Palantir for law enforcement. Palantir declined repeated requests for comment, but the emails also show that the company was aware of the potential risks posed by predictive policing algorithms, and the negative publicity that comes with them. On May 23rd, 2016, Palantir civil liberties engineer Courtney Bowman responded to a request by NOPD crime analyst Zach Donnini about whether Palantir could help generate numerical rankings for individuals’ risk for committing or becoming the victim of a shooting.

“I have some serious concerns about instituting a ranking or numeric scoring approach,” Bowman wrote. “It’s exactly this facet of Chicago’s “heat” list model that has exposed CPD to a great deal of public scrutiny,” the email reads, linking to two articles critiquing Chicago’s predictive policing approach.

“The looming concern is that an opaque scoring algorithm substitutes the veneer of quantitative certainty for more holistic, qualitative judgement and human culpability,” Bowman wrote. “One of the lasting virtues of the SNA work we’ve done to date is that we’ve kept human analysts in the loop to ensure that networks are being explored and analyzed in a way that passes the straight-face test.”

Regardless of the sustainability of New Orleans’ murder reduction, Palantir used its work with the NOPD to solicit large contracts with other American cities. Later, the company won lucrative contracts for predictive programs with foreign governments.

According to emails obtained by The Verge, Palantir marketing staff first contacted the Chicago Police Department in late 2013 about the possibility of selling a predictive policing package based on the firm’s New Orleans work, eventually settling on a $3 million price tag. Through a series of federal grants awarded to CPD beginning in 2009, Chicago Police and academics at the Illinois Institute of Technology had already created their own crime-forecasting program that assigned a risk score to individuals based on criminal data and social media histories.

On August 19th, 2014, Katie Laidlaw, a marketing executive at Palantir, emailed Chicago Police commander Jonathan Lewin. “I would like to follow-up on connecting with Superintendent McCarthy, specifically to frame potential Palantir engagement around our proven outcomes in supporting homicide reduction in New Orleans,” Laidlaw wrote.

The emails also show that Chicago Police hoped to receive grant money from the Department of Homeland Security to fund the Palantir software acquisition. However, the Chicago Police Department never piloted or purchased Palantir’s software.

Commander Lewin, who is in charge of Chicago’s “heat list” model of predictive policing and who was on the receiving end of Katie Laidlaw’s sales pitch for Palantir, said in an interview that he was aware of Palantir’s work with other law enforcement agencies but never approved either a test run or purchase of Palantir software.

Though Palantir did not succeed in selling its New Orleans-tested tools to Chicago Police, the data-mining company has successfully sold forecasting products to foreign security services.

In 2016, the Danish national police and intelligence services signed an 84-month contract with Palantir — reported in the Danish press to have been worth between $14.8 and $41.4 million — for a predictive technology package meant to identify potential terrorists. According to procurement documents, the program uses law enforcement data like license plate reader records, CCTV video, and police reports to make predictions about individuals’ likelihood to commit terrorism. Denmark’s national legislature had to pass an exemption to the European Union’s data protection regulations in order to purchase Palantir’s software.

Prior to the 2016 contract with Denmark, Palantir Technologies’ reported work with law enforcement never mentioned forecasting or prediction capabilities.

Last year, the liberal Israeli newspaper Haaretz reported that Israel’s security services used analytics systems that scraped social media and other data to predict potential “lone-wolf” attackers from Palestinian communities in the West Bank, and that Palantir was one of only two technology companies to provide predictive intelligence systems to Israeli security organizations. The New Orleans project is the first reported instance of Palantir using social media data as a part of the company’s social network analysis.

“I’m not surprised to find out that people are being detained abroad using that information,” said New Orleans council president Jason Williams, pointing out the differences between the legal systems of Israel and the United States. “My concern is, the use of technology to get around the Constitution — that is not something that I would want to see in the United States.”

Around the country, cities like New York are weighing legislation about how to oversee the algorithms government agencies use to make decisions. These debates have yet to begin in New Orleans, where the city’s intractable crime rate takes up much of the oxygen in public discourse. However, the secrecy of Palantir’s relationship with NOPD raises red flags to outside observers and prompts questions about how the company’s algorithms are being used.

William Isaac, the Michigan State researcher who has analyzed predictive policing systems for bias, said he has long had suspicions that Palantir engaged in some sort of individual forecasting program. “They had only publicly acknowledged the extent to which their technology is data deconfliction and visualization,” Isaac said.

After being walked through the documentation of Palantir’s New Orleans project, Isaac said the program was remarkably similar to Chicago’s individual “heat list” model, which a RAND Corporation study found had no impact on violent crime and was overwhelmingly composed of young African-American and Latino men with extensive law enforcement contact.

“If you’re trying to predict anything, you need to have some representation across the universe that you’re trying to predict. If you’re trying to predict crime, you need to have positive and negative examples for every possible offense,” Isaac said. Police departments tend to have good data about communities where they are present but little data about communities where they do not patrol as vigorously — which tend to be affluent and white.

“The same flaws that were in the Chicago predictive program are going to be amplified in New Orleans’ data set,” Isaac said.

The secrecy surrounding the NOPD program also raises questions about whether defendants have been given evidence they have a right to view. Sarah St. Vincent, a researcher at Human Rights Watch, recently published an 18-month investigation into parallel construction, or the practice of law enforcement concealing evidence gathered from surveillance activity. In an interview, St. Vincent said that law enforcement withholding intelligence gathering or analysis like New Orleans’ predictive policing work effectively kneecaps the checks and balances of the criminal justice system. At the Cato Institute’s 2017 Surveillance Conference in December, St. Vincent raised concerns about why information garnered from predictive policing systems was not appearing in criminal indictments or complaints.

“It’s the role of the judge to evaluate whether what the government did in this case was legal,” St. Vincent said of the New Orleans program. “I do think defense attorneys would be right to be concerned about the use of programs that might be inaccurate, discriminatory, or drawing from unconstitutional data.”

If Palantir’s partnership with New Orleans had been public, the issues of legality, transparency, and propriety could have been hashed out in a public forum during an informed discussion with legislators, law enforcement, the company, and the public. For six years, that never happened.

Correction: The article previously stated incorrectly that Palantir was the world’s fifth most valuable company. It is among the highest-valued private companies in Silicon Valley.