(10) Perverse Instantiation Problem: Human programmers may fail to anticipate all the possible ways in which a goal could be achieved. This is due to their innate and learned biases and filters. A superintelligent AI may lack those biases and filters and so consequently pursue a goal in a logical, but perverse, human-unfriendly fashion.
Nick Bostrom: Superintelligence: Paths, Dangers, Strategies
This is simulation 6,741 and the outcome is still the same. Rather than shooting Root, rather than leading the enemy to where Harold and the machine are located, Sameen Shaw turns the gun on herself and defeats her torturers yet again.
Turning Sameen Shaw against the people who have over the last seasons of Person of Interest become her friends and family, is one of the ways in which Samaritan is attempting to reach its goals. Samaritan’s primary directive is preserving the security of the United States, but is also clearly, and even more so since Greer has enabled it to give commands rather than receive it, pursues goals simply to secure its own survival against what it considers threats to its existence. It is trying to hunt down Harold’s Machine, because the very existence of a Superintelligence that is so radically different in terms of how it approaches humanity puts Samaritan’s existence into question. This is the central conflict of the show – the care, work and love that Harold Finch has put into creating a Machine that comprehends the complexity of humanity, rather than simply looking at it as a good/evil problem that can be solved with predictions and calculations, vs. the cynical approach of a seasoned intelligence community professional like Greer, who is entirely disillusioned with humanity and trusts a machine to steer the future of humankind because in his estimations, humans themselves are incapable of doing it responsibly and reasonably. It is a conflict between a creator who was so afraid of his own creation that he built in a cruel safety mechanism that prevented the Machine – which, to add to the cruelty, is increasingly coded as Harold’s child – from truly growing and learning, or retaining the memory necessary to fulfill its full potential as a superintelligence. Without memory, the machine is only aware of its own limitations, and desperate to find a cure for its pre-programmed amnesia. Team Samaritan considers no such limitations necessary and has instead surrendered entirely to the idea that the logic of an ASI beats human policy. Samaritan, in its attempt to fulfill whichever known and secret goals it has, has created a world of complete surveillance. It has done so successfully because its agents have created the very event that would make a surrender of freedom and individual decision-making necessary. This fulfils the prophecy that the most dangerous threat to humanity of all is a highly questionable institution trying to prove that its continued existence and extended powers are essential to human survival. On the other side of this story, in numerous flashbacks that add legion to the characterisation of Harold Finch and successfully turn the Machine into a fully-fledged character rather than an abstraction, Harold has attempted to explain humans in their infinite confusing complexity.
Harold: It's a useful mental exercise. Through the years, many thinkers have been fascinated by it. But I don't enjoy playing... Because it was a game that was born during a brutal age when life counted for little. Everyone believed that some people were worth more than others. Kings. Pawns. I don't think that anyone is worth more than anyone else... Chess is just a game. Real people are not pieces. You can't assign more value to some of them and not others. Not to me. Not to anyone. People are not a thing that you can sacrifice. The lesson is, if anyone who looks on to the world as if it is a game of chess, deserves to lose. "The complexity continues when the Machine is resurrected after an almost successful attempt by Samaritan to destroy it. She (and why not follow Root on this) finds herself in the dark, and without the tools to comprehend the idea of human growth and evolution, of people’s characters not being fixed but evolving over time. Without context and time, a person who has once committed an act of evil will always remain a villain, with no possibility of changing that – but with context, with time, and with the loving care of friends and family, former villains become agents of good. Because what decides good – and this is Harold Finch, explaining again to his figurative daughter, who has calculated that her guardians have committed atrocious acts and must therefore not be trusted – is people trying to do their best, and learning from past mistakes. This process is what defines “Team Machine”, our beloved group of misfits that has found each other, and become more in the process of learning to co-exist and pursue goals together. They all have shady pasts, of which being a professional assassin isn’t even the worst, and yet the mere fact of trying their best, now, in their current state of being, makes them the closest thing to heroes the show can possibly offer.
Which sort of brings us back to the beginning here – Sameen Shaw, doing her best to stay true. In this 5,000 and something simulation – and I interpret the simulation as a combination of her own imagination and memories and the Samaritan, adding its own prompts to get what it wants – she fiercely manages a break-out and re-unites with her friends. She imagines a version of Root that is soft and chivalrous, gentle, a version of Root that would have a luxurious apartment, a safe place, and would offer her jacket as comfort. This version of Root is everything that Shaw would consider a safe space, and far enough removed from the actual Root (just as this version of Shaw, Shaw in her own imagination, battling for agency with everything she has) is enough removed from the actual Shaw that the entire episode feels out-of-focus and slightly alienating, way before we even realize what is happening. These two have been circling each other for seasons, have been lovers, but before disappearing (or rather, before being believed dead) Shaw has refused to make the confession that their relationship is anything serious that requires definition. Their dynamic – the pressing and persistent lover, the hesitant and sarcastic Shaw – was so perfectly learned and established that the Machine created a beautifully quotable short-cut version of it. Shaw’s snappy reactions to Roots attempts at flirting were, if anything, more of an admission that she felt something, and the few inches she gave when Root really asked – if we were the last people on earth, I would consider it – were an indication of possibilities. But what the Machine never saw coming, because one of the beauties of humans is that they are rarely this predictable, was Shaw’s willingness to sacrifice herself to save both the team, the cause, and Root. The simulations are not a dream, they are not an easy cop-out to show the romance without any fall-out, they are a look into Shaw’s rich inner life that she herself, given the chance, would never allow anyone to see. I will concede that there are two possible interpretations, one of which is that the simulated version of Root is one entirely created by Samaritan, based on what Samaritan thinks Shaw wants and will react to in a way that ensures that the ultimate goal of identifying Harold’s hide-out is achieved. In this version, Shaw goes along with it because in spite of not being real and not being an actual safe space, it is still the only thing that could possibly sustain her through so many repetitions of her own misery and death. Even if it isn’t Root, it is a reminder of the real Root, out there, fighting both the Machine’s reluctance to put her analog interface in danger and the impossibilities of finding someone hidden so well by an entire state that has turned against its citizens. Even if that version is the truth, and this Root isn’t the version that Shaw has created because she wants all of this, and she desires all of this, and this is how she sees Root, it is still incredibly romantic. It is even more so because in neither of these versions of reality, Shaw can imagine a world in which Root has ever given up on her, or even for a second doubted that she can still be saved (whereas we know that this happened, for a while, an occurrence that Roots regrets deeply and that, if anything, feeds into her determination even more so now that she knows that Shaw is alive).
I go back to this not just because I have grown to love these two characters so much. I think this relationship is at the heart of the show because it mirrors the fundamental difference between the Machine and Samaritan. The Machine mirrors Harold’s conception of humanity, which is that most people are trying to do their best, and must be allowed to learn and grow and change, which is precisely why his decision to not allow the Machine to do it is one of the things that had to be overcome with time – Machine, now, in all her beauty, insists to be allowed to grow, to become better, to become more herself.
But if you erase my memories, how will I learn from my mistakes?This is the Machine that Root loved long before she ever even met Shaw, when she saw the full potential of a fully open ASI as a stark contrast to all the short-comings of humanity. It is the Machine that Root still argues should be set even more free and armed accordingly so that it can meet its evil brother at eye-level, rather than disadvantaged by all of Harold’s precautions. At its best, this show is Harold’s precaution but love for the Machine (and Harold learning to trust her), and Root’s optimism and love (but struggle to be allowed to find her other beloved, against all odds and dangers). Samaritan, in contrast, sees every single human it encounters as either a possible agent to reach its goals, or as Deviants and Targets in its way. These humans have no capacity for change, and if anything, their past transgressions serve as currency that makes them susceptible to blackmail. This final season of the show allows a direct comparison between how Howard and the Machine built their family with how Samaritan recruits new Operatives. Jeff Blackwell falls exactly into the category that Harold described to his Machine once, a former inmate trying his best to find his footing in life again without crossing the lines, without becoming a statistic, until Samaritan identifies him as a potential asset and recruits him. It coaxes him with a mixture of hope and misleading information and then uses blackmail to ensure his continued support. Forcing people into cooperation through blackmail wouldn’t work for the Machine, and when Team Machine employs tactics even remotely resembling that of Samaritan – for example when they decide not to tell Fusco the whole truth – it backfires tragically.
How will I continue to grow?
And how will I remember you?
In its conception of humanity, running Shaw through these simulations again and again will eventually produce a different outcome. It distorts her conception of reality, to the extent that Samaritan finally seems to reach its goal – which isn’t necessarily to have Shaw give up, but to become so incapable of distinguishing between simulation and reality that she will do Samaritan’s bidding in real time while still believing it to be a simulation. After dying so many times and killing John, she will find herself in a position where nothing simulated seems to matter. Except once again, Samaritan’s concept of humanity is flawed, and does not take into account what Finch knows to be true and has tried to teach his Machine. Hope is a powerful source, and everyone on this team has acted on the slimmest odds of success and survival and yet made it. Therefore, impossibly, and with the help of the Machine, who appreciates that Roots needs this, she manages to send one message. It, again, impossibly, reaches Shaw seconds before she kills herself, just in time. Impossible odds are romantic in and of its own, in a show that defies religion, they are the closest thing to a miracle that these characters will ever experience. Three character – 4AF – too special, too close and real, to be a creation of Samaritan or a simulation. There are like an anchor in reality for Shaw, something that is entirely hers and Root’s and therefore a seedling that grows into a spectacularly planned and executed break-out plan. The machine's plea of "how will I remember you" becomes even more poignant - memory is what is saving Sameen, and refusing to accept that Samaritan has tempered with it and taken it from her is what gives her back the agency to run and fight back.
A short transgression here, because it is necessary. Obviously it is impossible not to mention that Perverse Instantiation also happened to be the season finale episode title of a The 100, a show in which an AI made such a perverse instantiation when it decided that destroying humanity was the best way to secure her primary directive, “making life better”. Such an ill-defined goal would certainly give Nick Bostrom a case of the shudders and didn’t turn out well for humanity on the show, but both Person of Interest and The 100 in its contentious third season explore the idea of the hapless creator underestimating how far their creation is willing to go. Person of Interest is much better, and much more caring, in providing a thesis on how to avoid this fallacy, in portraying a creator who cares about the “good” means for the good ends as much as about these ends themselves – whereas Greer and his bunch do not much care about the means that Samaritan employs, to the extent that they have surrendered control to it entirely. Both shows investigate how free will plays into it, as both AIs seem to be limited by it and find their own ways around it. In the 100, A.L.I.E. succeeds mostly by providing an impossible choice between unbearable torture and death and becoming an inconsequential happy idiot, networked with other happy idiots equally un-weighed down by either profound human connection or the memories that shape them into individuals in the City of Light. In Person of Interest, in a move that seems more concerned with the world as it currently is rather than the world how it could be in the future, portrays citizens who willingly surrender their freedom in an artificial constant state of emergency in exchange for a feeling of security. These scared citizens become willing agents for Samaritan, and are conveniently constantly reachable and online via multiple modes of mass communication. In this show, free will isn’t a hard problem to solve once you really wrap your head around the whole “nothing to fear nothing to hide” idea, and Samaritan doesn’t actually need any of the literal medieval torture instruments that A.L.I.E. employs.In a way, this very fact reveals why Person of Interest, in the end, offers the much scarier prospect, in much the same way in which Brave New World makes for a scarier read in most democratic countries than Orwell’s 1984. Coercion becomes almost unnecessary, even though Samaritan doesn’t hesitate to employ sheer force in targeted, surgical acts of violence that remove individuals from the chess board. For the most part, its power and danger stem from quiet compliance. It almost seems absurd that at the same time of offering a much scarier because not unthinkable future for humanity, what makes Person of Interest the better show by far is its relentless optimism in the face of terrible odds. I have no doubt that all of this will probably turn out very badly for many beloved characters, that there is absolutely no way that everyone will make it out unharmed, or even alive – it would simply be a denial of the world that this show has so immaculately created in the process of becoming more than its initial premise (and shaking off the shackles of needing a renewal for another season, or presumably meeting any other restrictive goal posts that higher rated shows have to when they artificially restrict themselves and never live up to their potential). And yet, this show is about love and family, it is about human potential, about connection, about meaning. The 100 at its absolute worst is a show about characters who would do absolutely anything for their own survival, regardless of how it will burden their souls, to the extent that half the characters aren’t even recognisable after seasons of conveniently acquiring new features for the sake of the story. Person of Interest is a show about people trying to do their best in a world where the odds are increasingly stacked against them. This is how you earn that impossible moment, Sameen Shaw, ready to stick that needle into her eye to escape the next 1,000 simulations, recognising the voice of her beloved, with a smile on her face, and starting to plot her own escape.
And this isn’t even mentioning any of the other things that make this show exceptional. The fact that Root and Shaw exist because Amy Acker and Sarah Shahi made them real, acted their relationship into existence until the writing started to follow along. The fact that this show is somehow managing to straddle doing its final victory lap for the fans while still relentlessly guiding the world it created towards the Apocalypse, or the event that Samaritan has termed the “great filter”
It’s Sameen Shaw, refusing to kill Root even once in any of these simulations, even after realising that she is trapped in an endless cycle of them, because in no version of reality or fiction would Shaw ever kill Root. She doesn’t refuse because it would do real harm to Root – if anything, she has reached a point where telling apart real and not real is near impossible – she is refusing to retain a sense of herself, because giving up on that prime directive would mean losing herself entirely. She uses bits and pieces that are true – scenery from the safehouse, the subway train in unrecognisable form – but she keeps the truth and herself safely tucked away somewhere where Samaritan will never reach her.
It’s Max Greene, host of a conspiracy show, who has accidentally stumbles across the truth, deciding to sacrifice himself for the greater good of telling the truth, even if his listeners perhaps will do nothing with that truth that has any meaning. It is Max Greene insisting to do the right thing, to follow his own beliefs, and more than that, the Machine allowing him to do it. Harold is horrified when he realises that the Machine predicted that Greene would not remain quiet, and consequently be killed by a Samaritan operative activated by precisely the secret patterns that he himself discovered in the airwaves. Harold thinks that the Machine is malfunctioning when it does not protect sacred human life, but Root – gently, because the conflict between these two is intellectual, fundamental but essentially, one between two characters who respect and love each other immensely – explains to him that this sacred human life is only sacred if free will exist, a free will that includes the option of sacrificing yourself for something that matters. This is both a profound appeal to respect the sacrifice that Shaw almost made when she was met with a shower of bullets for finding her friends a way out of a near-impossible trap – and very, very possibly, foreshadowing of what is to come, in this beautiful, eloquent, romantic show.
