wagner, the dehumanization of international...
TRANSCRIPT
The Dehumanization of International Humanitarian Law:
Legal, Ethical and Political Implications of Autonomous Weapon Systems
Markus Wagner*
Table of Contents
I. Introduction ............................................................................................................... 3
II. The Road to and Degrees of Autonomy ................................................................... 5
1. Historical Development ........................................................................................................... 5
2. Variations of Autonomy: From remote control and automation to autonomy ......... 9
III. Legal Challenges to Autonomous Weapon Systems .............................................. 13
1. Introduction ............................................................................................................................... 14
2. The Principle of Distinction .................................................................................................. 18
3. The Principle of Proportionality ........................................................................................... 28
4. AWS and Individual Responsibility .................................................................................... 38
IV. Ethical Challenges to Autonomous Weapon Systems ............................................ 45
V. Political Challenges to Autonomous Weapon Systems .......................................... 58
VI. Conclusion .............................................................................................................. 59
* Associate Professor of Law, University of Miami School of Law. The author would like to thank Dapo Akande, Bill Boothby, Neil Brown, Dieter Fleck, Robin Geiss, Oren Gross, Tim McCormack, Fionnuala D. Ní Aoláin, Andreas Paulus, Daniel Reisner, Dan Saxon, Yuval Shany, Henry Shue and Silja Vöneky for their invaluable feedback and comments on previous version of this paper as well as the participants at workshops and conferences at the Oxford Institute for Ethics, Law and Armed Conflict (ELAC), at Hebrew University in Jerusalem on the occasion of a conference entitled New Technologies, Old Law: Applying International Humanitarian Law in a New Technological Age, and at workshops at the University of Göttingen / Germany and Freiburg / Germany. I would furthermore like to thank Brian Lechich, Bryan Doeg and Rachel Paul for excellent research assistance. The usual disclaimer applies.
Abstract
Today’s unmanned aerial drones are only a step towards greater automation and autonomy in military technology. In the future, a growing number of combat operations will be carried out by autonomous weapon systems (AWS) which no longer rely on direct human input. Taking humans out of the loop has the potential to disrupt fundamental assumptions of international humanitarian law.
The paper addresses this development towards greater autonomy along three dimensions: legal, ethical and political concerns. First, it analyzes the potential dehumanizing effect of AWS with respect to two cornerstones of international humanitarian law: the principle of distinction and the principle of proportionality. It argues that only in the rarest of circumstances can AWS be employed in light of the legal constraints inherent in these two principles. Moreover, the paper explores the dehumanization of international humanitarian law through the question of criminal responsibility for the conduct of AWS: who is to be held responsible should the action of an AWS lead to war crimes or crimes against humanity? AWS may dilute personal responsibility diluted to the point that deterrent effects for those that decide to deploy an autonomous weapon system are threatened to vanish. The paper will explore to what extent personal responsibility can be established at various stages of the design and deployment of AWS.
In a second step, the article turns to ethical questions. It explores the advantages and disadvantages of the deployment of AWS, independent of the legal considerations AWS. Authors from various fields, most prominently engineers and philosophers, have weighed in on this debate, but oftentimes without linking their discourse to the legal questions outlined above. This paper fills this gap by bridging these disparate discourses and suggests that there are important ethical reasons that militate against the use of AWS on ethical grounds.
Third, the article argues that the limitations on how force can be used constitute important barriers in deciding whether to enter an armed conflict. The introduction of AWS can substantially lower the risk that soldiers are exposed to although it does not eliminate it. This alteration of the risk calculus can make the decision over whether to engage in a conflict politically more palatable to the wider public and less risky for the political decision-makers.
Article
I. Introduction
In a seminal article published in the American Journal of International Law in 2000,
Theodor Meron, the current President of the International Criminal Tribunal for the
former Yugoslavia, expressed his hope that the direction of international humanitarian
law could undergo a development towards conducting combat in a more humane
fashion.1 This assessment was based on the inroads that had – maybe only apparently
– been made in the aftermath of the human rights tragedies in the former Yugoslavia
and Rwanda and the subsequent installation of international criminal tribunals in the
1990s. Since the publication of the article, numerous new conflicts have broken out.
More importantly, some of the fundamental assumptions underlying the conduct of
armed conflict have been put into question. We are witnessing changes with regard to
the adversaries that fight one another – consider the rise of what has been labeled
“asymmetric warfare”.2 Moreover, the means with which armed conflict is being carried
out has undergone significant modifications already with more, and potentially more
transformative, changes yet to come.
One of the most important modifications is already well underway – and has come
under some scrutiny. So-called unmanned aerial vehicles (UAVs) have taken on a vast
and increasing number of reconnaissance missions. A smaller number of missions carry
out armed attacks, with the operators of either type of mission connecting to their
1 Theodor Meron, The Humanization of Humanitarian Law, 94 AMERICAN JOURNAL OF INTERNATIONAL LAW 239, 239 (2000). 2 Steven Metz, Strategic Asymmetry, MILITARY REVIEW 23, 24 (July-Aug. 2001); Kenneth F. McKenzie, The Revenge of the Melians: Asymmetric Threats and the Next QDR (Institute for National Strategic Studies, National Defense University. 2000); Report of the Quadrennial Defense Review. 4 (1997).
aircraft via satellite link from thousands of miles away.3 Similar attempts have been
made by militaries around the world not only with respect to UAVs, but also regarding
sea and land warfare. All of these examples – whether they operate in the air, on sea or
on the ground – share one characteristic. These unmanned systems (UMS) are the
visible piece of a network that – at least until this point – still operates with direct human
input.
Building on these first steps towards greater autonomy for weapons systems, the next
generations of UMS are designed to operate wholly independently from human input.
The report specifically states that “the level of autonomy should continue to progress
from today’s fairly high level of human control/intervention to a high level of autonomous
tactical behavior that enables more timely and informed human oversight”.4 From target
selection to acquisition and the decision whether to employ weapons systems in the
particular moment in time and in that case which one, UMS are designed to be in a
position to carry out their missions in an autonomous fashion.
This changes the assumptions on which the law of armed conflict (international
humanitarian law) is based and has the potential to alter fundamentally the perceptions
of the law of armed conflict. The paper addresses this development in the context of the
3 See ___ below for a more detailed description of the current use of UAVs. 4 Department of Defense, FY 2009-2034 Unmanned Systems Integrated Roadmap, at http://www.acq.osd.mil/psa/docs/UMSIntegratedRoadmap2009.pdf, 27-36. Ibid., 27. For a report, giving UMS a more independent role, but mindful of the potential consequences of letting UMS making lethal decisions, see United States Air Force, United States Air Force Unmanned Aircraft Systems Flight Plan 2009-2047, at http://www.govexec.com/pdfs/072309kp1.pdf, 41.
dehumanization of the international humanitarian law. In a first step, it retraces the
history of autonomous weapons and differentiates future generations of autonomous
weapon systems (AWS) from the current generation of weapons (II.). It subsequently
deals with the potential effect of UMS with respect to two cornerstones of the
international humanitarian law: the principle of distinction and the principle of
proportionality. It argues that from a legal perspective UMS can be employed only in the
rarest of circumstances in light of the legal constraints inherent in these two principles.
Thus, their potential deployment is limited to such an extent as to render them useless.
Moreover, the article argues that another bedrock principle of the international
humanitarian law, the principle of personal responsibility, may be undermined by the
introduction of AWS (III.). The two subsequent sections deal with the ethical (IV.) and
political (V.) ramifications of the deployment of AWS and provide context to the legal
considerations. Through widespread use of AWS, personal responsibility is diluted to
the point that deterrent effects – both with respect to individual decisions over a
particular mission, but also the decision of engaging in armed conflict – are threatened
to vanish. Part VI. contains concluding observations.
II. The Road to and Degrees of Autonomy
1. Historical Development
Serious attempts at creating unmanned weapon systems have been made since the
end of the 19th century. Nikola Tesla constructed and patented the first remotely
operated boat capable of carrying an ordinance, though the device was never put into
service.5 The so-called Kettering Bug, a pilotless bi-plane carrying explosives and
developed after WW I, was also never deployed because of reliability issues.6 The
German army deployed a land-based cable-operated device during WW II called
Goliath, although it too was not considered to be a successful system.7 Nevertheless,
some already predicted at the time that unmanned systems would at some point be put
to widespread use. US Army Air Corps General Henry H. Arnold stated that while his
war may have been fought by “heroes flying around in planes”, future wars “may be
fought by airplanes with no men in them at all”.8
In hindsight, this prediction was premature, but the advances in remotely-operated
systems through innovations in advanced navigation and communications technologies
as well as the necessary satellite communication abilities allowed for a sharp increase
of the use of UMS, first in the air and increasingly on the ground and in marine
environments as well. Beginning in the 1970s, remotely operated airplanes were used 5 N. Tesla, Method and Apparatus for Controlling Mechanism of Moving Vessels or Vehicles, U.S. Patent No. 613,809 (filed July 1, 1898), available at http://patimg1.uspto.gov/.piw?Docid=00613809&homeurl=http%3A%2F%2Fpatft.uspto.gov%2Fnetacgi%2Fnph-Parser%3FSect2%3DPTO1%2526Sect2%3DHITOFF%2526p%3D1%2526u%3D%2Fnetahtml%2FPTO%2Fsearch-bool.html%2526r%3D1%2526f%3DG%2526l%3D50%2526d%3DPALL%2526S1%3D0613809.PN.%2526OS%3DPN%2F613809%2526RS%3DPN%2F613809&PageNum=&Rtype=&SectionNum=&idkey=NONE&Input=View+first+page. 6 Kendra L. B. Cook, The Silent Force Multiplier: The History and Role of UAVs in Warfare 2 (2007); John DeGaspari, Look, Ma, No Pilot!, 125:11 MECHANICAL ENGINEERING 42, 42 et seq. (November 2003); Laurence R. Newcome, Unmanned Aviation: A Brief History of Unmanned Aerial Vehicles (American Institute of Aeronautics and Astronautics, Inc. 2004); Diana G. Cornelisse, Splendid Vision, Unswerving Purpose: Developing Air Power for the United States Air Force During the First Century of Powered Flight 22 (Helen Kavanaugh-Jones ed., History Office, Aeronautical Systems Center, Air Force Materiel Command. 2002). 7 Jon Guttman, Goliath Tracked Mine, 28:2 MILITARY HISTORY 23, 23 (July 2011); P. W. Singer, Drones Don't Die, 28 MILITARY HISTORY, 66 for Dennymite, 67 for Kettering Bug, and 67-68 for Goliath, respectively (2011). 8 Lawrence Spinetta, The Rise of Unmanned Aircraft, HistoryNet.com, November 10, 2010, available at http://www.historynet.com/the-rise-of-unmanned-aircraft.htm.
on a regular basis when Israel used UAV for the first time in the Lebanon’s Bekaa
Valley.9 Further development was slow until the 1990s after which the number of UAV
has risen steadily. Since the late 1980s, the number of airborne UMS deployed by the
United States military has risen from 167 in 2002 to over 7,000.10 At the same time, the
expenditures for UAVs for the US military alone have risen from US$ 667 million to
US$3.9 billion between FY 2001 and FY 2012 for procurement and development
funding.11 This number can be expected to rise significantly given the apparent proclivity
of lawmakers to expand the use of unmanned systems.12 Globally, the total
expenditures for military UAVs – including research and development as well as
procurement – is expected to increase significantly from the current levels of roughly
US$ 6 billion to more than US$ 11 billion by 2020.13
Not only has the number of vehicles and the amount of money increased over the
years, their actual use in combat has seen a steady increase as well. First used almost
exclusively in a surveillance role as was the case in Kosovo, UAVs have increasingly
taken on combat roles as evidenced most prominently in the conflict in Afghanistan as 9 Ralph Sanders, An Israeli Military Innovation: UAVs, Winter 2002/2003 JOINT FORCE QUARTERLY 33(2002/2003); Elizabeth Bone & Christopher C. Bolkcom, Unmanned Aerial Vehicles: Background and Issues 2 (Novinka Books. 2004). 10 Jeremiah Gertler, U.S. Unmanned Aerial Systems 2 (Congressional Research Service. 2012). See also CQ Researcher, Drone Warfare – Are Strikes by Unmanned Aircraft Unethical?, 20:18 CQ Researcher, 653, 656. 11 Jeremiah Gertler, U.S. Unmanned Aerial Systems 2 (Congressional Research Service. 2012). 12 In 2000, Congress set forth that “[i]t shall be a goal of the Armed Forces to achieve the fielding of unmanned, remotely controlled technology such that— (1) by 2010, one-third of the aircraft in the operational deep strike force aircraft fleet are unmanned; and (2) by 2015, one-third of the operational ground combat vehicles are unmanned.” See P.L. 106-398, “Floyd D. Spence National Defense Authorization Act for Fiscal Year 2001,” section 220. This action was followed up by a legislative requirement for the Department of Defense to provide periodic roadmaps, the latest of which was released in 2009. Department of Defense, Unmanned Aircraft Systems Roadmap 2005-2030, at http://www.fas.org/irp/program/collect/uav_roadmap2005.pdf. 13 Teal Group Corporation, World Unmanned Aerial Vehicle Systems 2 (2011).
well as other countries.14 According to industry lobbyists, the importance of UAVs is
considerable: “every second of every day, 40 Predator-series aircraft are airborne
worldwide, while the hours that various UAVs by the Air Force are in operation has
more than tripled between 2006 and 2009, then standing at 295,000 hours per year”.15
These figures show that there has been a steady – and accelerating – increase in the
use of UAVs in particular over the last years. However, it is important to distinguish
between different levels of autonomy, at least for conceptual purposes. For the
purposes of this article, the different types of unmanned systems are grouped into three
different categories: remotely-operated systems, automated systems and systems
which actually function autonomously. The distinction serves an important purpose,
namely to separate the existing weapon systems which are either automated or
remotely operated from those that function in an autonomous manner. Each of these
categories implies different legal questions as there is not only a longer track record of
the automated and remotely operated weapons, but unlike autonomous weapon
14 For the controversy surrounding the use of UAVs in the elimination of known terrorists and terrorist suspects, Peter Finn, A Future for Drones: Automated Killing, Washington Post, Sept. 19, 2011, at A01; Siobhan Gorman, Drones Evolve Into Weapon in Age of Terror, The Wall Street Journal, Sept. 8, 2011, at A6; Ryan Vogel, Drone Warfare and The Law of Armed Conflict, 39 DENVER JOURNAL OF INTERNATIONAL
LAW AND POLICY 101(2010-2011); Peter Bergen & Katherine Tiedemann, Washington’s Phantom War, 90 FOREIGN AFFAIRS 12(2011). For the debate about use of drones and targeted killing, see Study on Targeted Killings - Report of the Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions, Philip Alston. No. A/HRC/14/24/Add.6; David Kretzmer, Targeted Killing of Suspected Terrorists: Extra-Judicial Executions or Legitimate Means of Defence?, 16 EUROPEAN JOURNAL OF INTERNATIONAL LAW 171(2005); Nils Melzer, Targeted killing in international law (Oxford University Press 2008); Orna Ben-Naftali & Keren R. Michaeli, "We Must Not Make a Scarecrow of the Law": A Legal Analysis of the Israeli Policy of Targeted Killings, 36 CORNELL INTERNATIONAL LAW JOURNAL 233(2003); Mary Ellen O'Connell, The International Law of Drones, 14:36 ASIL INSIGHTS 1(2010). 15 See statement by Michael S. Fagan, Chair, Unmanned Aircraft Systems (UAS) Advocacy Committee, Association for Unmanned Vehicle Systems International (AUVSI) before the House Oversight and Government Reform Subcommittee on National Security and Foreign Affairs, Rise of the Drones: Unmanned Systems and the Future of War. (23 March 2010).
systems, there is also a human operator in the loop.16 The distinction is complicated by
the fact that unmanned systems may operate in more than one and indeed all three
operating modes.
2. Variations of Autonomy: From remote control and automation to autonomy
As pointed out above, it is crucial to distinguish between different degrees of autonomy.
While these differentiations are difficult to maintain in certain situations given the ability
of different weapon systems to operate under each category, it is important to
remember that each subset faces different challenges.
Remote controlled systems are, as shown above not new. Their use however has
greatly proliferated, as evidenced by the combat operations in Afghanistan.17 Examples
of systems currently in operation include the MQ-1 Predator and MQ-9 Reaper, which
are capable of carrying weapons and are carrying out combat missions. It is these
vehicles which have garnered the strongest public debates.18 Most such systems are
operated from ground bases and communicate over satellite with their ground stations,
sometimes at a considerable distance. In addition to these larger aerial systems, a
16 A similar distinction is drawn by the International Committee of the Red Cross, see International Committee of the Red Cross, International Humanitarian Law and the Challenges of Contemporary Armed Conflicts 38 et seq. (2011). 17 Matt J. Martin & Charles W. Sasser, Predator: The Remote-Control Air War over Iraq and Afghanistan. A Pilot's Story (Zenith Press. 2010); Alan S. Brown, The Drone Warriors, 132:1 MECHANICAL
ENGINEERING 22, 24-25 (Jan 2010). 18 See only Jane Mayer, The Predator War - What are the Risks of the C.I.A.’s Covert Drone Program?, The New Yorker, 26 October 2009, at 36. See also in the wider context of targeted killing and its international legal implications, Study on Targeted Killings - Report of the Special Rapporteur on Extrajudicial, Summary or Arbitrary Executions, Philip Alston. No. A/HRC/14/24/Add.6.
considerable variety of smaller vehicles exist that are designed to be man portable,
typically carrying out short-distance reconnaissance missions and operated by ground
forces from portable devices.19 Land based systems are often used for explosive
ordinance detection (EOD), although increasingly UGVs are used for reconnaissance,
surveillance and target acquisition.20 Marine systems are also in operation, though
submarine vehicles face additional hurdles communicating under water.21 It has been
argued that the use of remotely operated UMS allows for longer reconnaissance and
more precise targeting and thus a reduction in civilian casualties,22 though there are
also reports that the use of remotely operated systems increases because of the
reduction of the risk to one’s own soldiers allows for attacks that would otherwise not
have been carried out.23 Other criticisms have centered on the potential for information
overload as well as the fact that the physical and emotional distance that is created by
remotely operated systems may lead to an increased propensity of attacks.24 This type
of system retains human involvement in the decision of whether to attack and how to
19 Elizabeth Quintana, The Ethics and Legal Implications of Military Unmanned Vehicles, Royal United Services Institute for Defence and Security Studies, at http://www.rusi.org/downloads/assets/RUSI_ethics.pdf, 2. 20 Elizabeth Quintana, The Ethics and Legal Implications of Military Unmanned Vehicles, Royal United Services Institute for Defence and Security Studies, at http://www.rusi.org/downloads/assets/RUSI_ethics.pdf, 2. For an overview of US ground UMS, Department of Defense, FY 2009-2034 Unmanned Systems Integrated Roadmap, at http://www.acq.osd.mil/psa/docs/UMSIntegratedRoadmap2009.pdf, 133. 21 Elizabeth Quintana, The Ethics and Legal Implications of Military Unmanned Vehicles, Royal United Services Institute for Defence and Security Studies, at http://www.rusi.org/downloads/assets/RUSI_ethics.pdf, 6. 22 Department of Defense, FY 2009-2034 Unmanned Systems Integrated Roadmap, at http://www.acq.osd.mil/psa/docs/UMSIntegratedRoadmap2009.pdf, 7-15 for general benefits of increasingly autonomous systems; Jack M. Beard, Law and War in the Virtual Era, 103 AMERICAN
JOURNAL OF INTERNATIONAL LAW 409(2009); Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 205-236 (CRC Press. 2009). 23 Peter Asaro, How Just Could a Robot War Be? 61 et seq. Current Issues in Computing and Philosophy: Frontiers in Artificial Intelligence and Applications (Adam Briggle et al. eds., IOS Press. 2008); Patrick Lin et al., Robots in War: Issues of Risk and Ethics 62 Ethics and Robotics (Rafael Capurro & Michael Nagenborg eds., AKA. 2009). 24 P. W. Singer, Wired for War: The Robotics Revolution and Conflict in the Twenty-first Century 395-396 (Penguin Press. 2009).
attack, although one operator may command more than one vehicle at a time.25 In this
scenario there is thus a clear line of responsibility as to who carries out the attack.
Automated systems have similarly been deployed for quite a long time. An early
example of such systems is the V-1 and the V-2 rockets that the German military fired at
the United Kingdom during WW II.26 More modern examples include automated sentry
guns as well as sensor-fused ammunition.27 Moreover, surveillance systems, such as
the Global Hawk, fall into this category.28 Capable of staying in the air for over 30 hours
and flying in altitudes of up to 65,000 ft it carries out surveillance missions in either an
automated or remote control fashion.29 While such systems do not require a human to
command the weapon system, there is still often very considerable human involvement
prior to deployment. Once that process is complete however, these systems are
capable of independently detecting the threat they are designed to counter and fire or
detonate following certain cues. It is these cues that raise concerns with these systems,
25 Department of Defense, FY 2009-2034 Unmanned Systems Integrated Roadmap, at http://www.acq.osd.mil/psa/docs/UMSIntegratedRoadmap2009.pdf, 7 and 28. 26 P. W. Singer, Wired for War: The Robotics Revolution and Conflict in the Twenty-first Century 47-48 (Penguin Press. 2009). 27 Elizabeth Quintana, The Ethics and Legal Implications of Military Unmanned Vehicles, Royal United Services Institute for Defence and Security Studies, at http://www.rusi.org/downloads/assets/RUSI_ethics.pdf, 1; Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 10-27 (CRC Press. 2009); Darren M. Stewart, New Technology and the Law of Armed Conflict, in International Law Studies (Blue Book) Series, Vol. 87: International Law and the Changing Character of War, 271, 276 et seq. (Raul A. Pedrozo & Daria P. Wollschlaeger eds., 2011). 28 Darren M. Stewart, New Technology and the Law of Armed Conflict, in International Law Studies (Blue Book) Series, Vol. 87: International Law and the Changing Character of War, 271, 276 et seq. (Raul A. Pedrozo & Daria P. Wollschlaeger eds., 2011). 29 Elizabeth Quintana, The Ethics and Legal Implications of Military Unmanned Vehicles, Royal United Services Institute for Defence and Security Studies, at http://www.rusi.org/downloads/assets/RUSI_ethics.pdf, 1-2.
thereby implicating the principle of proportionality according to which one has to
distinguish between military and civilian targets.30
The final category consists of autonomous weapon systems (AWS). AWS, unlike
remote controlled systems and automatic systems, do not depend on human input
either during or immediately prior to their deployment. What distinguishes their
functioning is the ability to independently operate and engage targets without being
programmed to specifically target an individual object. This ability to independently react
to a changing set of circumstances is at the very core of the definition of autonomy in
the context of AWS. While there is still some human involvement prior to sending an
AWS on a mission (e.g. refueling and arming), an AWS will be capable to carry out a
mission with a much higher degree of independence and indeed autonomy.31 Decisions
over which targets to engage and how and when to carry out an attack would be left to
the software which has been programmed in such a manner to be able to deal with a
myriad of situations and a changing set of circumstances. Whether or not it will be
possible to program such systems in a way that puts them in conformity with the
existing rules of international humanitarian law – specifically the requirements of the
principles of distinction and proportionality32 – remains to be seen. Unlike remotely
operated and automated systems, AWS have so far not been deployed in combat. More
importantly however, AWS represent a major shift in not only how wars will be fought,
but also in how we conceive of armed conflict. Moreover, their introduction into the 30 For a more detailed discussion, see ___ below. 31 For an early stage of development, see only Peter Finn, A Future for Drones: Automated Killing, Washington Post, Sept. 19, 2011, at A01. 32 For a more detailed analysis, see ___ below.
modern battle space may make it considerably more complicated to assign
responsibility to the action taken by an AWS – an aspect that AWS do not share with
either remotely operated or automated systems. As discussed below, whether to assign
responsibility to the programmer, to the decision-makers who allowed AWS to be used
in the military, the commanding officers on a tactical or strategic level or the soldiers
that actually deploy them is a question that will need to be addressed.
III. Legal Challenges to Autonomous Weapon Systems
The main challenges facing AWS from a legal perspective are twofold: on one hand,
AWS will have to comply with the principle of distinction, on another they will have to be
able to carry out the equally, if not more, demanding task of complying with the principle
of proportionality. Other provisions in AP I will also be applicable, such as verifying that
AWS will comply with the requirements of Article 36 AP I, mandating that prior to the
deployment of any weapon system, each State Party has to determine whether the
employment of a new weapon, means or method of warfare that it studies, develops,
acquires or adopts would, in some or all circumstance, be prohibited by international
law. This section, following a brief introduction locating these principles within
international humanitarian law (1.), focuses on the principle of distinction (2.) and the
principle of proportionality (3.) and attempts to outline the challenges that the
introduction of AWS into any combat roles brings about.
1. Introduction
International humanitarian law has been developed in great detail in a number of areas,
including what types of weapons are permissible for use in armed conflicts and what
types of targets are legitimate.33 While there is a number of other areas that cabin the
use of unmanned systems, this section focuses on the rules that apply to both
international and non-international armed conflict.34 By doing so it will also attempt to
delineate whether the current rules of international humanitarian law are sufficient to
deal with the paradigm embodied by AWS. If it turns out that the functional approach
taken by the body of IHL – i.e. one that does not focus on a single weapon systems or
technology – is adequate to deal with this new paradigm, then there may not be a need
for a new legal framework.35
The basic premise applicable with respect to the two principles in question is laid down
in Article 48 AP I:
33 Gary D. Solis, The Law of Armed Conflict: International Humanitarian Law in War 38 et seq. (Cambridge University Press. 2010). 34 This approach may be considered to be inadequate by some as it does not take account of the all the constraints that are in place to attempt to curb unwanted behavior. See Marchant et al. for a different perspective, focusing on constraints such as “military codes, professional ethical codes, and public “watch-dog” activities […] that might pertain to the present governance dilemma regarding military robotics” with less of an emphasis on the applicable rules of international law. Gary E. Marchant et al., International Governance of Autonomous Military Robots, UNPUBLISHED MANUSCRIPT 1, 15 (2010). 35 For authors staking out a position that the current legal framework is inadequate, see Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 72 (CRC Press. 2009); Gary E. Marchant et al., International Governance of Autonomous Military Robots, UNPUBLISHED MANUSCRIPT 1(2010); Armin Krishnan, Killer Robots: Legality and Ethicality of Autonomous Weapons (Ashgate. 2009).
“In order to ensure respect for and protection of the civilian population and
civilian objects, the Parties to the conflict shall at all times distinguish between
the civilian population and combatants and between civilian objects and military
objectives and accordingly shall direct their operations only against military
objectives.”
This very general and abstract rule is fleshed out in subsequent provisions, comprising
the cornerstone of the protection of civilians in the international humanitarian law – the
principle of distinction and the principle of proportionality. Both are important elements
in determining military necessity. Furthermore, there is an underlying element that
permeates the international humanitarian law that will be dealt with subsequently,
namely that combat be carried out in a humane fashion.
The following analysis takes account of the international humanitarian law as it can be
described today. It has been described as a tension between the elements of military
necessity and humanity.36 There is considerable disagreement where on this continuum
the balance should be struck. There is also disagreement to what degree extant
circumstances such as advances in military technology, the acceptability of civilian
casualties in the court of public opinion37 and potentially more fundamental changes –
36 Darren M. Stewart, New Technology and the Law of Armed Conflict, in International Law Studies (Blue Book) Series, Vol. 87: International Law and the Changing Character of War, 271, 272 (Raul A. Pedrozo & Daria P. Wollschlaeger eds., 2011); Michael N. Schmitt, Military Necessity and Humanity in International Humanitarian Law: Preserving the Delicate Balance 50 VIRGINIA JOURNAL OF INTERNATIONAL
LAW 795, 795 (2010). 37 See e.g. Wesley K. Clark, Waging Modern War: Bosnia, Kosovo, and the Future of Combat 444 (Public Affairs 1st ed. 2001). Clark notes that restrictive rules of engagement in the 1999 Kosovo conflict meant that "[t]he weight of public opinion was doing to us what the Serb air defense system had failed to do: limit our strikes".
e.g. in the role accorded to state sovereignty38 – may influence how this balance may
have to be struck.39 Indeed, the tendency in interpreting this area of the law appears to
be a move towards a less military-centric approach and rather one that takes
humanitarian considerations into account to a greater extent than ever before.40 This
may already be evident by the change in designation that this legal field has undergone
– from “law of war” to “law of armed conflict” and now “international humanitarian law”.41
This is somewhat counterintuitive given the large-scale atrocities that the international
community has witnessed in places such as Cambodia, Somalia, the former Yugoslavia,
Sierra Leone, Afghanistan and the Congo – all of which have seen civilians being the
center of military action.
There are numerous other rules that may further cabin the ability of military personnel,
such as the Rules of Engagement (RoE) of a particular military. These may differ over
time as evidenced by the changing RoE in place between different conflicts or even
during a conflict. The behavior of any AWS would have to replicate such differing rules –
assuming that the RoE remain within the permissible framework of the international 38 Prosecutor v. Tadic, Appeals Chamber Decision on the Defence Motion for Interlocutory Appeal on Jurisdiction, para. 97 (2 October 1995). Similarly, Prosecutor v. Delalic, Appeals Chamber Judgment, Case No. IT-96-21-A, para. 172 (20 February 2001). 39 Originally conceived in the Preamble of the Convention Respecting The Laws and Customs of War on Land, July 29, 1899, see the so-called Martens Clause: “Until a more complete code of the laws of war is issued, the High Contracting Parties think it right to declare that in cases not included in the Regulations adopted by them, populations and belligerents remain under the protection and empire of the principles of international law, as they result from the usages established between civilized nations, from the laws of humanity, and the requirements of the public conscience.” International Convention with respect to the Laws and Customs of War on Land, opened for signature 29 July 1899, [1901] ATS 131 (entered into force 4 September 1900). See also Theodor Meron, The Martens Clause, Principles of Humanity, and Dictates of Public Conscience, 94 AMERICAN JOURNAL OF INTERNATIONAL LAW 78(2000). 40 Theodor Meron, The Humanization of Humanitarian Law, 94 AMERICAN JOURNAL OF INTERNATIONAL LAW 239(2000). 41 Gary D. Solis, The Law of Armed Conflict: International Humanitarian Law in War 20-27 (Cambridge University Press. 2010).
humanitarian law. This means that AWS have to be programmed to conform to different
levels of aggressiveness, all of which have to be commensurate with the applicable
rules of international humanitarian law. A recent example of such a change occurred in
Afghanistan when, due to pressure from the Afghanistan government and after a
number of strikes that targeted civilian objectives, military forces changed their own
behavior considerably.42 Any program employed would have to be constructed in a way
to be able to conform with such policy decisions, which may vary considerably under
different circumstances, and may be subject to change within the same conflict
scenario.43
UMS must be able to operate in light of the principle of distinction as well as the
principle of proportionality. This means that the applicable rules must be converted into
a digital format that a computer applies in a given situation. And while a number of civil
law countries’ legal systems have attempted to make forays into this area, the
convoluted nature of armed conflict poses challenges that go beyond the disputes that
ordinary courts have to contend with. Computers are traditionally better at dealing with
quantitative elements than with qualitative assessments. While there have been
42 See generally Michael N. Schmitt, Targeting and International Humanitarian Law in Afghanistan, 39 ISRAEL YEARBOOK ON HUMAN RIGHTS 307, 312 et seq. (2009). For a theoretical exposition of how legal rules may change over time, including in the context of the LOAC, see ANTHONY D’AMATO, International Law, Cybernetics, and Cyberspace, in Computer Network Attack and International Law - Naval War College International Law Studies "Blue Book", 59 (FIND YEAR). For news report about this change, see Carlotta Gall, Afghan President Assails U.S.-Led Airstrike That He Says Killed 95, N.Y. TIMES, Aug. 24, 2008, at A6. These developments have contributed to a dramatic increase in the NATO forces' demand for UAVs and critical UAV surveillance capabilities. See David Ignatius, What a Surge Can't Solve in Afghanistan, WASH. POST, Sept. 28, 2008, at B7 (noting that Secretary of Defense Robert Gates has pushed for a major increase in ISR assets in Afghanistan); Anna Mulrine, Drones Fill the Troops Gap in Afghanistan, U.S. NEWS & WORLD REP., Sept. 25, 2008, at 30. 43 Afghanistan C.J. Chivers, A Changed Way of War in Afghanistan’s Skies, 16 January 2012, A1.
impressive advances in cognitive technologies, it remains to be analyzed whether the
principles of distinction and proportionality can safely be entrusted to a digital code. This
is even more important in light of the fact that these considerations are multi-faceted
and require that a plethora of factors be taken into account and weighed against one
another, including elements of conscientiousness and in which technological advances
are only in their infancy.
2. The Principle of Distinction
The principle of distinction mandates that any military action must distinguish between
combatants and civilians as well as between military and civilian objects. This distinction
between a person and an object that possesses a military character as opposed to one
that is of a civilian character therefore takes on crucial importance. Importantly,
international humanitarian law operates under the assumption that an individual who
does not qualify as a combatant is to be considered as a civilian. This bedrock principle
of international humanitarian law was already incorporated into the earliest international
instrument in the field of international humanitarian law, the 1868 St. Petersburg
declaration.44 It subsequently found inclusion into the Hague Convention Respecting the
Laws and Customs of War on Land through its annex,45 and finally into the additional
44 The preamble states partially that
“[…] That the only legitimate object which States should endeavour to accomplish during war is to weaken the military forces of the enemy […]”. Declaration Renouncing the Use, in Time of War, of Explosive Projectiles Under 400 Grammes Weight, Saint Petersburg, 29 November (11 December) 1868.
45 Regulations Respecting the Laws and Customs of War on Land, annexed to the Convention (IV) respecting the Laws and Customs of War on Land, The Hague, 18 October 1907. Allusions to the principle of distinction are prevalent throughout the preamble. E.g.
protocols to the 1949 Geneva Conventions. Additional Protocol I, applicable in
international armed conflicts, as well as Additional Protocol II, applicable in non-
international armed conflicts, contain the principle of distinction.
The former enshrines this general rule in Article 48 AP I which are refined in subsequent
provisions, namely by outlawing the targeting of individual civilians,46 unless they take a
direct part in hostilities,47 the targeting of historic monuments, works of art or places of
worship.48 Moreover, AP I prohibits not only attacks on civilians or objects of a civilian
nature, but goes further by prohibiting attacks that are “indispensable to the survival of
the civilian population”, as well as the natural environment and ‘installations containing
dangerous forces’.49 In addition, AP I contains provisions that prohibit certain methods
of attack, namely those that are by their nature indiscriminate.50 These rules are an
expression of the idea that underlies IHL, i.e. that not only must an attacker distinguish
between civilian and military targets, but an attacker must do so in a discriminatory
fashion. This would mean that if all that is available for attack is a weapon system that is
“Thinking it important, with this object, to revise the general laws and customs of war, either with a view to defining them with greater precision or to confining them within such limits as would mitigate their severity as far as possible”
or “these provisions, the wording of which has been inspired by the desire to diminish the evils of war, as far as military requirements permit, are intended to serve as a general rule of conduct for the belligerents in their mutual relations and in their relations with the inhabitants”.
46 Ibid Article 51(2). 47 Ibid Article 52(3). 48 Ibid Article 53. 49 See ibid, Articles 54, 55 and 56, respectively. 50 Ibid Article 51(4) states:
‘Indiscriminate attacks are prohibited. Indiscriminate attacks are: (a) those which are not directed at a specific military objective; (b) those which employ a method or means of combat which cannot be directed at a specific
military objective; or (c) those which employ a method or means of combat the effects of which cannot be limited as
required by this Protocol.’
designed to destroy a large-scale target, an attack can only be carried out in a fashion
that adheres to these principles.
These rules are not only contained in treaty law, but are generally held to form part of
customary international law as well. The ICJ’s Nuclear Weapons Advisory Opinion
posited that the principle of distinction was of utmost importance when it considered the
principle of distinction to be among “a great many rules of humanitarian law applicable
in armed conflict [that] are […] fundamental to the respect of the human person and
‘elementary considerations of humanity’”.51 Without making such an explicit reference,
the Court appears to consider the principle of distinction therefore to be at the level of
jus cogens, or at the very least, of a higher order52 when it considered them to
“constitute intransgressible principles of international customary law”.53 It is not
important whether the principle of distinction has reached the status of jus cogens.54 In
addition to the ICJ, the Eritrea-Ethiopia Claims Commission found that Article 48 API to
51 Legality of the Threat or Use of Nuclear Weapons, Advisory Opinion, I.C.J. Reports 1996, 226, para. 79. 52 For a powerful critique on a conceptual level, see Prosper Weil, Towards Relative Normativity in International Law, 77 AMERICAN JOURNAL OF INTERNATIONAL LAW 413(1983). For a counterpoint, see John Tasioulas, In Defence of Relative Normativity: Communitarian Values and the Nicaragua Case, 16 OXFORD JOURNAL OF LEGAL STUDIES 85(1996). 53 Legality of the Threat or Use of Nuclear Weapons, Advisory Opinion, I.C.J. Reports 1996, 226, para. 79; Dinah Shelton, International Law and "Relative Normativity" 159, 164, 166 International law (Malcolm D. Evans ed., Oxford Univ. Press 2. ed. 2006). 54 Arguments in favor are advanced e.g. by Jean-François Quéguiner, The Principle of Distinction: Beyond an Obligation of Customary International Humanitarian Law, in The Legitimate Use of Military Force: The Just War Tradition and the Customary Law of Armed Conflict, 161, 171 (Howard M. Hensel ed. 2008).
be an expression of customary international law.55 It appears thus evident that the
principle of distinction has become a rule of customary international law.56
Even though on this basis the principle appears clear, there are nevertheless a
considerable number of difficulties in practice in cases where a target can be both
civilian in nature as well as military in nature. The most often cited example for such
targets include bridges which are used for civilian purposes, but by which may also be
used to supply military units.57 Other such installations include broadcasting stations or
the energy network.58 As pointed out above, while the textual basis for the distinction
between civilian and military targets appears clear, realities on the ground oftentimes
leave ambiguous whether a target is legitimate or not.
Even when a decision has been reached to carry out an attack, the principle of
distinction requires that it be carried out with weaponry that are not indiscriminate – a
55 Eritrea-Ethiopia Claims Commission, Western Front, Aerial Bombardment and Related Claims - Eritrea's Claims 1, 3, 5, 9-13, 14, 21, 25 & 26, 45 ILM 396, 417, 425 (2006). 56 The fundamental nature of the principle of distinction, unlike other principles of the LOAC, has not been put into question. One commentator appears to have doubts as to the customary international law nature of the principle, raising constitutional concerns in the US context as being void for vagueness. W. Hays Parks, Air War and the Law of War, 32-1 AIR FORCE LAW REVIEW 1, 174 (1990). See also William H. Taft, The Law of Armed Conflict after 9/11: Some Salient Features, 28 YALE JOURNAL OF INTERNATIONAL LAW 319, 323 (2003). 57 Marco Sassoli, Legitimate Targets of Attacks Under International Humanitarian Law, International Humanitarian Law Research Initiative, Background Paper prepared for the Informal High-Level Expert Meeting on the Reaffirmation and Development of International Humanitarian Law, Cambridge, January 27-29, 2003, Harvard Program on Humanitarian Policy and Conflict Research (7-8). 58 Christine Byron, International Humanitarian Law and Bombing Campaigns: Legitimate Military Objectives and Excessive Collateral Damage, 13 Yearbook of International Humanitarian Law 183-186 (Michael N. Schmitt et al. eds., 2010).
prohibition that either derives from specific – if applicable – treaty law or more generally,
customary international law rules.
Furthermore, military objectives are considered to be those that by “nature, location,
purpose, or use make an effective contribution to military action and whose total or
partial destruction, capture, or neutralization, in the circumstances ruling at the time,
offers a definite military advantage”.59 Only when both the specific characteristic (i.e.
that “nature, location, purpose or use make an effective contribution to military action)
as well as the damage or neutralization offers a military advantage at the time is an
object a military one for the purposes of AP I. Each of these elements can be further
clarified.60 The term “nature” more closely describes the military significance of a
particular object, which could consist of weapons, depots or command posts.61 The term
“location” is a reference to a geographic space that has “special importance to military
operations”.62 An example for a particular location may be a river crossing or a mountain
pass, i.e. a place which possesses important characteristics without which a military
advantage cannot be gained or is considerably harder to obtain. When an object is to be
used for military reasons in the future, the object qualifies as a military objective through
“purpose”.63 Finally, “use” means that the enemy is presently utilizing an object
militarily.64 The element of use makes clear that the international humanitarian law
59 See Article 52 (2) AP I. 60 See generally Claude Pilloud et al., Commentary on the Additional Protocols of 8 June 1977 to the Geneva Conventions of 12 August 1949 para. 2020 et seq. (Martinus Nijhoff Publishers. 1987). 61 Ibid., para. 2020. 62 Ibid., para. 2021. 63 Ibid., para. 2022. 64 Ibid.
incorporates a dynamic element in that civilian objectives may become military ones if
they are being used by the enemy for military ends. It is clear that in these situations
there are heightened requirements in order for an attack commensurate with the
requirements of Article 52 (2) AP I. The weighing and balancing that must take place in
these instances requires a higher level of analysis compared to situations in which an
attack would be directed only against military objectives or combatants. In analogous
fashion, the same applies to individuals who, though once civilian, can potentially be
considered to “directly participate in hostilities”.65 Another element of the principle of
distinction is that attacks must be specific so as to treat “as a single military objective a
number of clearly separated and distinct military objectives located in a city, town,
village or other area containing a similar concentration of civilians or civilian objects.”66
The following example may shed light on this problem as well as the problem of the
differing values that individuals may place on a particular element of the analysis. In this
context, it should be borne in mind that it is often unclear what constitutes a breach of
international humanitarian law. One author describes a situation in which a high-ranking
general, after having watched two hours of footage streamed into his office on a
different continent gave orders to destroy a compound. He gave the order despite the
presence of civilians, because insurgents entered and left openly carrying weapons.67
The presence of insurgents should have been a signal to the civilians, according to the
general, that the compound was now a legitimate – and at least according to his 65 See Article 51 (3) AP I. 66 Article 51 (4)(a) and 51 (5)(a) AP I. 67 P. W. SINGER, Wired for War: The Robotics Revolution and Conflict in the Twenty-first Century 347-348 (Penguin Press. 2009).
analysis: a legal – target. This is a good example of the problems that the principle of
distinction – and in this case the principle of proportionality – carries with it: if a high-
ranking general interprets the international humanitarian law provisions in an – at the
very least – debatable manner, it is far from clear whose analysis should be used to
formulate a code that would eventually determine the outcome of an attack in similar
situations.
This is even more important given the increased complexity that today’s conflicts and
the conflicts of the future pose. As the traditional battlefield is being replaced by a battle
space with a much higher degree of complexity combat operations are no longer
confined to a particular area, but rather operations occur at different times or
simultaneously, in different geographical areas and by different means.68 Because of
this increase in complexity in which military and civilian objects become increasingly
used by combatants and because combatants become increasingly intermingled with
civilians, distinction becomes ever more crucial. At the same time, targets are
increasingly found in populated areas – making distinction more difficult. For the
purposes of UMS, it is therefore imperative to determine a priori how well a UMS can
distinguish between civilians and civilian objects on the one hand and combatants and
legitimate military objectives on the other.
68 Michael N. Schmitt, War, Technology, and The Law of Armed Conflict 137, 149 The Law of War in the 21st Century Weaponry and the Use of Force (Helm, Anthony M. 2007). Note that the term battle space has now been adopted by the US military to denote local conflicts as well as domestic soil, including the “global commons”. See National Military Strategy of the United States, 2004, 5, available at http://www.defense.gov/news/mar2005/d20050318nms.pdf. See for the use of the term in the 2011 National Military Strategy, Nicholas R. Krueger, The 2011 National Military Strategy: Resetting a Strong Foundation, National Security Watch, The Institute of Land Warfare, AUSA (May 2, 2011), available at http://www.ausa.org/publications/ilw/ilw_pubs/Documents/NSW%2011-2-web.pdf.
In the case of AWS this means that the underlying software would have to be able to
determine whether a particular target is civilian or military in nature.69 Moreover, the
AWS would have to be programmed so that it also takes account of the requirement
that in cases of uncertainty it would abort the attack.70 A number of weapons today are
capable of determining – based on pre-programmed characteristics, such as shape and
dimensions – a target’s military nature.71 Once a sufficient number of characteristics of
the target have been reconciled with the pre-programmed version, the weapon system
can initiate an attack. This type of matching is mechanical, based on quantitative data
and even if one were to argue that there is still an unacceptable amount of ambiguity, it
appears that the recent advances regarding this technology will enable such systems to
function with the required accuracy in the near future for certain types of targets.72 With
respect to individuals, the situation is much more problematic. Without going into the
details of the debate, it would be important to program software so that a system can
distinguish not only civilians and combatants, but also civilians from those that take
69 One interesting proposal is proffered is mandating that UVs would not target humans, but only weapon systems. See John S Canning, A Concept for the Operation of Armed Autonomous Systems on the Battlefield, (2006) 3rd Annual Disruptive Technology Conference <http://www.dtic.mil/ndia/2006disruptive_tech/canning.pdf>. While this may minimize the danger somewhat, it is unclear how this would alleviate the problem of, for example, someone carrying a rifle for safety reasons or for hunting purposes. 70 With respect to civilians, see Additional Protocol I Article 50(1), with respect to civilian objects, see Additional Protocol I Article 52(3). 71 Robert Sparrow, ‘Killer Robots’ (2007) 24 Journal of Applied Philosophy 62, 63. More recently, see Michael Lewis et al, ‘Scaling Up Wide-Area-Search Munition Teams’ (May-June 2009) 24 IEEE Intelligent Systems 10. 72 Note however that specifically with respect to Additional Protocol I Article 51(4)(c) there has been considerable controversy since it arguably contains elements of proportionality and thus may not be merely a quantitative assessment. See generally Stefan Oeter, ‘Methods and Means of Combat’, in Dieter Fleck (ed) The Handbook of International Humanitarian Law (Oxford University Press, 2nd ed, 2008) 119, 201 et seq.
“active part in hostilities”.73 While this is difficult enough for humans, it may be – at this
time – impossible for an AWS as this is again a move away from quantitative to
qualitative analysis.
One example74 may be a potential counter-insurgency operation in a village, the
inhabitants of which belong to a group the members of which carry a dagger, the
kirpan,75 although purely for religious reasons and not as a weapon. After receiving a tip
that insurgents are hiding in a home, a unit attempts to enter a home although
unbeknownst to the soldiers no insurgents are present. Just as the soldiers are about to
enter the compound, some boys carrying the kirpan are running after a ball that one of
them kicked towards the gate. When the soldiers enter the compound the father realizes
the situation and screams towards the boys – in a language that the soldiers do not
understand – to stay away from the gate and moves towards the gate to protect his
children. One would expect that a human is able to interpret the situation in a way that
indicates that these individuals are not a threat, but realizes the situation for what it is,
i.e. two children chasing a ball. On the other hand this could also be interpreted as two
quickly approaching targets carrying a weapon, with another potential target running
towards the unit in an agitated and potentially threatening manner. It becomes clear that
the ability to distinguish these two interpretations is crucial in combat. Distinguishing a 73 See generally Article 50 AP I, which negatively delineates civilians. See generally Sharkey, Lin, Chapter 7, 7. 74 Modeled after an example from Marcello Guarini & Paul Bello, Robotic Warfare: Some Challenges in Moving from Non-Civilian to Civilian Theaters 149, 150 Robot Ethics: The Ethical and Social Implications of Robotics (Patrick Lin et al. eds., MIT Press. 2012). 75 The kirpan is viewed as a ceremonial sword and can be a few inches or up to three feet long, representing the Sikh struggle against injustice and oppression. See Doris R. Jakobsh, Sikhism 60 (University of Hawaii Press. 2012); Rishi Singh Bagga, Living by the Sword: The Free Exercise of Religion and the Sikh Struggle for the Right to Carry a Kirpan, II-3 THE MODERN AMERICAN 32(May 2006).
weapon from a cultural or religious symbol, distinguishing a person in fear of his
children with an agonized face from someone with a threatening face, distinguishing the
children playing rather than being a threat requires cognitive abilities that – at least at
this stage – far surpass the abilities of robotics. The requirements for this capability are
especially demanding because at stake are human lives or at the very least human well-
being should the situation be interpreted according to the second scenario just
described.
Other examples that could elicit similar erroneous responses may be children who are
forced to carry weapons76 and who, for a system flying at even low altitude, may look
like combatants. It is also clear that situations may arise where seemingly innocuous
behavior may trigger a preprogrammed attack. An individual carrying a rifle may be
hunting or carry such a weapon for protective purposes. This is not at all uncommon
and while in such instances mistakes could equally be made by humans, these
situations are very context dependent. Thus, already at the stage of the principle of
distinction, qualitative elements become an important part of the analysis and a mere
reliance on quantitative factors is no longer sufficient.
76 Naomi Cahn, Poor Children: Child "Witches" and Child Soldiers in Sub-Saharan Africa, 3 Ohio State Journal of Criminal Law 413, 418 (2006).
3. The Principle of Proportionality
The principle of proportionality creates similar – and potentially even greater –
challenges. The reason for this can be formulated as follows: proportionality is a term
that cannot be defined in the abstract, but rather the determination of whether an attack
meets the requirements of the principle of proportionality depends on the particular
circumstances of the attack.77 This is further complicated by the fact that almost every
object, even though it may not be a military one in nature, can be transformed into a
military one.78 In this sense, AWS take the challenges that UAVs are currently facing
considerably further: while removing the individual combatant from the battlefield is in
itself not unproblematic as evidenced by the debates surrounding drone strikes,79 AWS
remove a combatant from the decision-making process over a particular situation
altogether. This shifts the burden of the decision-making progress. It is no longer a
combatant (be it the pilot in the case of manned aircraft or the operator in remote
controlled drones) that triggers the launch of weapon, but rather, in the case of fully
autonomous weapons, the decision shifts to the programming stage of the AWS system
software. It is at this stage where the decisions that would otherwise be left to
individuals would have to be made in an anticipatory fashion, including that of whether
an attack is proportional or not.
77 See William H. Boothby, Weapons and the Law of Armed Conflict 79 (Oxford Univ. Press. 2009); Oliver O'Donovan, The Just War Revisited 62 (Cambridge University Press. 2003); Yoram Dinstein, The Conduct of Hostilities Under the Law of International Armed Conflict 131 (Cambridge University Press 2nd ed. 2010). 78 Yoram Dinstein, The Conduct of Hostilities Under the Law of International Armed Conflict 130 (Cambridge University Press 2nd ed. 2010). 79 See ___, above.
Some of the issues that can be expected to emerge are the following: (1) Can it be left
to AWS to make decisions over whether to attack and if so with what type of weapons?;
(2) if the answer is affirmative, what limitations does the proportionality principle place
on their use?; (3) what precautions must be taken to avoid breaches of the
proportionality principle in situations that remove the decision-making process from
immediate human supervision?; (4) whose judgments flow into the a priori determination
over whether a particular attack is proportional?;80 and (5) can formulaic, software-
compatible rules be crafted so that important elements in determining proportionality
can form part of the equation? Each of these questions will have to be clearly answered
prior to the deployment of an AWS and at least at this stage it is far from certain that
these conditions can be met.
Versions of the principle of proportionality have been in existence for a long time. St.
Thomas Aquinas introduced the principle of double effect previously which contained an
early version of the principle of proportionality.81 Grotius posited that for the sake of
saving many one “must not attempt anything which may prove the destruction of
innocents, unless for some extraordinary reason”.82 The St. Petersburg declaration
contained language to the effect that because the “the only legitimate object which
States should endeavour to accomplish during war is to weaken the military forces of
80 See also Lin, Chapter 1, p. 8. 81 T. A. Cavanaugh, Double-Effect Reasoning: Doing Good and Avoiding Evil 2 et seq. and 181 (Clarendon Press ; Oxford University Press. 2006).For an even earlier example of precursors to the principle, see Thomas M. Franck, On Proportionality of Countermeasures in International Law, 102 AMERICAN JOURNAL OF
INTERNATIONAL LAW 715, 723 (2008). 82 Hugo Grotius, De Jure Belli ac Pacis Libri Tres, Book III, Chapter XI, para. VIII.
the enemy […] this object would be exceeded by the employment of arms which
uselessly aggravate the sufferings of disabled men, or render their death inevitable”.83
Similar rules were put in place – sometimes in a binding fashion, sometimes in a non-
binding fashion – until the middle of the 20th century.84 But it was not until the adoption
of the AP I to the Geneva Conventions of 1949 in the late 1970s that the principle of
proportionality – although without using that particular language – was specifically
included in a binding and wide-ranging document.85
The principle of proportionality does not find explicit mention in AP I, but rather finds
reflection in a number of provisions, most importantly in Article 51(5)(b) and Article 57(2)
AP I. The former prohibits an attack the results of which would be excessive in relation
to the military advantage anticipated. Specifically, the provision states:
“5. Among others, the following types of attacks are to be considered as
indiscriminate:
[…]
83 St. Petersburg Declaration Renouncing the Use, in Time of War, of Explosive Projectiles Under 400 Grammes Weight, Nov. 29/Dec.11, 1868, 1 AJIL Supp. 95 (1907). 84 Thomas M. Franck, On Proportionality of Countermeasures in International Law, 102 AMERICAN
JOURNAL OF INTERNATIONAL LAW 715, 723-724 (2008); William J. Fenrick, The Rule of Proportionality and Protocol I in Conventional Warfare, 98 MILITARY LAW REVIEW 91, 95-98 (1982); Judith Gail Gardam, Proportionality and International Law, 87 AMERICAN JOURNAL OF INTERNATIONAL LAW 391, 394 et seq. (1993). 85 Bernard L. Brown, The Proportionality Principle in the Humanitarian Law of Warfare: Recent Efforts at Codification, 10 CORNELL INTERNATIONAL LAW JOURNAL 134, 136 (1976-1977).
(b) an attack which may be expected to cause incidental loss of civilian life, injury
to civilians, damage to civilian objects, or a combination thereof, which would be
excessive in relation to the concrete and direct military advantage anticipated.”86
This provision functions as a general protection for civilians against non-discriminatory
attacks, but it obviously fraught with problems through the use of the terms
“excessive”.87 It is not clear what this term means in the abstract and can only be
determined in the specific circumstances of a particular situation.88 In order to minimize
the legal exposure of commanders, Article 57(2) AP I – itself reflective of numerous
proportionality concerns – was introduced. It refers to precautions that must be taken
with respect to avoiding or minimizing incidental loss of life or injury to civilians and
damage to civilian objects,89 or with respect to attack the civilian loss of life, injury or
damage would prove to be “excessive in relation to the concrete and direct military
advantage anticipated”.90 It is again, through the insertion of the term “excessive” not
86 Article 51(5)(b) AP I. 87 According to Fenrick, the terms “excessive” and “disproportionate” are more or less interchangeable. See William J. Fenrick, The Rule of Proportionality and Protocol I in Conventional Warfare, 98 MILITARY
LAW REVIEW 91, 97 (1982). Moreover, a number of countries reportedly held the view that the incorporation of the principle of proportionality was merely a codification of existing customary law. See Ibid., 104. 88 GARY D. SOLIS, The Law of Armed Conflict: International Humanitarian Law in War 273 (Cambridge University Press. 2010). 89 Article 57(2)(a)(ii) AP I. 90 Article 57(2)(a)(iii) AP I. The full provision reads:
“2. With respect to attacks, the following precautions shall be taken: (a) those who plan or decide upon an attack shall: (i) do everything feasible to verify that the objectives to be attacked are neither civilians nor civilian objects and are not subject to special protection but are military objectives within the meaning of paragraph 2 of Article 52 and that it is not prohibited by the provisions of this Protocol to attack them; (ii) take all feasible precautions in the choice of means and methods of attack with a view to avoiding, and in any event to minimizing, incidental loss or civilian life, injury to civilians and damage to civilian objects;
clear what this means in the abstract, but rather has to be determined on a case-by-
case basis. This choice of wording is a result of the tension mentioned above, between
the competing interests during armed conflict: gaining military advantage, while
protecting the civilian population.91
Importantly, what has to be weighed against one another in this context are the
following elements. On the one hand, there may be incidental loss of civilian life or
damage to civilian objects. However on the other hand, one has to weigh not the actual
outcome of the attack, but rather what had been the anticipated result of the attack.
What has to be taken into consideration therefore is the potential military advantage that
could be gained in the mind of the attacker.92 Importantly, it may be permissible to inflict
(iii) refrain from deciding to launch any attack which may be expected to cause incidental loss of civilian life, injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the concrete and direct military advantage anticipated; (b) an attack shall be cancelled or suspended if it becomes apparent that the objective is not a military one or is subject to special protection or that the attack may be expected to cause incidental loss of civilian life, injury to civilians, damage to civilian objects, or a combination thereof, which would be excessive in relation to the concrete and direct military advantage anticipated; (c) effective advance warning shall be given of attacks which may affect the civilian population, unless circumstances do not permit.”
91 This has led some authors to claim that the principle of proportionality is too vague a concept and proportionality would only be implicated when ‘acts have occurred that are tantamount to the direct attack of the civilian population’. W Hays Parks, ‘Air War and the Law of War’ (1990) 32 Air Force Law Review 1, 173; Michael N Schmitt, ‘Targeting and International Humanitarian Law in Afghanistan’ (2009) 39 Israel Yearbook on Human Rights 307, 312. For an opposing view, see Yoram Dinstein, The Conduct of Hostilities Under the Law of International Armed Conflict (Cambridge University Press, 2nd ed, 2010) 120-121. Problems relating to proportionality assessments in the context of targeted killings have been pointed out by Noel Sharkey, ‘Death Strikes From the Sky: The Calculus of Proportionality’ (Spring 2009) IEEE Technology and Society Magazine 17, 19. The idea that the principle of proportionality applies in armed conflict has been affirmed strongly by the Supreme Court of Israel. See HCJ 769/02 Public Committee against Torture in Israel et al v Government of Israel et al, [2006] especially 30-33, <http://elyon1.court.gov.il/Files_ENG/02/690/007/a34/02007690.a34.pdf>. 92 Yoram Dinstein, The Conduct of Hostilities Under the Law of International Armed Conflict 132 (Cambridge University Press 2nd ed. 2010).
extensive civilian casualties if at the time of the attack the concrete and direct military
advantage was not excessive in relation to the casualties inflicted.
This tension was pointed out in a 2000 report to the International Criminal Tribunal for
the Former Yugoslavia (ICTY) Prosecutor, which addressed the difficulty in applying the
principle of proportionality and professed that ‘[o]ne cannot easily assess the value of
innocent human lives as opposed to capturing a particular military objective’.93 Some
have suggested that the discrepancy between loss of life / injury / damage to objects on
the one hand and the direct military advantage anticipated must be clearly
disproportionate.94 The insertion of such a requirement does nothing to solve the
problem – and if anything, adds further confusion as the language is simply not
supportive of such an interpretation.95 The fluidity that characterizes the principle of
proportionality raises another question, namely whether a singular set of proportionality
assessments actually exists which could thus be programmed. The answer to this
question is obviously negative and it is clear that a military commander may arrive at
different conclusions in different situations and would most certainly differ in that
assessment from a human rights lawyer. Or, as one commentator put it, “[p]roportion is
93 International Criminal Tribunal for the Former Yugoslavia (ICTY), Final Report to the Prosecutor by the Committee Established to Review the NATO Bombing Campaign Against the Federal Republic of Yugoslavia, (June 8, 2000) 39 International Legal Materials 1257, [48]. 94 Henckaerts and Doswald-Beck, Customary International Humanitarian Law, 46. GARY D. SOLIS, The Law of Armed Conflict: International Humanitarian Law in War 274 (Cambridge University Press. 2010). 95 Whether reference to Article 8(2)(b)(iv) of the Rome Statute in this regard is helpful is an open question. The criminal law character of the Rome Statute would indicate otherwise. But see Gary D. Solis, The Law of Armed Conflict: International Humanitarian Law in War 274 (Cambridge University Press. 2010).
an elastic concept, but not indefinitely elastic”.96 And while it is clear that combatants
must have some discretion in deciding whether an attack is proportionate, that
discretion is not unfettered.97
It is not necessary to decide this long-running debate at this point. What is important for
present purposes is the question of how autonomous UMS could potentially carry out a
proportionality determination. As has been pointed out elsewhere, proportionality plays
a role in a variety of stages of attack:
“[P]roportionality is a factor in the selection of the target. If civilian losses are
inevitable, because of either the intermingling of civilian and military targets or
the dual character of the target itself, these must be balanced against the military
advantage. […] [T]he means and methods of attack must be assessed. Some
weapons are more likely to involve indiscriminate damage than others. Aerial
bombardment makes the distinction between combatants and noncombatant
more difficult and thus, in some circumstances, may be a disproportionate means
of achieving the military objective. Finally, even if these requirements are met,
96 Oliver O'Donovan, The Just War Revisited 62 (Cambridge University Press. 2003); William J. Fenrick, The Rule of Proportionality and Protocol I in Conventional Warfare, 98 MILITARY LAW REVIEW 91, 102 (1982). The latter posits that “[s]ince the quantities being measured, civilian losses and military advantage, are dissimilar, it is not possible to establish any reasonably exact proportionality equation between them.” See also Judith Gail Gardam, Proportionality and International Law, 87 AMERICAN
JOURNAL OF INTERNATIONAL LAW 391(1993). She points out that the law of armed conflict “is based on the fundamental principle that belligerents do not enjoy an unlimited choice of means to inflict damage on the enemy. Since the entry into force of Protocol I to the Geneva Conventions of 12 August 1949, and Relating to the Protection of Victims of International Armed Conflicts, proportionality has been both a conventional and a customary principle of the law of armed conflict.” 97 Thomas M. Franck, On Proportionality of Countermeasures in International Law, 102 AMERICAN
JOURNAL OF INTERNATIONAL LAW 715, 726 (2008).
the conduct of the attack itself must not be negligent and involve unnecessary
civilian casualties.”98
It should be borne in mind here that what was said about the principle of distinction
applies equally in the case of the principle of proportionality. First and foremost, the
principle of proportionality has attained the status of customary international law.99 In
addition, the increasing complexity of today’s battle spaces, as opposed to the
battlefields of conventional wars in the past, make determinations of proportionality
even more complex. As one commentator puts it, “[t]he more nebulous the military
objective is, the greater the need for caution in use of the weapons likely to cause
‘excessive’ collateral damage to civilians or civilian objects”.100
All of these considerations refer to qualitative assessments rather than those of a
quantitative nature. As some commentators observe, the principle of proportionality
“clearly highlights the difference between quantitative and qualitative decisions and the
need for human decision-making”.101 Whereas it is generally accepted that machines
are good at making quantitative calculations, humans are better adapted to making
these types of decisions which are by nature subjective.102 As has been pointed out by
98 Judith Gail Gardam, Proportionality and International Law, 87 AMERICAN JOURNAL OF INTERNATIONAL
LAW 391, 407 (1993). 99 Robin Geiß, MPEPIL, Land Warfare, mn. 15. 100 Yoram Dinstein, The Conduct of Hostilities Under the Law of International Armed Conflict 131 (Cambridge University Press 2nd ed. 2010). 101 Tony Gillespie & Robin West, Requirements for Autonomous Unmanned Air Systems set by Legal Issues, 4:2 THE INTERNATIONAL C2 JOURNAL 1, 13 (2010). 102 See in this regard also Sparrow, positing that “decisions about what constitutes a level of force proportionate to the threat posed by enemy forces are extremely complex and context dependent and it is
a number of authors, the problem lies in the fact that it is difficult to assign a value to the
destruction of e.g. a tank in exchange for the killing of civilians.103 Since there are –
often constantly changing – variables on both sides of the equation and since the
balancing of values depends on the individual making that calculation, it is by its nature
subjective.104 This subjective interpretation of the provision has been criticized as
allowing for abusive justifications which are ultimately hard if not impossible to disprove.
Cassese for example at the time argued for an objective standard105 and while the
standard may have been desirable, the wording of Article 51 AP I does not contain such
a more objective reference.
For AWS, proportionality considerations play a part in all of these stages as well. With
respect to target selection the program would have to be designed so as to anticipate all
potential decisions, either by programming them in or by designing decision rules that
are capable of making such decisions with a myriad of factors to be weighed.
Concerning the second element, i.e. the means to be employed, an AWS would have to
determine which weapon would have what type of effect under any given seemingly unlikely that machines will be able to make these decisions reliably for the foreseeable future.” William H. Boothby, Weapons and the Law of Armed Conflict 233 (Oxford Univ. Press. 2009); Robert Sparrow, Building a Better WarBot: Ethical Issues in the Design of Unmanned Systems for Military Applications, 15 SCIENCE AND ENGINEERING ETHICS 169, 178 (2009). 103 Yoram Dinstein, The Conduct of Hostilities Under the Law of International Armed Conflict 133 (Cambridge University Press 2nd ed. 2010); Thomas M. Franck, On Proportionality of Countermeasures in International Law, 102 AMERICAN JOURNAL OF INTERNATIONAL LAW 715, 729 (2008). For a different view, see the separate and dissenting opinion of Judge Nieto Navia in Prosecutor v Stanislav Gali6 (Separate and Partially Dissenting Opinion) (5 December 2003) IT-98-29-T (Trial Chamber, ICTY), para. 104 et seq. [Separate and Partially Dissenting Opinion of Judge Nieto-Navia]. 104 Yoram Dinstein, The Conduct of Hostilities Under the Law of International Armed Conflict 132 (Cambridge University Press 2nd ed. 2010); Waldemar Solf, New Rules for Victims of Armed Conflicts: Commentary on the Two 1977 Protocols Additional to the Geneva Conventions of 1949 310 (Michael Bothe et al. eds., Martinus Nijhoff Publishers. 1982). 105 Antonio Cassese, Means of Warfare: The Traditional and the New Law 161, 175 et seq. The New Humanitarian Law of Armed Conflict (Antonio Cassese ed., Editoriale scientifica. 1979).
circumstances. The question to be asked thus is whether there are other weapons or
means available at the particular moment in time that would cause less suffering while
still capable of achieving the same goal. While this may be easy in the abstract, the
close proximity of civilians in modern battle spaces and the constantly shifting
circumstances may make this determination much more difficult than it first appears.
Finally, an AWS would have to be capable of determining the extent of civilian losses
that are acceptable in any given situation. In either of these determinations the lack of
important information would have to trigger a veto so as to abort the mission or an AWS
would have to ask for human input for determining whether and if so, how to conduct a
particular mission.106
As pointed out above, there is no clear formula for either of these determinations and
given that even after a considerable passage of time and a considerable amount of
discussion there is no agreement over this element between the countries, it appears
curious at best and potentially naïve to believe that quantitative analysis would yield the
correct results. At least at this stage – and because of the problems outlined above
regarding qualitative considerations in the context of proportionality considerations this
may be true not only for the foreseeable future – it therefore is evident that despite the
impressive advances in computing technology the use of AWS would be limited to such
an extent as to render them ineffective for the large majority of operations. This means
that AWS that are fully autonomous may only be used in situations in which a target is
106 Tony Gillespie & Robin West, Requirements for Autonomous Unmanned Air Systems set by Legal Issues, 4:2 THE INTERNATIONAL C2 JOURNAL 1, 13 (2010).
remote and the situation appears with the potential for minimal civilian involvement, i.e.
in cases of certainty over such circumstances at the beginning of the mission.107 This in
turn would preclude the use of AWS in all other situations which constitute the very
large majority of cases, especially in today’s more complex battle spaces. More than
any other area, this aspect of the legal framework is a subjective one. Military
experience as well as legal expertise is essential to decision-making in this context.
4. AWS and Individual Responsibility
Throughout the history of organizational responsibility, it has more often than not been
the case that individuals who either fought or guarded the front lines were held
accountable for their actions rather than their superiors. This is true for military
operations as well as for police action. The problem in these instances have regularly
been that the direct participants in criminal acts oftentimes held positions with relatively
little power, yet those that designed the system were often able to evade criminal
responsibility. This changed somewhat with the trials of the Nuremberg and Tokyo
tribunals after WW II when a number of high-level officials were prosecuted, yet even in
more modern times, this phenomenon persists. There are many reasons as to why
holding individuals responsible for crimes committed in the battle space is important.
While one of the foundational ideas of armed conflict is that wounding or killing other
individuals – unlike in times of peace – is permissible, egregious behavior is
nevertheless sanctioned for reasons of reciprocity, deterrence and morality.
107 For a similar view, see William H. Boothby, Weapons and the Law of Armed Conflict 233 (Oxford Univ. Press. 2009).
The introduction of AWS creates paradoxical situations in which those who plan a
military operation are further removed from the actual combat operation.108 An
additional layer, the code upon which an AWS bases its decision, has been introduced
into the equation. It is far from clear how that challenge will play out in future combat
operations in which it can be presumed that breaches of the international humanitarian
law will take place.109 While military planners insist that a human will remain in the
loop,110 it appears clear that the current mode of operation, i.e. through remotely
operating those vehicles will be replaced by less direct oversight mechanisms. Several
models project that an operator no longer commands individual combat vehicles, but
rather that one operator is responsible for a much larger force. These plans are partially
due to projected budget constraints and their technical feasibility. It is also not
inconceivable that the time period for reacting to a particular cause of action that an
AWS has determined to take cannot be overseen by an operator directly. In such
instances, the question over who is criminally responsible becomes especially acute.
108 There have been documented problems of UAVs mistakenly attacking friendly forces. In the 2006 Israeli invasion of Lebanon, an Israeli drone attacked Israeli ground troops. There have also been two occasions in the US war in Afghanistan where drones have been used to target individuals who, because they stood out among other individuals because of their height, were mistaken to be Osama bin Laden. P. W. Singer, Wired for War: The Robotics Revolution and Conflict in the Twenty-first Century 399 (Penguin Press. 2009). 109 For a contrary view, see Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots (CRC Press. 2009). Tami Davis Biddle, Air Power, in THE LAWS OF WAR: CONSTRAINTS ON WARFARE IN THE WESTERN WORLD 140, 141 (Michael Howard, George J. Andreopoulos, & Mark R. Shulman eds., 1994) (suggesting that the emergence of more accurate bombing capabilities might lead to a convergence of "ethics and efficiency" that could "bolster the prospects for adherence to international norms"). See also Michael N. Schmitt, War, Technology, and The Law of Armed Conflict 137, 163 The Law of War in the 21st Century Weaponry and the Use of Force (Helm, Anthony M. 2007). 110 P. W. Singer, Wired for War: The Robotics Revolution and Conflict in the Twenty-first Century 123-124 (Penguin Press. 2009).
One of the most important problems that the introduction of AWS into the modern battle
space poses is the question of how to establish criminal responsibility when most legal
systems require the showing of intent, with some others adding a requirement of
showing individual guilt. At least one commentator has argued that because of the
absence of justly holding an individual criminally responsible, it would be unethical to
use AWS in warfare.111 This becomes especially important in situations where the
action of a human being would constitute a war crime. By way of example, picture an
AWS firing at a target despite its civilian nature or in a situation in which soldiers have
been wounded to such an extent that they are no longer capable of fighting. The AWS’s
algorithm may have been programmed to act in such situations for a variety of reasons,
including a calculation that the costs of watching over the soldiers may be too high in
comparison to the utility of the AWS in other parts of the battle space or because it was
programmed to instill fear in onlookers.112 In addition, it should be borne in mind that
traditional deterrence factors do not work vis-à-vis an AWS. It can neither be punished
nor does it possess any form of moral agency.113
That being the case, there is a range of actors that could be addressed in determining
whether command responsibility could arise. Setting aside the objection against holding
non-humans responsible, one could try to hold the program or the robot itself
111 Robert Sparrow, Killer Robots, 24 JOURNAL OF APPLIED PHILOSOPHY 62, 66 et seq. (2007). 112 See Robert Sparrow, Killer Robots, 24 JOURNAL OF APPLIED PHILOSOPHY 62, 66 (2007). The author provides additional rationales, such as “the robot […] seeking to revenge the ‘deaths’ of robot comrades recently destroyed in battle”. Given the purported a-emotionality of AWS, this type of rationale would require the very aspect that the proponents contend AWS do not have and make them superior to humans, emotions such as revenge. See only the discussion at fn. ___, below. 113 Colin Allen & Wendell Wallach, Moral Machines: Contradition in Terms or Abdication of Human Responsibility? 55, 62 (Patrick Lin et al. eds., The MIT Press. 2012). But see for a contrary view Samir Chopra & Laurence F. White, A Legal Theory for Autonomous Artificial Agents (University of Michigan Press. 2011).
responsible. In this context it should be pointed out that an AWS may be the cause for
harm,114 yet it is questionable to attribute blame for that action to an entity that does not
possess moral agency. For most legal systems, criminal culpability requires possessing
some form of moral agency, which in the case of AWS does not exist.115 As one author
puts it, attempting to impute moral agency to non-humans “offends not only the notion of
the rule of law, but also the more visceral human desire to find an individual
accountable”.116 In addition, there is the question of how an AWS could even be held
accountable. Even if were AWS that possess intellectual abilities – apart from using
algorithms to act in a discretionary manner – it is questionable whether a machine will
ever be able to suffer from any form of punishment, whatever form that may take.
Practically, one alternative may be to shut off the individual AWS though in all likelihood
the problem would be prevalent in all AWS that are based on the same code. It is
readily apparent that the high costs involved would make the shutting down of an entire
fleet of AWS based on the same code, highly improbable. These considerations show
that, at least for the foreseeable future, the idea of holding the machine itself
responsible may be theoretically possible, but there is little likelihood of this occurring.
114 See e.g. the killing of nine South African soldiers by an automated gun in 2007. Alex Eliseev, Army deaths investigated; Separate probes will determine how nine soldiers were killed in training, The Star, October 15, 2007, at 5; Staff Reporter, War of words after army base horror, Sunday Tribune, October 14, 2007, at 2. 115 But see Sparrow, Can Machines Be People? Reflections on the Turing Triage Test, Lin, Chapter 19, 4-5. 116 Darren M. Stewart, New Technology and the Law of Armed Conflict, in International Law Studies (Blue Book) Series, Vol. 87: International Law and the Changing Character of War, 271, 291 (Raul A. Pedrozo & Daria P. Wollschlaeger eds., 2011). This need for accountability is also stressed in Anthony J. Lazarski, Legal Implications of the Uninhabited Combat Aerial Vehicle, 16:2 AEROSPACE POWER JOURNAL 74, 81 (2002).
Alternatively, it may be possible to hold accountable the scientist or programmer who
developed the software upon which the robot relied. The design of the software, after
all, will ultimately be the foundation upon which the robot makes its determinations.
Leaving aside the situation in which the programmer acted with mens rea, responsibility
for a programmer’s action would have to be negligent. However holding a programmer
responsible for negligence may be a contentious premise given one of the core
characteristics of autonomy: if an AWS is supposed to act according to its code and in
truly autonomous fashion, it must be able to make discretionary decisions. It may not be
possible to predict the behavior of the AWS software in all its manifestations given the
changing nature of the battle space. Responsibility for negligence can be assumed only
for as long as the AWS is not designed to learn independently from past behavior or in
situations in which the designers can be proven to have acted negligently in supervising
the development of the AWS software when it comes to discretionary decision-
making.117
As a third option, one can turn to the military officers who set parameters for a given
engagement. This would be commensurate with the existing attribution mechanism in
which aberrant behavior of a weapon system is nevertheless attributed to the officer in
charge. This could be the case in which the weapon system can be set to a specific
target and if the circumstances were foreseeable at the outset of the mission. This type
of situation however does not reflect the characteristics of when AWS would be
deployed. Their advantage over automated systems is specifically that a target does not
117 Sparrow’s view in this regard may be too narrow, as it is possible to attribute negligent behavior for not monitoring the behavior of an AWS during the learning process. But see Robert Sparrow, Killer Robots, 24 JOURNAL OF APPLIED PHILOSOPHY 62, 70 (2007).
necessarily have to be pre-programmed, but rather that the decision-making process is
independent and leaves room for discretion. In the former situation, i.e. when an AWS
can engage more or less indiscriminately, the individuals who set such parameters will
likely be considered responsible for reasonably foreseeable violations of law the robot
causes. The latter situation however creates a number of complications. It appears
almost certain that soldiers on the frontline who deploy an AWS will not be able to fully
assess the complexities of the software upon which an AWS is built. This would lead to
the paradoxical situation to potentially hold individuals accountable for actions that they
had no control over, a risk that actually grows with a higher degree of autonomy.118
Finally, it may be possible to investigate higher-ranking military or civilian officials for
their action or inaction regarding the testing of AWS. Under Article 36 AP I, states are
under an obligation “to determine whether its employment would, in some or all
circumstances, be prohibited by this Protocol or by any other rule of international law
applicable to the High Contracting Party”.119 This obligation applies during the “study,
development, acquisition or adoption of a new weapon, means or method of warfare”.120
The provision’s aim is to “prevent the use of weapons that would violate international
law” and to “impose restrictions on the use of weapons that would violate international
law in some circumstances, by determining their lawfulness before they are developed,
acquired or otherwise incorporated into a State’s arsenal”.121 There is little state practice
118 Robert Sparrow, Killer Robots, 24 JOURNAL OF APPLIED PHILOSOPHY 62, 70 (2007). 119 Article 36 AP I. 120 Article 36 AP I. 121 International Committee of the Red Cross, A Guide to the Legal Review of New Means and Methods of Warfare: Measures to Implement Article 36 of Additional Protocol I of 1977, 88 INTERNATIONAL REVIEW
OF THE RED CROSS 931, 933 (2006).
as to the precise ramifications of this provision.122 The provision largely leaves it to
states as to how they analyze “whether the employment of a weapon for its normal or
expected use would be prohibited under some or all circumstances”.123 While there has
been rigorous debate about the meaning of this provision,124 it is clear however that
there is no requirement to hold an individual criminally liable for their action or inaction
regarding this provision.
It is important in this context to recall that the situation between AWS on one hand and
remotely operated and automated systems on the other are markedly different. In the
case of the latter, human input remains a crucial element, while AWS operate in an
autonomous manner. Remotely operated and automated systems retain the possibility
to assign individual responsibility. The situation for AWS however is different, as the
“premise underpinning automation is that the operation of the relevant device is capable
of being accurately predicted based on the programming and commands inputted”,
while AWS are built on self-selecting and self-determining systems.125 There is still the
122 Claude Pilloud et al., Commentary on the Additional Protocols of 8 June 1977 to the Geneva Conventions of 12 August 1949 424 (Martinus Nijhoff Publishers. 1987); James D. Fry, Contextualized Legal Reviews for the Methods and Means of Warfare: Cave Combat and International Humanitarian Law, 44 COLUMBIA JOURNAL OF TRANSNATIONAL LAW 453, 473 (2006). 123 Claude Pilloud et al., Commentary on the Additional Protocols of 8 June 1977 to the Geneva Conventions of 12 August 1949 424 (Martinus Nijhoff Publishers. 1987). The 2006 review by the International Committee of the Red Cross provides some guidance in this regard, see International Committee of the Red Cross, A Guide to the Legal Review of New Means and Methods of Warfare: Measures to Implement Article 36 of Additional Protocol I of 1977, 88 INTERNATIONAL REVIEW OF THE RED
CROSS 931, 948 et seq. (2006). 124 See Isabelle Daoust et al., New Wars, New Weapons? The Obligation of States to Assess the Legality of Means and Methods of Warfare, 84 Int'l Rev. Red Cross 345 (2002); Justin McClelland, The Review of Weapons in Accordance with Article 36 of Additional Protocol I, 850 INT'L REV. RED CROSS 397 (2003). 125 Darren M. Stewart, New Technology and the Law of Armed Conflict, in International Law Studies (Blue Book) Series, Vol. 87: International Law and the Changing Character of War, 271, 290 (Raul A. Pedrozo & Daria P. Wollschlaeger eds., 2011).
possibility of deviant behavior in those situations, which may be attributable to aberrant
behavior by the command programmers.126
IV. Ethical Challenges to Autonomous Weapon Systems
One essential question concerns the issue of the dehumanization of armed conflict
along the ethical dimension. From its inception, IHL was underpinned by an attempt to
make armed conflict more humane. And while these chivalristic notions may never have
been realized, there are moral and ethical implications when considering the use of
UMS that should be resolved before deploying this new type of weapon.
The bombing campaigns in the Balkans in the 1990s or Iraq in the first decade of the
21st century may serve as examples. While the latter was followed up by the use of
ground forces, both conflicts are characterized by a considerable reluctance to deploy
soldiers in ground operations. This was especially true with respect to Kosovo. A
bombing campaign ensued over a few months that led to the end of the conflict. The
option of conducting operations “surgically”, purportedly with a high degree of precision
was much more attractive for decision-makers than the messy and much more
dangerous nature of committing ground troops into action. As a consequence, the
operation was entirely carried out by airplanes, removing humans to a considerable
extent not only from harm, but also from the battlefield as such. UMS remove the
126 Darren M. Stewart, New Technology and the Law of Armed Conflict, in International Law Studies (Blue Book) Series, Vol. 87: International Law and the Changing Character of War, 290 (Raul A. Pedrozo & Daria P. Wollschlaeger eds., 2011).
human input further from the battlefield. Under this scenario it is no longer necessary to
send pilots on bombing campaigns, as was the case in Kosovo. It will also no longer be
necessary to employ a “pilot” for a particular drone. Rather, a UMS will operate
according to the code that functions as its basis. The responsibility therefore shifts
elsewhere, as outlined above, namely to the software engineer or those who advise a
software engineer who – today – is not considered to be a combatant, to the testing
units, to those overseeing a particular tactical situation or the military commanders /
political leadership.
One of the most vocal proponents of this move suggests that the outcome of such a
development is beneficial from an ethical perspective: “This effort has an over-arching
goal of producing an ‘artificial conscience,’ to yield a new class of robots termed
humane-oids—robots that can perform more ethically in the battlefield than humans are
capable of doing.”127
1. Dehumanization through Removal of Individual from the Battlefield?
One potential problem posed by the dehumanization of killing consists of lack of
distance and separation from actions even through the use of unmanned drones.128
Engaging an enemy within the framework of international humanitarian law through the
drone isn’t necessarily a problem, but it can easily become one. An example often
127 RONALD C. ARKIN, Governing Lethal Behavior in Autonomous Robots 16 (CRC Press. 2009). 128 P. W. SINGER, Wired for War: The Robotics Revolution and Conflict in the Twenty-first Century 395-396 (Penguin Press. 2009).
referred to in this context is the firebombing that took place during WW II. The physical
distance and technical aspect of their task is said to remove pilots to an extent that
enabled them to carry out their missions – which, with closer proximity to the battlefield,
they would not have carried out. As distance increases, it becomes psychologically
easier to commit an act which an individual would not otherwise be willing to do.129 This
is due to the fact that humans have a reluctance to kill one another and the physical as
well as psychological distance that long-range weapon systems have brought about,
combined with the technological nature of today’s combat operations where the direct
impact of an individual’s action is increasingly less visible, has circumvented this innate
reluctance.130 Other advantages include the lower number of human casualties in war
by placing robots in harm’s way instead of human beings,131 reducing the destruction
associated with armed conflict by potentially more precise machines as well as the
reduction in the accidental targeting of civilians or other non-combatants if the machine
was powerful enough and sophisticated enough to make accurate determinations.
Human error is often the cause of accidentally engaging friendly forces or civilians, so
129 Dave Grossman, On Killing: The Psychological Cost of Learning to Kill in War and Society 97-137 (Little, Brown 1st ed. 1995). 130 Dave Grossman, On Killing: The Psychological Cost of Learning to Kill in War and Society 107-110 (Little, Brown 1st ed. 1995); Gert-Jan Lokhorst & Jeroen van den Hoven, Responsibility for Military Robots, in Robot Ethics: The Ethical and Social Implications of Robotics, 145, 147-148 (Patrick Lin et al. eds., 2012); Noel Sharkey, Killing Made Easy: From Joysticks to Politics, in Robot Ethics: The Ethical and Social Implications of Robotics, 111, 111-112 (Patrick Lin et al. eds., 2012). See also the report by a drone operator after a Predator chase of a targeted enemy. Once the target was acquired, the author claims that the enemy “was so far away and only a high-tech image on a computer screen. The moral aspects of it – that I was about to assassinate a fellow human being from ambush – didn’t factor in. Not at the moment. Not yet.” Matt J. Martin & Charles W. Sasser, Predator: The Remote-Control Air War over Iraq and Afghanistan. A Pilot's Story 43-44 (Zenith Press. 2010). Furthermore, the authors contend that there is a gap between the reality of the war and the war that the operator was participating in from across the world: “The ability to kill people from such great distances, playing God, widened the gap between the reality of war and our perception of it. It was almost like watching an NFL game on TV with its tiny figures on the screen compared to being down there on the field in the mud and the blood in the rain getting your socks knocked off.” Ibid., p. 47 131 CQ Researcher, Drone Warfare – Are Strikes by Unmanned Aircraft Unethical?, 20:18 CQ Researcher, 653, 662 (statement by Barrett, claiming that the technology must be used as it is more humane, fn. 35).
removing the human (and human emotions) from the equation could be potentially
beneficial.
However, problems have been reported in the use of the current generation of UAVs
because individuals no longer fight on the battlefield and are stationed close to home,
making it psychologically difficult to separate between these spaces. It is no longer
possible to leave an experience behind through the distance that is being created by
traveling home. AWS change this paradigm even further. While the physical distance
may not be greater, the psychological distance no longer plays a significant role. In
contradistinction to automated weapons, such as missiles and rockets which may alter
their course, but have only one purpose, the role of AWS is different. Human operators
may oversee their action, but the very essence of an AWS is its ability to act on its own
and make decisions on its own.
Not all human participation however is eliminated. In an indirect manner, individuals
stay in the loop. The decision-makers over the parameters of the software essentially –
though not in real time – determines the operation of an AWS. Instead of an individual
making real-time decisions, it is the software that does so. Ultimately, the persons
responsible for maintaining and employing the AWS (or fighting alongside them) may be
targeted either by human beings or machines operated by the opposing side, potentially
broadening the scope for who would be considered to be a combatant. It is possible this
concern could keep the desire to protect human life sufficiently relevant to avoid
reducing historic gains in international humanitarian law. However it is also possible that
humanitarian concerns could become significantly less important as there would be less
tangible loss of life, especially in the developed states that can afford this technology.
2. Ethical Robots?
Various arguments are put forth to make the case for an increased use of AWS as their
use will lead to increase of the ethical behavior on the battlefield.132 Unlike humans, an
AWS does not have a need to protect itself. Since AWS are emotionless, nothing can
“cloud their judgment”.133 Walzer made a similar observation, namely that “fear and
hysteria are always latent in combat, often real, and they press us toward fearful
measures […].”134 Moreover, wider use of AWS would take the human psychological
element of “scenario fulfillment” out of the equation. This is believed to have contributed
to the downing of flight 655 by the USS Vincennes in 1988.135 “Scenario fulfillment”
leads to distortion or neglect of contradictory information in stressful situations, where
humans use new incoming information in ways that only fit their pre-existing belief
132 Elizabeth Quintana, The Ethics and Legal Implications of Military Unmanned Vehicles, Royal United Services Institute for Defence and Security Studies, at http://www.rusi.org/downloads/assets/RUSI_ethics.pdf, 13; Jack M. Beard, Law and War in the Virtual Era, 103 AMERICAN JOURNAL OF INTERNATIONAL LAW 409, 428 et seq. (2009); Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 29 et seq. (CRC Press. 2009). For similar arguments, see Gary E. Marchant et al., International Governance of Autonomous Military Robots, UNPUBLISHED MANUSCRIPT 1, 6 (2010). 133 Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 29 (CRC Press. 2009). 134 Michael Walzer, Just and Unjust Wars: A Moral Argument with Historical Illustrations 251 (Basic Books. 1977). 135 Scott Sagan, Rules of Engagement, in Avoiding War: Problems of Crisis Management, 443, 460 (Alexander L. George & Yaacov Bar-Siman-Tov eds., 1991). See also Elizabeth Quintana, The Ethics and Legal Implications of Military Unmanned Vehicles, Royal United Services Institute for Defence and Security Studies, at http://www.rusi.org/downloads/assets/RUSI_ethics.pdf, 13.
patterns, a form of premature cognitive closure.136 The official Department of Defense
report subsequent to the incident remarks finds that “[s]tress, task fixation, and
unconscious distortion of data may have played a major role in this incident” and that
the personnel “became convinced [that] track 4131 was an Iranian F-14” rather than a
civilian aircraft.137 Based on this information, the operator “appears to have distorted
data flow in an unconscious attempt to make available evidence fit a preconceived
scenario”.138 The design of AWS is such that they are not vulnerable to such patterns of
behavior. The proponents argue that since they are not prone to disregarding new
information because of the distortions that “scenario fulfillment” leads to, they are better
capable of handling situations in which new information may be contradictory to
previous information.139
Due to the increase in sensory abilities, AWS will have the ability to observe a large
number of relevant aspects. Today’s UAVs e.g. have a much longer “loitering time” over
a particular area (up to 40 hours). Even if they cannot observe everything, their abilities
are certainly greater than those of their human counterparts with respect to data
136 The action that led to the attack on the aircraft was furthermore attributed to a) confusion because of the proximity of military aircraft despite the civilian aircraft having engaged its civilian transponders and b) a permissive interpretation of the rules of engagement, possibly influenced by an attack on a US ship one year earlier, also in the Persian Gulf. Scott Sagan, Rules of Engagement, in Avoiding War: Problems of Crisis Management, 443, 460-461 (Alexander L. George & Yaacov Bar-Siman-Tov eds., 1991). 137 Formal Investigation into the Circumstances Surrounding the Downing of Iran Air Flight 655 on July 3, 1988. 45 (1988 ). 138 Formal Investigation into the Circumstances Surrounding the Downing of Iran Air Flight 655 on July 3, 1988. 45 (1988 ). 139 Ronald C. Arkin, Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture 6-7 (2011).
absorption and data analysis140. Data can arise from multiple remote sensors and
intelligence (including human) sources, as part of the Army’s network-centric warfare
concept and the concurrent development of the Global Information Grid.141 This
development can be expected to continue, summed up by one commentator: “[m]ilitary
systems (including weapons) now on the horizon will be too fast, too small, too
numerous and will create environments too complex for humans to direct.”142
This leads to a question, the answer to which may be crucial in the development of
AWS. Is it possible to devise an AWS that would be responsive to the requirements of
international humanitarian law and the underlying ethical requirements? The question
over whether it is possible to create code that would allow this is hotly debated. Arkin
suggests the creation of an “ethical governor”, a piece of program that would allow the
determination of whether a particular action by an AWS would be unethical, and if so,
would alert a human operator or constrain the action that would otherwise have been
carried out.143 The “ethical governor” would “not involve emotion directly […] as that has
been shown to impede the ethical judgment of humans in wartime”.144 Rather, he
proposes the introduction of an element akin to guilt which – over time, but not in
specific situations – can be re-programmed after a proper assessment of the system’s
140 Thomas Shanker & Matt Richtel, In New Military, Data Overload Can Be Deadly, New York Times, 17 January, at A1. 141 DARPA (Defense Advanced Research Projects Agency), Broad Agency Announcement 07-52, Scalable Network Monitoring, Strategic Technology Office, August 2007, available at https://www.fbo.gov/index?s=opportunity&mode=form&tab=core&id=b524ff8d8f7390061d4c5d5444c9e620&tab=documents&tabmode=list (last visited Sept. 22, 2010). 142 Adams, supra note 8, at 58. 143 Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 123 (CRC Press. 2009). 144 RONALD C. ARKIN, Governing Lethal Behavior in Autonomous Robots 118 (CRC Press. 2009).
behavior has been carried out.145 At the initial stage however, the role of emotions (and
with it, the attribution of intentions to an action – is the running of children towards a
group of soldiers which what looks like a weapon truly a threat or are they chasing a
soccer ball?) is diminished and the “ethical governor” will evaluate the options available
to it in light of the options available to it, with the guilt censor rejecting certain action as it
sees fit.146
There are a number of principled objections against this view, one of which is
recognized by Arkin himself, though there are others which I will turn to later. Arkin
himself realizes that IHL requires a certain level of compassion (in other words:
proportionality in that no more harm than necessary is inflicted), which is difficult to build
into the AWS.147 However, by abiding by the other rules of IHL he believes that a thick
enough web of rules is created and being followed that creates this compassion (or
proportionality). However, as was shown above, the widespread use of proportionality
throughout IHL begs the question why they exist to begin with if the other rules were
indeed capable of creating the web that Arkin presumes to exist.
Furthermore, while one can argue whether human problem solving in socially complex
situation should be the model of AWS design, human emotion has been shown to be an
important element in the process of human decision-making through experiments
145 Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 125 et seq. (CRC Press. 2009). 146 Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 140 (CRC Press. 2009). 147 RONALD C. ARKIN, Governing Lethal Behavior in Autonomous Robots 143 (CRC Press. 2009).
involving functions in certain parts of the brain.148 It is precisely in such situations where
the ability to empathize with a situation becomes important as they may play a
constructive or even decisive role in determining which of the options available at any
moment in time an individual will actually take. One instructive example involves
experiments with individuals that lack certain brain functions associated with emotions.
The lack of these functions, associated with damage in the prefrontal cortex would in
those situations most likely lead to disastrous results. In the situation described above,
a non-empathizing soldier may open fire, while a fully functional individual may have the
cognitive ability and empathetic reaction to not do so. The defect with Arkin’s argument
is that the AWS’s decision-making process would lead it to a particular course of action,
which may involve the use of force. If the “ethical governor” was designed to block this
behavior in this situation or situations of a similar nature, that veto will prevent the action
from occurring. The problem then becomes much clearer: the underlying assumption is
that emotions do not play a role in the filtering process of what options are being
considered in the first place.149
148 Marcello Guarini & Paul Bello, Robotic Warfare: Some Challenges in Moving from Noncivilian to Civilian Theaters, in Robot Ethics: The Ethical and Social Implications of Robotics, 129, 134 et seq. (Patrick Lin et al. eds., 2012). See also Susan A. Bandes, Is it Immoral to Punish the Heedless and Clueless? A Comment on Alexander, Ferzan, and Morse: Crime and Culpability, 29 LAW AND PHILOSOPHY 443(2010). Bandes generally discusses the role of the subconscious and unconscious in human beings. Bandes argues that there is no sharp divide between the two and that our ability to exercise control over conscious decisions is largely a myth and most decision-making comes from the subconscious (440). “Our conscious reasons are often post-hoc explanations rather than reflections of an actual process.” (445). 149 Marcello Guarini & Paul Bello, Robotic Warfare: Some Challenges in Moving from Noncivilian to Civilian Theaters, in Robot Ethics: The Ethical and Social Implications of Robotics, 129, 136 (Patrick Lin et al. eds., 2012).
Some authors also claim that when working in combination with humans UMS may
have the ability to reduce the number of unethical behavior by humans through
reporting mechanisms. They could be programmed to have the capability of
independently and objectively monitoring ethical behavior in the battlefield by all parties
and reporting infractions that might be observed.150 Even their presence alone is
expected to lead to a reduction in human ethical infractions.151 The question that must
be answered of course is whether this could or should be the benchmark for ethical
behavior? Arkin argues that AWS would be able to do the same tasks in a more ethical
manner. Arkin’s argument relies partially on a recent – and somewhat disturbing –
report published from the Surgeon General’s Office in 2006 supports the argument that
unmanned combat systems may undoubtedly play a vital role in enforcing many of the
ethical challenges that occur during combat.152 According to the report, appropriate
ethical behavior among Soldiers and Marines deployed in Operation Iraqi Freedom and
Operation Enduring Freedom appear to be questionable, despite a large number of
soldiers and marines reporting that they received adequate training.153 Some of the
findings include:154
150 Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 30 (CRC Press. 2009). 151 Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 30 (CRC Press. 2009). 152 Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 31 (CRC Press. 2009).See also P. W. SINGER, Wired for War: The Robotics Revolution and Conflict in the Twenty-first Century 401 (Penguin Press. 2009). 153 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. 37 (2006). Disturbingly, a large percentage of Soldiers and Marines reported that they non-commissioned officers and Officers in their own unit did not make it clear that mistreatment was impermissible, with 33% of Marines and 29% of Soldiers responding in such a manner, respectively. 154 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. (2006).
1. Approximately 10 percent of Soldiers and Marines report mistreating
noncombatants such as, purposely damaging or destroying civilian property when not
necessary or hit/kicked a noncombatant when not necessary.155
2. Only 47 percent of Soldiers and 38 percent of Marines agreed that
noncombatants should be treated with dignity and respect.156
3. Over one-third of Soldiers and Marines reported torture should be allowed in
order to save the life of a fellow Soldier or Marine or to obtain important information
pertaining to the enemy.157
4. 45 percent of Soldiers and 60 percent of Marines did not agree that they would
report a fellow Soldier or Marine if he had injured or killed an innocent noncombatant.158
5. Only 43 percent of Soldiers and 30 percent of Marines agreed that they would
report a unit member for unnecessarily damaging or destroying private property.159
6. Less than one-half of Soldiers and Marines would report a team member for
unethical behavior.160
7. 28 percent of Soldiers and 31 percent of Marines reported ethical dilemmas in
which they did not know how to respond.161
8. Immediate loss of a fellow Soldier or Marine during extreme violence was
associated with an increase in ethical violations.162
155 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. 4 (2006). 156 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. 35 (2006). 157 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. 35 (2006). 158 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. 37 (2006). 159 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. 37 (2006). 160 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. 36-37 (2006). 161 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. 37 (2006). 162 Mental Health Advisory Team (MHAT) IV Operation Iraqi Freedom 05-07, Final Report. 39-41 (2006).
Based on this, Arkin argues that if it were possible to program an AWS that behaves
better than those numbers it would be unethical not to employ an AWS as opposed to
the soldiers that display such behavior.163 One doesn’t need to be a philosopher to
understand the consequentialist or utilitarian argument that is being made here. And
one doesn’t have to be a deontologist in order to at least raise some issues with this
approach. There is a difference between the admission by individuals to having
committed what could amount to war crimes – ex post facto – and an a priori built-in
failure rate which may lead to the commission of war crimes. The former could
potentially be remedied through better education prior to sending individuals into
combat. Furthermore, states are under an obligation to prevent or in the alternative
prosecute war crimes, making it at the very least possible to retroactively punish
offenders. Such criminal sanctions may serve as a deterrent for some.
In addition, the problem with this approach is that it only moves the question further
backward in the design process, without actually solving it. In order to determine
whether an ethically problematic situation even takes place, the AWS would have to be
cognizant of the ethical implications of its actions. It would have to be able to compare
the ethical implications of various courses of actions and decide which of these courses
of action is more appropriate. The dilemma thus consists in the “ethical governor” not
being able to recognize ethically problematic situation in the first place, or it would have
to be able to think ethically on its own – and act accordingly.164 One problem that may
163 Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots 33 et seq. (CRC Press. 2009). 164 Rob Sparrow, Can Machines Be People? Reflections on the Turing Triage Test, 301, 304 et seq. (Patrick Lin et al. eds., 2012).
arise is that such a calculation may be impossible to be carried out in the time period
that is available for the AWS, or because of the short time frame, an AWS may use only
insufficient information to take action.165
This would then require a whole new set of answers to questions such as whether it is
possible to punish an AWS, as punishment requires moral agency.166 It is here where
some researchers claim that the – oftentimes overlaid by an instinctual decision-making
system – deliberative system is what makes humans moral agents. The ability to
cognitively assess a situation allows for “structur[ing] alternative possible futures as
mental representations, and then to choose our actions based on which of the
representations that we wish to become our experienced reality”.167 Based on this,
some argue that the development of such a deliberative system is a requirement for
moral personhood and that the “key to moral responsibility and personhood is the
possession of moral agency”.168
165 Keith Abney, Robotics, Ethical Theory, and Metaethics: A Guide for the Perplexed 35, 43 (Patrick Lin et al. eds., The MIT Press. 2012). 166 Colin Allen & Wendell Wallach, Moral Machines: Contradition in Terms or Abdication of Human Responsibility? 55, 62 (Patrick Lin et al. eds., The MIT Press. 2012). 167 Keith Abney, Robotics, Ethical Theory, and Metaethics: A Guide for the Perplexed 35, 47 (Patrick Lin et al. eds., The MIT Press. 2012). 168 Keith Abney, Robotics, Ethical Theory, and Metaethics: A Guide for the Perplexed 35, 47 (Patrick Lin et al. eds., The MIT Press. 2012). See also the contribution of Allen and Wallach in the same volume.
V. Political Challenges to Autonomous Weapon Systems
It could be argued that the decision to employ more sophisticated technology carries
with it a higher burden for political decision-makers.169 As Stewart puts it, the “perverse
effect for States and the senior civilian and military command echelon who promote the
development and implementation of new technology as a means of ‘casualty free’
warfare is that they may well find themselves with nobody to stand between the actions
of such autonomous systems and themselves when things go wrong”.170 One of the
very real dangers in relying on autonomous systems is a perception of low or no risk
that may be created for those who no longer have to fight in conflicts. In the process, it
has the potential of lowering the costs for political decision-makers to engage in armed
conflict given that the political calculus would not have to take into account the number
of fallen soldiers.
Historically speaking, IHL has been an anthropocentric endeavor. Taking individuals out
of the equation (or no longer retaining them “in the loop” or “on the loop”) by relying
upon computer software could increase the likelihood that a state will resort to force as
its citizens are no longer directly being placed at risk. This development has been going
on for some time now in armed conflicts through the increased reliance of developed
states on bombing campaigns in the late 20th century, e.g. the prolonged NATO
bombing campaigns in the Balkans in the 1990s and US/UK bombing campaigns 169 Darren M. Stewart, New Technology and the Law of Armed Conflict, in International Law Studies (Blue Book) Series, Vol. 87: International Law and the Changing Character of War, 271, 291 (Raul A. Pedrozo & Daria P. Wollschlaeger eds., 2011). 170 Darren M. Stewart, New Technology and the Law of Armed Conflict, in International Law Studies (Blue Book) Series, Vol. 87: International Law and the Changing Character of War, 271, 293 (Raul A. Pedrozo & Daria P. Wollschlaeger eds., 2011).
against Iraq. The – relatively speaking – lower risk to its armed forces made the
decision over whether to engage in a conflict politically more palatable to the wider
public and less risky for the politicians involved. In the context of UMS, this pre-conflict
proportionality calculation becomes even more important to take into account.171 A
fundamental impediment to war is the loss of human life, especially the lives of fellow
citizens; casualties are a significant reason why wars are not more common. Sending
an army of machines to war – rather than friends and relatives – may not exact the
same physical and emotional toll on a population.172 Assuming the existence of a just
cause, one could celebrate this new calculus, which more readily permits legitimate self-
defense. However, this reduced cost may, in turn, reduce the rigor with which non-
violent alternatives are pursued and thus encourage unnecessary – and therefore unjust
– wars.
VI. Conclusion
While the predictions made at the end of WW II about the future of air combat was
premature, the advances in computer technology has enabled the development of
weapon systems are designed to act in an autonomous manner. While uncertainties
about the capabilities and the time frame for the deployment of such weapon systems
exist, the inclusion of such weapon systems in the main report for the 31st International
Conference of the Red Cross and the Red Crescent is indicative of the importance of
171 Peter Asaro, How Just Could a Robot War Be?, in ADAM BRIGGLE, KATINKA WAELBERS, AND PHILIP BREY (eds.), CURRENT ISSUES IN COMPUTING AND PHILOSOPHY 1, 7-9 (2008). 172 Robert Sparrow, Predators or Plowshares? Arms Control of Robotic Weapons, IEEE TECH. SOC’Y
MAGAZINE, Spring 2009, at 25, 26 (hereinafter “Sparrow, Predators or Plowshares?”).
this issue.173 It therefore appears short-sighted to dismiss concerns about the legal,
ethical and political implications of the introduction of AWS into the modern battle
space.
The technology to implement such devices is currently either available, or being
developed and tested. In the near future, advanced militaries will have the capability to
employ AWS. The paths they choose to follow could undermine the humanization of
international humanitarian law as described by Theodor Meron in 2000 unless care is
taken to ensure, with a reasonable degree of certainty, compliance with international
legal rules applicable in armed conflict. As has become evident, this will – from a
technological perspective – be either difficult or impossible to achieve. The current
design architecture does not build these concerns in at the front end of such projects to
a sufficient degree. This leads to the question that should be asked and one to be taken
more seriously. Rather than asking how and to what extent AWS can be used and to
maximize their perceived utility, it may be necessary to first answer the question if such
systems should be allowed to harm – and kill – humans in the first place.
173 International Committee of the Red Cross, International Humanitarian Law and the Challenges of Contemporary Armed Conflicts 38‐40 (2011).