Ethical Control of Unmanned Systems

Ethical control of unmanned systems can be accomplished through structured mission definitions that are consistently readable, validatable and understandable by humans and robots. Responsible humans must remain in charge of lethal/lifesaving force, and then robots become more effective.

Synopsis | Briefings | Design Overview | Documentation | Download | Missions | Ontologies | OwlDoc | Queries | Resources | Savage Developers Guide | Visualization | Contact

🔖 Project Synopsis

Project Motivation: ethically constrained control of unmanned systems and robot missions by human supervisors and warfighters.

Precept: well-structured mission orders can be syntactically and semantically validated to give human commanders confidence that offboard systems will do what they are told to do, and further will not do what they are forbidden to do.

Project Goal: apply Semantic Web ontology to scenario goals and constraints for logical validation that human-approved mission orders for robots are semantically coherent, precise, unambiguous, and without internal contradictions.

Long-term Objective: demonstrate that no technological limitations exist that prevent applying the same kind of ethical constraints on robots and unmanned vehicles that already apply to human beings.

Abstract. Ethical human supervision of unmanned maritime systems is foundational in future undersea warfare. Forward deployed unmanned systems in the human-machine team must comply with their Commander's intent throughout the duration of their existence in future Undersea Warfare - in which harsh physical domains, long distance from the Commander, and prolonged time on-station stress the capabilities of the unmanned systems and limit their operator's control. Therefore, in order to apply ethical control of unmanned systems in future undersea warfare, we develop an ontology for unmanned systems mission execution and design. We study four canonical missions for unmanned maritime systems with progressive sophistication in order to test and evaluate Ethical Control design on the autonomy of the unmanned systems. The goals of our research are to ensure unmanned maritime systems comply with existing policy guidance of the U.S. Department of Defense and relevant international organizations, and provide inputs to emerging policy guidance. Our vision is for Commanders to be confident in authorizing life-saving or lethal force from unmanned systems that operate under ethical control in collaboration with human forces. Ultimately, "Ethical Control leads to better warfighting." Simulation playback of multiple key scenarios demonstrates these principles in action.

🔖 Briefings

Comprehensive presentation: Ethical Control of Unmanned Systems overview describes all aspects of this project, along with related work and relevant resources.

We are now briefing research progress publicly, with all work available under an open-source license.

Videos	Dates, Events, Slidesets, Descriptions
	🔖 Python Mission Evaluation, Exhaustive Test Analysis and Connecting to AI-based Opponent Systems (slideset) Implementation Demonstration and Discussion Video, Jon Cefalu, neurobinder.com 16 December 2020, duration 59:41. (0:00) Attendee introductions, (4:30) Ethical Control of Unmanned Systems context, (6:00) Jon shows new capabilities include a Python implementation for exercising decision logic in Autonomous Vehicle Command Language (AVCL) mission. Additional features include the ability to exhaustively test mission variations, checking functional mission correctness and detecting decision loops. (20:00) Mission testing considerations appear to provide an initial basis for evaluating human-machine mission logic and code coverage, building stepping stones towards model-based testing as well as verification and validation, (27:30) Jon also demonstrated cross-connecting the Pirate mission to AI Dungeon (Wikipedia), a text-based adventure game engine based on the Generative Pre-trained Transformer 3 (GPT-3) language-prediction model by OpenAI that uses deep learning to produce human-like text responses. Despite only superficial configuration, scenario exploration became possible using interesting (and occasionally outlandish) text responses from the tool. (48:30) Group discussion on future design considerations for connecting human-machine teaming systems with realistic wargaming systems for sensitivity analysis, massively repeatable testing, and analysis of alternatives. Further reading: David Walton, Three Laws Lethal, Pyr publishing, Hoboken New Jersey, 2019 (reviews).
	🔖 Ethical Control of Unmanned Systems using Formal Mission Ontologies for Undersea Warfare (slideset) 4 September 2020, duration 1:06:20. National Defense Industry Association (NDIA) Undersea Warfare (USW) Virtual Conference with follow-on discussion. Event dates 22-23 September 2020. Where must we go next: Massive testing of unmanned hardware + software ability to follow both orders and constraints in physically realistic virtual environments. Certify capabilities via field experimentation (FX), confirmed by USW range exercises and regular force operations. Human warfighters and commanders (not just engineers) review and approve unmanned systems as… qualified. New normal will be human + machine teaming. Mainstream capabilities in all aspects of acquisition and deployment. Thanks to Raytheon Technologies for Cooperative Research and Development Agreement (CRADA) support. Research results and insights are available in jointly approved Technical Report NPS-USW-20-001, Ethical Control of Unmanned Systems: lifesaving/lethal scenarios for naval operations, August 2020.
	🔖 NPS CRUSER Overview, Ethical Control of Unmanned Systems (slideset) 4 May 2020, duration 21:10. NPS Consortium for Robotics and Unmanned Systems Education and Research (CRUSER) monthly meeting, 91 people. Mission orders must be understandable by both humans and machines. Logical testing and command trust are possible using Semantic Web standards. Thanks to CRUSER and Raytheon for strong support during this many-year project. NPS-Raytheon CRADA partnership offers new possibilities for influence. Research insight: our university needs to establish a cross-disciplinary, cross-service NPS Center for Ethical Warfare. Your NPS efforts matter. Talk to us about a potential thesis! This work holds multiple opportunities for applied student research in any major.
	🔖 Data-Centric Security for Ethical Control of Unmanned Systems (slideset) 16 April 2020, duration 22:08. Briefing to Raytheon and NPS colleagues. Data-centric security enables strong command authority and trust between deployed commands and remote unmanned systems. Trusted communications must occur regardless of intervening challenges to network connections by the ocean-atmosphere environment or hostile countermeasures. Ethical control of unmanned systems requires reliable message exchange across long distances and durations of time.
	🔖 Mission Design and Semantic Web Exemplars for Human Supervision of Lethal/Lifesaving Autonomy (slideset) 6 April 2020, duration 1:01:31. Project briefings reprised to Raytheon colleagues, providing project overview of sponsored work performed as part of an NPS-Raytheon Cooperative Research and Development Agreement (CRADA). 25 March 2020. Online brief to Naval Unmanned Vehicle and Autonomous Systems (UVAS) Working Group, providing project overview with emphasis on relevance to much naval research and future operations. 9‑11 March 2020. Conference panel. US Semantic Technologies Symposium (US2TS) panel session: Hybrid AI for Context Understanding, North Carolina State University, Raleigh, NC. Also presented related work X3D Ontology for Semantic Web as a "Lightning Talk" there. (slideset)

• Event cancelled due to COVID lockdown. 29 March - 1 April 2020, NDIA 2020 Undersea Warfare (USW) Spring Conference, Admiral Kidd Conference Center, San Diego CA.

🔖 Design Overview

The essence of this work is defining missions that are clear, unambiguous, validatable as syntactically correct, and verifiable as logically correct.

Key insights:

1. Humans in military units are able to deal with moral challenges without ethical quandaries, using formally qualified experience and by following mission orders that comply with Rules of Engagement (ROE) and Laws of Armed Conflict (LOAC).
2. Ethical behaviors don't define the mission plan. Instead, ethical constraints inform the mission plan.
3. Naval forces can only command mission orders that are understandable by (legally culpable) humans, then reliably and safely executed by robots.

Design and development of these capabilities has been ongoing for many years. Key language components include:

• Autonomous Vehicle Command Language is an XML-based language for defining unmanned-system missions.
• RDF/OWL are standardized Semantic Web languages to express properties and logical relationships.
• Terse Triple Language (Turtle) is a concise syntax for RDF/OWL.
• SPARQL is a standardized query language that enables us to write semantic tests.

Life-saving missions and missions with lethal force are complementary. Human-robot activity can result in lethal or life-saving outcomes.

In this work, ethical theory meets professional practice. Each step must work for human commanders and unmanned systems alike.

Numerous assets are provided here to explain how this approach works and continues to mature.

🔖 Documentation

• Synopsis report: Donald P. Brutzman, Curtis L. Blais, and Hsin-Fu Wu (Raytheon), Ethical Control of Unmanned Systems: Lifesaving/Lethal Scenarios for Naval Operations, Technical Report NPS-USW-2020-001, Naval Postgraduate School (NPS), Monterey California, August 2020. Now available in NPS Dudley Knox Library Calhoun Digital Archive.
• Comprehensive presentation: Ethical Control of Unmanned Systems overview describes all aspects of this project, along with related work and relevant resources.
• Ethical Control flyer and project quad chart (.pdf).
• Network Optional Warfare (NOW) for deliberate, stealthy, minimalist tactical communications.
• Network Optional Warfare (NOW): Ethical Control of Unmanned Systems overview

Presentations, papers, figures, flyers and reports are all available in the documentation section of the project archive. Also available: mission diagrams (.pdf).

Data model documentation:

• AVCL 3.0 Schema documentation (and AVCL.3.0.changes.txt) for Autonomous Vehicle Command Language (AVCL)
• Protege Owldoc for Mission Execution Ontology (MEO)

🔖 Download

EthicalControlArchive.zip (120MB) provides full website for download and local testing.

Version control for all project assets is publicly available at https://gitlab.nps.edu/Savage/EthicalControl

🔖 Missions

Unmanned systems working in tandem with human forces, authorized by commander for life-saving or lethal force, can handle progressive challenges in distance and time.

The following missions carefully define and test such capabilities.

• Sailor Overboard. Perform SAILOR OVERBOARD operations, carried out in concert with shipboard emergency procedures.
• Lifeboat Tracking. Provide remote presence for locating, tracking, communications and beaconing an adrift lifeboat.
• Pirate Boats Attack. Overtake pirate small-boat gang attempting to capture threatened merchant ship, provide warning and counterattack if escalation of hostilities is warranted.
• Hospital Ship EM Decoy: Sense-Decide-Act Loop. Immediate reaction using Sense-Decide-Act cycle results in blue-on-blue robot swarm attack and war crime.
• Hospital Ship EM Decoy: OODA Loop. Orient-Observe-Decide-Act (OODA) loop tactics and Ethical Control constraints prevent automatic erroneous counterattack against false flag placed on friendly ship, and thus improves defense.

Exemplar Missions	Sailor Overboard	Lifeboat Tracking	Pirate Boats Attack	Hospital Ship EM Decoy Sense-Decide-Act Loop	Hospital Ship EM Decoy OODA Loop	Descriptions
🔖 Diagrams						AVCL mission diagrams show ternary logic for decision flow. Each goal can only result in success, failure or exception. Visual representations are quite useful for checking mission logic. This carefully designed mission structure is able to express all possible orders while retaining traceable logic and accountability with rules of engagement (ROE).
🔖 AVCL XML Missions	SailorOverboard.xml	LifeboatTracking.xml	PiratesSeizing MerchantDefense.xml	HospitalShipEmDecoy2. Defender. SenseDecideAct.xml	HospitalShipEmDecoy3. Defender. EthicalControlOODA.xml	AVCL XML documents define machine-readable and human-readable missions in a manner that can be syntactically validated as well-formed and well-structured, using strictly controlled terms of reference.
🔖 Turtle Triples	SailorOverboard Converted.ttl	LifeboatTracking Converted.ttl	PiratesSeizing MerchantDefense Converted.ttl	HospitalShipEmDecoy2. Defender. SenseDecideAct Converted.ttl	HospitalShipEmDecoy3. Defender. EthicalControlOODA Converted.ttl	Turtle triples are created by a AvclToTurtle.xslt conversion stylesheet that essentially "explodes" a mission into each component relationship. This form allows semantic queries and reasoning to occur.
🔖 Lisp Test Programs	SailorOverboard Converted.cl	LifeboatTracking Converted.cl	PiratesSeizing MerchantDefense Converted.cl	HospitalShipEmDecoy2. Defender. SenseDecideAct Converted.cl	HospitalShipEmDecoy3. Defender. EthicalControlOODA Converted.cl	Lisp is a functional programming language for AI research. The AvclToLisp.xslt conversion stylesheet reads AVCL XML to produce Lisp source code, encouraging AVCL support in multiple robots.
🔖 Prolog Test Programs	SailorOverboard Converted.pl	LifeboatTracking Converted.pl	PiratesSeizing MerchantDefense Converted.pl	HospitalShipEmDecoy2. Defender. SenseDecideAct Converted.pl	HospitalShipEmDecoy3. Defender. EthicalControlOODA Converted.pl	Prolog is a logic programming language associated with AI research and computational linguistics. The AvclToProlog.xslt conversion stylesheet reads AVCL XML to produce Prolog source code, encouraging AVCL support in multiple robots.
🔖 Python Test Programs	README.md and MissionExecutionEngine.py	test.all.missions.python.log.txt exhaustively tests all XML mission definitions Video: Python mission evaluation demonstration shows how a mission execution engine can read and simulate AVCL XML missions for individual runs or exhaustive testing. SailorOverboardMission8State.py example single mission SailorOverboardMission8StateInitialRun.txt SailorOverboardMission8StatePythonSyntaxHighlighted.html				Python is a strictly defined high-level, general-purpose programming language.

🔖 Ontologies

The Mission Execution Ontology (MEO) is available in Turtle and RDF XML forms.

Original	Current	Under Consideration

The Protégé tool is used to create OwlDoc that fully documents internal ontology relationships.

🔖 Queries

Initial queries are checking the soundness of this approach. Future queries will perform in-depth analysis of structural soundness and perform diagnosis that necessary ethical constraints are indeed present for valid mission definitions of arbitrary complexity.

Please see ExampleReasoningQueryingProtege.pptx (.pdf) for further details on design and debugging of these SPARQL queries.

Ant is invoked via build.xml targets to perform all queries. The test framework is sufficiently mature that addition of new diagnostic queries is relatively straightforward.

Log file build.all.log.txt is maintained as a log of all conversions, queries and responses Tracking version control history for all assets is an excellent form of regression testing to confirm that corrections and improvements are confirmable in future builds.

🔖 Resources

The Savage Developers Guide describes how to install and configure commonly used software-development tools Ant, Java, Netbeans and XMLSpy.

Additional tools include the following.

• Allegro Common Lisp by Franz Inc. is a development system for both Lisp and also Prolog with a free edition available.
• Apache ARQ is an open-source query engine for SPARQL.
• Armed Bear Common List (ABCL) is a full implementation of the Common Lisp language, written in open-source Java and runnable via both command line and Ant.
• Protégé is a free, open-source ontology editor and framework for building intelligent systems.
• GNU Prolog is a free open-source compiler for the ISO-standard Prolog programming language.
• ROBOT is an OBO Tool (ROBOT) (with tutorial) is a free open-source command-line tool for working with Semantic Web languages.

🔖 Visualization

We are also working to show visualization capabilities using the AUV Workbench so that human operators might rehearse and replay missions for meaningful assessment. Initial exemplar follows.

How can we rapidly test missions and visualize their progress?

• AUV Workbench Demo mode resets each goal's Search type to EXPANDINGSQUARE.
• Slight change in criteria: reaching the center of the goal is sufficient to declare goal success, skipping the actual search.
• Demo mode thus lets a mission proceed as written in rapid fashion to illustrate decision-tree logic checking.

Demo: 2D map display above shows

• Green arrows are nextOnSuccess,
• Orange arrows are nextOnFailure,
• Bolded arrows indicate which of choices was taken during mission conduct.
• _S_uccess _F_ailure letters (upper-left panel) also report results of each Goal's conduct.

Goal-by-goal narrative of completed mission conduct

• [0] Launch from launch position.
• [6] Avoid first obstacle, next proceed to goalA boundary and then center.
• [2] Once at center: success goalA (Search), next proceed to goalB boundary and then center.
• [3] Once at center: success goalB (Sample Environment), next proceed to goalC boundary and then center.
• [4] Once at center: failure goalC (Search), next proceed to goalD boundary and then center.
• [5] Immediate failure without transit to goalD (Rendezvous) since capability is declared as not supported by robot controller.
• [1] Proceed directly to nextOnSuccess Recovery Position, once there finished in terminal state.

Methodology summary

• Read an AVCL mission document
• Test decision logic using a virtual Software In The Loop (SITL) robot controller,
• Run within an evaluation virtual environment (AUV Workbench).
• Human evaluators then evaluate proper execution of the AVCL ethical mission, specifically assessing that robot-controller SITL within the AUV Workbench virtual environment.

🔖 Contact

Questions, suggestions, additions and comments about this Ethical Control of Unmanned Systems page are welcome. Please send them to Don Brutzman and Curt Blais (email brutzman at nps.edu and clblais at nps.edu).

Master version of this page is available online at
https://savage.nps.edu/EthicalControl and available in GitLab version control.

Updated: 10 February 2021