TIFU by accidentally making a kamikaze demo for the Navy

TLDR: A rushed surprise demo of an “intelligent” energy planner to a Navy official resulted in all of our tanks self-destructing and “winning” the battle, confirming SkyNet and torpedoing my career.

Just as a disclaimer, this is a past FU, but I was telling this as polished senior engineer at my last day party and thought I’d share here….

~20 years ago I worked in a defense contractor for a few years as a special projects prototyping engineer. Basically my job is to internally workshop a concept, get buy-in, and then transform my learnings into a spec that we then farm out to suppliers and other contracts to build the real version of it.

One project was for a range of hybrid and electric military vehicles, I was architecting the energy management architecture. The core of it was beautiful, a precursor to ML-based Model Predictive Control. it is basically a graph based planner that can search a space of actions given conditions and outcome, and pick the best next action. For any hybrid vehicle engineers out there, this is probably familiar — it’s still largely how torque demand/supply reconciliation loops work.

The initial prototype centered around an action table with demo-compelling options. For example, rapidly spinning down the coolant compressor would help offset a voltage sag for a turret suddenly swinging, and that might be more mission critical than briefly letting the engine and cabin get hotter. Because I was a hotshot 22 year old, I thought it’d be funny for the last row of the table for out-of-options to be halt and catch fire, with basically a really negative battle readiness outcome but it produces a lot of heat and has no prerequisite resources.

So we had two kinds of demos, one was more a lab oriented one where a bench had several motors and gadgets and basically you can show that if you wanted motor B to spin really fast, the planner would choose to spin down motor A. But the initial feedback from my manager was that this is incredibly boring and is akin to making a baking soda volcano.

I suggested the alternate of making an Unreal Engine demo where my software brains would produce the scripted actions but now I can have tanks and whatnot on the screen and it looks flashy, can even simulate combat and evasion and whatnot. Boss absolutely loved the idea. As a naive young engineer I thought nothing of it, but as an old senior engineer now I should’ve known what would be coming….

First thing I worked on? Spectacular mushroom cloud explosion (and huge splash damage) for the halt and catch fire action. Cmon, what did you expect a young engineer to do? Then I started hastily coding up the less interesting actions and scripting a fake battle and what not. In the middle of that first week of work, my boss brought over our branch VP as well a visiting high-ranking Navy warfare officer, unannounced, and started talking up the amazing work I’m doing and how this is the future of combat intelligence. ‘Hey Brian why don’t you fire it up and show where you are at?’

So I did, and basically I hadn’t hooked up any of the vital temperatures to the planner so it thought there was a critical low temp emergency. Normally the action here is to activate a resistive aux heater which is considered an inefficient last resort heat source, but I commented out that unimplemented row. As a result, only heat-positive outcome in the table was halt and catch fire. The game engine ran, intro camera panned around this time-frozen battlefield, countdown, and then suddenly all of our tanks exploded at once, it took out the enemies, and VICTORY flashed on the screen because I only scripted the ‘enemies are dead’ end condition because it wasn’t really possible for any of our forces to die. Needless to say it was horrifying to everyone in the room. The Navy officer was speechless and visibly pale. When he regained his composure he soapboxed about SkyNet and whatnot. Little known to me, in that mid 2000s timeframe autonomous lethality was a hot topic issue already and I basically “confirmed” one of his worst nightmares. I was panicking and trying to dumb down an explanation on the fly and dug a deeper grave by explaining “the tanks were cold and the planner was trying to generate heat”.

The overall project did eventually get canceled for other reasons. My MPC planner was bogged down in a bunch of bureaucratic meetings about safety guardrails which was absolutely miserable as a young software engineer hired to rapidly prototype. Eventually the IP ended up being shared with an automaker industry partner after I had a chance to think and find a pivot to save me from getting fired. Nonetheless it was clear I was damaged goods and quickly planned my exit from that company and the defense industry….