TL;DR: I spent part of my Unity years building the ERIS XR platform and AR tools at ARCortex, work that reached the U.S. Fire Administration. Building augmented reality for firefighters is nothing like building a demo. The headset has to survive a bad connection, a distracted operator, and a life-or-death decision, and it has to do that outdoors, in daylight, with gloves on. This is what public-safety AR taught me about latency, trust, and knowing when the map is wrong, lessons I carry straight into the AI systems I build now.
Most AR you see in a keynote is a beautiful thing floating in a clean room. Public-safety AR is the opposite discipline. The user is stressed, the environment is hostile, and a wrong overlay is worse than no overlay. When I joined ARCortex, I moved from consumer and game AR into systems where the stakes were real: the ERIS XR platform for the California Fire Brigade, AR mapping tools, and defense work including MQ-9 Reaper tracking on AR maps and property visualization from planes. Here is what that world demanded that a demo never does.
What ERIS actually had to do
ERIS was an XR platform built for fire response, not for a trade show. The core job of this class of tool is to take a flood of spatial data, unit positions, terrain, structures, incident markers, and put it in front of a human who is already at capacity, without adding to the load.
That constraint reorders every engineering priority you thought you had. In a game, the frame is the product. In a game, a dropped frame is an annoyance. In fire response, a stale marker is a person told a building is clear when it is not. So the first thing public-safety AR teaches you is that correctness and freshness of data outrank visual polish by a wide margin. A plain, correct overlay beats a gorgeous, slightly-late one every time.
The map is a claim, not a fact
The hardest idea to internalize was that every overlay is a claim about the world that can be wrong. GPS drifts. A structure was renovated since the last survey. A unit moved but the packet did not arrive. In a consumer AR app you paper over this with animation and forgiveness. In public safety you have to design for the overlay being wrong, because sometimes it is.
That meant building in honest signals of confidence and staleness. When a position had not updated recently, the interface had to say so rather than keep rendering a crisp, confident dot that lied. I would rather show an operator a marker that admits "last seen 40 seconds ago" than one that looks authoritative and is not. This is the exact instinct I now bring to AI agents: a system that knows the edge of what it knows is far safer than one that is always confident. I wrote about the same boundary problem for voice agents that must escalate instead of guess, and it is the same principle wearing a different hat.
Latency is a safety property, not a benchmark
In web work, latency is a number you optimize for conversion. In AR for a moving operator, latency is a physical safety property. If the overlay lags the head by even a fraction of a second, two things happen: the illusion breaks, and worse, the user starts to distrust the whole system. Once a firefighter decides the headset lies, they stop looking at it, and you have shipped dead weight strapped to someone in danger.
So the latency budget was sacred and split across the whole chain:
- Tracking to render. Head and device motion had to map to the overlay with as little delay as the hardware allowed, or the world would swim.
- Data to display. Positions and markers coming over the network had to be reconciled and drawn without stalling the render loop, which means the networking never blocks the frame.
- Interaction to feedback. Any input, a selection, a mode switch, had to acknowledge instantly, even if the underlying action took longer to complete.
The lesson that generalized: decouple the thing the user feels from the thing the system does. Acknowledge immediately, reconcile in the background, and never let slow work freeze the surface a person is depending on. I use the exact same architecture in my agent work now, where the automation layer runs behind the conversation rather than blocking it.
Designing for gloves, sun, and stress
A surprising amount of public-safety AR is not graphics. It is human factors under conditions no office user ever faces.
Gloves. Fine gestures and small targets are useless when the operator is wearing thick protective gloves. Interaction had to collapse to a few large, unmistakable actions. Every clever multi-finger gesture I might have used in a consumer app was off the table.
Daylight. Outdoor AR competes with the sun. Subtle, low-contrast interfaces that look elegant indoors vanish in bright field conditions. Overlays had to be high-contrast and legible in the worst lighting, not the best.
Cognitive load. The operator is already running an incident in their head. The interface's job is to subtract load, not add it. That meant ruthless information hierarchy: show the two things that matter now, tuck everything else away, and never make someone hunt through a menu during an emergency.
None of this shows up in a portfolio screenshot. All of it is the actual work. Building for the hardest user in the hardest moment is a design discipline that quietly makes everything else you build better.
The AR tools around ERIS taught the same lessons at smaller scale
Alongside the platform, I built a family of AR tools at ARCortex, AR Planes and AR Properties for spatial visualization, plus experimental viewers, and on the defense side, MQ-9 Reaper tracking rendered on AR maps and property visualization from planes. Different domains, same core problem: fuse a live data stream with the physical world and present it to someone who has to act on it.
Tracking a moving aircraft on an AR map is, structurally, the same challenge as tracking a fire unit: a stream of positions, uncertainty in each one, and a human who needs the current truth at a glance. Once you have solved the honest-uncertainty and latency problems for one, you have the pattern for all of them. That is the compounding value of hard constraints. Solve the strict version once and the relaxed versions come for free.
What public-safety AR gave my AI work
I have since moved from Unity and headsets into AI agents and my own products, an arc I traced in my freelance lessons piece. People assume that was a clean break. It was not. The mindset transferred almost intact:
- Design for being wrong. Every AR overlay could be wrong, so I built confidence and staleness into the surface. Every LLM output can be wrong, so I build the same escalation and human-in-the-loop boundaries into agents.
- Latency is trust. A slow overlay loses the operator; a slow agent loses the caller. Decouple what the user feels from what the system does, in both worlds.
- The hardest user sets the spec. A gloved firefighter in the sun forced clarity that made every interface better. A stressed customer on a bad phone line does the same for a voice agent.
- Freshness beats polish. Correct and current beats pretty and late, whether it is a unit marker or a booking slot.
The tools change. I have been paid for C++, Unity, AR headsets, and now LLM agents, and I expect that list to keep growing. The durable skill is reading what a system truly cannot afford to get wrong and building around that first. Fire response taught me that lesson at the highest possible stakes, and I have never built the same way since.
I build immersive and AI systems for teams that cannot afford to get the hard parts wrong, from XR platforms to voice and chat agents. If that is you, my inbox is open: more about my background here, or book a call.