Why is my geospatial AR overlay pointing in the wrong direction?

Almost always heading, not position. GPS position error is a few metres in the open, which is invisible for a target kilometres away, but a compass heading that is off by five degrees rotates everything in the sky by five degrees, and at distance that is a huge on-screen offset. The magnetometer is the least reliable sensor in the stack and is thrown off by nearby metal, vehicles, phone cases and buildings. Treat heading as a continuously fused value rather than something sampled once at session start, give the user a calibration gesture to align the overlay against a landmark they can see, and surface low confidence on the interface instead of rendering a crisp marker you cannot justify.

How do you render objects that are kilometres away in Unity AR?

Do not place them at their true metric distance. Floating point precision degrades as objects move away from the session origin, so distant geometry jitters, and widening the camera far plane enough to include them destroys depth buffer precision for everything near the user. The practical approach is to keep the rendered scene compact by placing markers on a fixed radius around the user and carrying the real distance in the label text, and to separate near-range and far-range drawing rather than stretching one camera across both. At that range users are judging direction and identity, not parallax, so legibility beats realism.

How should an AR app handle late or missing position updates?

Interpolate between fixes so the target moves smoothly, extrapolate through short gaps, and cap how far forward you are willing to predict. Without that cap a dropped feed produces a marker that keeps confidently flying on a heading nobody actually reported, which is worse than showing nothing. Past the cap, degrade the marker's appearance so missing data is visible rather than silent, and show data age wherever it affects a decision, because an operator who can see a position is nine seconds old can compensate for it. The underlying rule is to decouple what the user sees from what the system is waiting on.

Geospatial AR Is a Coordinate Problem, Not a Graphics Problem

TL;DR: Geospatial AR, the kind where you point a device at the sky or the street and see a real aircraft, drone or property labelled in place, is almost never a graphics problem. It is a coordinate problem wearing a graphics costume. You are asked to reconcile a global coordinate system that knows where the Earth is with a local one that only knows where the device started, using sensors that disagree with each other. Heading is your worst error source, distance breaks your renderer, altitude means three different things, and your telemetry is always a little late. Here is what I learned building AR Planes, AR Properties and MQ-9 Reaper tracking on AR maps at ARCortex, and why the discipline it forced still shows up in the AI agents I build now.

At ARCortex in San Francisco I spent most of my time on the ERIS XR platform for the fire service, but the work I get asked about most is the family of tools around it: AR Planes and AR Properties for spatial visualization, and on the defense side, MQ-9 Reaper tracking rendered onto AR maps and property visualization from planes.

People assume the hard part was the rendering. It never was. Unity draws a labelled marker in the sky perfectly well on the first afternoon. The hard part is being able to claim, honestly, that the marker is in the right place.

You are reconciling two coordinate systems that do not agree

Every geospatial AR app lives across a seam. On one side is the world's coordinate system: latitude, longitude, altitude, an Earth-shaped model, a stream of telemetry from something real. On the other side is the AR session's coordinate system, which is local, arbitrary, and defined entirely by wherever the device happened to be when tracking initialized.

The AR framework knows a great deal about the two metres around the user and nothing whatsoever about the planet. GPS knows about the planet and almost nothing about the two metres. Your entire job is the transform between them.

Practically, that means converting geodetic coordinates into a local tangent frame, usually east-north-up, centred on a reference origin you pick yourself, then aligning that frame with the AR session's axes. The maths is textbook. The difficulty is that both inputs to the transform are noisy, and one of them is much noisier than developers expect.

Heading is the error you actually see

Position error from GPS is a handful of metres in the open, and for a target three kilometres away that is invisible. Nobody notices a plane rendered five metres off when it is thousands of metres out.

Heading is a different story. If your device's idea of north is off by five degrees, everything in the sky is off by five degrees, and at distance that is enormous. The user does not perceive it as a heading error either. They perceive it as "your app is wrong", because the aircraft they can see with their eyes is over there and your label is over here.

The magnetometer is the culprit. It is affected by nearby metal, by the vehicle you are sitting in, by the phone case, by the building. It is the single least trustworthy sensor in the stack, and it happens to control the axis your users judge you on.

Three things helped more than any amount of filtering:

Do not treat heading as a fixed input. Fuse it, smooth it, and re-evaluate rather than sampling once at session start and trusting it for the next ten minutes.
Let the user correct you. A simple calibration gesture, aligning the overlay against a landmark the user can see, buys more perceived accuracy than any sensor work. Humans are excellent at spotting a constant offset.
Show your confidence. If heading is unstable, say so on the surface instead of rendering a crisp, confident, wrong marker. That is the same honest-uncertainty principle I described in the ERIS piece, and it applies to every overlay that claims to know where something is.

Distance breaks the renderer before it breaks the maths

Once your targets are kilometres away, ordinary Unity assumptions start failing quietly.

Floating point precision degrades as you move away from the origin, so a naive approach that places a plane at its true metric distance from the session origin gives you jittering geometry. The standard fix is not to place things at true distance at all. Keep the rendered scene compact, place markers on a fixed radius around the user, and let the label carry the real distance as text. The user is not judging parallax at four kilometres. They are judging direction and identity.

Camera clipping planes are the other trap. Widening the far plane far enough to include distant objects wrecks depth buffer precision for everything near the user, and a mixed scene with a hand-scale UI and a horizon-scale target is exactly the case that suffers. Separating what draws in which range, rather than stretching one camera to cover both, is the cleaner answer.

And there is a perceptual problem underneath both: a correctly rendered object at four kilometres is a few pixels. Realism is not the goal. Legibility is. The marker has to be readable at a glance by someone who is standing outside, in the sun, doing something else, which is the same design constraint I ran into designing for gloved firefighters.

Altitude means three different things

For ground-level AR you can mostly ignore altitude. For anything airborne you cannot, and this is where geospatial AR gets genuinely fiddly.

An altitude number can mean height above the reference ellipsoid, height above mean sea level, or height above ground level, and different data sources report different ones without always announcing which. Mix them up and your aircraft sits hundreds of feet off vertically, which at moderate distance is a visible and embarrassing error.

The lesson generalizes past AR: when a data source hands you a number with a unit but no datum, the datum is the part that will hurt you. Pin it down in writing at integration time, and render the vertical axis against a reference you have verified rather than one you assumed.

Telemetry arrives late, irregularly, and sometimes not at all

A moving target is a stream of positions with gaps. Updates arrive at uneven intervals, out of order, occasionally stale by seconds. If you render each fix as it lands, the target teleports.

So you interpolate between fixes and extrapolate forward through gaps, which means for most frames you are displaying a prediction rather than an observation. That is fine, as long as you are honest about it. What is not fine is letting extrapolation run indefinitely, because a dropped feed then produces a marker that keeps confidently flying on a heading nobody reported.

The rules I settled on:

Smooth between fixes, but cap how far forward you will predict. Past that cap, degrade the marker's appearance rather than hiding it, so absence of data is visible instead of silent.
Show data age when it matters. An operator seeing a nine-second-old position can compensate. An operator seeing an unlabelled position cannot.
Separate what the user feels from what the system waits on. That decoupling is the same instinct behind the way I structure a voice agent's latency budget. Different domain, identical shape: never let a slow upstream turn into a frozen surface.

The map view is not a fallback, it is the honest view

Rendering targets onto an AR map, rather than into the live camera, felt at first like the lesser feature. It is not. A map is a top-down projection where you control the frame, and it never lies about heading because it does not depend on the magnetometer at all.

For anything operational, the AR camera view is the intuitive interface and the map is the reliable one, and people under pressure switch between them constantly. Building both, and keeping a single shared state behind them so they can never disagree, turned out to be one of the better decisions on that work. It is also the same architectural instinct as keeping one source of truth behind a client dashboard: two views, one model, no chance of a contradiction the user has to referee.

What geospatial AR gave the rest of my work

I have moved from Unity and headsets into AI voice agents and my own products, and this specific work still shapes how I build:

Know which of your inputs is the bad one. In geospatial AR it was heading. In a voice pipeline it is usually transcription of names, numbers and addresses. Every system has one input that dominates the error budget and deserves disproportionate defence.
Never render confidence you do not have. A crisp marker for a stale position and a fluent sentence for a fact the model does not know are the same failure. Both are far worse than visibly saying "I am not sure".
Give the user a second view. The map beside the camera, the transcript beside the call, the dashboard beside the agent. People trust a system they can check.

That transfer, and how much of it survived the jump from real-time systems to LLM systems, is something I have written about across clinical XR and cross-platform SDK work. The stacks look nothing alike. The engineering judgement is almost entirely portable.

I have built AR and XR systems for public safety, defense and healthcare, and now build production AI voice and chat agents for startups in the US and UK. If you have a hard real-world data problem, whether it is spatial or conversational, more about my background is here, or book a call.