Augmented reality programming: how AR apps work and how to build one

What an AR app actually does

Apple's ARKit documentation describes augmented reality as adding 2D or 3D elements to the live view from a device's sensors so that they appear to inhabit the real world. Every AR framework, on a phone, a headset or in a browser, splits that into three jobs that repeat on every frame:

1. Track the device. Work out where the camera is and which way it points, many times a second.
2. Understand the scene. Find floors, tables and walls, estimate depth and light, and recognize known images or objects.
3. Render the content. Draw virtual objects from a virtual camera that matches the real camera's pose, then combine them with the view of the real world.

The platform does most of the first two. Your application code decides what appears, where it is anchored, how people interact with it and what data leaves the device.

How the result reaches the eye depends on the display. The W3C's WebXR Augmented Reality Module separates two kinds. Pass-through displays, such as phones and the Varjo XR-3 headset, show a camera image and composite the virtual content over it in software. See-through displays with additive light, such as HoloLens 2 and Magic Leap devices, let the user look through transparent optics while virtual content is added on top. The platform tells your code which one it is running on: in WebXR, the session's environmentBlendMode reports alpha-blend for pass-through and additive for see-through.

How AR tracking works: visual-inertial odometry and SLAM

AR tracking fuses two kinds of sensor. The camera sees the room; the inertial measurement unit (IMU) measures how the device accelerates and rotates. Together they produce what Apple calls a high-precision model of the device's position and motion.

Apple calls its technique visual-inertial odometry (VIO): ARKit finds notable features in the camera image, tracks how their positions change from frame to frame, and compares that with the motion-sensor data. Google describes ARCore's version as simultaneous localization and mapping (SLAM): visually distinct feature points are combined with IMU readings to estimate the camera's pose, its position and orientation, relative to the world over time. Either way, the output is a pose for every frame: a position and an orientation, six degrees of freedom in all. ARKit expresses it in a right-handed coordinate system with the y-axis pointing up.

Because tracking is image analysis, it fails in predictable ways, and good AR apps design for them:

• Too little detail. A blank wall or a dark room gives the tracker too few features. ARKit reports limited tracking with the reason insufficientFeatures, and ARCore warns that textureless surfaces such as a white wall may not be detected as planes.
• Too much motion. Moving too far, too fast or shaking the device blurs the image and reduces tracking quality (excessiveMotion in ARKit).
• Start-up and interruptions. A session begins with no pose, passes through a limited state while it initializes, and only then reaches normal tracking. After an interruption, such as switching apps, ARKit can try to relocalize, which only works if the device returns to roughly where it was.

While tracking is limited, ARKit adds no new planes and hit tests return nothing, so the app should tell the person what to do: move more slowly, turn on a light, point at a surface with some texture. Apple provides a standard coaching overlay, ARCoachingOverlayView, for exactly this. To resume a session later, even after the app relaunches, ARKit can save an ARWorldMap (iOS 12 and later), though Apple warns that relocalizing to it is reliable only when the lighting and the room have not changed much.

Scene understanding: planes, depth, meshes and anchors

Knowing where the device is only matters once the app knows what is around it. The frameworks expose that understanding in layers.

Planes. ARCore looks for clusters of feature points that lie on a common horizontal or vertical surface, such as a table or a wall, and reports each one as a plane with a boundary. ARKit does the same when plane detection is on. A new plane's position and size are rough at first and are refined over time, which is why Apple advises giving plane detection time and switching it off once you have what you need.

Hit tests. To place something where a person taps, the app casts a ray from that screen point into the scene and asks which plane or feature point it meets; the answer is a pose in world space. ARKit offers this as ray casting; ARCore and WebXR call it a hit test.

Anchors. This is the part newcomers miss. The framework's estimates keep changing: as its map improves, planes and feature points shift. Content placed at raw world coordinates slides when those corrections land; content attached to an anchor moves with them and stays on the table. Google's performance guide is direct about it: use an anchor whenever possible. Its fundamentals page adds the other half: to reduce CPU costs, reuse anchors and detach the ones you no longer need.

Depth. ARCore's Depth API builds depth maps from the main RGB camera with a depth-from-motion algorithm, and merges data from a time-of-flight sensor where the device has one. Google says it is most accurate between about half a metre and five metres from the scene. Depth lets real objects hide virtual ones (occlusion), lets virtual objects collide with real surfaces, and improves hit tests on low-texture surfaces where plane detection fails.

Meshes. On devices with a LiDAR Scanner, such as the fourth-generation iPad Pro, ARKit's scene reconstruction turns depth into a polygon mesh of the room, optionally classified by object type, which the app can use for occlusion, physics and more accurate placement. Apple Vision Pro offers scene reconstruction through ARKit as well.

Light, images and objects. ARCore estimates the average intensity and colour correction of each camera image so virtual objects can be lit to match. Both platforms can recognize known 2D images, such as a poster or product packaging, and anchor content to them. On Vision Pro, ARKit can also track known 3D objects, and apps with Apple's enterprise entitlement can locate and read barcodes and QR codes.

Rendering: making virtual content look like it belongs

Rendering is where AR comes closest to game development. On each frame, the app sets its virtual camera to the pose the tracker reported and draws the scene from that viewpoint; on a phone, the result is laid over the camera image. If the virtual camera and the real one disagree, content visibly slides.

The budget is tight. Apple's Human Interface Guidelines ask AR apps on iPhone and iPad to update the scene 60 times per second so that objects do not jump or flicker. That leaves about 16.7 milliseconds per frame for your logic and drawing, on a device that is running the tracker at the same time. Google adds a useful rule: users generally prefer a frame rate that is lower but consistent over one that is higher but variable.

Realism comes from a few specific techniques rather than detail for its own sake:

• light virtual objects with the platform's light estimate, and cast soft shadows onto detected surfaces;
• use depth or a scene mesh for occlusion, so a real chair can stand in front of a virtual lamp;
• keep reflective surfaces small or coarse, because the reflections ARKit provides are approximations built from the camera image;
• keep text out of the 3D scene where you can, and put controls in screen space instead.

Most teams use an engine rather than drawing with raw graphics APIs. RealityKit is Apple's 3D rendering engine for AR and visionOS; it is built on an entity component system (ECS), in which entities carry components and systems update them every frame, and it has adopted MaterialX, an open standard for materials. Unity and Unreal Engine are the usual cross-platform choices: ARCore, Android XR and OpenXR all list both. In the browser, a JavaScript 3D library such as three.js, which ships WebGL and WebGPU renderers, does the drawing.

3D assets are a large share of the work, and the file format follows the platform. Apple's AR Quick Look takes .usdz or .reality files, and Reality Composer Pro prepares scenes for RealityKit. Most other tools use glTF, the Khronos Group's royalty-free format for delivering 3D scenes, usually as a single binary .glb file; glTF 2.0 is also the international standard ISO/IEC 12113:2022. Keep models light: on a phone, every polygon and texture competes for the same processors as the tracker.

AR platforms, SDKs and languages in 2026

The platform you target decides the SDK, and the SDK decides the language. As of September 2026:

Apple: ARKit, RealityKit and visionOS

On iPhone and iPad, an app runs an ARSession with a configuration that says what to track, and shows the result in a RealityKit view. The core setup is short:

let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = [.horizontal, .vertical]
if ARWorldTrackingConfiguration.supportsSceneReconstruction(.mesh) {
    configuration.sceneReconstruction = .mesh // devices with a LiDAR Scanner
}
arView.session.run(configuration)

Apple Vision Pro changes the model. visionOS apps are built with SwiftUI and RealityKit and present content in windows, 3D volumes and spaces. Apps launch into the Shared Space, side by side with other apps; an app that opens a Full Space and asks permission can use ARKit for plane detection, world tracking, hand tracking, scene reconstruction, and image and object tracking. Unity projects can target visionOS too, AR Foundation included. Vision Pro can also hold the editor you already use: Mac Virtual Display shows a Mac's screen as a window of up to a 32:9 ultrawide layout.

Google: ARCore and Android XR

ARCore provides motion tracking, environmental understanding and light estimation on supported Android phones running Android 7.0 or later, with SDKs for Android (Kotlin or Java), the Android NDK (C), Unity, iOS, Unreal Engine and the web. Two of its cloud services reach beyond a single device.

The Geospatial API localizes a phone against Google's Visual Positioning System, a model built from Street View imagery, so content can be anchored by latitude, longitude and altitude, relative to the terrain, or to a rooftop, anywhere Street View covers, without mapping the place first. Cloud Anchors let one device host an anchor, uploading a feature map of the space in exchange for an ID, so that other Android or iOS devices can resolve the same anchor in the same place.

Android XR is Google's platform for headsets and glasses. Its Jetpack XR SDK includes Jetpack Compose for XR for spatial UI, Jetpack SceneCore for the 3D scene graph, ARCore for Jetpack XR for perception (planes, anchors, hands, depth) and Jetpack Compose Glimmer for display glasses; Unity, Godot, Unreal Engine, OpenXR and WebXR are supported as well. As of September 2026 the SDK is in Developer Preview 4, so expect API changes before you commit a product to it.

Cross-platform: Unity AR Foundation and OpenXR

Unity's AR Foundation gives one API, written in C# like the rest of a Unity project, for sessions, device tracking, planes, images, anchors, meshing, occlusion and more. It implements none of them itself. Each target needs its provider plug-in: Google ARCore XR Plug-in on Android, Apple ARKit XR Plug-in on iOS, the Apple visionOS XR Plug-in, and OpenXR plug-ins for HoloLens 2, Meta Quest and Android XR. Features also differ by platform: in the AR Foundation 6.5 feature table, body tracking and shared-session participants are listed for iOS only, and meshing is not listed for Android.

OpenXR is the Khronos Group's royalty-free standard for XR devices. An application creates an instance, selects a system (a display plus input and tracking devices), then creates a session and runs its render loop against the device's conformant runtime. Version 1.1 arrived on April 15, 2024, and in June 2025 Khronos published ratified spatial entities extensions for plane and marker tracking, spatial anchors and anchors that persist across sessions. Conformant runtimes include Meta Quest, Android XR headsets, Pico, HTC Vive, Magic Leap 2, Microsoft HoloLens and SteamVR. Apple devices are not on the list, so Vision Pro work stays with Apple's frameworks or Unity. You usually reach OpenXR through an engine: Unity has shipped an OpenXR plug-in since Unity 2020 LTS, Unreal Engine has supported it since version 4.24, and Godot has had it built in since 4.0.

AR in the browser: WebXR, Quick Look and model-viewer

WebXR is the W3C's browser API for VR and AR, and it is still a draft: the core WebXR Device API is a Candidate Recommendation Draft (latest June 9, 2026), the Augmented Reality Module that adds the immersive-ar session mode is one too (April 25, 2025), and the Hit Test Module is a Working Draft (December 11, 2025). Further modules cover anchors, depth sensing, DOM overlays and lighting estimation. In Chrome on Android, WebXR's AR features run on ARCore.

A page has to be served in a secure context (HTTPS, or localhost during development) with a certificate that never lapses (see how to renew an SSL certificate). It checks for support, then requests the session in response to a user action such as a tap:

arButton.addEventListener("click", async () => {
  if (!(await navigator.xr?.isSessionSupported("immersive-ar"))) return;
  const session = await navigator.xr.requestSession("immersive-ar", {
    requiredFeatures: ["hit-test"],
    optionalFeatures: ["anchors", "dom-overlay"],
    domOverlay: { root: document.getElementById("ar-ui") },
  });
  // Next: bind a WebGL layer, request a reference space and
  // draw each frame from session.requestAnimationFrame().
});

MDN still marks the WebXR Device API as experimental and not Baseline, because it does not work in some of the most widely used browsers, so plan a fallback.

For the most common business case, letting a shopper see a product in their own room, you often need no AR code at all. On iPhone and iPad, a link to a .usdz file with rel="ar" opens the model in AR Quick Look from Safari. On Android, Scene Viewer does the same from a link or an app. Google's <model-viewer> web component wraps both: it displays a glTF or GLB model in any evergreen browser and, with the ar attribute, tries WebXR, then Scene Viewer, then Quick Look, which is its default ar-modes order, generating the USDZ file on the fly if you do not supply one.

Choosing an approach: native, engine or web

Two rules follow from the table. First, choose by where your users already are: a customer-facing feature has to run on the phones your customers own, while an internal tool can standardize on one device. Second, keep what you own portable: models in glTF or USD, content and business logic in your own back end, and the platform-specific code in a thin layer on top.

How to build and test an AR app

AR adds a variable that ordinary app testing does not have: the room. A workflow that holds up:

1. Write down the physical conditions. Room size, the surfaces content will sit on, indoor or outdoor light, and how long people will hold the device. Apple's guidelines ask you to expect cramped spaces without large flat surfaces, and to say up front what the experience needs.
2. Make and test the 3D assets early, in the target formats and at the scale they will appear. Apple's Object Capture API builds models optimized for AR from photographs. A model that looks wrong in a viewer will look worse in a room.
3. Gate by device. ARKit needs iOS 11 or later and an A9 processor or later. On iOS, add the arkit device capability if the whole app needs AR, or check isSupported if AR is optional, and hide the feature on devices that cannot run it rather than show an error.
4. Iterate in a simulator. Unity's XR Simulation runs AR Foundation apps in the Editor inside simulated rooms. The visionOS simulator in Xcode lets you try different room layouts and lighting. The Android Emulator runs ARCore apps in a virtual scene, on x86 system images at API level 27 or later. WebXR does not run in the Android Emulator, so use Chrome DevTools port forwarding to a real phone, which also gives you a secure localhost origin.
5. Record real sessions and replay them. ARCore's Recording and Playback API saves the camera video and IMU data to an MP4 file that ARCore can later treat as a live feed, so a bug found in a shopping centre can be reproduced at a desk.
6. Test on real devices in real rooms. Cover the device classes your users own, low light, blank walls, fast movement, a phone call in the middle of a session, relocalization after an interruption, and sessions long enough to heat the device.
7. Ship the fallbacks and notices. Give people who decline camera or sensor access another path, and include the privacy disclosures your platform requires (see below).

Performance: keep the frame rate steady while the tracker runs

On a phone, your app and the tracker share the same CPU and GPU. Google warns that CPU-bound apps can starve ARCore's motion tracking, and gives a direct check: if this message appears in the device log, SLAM is not getting enough time.

adb logcat | grep 'VIO frequency low'

What helps:

• Anchor content and reuse anchors. Anchored objects stay stable when the map is corrected, and detaching anchors you no longer need reduces CPU cost.
• Turn features off when you are done with them. Google names Instant Placement (disable it once full tracking is established) and Augmented Images (disable it when not needed) as CPU-heavy. Apple suggests stopping plane detection once you have the planes you need.
• Watch thermals. During QA, read Android's PowerManager.getCurrentThermalStatus() (API level 29) and getThermalHeadroom(int) (API level 30) on a production build, because debug builds perform differently.
• Test across device classes. ARCore phones differ in CPU and GPU, memory, camera and IMU quality, and drivers. Test on devices that represent your users, not only on the newest one.

Privacy and safety in AR apps

An AR app sees more than most apps: a live camera feed, the layout of someone's home or workplace, and sometimes their face and hands. The platforms put rules around each of these, and a well-built app treats those rules as the minimum.

• Camera access on iOS. ARKit needs camera permission and asks for it the first time a session runs, and the app must explain why in NSCameraUsageDescription. An app that uses face tracking must also have a privacy policy that describes how it uses face data.
• Sensor data on Vision Pro. ARKit data is available only when the app presents a Full Space and other apps are hidden, and world sensing needs a usage description and the person's authorization (plain world tracking does not). People can decline or revoke access in Settings, so anything that depends on it needs a fallback. Access to the main camera is an enterprise entitlement that Apple grants only for apps used in a business setting.
• Google's disclosures. Apps that use ARCore must say so prominently, and apps that use Cloud Anchors or the Geospatial API must show Google's notice that Google processes sensor data, such as camera and location, to power the session, with a link to learn more.
• The browser. WebXR runs only in a secure context, and immersive sessions are requested in response to a user action such as a tap.

Beyond the platform rules, decide what your own system keeps. World maps, scene meshes and hosted anchors describe the inside of real rooms. Store them only when a feature needs them, keep them only as long as it needs them, and protect the API that serves them like any other: secure coding practices apply to an AR back end exactly as they do to a web app.

Where AR earns its place in a business

AR is worth building when seeing something in place answers a question that a photo cannot. The strongest patterns map directly onto the capabilities above.

Seeing a product before buying it. Will the armchair fit? Google uses exactly this example to explain its Depth API. For most stores this starts as a 3D viewer on the product page, and AR Quick Look can show a banner with an Apple Pay button or a custom action inside the AR view.

Wayfinding and location content. The Geospatial API is designed for guiding people to a place, such as finding a car in a dense city or getting around a busy venue, with content attached to real-world coordinates instead of a hand-built map.

Shared and persistent content. With Cloud Anchors, one person places content and others see it in the same spot, either at the same time or weeks later. ARKit has collaborative sessions for nearby devices, and RealityKit supports group experiences over SharePlay. In every case, the second device has to recognize the place before it can show what was left there.

Hands-busy work on headsets. Training, inspection and guided procedures suit head-worn devices, where ARKit's object tracking can find a known machine. For apps that a business distributes privately to its own staff, Apple's enterprise APIs for Vision Pro add capabilities such as main camera access and spatial barcode and QR code scanning. Whether a job suits AR or VR at all, and what the training evidence shows, is covered in AR vs VR.

Plan for the platform changing under you. Microsoft states that, as of December 2024, HoloLens devices are no longer manufactured, and 8th Wall retired its hosted WebAR platform on February 28, 2026, ending paid subscriptions and moving to an open-source release. Neither had to end a project that kept its 3D assets in open formats and its content and logic in its own back end. That is the architecture to ask for.

An AR feature is rarely a product on its own. It needs product data, accounts, an API that delivers models and somewhere to keep anchors, which is ordinary custom software development: the APIs and back ends behind platforms, portals and internal tools. If your first step is a 3D viewer on a product page, that page still has to load quickly on a phone; our web design and development work holds pages to Google's Core Web Vitals thresholds on real phones, and a model viewer has to fit inside that budget.