Recognize objects in ARKit with a Core ML model

WHAT THIS TESTS This checks systems thinking: composing ARKit, Vision, and Core ML in real time, respecting the render loop, and bridging a 2D image detection into 3D space, where the hard problem is depth.

A GOOD ANSWER COVERS ARKit runs an ARSession that vends ARFrame objects through session(_:didUpdate:) or via sceneView.session.currentFrame. Each frame's capturedImage is a CVPixelBuffer in camera orientation. You feed it to a VNImageRequestHandler with a VNCoreMLRequest wrapping your custom model, but you run inference on a background queue and typically throttle to every Nth frame, because the render loop targets 60 frames per second and synchronous ML would stall it. You must pass the right CGImagePropertyOrientation derived from device orientation and the camera. When you get a detection with a normalized bounding box, you take a representative 2D point, convert it to view coordinates, and use raycasting, an ARRaycastQuery against existing plane or feature geometry, to find a 3D world position. You then create an ARAnchor at that transform and attach content. The major challenge is depth ambiguity: a single 2D point corresponds to a ray, not a point, so you need plane detection, scene depth (LiDAR), or feature points to resolve where along the ray the object sits, and detections can be noisy frame to frame.

COMMON WRONG ANSWERS Running inference synchronously inside the render delegate, tanking frame rate. Ignoring image orientation, producing wrong detections. Assuming a 2D pixel maps to exactly one 3D point without raycasting or depth. Adding anchors every frame so the scene fills with duplicates.

LIKELY FOLLOW-UPS How LiDAR sceneDepth simplifies the depth problem. Smoothing or debouncing detections to place one stable anchor. Throttling inference and the throughput tradeoff. Coordinate conversion between normalized Vision space and view space.

ONE CONCRETE EXAMPLE Every few frames you grab frame.capturedImage, hand it to a background VNImageRequestHandler with orientation from the interface, and run a VNCoreMLRequest. On a confident detection you take the box center, convert it to the sceneView's coordinate space, build an ARRaycastQuery from that point against estimated planes, take the first result's worldTransform, and add an ARAnchor with a label node. You debounce so the same object is not re-anchored, and use LiDAR depth when available to fix the distance precisely.