Apple Vision Pro starts at $3499. Apple’s current VR experience has so far been challenged in terms of content. Apple figured this would be a problem, so it introduced Object Capture APIs in 2021. Unfortunately, it appears this tech has seen little use by mainstream app developers. Back when it launched, only macOS could crunch the data to create USDZ files and only in apps that specifically implemented it. iPadOS couldn’t use it despite iPad Pro sharing the same M1.
Are developers actually using Object Capture in mainstream apps? So far, we only know about the PhotoCatch app for Mac. This app is not mainstream and terrible SEO made it difficult to find, but it runs Object Capture APIs to create 3D objects from 2D still images. You dump in some photos and the APIs do the work. There are relatively few settings compared to other other photogrammetry apps, so it’s a good thing the app is free.
Apple’s Object Capture API has struggled with asphalt shingles. It may seem minor, but we suspect many users will want to capture images of their homes for the ultimate before and after. Object Capture will usually give up on the roof and fail to render anything. You can sometimes see through massive spaces where roof should be. Oops!
Some objects types cannot be rendered at all. It’s nearly impossible to capture reflective objects or through glass or objects that constantly move. This limits the types of objects we can capture or prevents us from capturing some objects entirely. It’s difficult to appreciate the implications of these limitations until you begin to experiment with rendering. These limitations apply to all photogrammetry software, not only Apple’s. Glass buildings and wavy trees are not ideal.
Rendering 3D objects from 2D image files is traditionally called photogrammetry. We’ve run these operations using more traditional CUDA-powered software and cameras that capture in RAW for maximum control. On an industrial scale, photogrammetry has been used to render objects in modern games like Forza Horizon 5 or Microsoft Flight Simulator.
Traditional capture flows can be relatively intensive. The camera settings and lighting conditions must be consistent for the duration of the capture session, so captures should occur within the span of an hour rather than days. You must consider all the angles needed to render every surface of an object. It may be best to use prime lenses, avoid lenses that show distortion, and use a narrow apertures to capture as much detail as possible.
The post production flow can be challenging, too. You will usually need to batch edit the photos before you ingest the images into software. There’s also a chance you’ll capture some bad shots that’ll need to be removed. There are other tricks for specific conditions, like adjusting blues to eliminate sky detail in outdoor captures to help avoid blue bits on tree leaves. You’ll want a powerful system with plenty of RAM and a strong GPU.
We’ve created 3D renderings using photogrammetry. It’s fun, but it’s also time consuming and we think it’s unusual for one person to possess all the skills needed to do it well. It’s very niche and there are few job postings looking for the types of skills required to do it.
We don’t expect people to wear a headset to look at 3D content on this site, but Apple wants developers to try and Apple wants to call it XR. Is this the future? Do people want this? Perhaps, or perhaps not.
VR headsets have some inherent accessibility challenges, whether you need to charge the battery, run a software update, dig out accessories, move furniture, prepare hair and makeup,… it’s a substantial investment in time and effort to look at these objects in a headset. And that’s all before we consider the download size of the content or whether the headset gives you a headache. Then we expect people to pay for a subscription? It’s a gamble for headsets of any price, especially considering most subscription apps don’t make money.
Apple has a plan for accelerated 3D content creation, but does it matter? It appears Apple has sold fewer than 300k headsets in total. Developers appear to have little incentive to create compelling content for such a small user base, but the elements are mostly there. We should want photogrammetry to be possible using only the local hardware phones in our pockets and purses. It’s neat idea for the future, but current market forces may not allow this software to go mainstream. It’s a shame because Apple has dramatically improved the capture experience, at least in theory.
Video games have come a long way since Mario and Sonic. As game studios seek to create richer, more realistic environments and experiences, photogrammetry saves teams of designers time in trying to digitally replicate objects that already exist in physical space. Consumer-grade photogrammetry tools could improve accessibility for the general public, but compelling use cases for individuals remain unclear.