Spatial computing

Spatial computing refers to 3D human–computer interaction techniques that are perceived by users as taking place in the real world, in and around their bodies and physical environments, instead of constrained to and perceptually behind computer screens or in purely virtual worlds. This concept inverts the long-standing practice of teaching people to interact with computers in digital environments, and instead teaches computers to better understand and interact with people more naturally in the human world. This concept overlaps with and encompasses others including extended reality, augmented reality, mixed reality, natural user interface, contextual computing, affective computing, and ubiquitous computing. The usage for labeling and discussing these adjacent technologies is imprecise.^[1]

Spatial computing devices include sensors—such as RGB cameras, depth cameras, 3D trackers, inertial measurement units, or other tools—to sense and track nearby human bodies (including hands, arms, eyes, legs, mouths) during ordinary interactions with people and computers in a 3D space.^[2] They further use computer vision to attempt to understand real world scenes, such as rooms, streets or stores, to read labels, to recognize objects, create 3D maps, and more. Quite often they also use extended reality and mixed reality to superimpose virtual 3D graphics and virtual 3D audio onto the human visual and auditory system as a way of providing information more naturally and contextually than traditional 2D screens.

Spatial computing often refers to personal computing devices like headsets and headphones, but other human-computer interactions that leverage real-time spatial positioning for displays, like projection mapping or cave automatic virtual environment displays, can also be considered spatial computing if they leverage human-computer input for the participants.

History

The term "spatial computing" apparently originated in the field of GIS around 1985^[3] or earlier to describe computations on large-scale geospatial information. Early examples of spatial computing in GIS include ArcInfo and its iterations, initially released in 1981,^[4] a part of ArcGIS along with ArcEditor, which together provide mapping, analysis, editing, and geoprocessing for geodatabases.^[5] This is somewhat related to the modern use, but on the scale of continents, cities, and neighborhoods.^[6] Modern spatial computing is more centered on the human scale of interaction, around the size of a living room or smaller. But it is not limited to that scale in the aggregate.

In the early 1990s, as field of virtual reality was beginning to be commercialized beyond academic and military labs, a startup called Worldesign in Seattle used the term Spatial Computing^[7] to describe the interaction between individual people and 3D spaces, operating more at the human end of the scale than previous GIS examples may have contemplated. The company built a CAVE-like environment it called the Virtual Environment Theater, whose 3D experience was of a virtual flyover of the Giza Plateau, circa 3000 BC. Robert Jacobson, CEO of Worldesign, attributes the origins of the term to experiments at the Human Interface Technology Lab, at the University of Washington, under the direction of Thomas A. Furness III. Jacobson was a co-founder of that lab before spinning off this early VR startup.

In 1997, an academic publication by T. Caelli, Peng Lam, and H. Bunke called "Spatial Computing: Issues in Vision, Multimedia and Visualization Technologies" introduced the term more broadly for academic audiences,^[8] focusing on a variety of topics such as image processing, dead reckoning navigation, object recognition, and visualizing spatial data.

The specific term "spatial computing" was later referenced again in 2003 by Simon Greenwold,^[9] as "human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces". MIT Media Lab alumnus John Underkoffler gave a TED talk in 2010^[10] giving a live demo of the multi-screen, multi-user spatial computing systems being developed by Oblong Industries, which sought to bring to life the futuristic interfaces conceptualized by Underkoffler in the films Minority Report and Iron Man.

Google Earth, initially released by Keyhole Inc. in 2001 and re-released by Google in 2005 can be considered a capable GIS and includes advanced geospatial tools and capabilities.^[11]

In February 2014, Google announced Project Tango, a prototype Android smartphone equipped with depth sensors and motion-tracking cameras capable of making over 250,000 3D measurements per second, with the stated goal of giving mobile devices "a human-scale understanding of space and motion."^[12] While Tango demonstrated that spatial computing was feasible on a handheld device, its reliance on specialized hardware limited adoption to a small number of devices.^[13] In June 2017, Apple introduced ARKit at WWDC, a software framework that brought augmented reality capabilities to hundreds of millions of existing iPhones and iPads using their built-in cameras and motion sensors, without requiring specialized hardware.^[14] Google followed in August 2017 with ARCore, a software-based augmented reality platform for Android built on Tango's research but designed to run on standard smartphones, targeting 100 million devices at launch.^[15] Google deprecated Tango on March 1, 2018, shifting its augmented reality efforts entirely to ARCore.^[16]

Notable instances of the use of spatial computing

In 2019, Microsoft HoloLens released a video outlining Airbus' partnership with Microsoft Azure to utilize the latter's mixed reality services for streamlining and improving the aircraft design process, as well as reducing the error in development.^[17] Airbus utilized the HoloLens 2 to this end, and the executive vice president of engineering claimed that their design process' validation phases were "hugely accelerated by 80 percent", as well as "strongly believe[d]" that up to 30% improvements in their industrial tasks could be attained with the HoloLens 2. During the presentational video, Airbus cited the maturity of Microsoft Azure services as "key" for their usage of the HoloLens 2.

Also in 2019, the U.S. army partnered with Microsoft to produce a HoloLens based Integrated Visual Augmentation System (IVAS) to enhance infantry members by giving troops various abilities, including but not limited to using holographs to train, projecting 3D maps into their vision, and seeing through smoke and corners.^[18] Microsoft received tens of thousands of hours of feedback for their systems by 2021. Sergeant Marc Krugh at the time claimed that Microsoft's partnership has already caused the army to rethink some of its troops' operation strategy.

In July 2016, Niantic released Pokémon Go, a mobile game that used GPS, the device camera, and inertial sensors to overlay virtual characters onto real-world locations. The game reached over 500 million downloads within two months of launch and was widely credited with introducing augmented reality and spatial computing concepts to a mainstream consumer audience.^[19]

Products

Apple Vision Pro

Apple announced Apple Vision Pro, a device it markets as a "spatial computer", on June 5, 2023. It includes several features such as Spatial Audio, two 4K micro-OLED displays, the Apple R1 chip and eye tracking, and released in the United States on February 2, 2024.^[20] In announcing the platform, Apple invoked its history of popularizing 2D graphical user interfaces that supplanted prior human-computer interface mechanisms such as the command line. Apple suggests the introduction of spatial computing as a new category of interactive device, on the same level of importance as the introduction of the 2D GUI.

Apple Vision Pro runs on a new operating system called visionOS, which combines eye tracking, gesture recognition, and voice input to enable immersive interaction without physical controllers. The platform is aimed at productivity, entertainment, collaboration, and enterprise use cases.

Meta Quest

The Meta Quest 3, a mixed reality gaming headset that includes spatial audio, two color cameras, and grants the ability to interact with virtual characters released on October 9, 2023, at a notably cheaper price than the Apple Vision Pro, but with reduced capabilities.^[24][25]

See also

• Brain–computer interface – Connection between brain and computer
• Cyberspace – Concept describing a widespread, interconnected digital technology
• OpenXR – Standard for access to virtual reality and augmented reality platforms and devices
• Transhumanism – Philosophical movement
• Virtual community – Virtual community that exists onlinePages displaying short descriptions of redirect targets
• Virtual world – Large-scale, interactive computer-simulated environment
• WebXR – Experimental JavaScript API for augmented/virtual reality devices
• Wirehead – Concept in fiction or futuristic applications

References