Instructions

This page will hold purchase, construction, setup, and calibration instructions for an Augmented Reality (AR) Sandbox.

Hardware Requirements

An AR Sandbox requires the following hardware components:

Computer

The ideal computer for an AR Sandbox is a dedicated desktop PC running a version of Linux, with a consumer-level, e.g., Nvidia GeForce, graphics card. The PC should have a good CPU, but does not need large RAM (4GB is sufficient to run the AR Sandbox software) or a large hard drive (20GB is sufficient to install Linux and the AR Sandbox software). While the AR Sandbox software runs under Mac OS X, we strongly recommend Linux because Linux-based installations are more stable.

We recommend an Intel Core i5 or Core i7 CPU running at at least 3GHz (as of 11/2016), an Nvidia GeForce GTX 1060 graphics card (as of 11/2016), and the current release of the 64-bit version of Linux Mint with Mate desktop (18.3 "Sylvia" as of 04/2018). The AR Sandbox requires that the vendor-supplied binary drivers for the graphics card be installed. We advise against using AMD/ATI (Radeon) graphics cards, as their Linux drivers can cause problems, specifically with the AR Sandbox's water simulation.

We advise against using laptop computers to run AR Sandboxes, for the reasons listed above. If using a laptop is a requirement for an installation, ensure that the laptop does not contain Nvidia Optimus technology, or at least allows turning it off and granting access to the discrete graphics card via the computer's BIOS configuration.

With the recent renaissance of consumer virtual reality (VR), it is now easy to buy a pre-built PC matching the AR Sandbox's requirements. Many online electronics retailers now sell PCs under the "VR ready" label, such as Amazon.com and newegg.com.

The AR Sandbox has two main components: the topographic map renderer, and the water flow simulation. The former is comparatively easy on CPU and graphics card, and works on most current laptops or mid-range PCs. The water simulation, on the other hand, requires a high-end graphics card like the recommended GeForce GTX 1060. While the water simulation can be disabled to allow running the AR Sandbox from a lower-end computer, we do not recommend doing so.

The benefit of a dedicated PC is that the AR Sandbox becomes a computational appliance. Since the AR Sandbox software does not require a live Internet connection, it is possible to install the OS and AR Sandbox software, and then disconnect the computer from the Internet and never update the operating system or AR Sandbox sofware. The AR Sandbox software can be set up to start automatically when the computer boots, in which case it does not even require a monitor, mouse, or keyboard. Should newer versions of the AR Sandbox software become available, they can be uploaded via removable media.

Projector

Ideally, the projector should have a short throw length and a native 4:3 aspect ratio to match the field-of-view of the Kinect camera. The projector's native resolution is secondary; XGA (1024x768 pixels) is sufficient, as the sandbox overall's resolution is limited by the Kinect camera's 640x480 pixels. For practical reasons, short-throw projectors generally project above centerline, i.e., the bottom edge of the projected image appears above an imaginary horizontal plane through the projector's lens. The ideal projector for an AR Sandbox would be a centerline projector, so it could be mounted directly next to the Kinect camera. Since centerline short-throw projectors are rare and typically very expensive, the compromise is to mount an above-centerline projector above the sandbox's rear long edge, while the Kinect is mounted above the sandbox's center (see Figure 1).

We recommend the BenQ MX631ST above-centerline short-throw XGA DLP projector with 13000:1 contrast ratio and 3200 ANSI lumens, which generally sells for around $550. At maximum zoom, it matches the Kinect camera's field-of-view and can thus be mounted at the same height as the Kinect camera, simplifying setup.

We strongly recommend that the projector be connected to the PC's graphics card via a digital video connection, i.e., using an HDMI port on the projector, and an HDMI, DVI, or DisplayPort port on the graphics card. An analog connection, such as using a 15-pin VGA port on the projector, leads to degraded image quality and can cause misalignment between the projected image and the sand surface.

Sandbox

The sandbox itself should have a 4:3 aspect ratio, to match the fields-of-view of the Kinect camera and the projector. The size of the sandbox is limited by the Kinect camera's minimum and maximum sensing distances, and the desired sandbox resolution. Due to the Kinect camera's approximately 90° field-of-view, the Kinect camera has to be mounted about as high above the sand surface as the sandbox is wide. The Kinect camera should be mounted directly above the sandbox's center point, looking straight down (see Figure 1).

Figure 1: Typical arrangement of projector and Kinect camera above a 40"x30" (1mx0.75m) AR Sandbox. The short-throw projector is mounted at the same height as the Kinect camera, but above to the rear long edge of the sandbox to account for its above-axis projection.

We recommend a sandbox size of 40"x30" or 1mx0.75m, leading to a Kinect camera mounting height of approximately 40" or 1m. At this height, the camera's nominal horizontal resolution is 1.56mm, and its effective horizontal resolution is high enough to resolve features on the order of 5mm. Vertical resolution at the same height is 2.79mm. Increasing the size of the sandbox increases the required height of the camera/projector mount by the same factor, and not only reduces horizontal resolution, but also vertical resolution. In a 2mx1.5m sandbox, for example, nominal horizontal resolution is 3.12mm, and vertical resolution drops to 11.16mm (vertical resolution is roughly proportional to height squared).

Sand

The sandbox should be filled with sand to a depth of around 4" or 10cm. At 40"x30", this totals 2.77 cubic feet or 75dm3 or 75l of sand, weighing approximately 198lb or 98kg. We recommend Sandtastik White Play Sand, for between $15 and $25 per 25lb, or $120-$200 total. Sandtastik sand has excellent projection properties, but a shallow angle of repose when dry. We recommend keeping the sand slightly moist to make it moldable. Adding 1 cup or 0.25l of water to 198lb or 98kg of sand and mixing thoroughly is sufficient.

Health Concerns

Regular sand is basically crystalline silica, primarily in the form of quartz. While silica is non-toxic when ingested orally, inhaling very fine silica dust can cause adverse health effects.

The recommended Sandtastik play sand does not, according to the manufacturer, contain or release fine silica dust. Neither should alternatives such as Moon Sand or Kinetic Sand, as they are either not made from actual sand, mixed with a binding agent, or surface-treated with a polymer. Regular sand, such as bought in bulk from hardware or home improvement stores, should be washed before use to reduce the amount of fine dust particles contained in it. Here is a how-to guide on washing play sand (step 5, baking, is optional).

Hardware Construction

Building the sandbox itself, and mounting the Kinect camera and projector above it are left as exercises to the reader (see Figure 1 for a rough layout sketch). Our own prototype AR Sandbox is built from wood and some metal. The sandbox proper is plywood on top of a sturdy wheelbase (for mobility), slathered generously with polyurethane to make the sandbox waterproof and rot-resistant (We are using small amounts of water to make the sand moldable). The projector and camera head assembly is made from aluminum slats, and the entire assembly is suspended above the sandbox from a vertical steel pipe. The head assembly offers limited adjustment for camera and projector position and orientation to allow physical alignment between sand surface, camera field-of-view, and projected image. Most importantly, the projector should be oriented such that it projects on-axis onto the "ideal" level sand surface. This minimizes distortion effects and focus problems.

Software Installation

The basic process is to install the Vrui, Kinect, and SARndbox sofware packages, in that order, on top of a Linux or Mac OS X operating system. See the README file included with the AR Sandbox software package for details.

If you are a first-time Linux user, please follow these complete step-by-step software installation instructions.

Kinect Camera and Projector Alignment and Calibration

As mentioned above, the projector should be aligned physically such that it projects orthogonally or on-axis onto the "ideal" level sand surface, and that its projected image exactly fits the sandbox. This usually means that the projector has to be mounted vertically above the back long edge of the sandbox, due to short-throw projectors' usual above-axis projection. The Kinect camera should be aligned such that its field-of-view exactly matches the sandbox as well. This is most easily achieved by mounting the Kinect vertically above the center of the sandbox, orienting it straight down, and then fine-tuning its position and orientation while observing its depth image in the RawKinectViewer application.

Kinect Camera Intrinsic Calibration

Ideally, the Kinect camera should be calibrated intrinsically to capture a proper 1:1 representation of the sand surface. All Kinect cameras are pre-calibrated at the factory, but while that calibration is servicable, it is not very good. Most importantly, Kinect cameras benefit from per-pixel depth correction; without that, a Kinect will capture a completely flat surface as a bowl-like shape, which noticeably affects elevation contour lines and water flow. Intrinsic calibration is described in detail in the Kinect package's README file, and the following two videos:

  1. Intrinsic Kinect Camera Calibration with Semi-transparent Grid
  2. Intrinsic Kinect Camera Calibration Check
There is no video for the per-pixel depth correction calibration step, which needs to be done first, but it is fairly simple and explained in the README file.

For simplicity, intrinsic calibration should be performed before the Kinect is mounted above the sandbox.

Kinect Camera Extrinsic Calibration

Extrinsic calibration establishes the position and orientation of the Kinect with respect to the sandbox, and the mathematical plane equation of the "ideal" level sand surface and its position and size in 3D space. It is explained in the following two videos:

  1. AR Sandbox Calibration - Step 4
  2. AR Sandbox Calibration - Step 5

Unlike intrinsic calibration, extrinsic calibration has to be performed with the Kinect mounted above the sandbox, and needs to be repeated any time any changes are made to the physical sandbox layout, i.e., any changes to the position or orientation of the projector or camera or the sandbox itself. Also, depending on the sandbox assembly's mechanical stability, it might have to be redone on a regular basis (every month or so). Fortunately, the process only takes a few minutes.

Kinect Camera and Projector Calibration

The final calibration step is to measure the precise alignment of the Kinect camera's field of view and the projector's image. This is done using a calibration prop and a dedicated calibration utility, explained in detail in the following video: New AR Sandbox Calibration Procedure.

Like extrinsic calibration, this procedure has to be performed any time the AR Sandbox' physical layout changes in any way.

Running the Augmented Reality Sandbox

See README file included with the AR Sandbox software package.