ARToolKit for Unity's C# scripts communicate with this DLL by use of P/Invoke. This is the same method used on all other platforms supported by ARToolKit for Unity. Structure of an ARToolKit for Unity Windows Phone 8.1 and Windows Store 8.1 project. ARToolKit is a set of augmented reality software tools that can be used in AR apps. Its main benefit is an open-source code that implies free access to the library. ARToolKit supports: 2D recognition; mapping additional elements via OpenGL.
This article discusses APIs that are not yet fully standardizedand still in flux. Be cautious when using experimental APIs in your own projects.
Introduction
Artoolkit Tutorial
This article is about using the JSARToolKit library with the WebRTC getUserMedia API to do augmented reality applications on the web. For rendering, I'm using WebGL due to the increased performance it offers. The end result of this article is a demo application that puts a 3D model on top of an augmented reality marker in webcam video.
JSARToolKit is an augmented reality library for JavaScript. It's an open source library released under the GPL and a direct port of the Flash FLARToolKit that I made for the Mozilla Remixing Reality demo. FLARToolKit itself is port of the Java NyARToolKit, which is a port of the C ARToolKit. Long way, but here we are.
JSARToolKit operates on canvas elements. As it needs to read the image off the canvas, the image needs to come from the same origin as the page or use CORS to get around same-origin policy. In a nutshell, set the crossOrigin-property on the image or video element you want to use as a texture to ' or 'anonymous'.
When you pass a canvas to JSARToolKit for analysis, JSARToolKit returns a list of AR markers found in the image and the corresponding transformation matrices. To draw a 3D object on top of a marker, you pass the transformation matrix to whatever 3D rendering library you're using so that your object is transformed using the matrix. Projectlibre change language. Then, draw the video frame in your WebGL scene and draw the object on top of that and you're good to go.
To analyze video using the JSARToolKit, draw the video on a canvas, then pass the canvas to JSARToolKit. Do this for every frame and you've got video AR tracking. JSARToolKit is fast enough on modern JavaScript engines to do this in realtime even on 640x480 video frames. However, the larger the video frame, the longer it takes to process. A good video frame size is 320x240, but if you expect to use small markers or multiple markers, 640x480 is preferable.
Demo
To view the webcam demo, you need to have WebRTC enabled in your browser (on Chrome, go the about:flags and enable MediaStream). You also need to print out the AR marker below. You can also try opening the marker image on your phone or tablet and showing it to the webcam.
Here is a demo of what we're aiming for. This demo creates an image slideshow using AR markers. Show a marker to the camera and it will place a photo on it. Move the marker out of the camera's view and show it again and the image has changed.
Setting up JSARToolKit
The JSARToolKit API is quite Java-like, so you'll have to do some contortions to use it. The basic idea is that you have a detector object that operates on a raster object. Between the detector and the raster is a camera parameter object that transforms raster coordinates to camera coordinates. To get the detected markers from the detector, you iterate over them and copy their transformation matrices over to your code.
The first step is to create the raster object, the camera parameter object and the detector object.
Using getUserMedia to access the webcam
Next, I'm going to create a video element that's getting webcam video through the WebRTC APIs. For pre-recorded videos, just set the source attribute of the video to the video URL. If you're doing marker detection from still images, you can use a image element in much the same way.
As WebRTC and getUserMedia are still new emerging technologies, you need to feature detect them. For more details, check out Eric Bidelman's article on Capturing Audio & Video in HTML5.
Detecting markers
Once we have the detector running a-ok, we can start feeding it images to detect AR matrices. First draw the image onto the raster object canvas, then run the detector on the raster object. The detector returns the number of markers found in the image.
The last step is to iterate through the detected markers and get their transformation matrices. You use the transformation matrices for putting 3D objects on top of the markers.
Matrix mapping
Here's the code to copy JSARToolKit matrices over to glMatrix matrices (which are 16-element FloatArrays with the translation column in the last four elements). It works by magic (read: I don't know how ARToolKit matrices are setup. Inverted Y-axis is my guess.) Anyway, this bit of sign-reversing voodoo makes a JSARToolKit matrix work the same as a glMatrix.
To use the library with another library, such as Three.js, you need to write a function that converts the ARToolKit matrices to the library's matrix format. You also need to hook into the FLARParam.copyCameraMatrix method. The copyCameraMatrix method writes the FLARParam perspective matrix into a glMatrix-style matrix.
Three.js integration
Three.js is a popular JavaScript 3D engine. I'm going to go through how to use JSARToolKit output in Three.js. You need three things: a full screen quad with the video image drawn onto it, a camera with the FLARParam perspective matrix and an object with marker matrix as its transform. I'll walk you through the integration in the code below.
Here's a link to the Three.js demo in action. It has debugging output enabled, so you can see some internal workings of the JSARToolKit library.
Summary
Artoolkit
In this article we went through the basics of JSARToolKit. Now you are ready to build your own webcam-using augmented reality applications with JavaScript.
Integrating JSARToolKit with Three.js is a bit of a hassle, but it is certainly possible. I'm not 100% certain if I'm doing it right in my demo, so please let me know if you know of a better way of achieving the integration. Patches are welcome :)
References
Artoolkit Tutorial
Welcome to the ARToolKit community!
ARToolKit is software that lets programmers easily develop Augmented Reality applications. Augmented Reality (AR) is the embedding of computer generated content into the natural environment, and has many potential applications in entertainment, media, advertising, industry, and academic research.
The source code for this project is hosted on GitHub and the compiled SDKs for all platforms (Mac OS X, PC, Linux, Android, iOS), along with the ARToolKit plug-in for Unity3D, are available at our Downloads page.
Each of the individual platform downloads include a README file and example applications to help you get started immediately. The documentation pages here include: detailed APIs, tutorials, and examples from novice to expert levels. Further support can be found by joining the ARToolKit Community Forum.
Features
- Robust Tracking, including Natural Feature Tracking
- Strong Camera Calibration Support
- Simultaneous tracking and Stereo Camera Support
- Multiple Languages Supported
- Optimized for Mobile Devices
- Full Unity3D and OpenSceneGraph Support
One of the most difficult parts of developing an augmented reality application is precisely calculating the user’s viewpoint in real time so that the virtual images are exactly aligned with real world objects. ARToolKit uses computer vision techniques to calculate the real camera position and orientation relative to square shapes or flat textured surfaces, allowing the programmer to overlay virtual objects. ARToolKit currently supports classical square marker, 2D barcode, multimarker, and natural feature tracking. Furthermore, ARToolKit supports any combination of the above together. The fast, precise tracking provided by ARToolKit has enabled the rapid development of thousands of new and interesting AR applications.
The documentation contains a complete description of the ARToolKit library, how to install it, and how to use its functionality in AR applications. Several simple sample applications are provided with ARToolKit to enable the programmer to get started immediately. ARToolKit includes the tracking libraries and complete source code for these libraries enabling programming to port the code to a variety of platforms or customize it for their own applications.
ARToolKit is multi-platform, running on the Windows, Mac OS X, Linux, iOS and Android operating systems. The functionality of each version of the toolkit is the same, but the performance may vary depending on the different hardware configurations. ARToolKit can be easily ported to other new and experimental platforms.
ARToolKit supports both video and optical see-through augmented reality. Video see-through AR is where virtual images are overlaid on live video of the real world. The alternative is optical see-through augmented reality, where computer graphics are overlaid directly on a view of the real world. Optical see-through augmented reality typically requires a see-through head mounted display and has more complicated camera calibration and registration requirements.
Tutorials and Examples
Quick Links
Getting Started
Marker Types and Usage
Advanced Topics
General Topics
Related Documentation (Off-Site)
- Inside ARToolKit Tutorial given at ART 02 by Hirokazu Kato
- Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System by Hirokazu Kato and Mark Billinghurst.
- A Registration Method based on Texture Tracking using ARToolKit by Hirokazu Kato, Keihachiro Tachibana, Mark Billinghurst, and Michael Grafe.