My name is Nicolò Carpignoli, I'm a WebXR developer from Italy. I'm working for Chia lab, a design and software company based in Bologna. You can find more about me on Github at nicolocarpignoli, On twitter at nicolocarp, On medium at nicolcarpignoli. I'm here to make easier for you to start with Augmented Reality on the Web. From now on we will refer to it as 'WebAR'.
Let's start from definitions. What is AR? AR for me is a new communication medium. It's new for the market, not for the tech. It makes you see content over the reality. This can make you learn in different ways, see things differently, you can feel more engaged with the content itself, because it's situated in your environment. You can feel it closer to you and more real. Situated content can be also more accessible (for example, Augmented sign language books).
Some examples of AR use cases can be: "I’d like to see how this new furniture fits on my room, the colors, the size, etc... before buying it." "I’d like to learn biology, or anatomy: this 3D model I see, in real sizes, over my textbooks, is really helping me to understand sizes, to see every side of it, every detail, etc… and I can interact with it, it’s so cool and helpful!" "I want to explore my city, walking in the historic center and see new places I do not know. To see anchors on my phone camera, where places really are, with some indications and distance, is helping me to find them." "I’m abroad and I have text in front of me in a language I do not know, I don’t have time to type it all on a translator: I will use this AR app that translates that to me in real time. Also, it’s chinese and I’m italian, how do I even type it?"
AR comes in different types: markerless, image tracking, location based, marker based. We will see each of them later, in detail, talking about technologies.
Why are we talking about AR on the Web? Because now it's possible to use AR on every user's phone, and it makes sense, because this way, we can make AR content and apps collaborative, and build them easily. For example, to have a mobile apps instead of a web app, it would mean: For developers, to have High costs of development, build and release; To have a Macbook for iOS, developer licenses for both Android/iOS; To release a new app at every bugfix or feature.; For users, it means: To find an app on the stores; To install an app (storage issues); To keep it updated; To remember to delete it. So the AR future, in my opinion, relies on the Web.
Today we're going to talk about several, free and open source WebAR technologies. AR.js, WebXR Device API, jsartoolkit, model viewer and jeelizAR. Let's start from the first of them.
AR.js is written in Javascript and it features Marker Based, Image Tracking and Location Based AR. It's cross browser, working on almost every android and iphones of last 5-6 years. Marker Based means that when we scan a marker, so a simple image sorrounded by a black border, we can show content on top of it. Content will disappear if the camera lost the marker. Instead Image Tracking uses the same 'marker' behaviour but in this case we can use real pictures and not simple barcodes or qrcodes like images. So we can augmented drawing, pictures, art, and make them live, for example starting from a movie poster and animate it with a 2D video overlayed on it. Content can be 3D models, videos, images, even audio. It uses aframe, a framework based on HTML and Javascript, that makes easy to create content on the Web for AR and VR - VR stands for Virtual Reality. Instead, location based AR uses real-world coordinates to place AR content in context. Users can move freely (outdoors for better precision) and content associated with their location will be scaled and placed accordingly (e.g: content will render bigger / smaller based on distance to the user). It can be used to: Show Augmented Reality informations of places walking on a park, a street, etc., For Cities and museum tours, restaurant guides, treasure hunts, biology or history learning games or place virtual art on any real world location, to create GPS based games (Pokèmon GO-like).
You can even create markers inside QRCodes with custom qr code creators, in order to make the user, with a single scan, to reach your website thanks to the qrcode and to see AR content with the marker within it. AR.js is cross browser and features three important AR types, but still requires programming knowledge to be used, to create web ar with it and deploy it live on a website.
AR.js Studio instead is an authoring platform to create Web AR Experiences. It uses AR.js under the hood, but it's designed to develop AR without coding knowledge. You just choose your experience type, marker or location based, define anchors (for example places of interest if location based) and load your content, like 3D models, images or videos. You can then choose to publish it live on a website and it will output you a public URL, or you can download the zip archive and play with it. AR.js Studio is still not ready, me and all the AR.js organization are working to release a first demo in June/July 2020.
Now we're going to talk about WebXR Device API. The future of Augmented Reality is the Immersive Web Community Group and the WebXR Device API This new API is aiming to be a standard for both Web AR and Web VR technologies on browsers. Currently it works on latest Chrome and Firefox versions, with ARCore compatible devices. No rumors about Safari/iOS support. We’re still from have it cross-browser, but it’s time to give it a look. It features Markerless AR, hit-testing and soon DOM Overlay. With Markerless AR we mean to see 3D models around you without needing a marker or an image as anchor.
jsartoolkit is a Javascript port for the Web of ARToolkit, the first of all OSS project of WebAR around. jsartoolkit it's the base of AR.js, for example, and of more projects. It features the same AR types of AR.js but lacks of aframe support and location based. But it gives you a better control on detection and tracking API. It's suggested if you want to control tracking/detection better than with AR.js and you know JS/ComputerVision programming.
Instead, WebARStudio it's the authoring platform for jsartoolkit. It's different from the previously showed AR.js Studio. In this case, we have a more complex interface, but it makes it possible to create 3D complex scene. So you can do more, talking about 3D stuff and then publish them as WebAR. But again, it's a bit more complicated. It depends on your needs and your 3D design knowledge.
Now it's time to talk about Model Viewer. Model Viewer means WebXR (when supported) + Quick Look (on Apple) + Scene Viewer (on Android) + fallback experience. It Features: markerless AR, place/scale objects, take pictures of the scene. This is practically a layer on top of Google's and Apple's markerless solution for WebAr, although their solution is not a real WebAR, because it opens a file with a native mobile view. Anyway, it can still be thought of 'web' based, as files are served through the web. This is the solution used by Google on Google Search, introduced in recent times: try to search for an animal on Google, on the results you will find the possibility to see it on your room. That's Model Viewer. Unfortunately it's not very cross browser and it's limited, because it's not possible to modify the scene when the camera starts the AR scene. But it's possible to modify the web page that triggers it. Code to use this solution is very simple, although you will need 3D models with GLTF and USDZ formats (latest for Apple devices).
Last but not least, jeelizAR and JeelizFaceFilter. They are based Machine Learning and are featuring Markerless AR and Face Filters. These are two solutions based on neural networks. You have to use pre-trained neural networks or train one on your own. Then, when you have a neural network in order to recognize a specific kind of objects (like a cup, a table, etc.) you can use JeelizAR to track it real-time. It's very fast, faster than Tensorflow.js, authors said. It comes also with face-tracking and face-detection and this specific library is called JeelizFaceFilter. It's a great library and the only face filter AR around on the web for free. But it requires programming knowledge and also a bit of Machine Learning basics to use it.
I hope you enjoyed. These were the most used Web AR technologies on Open Source as of mid 2020. We have seen pros and cons, features and support on devices for each of them. Some of them can be used without programming knowledge, other requires a bit of Javascript and Web programming basics. You can learn more following me on Twitter at nicolocarp, on Medium at nicolcarpignoli or on these references:
Write me an email at nicolocarpignoli dot gmail dot com for collaborations, questions or just a chat! Bye!