This is the second post in four-part series we’ll be posting this week. Read part one and check back tomorrow for part three.
Beyond simple audio and video playback, there’s a lot we can do to manipulate the content to truly integrate it with the rest of the web. Back when we relied on Flash and QuickTime plugins, video acted as a black box, impenetrable and separate from the web page. But with the HTML video element, we can manipulate video with CSS just like any other HTML element, even 3D effects. CSS layout works well across all browsers, with the notable exception of the iPhone. As I mentioned in part one, video on the iPhone can only be played in full screen, not within the web page layout.
Although WebGL now has wide support, it remains broken for video processing in Internet Explorer and Safari, in both desktop and mobile versions. Microsoft has confirmed that it plans to fix this, and for now there is a workaround that draws the video to a 2D canvas, which is then passed to WebGL. It runs slower than drawing the video directly with WebGL, and it is very slow in mobile and desktop Safari.
Because WebGL gives direct access to the graphics hardware, there is a lot that can go wrong. Every device and platform has different capabilities, and there are various methods and extensions on the WebGL context that give information on these capabilities. WebGL Stats lists many of the varying capabilities and shows how widely different devices support them. But it’s important to test WebGL on as many devices as possible, as I often see GPU (Graphics Processing Unit) shaders failing to compile or presenting incorrect output due to poorly documented quirks, especially in Internet Explorer.
Audio can be loaded into the Web Audio API with XMLHttpRequest, but it’s best for very short clips and doesn’t allow for pausing and seeking. There is a way to pull audio data from an audio or video element, which is much more useful, but it’s broken in Safari and in Firefox when the source file is loaded from a cross-origin source. Web Audio API is available on all modern browsers, except for Internet Explorer, but Microsoft says it’s in development.
As with video playback, Web Audio won’t work in iOS unless you trigger it inside a touch event handler.
When directly manipulating audio, image or video data, we need to be concerned with cross-origin restrictions. For security reasons, the browser does not allow a script to modify any of these resources unless they come from the same server, on the same domain name and port. For example, a malicious site might try to reference an image on your bank’s server of a scanned check. If you’re logged in to the bank site, the malicious site would be able to show you the check image, assuming somehow it had the URL, but it would not be able to read the actual contents of that image because of cross-origin restrictions.
There is an exception to these rules that sometimes comes in handy, because sometimes we want to access media from another server. You might want to publish a video for people to remix, or more likely, you just want to serve a video with a content delivery network (CDN), which is a very good idea for performance. It requires setting the crossorigin attribute on the video (or img or audio) tag, and the server has to give permission to use the resource by setting the appropriate headers. Mozilla Hacks has an article on how to do this with images, and it works the same way for video.
Unfortunately, neither Safari nor Internet Explorer supports this exception for video, though they do support it for images. So our workaround for video textures is limited to videos on the same origin. I filed a bug report for Internet Explorer, but Microsoft has not yet responded. Firefox has the aforementioned cross-origin audio bug, which will hopefully be fixed soon.
Tomorrow in part three, I’ll discuss upcoming video features such as streaming, camera access and recording.