Web Platform Team Blog

Adobe

decor

Making the web awesome

Test the Web Forward Seattle!

Test the Web Forward Seattle!

Come Join Us!

Test the Web Forward is throwing an event in Seattle on April 12–13. The focus of this event is on learning, hacking, and writing tests for W3C specifications. During the event, experts will teach you about W3C specs & W3C testing and will guide you as you help make the Web a better, more interoperable place.

The Experts and Speakers will be posted shortly on the Test the Web Forward website.

Also, follow us on @testthewebfwd for updates and be sure to sign up for the W3C Test the Web Forward mail list.

To learn more about the past events, check out our posts on Test the Web Forward ParisBeijing and San Francisco.

Get Registered

Registration is now open on Eventbrite, so get signed up! Help make the Web a better place!

Can’t make it?

Hey, if you are busy or can’t make it for any reason, it is a bummer, but be sure to sign up for the W3C Test the Web Forward mail list so that you learn about future happenings!

Regions feature support matrix revisited – keeping it clean

Welcome to the Real World™

A while ago, I wrote a blog post on how we used Browserscope to create a feature support matrix for tracking the level of support for CSS Regions in different browsers. The initial plan was to have both the feature detection tests and the submission mechanism available to the public so that anyone could run the tests and submit their results.

Fast-forward a couple of months and the initial plan didn’t seem to hold very well to the rigors of the real life Internet: submissions for irrelevant browsers (like the one running on Nokia 72), false negatives on supported browsers (as the feature is under a runtime flag), or plain empty results as people used our test key for personal experimentation. In this context it became apparent we needed to revise both the process and the data we were exposing.

Fixing the process

Revising the process was fairly easy. We just moved the code responsible for the results submission on an internal server, while leaving all the other bits for running the tests and viewing the aggregated results in place.

The tricky part was cleaning up the existing data without re-running the tests on the whole list of browsers – some of which already unavailable due to the fast-paced release cycle of modern browsers.

Fixing the data

The reasons we initially chose Browserscope for the feature support matrix was that it did almost all the heavy lifting for us: user-agent parsing, results storage and aggregation, all through a very simple API. The downside however, was that one has almost no control over the data once it got inside Browserscope. The only exposed operations are reordering or removing test categories (basically, columns from the results table).

Browserscope’s way of keeping the results relevant – in spite of the unavoidable mis-reports – resides in the large number of results submitted, as the suites are usually public. For abuse protection they’re capping the number of results pushed from any given IP address and protection against outliers is provided by the statistical processing of the results submitted.

In our case, given that WebKit nightly is by no means a mainstream browser and Chrome has the regions implementation under a run-time flag meant that we were more likely to get inaccurate results as more people run the tests. So besides “hiding” the submit code from the general public, we also had to clean-up the results ourselves.

The main challenge was removing irrelevant browsers from the results list (basically, rows from the results table) – there’s simply no nice way to trick the system. You could hack your way around it by manually listing the user agent strings you want to see/aggregate over. But that involves updating the list every time you land some results with a new user agent. Ugh!

Delete as selective copy

Instead of actually deleting results, we resorted to creating a new results table – one that would contain only the relevant results/browsers. Once the copy would be complete, we would start using that table further on.

However, what started as an “I know, I’ll use Python” moment, turned to bitter reconsideration once I realised that:

  1. currently there’s no way to retrieve all the test results in a machine-friendly format, like CSV;
  2. Browserscope relies solely on the User-agent HTTP header to aggregate results, so I had to use some API that would allow me to fiddle with that header, too;
  3. and last, but not least: the “API” for submitting the results relies on executing some JavaScript code from the Browserscope server that has, among other things, some CSRF protection magic. So I actually had to execute that code in a real JavaScript environment – which actually limited my options quite a bit.

Most of the things above were things that I knew in a corner of my mind, but only became apparent at a closer inspection of the problem.
What I needed was basically an environment where I could

  1. retrieve some HTML, scraping it for useful data
  2. filter said data and then
  3. execute some JavaScript in a browser-like environment whose user-agent string I could change at will.

After a short initial panic moment :) I realized a platform that would allow me to do all those things actually exists, and I’m actually familiar with it, being an Adobe-built technology. This technology is Adobe AIR. One of the cool things about AIR is that it embeds a WebKit port, allowing developers to write desktop apps using just HTML/CSS and JavaScript.

HTMLLoader to the rescue

Browserscope submitter

Armed with this new-found hope, I created a helper app (pictured above) that does just a little more than the three steps I previously listed as requirements for the solution. More specifically:

  • it takes as input: the key of the source results table, the maximum number of results to fetch, the key of the destination results table and the sandbox ID for the destination table
    • the sandbox ID is required to bypass the per-IP cap on results submissions, since most probably we’re gonna submit a lot of results in a short period of time from the same IP
  • based on the key of the source table it fetches its raw (not aggregated) contents, nicely formatted as HTML (something like this). This is done via an AJAX call and the response is parsed using jQuery, extracting the list of tests and the individual test results
  • based on the list of tests and the individual results previously extracted, an interactive table is displayed that allows excluding irrelevant results
  • the non-excluded results are submitted to Browserscope using the AIR-specific air.HTMLLoader object
    • for each result, the userAgent property of an HTMLLoader object is set to the user-agent string reported by the browser that submitted the result and then, using the loadString() method, a small piece of JavaScript is injected and executed that does the actual result submission
    • results are submitted one-by-one, asynchronously, using a pool of HTMLLoader instances

Wrapping it up

Now that the cleanup is completed, you can see what CSS Regions features are supported in which browsers and run the tests yourself, over on our gitHub page, just as before.

If there’s one important thing this clean-up/refactor taught me, is that when you know you’re (ab)using a technology, you’d better be prepared for the moment you’ll hit the hard wall of limitations you knew existed from the very beginning. And also, as long as it gets the job done, no technology should be discarded just because it doesn’t seem appropriate. Oh, and most of the times, the path you already know is usually the shortest.

W3Conf 2013: A Summary

The Adobe Web Platform team helped in organizing the W3Conf at the beautiful Regency Center in San Francisco on 21-22nd Feb 2013. Here are some notes from my perch behind the curtains (I was introducing the speakers!).

By all standards, the conference was a great success. Here are some metrics:

  1. We had about 20,000 viewers watching the conference live in total (there were on an average about 300 viewers watching every second of it).
  2. About 1,500 people tweeted about the conference. Notably, Faruk Ateş and Kevin Marks were live-tweeting the event.
  3. 300 people attended the event (including Tim Berners-Lee) in total. Chris Coyier live-blogged day 2 of the event.

What was talked about?

Speakers such as Joshua Davis, Eric Meyer, Lea Verou, Brad Hill, Vicki Murley, Leoni Watson (and more!), gave excellent talks on various aspects of Open Web technologies, including CSS, SVG, Accessibility, Security, and Performance. All the talks are archived on the W3Conf Youtube Channel.

We also had Alexis, Alex, and CJ from our team speaking about various aspects of the web:

Opportunities to Contribute

Larry McLister & Rebecca Hauck from our team were at the event speaking to the participants about Test the Web Forward and Webplatform.org efforts. We had about 100 people registering their interest in participating in Test the Web Forward events (you can too, just subscribe to this mailing list)

Thanks

It would be insane not to thank the valiant and super human efforts of Susan Parini, Marie Nedich, and Liz Arroyove-Federick who took care of all the logistics and the W3C for letting us be the host of this year’s event.

Wait, there is more!

We also had 2 other events that occurred just after the W3Conf: A workshop by Joshua Davis on using SVG with Processing for creative coding, and a Docsprint to add more exhaustive references to webplatform.org.

You can work through the files from Joshua Davis’s workshop.

Join us

We are participating in many more events, and we would love to have you join us at more events.

“The Making of CSSFilterLab” at W3Conf 2013

Link to the CSS FilterLab Presentation - W3Conf 2013

I had the honor to speak at the W3Conf 2013 last week. My talk was about a fork of CSS FilterLab that I’ve been working on recently. The project is written entirely in JavaScript, CSS and HTML and targets mobile devices with touch based interfaces. In order to work with all the touch API, I had to implement a framework that I’ve later reused to create the slides. Yes, the slides are also using HTML5 and the presentation was given on a tablet device. I’ve also covered the performance problems I had while porting to mobile and the workarounds I used in CSS FilterLab.

The talk was recorded and is embedded below. The source code for the slides are available on my GitHub account and you can also browse the slides online. Make sure you use left/right on your keyboard to navigate or swipe to advance to the next slide.

“Do Androids Read Electric Books?” at W3Conf 2013

Screen-Shot-2013-03-05-at-1.27.58-PM

I was fortunate to have the opportunity to speak at W3Conf in San Francisco this year. My talk was about reading experiences on the web, ways we can improve them and where we see this going in the future. In the talk I show off how we can use Regions for visually stunning magazine layout and responsive pagination. There are also some examples of Exclusions being used in creative designs for web based children’s books/games or responsive comic books. You can view the video of my talk below and find the slides on github along with the source. Be sure to check out all the other amazing speakers from the conference as well.

“The top web features from caniuse.com you can use today”

First slide of W3Conf slideshowAt the 2013 W3Conf I gave a talk about the browser compatibility site Caniuse.com. The talk consisted of two parts, starting with an explanation of how caniuse.com started, how it’s useful to developers, and what’s to come on the site. The next and larger part discusses top web features now supported by enough browsers that you can safely start using them without worry or guilt. Features include CSS Tables, Pseudo-element & generated content, CSS counters, CSS outline, sessionStorage & localStorage, hashChange event, text-overflow, and Data URIs. Lesser supported but related features are also addressed.

The talk was recorded (together with all other W3Conf talks) and can be viewed here:

“Magazine-like layouts with CSSRegions” and the WebVisions NewYork show

At the end of February, i attended the WebVisions conference, hosted in the “Theater for the New City”, New York. The audience consisted of half-developers half-designers and Adobe, a key sponsor for the event, had two speakers: Kevin Hoyt and myself.


While Kevin had a presentation about PhoneGap and how it can be used to build mobile applications that are standards compliant, I spoke about CSS Regions: what are they, what are the problems they are trying to solve and what are the solutions proposed by regions. I also talked about how a developer can combine regions with media queries to create adaptive layouts and how regions can be used with the new CSS3 layout modules like multicolumn, flexbox and grid to achieve maximum layout flexibility. Last but not the least, I showed how regions could be combined with CSS Exclusions to create more compelling layouts on the web.

People seemed to enjoy the presentation and like the power and flexibility brought by CSSRegions. The predominant question was about the availability of CSS Regions in browsers, therefore it becomes important to remove regions from the experimental flag in Chrome this year. CSS Regions polyfill, a JS library that allows developers to experiment with basic CSS Regions support in non-WebKit based browsers like Firefox, was welcomed and some people wanted to experiment with a CSS Exclusions polyfill as well. While discussing CSS Exclusions, people loved the idea of defining an exclusion based on the alpha channel of an image.

The strongest argument favoring regions over a solution that was image-based was that a smaller content payload matters, since it is unpleasant for the user to download 100 MB of images for a magazine issue, sometimes on a mobile connection.

Some highlights from the conference:
* The final keynote was delivered by Ethan and Malachai Nicolle, the creators of AxeCop. It was an impressive talk in which they shared how they created AxeCop and how they evolved from some nice comics to a site with very high traffic and further to a show on Fox channel. Malachai is an 8-years old child :) with lots of ideas illustrated by his brother Ethan.
* Douglas Rushkoff had an interesting keynote about how the society has changed its focus from future to present (I can’t wait to read its last book “Present shock: when everything happens now”) and Andy Bichlbaum showed some very funny movies related to the activity of the group he is leading, “The Yes Men”.
* Thomas Phinney, the fontexpert, ex-Adobe now WebInk, spoke, of course about fonts and how to wisely use them on the web. His examples of sites and movies included both good and not-so-good use of fonts.
* I enjoyed watching Rachel Nabors presentation about how to play with animations in CSS3 and the presentation from Shay Howe, a professional creative designer, about writing performant code with HTML/CSS.
* Jason Cranford Teague had a workshop in the first day about responsive design and Sara Wachter-Boettcher explained why we need better, not more, content and showed some techniques to achieve that during her workshop.

It was a great event and I am really glad I had the opportunity to speak about the work we are doing in Adobe’s WebPlatform group.

Links of Interest

Here at Adobe we’ve started collecting links being shot around in the office and compiled a list of some of the interesting ones. There’s a lot of exciting news happening around the open web platform recently so be sure to check out these weekly link digests:

  • Haxe 3.0 RC - now available with improved JS and C++ transpiling and better macros
  • Early benchmarks with ASM.JS
  • FAMO.US – This product highly improves animation performance on mobile devices by computing matrix transforms natively and mapping the results back into post-calculated CSS matrix transforms. This allows super-fast DOM elements without needing the html5 canvas
  • Bug Report – Webkit core has been identified with having very poor performance with passing data to and from worker threads
  • Ejecta - A fast, open Source JavaScript, canvas & audio implementation for iOS
  • BECK360 - An amazing demo that pushes the limits: audio and video recorded in 360 degrees and playback with real-time webcam head tracking for the experience

Are we missing any noteworthy links from this week? Feel free to post them in the comments!

How I Came To Care About Ellipsis

The Adobe Web Platform Team has some features and specifications that we focus on implementing in WebKit, but we also like to contribute to the quality of the project as a whole. To this end, we often work on bug fixes for things that are not related to one of our features. Not only is this a good way to contribute to the project in a more general way, it also is an excellent way to get broader experience with the code.

As part of these efforts, I recently fixed Bug 101879. This bug was filed by a user who discovered that WebKit didn’t correctly implement the CSS 3 UI text-overflow property when a block contained a mixture of inline content and block content. Specifically, it fails to render one of the examples from the specification properly:

Incorrect rendering of spec example

The correct rendering is as follows:

Correct rendering of example

If you want to know how your browser fares, you can go to the text-overflow section of the specification and scroll down a little to find the table with this example and others.

It turns out this happens because a block element can only contain either all inline content or all block content, so when the block contains both inline and block content, WebKit creates anonymous blocks to contain the inline content. (Anonymous block creation is explained in Section 9.2.1.1 Anonymous block boxes of the CSS 2.1 spec) Since the value of the text-overflow property is not inherited by child blocks by default, it was getting lost when laying out the text in the anonymous blocks. I changed the code to check for a block parent that has text-overflow set when laying out the content in an anonymous block, and use that value to determine if the inline content overflow in the anonymous block should get an ellipsis or not.

So that’s the short story of this one bug. If you’ve been having some issues with ellipsis in WebKit, this is good news for you. If you have some other bug that’s been bothering you, perhaps you should take a look at the code: you might find it easier than you expect to get started and fix a bug. If you think that sounds interesting, you should check out my post on Getting Started with the WebKit Layout Code.

CSS Fragmentation In WebKit

What is fragmentation?

The CSS 2.1 specification defines a box model to represent the layout of a document and pretty much everything is a box. Normal flow nodes (e.g. not absolutely positioned) are laid out child by child starting at the top of their parent element box. If an element’s box is too small to fit all the content, the content is said to overflow and this overflow can either be visible or get clipped.

Fragmentation is different from overflow because it allows flowing the content through multiple boxes called fragmentation containers – fragmentainers for short. When the end of the current fragmentainer is reached a break occurs and the layout continues within the next fragmentainer. Using CSS, authors can also force breaks to occur after or before an element and even avoid them altogether.

The important detail to remember is that fragmentation doesn’t mean taking the overflow of a box and visually moving it to another one. Fragmentation happens during the layout and affects the dimensions of the content boxes.

There are a few specifications in CSS based on fragmentation:

  • CSS3 Pagination - the document is laid out so it can fit on pages; the pages act as fragmentainers.
  • CSS3 Multi-column - the element defines a number of columns that fill its box and where the content is laid out; the columns act as fragmentainers.
  • CSS3 Regions - selected content forms a flow that’s laid out inside boxes called regions; the regions act as fragmentainers.

All of these specifications share common concepts covered by the CSS3 Fragmentation specification.

Paragraph in regions.

Example of a paragraph flown in two regions.

Fragmentation in WebKit

The Layout Process

To fully understand the concepts presented here you should know a bit about how layout works in WebKit. There are some nice articles on the Web covering the basics such as Bem’s article.

Long story short, the DOM tree is internally mapped to a render tree that is used to build up the box properties for every node. The render tree is made up from objects like RenderBox, RenderBlock, RenderInline etc. that represent the concepts defined in CSS2.1. During layout this tree is traversed and the various geometrical properties of the renderers are computed based on the style values. In the end all the elements have their box information computed.

As a side note, you’ll notice the symbols inside WebKit are usually named using the “pagination” or “page” terminology. This is for historical reasons, but most of the time the pagination concepts map to the fragmentation ones – pages, columns or regions.

There were a couple of ways the fragmentation behavior could have been implemented in WebKit. One of them was to have a renderer for each element fragment. For instance a simple paragraph with ten lines where there’s a break after the fourth one, we would have had one renderer with four lines in the first fragmentainer and a second renderer with six lines in the second fragmentainer. Both of the renderers would belong to the initial element. This approach is difficult to implement correctly and maintain because of the complexity it brings to the codebase. Additionally, it is also very risky from a security standpoint as it can introduce many subtle memory management bugs.

A rule of maximum one renderer per DOM node was created because of these concerns. Fragmentation is implemented by shifting the monolithic boxes (boxes that can’t be fragmented, such as line boxes – rectangles wrapping each line of text) so that they don’t overlap with the breaks. The correct rendering is obtained during the painting phase by placing each fragment exactly where it is supposed to appear in the fragmentainer. More about this topic in the next section.

In the case of unforced breaks, a position adjustment, called pagination strut in the codebase, is attached to the boxes during layout. This value represents a shift offset from the default layout position to the next fragmentainer in case the box doesn’t fit the current one. This offset needs to be stored separately on each box, not as a part of the top position because it’s not an attribute of the renderer. It’s a layout artifice that helps measuring the fragmented boxes. For example, if a block is collapsing at the top margin with its container (e.g. it’s the first child) and it also has a pagination strut, that offset will be transferred to the container. This makes sense because both the block and the container need to be placed in the next fragmentainer even though it’s the child that won’t fit.

The same rule applies to the first line box of a block. Also, during subsequent layouts it’s possible for an element to shift in the block-flow direction. The line boxes pagination struts need to be recomputed because the shift offsets are most likely different after the element changed position.

When a line doesn't fit the current fragmentainer it is shifted using the pagination strut.

When a line doesn’t fit the current fragmentainer it is shifted using the pagination strut.

As an example, in the case of the example above, the first four line boxes would be positioned normally in the first fragmentainer. The fifth line box may need to have a pagination strut defined if it fits only partially in the first fragmentainer. This pagination strut would logically shift the line inside the next fragmentainer. The rest of the lines are all placed normally in the second fragmentainer.

In the case of forced breaks, the pagination strut is no longer used because the authors directly specify where the breaks occur. The boxes are positioned inside their container so they respect the break condition.

Each type of fragmentation layout has its own specialized behavior. The multi-column elements have the ColumnInfo object attached to them. It contains information about the number and the width of the columns, the distance between breaks (used to balance the columns if they have auto-height) etc. This object is also pushed on the layout state stack so it can be accessed from any point inside the render tree.

The regions implementation is currently the most complex type of layout that makes use of fragmentation. The content node renderers are not attached to their DOM parent renderer. Instead, they are moved to a special object, RenderFlowThread, that sets up a pagination context when it is laid out. These renderers need to hold and use various information about the flow thread: the regions size, how the descendant boxes change width in every region they are flowing into etc. Because the concept of the flow thread is so generic there are plans to port the multi-column implementation on top of it.

The Layout Performance

From a performance perspective, the layout process will always be slower for fragmented content because the engine can’t apply the same optimizations that are made for continuous vertical content. For example, without the possibility of fragmentation, updating the top margin of an element can shift it on the vertical axis but no relayout may be needed. If the element is enclosed in a fragmentation context, by shifting the element it’s possible it won’t fit any more inside the fragmentainer, triggering a break. To cover this case, a relayout of the element is always required.

The layout engine is usually optimized for both speed and memory consumption. This means fragmentation code is used only when necessary: if there is a printing context, if the node being laid out is a part of a multi-column element or if the renderer belongs to a flow thread. The logic for enabling the fragmentation code can be found inside the LayoutState class. During layout, a stack of LayoutState objects is created and stored on the root of the render tree, the RenderView. Any renderer can query the top of this stack to determine if the content can be fragmented or not.

The Painting Phase

The painting of the renderers in WebKit is handled by the RenderLayer tree. For some renderers the engine creates layers that are used to paint the document in the correct order (e.g. with respect the z-index restrictions). Layers are always created for the renderers that fragment, such as multi-column blocks or RenderFlowThread objects. Because of this, multi-column elements and regions create stacking contexts and are painted as a single item.

For regions, when the painting operation occurs, the engine takes into account the fragmentation properties of the renderer and shifts the layer to the correct position so the painting is always executed inside the correct fragmentainer. The content not belonging to the current fragmentainer is clipped so the engine always paints only the content that should be displayed in a certain area. The mechanism is in some ways similar to how sprites (e.g. CSS Sprites) work.

The RenderFlowThread is painted in the two regions at the offsets corresponding to each content fragment.

The RenderFlowThread is painted in the two regions at the offsets corresponding to each content fragment.

For example, let’s say the paragraph above is flown into regions A and B. When region A is painted, the flow thread layer is called to paint in the fragmentainer box. It will paint the first four lines and then blank space until the bottom of the fragmentainer because the fifth line was shifted at layout-time using the pagination strut. When region B is painted, the flow thread layer is called again to paint inside the fragmentainer box, but at a new offset, where the region B fragment starts (top of the fifth line). The last six lines of the paragraph are painted.

Something similar happens with the multi-column blocks. However, the main difference is the fragmentainers for multi-column blocks don’t have their own layers. By using the same layer as the multi-column block the content is able to use advanced graphics features such as 3D transforms and video tags (see WebKit Compositing for more details about how accelerated layers work).

In the case of a flow thread, making its layer work with the region layers is one of the major challenges that need to be solved. As a consequence of this problem, content layers that require accelerated compositing (e.g. video layers) will not work correctly inside flow threads.

Conclusions

The modern fragmentation concepts are not yet fully implemented inside WebKit. There are some issues with the handling of forced breaks and the avoid value is implemented only for the break-inside property. The widows and orphans properties were just recently fully enabled.

On the CSS Regions side there’s some work left to do to achieve a smooth integration with the layers and compositing subsystem. Testing is also much needed to ensure good integration with other Web features.