At work, I am still working on finding and squashing fun test failures. In this case, “fun” means tests that have an intermittent failure rate of 5% (or less!). The test issue I worked on today had to do with the “lost update” problem.
The lost update problem
Growing Object-Oriented Software, Guided by Tests has a great description and diagram of the problem:
The short version is that when you poll a system for its state, it’s entirely possible to miss the state you are looking for. In the diagram, the color changes to red and then to blue before the test ever has a chance to see that it was red. Since this system will never go back to red, the test will incorrectly fail.
The lost update problem in Capybara
Like many other sites, we use the DataTables jQuery plugin to show tabular data. A test that ensured that the filtering worked looked something like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Enabling filtering triggers some slow backend activity, which brings up the loading dialog. The test waits for that dialog to appear and disappear before continuing on. Now the entire table is populated and we can safely see how many elements are in the table.
However, the test will fail if the backend is too fast. The loading dialog will appear and disappear almost immediately. The test will time out waiting for the loading dialog that will never appear again. This behavior can be reliably replicated by adding a sleep to the test between lines 13 and 14.
A Capybara solution
In order to make the test more robust, I rewrote it as:
1 2 3 4 5
The test now ignores the loading dialogs completely, instead asking Capybara to find a particular number of elements. Asking Capybara to find things in this manner will let the test leverage the built-in waiting facilities of Capybara.
In this test, the number of data items won’t change once the table is loaded, so it is a safe state to poll. As an additional benefit, the test now has fewer lines of code and is clearer.
As a downside, when the test fails, the Capybara error message doesn’t include how many items were found, which isn’t as informative as the equivalent message from the RSpec matcher.
Also, this test still ultimately relies on polling the DOM, so it’s possible for similar bugs to pop up in the future.
The GOOS solution to the lost update problem
GOOS provides a solution to the lost update problem that can avoid the problems with polling completely:
The system under test must be modified to provide notifications when something interesting happens. This system now has a listener that is notified when the color changes and what the color is changed to. The test supplies a simple listener that accumulates the changes and offers a nice API suited for the tests.
A hypothetical Capybara solution without polling
I can imagine a Capybara test that looks something like this:
1 2 3 4 5 6 7
running Selenium session that creates an event listener and binds it
to the given event. This listener just collects all the events it
receives. After yielding the block, the test code then polls the event
listener, waiting for the event to be captured.
This might be a useful thing for Jasmine tests, so it might already exist in that ecosystem.