Testing is an important part of the software development process, helping to ensure that applications are working as they’re intended. But what happens when those tests aren’t trustworthy?
A “flaky test” is a test that sometimes passes and sometimes fails under the same exact circumstances, according to Trisha Gee, lead developer advocate at Gradle, in a recent episode of our podcast.
“The most annoying thing about this is you’re not sure if it’s the passing thing that’s correct or the failing thing that’s correct,” she said.
According to Gee, running a test multiple times can help identify if it’s a flaky test. A lot of build systems give you the option to rerun a test when it fails, so if you re-run a failed test and it passes the second or third time during the same build, then you’ll know it’s a flaky test.
Another way to identify flaky tests it to look at the same test across multiple builds, and if you determine it has the same inputs and outputs, including the same production code, test code, data and infrastructure, and it passes on one build and fails on another, it can be flagged as flaky.
Once it’s been determined that a test is flaky, it’s important to flag it as such so that if it fails in a future build, you know it wasn’t anything to do with you and the changes you made. This isn’t to say that tests should be ignored, as a flaky test is often telling you something, whether that be that the test is not well written or that the infrastructure doesn’t work under certain circumstances. “You really ought to take a look at why this particular condition sometimes is successful and and sometimes is not,” Gee said.
Gee believes that identified flaky tests shouldn’t be ignored because you don’t “really want yourself or the team thinking some tests failed, it’s probably fine. I mean, if tests failed, it’s probably not fine. You want to get used to the idea that a failure is a failure that needs to be looked at.”
There is also the worry that flaky tests erode confidence in your test suite. “As soon as you’ve got a test which sometimes passes and sometimes fails, you’re a bit like, ‘well, tests are just a bit dodgy, and we just don’t trust them anymore.’ And that’s not what you want. Get them out of the way, get that confusion out of your life,” she said.
To avoid flaky tests from entering your test suite in the first place, one piece of advice would be to look at all of your integration and end to end tests and determine if they really need to be those types of tests, or if they could be turned into multiple unit tests. There’s a temptation when dealing with complex systems to want to test a lot all at once, but that setup makes tests quite prone to flakiness.
“There’s a lot of misunderstanding around unit tests,” Gee said, “A unit test doesn’t have to be a single test; a unit test can be a sociable unit test, so you have a whole bunch of classes interacting with each other, but what they can’t be is they can’t be tests that include databases or external APIs or other modules. A unit test should be a single, encapsulated unit and they run quickly and they run reliably.”
Another thing that may cause flakiness is having tests that rely on date and time. “I spent a lot of time coding when I lived in London, and in London for six months, the year is in the UTC time zone. So we don’t have to worry about plus one, plus six, plus 10. Every single year when we had summer time, half the tests would fail that had dates because we’d forgotten to take into account time zones.” Because of this, Gee recommends subbing out the system clock in tests.
Encapsulation can also be helpful so that multiple tests aren’t all relying on the same database with the same data and trampling over each other.
Being able to consistently set up the right data and environment, and mocking and stubbing for external dependencies, can help testers ensure they can control exactly what’s happening with their tests.
The post The best ways to deal with flaky tests appeared first on SD Times.
Source: Read MoreÂ
