Balancing Tests in Software Development
The Dichotomy of Correctness and Usefulness
Bottom Line Up Front:
Managers: Most software tasks aren’t simple. Do you have a strategy for UX? Your team is delivering UX whether you have a strategy for it or not.
Senior Developers: Consider being explicit about the factors you use in balancing tests. If you cannot verbalize the factors, it may be a valuable exercise to take the time to do so.
Junior Developers: Look for the balancing factors that senior developers use. If you don’t see them, ask for them.
“The Bug, Assigned”
A junior developer on a previous team was assigned a bug “There are Too Many Red Boxes Popping Up on the Screen”. The developer was diligent and found the service that was returning HTTP 500 status responses to the UI. The developer was diligent and found the code in the service responsible for this particular 500 error. The developer was very diligent in doing technical discovery associated to the bug. They had a fix in mind and were ready to approach the owner of the shared codebase to remove the 500 error.
Does this seem reasonable?
Should the shared codebase stop returning errors and always return an empty success?
What kind of bug is this?
“The Bug, Discovered”
When a developer is assigned a bug or other task, they are not given a roadmap to a fix. The developer must build context for the bug and for the fix. It’s not a trip to the home improvement store, it’s a trip into the jungle. They hack and slash their way to a point of some understanding and then come back out of the jungle with hard won knowledge.
The developer has to learn what the UI stack is doing, then what the backend service stack is doing. It is difficult to navigate all the different technologies, especially when you first start out. Once they have cut the path and know how the information flows, they pick a spot to divert the data stream.
As Ian Malcolm famously said “What's so great about discovery? It's a violent, penetrative act, that scars what it explores. What you call discovery, I call the rape of the natural world.”
“The Bug, Observed”
Users hate error pop ups. Most developers will read “Errors should be actionable” within a couple years of joining the profession. So why do apps still pop up errors? Because end users are pretty good at monitoring our apps.
“Surely you monitor your app with a real monitoring tool!?” - I hear you gasp at your monitor, in the same tone as “Surely you aren’t testing in production!?”
Look at most recent outages and you will see complaints of “green status bubbles” while the app is down. EVERYONE uses end users for monitoring in the same way that EVERYONE tests in production. It’s not Ok, but it is how the world works.
When you bug your users with error popups, they will bug you right back with bug reports. There is value in bug reports, but beware of eroding trust in your application.
“The Bug, Fixed?”
This story is not over. It never will be, until the app is retired and the codebase finally moved to cold storage. For now, a way to reduce the number of errors is to cache those failures for some time at the UI level. In this case, the underlying data is not critical to the user and the failure is intermittent. The team that owns the system returning the failures is not responsive. We can fall back to an identification number instead of the value associated to that number. We could also color the element associated to the id number to indicate a failure, rather than popping errors over the app.
Don’t show useless errors to the user too often (what is too often) - UX
If there is an error, it should be observable in some meaningful way - UX
Be careful of breaking a contract in shared services - Technical
Shared services should behave according to a contract - Technical
So, is it better to be correct, or to be useful? Is it possible to be both? Is it useful to show the user an error about connection status to a tertiary system? A Primary Key Error in a database? Those may be useful errors in troubleshooting, but not to an end user.
For the end user, tend towards usefulness
For core technical products, tend towards correctness
If possible, be both!
Would you have explicitly called out the UX considerations of this bug?
Where would you have “fixed” the bug?
Do you think a technical solution is sufficient?
Are there other social or UX aspects that you think should have been explored?
Do you have any balancing tests that you find useful?