The Assumptions Programmers MakeNick RogersBlockedUnblockFollowFollowingMay 31“Tractor” by Leszek Leszczynski is licensed under CC BY 2.
0In the summer of 2017, we at DealerOn had just flipped the switch on our spiffy new inventory management system, bringing to life an event-based web UI for managing and merging vehicle updates from our automotive dealers’ respective systems.
These updates now could go live on the websites we hosted in near real-time, rather than the old system where you were lucky to get one update every 12 hours.
After some initial bumps, we successfully migrated all of our dealers away from the legacy system, and everyone was happy: Our customers, who could now have a vehicle up for sale the minute it arrived on their lot; our customer support staff, who could correct issues without needing to cross their fingers and wait until the next day; and especially the programmers, who put tons of work into something that looked and worked great.
Or so we thought.
A key part of the inventory update process for each dealer involves downloading photos for each vehicle, either from the dealer’s management utility or from a third party, and formatting them for display on our dealer’s websites.
We made the decision to store each photo on our assets server in a directory named by the vehicle’s VIN.
For example, a vehicle with VIN 1C8RACDN8KW520476 would have all the photos downloaded into a directory with the name “/photos/1C8RACDN8KW520476”.
We decided a bit later on that we wouldn’t want to run out of storage space, so we wrote a quick little method to delete photos from our assets after the vehicle was sold.
The code was tested and deployed without any issues.
That is, until one morning when we rolled into the office and learned our client support team was getting hammered with calls from angry customers asking us why all of their vehicles were displaying a “Vehicle Photos Unavailable” stock image instead of the photos that they had taken.
(Apparently car shoppers actually want to see the car before they’d consider buying it, who knew?)Our dev team sprang into action to figure out why the photos weren’t showing, thinking it was simply a bug in the website code.
After some initial debugging, it was discovered to be a much bigger problem than first realized: nearly all the inventory photos for all of our websites had been deleted from our assets server!.The “/photos” directory was nearly bare.
We quickly began to restore a backup of the missing photos so they would begin showing again, but where did the photos go, and would it happen again?Ruh roh.
There was a scramble to find out if anyone had accidentally deleted the photos on the production assets server, or if the quick method to delete photos for sold vehicles was going rogue and deleting way more than it should have been.
We weren’t having much luck, and hearing from client support about all the ticked-off dealers who wanted to know what gives wasn’t helping.
By that afternoon we had ripped out every single capability of our inventory importer to delete any type of file whatsoever, but we still didn’t know the root cause of all this.
On a whim, we pulled up the exact time our assets server logs started recording a sharp increase of free disk space and we tried to figure out exactly what happened at that second.
Eventually, we spotted something strange.
There was a vehicle removed from a dealer’s inventory right before the photos were all deleted, and this vehicle had a VIN of a single period.
Even though VINs were not a national standard until 1981, our system requires something (i.
not necessarily a valid VIN, but it can’t be blank) in the VIN field for each vehicle.
We realized what had happened, after the vehicle with a vin of “.
” was sold, we gave our system a command to delete “/photos/.
”, which translates to “all files and directories contained in the directory ‘/photos’”.
After we were somehow able to extract our palms from our foreheads, we deployed a fix and then manually refreshed photos for any sites still missing them, and the phone calls stopped.
The trouble started when this assumption was not challenged…It turned out that this “.
” vehicle was a vintage tractor which was for sale in a rural dealer’s inventory.
It didn’t have a VIN, so the dealer entered only a period because our new inventory system required him to enter something.
It raised an interesting question, though: could this really be called a bug?.Sure, we definitely should have been checking for the path traversal vulnerability, but entering a period for a VIN was perfectly valid and well within the bounds of our expected processing behavior.
At some point, one or more persons made the (wrong) assumption that “every vehicle has a valid 17 character VIN” and decided it was OK to categorize information based on it.
The trouble started when this assumption was not challenged, causing the inventory system functionality based on this faulty assumption to come crashing down.
Surely if a developer had been asked point-blank, “What happens if the photo processor tries to delete a folder for a VIN with a path character in it”, they would have immediately seen the problem and put in a ticket to correct it before something like this happened.
Why wasn’t that question ever asked?.Especially on long-running projects, it’s tough for developers and architects to avoid getting tunnel vision.
Even when there’s a spec or ticket to work off of, workers assume that the ticket is gospel truth and has been completely fleshed out.
In truth, no spec will be 100% complete and oversights are bound to happen.
Assumptions will be made, but regular status reports between project stakeholders can help highlight them at the outset of the project rather than at the end, when frustrated developers will need to toss out hours of work over something that could have been cleared up with a conversation.
Developers can even get caught off guard by assumptions they’ve made about societal customs, like names, gender, addresses, and calendars.
Following the legalization of same-sex marriage nationwide in the United States in 2015, several local governments were unable to issue marriage licenses because their systems were not able to properly record a marriage between two people of the same sex.
Depending on when these systems were launched, the assumption of “marriage will always be between two people of the opposite sex” may have been completely valid to make at that time, but it became invalid in the years immediately prior to 2015.
Because this assumption wasn’t corrected in time, whether due to inattention or lack of priority, these systems needed to undergo costly upgrades.
It’s unlikely that a programmer will be able to think of every possible way that bad input can cause problemsTo help uncover hidden assumptions, there’s a great list of Falsehoods Programmers Believe with common assumptions to keep in mind.
Plenty of the listed assumptions seem obvious when reading them, but developers can still be blind to them.
(I wish I had this list when I wrote some code that I thought was great at the time, before it folded like a lawn chair once the clocks changed for Daylight Savings Time.
)Of course, it’s unlikely that a programmer will be able to think of every possible way that bad input can cause problems.
Still, it’s just as unlikely that the data a program operates on will never stray from the golden path.
Sometimes it’s a good idea to take a step back and look at one’s own code from a distance and consider the unexpected.
The best time to ask questions such as “What happens if this variable is null?” or “What if the database goes down during this call?” or “What if an unauthorized user tries to view this page?” is while looking at the big picture.
Otherwise, implicit assumptions are made, which if left for several years before being uncovered can lead to resorting to hacks or expensive redesigns.