Building scalable systems isn’t limited only to technology. Often you can take the lessons learned from Computer Science and apply those same techniques to business systems and dramatically improve the effectiveness of a business.
During a company meeting at one of my previous jobs one, of the executives was berating the company as a whole for our performance of late. In particular he stressed that “we know the processes work, people just need to be more careful”. This statement struck me as odd, from a software perspective if you are experiencing similar and frequent errors with different hardware in place, you would expect there is some problem with the software itself. Yet rather than looking at the systems themselves they chose to blame the hardware.
During a discussion I had with a director at this same company, he was lamenting that people were not careful enough when they made modifications to production data. I explained that when any data modification is made there is a non-zero chance that someone will mess up the data. Yet with a database you want your data to be as error free as possible. So rather than relying on people to be perfect, you build your systems around the idea that people are imperfect. Good backups, primary and foreign keys, check constraints, triggers, type safety on data, these are all methods to help ensure the consistency of your data. More importantly though they are put in place so you can state expectations about what can go into the database and to prevent people from inadvertantly messing things up. In the case of backups you have a third level of redundancy in case your first two lines of defense fail (people, constraints).
The logic of redundancy and checks can be applied to almost any process. I read this fantastic article about checklists a few days ago. Essentially by implementing checklists for certain procedures, and having those lists monitored and enforced they reduced the error rate nearly to zero. The lists weren’t for some low level uneducated worker. Rather they were for highly trained and specialized doctors.
The moral of the story then is that as complexity increases, it’s important to build scalable business systems that will account for the imperfection of people. Even something as simple as implementing a checklist and having people actively monitor that all of the steps were done correctly can yield much more fault tolerant business systems.