Programming and Allegory
The universe is filled with disorder. With a little hand waving and a lot of generalization, you might consider the mark of an intelligent species as one that attempts to bring order to the chaos. Order is in the eye of the beholder. For the thousands of years that humans have been on Earth, order has been synonymous with civilization. In fermenting the uniquely human perception of civilization, we erected buildings where forests once stood, developed cars that guzzle fossil fuels, and marred the environment in irreversible ways. The imposition of human order and the development of civilization are two sides of the same coin.
The human need to impose order is everywhere. From the bridges we build, to the software we write. While the connection between physical and digital infrastructure may seem transient, they share many similarities. In fact, digital infrastructure is often inspired by physical infrastructure, relying heavily on physical metaphors in its composition. Obviously, physical infrastructure predates its digital brethren – humans have been building things in the physical world since we could walk upright – which is to say, thousands of years. Comparatively, we’ve only been writing software for a tiny fraction of that time. Thus, when programmers built our earliest digital infrastructure, they looked to the physical world for inspiration through allegory.
In the physical world, we use bricks, cement, stone, and steel to erect structures. When building software, we work with data. This was true in the 1950s, when computer science was in its absolute infancy and the first programs were being written, and it’s true today, as software is eating the world. As we wrote increasingly complex software, from automating clerical work and managing organizations and logistics to complex operating systems and distributed web servers, programmers have constantly sought better ways to organize software (see containerization and microservices). Digital infrastructure, like its physical kin, requires scalable building blocks as well as tried and true methods of putting them together.
Lessons from Physical Infrastructure #
Humans have been constructing buildings for a long time, and the hard-won fruits of that labor manifest themselves in the longevity and extravagance of our modern architecture. We borrow on centuries of architectural prowess from great minds all over the world in order to construct bigger and better buildings. Knowledge trickles down the generations, becoming refined through evolution, interaction, and diffusion. We wouldn’t have the Burj Khalifa if the ancient Egyptians hadn’t built the Great Pyramids.
The influence of the old in the construction of the new is all around us. Structures like the Santa Maria del Fiore in Florence, the Haiga Sophia in Turkey, and the Pantheon in Rome continued to influence architectural works and techniques centuries after they were erected. Countless architects and their creations have looked to these structures as inspiration, and more significantly, as learning material. The dome of the Santa Maria del Fiore was a magnificent achievement in 15th century Florence – the very first of its kind. The weight of the dome is estimated to be approximately 37,000 tons – its completion in 1436 is widely regarded as the start of the European Renaissance. In constructing the Duomo, Filippo Brunelleschi, the man in charge of the project, had to develop new techniques that would influence architecture and design long after his death. Necessity is the mother of invention: to build something truly innovative, you must consider the problem from previously unexplored dimensions.
Embracing Innovation #
We use old techniques for as long as they meet the demands of our ever expanding imaginations. The transition to new and innovative methods is never simple, as illustrated by periods of history marked by technological progress. The industrial revolution was a time of great innovation. New machines were being developed, and gone were the days of dangerous, manual drudgery. But were they? The period immediately following the industrial revolution was one defined by a stagnation in productivity, not an overwhelming increase, as many pundits hypothesized. The immediate impact of the era’s technological innovation produced an underwhelming effect on productivity because people were using new technologies in old ways.
As time went on and innovation continued, the bump in productivity promised by the wave of technological progress ushered in by the Industrial Age was finally seen. The earlier trends in the period would be felt again at the start of another era of technological progress.
Robert Salow, the 1987 Nobel Prize winner in Economics, captured this tension in what’s now referred to as Salow’s paradox. In describing the advent of computing and the onset of the Internet Age, Salow famously said, “You can see the computer age everywhere but in the productivity statistics.” The Internet Age, like the immediate period following the industrial revolution, was marked by stagnation in productivity. While there are many theories as to why this is and many proposed solutions, it fundamentally boils down to the use of new technology in old ways.
While innovations like the cloud have revolutionized the way we do business by democratizing compute resources and capabilities, we won’t fully break from Salow’s paradox until we rethink the fundamental building blocks of the software that is now eating our world.
The Limits of Classical Computing and Software #
The code that underlies modern applications bears too much resemblance to code written in the second half of the last century. While modern apps are written in higher level programming languages than older digital infrastructure, the patterns and techniques have not changed. Data are the building block of software applications, and just as humans have sought to bring order to our world by carving civilization out from the environment around us through the development of structures, institutions, and social systems, software applications also require methods of imposing order on the data that these digital systems touch.
Early programmers developed the notion of data structures, that is, ways of organizing data within software applications. This gave programmers the scalable building blocks they needed in order to construct the digital equivalent of the Duomo of Florence. Writing organized and well structured code not only improved the overall quality of software, it also made the development of software more efficient by making it easier for other people to understand and contribute to existing code. Data structures are ubiquitous today. They underlie practically every application you use. Data structures are covered in every undergraduate Computer Science curriculum in the world, and in many ways, you cannot think about software without thinking of data structures.
While the concept of data structures is profoundly important, it’s another example of the use of new technologies in old ways. The ways we structure data within software is inspired by the physical world. To see this paradigm, look no further than the names of some canonical data structures: we have sets, lists, and stacks that function like physical stacks of objects. We also have bags, queues, and a variety of trees. We have branches in our version control systems, and forks in our repositories. In many ways, the physical analogies of our digital building blocks are invaluable: they bring a semblance of familiarity to the complex business of writing software. Good design optimizes for reduced cognitive load on end users – in designing data structures that borrow from the physical world we know so well, we ensure that programmers have intuitive building blocks to play with. This is a common pattern in the software world – API designers take pains to develop interfaces that repeat ubiquitous patterns – it is easier to learn a new technology and become productive with it when that piece of technology resembles something you already have experience with. Thus, it makes sense for data structures, in all their ubiquity, to serve as allegories to the physical world.
But what if, in the pursuit of simplicity, we’ve forsaken greater innovation? While quantum computing research is accelerating and there are whispers of a quantum age on the horizon, there are still many optimizations to be made in the classical computing era. Perhaps our code has optimized the use of physical allegory and now requires something more. The next generation of code will be written with innovative data structures and design patterns that have no analogy to the physical world. While you can compare data to bricks or stones in a myriad of ways, there are many properties of a datum that a simple brick could never have. Data can be immutable or not, they can be duplicated with ease, they can be combined, transformed, and interpreted. Revisiting the fundamental building blocks of software and thinking about new ways of structuring data will give us a new class of software that is more efficient, more productive, and more useful to end users. The advent of these methods won’t obviate the need of traditional data structures – there will always be a need for intuitive allegory to the physical world while building something as complex as software – but these methods will be combined with newer conceptions, opening a hitherto undiscovered frontier of classical computing capabilities.
The fundamental redesign of how we structure data in programs in order to increase productivity is not an isolated trend in software. The governance of software projects has undergone evolution in order to better fit stake-holder needs. The traditional waterfall method, which entailed rigorous planning and design followed by implementation, verification, and maintenance was not well suited for the iterative nature of software development. The method was borrowed from the design and development practices used for physical infrastructure, such as buildings and bridges. When building bridges, iteration is financially and sometimes physically prohibitive, so an emphasis is placed on the planning phase. While the waterfall approach worked and early software organizations used it, it was clear that a more iterative method with a focus on the nuances of software was necessary. Today software organizations emphasize agility and testing, as well as constant stake-holder buy-in and user motivations – all fundamental to the development of software products.
If we could innovate the ways in which we govern our software projects to increase productivity and meet consumer demand, then we can also innovate the ways in which we write software as well. The physical allegories we rely on while building software are dated, and the goal of discovering new methods is to optimize classical computing capabilities by re-imagining how we compose code.