Accepting My Fate As A Millennial Software Engineer

(back-ported from Medium - May 19, 2019)

I used to get rather upset about how I was born so late into the game of computer science. I mean look at it, all the coolest stuff that involves computer science fundamentals we learned in school have all been invented — programming languages, operating systems, data structures and algorithms, storage systems, cryptography, computer graphics, world wide web, etc. Our day-to-day job involves taking these wonderful tools built by other brilliant people and then stitching them together with if/else statements. Everything is becoming a blackbox and these boxes just keep on getting bigger and bigger.

Some examples:

binaries → high level languages & frameworks
formal proof → chaos monkey
on-prem → cloud
decision trees → neural networks

Because of how high the piles of abstraction are, there is simply no way to understand everything. So we have no choice but to use the lazy evaluation approach — where we trust the systems and tools to work, and only when it doesn’t will we then start digging into the layers of abstraction in an attempt to understand and fix the problem. If we are lucky, we can dig all the way to the root of the problem. Other times it is a true blackbox (be it due to a mental constraint, like neural networks, where our brain simply doesn’t have that kind of capability to process all the information; or be it due to a physical constraint, like cloud products, where the source code is locked away as trade secrets), where all we could do is trial and error — we form an arbitrary hypothesis, run it against the box, and check its output to see if matches the hypothesis. And perhaps with this newly concluded confirmation bias we fixed the problem, but we still don’t fully understand the box so we are left with a hint of sadness. Theoretically we could build everything from scratch, but working in the industry means productivity is king; we certainly won’t be at our jobs for long if we insisted on doing that.

It took my ego a while to accept this reality. But once I did, I realized: how close-minded of me to think that all the coolest things have been invented! The point of invention is that it does not exist yet. So that assumption all the coolest things have been invented is absurd (and cocky). Instead, the area for innovations simply moved up in the abstraction layer, while I grudgingly stayed below. I learned to marvel at the existing robustness and intricacy of the systems and tools we already have, and at the same time try to take learnings from it in attempt to identify gaps and improve the ecosystem further.

Thinking on a higher level today, there are so many potential areas for improvements off the top of my head (from the perspective of an infrastructure engineer, so these ideas are rather limited in scope):

Why are the most popular programming languages still designed for a single machine and have to rely on libraries built on top of application protocols to communicate between machines? I could imagine a rise of distributed + declarative programming languages in the near future that abstract all that away.
Why are we still stuck on application protocols like HTTP/FTP/etc. anyways? I could imagine higher level of protocols that communicate more with less code/data.
Why are we replicating data on so many heterogenous storage systems in order to optimize for different use patterns. I can imagine a one-size-fits-many database in the future which stores the data once but leverage machine learning to optimize for different usage patterns.

In conclusion, I have come to accept my fate as a millennial software engineer that stands on the shoulder of giants and stitches their work together, because I realized that the giants before us were no different — they were also building abstraction layers on top of the work known before them. There is always room for innovation, as long as one sets the mind to it. This is what I love about this field of work — formulating ideas and then making them concrete via code.