Why Software Engineering Skills Are Rapidly Becoming Obsolete

Why every software engineer must upskill.

‍

Why every software engineer must upskill.
‍

‍The Challenges of the Rapid Advance in Technological Progress

A few decades ago, the widely accepted belief was that in order to succeed you should study hard and get a college degree. This would then set you up for your career and essentially the rest of your life. Life was separated into a “learning” part and a “working” part.

Those days are long gone. And every software engineer knows this well.

From the invention of personal computers and the Internet to smartphones, DNA sequencing, self-driving cars, robotics, and now artificial intelligence and large language models (LLMs), the number of transformational, ground-breaking technologies are being invented and adopted at an unprecedented rate. Not only has technology advanced tremendously in the last few years, but the rate of technological progress and the rate of technology adoption has itself accelerated. In other words, the fundamental changes we used to observe at a societal level over decades are now happening in the order of a few years.

The rapid advance in technological progress and the increasing rate of technological progress and the rate of technology adoption comes with a lot of promise, but it also creates a multitude of new challenges for humanity and the future of work. One of the biggest challenges here lies in our ability to adapt to these changes.

‍

‍

We’ve all read the stories about how artificial intelligence will take over 100s of millions of jobs. Just a few weeks ago, Goldman Sachs reported that the latest wave of artificial intelligence could replace the equivalent of 300 million full-time jobs within a decade (see here).

But if we take a closer look, we’ll see that this is just the tip of the iceberg.

‍

‍Technological Progress Leads to Decreasing Half-Life of Skills

What’s really happening is that the half-life of any given skill (or piece of knowledge) is decreasing every year. In other words, if you learn a new skill today it will be valuable for a shorter period of time than a comparable skill you might have learned in the past.

Let me give you an example. A millennium ago someone might have learned the skills to be a carriage driver and worked in that profession for a life-time. They might have taught their children, grandchildren, great grandchildren and so on all the way up until the invention of the automobile. At this point in time, society transitioned from carriages to automobiles and with that the skills of a carriage driver lost their value. A person could now learn the skills of a taxi driver and have a career for a life-time in this new profession. With the invention of ride-sharing apps like Uber, the taxi drivers are now being displaced by a new generation. The value of acquiring the skills of a taxi driver is worth a lot less now, because the taxi driver profession is shrinking and because almost anyone can now acquire the skills to become an Uber driver. But the skill set of an Uber driver today is worth even less than that of a taxi driver 50 years ago, because that industry itself is on the verge of being disrupted by the invention and mass adoption of self-driving cars. This may still be a few years out, but it’s certainly not a century away.

‍

*Source:* *https://www.nathanlustig.com/half-life-jobs/*

The key is that the rate of change is also increasing. In other words, the speed of change (or the derivative of the curve) is itself changing. The life-time of these skill sets moved from say 3,800 years (the carriage driver) to 120 years (the taxi driver) to 7 years (the Uber driver). This can be hard to grasp as it’s well-known that humans struggle with non-linear thinking. There is even a term for this cognitive bias called the “linearity bias” (see here).

‍‍

The Half-Life of Software Engineering Skills and Software Libraries

Let’s look at an example relevant to software engineering. Software libraries tend to follow a similar pattern. I've spent years building software and ML systems in Python, so I am going to focus on this as a concrete example. In particular, let’s take a look at the public data available from the famous Stackoverflow website.

Stackoverflow has a tool allowing us to measure the % of questions for any given tag over time (see here). With this tool, we can look at any given library (if it has a tag) and see how popular it is. We can also easily measure when it reaches its peak popularity. Here’s an example of the scikit-learn library, which was released in 2007 and appears to have peaked in popularity around 2020 (i.e. 13 years after its release).

*Source:* *https://insights.stackoverflow.com/trends?tags=scikit-learn*

If we assume that the popularity of every library follows a Normal distribution over time, we can define the “half-life of its popularity” as the point in time where less and less people will be using the library. Then the half-life becomes the number of years between its release and its peak in popularity. This is extremely intuitive because beyond this half-life point, fewer and fewer people will use that library every year.

Following this methodology, I’ve taken 30 of the most Python libraries and plotted their half-life here. The graph could certainly be improved, but the trend is absolutely clear. The newer Python libraries have a far shorter half-life than the older ones. For example, the data science library SciPy appears to have twice the half-life of newer libraries like Gensim or Spacy.

‍

This phenomenon is by no means specific to Python. I’m no Java expert, but I can do the same analysis for popular Java libraries and frameworks. Below you’ll see the same plot for 30 of the most popular Java libraries and frameworks. There is again a strong trend showing that older libraries tend to have a far longer half-life than newer ones.

‍

Now with that said, it should be noted that this methodology has lots of drawbacks. The methodology exclusively considers software libraries, but ignores fundamental skills and conceptual understanding (including computer science theory). Further, the underlying assumptions are bound to be incorrect. The peak popularity period of each library is estimated and this is done solely on Stackoverflow data. Some of the libraries likely also haven’t peaked yet. And then there are many dependencies between the libraries. Some libraries depend on each other (one literally imports the other) and other libraries work better together, which means we can’t really treat them as “independent data points”. There may also be a “survival of the fittest” bias going on where the libraries I looked at here are the ones that have stood the test of time, and other libraries have simply been forgotten by the community and myself.

Finally, do we really care about the half-life of a library's popularity? Isn’t it possible that a library could be getting less popular relative to others, but that the number of open jobs for the corresponding skill set keeps growing? Or isn’t it also possible that even if the library is less popular, the pay for the skill set keeps growing? Yes, all of that is certainly possible.

The software industry is a highly complex and interconnected world. There are many more variables at play and it’s extremely difficult to isolate a single variable (such as “release year”) and use that to predict the half-life of any given software library.

Nevertheless, we have now looked at the strong trends and the key point remains. As the rate of technological progress and the rate of technology adoption increases, the half-life of each software skill is shortened.

‍‍

Software Engineering Will Be Hit Hard

The impact of technological progress is bound to be particularly strong in the software industry. The advances in computing (chips, cloud computing, AI, LLMs etc.) are all accelerating the rate of change in the software industry. Among other things, we should expect an acceleration in the development of new tools, new programming paradigms and ever more automation driven by AI in the coming years. The fact that this industry is already digitalized with millions of open source repositories and that LLMs can now understand and generate complex source code, means we should expect the rate of change to accelerate even more than in other industries.

This acceleration in turn will shorten the half-life of each skill.

Software engineers who want to stay relevant in this brave new world will have no choice but to upskill themselves. They need to keep learning new skills, all the time and on every topic.

As Thomas Friedman recently said: “When the pace of change gets this fast, the only way to retain a lifelong working capacity is to engage in lifelong learning.”

That’s why we’re building an AI Mentor for Software Engineering at Korbit. The AI Mentor analyzes the software engineer’s code, identifies issues and skills gaps and upskills them.

You can sign up here: www.korbit.ai

‍