Humans have reached the upper limit of silicon computing architecture, and AI is expected to consume 50% of the world's electricity supply by 2030

Edited by Aeneas

We have already begun to experience the feeling that the silicon computing experience has reached the upper limit. In the next 10 years, there will be a serious computing power gap, and neither existing technology companies nor governments have been able to solve this problem.

Now, we're so used to computing getting cheaper that we never doubted that maybe one day we wouldn't be able to afford it.

Now, Rodolfo Rosini, CEO of a startup, asks a question that shocks us: What if we're reaching the fundamental physical limits of classical computing models, just as our economies rely on cheap computing?

The stagnation of big computing

Now, due to a lack of technological innovation, the United States has reached a plateau.

Wright's Law holds true in many industries — every 20 percent or so improvement in manufacturing processes doubles productivity.

In the field of technology, it manifests itself in Moore's Law.

In the 1960s, Intel co-founder Gordon Moore noticed that the number of transistors in integrated circuits seemed to double year-over-year, proposing Moore's Law.

Since then, this law has become the basis of a contract between the market and engineering, taking advantage of excess computing power and shrinking size to drive the construction of products in the computing stack.

The expectation was that with faster and cheaper processors, computing power would increase exponentially over time.

However, the different forces that make up Moore's Law have changed.

For decades, the driving force behind Moore's Law was Dennard's law of scaling. Transistor size and power consumption are halved simultaneously, doubling the amount of computation per unit of energy (the latter is also known as Koomey's LawKoomey's Law).

50 years of microprocessor trend data

In 2005, this scaling began to fail due to the heating of the chip due to current leakage, and with it, the performance of the chip with a single processing core stagnated.

To maintain the trajectory of computing growth, the chip industry has turned to multicore architectures: multiple microprocessors "glued" together. While this may lengthen Moore's Law in terms of transistor density, it adds complexity to the entire computing stack.

For some types of computing tasks, such as machine learning or computer graphics, this brings performance gains. But for many general-purpose computing tasks that are poorly parallelized, multicore architectures are powerless.

In short, the computing power of many tasks is no longer growing exponentially.

Even in the performance of multi-core supercomputers, from the TOP500 (ranking of the world's fastest supercomputers), there was a clear inflection point around 2010.

What is the impact of this slowdown? The increasingly important role that computing plays in different industries shows that the impact is immediate and will only become more important if Moore's Law is further shaken.

To take two extreme examples: increased computing power and lower costs have led to a 49% increase in productivity in oil exploration in the energy industry, and a 94% increase in protein folding forecasts in the biotechnology industry.

This means that the impact of computing speed is not limited to the tech sector, and much of the economic growth of the past 50 years has been a second-order effect driven by Moore's Law, without which the world economy may stop growing.

Another prominent reason for the need for more computing power is the rise of artificial intelligence. Today, training a large language model (LLM) can cost millions of dollars and take weeks.

Without the continued addition of number crunching and data expansion, the future promised by machine learning cannot be realized.

With the growing popularity of machine learning models in consumer technology, heralding a huge and possibly hyperbolic demand for computing in other industries, cheap processing is becoming a cornerstone of productivity.

The death of Moore's Law could lead to a big stagnation in calculations. Compared to the multimodal neural networks that may be required to reach AGI, today's LLM is still relatively small and easy to train. Future GPTs and their competitors will require particularly powerful high-performance computers to improve, and even optimize.

Perhaps many people will be skeptical. After all, the end of Moore's Law has been predicted many times. Why should it be now?

Historically, many of these predictions have stemmed from engineering challenges. Previously, human ingenuity had overcome these obstacles time and time again.

What's different now is that we're no longer facing engineering and intelligence challenges, but limitations imposed by physics.

MIT Technology Review said on February 24 that we are not ready for the end of Moore's Law

Overheating makes it impossible to handle

Computers work by processing information.

When they process information, some of it is discarded as microprocessors merge branches or overwrite the registry. It's not free.

The laws of thermodynamics have strict limits on the efficiency of certain processes, and they also apply to calculations as they do to steam engines. This cost is called Landauer's limit.

It is the tiny amount of heat dissipated during each calculation operation: about 10^-21 joules per bit.

Given that this amount of heat is so small, the Randall limit has long been considered negligible.

However, engineering capabilities have now evolved to the point where they can reach this energy scale, as real-world limits are estimated to be 10-100 times larger than Landauer's boundaries due to other overhead such as current leakage. Chips have hundreds of billions of transistors running billions of times per second.

Add up these numbers, and perhaps Moore's Law may have an order of magnitude left before reaching the thermal barrier.

At that point, existing transistor architectures will not be able to improve energy efficiency further, and the heat generated will prevent the transistors from being packed more compactly.

If we don't figure this out, we can't see what will happen to industry values.

Microprocessors will be limited and the industry will compete for lower rewards for marginal energy efficiency.

The chip size expands. Take a look at the GPU card of the NVIDIA 4000 series: despite using a higher-density process, it is only the size of a puppy and has a whopping 650W.

This prompted NVIDIA CEO Jensen Huang to declare "Moore's Law is dead" at the end of 2022 — a claim that other semiconductor companies have denied while this claim is mostly true.

The IEEE publishes a semiconductor roadmap every year, with the latest assessment being that 2D scaling will be completed in 2028 and 3D scaling should be fully launched in 2031.

3D miniature (in which chips are stacked on top of each other) is already common, but it is in computer memory, not in microprocessors.

This is because memory has much lower heat dissipation; However, heat dissipation is complex in 3D architectures, so active memory cooling becomes important.

Memory with 256 layers is on the horizon and is expected to reach the 1,000-layer mark by 2030.

Going back to microprocessors, multigate device architectures that are becoming the commercial standard (such as Fin-field effect transistors and gates-all-round) will continue to follow Moore's Law for years to come.

However, due to inherent thermal problems, true vertical scaling is not possible after the 2030s.

In fact, current chipsets carefully monitor which parts of the processor are active at all times, avoiding overheating even on a single plane.

2030 crisis?

A century ago, the American poet Robert Frost asked: Will the world end in frost or fire?

If the answer is fire, it almost heralds the end of computing.

Or, just accept the fact that electricity use will increase, and then scale up the manufacture of microprocessors.

For this purpose, humanity already consumes a large part of the earth's energy.

Perhaps another option is to simply embrace increased power use and scale up the manufacturing of microprocessors. We already use a large part of the Earth's energy supply for this purpose.

In Ireland, just 70 data centers consume 14% of the country's energy. By the 2030s, 30-50% of the world's electricity production is expected to be used for computing and cooling — not counting the energy consumption of cryptocurrencies.

(Interestingly, after the March 19 blog post, the authors deleted this prediction.) His explanation is that this is an inference based on the worst-case scenario of the Nature paper, which has now been removed for clarity and precision of the argument)

The current rate of scaling up energy production will lead to a slight increase in the cost of scaling Moore's Law.

A series of one-off optimizations at the design (energy efficiency) and implementation level (replacing old designs still in use with the latest technology) will allow developing economies such as India to catch up with overall global productivity.

After the end of Moore's Law, humans will run out of energy before the manufacturing of microprocessor chips reaches its limit, and the pace of computing cost decline will stagnate.

While quantum computing is touted as an effective way to surpass Moore's Law, there are too many unknowns to be commercially available, at least for the next 20 to 30 years.

Obviously, there will be a serious computing power gap in the next 10 years, and existing technology companies, investors or government agencies will not be able to solve it.

Moore's Law and Randall's limit collided for decades and is arguably one of the most significant and critical events of the 2030s.

But now, it seems that not many people know about this.

Resources:

https://www.exponentialview.co/p/the-great-computing-stagnation