Once anything enters a bubble, you inevitably start worrying about when it will burst, and AI chips have by now clearly entered one.
From Cambricon's DianNao at ASPLOS'14 to Google's current TPUv3, AI chips took only five years to become a huge success. Riding the explosion in demand for AI compute, and with everyone shouting that Moore's Law is over, Domain Specific Architecture seems to have become the only way out.
But as countless giants and startups design one AI chip after another that all look much the same, we need to answer a question: do we really need this many AI chips?
Software Complexity
One problem the fast growth of AI chips cannot avoid is the exponential rise in software complexity. Plenty of companies spend two years or even less building a chip, then find they need far longer to support a pile of frameworks, keep up with algorithm progress, and adapt to platforms ranging from phones to data centers. Once you miss the window for deployment and volume production, the chip you built goes out of date fast.
Unlike a general-purpose architecture, a specialized one like an AI chip has to be designed with software design and optimization in mind at the same time. Chip companies are often optimistic about what software porting and tuning will cost, and hope that middleware and compilers will solve everything. In practice, from Intel to Google to Nvidia, large numbers of software engineers are being poured into porting to various platforms and hand-tuning network performance. And among startups, chips that taped out long ago but keep slipping their delivery dates are everywhere.
The deeper we dig into the potential of a chip architecture, the harder the software abstraction gets, because it has to pull models or parameters from the underlying architecture up into the higher-level abstraction. The usual approach today is middleware sitting between the low-level chip architecture and the software above it, but the cost of building that middleware is usually underestimated. A while ago someone from a chip startup asked me how many people and how much time it takes to build an Inference middleware layer like TensorRT. That is not an easy question to answer, so I asked back how many resources they had for the project. To my surprise, their boss had given them only three or four headcount, because they assumed they already had a low-level compiler and a model conversion tool on top, so a middleware layer for architecture abstraction would not take much effort. I suspect that much investment can produce something functionally complete, but I do not believe the final product will hit the performance numbers they want in real applications. After all, you do not build a chip just to run a Benchmark like ResNet-50.
Fragmentation
Write one set of code and run it on different platforms: that is what software engineers have wanted for a long time. The fragmentation created by AI chips with different architectures does a lot of damage to their willingness to put AI into real software products. Unlike what we are used to, deep learning's poor interpretability brings all kinds of unexpected defects. Here is a common headache: a proprietary model gives satisfying results on a local CPU, but its performance drops sharply once it is deployed to some device. How do you debug that, who is responsible for debugging it, what tools do you debug it with, and can the debugging engineer even get hold of the proprietary model? None of these questions have easy answers.
Fragmentation also shows up in the way proprietary architectures give up forward compatibility in order to squeeze out absolute performance. The middleware above has fragmented AI frameworks on one side and generation after generation of chip architectures on the other. How do you maintain several partly incompatible instruction set architectures at once and guarantee that every software update covers every device completely? There is no answer other than throwing more people at it. A common line is to do what consumer chips do today and offer only short-term software support, two or three years. But in the fields where AI chips are actually used, smart cameras, industrial AI and autonomous driving, a chip's life cycle can run to ten years. It is hard to imagine what scale a company has to reach before it can provide support that lasts, and if you expect a startup not to survive two or three years, how do you feel comfortable putting its product into a consumer car in volume production?
AI Chips Are Transitional
From a software engineer's point of view, I firmly believe custom AI processors will only ever be a transitional product. A unified, programmable, highly parallel architecture is the direction we should be aiming at. Look back at the past twenty years: we watched the minicomputer market and its specialized architectures shrink, we watched graphics processors grow into general-purpose vector processors, and even our phone and PC platforms are converging. There is good reason to think that putting resources into custom AI chips now is not a good investment at all.