It’s late 2024 and the NPU still feels silly. Tech companies like Qualcomm, Microsoft, AMD, Intel, and even Apple want casual users to care about running LLM or image generation models on local hardware like laptops. As we’ve mentioned, they want to avoid NVidia’s CUDA. While we always appreciate competition, it does feel like these companies are inadvertently screwing themselves or each other. The mention of the TOPS acronym only enhances the irony. Welcome to the nightmare story of silly specifications!
First there’s Qualcomm and Windows on ARM. Microsoft’s partnership with Qualcomm was not entirely surprising. Windows on ARM has been a persistent side project going back to Windows RT in 2012. The Snapdragon Dev Kit remains elusive, but the laptops themselves appear genuinely remarkable in terms of display quality and offer stellar battery life, representing PCWorld’s 4 of 5 in best battery life. For casual users with browser-based workflows, Snapdragon PCs are worth consideration, but developers still have a right to be annoyed.
Microsoft appeared to annoint Qualcomm for the first functional NPU. Intended to run chatbots or generate images using rough sketches in Paint. AMD and Intel’s NPUs, previously discussed, were branded Ai PC. Qualcomm machines have been declared CoPilot+ PCs by Microsoft and include a new button.
The system’s central feature, Recall, has been called a ‘Privacy Nightmare’ and delayed indefinitely. But hey, at least the button got pushed to prod, right?
AMD’s first Ai PCs were stunted. After launching the NPU-inclusive 8600G and 8700G for desktop, Microsoft announced CoPilot+ PCs would require 40+ TOPS. These two NPUs only offer 16 TOPS, meaning our first-ever Ai PC build will presumably never be a CoPilot+ PC. Perhaps more sad is the Ryzen 7 8845HS, offering 38 TOPS. Womp, womp!
For AMD’s desktop Zen 5 chips, Ryzen 9000 series, the NPU is either missing or undocumented, even in the press release. Will AMD’s future desktop chips include NPU? Only time will tell. Luckily, the new AMD laptops look fantastic in terms of performance and efficiency.
Intel fell short in the TOPS game, too. Despite inclusion in Microsoft’s own Surface Laptop 6, the Core Ultra 165H only reaches 34 TOPS. So close, yet not CoPilot+ compatible!
The next generation of Intel’s laptop chips will supposedly extend “up to” 120 TOPS. Lunar Lake looks promising for game demos.
Why should consumers care about the NPU? This new NPU hardware may eventually offer a benefit to third-party developers for some types of operations in the future, but only if the stars align. Consumers need to buy these machines in number before developers see a need to build apps that depend on NPU.
The current NPU play is about demo and delay. Shareholders want to see demos of the new chip in action. A controlled demo video is potentially enough. Companies want potential shareholders to buy the stock today based on demos of future functions. If the product is embarrassingly bad, the company should and will delay public release. Until the features ship, it’s only about adding shareholder value. The consumer doesn’t matter until the product exists in a functional state in public release.
Some of these machines seem great. The graphical hardware, efficiency, and displays look fantastic. And it’s a good thing because functionally, the NPU isn’t yet providing benefit. Consumer-oriented Ai needs to be real and widely available. As consumers, we all need to know someone who bought one of these machines and we need to have hardware and software envy. That’s a big part of what would drive hardware sales.
What happens if these software features never ship? It’s certainly possible, and you’ll notice every product demo begins with disclaimer text explaining that features are subject to change. Typical users don’t want to run betas that could destroy their battery life or put their critical software at risk. Companies don’t usually provide any meaningful support for betas, aside from telling you to roll back to public release when something doesn’t work properly.
We believe consumer tech is how Ai breaks even and becomes profitable, if ever. The industry cannot support the expenses of Ai model training (hardware, electricity, and the cost of having humans generate). Investors and shareholders will eventually want their money back in the form of profits, likely in the form of recurring revenue through SaaS. And it’s important to remember that most subscription apps don’t make money. We expect this will remain true for projects in this latest Ai era.
Full Disclosure: Devin owns some NVDA and META stock.