Nvidia’s Rise Surprises Even the Researchers Who Started AI Boom

This week I attended a very of-the-moment Silicon Valley event. But first…

Three things you need to know today:

• Astera Labs jumped 72% in its first day of trading on demand for AI
• Microsoft attracted growing criticism for censoring Bing in China
• Epic Games will open mobile stores on the iOS and Android platforms

Unforeseen consequences

A huge line of people snaked across the floor Wednesday morning at AI chipmaker Nvidia Corp.’s tech conference in San Jose, California. They all hoped to get into the room where Chief Executive Officer Jensen Huang was to preside over a reunion of nearly all the authors of a key 2017 research paper that led to the current boom in artificial intelligence. It was, in a sense, getting the band back together.

Huang gathered seven of the eight authors of “Attention Is All You Need,” which became massively influential in AI in the years since it was published. The paper, the work for which was completed when all its authors worked at Google (they’ve all since left), introduced the concept of transformers – technology that allowed companies such as OpenAI to build large language models (that’s the “T” in GPT-4, which helps power ChatGPT), and Nvidia to sell oodles of the chips needed to train such models.

Huang, clad in his signature black leather jacket, took the stage and began by gently haranguing late-comers trying to find seats in the back (“C’mon you guys, hurry up, I’m going to start!”). Then he asked the members of the panel — Ashish Vaswani, Noam Shazeer, Aidan Gomez, Lukasz Kaiser, Illia Polosukhin, Jakob Uszkoreit, and Llion Jones — about the origins and impact of their work. Paper author Niki Parmar was absent due to a family emergency, Huang said.

Polosukhin, co-founder of NEAR Protocol, said the paper originated from efforts to improve machine-aided answering of users’ questions. Google requires quick answers to questions, he pointed out, so AI models needed to be designed to process information rapidly. Existing models at the time were simply not nimble enough.

Jones, co-founder and chief technology officer at Sakana AI, said the researchers were trying to build technology that would be able to generalize across a wide range of tasks. But they didn’t predict how well it would work, he said.

Several of the paper authors left Google to start their own AI companies, precisely because they felt they needed to be outside the research lab to build the technology they developed at Google into something that could be useful for many people.

“You couldn’t make these models smarter in the vacuum of a lab,” said Vaswani, who co-founded Essential AI with Parmar.

Only Kaiser, who is a member of the technical staff at OpenAI, didn’t strike out on his own — but he’s working at the company that has arguably benefited the most from the development of the transformer.

Nvidia’s powerful chips, and the company’s rise to become one of the five most valuable global companies, has pushed Huang into prominence as an evangelist for the benefits of AI. And quips and reminiscences aside, the interactions among Huang and the group on stage made one thing crystal clear: While the Nvidia CEO expressed appreciation for their technological development — “We have a whole industry that’s grateful for the work you guys did,” he said at one point — the researchers, now almost all entrepreneurs themselves, need him as much as he needs them.

“Thank God for giving us this incredible technology, and thank Jensen,” said Shazeer, co-founder and CEO at Character.ai.

He might have been exaggerating, but then again, maybe not.

The big story

Apple is under antitrust pressure. The US Justice Department is set to sue the iPhone maker as soon as Thursday, accusing it of blocking rivals’ access to the popular smartphone. Separately, rivals including Meta and Microsoft joined forces and asked a federal judge to reject Apple’s plan for opening its App Store to outside payment options, the latest volley in the legal back-and-forth regarding Apple’s control over its app marketplace.