Inside Meta’s AI strategy: Zuckerberg highlights compute, open source and training data

Big Tech’s earnings conferences this week all offered insight into their companies’ AI efforts. Google focused its generative AI efforts in search and cloud. Microsoft has delved into the integration of AI across its technology stack. And Amazon talked about chips, Bedrock, and, oh yeah, Rufus, its new AI-powered shopping assistant. But I think Meta beats them all when it comes to providing the most in-depth coverage of AI strategy.

The Meta AI Playbook is unique in many ways, thanks to our consistent focus on open source AI and the vast and growing amount of AI training data we derive from public posts and comments on Facebook and Instagram.

So it’s interesting that yesterday during Meta’s Q4 2023 earnings call, CEO Mark Zuckerberg was the first to tout the company’s advantageous position in one of the most competitive areas of AI development: computing. was.

Zuckerberg said Meta has a clear long-term strategy to become a leader in building the most popular and cutting-edge AI products and services, and what he claimed is necessary for that effort. He also said he would build “complete general intelligence.” The first key aspect of this is “world-class computing infrastructure,” he said.

Zuckerberg went on to reiterate what he recently revealed in his recent Instagram Reels. That means by the end of this year, he expects Meta to have about 350,000 of his H100s, and if you include other GPUs, that’s about 600,000 H100 equivalents of compute in total. Why would Meta do such a thing? Surprise, surprise — Instagram Reels.

“We are in a good position now because of the lessons we learned from Reels,” he explained. “Originally, we weren’t building enough GPU clusters for Reels, but as we were looking at it, we realized we had enough GPU clusters to support both Reels and other Reels-sized AI services that we expect to come out in the future. We decided that we needed to build more capacity. That situation again.”

The meta is “playing to win,” Zuckerberg added, noting that training and operating future models will become even more compute-intensive.

“We don’t yet have a clear estimate of how much this will be, but state-of-the-art large-scale language models tend to be trained with about 10 times the amount of compute each year,” he said. . “Our training cluster is only one part of our overall infrastructure, and the rest is obviously not growing as quickly.” The company plans to continue to invest aggressively in this area. “We are also designing new data centers and designing our own custom, workload-specific silicon to build state-of-the-art clusters,” he explained.

Open source AI strategy was at the forefront

Zuckerberg then highlighted Meta’s unwavering open source strategy. Despite being criticized and even criticized by lawmakers and regulators over the past year for this issue, including an initial leak of the first version of Llama, Meta has made it available only to researchers. Masu.

“Our long-standing strategy has been to build and open source our general infrastructure while keeping specific product implementations proprietary,” he said. “For AI, common infrastructure includes Llama models, including Llama 3, which we are currently training and looking great so far, and industry standard tools such as PyTorch, which we developed. Open Source We deeply believe that this approach will bring about a lot of innovation across the industry.”

Zuckerberg also provided important details about Meta’s open source approach to business, a statement that has already been widely shared on social media.

“There are several strategic advantages. First, open source software is typically not only safer and more secure, but also operationally more computationally efficient, due to continuous feedback, scrutiny, and development from the community. This is a big problem because safety is one of the most important issues in AI. Increased efficiency and reduced computing costs benefit everyone, including us. Second, Open source software often becomes an industry standard, and as companies standardize building on our stack, it becomes easier to integrate new innovations into our products.

It’s subtle, but being able to learn and improve quickly is a huge advantage, and being an industry standard allows that. Third, open source is very popular among developers and researchers. We know that people want to work on open systems that are widely adopted. Therefore, this will help Meta to recruit top talent. This is extremely important when leading in new technology fields. Again, we typically have our own data and build our own product integrations anyway, so even offering infrastructure like Llama as open source doesn’t help our main The benefits are never diminished. This is why our long-standing strategy is to open source our general infrastructure, and why we expect it to continue to be the right approach for us. ”

Finally, I was fascinated by Zuckerberg’s emphasis on the “unique data and feedback loops” in Meta’s products.

Regarding the large corpus on which to pre-train the model, Zuckerberg said that Facebook and Instagram have “hundreds of billions of publicly shared images and tens of billions of publicly shared videos, and that number is larger than the Common Crawl dataset and people We estimate that it is more than that.” You can also share many public text posts across our services in comments. ”

The Common Crawl dataset contains petabytes of web data (raw web page data, metadata extracts, and text extracts) collected regularly since 2008. It’s huge. So the idea that the meta has access to a potentially even larger corpus of its own is literally huge.

But Zuckerberg went on to say: “Even more important than a pre-training corpus is the ability to establish a good feedback loop with the hundreds of millions of people who interact with AI services across our products. Reels and ads are a big part of how we were able to improve our AI system so quickly over the last two years when we had to rebuild our systems.”

Yesterday’s Bloomberg article highlighted the fact that the success of Meta’s llama model has led to a real llama becoming the unofficial mascot of open source AI events.

But if Meta’s earnings report is anything to go by, Meta’s 2024 capital investment will go even further than a cute fuzzy camel to win in a highly competitive enterprise. According to hints, the company is looking to move forward with billions of dollars more. AI racing gets faster and faster.