Meta made a remarkable claim in an announcement released today intended to shed more light on its content recommendation algorithms. It is preparing for behavior analysis systems “orders of magnitude” larger than the largest big language models out there, including ChatGPT and GPT-4. Is that really necessary?

From time to time, Meta decides to refresh its commitment to transparency by explaining how some of its algorithms work. Sometimes this is revealing or informative, and other times it just leads to more questions. This time it’s a bit of both.

In addition to “system cards” that explain how AI is used in a given context or application, the advertising and social network posted an overview of the AI ​​models it uses. For example, it may be worth knowing whether a video depicts roller hockey or roller derby, even if there is some visual overlap, so that it can be appropriately recommended.

In fact, Meta has been among the most prolific research organizations in the field of multimodal AI, which combines data from multiple modalities (visual and auditory, for example) to better understand content.

Few of these models are published, though we often hear about them being used internally to improve things like “relevance,” which is a euphemism for targeting. (They allow some researchers to access them.)

Then comes this interesting little fact that describes how you’re building your computing resources:

To deeply understand and model people’s preferences, our Recommendation Models can have tens of trillions of parameters, orders of magnitude larger than even the largest language models in use today.

I pushed Meta to be a little more specific about these ten-trillion theoretical models, and that’s exactly what they are: theoretical. In a clarifying statement, the company said: “We believe our recommendation models have the potential to reach tens of trillions of parameters.” This phrase is a bit like saying your burgers “can” have 16-ounce burgers, but then admitting they’re still at the quarter-pound stage. However, the company clearly states that its goal is to “ensure that these very large models can be efficiently trained and deployed at scale.”

Would a company build expensive infrastructure for software it has no intention of building or using? It seems unlikely, but Meta declined to confirm (though they also didn’t deny) that they are actively looking for models of this size. The implications are clear, so while we cannot treat this model on the scale of tens of trillions as existing, can treat it as genuinely aspirational and likely in the works.

“Understanding and modeling people’s preferences”, by the way, should be understood as user behavior analysis. Your actual preferences could probably be represented by a hundred word plain text list. It can be hard to understand, at a fundamental level, why you would need such a large and complex model to handle recommendations for even a couple billion users.

The truth is, the problem space is huge: there are billions and billions of pieces of content, all with corresponding metadata, and certainly all sorts of complex vectors showing that people who follow Patagonia also tend to donate to the World Wildlife Federation, buy more and more expensive bird feeders, etc. So perhaps it’s not so surprising that a model trained on all this data is quite large. But “orders of magnitude larger” than even the largest out there, something trained in virtually every available written job?

There is no reliable parameter count in GPT-4, and leaders in the AI ​​world have also found it to be a performance-reducing measure, but ChatGPT is at around 175 billion and GPT-4 is believed to be more taller than that but lower. than the wild 100 billion claims. Even if Meta is exaggerating a bit, this is still terrifying.

Think about it: an AI model as big or bigger than any created so far… what goes in on one end is every action you take on Meta platforms, what comes out on the other is a prediction of what you will do or you will like below. A bit creepy, isn’t it?

Of course they are not the only ones doing this. TikTok led the way in algorithmic following and recommendation, and has built its social media empire on its addictive feed of “relevant” content meant to keep you scrolling until your eyes hurt. His competitors are openly envious.

Meta clearly aims to blind advertisers with science, both with the stated ambition to create the biggest model on the block and with passages like the following:

These systems understand people’s behavioral preferences using large-scale models of attention, graphical neural networks, few-shot learning, and other techniques. Recent key innovations include a novel hierarchical deep neural retrieval architecture, which allowed us to significantly exceed several next-generation baselines without regressing inference latency; and a new ensemble architecture that leverages heterogeneous interaction modules to better model factors relevant to people’s interests.

The above paragraph is not meant to impress researchers (they know all this) or users (they don’t understand or care). But put yourself in the shoes of an advertiser who begins to question whether his money is well spent on Instagram ads over other options. This technical mumbo jumbo is meant to dazzle you, to convince you that not only is Meta a leader in AI research, but that AI really excels at “understanding” people’s interests and preferences.

In case you’re wondering: “Over 20 percent of the content in a person’s Facebook and Instagram feeds is now recommended by AI from people, groups, or accounts they don’t follow.” Just what we asked for! So that’s all. The AI ​​is working very well.

But all of this is also a reminder of the hidden apparatus at the heart of Meta, Google and other companies whose main motivating principle is to sell increasingly granular and precise targeted ads. The value and legitimacy of that targeting must be constantly reiterated, even as users rebel and advertising multiplies and insinuates rather than improves.

Meta has never done anything sensible like presenting me with a list of 10 brands or hobbies and asking me which one I like. They’d rather look over my shoulder as I browse the web for a new raincoat and act like it’s some advanced artificial intelligence feat when they show me raincoat ads the next day. It’s not entirely clear that the latter approach is superior to the former, or if so, how superior? The entire web has been built around a collective belief in precision ad targeting and now the latest technology is being deployed to prop it up for a new, more skeptical wave of marketing spend.

Of course, you need a model with ten trillion parameters to tell you what people like. How else could you justify the billion dollars you spent training him!