UAE’s Technological Innovation Institute (TII) yesterday released Falcon 180B, a highly scaled-up version of Falcon 40B. According to the official blog post, this is the largest open-source language model, boasting a staggering 180 billion parameters.
According to TII, Falcon 180B is trained on 3.5 trillion tokens on 4096 GPUs simultaneously, using Amazon SageMaker for a total of ~7,000,000 GPU hours.
To put it in perspective, Falcon 180B is 2.5 times bigger than Llama 2 and required four times more computational power for its training. It’s certainly intriguing how UAE’s TII manages to obtain such substantial computing power.
UAE has Oil Money
The UAE, as an oil-rich nation, has ample financial resources at its disposal. According to a report hydrocarbons continue to play a critical role in the UAE economy, with 30% of the UAE’s GDP directly based on oil and gas industry and 13% of its exports.
The UAE is allocating the money earned from oil to fund AI projects. Six years ago, they launched the National Strategy for AI 2031, aiming to make AI contribute significantly to their economy, targeting up to 13.6 percent of their GDP by 2030.
In 2020, UAE government established ARTC (Advanced Technology Research Council) to promote scientific research and innovations in AI. Few months later ARTC established TII which today is behind the creation of Falcon 180B.
There is no doubt the UAE is bullish on investing in AI initiatives. In June, when Openai CEO Sam Altman visited Abu Dhabi, he praised the nation’s foresight in recognizing the potential of AI, stating that the city “has been talking about AI since before it was cool.”
While the world has been struggling to Procure NVIDIA GPUs, UAE secured access to thousands of NVIDIA chips which it used to build the Falcon model in May. Moreover, the report added that UAE wants to control and own its own computational power and talent without depending on Chinese or Americans. There is no doubt that they have capital, energy resources and talent to do it.
Similarly, Saudi Arabia also purchased no less than 3,000 of NVIDIA’s H100 chips. These processors are valued at $40,000 each. The acquisition was facilitated through the public research institution, King Abdullah University of Science and Technology (KAUST).
When we crunch the numbers, it becomes apparent that Saudi invested a staggering $120 million to secure this impressive array of GPUs.
This is the reason why when the U.S tried to ban exports of AI chips to middle east nations, AMD and NVIDIA both raised eyebrows. All the major economies of the world right now are engaged in the LLM race which has led to cold war with the US trying its best that its domestic AI chip manufacturers do not lend support to their competitors.
Not only this, UAE’s G42 recently launched Arabic language AI model Jais which contains 13 billion parameters. Jais was created with the help of supercomputers produced by the Silicon Valley-based Cerebras Systems for which it had signed a $100 million deal with G42. As NVIDIA’s chips were short in supply, UAE was smart enough to seek alternatives.
Moreover, G42 in 2021 raised $800 million from U.S. tech investment firm Silver Lake, which has backing from Mubadala, the UAE’s sovereign wealth fund.
What about OpenAI?
Coming to OpenAI, the company’s progress is largely dependent on the multi-billion dollar investment it received from Microsoft at the beginning of the year. However, with the recent developments it appears that it has exhausted the investment. Recently, Sam Altman posted on X that the company is not coming up with GPT-5 or GPT- 4.5 in the near future and asked people to calm down.
According to The Information report, OpenAI losses roughly doubled to around $540 million last year as it developed ChatGPT and GPT-4. According to reports, training GPT-3 with 175 billion parameters cost them more than $4 million.
Now with GPT-4 rumoured to have about 1.76 trillion parameters, the cost of building the model comes up close to $46.3 billion, assuming a linear increase in cost per parameter. Again, this is a simplified estimate, and the actual cost may vary based on various aspects, including research and development costs, talent, hardware improvements, and more.
This explains why OpenAI has been shying away from releasing the multimodal capabilities of GPT-4 into the public, or disclosing the parameter size, which the team seem to be hiding deliberately to avoid unwanted attention. Who knows, OpenAI fooled us all and we never actually got GPT-4.
Altman previously had suggested OpenAI may try to raise as much as $100 billion in the coming years to achieve its aim of developing AGI. Maybe, OpenAI should also attract some oil money, or probably expand to the middle east. Interestingly, Microsoft is already planning to do that.
As of now, OpenAI is trying to attract enterprises in order to stay in business. It announced its inaugural developer’s conference which is supposed to take place in San Francisco on November 6, 2023, where it is hoping developers from around the world will come up with new ideas and tools for ChatGPT and APIs.
The post Soon UAE Will Dethrone OpenAI appeared first on Analytics India Magazine.
