After more than a year of planning and training, a volunteer-led project has produced an open source language model that they say is as powerful as OpenAI’s GPT-3, but free and open to anyone. person (if he has the computing power). . Called Bloom, the model is available in open source along with the code and data sets used to create it. Brooklyn-based AI startup Hugging Face has launched a free web app that allows anyone to try Bloom without having to download it.
Bloom is the creation of BigScience, an international community-driven project aimed at making great natural language models available for research. Large language models, or “LLMs” for short, can translate, summarize, and write text with more or less human nuances. (See GPT-3.) But historically they have been costly to create, keeping them out of the reach of researchers and firmly in the hands of big companies like Meta, Google, and Microsoft.
That is finally changing, thanks in part to the efforts of BigScience. The group’s more than 1,000 volunteer researchers, backed by ethicists, philosophers, lawyers and startup engineers and large technology companies alike, spent months working for Bloom, which competes in scale LLMs made by companies like OpenAI and Alphabet’s DeepMind. One of the largest open source models to work in multiple languages, Bloom is designed to be applied in a number of research applications, such as extracting information from historical texts.
“Bloom is capable of generating text in 46 natural languages and dialects and 13 programming languages,” says a blog post shared with TechCrunch prior to launch. “Although you were never trained in any of these specific tasks, Bloom can be asked to compile summaries or text translations, issue code based on instructions, and follow instructions to perform original tasks such as writing recipes, extract information from a news article or compose sentences with a newly defined invented word … Bloom’s performance will continue to improve as the workshop continues to experiment and advance above Bloom.
BigScience sponsors also hope Bloom will push for new research on ways to combat the problems that affect all LLMs, including bias and toxicity. LLMs tend to cast falsehoods and show prejudice against religions, genders, races, and people with disabilities. They also struggle with the basics of writing, often changing the subject of a conversation without a continuation and repeating themselves non-stop, or even contradicting each other.
“[Bloom] shows the continued power of open source and open science even for big, expensive basic models, “Richard Socher, CEO of You.com and former chief scientist at Salesforce, told TechCrunch by email. he is not involved with BigScience. “It also shows that in AI, no organization has a significant advantage for a long time. Once an organization demonstrates that something is doable, the same capabilities will appear six to 12 months later elsewhere. “
Humble beginnings
The origins of BigScience lie in conversations years ago between the scientific director of Hugging Face, Thomas Wolf, Stéphane Requena of GENCI and Pierre-François Lavallée of IDRIS. The founders envisioned creating software, datasets, LLMs, and tools to explore the social impact of AI, which has only received more attention from the research community in recent years.
Soon, steering committees were formed to offer BigScience members, who came from more than 60 countries and 250 institutions, scientific and general advice, design collaborative tasks, and organize workshops, hackathons, and public events. Different working groups were tasked with addressing challenges such as data governance, the demonstration of theorems in mathematics and archiving strategies, as well as privacy and informed consent and other legal issues.
Bloom is the sum total of his work. He trained with a public-funded computing time (through grants) worth $ 7 million at the supercomputer Jean Zay located near Paris, France, which is among the most powerful machines in the world.
There is a solid discussion in academic circles about the carbon impact of training in AI; data centers are not particularly environmentally friendly. But BigScience says Jean Zay, thanks to his unique cooling system and nuclear power source, was able to train Bloom with a carbon footprint equivalent to a flight from Paris to New York.
Like all language models, Bloom is essentially a statistical tool for predicting words. With a large number of examples of a 1.6 terabyte training dataset, Bloom learned how words are likely to occur based on patterns, including the semantic context of the surrounding text. For example, considering a typical email that ends with the snippet “Looking forward …”, Bloom could complete it with “… to receive a response.”
One of the goals of the BigScience working groups was to collect data representative enough to form Bloom. Due to systemic biases in public data sources, non-English LLMs have traditionally not performed as well as their English counterparts. From books, academic publications, radio transcripts, podcasts, and websites, the 341 billion word set used to form Bloom aims to encode different cultural contexts in different languages, such as Swahili, Catalan, Bengali and Vietnamese.
BigScience groups hand-picked nearly two-thirds of the data set from 500 sources, soliciting suggestions from community groups such as the African natural language processing community Masakhane, LatinX in AI, and Machine Learning Tokyo. They drafted privacy and filtered quality, for example trying to reduce the overrepresentation of pornographic sites, which tend to contain sexist associations.
Bloom is not completely free of bias; no LLM is. But the hope is that by maintaining transparency around training data, it will be easier for researchers to get to the root of Bloom’s predictions and decision-making.
Great size
With 176 billion parameters, Bloom is approximately the size of GPT-3. Machine learning parameters are the parts of the LLM learned from training data and tend to correlate with model effectiveness in a task such as text generation.
In general terms, higher parameter models require more computing power to train. A 2020 study by AI21 Labs set the costs to develop a text generation model with just 1.5 billion parameters up to $ 1.6 million; Bloom trained with 384 Nvidia A100 GPUs for three months. This has made it difficult for the community to use large state-of-the-art language models, such as Microsoft’s and Nvidia’s Megatron-Turing (MT-NLG) natural language generation, which has 530 billion parameters.
BigScience claims that researchers will be able to use Bloom for less than $ 40 an hour at a cloud provider. But with the goal of removing even this access barrier, the organization plans to launch smaller, less hardware-intensive versions of Bloom and is developing a distributed system that will allow labs to share the model between their servers. . An API is also being worked on.
Bloom joins a growing ecosystem of open source and highly capable LLMs with extensive commercial and research uses. In February, AI’s EleutherAI open research group launched GPT-NeoX-20B, which at the time surpassed other public language models at several benchmarks. Months later, open source Meta OPT-175B, which the company claimed was the first 175 billion-parameter language model to be made available to the AI community.
They have made good use of it: companies have already sprung up around EleutherAI models. But some investigators fear abuse. At the University of Maryland, researchers found that it is possible for LLMs to generate false news and cybersecurity reports convincing enough to deceive experts. Another article written jointly by Meta researchers explores the potential harm that could result from LLMs that give poor advice, especially medical or psychological prognosis.
Many companies that offer access to LLM via an API, such as OpenAI, apply filters to remove problematic text. But open source models obviously don’t have those protections.
In recognition of the potential for misuse, Bloom includes documentation describing its capabilities and limitations. Its use requires accepting a legal license that commits researchers not to use the model for malicious purposes. BigScience plans to monitor how the model is applied and adjust the license and documentation as needed.
“We are scheduled to add more languages, make the model smaller so that it is easier to use with the same level of performance, and we will support community efforts to expand it,” the blog post continues. “Bloom is a living family of models that will grow, not a unique model.”