More public healthcare companies will be taken private. Here's what PE firms are betting on.

Experts think private equity firms could seek out more big healthcare acquisitions this year.

Prostock-Studio/Getty Images

• A public healthcare company just announced it's getting bought for $8.9 billion.
• It's the third healthcare player to go private through a buyout this summer.
• Experts say falling debt rates and private equity interest in healthcare AI could spur big deals.

Revenue cycle management company R1 RCM just announced its acquisition by two private equity firms for a whopping $8.9 billion — signaling a long-awaited resurgence of big healthcare deals.

It's the third public healthcare company to say it's taking a buyout and going private this summer. Medical scribe company Augmedix announced its acquisition by $6 billion startup Commure in July, and digital health-tracking platform Sharecare said in June PE firm Altaris would buy it.

Experts say the recent string of take-privates is good news for startups. M&A activity has been ticking up since the beginning of the year, but hasn't rebounded from 2023 lows as quickly as dealmakers hoped.

Now, healthcare startups dreaming of an IPO might be encouraged to stay private longer with renewed chances for big returns through M&A.

"Every time we see one of these bigger deals happen, it's a really powerful validation, because firms know they're going to be in it for a little bit longer term," said David Marks, a private-equity-focused partner at Holland & Knight. "They're only willing to do it if they really have a high degree of confidence in it."

Private equity's return to the scene

The debt market is finally correcting for the types of deals healthcare companies have been waiting for.

Interest rates are coming down for the large amounts of debt necessary for firms to conduct leveraged buyouts, or LBOs, said Thad Davis, a senior managing director at Leerink Partners.

"Cost of capital in the LBO market is declining right now, and equity values have ticked up but haven't gotten super hot. There's an opportunity to take these companies private," Davis said.

Commure CEO Tanay Tandon. Commure was last valued at $6 billion after merging with AI-powered billing startup Athelas in October.

Athelas

The Sharecare and Augmedix deals were much smaller than R1 RCM's, at $518 million and $139 million, respectively. And Commure, Augmedix's buyer, isn't a private equity firm, although it's been unusually acquisitive for a healthcare startup, making six deals since its launch out of stealth in 2020.

Still, debt rates are expected to keep falling, which should open up more windows for healthcare companies and private equity firms alike to make deals of all sizes.

PitchBook Data's lead healthcare analyst Rebecca Springer thinks we'll likely see more large deals like R1 RCM's buyout this year.

"Folks have been sitting on the sidelines with dry powder, and are feeling more confident about the interest rate environment going forward," she said.

Where we'll see more deals

Private equity firms looking for more deals right now are interested in many of the same sectors that VCs are right now — especially AI.

In healthcare, startups have emerged with AI-driven profit structures that are more up private equity's alley, with less cash burn and a theoretically quicker path to breakeven.

Revenue cycle management has long been a hot area for private equity firms. Medical scribe companies like Augmedix are also picking up steam as dozens of clinical documentation players compete for health system contracts.

Abridge CEO Dr. Shiv Rao. The medical scribe startup raised a $150 million Series C round in February, and has since announced collaborations with healthcare giants Epic and Mayo Clinic.

Abridge

Springer said she'd put pharma services companies at the top of the list of potential acquisition targets right now for private equity, followed by healthtech players, and then behavioral health and home-based care providers.

Springer said she thinks private equity firms are "very unlikely to buy a care delivery company in this environment."

And while bigger deals may be coming back, she said she also anticipates more discounted acquisitions of venture-backed digital health companies that notched too-high valuations during healthcare's funding boom.

"I'm seeing a lot of companies that took a couple of rounds of venture funding and then couldn't quite get to the next revenue milestone, sell to a private equity-backed platform or sell directly to private equity if they're willing to meet in the middle on valuation," she said. "I think you'll see more of those VC-to-PE deals as the venture landscape shakes out and as PE looks more actively at areas like digital health."

Skirting regulatory scrutiny

A stricter regulatory environment has prompted some dealmakers to sit on the sidelines rather than risk getting a deal blocked.

Private equity's deepening presence in healthcare has been a central focus of that scrutiny, after nearly two decades of buying up hospitals and raising prices. Two congressional bills introduced this year looked to block private equity rollups in healthcare; a fresh wave of criticism followed PE-owned Steward Health's bankruptcy in May and its hospital closures announced last month.

Sen. Markey introduced the Health Over Wealth Act in July that would limit private equity investment in healthcare providers.

Brian Snyder/Reuters

But Holland & Knight's Marks said private equity leaders he's spoken with recently aren't necessarily deterred by regulatory scrutiny as they look for their next buys.

He said the acquisition of R1 RCM, which sells a tech platform rather than healthcare services, simply reflects the growing traction of AI-powered tech for healthcare administration rather than an effort by private equity firms to seek out FTC-friendly deals.

"When I've asked them point blank, they're seeing the regulatory shifts and they know they need to account for it, but it's not really fundamentally changing their investment pool," Marks said. "I think it's just a true, organic interest in the spaces."

Read the original article on Business Insider

from Business Insider https://ift.tt/SIHLQpu

Reddit cofounder reveals his best-ever investment — and his biggest flops

Reddit cofounder Alexis Ohanian talked about his single best investment in a recent podcast interview.

Noam Galai/Getty Images

• Alexis Ohanian said his early investment in Ethereum's pre-token sale is now worth $17.1 million.
• The Reddit cofounder called it his best-ever investment.
• Ohanian cited past investments in Zenefits and Clubhouse as two that didn't pan out.

Alexis Ohanian recently revealed his single-best investment — and also some financial moves he now regrets.

In a podcast interview with venture capitalist Harry Stebbings, the Reddit cofounder said that he made millions from investing early in Ethereum.

Ohanian said he invested $10,000 at the company's pre-token sale — which turned out to be a highly lucrative move as the company's cryptocurrency, ether (ETH), is now the second largest in the world by market cap.

At the time of the podcast recording, Ohanian said his ether stake is now worth $17.1 million.

The SevenSevenSix venture capital firm founder also said that, along with some ether, he also sold a lot of bitcoin — another profitable move for a more romantic reason.

"I sold it for a wedding and engagement ring and all that stuff, so it was a good investment," he said.

Ohanian got engaged to tennis champion Serena Williams in 2016, and the two married the following year.

Biggest investment regrets

While he successfully predicted Ethereum's rise, financial risks don't always pay off — and Ohanian talked about some examples of when he wished he had exited an investment sooner.

Ohanian said one of his biggest investing regrets is Clubhouse, the social audio app that gained mass popularity during the COVID-19 lockdown. The startup raised over $100 million and at one point enjoyed a reported $4 billion valuation.

The app, which reached over 10 million downloads, inspired competing features such as Spaces on X, then Twitter, and Spotify's Greenroom.

"I had like the Series A," Ohanian said. "And I remember when that multibillion-dollar round came up, I actually put more money in because I'm an idiot."

However, the post-pandemic return to normal life and outdoor socializing hit Clubhouse particularly hard, with daily downloads and users dropping significantly by 2021.

But the "one that hurt the most" for Ohanian was the online HR platform TriNet Zenefits, formerly called Zenefits.

After launching in 2013, Zenefits quickly became one of Silicon Valley's unicorn startups. Within two years, the company raised over $500 million and was valued at $4.5 billion until it started unraveling amid various controversies.

In 2016, Fidelity marked down the value of several startups it had investments in, including Zenefits, by as much as 38%, The Wall Street Journal reported.

"We could have returned the fund once over whenever the Fidelity markdown was," Ohanian said. "Literally putting an SPV together to sell Zenefits, and then the Fidelity markdown happened, and it killed the SPV."

Read the original article on Business Insider

from Business Insider https://ift.tt/fUgudt0

A private-equity takeover is coming for startups

The exit doors are closing for venture-backed software companies.

Getty Images/Alyssa Powell/BI

• More startups are about to get gobbled up by private equity.
• A sluggish economy has fewer corporates making acquisitions. Plus the IPO market is looking shaky.
• Tech-focused private equity is seizing the moment, with roughly 59 buyouts in the first quarter.

Scott Arnold was preparing to take his company, AuditBoard, a Southern California provider of audit and risk management software, to the public markets.

Hg had other ideas. The European private equity firm had been eyeing the late-stage startup for five years, impressed by its traction and focus on customer success. When word got around that AuditBoard was seeking potential investors for its initial public offering in early March, Hg pounced. It wanted to snap up the company and spur on the next phase of growth, and it was willing to pay through the teeth for it.

The parties hashed out an offer over a half hour at an airport bar. Four weeks later, they landed on a sale price of roughly $3 billion, over 20 times AuditBoard's valuation when it last raised primary funding from VCs.

The speed of the deal, and the flashy multiple, illustrate how private equity funds are able to swallow startups more often lately. Though many founders still pine for an exit in the public markets, a period of high interest rates has late-stage startups staying private longer. Meanwhile, an antitrust crackdown, especially in Big Tech, has starved some venture-backed companies of another way to exit.

The exit freeze has some founders giving private equity a second look, said Michael Brown, a general partner at Battery Ventures, a firm that backs companies at all stages, from seed and early to growth and buyout. Battery was also AuditBoard's largest institutional shareholder.

"They move very quickly. They're actually paying very good prices for things — as attractive as a strategic," Brown said, adding, "and you get immediate liquidity, whereas if you go public, management can't just sell day one."

PitchBook data shows software buyouts are on the rebound, with an estimated 59 deals in the first quarter. That might not sound like much, but it's significant as a growing share of all outcomes, said Derek Hernandez, a PitchBook senior emerging technology analyst.

The number of corporate mergers and acquisitions of software companies has fallen to about 20% below pre-pandemic levels, while software buyouts are trending toward a five-year high, according to PitchBook.

The software-as-a-service category is especially ripe for consolidation, said Aaron Fleishman, a software investor at Tola Capital. The software market exploded during the pandemic, with people working from home and businesses spending more on all things cloud. But in the face of rising inflation and interest rates, software customers from tech startups to mom-and-pop shops trimmed their budgets.

Fleishman noted that when software spending saw a significant pullback, many companies with subscription-based revenues in the $20 million to $50 million range found themselves at an impasse. Their slowing growth made it difficult for these companies to appeal to new venture investors. They're not big enough to go public, and they're not likely to be picked up by an incumbent.

"There's not a lot of buyers for those assets because it just feels like the last generation at this point," Fleishman said.

Their tricky spot is private equity's gain. Firms like Thoma Bravo or Vista Equity Partners — leading providers of private equity in tech — may buy a company, strip it, build it to a few hundred million of annual recurring revenue, and flip it or take it public. They might also bolt on other businesses to create a new Frankenstein software giant.

"You're going to see a lot of that consolidation over the next year, two years," Fleishman said.

The bought-out becomes the buyer

In this market, some software companies are looking to sell to private equity more. Others are doing the buying.

This spring, Metropolis, a startup building a better parking app, followed private equity's playbook by taking private SP+, one of the largest parking networks in North America.

Yoni Rechtman, an investor at Slow Ventures, which first backed Metropolis at the seed stage, said the growth buyout has helped Metropolis not only expand its market presence but also capture more value. "Owning the assets means owning all the upside," Rechtman said.

The deal follows the October news that Metropolis raised $1.8 billion in funding led by Eldridge Industries, a provider of equity and debt financing. Flush with cash, Metropolis is effectively "using our equity to acquire companies rather than acquire customers," Rechtman said.

But it's not just startups mimicking private equity. Sequoia changed its model so it could hold on to public companies longer. General Catalyst bought a healthcare system. Andreessen Horowitz plans to invest in the private equity asset class through its family office division.

These trends point to the worlds of private equity and venture capital colliding in new ways. Startups are caught in the middle, weighing the allure of faster liquidity against the traditional dream of going public.

With high interest rates and a liquidity crunch reshaping the landscape, private equity is seizing the moment.

Read the original article on Business Insider

from Business Insider https://ift.tt/rnBuZcX

Californians are buying fewer Teslas. That could be a bad sign for Elon Musk.

Tesla registrations fell in California.

Newsday LLC/Getty Images

• Tesla's new car registrations in California dropped 24% in the second quarter.
• Governor Gavin Newsom said Tesla is finally starting to see competition from other companies in California.
• Tesla's troubles in California could be a symptom of a larger issue, experts say.

Tesla has a California problem — its residents aren't buying as many of Elon Musk's electric cars as they used to.

Tesla's new car registrations in the Golden State dropped for the third straight quarter, with a 24% fall between April and June, according to the California New Car Dealers Association.

And California isn't the only place where Tesla's chokehold on the EV market is beginning to slip. The carmaker's delivery numbers slumped for the second straight quarter in July and Tesla's slice of the US EV market fell below 50% for the first time.

Tesla CEO Elon Musk has attributed lackluster sales to high interest rates and economic uncertainty in the US. But, while Tesla's sales are dropping, the EV market as a whole is expanding, including in California.

Data from Cox Automotive shows that total EV sales climbed 23% from the previous quarter and more than 11% compared with the previous year. Similarly, data from CNCDA shows that EV sales in California hit their second-highest quarter on record in July.

'Dramatic shift'

California Gov. Gavin Newsom made light of Tesla's declining popularity and hyped up Rivian and Ford's progress in the state this week.

"Tesla is not the exclusive manufacturer any longer in this space," Newsom said in comments reported by Forbes. "You're seeing a dramatic shift in competition across the sector. This is exactly what was predicted. This is exactly what we have been promoting: competition in the space."

California Gov. Gavin Newsom.

Noam Galai/Getty Images

Tesla did not respond to a request for comment from Business Insider.

California is the largest EV market in the US and accounts for nearly a third of electric car sales, according to data from Experian Automotive.

Meanwhile, Musk has attempted to distance himself from the state. In July, Musk said on X that he planned to move SpaceX and X's headquarters out of California. The billionaire has been critical of Newsom and cited a bill that was signed by the governor in July that prohibits "forced disclosure" rules in public K-12 schools as his reasoning for leaving the state.

Musk has become increasingly vocal about politics in recent years and it's unclear whether that may have had some sway on sales in California — a Democratic Party stronghold.

On Tuesday, European drugstore chain Rossman announced plans to stop buying Teslas due to Musk's support for former president Donald Trump.

Earlier this year, some Tesla owners even said they were considering ditching the brand after Musk made a problematic post on social media and some have taken to putting "I bought this before we knew Elon was crazy" bumper stickers on their Teslas.

"For most people, Elon's politics wouldn't be an issue, but when people who have purchased are trying to disassociate that says something. Would that person really want to buy another one?" said Ivan Drury, an automotive analyst for Edmunds.

"It could be the thing that pushes someone over the edge who's a little on the fence," he added. "Do they really want to get wrapped up in that when there's competent competition?"

And Tesla might not be keeping up with the competition.

Lack of updates

"In California there's always the question of 'is this cool? is this the newest thing?' And Tesla really hasn't done much to keep up with that," Auto Trader editor Brian Moody told Business Insider.

The carmaker hasn't updated many of its vehicles in years and its Cybertruck isn't exactly geared toward the average consumer.

What's more, California registration numbers are just another sign that traditional automakers are on Tesla's heels, Moody said. "The companies that have had a century of experience manufacturing things in the real world, engineering things in the real world, marketing and distributing those things in real time, now they're catching on."

Do you work for Tesla or have a tip? Reach out to the reporter via a non-work email and device at gkay@businessinsider.com or 248-894-6012.

Read the original article on Business Insider

from Business Insider https://ift.tt/qnUBWyz

A Gen Z data scientist says AI jobs require more than coding — and don't expect to be a loner at work

Pranjali Ajay Parse said AI jobs require robust skills and the ability to collaborate with many teams.

Pranjali Ajay Parse

• Data scientist Pranjali Ajay Parse explained what a role in AI looks like in reality.
• She said AI roles require a number of interdisciplinary skills and wide-range collaboration.
• Parse said employees also need to consider privacy concerns and ethical implications of their work.

There's more to an AI job than simply writing code.

Just ask 25-year-old Pranjali Ajay Parse, who works as a data scientist for Autodesk. She's been developing an AI tool that provides employees with insights into their work patterns, such as meeting trends and work routines.

After getting her master's degree in computer science and working at Autodesk for over a year, Parse has been able to grasp what it's like to actually work in an AI role — and she said it's not what people may expect.

Parse said that working in AI is largely interdisciplinary and dependent on collaboration; and while you may be working in tech, the job also requires a heavy focus on ethics. In a conversation with Business Insider, she debunked some of the myths about AI roles.

It's not just coding

Pranjali said proficiency in Python won't cut it if you're looking for a job in AI.

Parse said candidates don't necessarily need a degree in AI to get a job in the field. But she said you need to know how to do case study analysis, SQL querying, and coding. She said candidates can try boot camps or personal projects to skill up in those areas.

"AI is inherently interdisciplinary," Parse said. "It draws from various domains, including mathematics, computer science, statistics, and domain-specific knowledge."

Parse said about 70% of her job is data science, which requires reviewing and analyzing data sets. She said the rest of her time is split between software engineering, building pipelines, data engineering, architectural design, and a lot of math.

Parse also added that it's important to stay updated with advancements in related fields because the technology is constantly evolving.

AI roles are often highly collaborative

Software engineers have been known to be loners, but don't count on solitude if you're working in AI.

While some engineering roles tend to be independent, Parse said, "AI projects are rarely done solo." Part of this is because AI is a new technology that requires collaboration among a variety of teams and stakeholders, she said.

For example, Parse said she has to interact with seven or eight teams to build an AI recommendation system project.

In her experience, the process begins with data collection and preparation by a data analysis team. Then, data scientists apply statistical methods and modeling. The machine learning team then develops and refines the model. Once the model is ready, UX and UI experts design the user interface, followed by software engineers who build the front end.

Finally, the marketing team determined the product's launch strategy.

"An end-to-end AI project requires a lot of communication and collaboration," Parse said.

You need to be thinking about ethics

Privacy teams are often deeply embedded in the process when sensitive data is handled during AI development.

Parse said the privacy protocols are extensive. When working with a person's data, employees need to receive permission for tasks. Projects also require robust production measures, like pseudonymizing identities and ensuring models don't "inadvertently recreate biases or create inequitable outcomes."

This requires complying with legal and regulatory requirements, she said. It also means thinking about the long-term implications of projects, including potential unintended consequences and ethical dilemmas.

While privacy may seem like an obvious consideration for those working in AI, Parse said it can be easy to get caught up in how the models perform. Also, since so many teams contribute to the product, it can be easy to focus on your specific task rather than the overarching implications, she added.

Parse said it's up to companies to train employees on proper privacy and ethical guidelines. But it's also important for employees to consider a third-person perspective on the work they're doing.

Read the original article on Business Insider

from Business Insider https://ift.tt/IPj5Lsu

A top 1% fund manager over the last 15 years warns a selling spiral threatens to sink the market — and shares 4 stocks he's betting on to capitalize on the turmoil

Johannes Eisele/AFP via Getty Images

• Bill Smead warns of a major market bubble, predicting poor long-term S&P 500 performance.
• Smead cites high stock holdings in household portfolios and elevated valuation measures.
• He suggests investing in homebuilders and oil stocks to capitalize on a falling rate environment.

Bill Smead doesn't like where the market sits right now.

Yes, the S&P 500 is just off of all-time highs following a 36% percent rally from October to July. Yes, earnings growth is strong. But Smead is skeptical about how long the good times can last.

In an interview with Business Insider on Wednesday, Smead said we're in one of the largest bubbles in market history.

"We view the today's circumstances as the fourth major financial euphoria of the last 100 years: 1929, 1969, 1999, and now," Smead, who manages the Smead Value Fund (SMVLX), said. The fund has outperformed 99% of similar funds over the last 10- and 15-year periods, according to Morningstar data.

There are plenty of data points backing up Smead's outlook. He cited one in an August 6 note: the percentage of stock holdings making up household portfolios. Right now, it's at its highest-ever level.

Smead Capital Management

Or take two famous valuation measures, for example.

The Shiller cyclically adjusted price-to-earnings ratio, which is a 10-year moving average of stock valuations, exceeds levels in 1929 and trails only its 2021 and 1999 heights.

GuruFocus

And the so-called Warren Buffett Indicator, or total market cap-to-GDP, hovers just below its all-time highs.

GuruFocus

The extended nature of the current market make Smead certain of one thing — the S&P 500 will perform dysmally over the next decade or longer. While valuations are poor predictors for near-term market performance, they tend to be the most important factors in deciding where stocks go in the long-term. According to a Bank of America analysis, 80% of the market's performance over a 10-year period can be attributed to starting valuations.

"We're completely convinced that people will not make money in the next 10-15 years in the S&P 500 index, other than for trading purposes," Smead said.

"History bears that out," he continued. "From '64-71 the Dow went nowhere. And from 1999 to 2009 the S&P 500 made no money."

But Smead has also become bearish on the near-term prospects of the market, largely due to the weight of tech stocks in the index. With poor tech-sector earnings recently, the sector has endured its worst period since 2022. From June 10 to August 7, the Technology Select Sector SPDR (XLK) Fund fell 17%.

Given the sector's 31% weight in the S&P 500, Smead thinks its poor performance will continue to drag down the index, causing more investors to sell the index, further hurting tech stocks, and so on.

"Selling begets more selling and feeds on itself. Major stock prices begin to perform poorly, which drags down the S&P 500 Index, which in turn causes the selling of the index. The index selling puts a drag on the largest-cap stocks, which adds more legs to the index selling."

What's more, inflation is "entrenched" in the economy, he believes, and once the Federal Reserve begins to lower interest rates in an effort to stave off a recession, prices will begin rising more quickly again. This will hurt the market's valuation, which currently sits at 21x earnings, well above its long-term average of 14x.

Stocks Smead is betting on

Smead's views align him with ultra-bearish money managers like Jeremy Grantham and John Hussman, who both believe stocks are in a bubble and will deliver lackluster returns over the long-term.

While Smead may turn out to be right over the next decade-plus — a duration to which his track record lends credibility — it's difficult to say whether his near-term skepticism about the market will play out. The S&P 500 is up 49% since late 2022 as bears have continued to warn of downside.

Stocks suffered from ongoing selling for a few days earlier this month as July's jobs report came in cooler than expected, raising questions about the economy's health. But the market has rebounded in recent days, and investors will wait for more data to confirm whether or not labor market deterioration is substantial.

If stocks do indeed continue to sell-off, Smead has a couple of ideas to capitalize on the downside. First, he likes homebuilders, which he believes will benefit from lower mortgage rates as the Fed gets ready to slash its overnight lending rate for the first time since its aggressive hiking cycle began in 2022.

Two in particular that Smead is betting on are DR Horton (DHI) and Lennar (LEN).

Second, lower interest rates would spark demand and could cause inflation to resurface, which would be good for oil stocks, Smead said. He has positions in Occidental Petroleum (OXY) and Apache (APA).

Read the original article on Business Insider

from Business Insider https://ift.tt/YJxAp87

The AI world's most valuable resource is running out, and it's scrambling to find an alternative: 'fake' data

AI leaders like ChatGPT boss Sam Altman are hoping synthetic data will make their AI models smarter.

Andrew Caballero-Reynolds/AFP/Getty Images

• The AI industry has a major problem: the real-world data used to make smarter models is running out.
• Companies scrambling for an alternative think synthetic data could offer a solution.
• Research suggests synthetic data could poison AI with low-quality information.

The AI world is on the cusp of running out of its most valuable resource — and it's leading industry leaders into a fierce debate over a fast-growing alternative being touted as a replacement: synthetic data, or essentially "fake" data.

For years, the likes of OpenAI and Google have scraped data from the internet to train the large language models (LLMs) that power their AI tools and features. These LLMs digested reams of text, video, and other media online produced by humans over centuries — be it research papers, novels, or YouTube clips.

Now, the supply of "real," human-generated data is running dry. Research firm Epoch AI predicts textual data could run out by 2028. Meanwhile, companies that have mined every corner of the internet for usable training data — sometimes breaking their policies to do so — face increased restrictions on what remains.

To some, that's not necessarily a problem. OpenAI CEO Sam Altman has argued that AI models should eventually produce synthetic data good enough to train themselves effectively. The allure is obvious: training data has become one of the most precious resources in the AI boom, and the possibility of generating it cheaply and seemingly infinitely is tantalizing.

Still, researchers debate whether synthetic data is the magic bullet, with some arguing this path could lead to AI models poisoning themselves with poor-quality information and a "collapse" as a result.

A recent paper published by a group of Oxford and Cambridge researchers discovered that feeding a model with AI-generated data eventually led it to produce gibberish. AI-generated data was not unusable for training, the authors claimed, but it should be balanced with real-world data.

As the well of usable human-generated data dries up, more companies look into using synthetic data. In 2021, research firm Gartner predicted that by 2024, 60% of data used for developing AI would be synthetically generated.

"It's a crisis," said Gary Marcus, an AI analyst and emeritus professor of psychology and neural science at New York University. "People had the illusion that you could infinitely make large language models better by just using more and more data, but now they've basically used all the data they can."

"Yes, it will help you with some problems, but the deeper problem is that these systems don't really reason, they don't really plan," Marcus added. "All the synthetic data you can imagine is not going to solve that foundational problem."

More companies create synthetic data

The need for "fake" data hinges on the notion that real-world data is fast running out.

This is partly because tech firms have been moving as fast as possible to use publicly available data to train AI in an effort to outsmart rivals. It's also because online data owners have become increasingly wary of companies taking their data for free.

OpenAI researchers revealed in 2020 how they used free data from Common Crawl, a web crawler that contains "nearly a trillion words" from online resources, to train the AI model that would eventually power ChatGPT.

Research published in July by MIT's Data Provenance Initiative found websites now putting restrictions in place to stop AI firms from using data that didn't belong to them. News publications and other top sites are increasingly blocking AI companies from freely cribbing their data.

To get around this problem, companies such as OpenAI and Google are cutting checks worth tens of millions of dollars for access to data from Reddit and news outlets, which act as conveyor belts of fresh data for training models. Even this has its limitations.

"There are no longer major areas of the textual web just waiting to be grabbed," Nathan Lambert, a researcher at the Allen Institute for AI, wrote in May.

This is where synthetic data comes in. Rather than being pulled from the real world, synthetic data is generated by AI systems that have been trained on real-world data.

In June, for instance, Nvidia released an AI model that can create artificial datasets for training and alignment. In July, researchers at Chinese tech giant Tencent created a synthetic data generator called Persona Hub, which does a similar job.

Some startups, such as Gretel and SynthLabs, are even popping up with the sole purpose of generating and selling troves of specific types of data to companies that need it.

A chat powered by Meta's Llama 3 AI model.

Anadolu/Getty Images

Proponents of synthetic data offer fair reasons for its use. Like the real world, human-generated data is often messy, leaving researchers with the complex and laborious task of cleaning and labeling it before it can be used.

Synthetic data can potentially fill holes that human data cannot fill. In late July, Meta introduced Llama 3.1, a new series of AI models that generate synthetic data and rely on it for "finetuning" in training. In particular, it used the data to improve the performance of specific skills, such as coding in languages like Python, Java, and Rush, as well as solving math problems.

Synthetic training could be particularly effective for smaller AI models. Microsoft last year said it gave OpenAI's models a diverse list of words that a typical 3-4 year old would know, and then asked it to generate short stories using that data. The resulting dataset was used to create a group of small but capable language models.

Synthetic data may help offer some effective counter-tuning to the biases produced by real-world data, too. In their 2021 paper, "On the Dangers of Stochastic Parrots," former Google researchers Timnit Gebru, Margaret Mitchell, and others argued that LLMs trained on massive datasets of text from the internet would likely reflect the data's biases.

In April, a group of Google DeepMind researchers published a paper championing the use of synthetic data to address problems around data scarcity and privacy concerns in training, adding that ensuring the accuracy and lack of bias in this AI-generated data "remains a critical challenge."

'Habsburg AI'

While the AI industry found some advantages in synthetic data, it faces serious issues it can't afford to ignore, such as fears synthetic data can wreck AI models entirely.

In Meta's research paper on Llama 3.1, the company said that training the 405 billion parameter version of the latest model "on its own generated data is not helpful," and may even "degrade performance."

A new study published in the journal Nature last month found that "indiscriminate use" of synthetic data in model training can cause "irreversible defects." The researchers called this phenomenon "model collapse" and warned that the problem must be taken seriously "if we are to sustain the benefits of training from large-scale data scraped from the web."

Jathan Sadowski, a senior research fellow at Monash University, coined a term for this idea: Habsburg AI, in reference to the Austrian dynasty that some historians believe destroyed itself through inbreeding. Since coining the term, Sadowski told BI he has felt validated by the research backing his assertion that models heavily trained on AI outputs can become mutated.

"The open question for researchers and companies building AI systems is how much synthetic data is too much?" said Sadowski. "They need to find any possible solution to overcome the challenges of data scarcity for AI systems—even if those solutions are just short-term fixes that could do more harm than good by creating low-quality systems."

However, findings from a paper published in April showed that models trained on their own generated data don't necessarily need to "collapse" if they are trained with both "real" and synthetic data. Now, some companies are betting on a future of "hybrid data," where synthetic data is generated by using some real data in an effort to stop the model going off-piste.

Scale AI, which helps companies label and test data, said the company is exploring "the direction of hybrid data," using both synthetic and non-synthetic data (Scale AI CEO Alexandr Wang recently declared: "Hybrid data is the real future.")

In search of other solutions

AI may require entirely new approaches, as simply jamming more data into models may only go so far.

A group of Google DeepMind researchers may have proven the merits of another approach in January when the company announced AlphaGeometry, an AI system that can solve geometry problems at an Olympiad level.

In a supplemental paper, the researchers explained how AlphaGeometry uses a "neuro-symbolic" approach, which meshes the strengths of other AI approaches, landing somewhere between data-hungry deep-learning models and rule-based logical reasoning. IBM's research group said it sees it as "a pathway to achieve artificial general intelligence."

What's more, in the case of AlphaGeometry, it was pre-trained on entirely synthetic data.

The neuro-symbolic field of AI is still relatively young, and it remains to be seen if it will propel AI forward.

Given the pressures companies such as OpenAI, Google, and Microsoft face in turning AI hype into profits, expect them to try every solution possible to solve the data crisis.

"People had the illusion that you could infinitely make large language models better by just using more and more data, but now they've basically used all the data they can," said Marcus. "We're still basically going to be stuck here unless we take new approaches altogether."

Read the original article on Business Insider

from Business Insider https://ift.tt/DHkU84M