Selling the assets could be challenging, because the big tech and media companies that could afford such a deal—from

Microsoft Corp.

to

Comcast Corp.

—might face antitrust hurdles of their own, the executives said. Some observers said that a spinoff would be more likely and that the resulting enterprise could become an acquirer in an ad-tech business crowded with small firms.

“It has to be a business that’s viable on its own, and is more likely to be an acquirer than an acquiree,” said Dave Morgan, chief executive of Simulmedia Inc., a marketing technology company.

Splitting up the Google ad machine would give publishers and advertisers more choice and flexibility to go outside the Google ecosystem without risking any harm to their businesses, some of the executives said. Google declined to comment.

The Justice Department, led by Attorney General

Merrick Garland,

has alleged that Google has abused its market power in online ads and has called on the company to divest itself of two key pieces of its interlocking systems of tools: its advertising server, which publishers use to put ad space on their sites up for sale, and its ad exchange, where automated transactions between buyers and sellers occur in milliseconds, as users’ webpages are loading.

Given how long antitrust legal battles can take to unfold, the timing of any spinoff or sale could be years away, unless Google tries to reach a settlement with the government. Google could also prevail in the case and hold on to its entire ad business.

Google has denied wrongdoing and described the lawsuit as attempting to “pick winners and losers in the highly competitive advertising-technology sector.” The company has hired an arsenal of top lawyers to fight the case.

Rajeev Goel,

co-founder and chief executive of the ad-tech firm PubMatic Inc., said the case, combined with revelations from other legal battles Google is facing, is nudging potential customers to explore alternatives to Google.

“This case will take a long time, but the implications will be felt much sooner,” he said. PubMatic was named in the government’s suit as a victim of Google’s allegedly anticompetitive conduct.

Google’s ad server is by far the industry’s behemoth, used by 90% of large publishers, while its ad exchange has a more than 50% market share, according to the government’s complaint. In recent years, Google has merged these tools into a single product called Google Ad Manager, which, together with Google’s mobile-ad and network businesses, booked $31.7 billion in revenue in 2021.

Several large tech and media companies are pushing into the online-ad business and could accelerate their efforts by buying a piece of Google’s systems.

Comcast, whose FreeWheel unit is one of the biggest players in video-ad sales, could be interested in Google’s assets, but some executives worry about winning over antitrust enforcers, according to people familiar with the situation.

Microsoft last year purchased the ad-tech company Xandr, augmenting a digital- advertising business that touches its search engine Bing, the professional networking site LinkedIn and the digital-media company MSN.

Amazon.

com Inc. has rapidly become a major player over the past few years, taking market share from Google and

Facebook.

Netflix Inc.,

which recently launched a tier of its service that features ads, has joined with Microsoft in the early going but indicated an interest in developing its own ad technology down the line.

Microsoft and Netflix didn’t respond to requests for comment.

Amazon

and Comcast declined to comment.

Big companies are already battling on various fronts with regulators in Washington and Europe. “The last thing Big Tech companies would want to do is expose their businesses to DOJ regulators,” said Richard Kramer, the main founder of Arete Research, an equity-research firm focused on the tech sector.

The ad-tech industry has many companies that compete with Google to broker ad deals. Companies such as Index Exchange Inc.,

Magnite Inc.

and OpenX Technologies Inc. run marketplaces like Google’s AdX exchange.

Companies that specialize in tools to help advertisers purchase ad space—known as “demand-side platforms”—could find it appealing to buy Google’s sell-side assets, the ad server and the marketplace. The Trade Desk Inc., the largest independent ad company with buying tools, has a market capitalization of around $25 billion.

Smaller players and private-equity firms would likely have a hard time coming up with the capital to pull off a deal for Google’s assets, several industry executives said.

“It needs to be a spinoff to Alphabet shareholders,” said Terence Kawaja, founder of LUMA Partners, an investment bank focused on the digital-media and advertising sectors.

Splitting off Google Ad Manager from the rest of Google would likely deal a blow to the business itself, said Ari Paparo, founder and chief executive of the advertising and marketing-media company Marketecture Media Inc. and a former Google ad executive.

As the Justice Department outlined in its complaint, the connections between Google’s tools were one reason for the company’s dominance. Publishers who use Google’s ad server and exchange to sell their ad space are able to plug into the fire hose of demand that comes from Google’s ad-buying tools. A 2019 Wall Street Journal article explained those connections in detail.

“If you remove all of the demand, you could reduce the revenue very significantly, if you don’t do it carefully,” Mr. Paparo said.

Justin Wohl, chief revenue officer of the web publication

Salon.com

LLC, said he currently feels no choice but to use Google’s ad server to get the 25% of his ad revenue that comes from Google’s ad exchange, AdX. A breakup of the Google assets might allow him to switch ad servers while still being able to sell in Google’s ad exchange.

“My hope as a publisher is that some back office at Google has already been working on this divestment,” he said.

—Jessica Toonkel contributed to this article.

Write to Patience Haggin at patience.haggin@wsj.com and Keach Hagey at Keach.Hagey@wsj.com