Google has adopted rules similar to ones that Apple Inc. requires for its software that powers smartphones and distributes apps to them. That is getting it into trouble.

The antitrust case brought against Google Wednesday by attorneys general from 36 states and the District of Columbia says the Alphabet Inc. unit’s Android mobile operating system and its app marketplace—the Play Store—have increasingly evolved to become a so-called walled garden. That is a phrase commonly applied to Apple and how the iPhone maker pushes consumers to remain within its ecosystem of hardware, software and services.

Google pitched Android at its outset more than a decade ago as the “first truly open and comprehensive platform for mobile devices.” Unlike Apple, Google licenses Android and its proprietary apps such as the Play Store to mobile device makers. Competing app stores can also be developed on the Android platform.

Those factors helped boost the use of Android, which is active on more than three billion devices world-wide and, according to Statcounter, to the most popular operating system in terms of internet usage.

In their 144-page complaint, the attorneys general accused Google of working to diminish competition in Android app distribution and wrest increasing amounts of money from developers.

Google last year said it would soon require all apps selling digital goods or subscriptions to use the company’s in-app billing system to process payments—similar to the way Apple operates its App Store. Google previously exempted from the requirement apps that provided digital content that could be consumed outside of the app such as songs that could be ported to other music players. The company said the change reflected an interest in applying rules “consistently and fairly to all developers.”

The states, led by Utah, disputed Google’s claim, saying the move had financial motivations. The rule, slated to take effect Sept. 30 for existing Android apps, would require companies such as Netflix Inc. and Spotify Technology SA to use Google Play Billing for their streaming services. That would enable Google to collect up to a 30% cut of revenue collected from digital purchases and subscriptions, the lawsuit said.

A Netflix representative declined to comment. A representative for Spotify didn’t respond to requests for comment.

“This lawsuit isn’t about helping the little guy or protecting consumers,” said Wilson White, Google’s senior director of public policy, in a Wednesday blog post. “It’s about boosting a handful of major app developers who want the benefits of Google Play without paying for it.”

A Google spokesman declined to comment further.

Proton Technologies AG, a small maker of communication apps, has avoided Google’s commission by letting users know they can purchase its premium services through its website and then access them by downloading its free app via Google Play. Come September, Google will begin prohibiting ProtonMail from informing users of that option, said Matt Fossen, a spokesman for the developer.

“This will affect us negatively because most people find products on app stores rather than signing up directly on the vendor’s site,” Mr. Fossen said.

Google in late June said it would require developers distributing new Play Store apps to use its proprietary app-publishing format. The Android App Bundle, Google said, grants access to advanced features while potentially reducing the app’s size. Critics said the change prevents developers in this format from installing their own payment processors and avoiding Google’s commission on paid apps, in-app purchases of digital goods and subscriptions.

The lawsuit also said Google makes the process of downloading Android apps from competing app stores onerous with security warnings and requests to grant permission multiple times in a single installation. Google has said that its security warnings are designed to protect consumers from the risk of downloading malware. Apple doesn’t allow apps to be downloaded from third-party app stores on iPhones and iPads.

President Biden signed an executive order Friday that he says will promote competitive markets by limiting corporate concentration that hurts consumers, workers and small businesses. Photo: Evan Vucci/Associated Press The Wall Street Journal Interactive Edition

Software developers and Big Tech critics have sharpened their focus on the app economy as consumer spending on digital goods and services has surged in recent years. About $38.8 billion was spent in the Play Store and $72.3 billion was spent in Apple’s App Store last year, according to industry research firm Sensor Tower Inc., generating billions of dollars of revenue to both companies.

Apple and Google are facing lawsuits from developers over their app-marketplace practices and been targets of antitrust scrutiny by U.S. lawmakers as well as regulators in Europe and elsewhere. President Biden also signed a broad executive order Friday calling for regulators to sharpen scrutiny of a range of companies including large technology firms.

Both companies have defended their conduct, including taking a cut of some sales from their app marketplaces, saying the fees help cover costs to maintain platform security, among other things.

The states’ lawsuit filed Wednesday marks the third government action within the last year against Google.  Damien Geradin, a European competition attorney who has studied the evolution of Android, supports antitrust actions against both companies but said he believes Apple has so far avoided the U.S. scrutiny facing Google because Google is an easier political target.

“Apple hasn’t exhausted its political capital,” said Mr. Geradin, who represents the Coalition for App Fairness—a nonprofit formed by app developers—as outside counsel.

He speculated that, regarding Apple, the states could be awaiting the outcome of an antitrust case involving Apple and videogame company Epic Games Inc. Closely held Epic sued both Apple and Google last summer after they booted its hit videogame “Fortnite” from their app marketplaces, citing rules violations.

Some themes presented in the states’ lawsuit against Google mirror those advanced by Epic. An Epic representative declined to comment on the case.

David Hoppe, a media and technology lawyer, said the states’ case is novel because the court is being asked to delve into the details regarding the user experience in software and how such designs are connected to alleged antitrust conduct. It also raises questions about whether the legal system is the best venue to address these issues, he said.

“Do we want courts deciding details of how an app is going to work, like what messages are shown to users,” said Mr. Hoppe, managing partner of Gamma Law in San Francisco. “From a consumer perspective, I may be frustrated with the barriers put up. But at the same time, is this the province of the court?”

Write to Sarah E. Needleman at sarah.needleman@wsj.com and Bowdeya Tweh at Bowdeya.Tweh@wsj.com