California lawmakers are seeking to lead on the oversight of artificial intelligence with a sweeping bill that would monitor how employers and industries use automated decision tools, from algorithms that filter out job applicants to programs that detect academic cheating.
Assemblymember Rebecca Bauer-Kahan (D) has sponsored legislation (A.B. 331) that is the lone AI-related proposal in the California state legislature and one of the few measure across the nation that would impose assessment requirements on the private sector’s use of such software.
The state’s bill comes as federal action on the topic has been slow to develop. While AI has garnered more public attention amid fears over potential abuse, the Biden administration only recently asked for public input over how to build a framework to oversee AI programs. Legislators in multiple states are stepping in with a wave of bills, but most measures would either ask to explore the topic or only affect how government uses AI.
The Bauer-Kahan legislation is more far-reaching because it targets discrimination from AI software in employment, education, housing, utilities, health care, financial services, and other areas.
“Fifty percent of companies are now using these automated decision tools to make consequential decisions. This is happening today in many sectors that affect people’s lives, and so we need to move forward,” said Bauer-Kahan. “We’re not ahead of the ball. We’re already behind the ball.”
Assessments and Safeguards
Business lobbyists have taken notice of the legislation, which is moving through committees in the state Assembly. The measure takes guidance from the Biden administration’s AI Bill of Rights framework, Bauer-Kahan said. The Biden document outlines broad principles to prevent an automated decision system from causing discrimination and other harm.
The legislation would spell out how creators and users vet AI systems. Instead of relying on an independent third-party audit, Bauer-Kahan’s measure would require developers — the ones who create or code the automated tool — and users of the tool to each submit annual impact assessments to the California Civil Rights Department by 2025. The bill would be the first in the nation to divide those responsibilities, observers said.
“It’s an interesting choice. I think it’s a smart choice,” said Sorelle Friedler, a computer science professor at Haverford College who helped craft the Biden AI document. “There’s going to need to be a certain sort of assessment done by the people who develop that model. But then depending on how it’s used, if it’s used for health care versus if it’s used to write poetry, there are going to need to be really different assessments.”
Under the bill, the assessments would have to include how an automated decision tool is being used, what data is being collected, what safeguards are in place, what potential adverse impacts may result, and how the tool was evaluated. In addition, companies would need to implement a governance program putting those safeguards into practice.
The measure would require those using AI to write a publicly available policy listing the types of automated tools used and how the company manages the risk of illegal discrimination. If a decision is made solely based on AI, an affected individual would have a right to opt out if the request was “technically feasible.”
Business Concerns
Industry groups argue many terms in the legislation are unclear and would create compliance problems, such as determining what exactly constitutes a violation or how to decide what is “technically feasible” in an opt-out request. The frequency and breadth of the assessments could also be burdensome for small businesses, said Ronak Daylami, policy advocate for the California Chamber of Commerce, at an April 11 hearing.
The bill partly addresses such concerns by applying its requirements to businesses that use the technology and have 25 or more employees, unless the tool impacted at least 1,000 people per year.
The biggest concern from businesses is that the punishments under the bill could produce a chilling effect on innovation in the space because of its potential financial penalties.
Under the measure, the state Civil Rights Department can impose a $10,000 fine each day an impact assessment is not submitted. The bigger fear among business lobbyists, however, is the private right of action that would allow state resident to bring suit, a provision that hasn’t been included in any other AI proposals, said Friedler. A business or developer would get 45 days to correct the violation to ward off a lawsuit for injunctive relief, though business groups say it’s not enough time.
Bauer-Kahan defended those provisions but was open to compromise.
“I mean, the impact assessment itself is a corrective mechanism, right? The whole goal is there to catch it and fix it and that’s what the bill asks you to do,” said Bauer-Kahan at the April 11 hearing. “There is currently as it relates to injunctive relief, an offramp, but I know the opposition would like to see more of that and we’re in conversations about that.”
Assemblymember Bill Essayli (R) noted that California law already makes discrimination illegal, whether from a human or algorithm, and he asked during the hearing if creating a new private right of action was even necessary.
Upcoming Work
The California Civil Rights Department’s Council is in the process of adapting existing employment discrimination regulations around the usage of AI. There’s also yet-to-be-created rules from the California Privacy Protection Agency on privacy protections against automated systems. Industry groups said they fear that Bauer-Kahan’s bill would conflict with those regulatory efforts.
But privacy advocates contend the bill could also complement the forthcoming rulemaking. For instance, draft regulations from the Civil Rights Council would hold employers liable for discrimination from an automated tool, even if it’s created by a third-party vendor. That provision would place additional burdens on an employer to understand how another company’s automated system works.
Bauer-Kahan’s bill reinforces that proposed language because it would require a developer of the AI tool to send certain information to the deployer of the technology. The bill also clarifies that trade secrets don’t have to be disclosed as part of compliance, which is a concern around the council’s draft rules.
More importantly, Bauer-Kahan’s legislation mirrors the Biden proposal in that it would require advance notices to people subjected to an automated decision tool. That provision would give teeth to the potential state regulations because of the difficulty in enforcing protections against AI abuse if residents don’t know it’s being used.
Bauer-Kahan said she expects the bill will still be revised, but fellow Democratic lawmakers in the state Assembly Privacy and Consumer Protection Committee were eager to approve the measure on an 8-3 vote at its April 11 hearing. Despite the business opposition, California should take the lead on AI as the technology constantly evolves, lawmakers said during the debate.
“It’s never going to be perfect. Like let’s make sure that we’re clear about that,” said Assemblymember Josh Lowenthal (D) on regulating AI. “This is going to be a work in progress forever. It will never, ever, ever be completely perfected.”
