This time of year, the California tax office is always buzzing with activity as hundreds of thousands of residents and businesses seek tax advice. Phones ring and keyboards clack.

Call volume can reach up to 10,000 per day, quadrupling the average wait time from four minutes to twenty. Chief of the call center Thor Dunn stated, “When the bell rings at 7:30 you (already) have a wait.” He also mentioned that employees with other occupations are trained to pick up the phone during busy times. “Everyone is on deck.”

Therefore, California’s 3,696-person Department of Tax and Fee Administration intends to employ generative artificial intelligence to assist its about 375 call center agents on state tax code later this year—for the upcoming tax season. The information they provide to California business owners seeking tax advice will then be informed by the AI.

Generative artificial intelligence models, trained on vast datasets frequently stolen from the internet without authors’ permission, are capable of producing text, image, and audio material. With its debut in fall 2022, OpenAI’s ChatGPT, a big language model, sparked interest in generative AI by effectively predicting the next word in a sequence of input text and then producing or generating text that represents the training data.

For you, the individual phoning the tax center, what form would that be? Although a slide in the tax department’s request for proposals requesting a vendor states that any AI solution must “be able to provide responses to incoming voice calls, live chats, and other communications,” the tax department informed CalMatters that this technology will not be used without a call center employee present to review the answer.

That call for proposals was issued last month with the goal of using AI to assist the state with taxes. This week is the deadline for initial proposals, and the process should be completed by April. 100 people attended a meeting with possible vendors last month, according to department spokesperson Tamma Adamek, who talked with CalMatters.

The tax guidance Acting head of the California Department of Technology and state chief information officer Liana Bailey-Crimmins says the AI proposal is one of five proofs of concept the state has started investigating how state agencies can employ generative AI. The state’s Health and Human Services Agency is conducting two trials to investigate whether generative AI can facilitate public benefit understanding and attainment, as well as aid in health care facility inspections. Caltrans is also working on two projects to investigate whether generative AI can lessen traffic congestion and deadly accidents.

The vendor who wins the AI tax proposal will be awarded a six-month contract; after that, state representatives will decide whether or not to award a longer one. In addition to vendors having to “monitor and report on GenAI solution responses for factual accuracy, coherence, and appropriateness,” the initiative needs to show reduced call times, wait times, and abandoned calls.

The initiative marks the beginning of an iterative, multi-year process for AI regulation and implementation, which was initiated by Governor Gavin Newsom last autumn. By July, governmental agencies must investigate the use of generative AI, according to the executive order he issued.

Contract-awarded private enterprises will train AI models in a “sandbox” situated on state servers, designed to adhere to information security and monitoring guidelines established by the technology department, in order to reduce risks. The IT department is required under Newsom’s executive order to make the sandbox available for usage by contract-awarded companies in March.

AI Risk Assessment

In November 2023, the Government Operations Agency of the state assessed the advantages and disadvantages of generative AI. The paper issues a warning, stating that generative models may yield plausible but erroneous results, provide distinct responses to the same question, and experience model collapse when predictions deviate from true outcomes. The use of generative AI also runs the risk of automation bias, which occurs when users become unduly dependent on and trusting of automated decision-making.

It’s unclear exactly how call center staff for tax agencies will identify which responses from massive language models to believe.

According to Adamek, the tax department’s spokesperson, they receive training on fundamental tax and fee programs and are able to seek assistance from more seasoned team members when they have questions about a particular topic. The technology department is slated to assist in training state personnel on identifying incorrect or fraudulent text in July, working with other state departments.

According to Adamek, the tax department does not view its intended use of generative AI as high risk because it is primarily concerned with improving state business processes and all relevant data is accessible to the general public. Later in the procedure, the tax department will evaluate risk, according to her. In the upcoming weeks, standards guidelines for state entities that enter into contracts with private enterprises are scheduled to be released.

The technology department may not agree, but the tax department does not view the use of generative AI as highly risky.

According to Newsom’s directive, all state agencies must provide the Department of Technology with a list of the high-risk generative AI applications they are utilizing in less than 60 days. CalMatters was informed by Bailey-Crimmins that none of the governor’s agencies are utilizing high risk generative AI.

A new rule mandates that by September, at the latest, the technology department must catalog all high-risk AI applications and automated decision making systems used by state entities.

However, some people outside of government are concerned about some of California’s AI initiatives. Among them is Justin Klozcko, a Los Angeles-based author of the Consumer Watchdog report Hallucinating Risk, which explores the possible risks associated with AI patents held by banks and used in financial services. He points out that OpenAI, the San Francisco-based company that created ChatGPT, has issued warnings in its documentation that using AI to provide financial advice or offer basic services carries a significant risk.

“There’s still a lot we don’t know about generative AI and what we do know is that it makes mistakes and acts in ways that people who study it don’t even fully understand,” Klozcko said. He also questioned the ease of determining whether that information is accurate in the hands of the call center employee who may not be qualified to determine whether text output by a large language model — made to sound convincing — is in fact inaccurate or false.

“I worry that workers in charge of this won’t understand the complexity of this AI,” he said. “They won’t know when they’re led astray.”

“We take those risks seriously,” according to Bailey-Crimmins, who also stated that possible drawbacks will be taken into account when deciding what to do after the six-month trial project.

“We want to be excited about benefits, but we also need to make sure that what we’re doing is safeguarding… the public puts a lot of trust in us and we need to make sure that the decisions we’re making (are) not putting that trust in question.”