Tuck School of Business | How Gen AI can improve customer service

by Kirk Kardashian
December 5, 2024

Finding a satisfactory solution to customer service queries is crucial to the success of an e-commerce business. Since the customer cannot visit a brick-and-mortar store, the human agents are the manifestation of the company in the minds of consumers. If the agent cannot solve the customer’s problem, the customer may not come back. Scale this dynamic to hundreds of thousands of inquiries per day, and you face the daunting challenge of earning buyer trust and loyalty.

Curious about whether generative AI (Gen AI) could improve customer service, Taobao conducted a large-scale, randomized field experiment in early 2024. The company gave some of its customer service agents access to a genetic AI assistant, while other agents (the control group) did not have access to the assistant. The Gen AI assistant would analyze customer order data and previous customer interactions in real time and give the agent a quick summary of the problem in the form of a written message that the agent could send directly to the customer. At the same time, the assistant prepared another message with a suggested solution to solve the customer’s problem. The agent has the option to use, modify or ignore these messages formulated by the Gen AI assistant.

Tuck Professor Lauren Xiaoyuan Lu studies the operational drivers of organizational performance in healthcare, retail, and supply chain. She teaches Supply Chain Management in the Tuck MBA program and teaches in Tuck Executive Education.

Tuck Professor Lauren Xiaoyuan Lu, along with several colleagues in China, worked with Alibaba to analyze the data from this experiment. They report their findings in a new working paper titled “Generative AI in Action: Field Experimental Evidence on Worker Performance in E-Commerce Customer Service Operations.” In analyzing the experiment, Lu and her colleagues sought to answer two questions that are now of utmost importance to both academics in many fields and retail practitioners: “How does genetic AI affect the performance of human agents in real-world customer service?” Interactions, and are these effects consistent across different types of agents?”

Their first key finding was quite predictable: the use of the Gen AI assistant improved both the speed and quality of service in this environment. This was reflected in agents spending less time in customer chats and customer satisfaction increasing, reflected in higher customer ratings and lower dissatisfaction rates.

The remaining results were more nuanced, but no less significant. The researchers realized that the genetic AI assistant served to complement the human agents rather than replace them. One might imagine that human agents would slack off if an automated assistant provided all the answers, but they didn’t. Statistically, agent typing time did not decrease and agent message volume actually increased. “So both indicate that the engagement of these agents increased after using the assistant, which was somewhat of a surprise,” says Lu. Lu plans to further explore this phenomenon in future work; It currently hypothesizes that the next generation AI assistant will relieve both the customer and the agent of the burden of routine communication and allow the agent to focus more on the customer’s specific needs.

Lu’s third main result was also a bit of a surprise. She found that the generational AI assistant reduced the achievement gap between high and low performers, but it was not a unilateral change. Yes, the generation AI assistant improved the performance of low-performing agents, but it also slightly worsened the performance of high-performing agents. According to Lu, this can probably be explained by the fact that the recommendations of the genetic AI assistant are less sophisticated than those of the most powerful agents. So when these top agents relied on the assistant, it resulted in less optimal solutions. This presents a management challenge for e-commerce companies using genetic AI. Since the Gen AI Assistant’s message formulation is heavily influenced by previous messages from top-performing agents, if these agents distort the data by becoming too rely heavily on the assistant. “Companies need to think about how they can continue to incentivize their top performers to perform and even improve,” Lu suggests.

While the results of the Alibaba experiment generally bode well for e-commerce companies that can use genetic AI to improve the speed and quality of customer service, Lu still senses a looming dark side to this technology when it comes to workers goes. In the e-commerce world, margins are very low and a large part of a company’s costs are related to logistics, yet customers expect ever faster service. Something has to give. Companies have already reduced labor costs by moving customer service activities offshore; As generational AI assistants improve, it will be tempting for companies to reduce their workforce even further.

“I have concerns about the displacement of workers,” Lu said. “Companies should think about how they can redeploy these workers and upskill them for other value-added services.”