The challenge of AI decision-making predates AI itself. Back in 1985, Jarvis Thomson came up with the ethical dilemma of the runaway trolley, which presents a seemingly impossible decision between death and more death. With AI based decision-making becoming mainstream, we are now forced to urgently come up with regulatory and legal frameworks that speak to challenges as complicated as the trolley problem.
A colleague recently set me this screenshot that was circulating on social media:
I cannot vouch for the source of the content, but regardless of its validity, it presents a plausible retail scenario with far-reaching implications. Given the current state of consumer legislation, it seems possible that a consumer can legally prime the decision of an AI algorithm. If the colour of my eyes or my historic credit rating can be used in AI based decisions, can my needs and desires I express during the negotiation of a transaction, also be considered? The individual in the conversation with Chevrolet’s AI bot, clearly felt that buying a 2024 Chevy within a budget of $1 is acceptable. And succeeded in negotiating a deal.
“It does highlight the danger of generative AI in these situations”, says Richard Hammond, Head of Legal at First Technology. “My opinion would be that it could certainly be a binding commitment. As the AI is representing itself as acting on behalf of the business, a consumer would have the reasonable impression that they are interacting with a representative of the business, who is able to bind the business to the transaction”, he says, tongue in cheek.
He continues, “However, in the context of this scenario, Chevrolet probably wouldn't be held to that contract, because it is clear that the consumer understands that he is engaging with an AI bot and is effectively just trying to ‘snatch a bargain’, which, at least in South Africa, isn't permitted under consumer law. This also wouldn't have been an advertised or public price, further weakening any argument of the consumer that the business should be held to that price.”
Retailers will continue to invest new technologies, especially in cost saving online user experiences. According to the ARK Investment Management report of 2024, AI is yielding discounted costs of up to $0.25 per user-interaction-hour, compared to in-person sales interactions.
Underscoring the risk, Richard concludes, “As AI evolves and takes over more customer facing functions, there is no doubt that businesses will be bound to transactions concluded via AI. Although the Chevrolet scenario is a bit too extreme for the company to be bound, in situations where it is not so obvious, I do believe a retailer might find themselves in a binding scenario that they otherwise would have not considered.”
Gabriel Malherbe is the Managing Executive of First Digital, a First Technology Group company focusing AI enabled software development, digital transformation consulting and the design, development and deployment of enterprise solutions to various international clients.
Richard Hammond is the Head of Group Legal at First Technology, focusing on the legal, contractual, and risk and compliance requirements for all First Technology group companies.
Related: