As Insurance Insider canvassed the views of CROs, InsurTechs, lawyers and consultants, the risk of a data breach was among the pervading issues that emerged as they laid out the risks and oversight measures required for any firm piloting ChatGPT or similar AI tools.
Insurance firms are still examining the full implications of the advent of ChatGPT and other generative pre-trained transformer (GPT) models – essentially the tech that enables chatbots to provide human-like answers.
In this second of a two-part analysis on the proliferation of these models – also described as large language models (LLMs) – this publication explores the risks inherent in using them, after part one set out the use cases.
Sources explained that a hierarchy of risks comes with using ChatGPT.
Whether it's being used in a pilot or live environment, a potential major data breach whereby sensitive policyholder information inadvertently ends up in ChatGPT's data universe is number one on this hierarchy.
Each individual this publication spoke to warned that when information is fed into the chatbot, it joins the vast corpus of data ChatGPT can trawl for future queries by any user. The data is stored for 30 days and then deleted.
In the broader corporate world beyond insurance, this wasn't a huge concern for many early adopters trying it out.
Data security firm Cyberhaven recently published a study showing that on April 12, its product detected a record 6,352 attempts to paste corporate data into ChatGPT per 100,000 employees. Only weeks ago, Samsung banned all employees from using ChatGPT after an embarrassing story broke that staff were feeding in confidential data, including source code for debugging and transcripts of internal meetings for summarisation.
As one CEO noted, the biggest fear is the ability to control what data staff are feeding into free-access tools like ChatGPT, among a headcount of tens of thousands. Notably, CROs of London market firms told this publication that several months ago, they had instructed all staff not to enter any personal or sensitive information into ChatGPT if they were trying it out of curiosity.
The data protection risks are heightened when firms begin to push blocks of data into ChatGPT to determine how best to mine it.
Doug McElhaney, a partner at McKinsey who leads transformational programs in AI and analytics for insurance firms, said: "Generally the only safe way to perform that kind of task is to licence an independent version of a GPT model, or use an integration from a tech vendor, and then deploy it on an isolated protected data set within a private cloud infrastructure, and then carefully push the data in.
“Even then, an organisation should strip this data down to minimise the risk of any sensitive information being incorporated in that data.”
One executive highlighted that once any sensitive data is entered into ChatGPT, many firms could find it very difficult to extract that information back out, or delete it, once a GDPR issue is discovered.
Ultimately, precise guidelines are required for any use of a public-access AI tool like ChatGPT.
This has been the case among insurance technology firms, many of which are trialling conservative experiments to establish more confidence in deploying ChatGPT for live use.
Doug Marquis, CTO at Zywave, said the company has been experimenting with ChatGPT for a range of purposes, including data extraction from policy forms, an internal HR chatbot and marketing campaign emails for brokers.
"We are very prescriptive on what data staff can put in, and the tools they can use at this point in time. We're trying to strike a balance between allowing people to innovate but keeping sufficient guardrails around it. We are working on it from a legal and cybersecurity perspective and with outside counsel to establish and refine our company policy."
Data protection in practice
In the London market, GPT models are largely being tested at an early stage behind the scenes, but in one case that Insurance Insider has reported on, the algorithmic underwriting platform provider Artificial Labs has been trialling different LLMs, including ChatGPT and others used in private enterprise. The company opted out of all data sharing with its secure, cloud-based LLM providers.
The firm said that as the mandatory 30-day data retention policy on OpenAI’s servers cannot be circumvented, any data it shared was decontextualised to ensure it was meaningless in isolation and did not contain any sensitive or personal information.
In its tests of ChatGPT, Artificial broke segments of documents into small, anonymised snippets during ingestion to remove all context that would contain any identifiable information. The pilot also involved dummy data that was reflective of real data.
A spokesperson added: "We are working with alternative LLMs that are private and enterprise-focused and plan to use a self-hosted model for our live production environment. This will be a simple switchover and will not alter the effectiveness of the tool. We have received extensive legal advice on this subject and are comfortable that we are compliant with current laws and regulations."
Data hallucinations and prompt injection attacks
While data protection remains the number one concern, ChatGPT's weakness in hallucinating false responses is a close second. However, sources said this will probably slow adoption of ChatGPT but not necessarily block it altogether, especially if future iterations of the application prove its accuracy and reliability has been substantially enhanced.
One executive highlighted that the prevailing mistrust of ChatGPT due to this issue may lead firms to pursue alternative LLMs such as Google’s Bard at this stage, particularly for purposes where accuracy is essential.
An equally prominent fear is around prompt injection attacks.
These constitute a new form of cyberattack, and like much around ChatGPT, the language used to describe them is evolving rapidly.
In essence, they involve a hacker manipulating the user's input into ChatGPT to make the model ignore a genuine instruction or perform an unintended response, resulting in various outcomes such as the creation of executable malicious code, or a leak of sensitive data.
Blog posts and cross-industry discussions around prompt injection attacks have started to appear only in the past week, indicating the frequency of new developments that need to be factored into governance.
Governance frameworks
Insurers have been rapidly establishing frameworks for what level of interaction with ChatGPT is permissible for experiments and what kind of data can be entered into it. Above this, they're looking at what sufficient governance of GPT models looks like.
Marcus Evans, partner and EMEA head of information governance, privacy and cybersecurity at the law firm Norton Rose Fulbright, said a key challenge for firms will be translating technical outputs from GPT models into concepts that can be understood and evaluated by non-mathematically gifted business, compliance and legal professionals involved in the decision-making process.
He added: "A governance process could involve first capturing the three main building blocks: the use case (current and future state); the model qualities/drawbacks; and the data representativeness/drawbacks.”
Evans explained that the use cases could then be evaluated against risks including inaccurate/inappropriate outputs, unexplainable responses, outputs that were not subject to appropriate human oversight and any potential disproportionate effects of these risks on different business functions.
Pravina Ladva, Swiss Re's group digital and technology officer, said that solid data protection standards, cybersecurity and standardised systems “are paramount" for AI oversight, adding: “Consent to use the data, data quality and quantity are among the key factors to succeed with AI.”
The regulatory angle
As insurers review governance frameworks, regulators are also working out how they can get their arms around the explosion of AI in financial services. In the UK, both financial watchdogs have indicated plans to regulate what they call "critical third-party" technology vendors, and their definition may ultimately include AI solution providers, though this attempt might only scratch the surface.
Eventually though, the insurance industry and regulators will inevitably run into the "explainability" challenge, whereby it could become impossible for firms to explain the full inner workings of LLMs to the watchdogs.
In Insider Engage's recent webinar on the use of AI in insurance, Chris Mullan, SVP product at Eigen Technologies, explained that regulators may just have to get comfortable with the “unexplainability” of LLMs like ChatGPT, and insurers might have to embed a second line of defence model around the use of them.
The winners
The ultimate winners in the AI adoption race, a source said, will be companies that can train a GPT model based off their own data set, while at the same time ensuring that by working within their own IT infrastructure, they're avoiding any client data slipping out to the outside world via a public access AI tool like ChatGPT.
One source gave the example of GPT models giving brokers the ability to extract whatever number of data points they want from thousands of slips, with codified rules, to provide a data set to train the tool. This data could be then stored to examine and act upon in future, safely.
Beyond the opportunities and operational risks, one CEO flagged a specific implication for insurance technology vendors: "The question carriers will increasingly put to providers of underwriting workbenches, for example, is whether they have integrated a GPT model within their platform. If they haven't, and their competitors have, the consequences will likely be severe," the source said.
For the longer term, as underwriters amass ever more data points from GPT models and multiple other sources, one CEO warned that underwriters could be at risk of "burn out" in future, by having to process more information at increasing velocity for every underwriting decision.
In the shorter term, this current period was described by an executive as a second phase of the insurance industry's adoption, whereby firms are quantifying the benefits of deploying LLMs after a discovery phase.
The third period, of active usage in a live environment, is not far away.
Insurance Insider delivers global wholesale, specialty, and (re)insurance Intelligence that enables you to act first. Redeem your complimentary 14-day trial for more premium content from Insurance Insider.
Scan here to download the app