ALL MATERIAL COPYRIGHT BEYOUROWN 2026

The Risks Generative AI Poses To Business Security, By Louis Blackburn, Operations Director At CovertSwarm

pexels-sora-shimazaki-5935794
Generative AI is the single biggest source of opportunity in the tech space right now. The entire population, including both business leaders and fraudsters, are in the process of understanding and exploring these new opportunities. ‌CSOs reported AI-powered attacks as one of several significant security concerns to business, with 85% of security professionals reporting an increase in AI-enabled attacks over the past year and 67% admitting AI will provide new avenues for attacks. Moreover, the quality of all types of online attacks across the board is only going to improve, making it harder than it has been before to spot the telltale signs of a fraudster. ‌With AI-powered cybersecurity attacks on the rise, Louis Blackburn, Operations Director at CovertSwarm, sheds light on the types of AI-fraud businesses should be aware of, how organisations can mitigate these attacks, and the roles CISOs and auditors play in implementing an appropriate security strategy. The quality of fraud is improving AI fraud differs from typical online fraud through the use of new AI-based tools, like using voice synthesis for creating fake conversations. Criminals mimicking real people is nothing new, but now there are easily accessible tools that can essentially create a conversation that never happened by using just one minute of sampled speech from someone's voice. ‌McAfee’s recent global study found that, out of 7,000 people surveyed, one in four said that they had experienced an AI voice cloning scam, or knew someone who had. ‌Video AI is another common tool used in the fraud and criminal space, but audio and voice synthesis is beginning to become even more prevalent, as it’s much harder to spot a fake person on the other end of the phone than it is on a webcam. ‌In the past, fraudsters have struggled to create emails in the language of the target, and commonly included misspellings or grammatical errors that acted as giveaways, but AI and LLMs (Learning Language Models) are all working to correct these issues and, unfortunately, improve the quality of fraud. Essentially, AI now allows fraudsters to write a phishing email in French that perfectly translates to the correct phrase in English, making it extremely difficult for someone to notice that anything is amiss. AI fraud will have devastating impacts Global cybersecurity crime is expected to cost the world $10.5 trillion USD annually by 2025, up by 15% from the cost five years ago; and the average global cost of a single data breach costs a business around $3.62 million, with customer trust taking a huge hit. ‌It’s essential that organisations of all sizes learn about the security risks that AI can pose to their business, and establish a plan to deal with any threats. ‌Voice synthesis tools are being used to mimic employees in organisations, such as service desk workers, to gain people’s trust. Businesses are at threat of simple transactions like a password reset taking place over the phone and getting into the hands of somebody who, unbeknown to them, is not the actual colleague a person thinks they’re speaking to. ‌AI can be used to perform reconnaissance against organisations in the future. Collating information about a target business at the moment is a very manual process, but in the near future attackers will be able to use AI to quickly find out the relevant information about an organisation — like the IP address, open ports, security software, and hardware in use, and vulnerabilities in these systems. ‌In the future, this will develop into hackers being able to use AI and OpenSource to look into a company’s computer vulnerabilities and other areas that may be insecure. Organisations need to be proactive with regard to all digital security, and perform continuous testing to find the problems before AI does. How can organisations assess the risks? According to the ONS March 2024 dataset, 79% of UK businesses are not currently using AI and have no plan to do so soon, indicating a lack of preparedness for AI-powered cybersecurity attacks nationwide. SMEs are at even greater risk, with less budget to put defences in place making them keen targets for hackers. Forty-three percent of SMEs have no cybersecurity defences in place at all — now is the time for leaders to make smart investment decisions that will protect their organisation. ‌The security measures that organisations need to put in place depend on the organisations. Larger organisations are likely to have an internal intelligence department that assesses the latest threats and maps them against vulnerabilities internally. The most important measure any business can take — regardless of size — is to educate employees on the existing risks fraud already poses to the business, and what AI is bringing to the table. ‌If your organisation conducts a significant amount of business over the phone, it’s paramount that employees have an understanding of the aforementioned voice synthesis technology, and realise what its capability is now and what it’s due to be soon. ‌Using that knowledge, intelligence teams or IT departments will understand what the key risks to the company are at the moment, and can make sure the right education and awareness for their staff is in place. ‌While the first step towards mitigating fraud risks needs to happen through internal education, putting extra technical controls in place to give employees a means of assuring that who they’re speaking to really is that person — in a similar way to authentication methods put in place for telephone banking — is another effective barrier businesses can put in place to deter attackers. Auditors play a vital role in managing the risk Auditors should be turning their focus to the organisation’s overall approach to embracing AI, and what that should look like. ‌Organisations are at risk from their own users inputting confidential data into AI and LLMs. The risk may be manageable with Microsoft and Google-based LLMs mainly available at the moment, because these are well-established companies. In the future, howevfer, when more edge-case AI and LLMs go live — and in the case that these are accessible to employees — companies are at risk of data breaches internally, where colleagues may innocently input information, like customer data, into one of those. ‌It’s essential for auditors to look at the risk of information disclosure to AI and LLMs, and ensure that there’s a company-wide strategy in place for embracing the tooling. The first port of call should be making sure that the relevant staff are aware of the risks, and then bringing more general HR processes up to date with the risks presented. Your security strategy needs to be watertight ‌Ensure an umbrella policy or procedure for the introduction of AI and LLMs into the organisation is put in place, and the relevant staff are educated on the risks of their use. If there is no company policy in place for the use of AI and LLMs, this raises the potential for the event of a data loss or breach taking place. Once the policy is in place and an LLM is introduced for internal use, technical barriers should be put in place to limit or at least control the type of information that can be copied in. Effective intelligence feeds need to be introduced into the organisation to inform IT departments and intelligence teams about the existing risks AI and LLMs pose to the business and the latest developments; policies and re-education can then happen accordingly. Organisations should continuously test the efficacy of technical controls around AI and LLMs, and use continuous red team testing to ensure there are no security vulnerabilities. ‌AI presents both immense new opportunities and significant security risks for organisations. Understanding the risks that AI-related fraud poses to your business through effective intelligence feeds and constant education is paramount for mitigating the risks. ‌By staying on top of evolving threats, and implementing a robust security strategy focusing on continuous testing and education, and with controls in place to limit employee access to AI tools, organisations will be able to safely navigate security risks and utilise AI for its benefits.   By Louis Blackburn, Operations Director At CovertSwarm

Read More
Membership Personal Invitation

Leading Like a Pro: Enhancing Leadership Skills for Female Founders

|BEYOUROWN Team
In the dynamic world of entrepreneurship, female founders face unique challenges and opportunities. Navigating the path to success requires not only a compelling vision but also robust leadership skills. For...