Apple’s Artificial Intelligence helps criminals
Read for free: https://acuity.design/apples-artificial-intelligence-helps-criminals/
Apple Intelligence was helping someone steal my payment details today.
In my emails this morning, one had a Priority sticker added with the message that I should deal with it to stop service disruption. It was apparently from Weebly, a website hosting company which I have used in the past.
It was early, Apple had added its Priority message, it was a provider I have used. Dealing with it now seemed a good idea.
And yet…
I do automatically long press web links in emails just to check. Long press brings up a preview of where an email link goes. It is a good piece of design.
The link went to some strange website which did look like the Weebly sign on page but definitely was not the official page.
Apple and its artificial intelligence (AI) was helping someone steal from me
Dumb AI and not particularly clever criminals
The AI was superficially correct. The email did say that Weebly was suspending my service due to payment detail update issues.
However, Apple Intelligence was failing in about the same way every person who had fallen for a phishing email scam has failed. It ignored the obvious error. It did not look at the website link and ‘think’ critically.
This scam is not particularly hard. It is performed probably millions of times a day. It is something that can be stopped with a little bit of self awareness and wariness.
This AI is dumb. It is superficial and believes the scammers.
Apple loses credibility and trust while customers lose money
What is worse is that Apple has foolishly decided to enable an AI that destroys its own corporate credibility.
Phishing scams depend upon criminals abusing the trust and authority of credible organisations. They imitate the look and feel of a powerful organisation to convince people to act quickly and to hand over their payment details.
This is a form of emotional design that scammers have practiced for years. The rush of panic and the offer of calm authority to help.
What Apple is doing is helping the criminals by adding a Priority sticker.
Apple’s authority and trust is being used to turbocharge the scam.
What could be done?
As a first step, Apple should turn off this feature. It is clearly open to criminal abuse. It benefits no one to have Apple enabling more fraud and loss.
Secondly, there is a possible feature that could work. If Apple Intelligence actually compared who the email said it was from and the weblink that is hidden behind the email button then it could stop phishing and fraud. Such emails could be quarantined and sent to the relevant authorities for investigation. Apple could build greater trust and use its technology and authority to help customers.
The current AI is not capable of doing the job. It is not reasonable for Apple to deploy a technology that clearly helps criminals. It should be stopped until it is safe.
Originally published at https://acuity.design on December 22, 2024.
