AI gedachten | thoughts

Anyone with a food allergy can participate perfectly well in our society. There are laws on allergen labelling, so this group can get through life without any problems. Right?

Well, not really. I know someone who is allergic to cloves. A rare allergy, but imagine having it. Those fine labelling laws do not help much here. Or rather: not nearly enough. Of course, “clove” is regularly mentioned explicitly as an ingredient. But quite a lot of products contain “spices” without specifying which spices.

And then try finding out through customer service whether cloves are among them.

“All allergens are listed on the label.”
No, friendly chat assistant from a major supermarket: there are 14 official allergens in the EU, but there are many more besides.
“The unique flavour of our product is based on a spice mix that we source from a third party and whose composition we cannot disclose.”
Hey, I’m not a competitor trying to copy your product! But I checked, and apparently the law does allow this.
“We are sorry that we were unable to meet your expectations.”
Indeed, major manufacturer of meal components, you have lost a customer.

These are all things I never have to deal with as a standard user. That changes if you are blind or deaf, use a wheelchair, have low literacy skills, or belong to some other group.

Then it becomes a question of how well a medicine works. Or whether the safety of a product has been properly tested: it took a surprisingly long time before female crash test dummies became widely available and were actually used. It is about tactile paving at stations (so you can “see” it with a white cane) and braille on handrails or in lifts. It is about wheelchair accessibility. Sure, we do our best, but it almost never works out right by itself.

Put simply: technology excludes groups of people. Exclusion is a consequence of bias—a conscious or unconscious tendency on the part of the people behind the technology to look in only one direction. And, to make matters worse, not to look any further than the end of their own nose.

Bias in degrees

Every kind of bias is annoying, but there are some categories you can distinguish.

A very practical form of bias is a certain kind of inattentiveness in the design of everyday things. Scissors that only work comfortably for right-handed people are one example. All sorts of obstacles in public spaces that get in the way of wheelchair users are another, as is the accessibility of information for blind people. Sometimes the solution is genuinely difficult to come up with, and sometimes you need laws and regulations to force it through. You can accuse the designers of this technology of ignorance, but I also recognize something of my own behaviour in it: “oh no, I just didn’t think for a moment that you were allergic to apples!”

A more problematic form of bias is when laziness plays a role. Blood oxygen meters, for instance, work less well on people with darker skin tones. Medicines were certainly not always tested on all population groups, especially in the past. Bulletproof vests in the army do not fit women’s body shapes. I already mentioned seatbelts and crash test dummies. With a bit of thought and attention, designers could perfectly well have taken these things into account, but cost-cutting, convenience, or blinkers kept them from doing so.

The most malicious form consists of prejudice. Malicious, because the person causing the harm ought to know better. This can arise in interaction between people, but technology can also implicitly pass prejudice on. Just look at the design of devices, and compare kitchen appliances with power tools for DIY jobs. Soft rounded shapes in pastel colours versus angular chunky machines in green, dark blue, and black. The stereotypes in manuals and advertising material belong here too.

So what about AI?

AI is also “just” a technology, so let’s unpack that.

When it comes to the inattentiveness type of bias, things are not too bad. For the time being, we mainly interact with AI through screens. The digital industry has had decades to address accessibility problems. In fact, it is striking how much AI actually helps certain groups to participate. Familiar examples are screen readers that turn written text into speech. Or, the other way around, speech recognition for people who otherwise could not operate a computer or phone. An even lovelier example is the beautiful solution created by the young Dutch company Whispp, which helps people who stutter. It turns out that many people who stutter have far less trouble when they whisper. Through an app, Whispp instantly converts whispered speech into voiced speech, making a “normal,” fluent phone conversation possible.

The second kind of bias, caused by laziness, was until quite recently the kind AI was most notorious for. Usually this happens because the training data for a model has not been thought through properly. We know the examples of Black people who were recognised as gorillas, because the model’s training data consisted almost entirely of white people. Amazon also produced a fine example: they wanted to train a model to automatically pick out the best job applicants. That model was trained on earlier applicants who had turned out to perform well. The only problem was that those earlier applicants were mostly men. So the model “automatically” selected men as well. You can understand it a little: collecting data takes a huge amount of time and effort. Most developers are happy if they have anything at all, and if their model also seems to perform well—tested on that same badly delimited and poorly selected dataset—that already feels like quite an achievement.

With the arrival of chatbots, prejudice bias has entered the AI world as well. The first incident was the chatbot “Tay” made by Microsoft, which people could talk to on Twitter. Tay went off the rails because it was designed to learn from interactions. That was, of course, immediately picked up and stress-tested, with the result that within a few hours Tay had turned into a racist and sexist Holocaust denier. (And this was back in 2016! Long before Mr. X took over the platform.)

After Tay, things stayed quiet around chatbots for a while, until of course ChatGPT appeared in 2022. Because language models are trained on “all” available texts, it is hardly surprising that “all” the prejudices contained in that training data also make their way into the model. A familiar example is that a “doctor” is more often referred to as “he” than “she”, but cultural and political prejudices can also shine through in the output of these models. The problem is that many large language models (though not all) are trained on data gathered with very little selection. Their designers simply do not know exactly which ideas are lurking in the training material.

You can still do something about that with fine-tuning, but that is really a bit like correcting afterwards something that was not entirely OK to begin with.

A language model is trained by “feeding” it huge amounts of text: in this way the model learns which words tend to appear together in which combinations. In this blog I compared that process to drawing a map. After this training—which takes an enormous amount of time and energy—the model has, as it were, learned a language. Astonishingly enough, the model has also learned quite a lot of facts along the way. Only: the model is not yet very good at holding a conversation, or summarising, simplifying, or translating a text. That happens through fine-tuning, in which examples are given of what good answers look like and what less good answers look like. Or how instructions ought to be followed. To stay with the map metaphor: after training, all the roads and places are on the map, but the map still does not tell you how to get from A to B in the fastest way, the shortest way, or the most scenic way.

I have argued before that GenAI (chatbots) gets all the attention now, whereas “classical” AI has in fact been applied much more broadly for quite some time. For many applications, bias is not such a big problem. If AI is used to inspect bridges, predict forest fires, or sort waste, a wrong decision caused by bias is mainly a matter of accuracy and reliability. A question of adjusting the settings and making sure overall accuracy is as high as possible—while having to accept that a small percentage will inevitably go wrong.

AI that interacts with people has a hard time

Things become complicated when those successful AI applications are also applied to people. For a bridge or a forest fire it does not matter very much whether it is treated with bias or not. But as soon as people are involved, that changes: the bridge itself does not care very much, but the people driving over it certainly do. Then bias translates into disadvantaging particular groups.

The bad news is that there are situations in which you simply cannot eliminate this kind of bias. That is because you can look at bias in different ways, each of them perfectly reasonable in itself. But those perspectives cannot all be satisfied at the same time.

Compare it to map projections: the Mercator projection, in which a straight line on the map corresponds to a straight line at sea. That is very useful for navigation (which was Mercator’s purpose, after all), but it means the surface areas are completely wrong: Greenland looks as large as Africa. The alternative is a projection where the areas are correct but straight lines become curved (for example the Aitoff-Hammer projection).

There is no map projection in which straight lines remain straight and areas are also accurate. Likewise, there is a group of selection systems—whether they use AI or not—that will always treat groups with different characteristics unfairly. The (in)famous COMPAS system, used in the United States to decide whether a suspect should be released on bail, suffers from this as well. Yes, from one perspective it is fair. But from another perspective, suspects of colour are disadvantaged. And if you fix that, the first perspective becomes unfair again. (This situation is too complicated to explain here to a fuller extent. I hope to go into it more deeply in a later blog.)

Besides this rather fundamental problem, there are other annoying causes of bias as well.

Another irritating effect is that AI can be so “smart” that it starts looking for a fragment of prejudice all by itself. You can deliberately leave out certain information in order to avoid discrimination on that basis, such as ethnicity. But then it turns out that a model can reconstruct something suspiciously similar to “ethnicity” from a combination of postcode, income, and education level. It is a bit like the “spices” at the beginning of this blog: superficially it all looks perfectly fine, but if you look more closely there is still a problem.

Sometimes an AI system creates its own bias. Think of a navigation system that suggests an alternative route to masses of road users, thereby creating a new traffic jam. Or the criticism of predictive policing systems, which select certain neighbourhoods for extra patrols on the basis of the probability of crime—and in doing so may ensure that even more offenders are caught there.

But people have bias too, don’t they?

Using AI to make decisions involving people is therefore a minefield. The issues are obvious. You can assess such an AI system quite easily on its overall behaviour. And you will find that it is biased – just look at the examples above.

By contrast, when people make decisions, one person will do it one way and another person another way. That is what very often happens in the non-AI world as well: school advice, prescribing medicines, assigning athletes to teams, judicial rulings…

There will certainly be prejudices there too. But because of the differences between all those human decision-makers, it is usually hard to prove that things are going wrong in a structural way. (If that is the case, we talk about institutional prejudice; it often takes long and grinding processes before that is recognised.) The main difference is this: in a human decision, other points that are not measurable and may not be objective will also be taken into account. The human touch, indeed.

So now what?

So here we are. Technology is invented to make our lives more comfortable and safer, and then it turns out that discrimination is almost impossible to avoid. So now what?

In an earlier blog about the reliability of AI, I introduced the trio of labor, laws, and literacy. That turns out to be useful here as well.

A great deal of technology is brought to market in a free market. A product that excludes groups of people will be bought by fewer people. Or a product will appear on the market for the excluded group. For bias caused by inattentiveness, that is a solution. That is why there are scissors for left-handed people, bicycles with low step-through frames, and shampoo for people with curly hair. In short: we have some labor to do on improvements!

If that does not work well enough, then most societies will start making laws. You can see that with the “laziness” type of bias it turns out to be necessary to establish laws, rules, and standards. About the accessibility of websites or public places. About standards made mandatory in the testing of products. And indeed: the AI Act imposes all sorts of requirements on AI products. The requirement of “meaningful human oversight” is one of them. That is meant to prevent AI systems from making decisions about groups of people without a (hopefully sensible) human having seriously looked at them. It addresses the impossibility of truly eliminating bias.

The requirement that AI systems should be as free from bias as possible is also laid down in law. But even if technology sometimes excludes groups of people, should you also deny the solution to the group that does benefit from it? That would not be fair either. We know of medicines that only work for certain groups of people – should such a medicine be sent back to the developers with the instruction to “remove the bias”? The AI Act does not go that far: it states that bias must be “avoided as much as possible” and that the training data must be as representative as possible. In other words, not that it must be entirely free from bias – and that wording is deliberate. We now know why. (This aspect is taken rather seriously. It is even permitted to process special categories of personal data, such as ethnicity, in order to remove as much bias as possible. Under privacy law, special categories of personal data are protected extremely strictly.)

Despite working on improvements and drawing up laws, it will remain necessary – especially on the subject of bias – to be literate. Laws and regulations cannot 100% prevent AI applications from coming onto the market that have not been tested well enough and that introduce new biases. An extra unpleasant complication is that biases sometimes reinforce one another. That is called intersectionality: groups of people who already face certain disadvantages are often hit even harder when they are excluded once again.

All of this makes bias in technology a stubborn problem, deeply embedded in our society. AI has a tendency to reinforce it once again. But fortunately, not all is lost: we can reject those AI applications if we do not like them. Then we will simply have to make do again with our own familiar prejudices.