People have made this argument against FICO, for what it's worth. (FICO is likely the most consequential AI employed in the United States outside of the Googleplex. It is not called AI because AI as a field demotes anything that ships into mathematics or engineering.)
FICO operates off of credit reports, which have no explicit way to encode race in them. It is fundamentally a clustering algorithm. It is very good at clustering.
(There's a good counterargument to be made, which is that FICO actually clusters people based off observable behavior and the system which FICO supplements/replaces -- human underwriters -- cluster people based on "Tom is an upstanding Christian who goes to my church; of course he is good for the money.")
FICO operates off of credit reports, which have no explicit way to encode race in them. It is fundamentally a clustering algorithm. It is very good at clustering.
(There's a good counterargument to be made, which is that FICO actually clusters people based off observable behavior and the system which FICO supplements/replaces -- human underwriters -- cluster people based on "Tom is an upstanding Christian who goes to my church; of course he is good for the money.")