Should-Read: Janelle Shane: Do neural nets dream of electric sheep?

Should-Read: Janelle Shane: Do neural nets dream of electric sheep?: “Neural network[s]… used for everything from language translation to finance modeling. One of their specialties is image recognition…

…make really bizarre mistakes…. They can find sheep easily in fields and mountainsides, but as soon as sheep start showing up in weird places, it becomes obvious how much the algorithms rely on guessing and probabilities. Bring sheep indoors, and they’re labeled as cats. Pick up a sheep (or a goat) in your arms, and they’re labeled as dogs. Paint them orange, and they become flowers…. And if goats climb trees, they become birds. Or possibly giraffes. (It turns out that Microsoft Azure is somewhat notorious for seeing giraffes everywhere due to a rumored overabundance of giraffes in the original dataset)….

Neural networks match patterns. They see patches of furlike texture, a bunch of green, and conclude that there are sheep. If they see fur and kitchen shapes, it may conclude instead that there are cats. If life plays by the rules, image recognition works well. But as soon as people-or sheep-do something unexpected, the algorithms show their weaknesses.