Forbes on next generation of AI

What do you think about the ideas in Forbes article on the next generation of AI?

See: https://www.forbes.com/sites/robtoews/2020/10/12/the-next-generation-of-artificial-intelligence/

Forbes believe in unsupervised learning, federated learning, and transformers for neural networks.

Unsupervised learning (aka self-supervised learning) is based on “predicting everything from everything else”, e.g. language models from billions of documents. This avoids the bottleneck of having to label data for supervised learning, and is more flexible in allowing the learning system to figure out its own labels and "being able to explore and absorb all the latent information, relationships and implications in a given dataset.”

Federated learning is about services that support privacy friendly machine learning by a third party across training data without having to transfer the data to that party. Instead, the learning process is applied locally to the data, and the results transmitted to the third party for aggregation with the overall model.

Transformers are a technique for learning across sequences of things, e.g. words in text or frames of video, that is readily executed in parallel and computationally more efficient that previous techniques. This was first applied to language models to predict text following a previous text extract (e.g. BERT and GPT-3), but is now being applied more widely. e.g. to video.

Whilst GPT-3 is pretty amazing in the quality of the text it can generate, it is limited in the kinds of reasoning it can apply. It knows simple generalisations, but is very limited in respect to reasoning about time, and is unaware as to what it doesn’t know. As an example, asking for the sum of two large numbers returns a large number, but not the actual sum, asking for the US president in 1610 returns a historical figure rather than stating that the question doesn’t make sense as the USA wasn’t in existence then.

This is unsurprising as language models are not the same as higher level reasoning that children are taught at school and through interaction with their parents and peers.

What do you think?

Dave Raggett <dsr@w3.org> http://www.w3.org/People/Raggett
W3C Data Activity Lead & W3C champion for the Web of things