Next generation ANNs

I am on my way back from the Edge AI summer school in Pisa, where I gave a lecture on cognitive approaches for low-code control in swarms of digital twins. See: https://github.com/w3c/cogai/blob/master/demos/Swarms/tasks/README.md

Discussion on the massive electrical demand of AI got me to look at field of photonics as a game changer for AI:

Today’s ANNs use a lot of electrical power especially for training. This is creating a big demand on the national grid and acts as a downside for the expanded use of AI. The brain uses just a few watts, perhaps 20W, and is five orders of magnitude more power efficient than GPUs. The brain features local analog processing and spikes for long range communication. Can we match that?

A very promising approach is to use light pulses conveyed via optical waveguides as a replacement for electrical conductors within and between integrated circuits. This is faster and uses less power. We can build upon the extensive experience with CMOS fabrication for hybrid photonic devices that combine electronic and optical components in the same chip.

Silicon quantum dots (SiQD) can be used for emitting and detecting light in a wide range of wavelengths from the near infrared to the near ultraviolet. SiQDs can further be used for beam splitters and other optical devices. Waveguides, just nanometres across, can efficiently transport photons, even around tight bends, enabling a high density of communication paths. Resistive random access memory (RRAM) is based upon memristors, and can be integrated alongside CMOS electronic gates and optical waveguides. Memristors can also be used to mimic synapses for neuromorphic computing.

Of course there are many challenges to address before this technology can become widespread. This includes connectivity to photonic chips compared to the ease of bonding wired connections. We are also likely to need new architectures for neural networks that feature continual learning through continual prediction and weight adjustments at a layer by layer basis, replacing today’s gradient-based back propagation. Another motivation is that adversarial attacks suggest that today’s AI is significantly different from human cognition, e.g. adding some carefully chosen noise to an image of a panda causes the model to misclassify it as a gibbon with a high degree of confidence, despite the noise being imperceptible to the human eye. Similar considerations apply to jail breaks that evade the alignment training for LLMs.

In Pisa, I presented my recent work on extending chunks and rules to support messaging and task synchronisation. This is a symbolic approach that emerged well before recent advances in generative AI.  I now want to recast this in terms of artificial neural networks with fuzzy rules and explore how to integrate machine learning to mimic the complementary role of the cortico-basal ganglia circuit for deliberative cognition and the cortico-cerebellar circuit for motor skills. Before that however, I need to extend the chunks & rules test suite and spec to cover the recent extensions. As always, offers of help would be much appreciated.

Dave Raggett <dsr@w3.org>