Associative Memory

This is further to my previous email on the role of associative memory for continual learning, including sequence learning. I am now scouring the research literature for useful accounts of associative memory that I can implement to gain practical experience.  The established approach uses recurrent neural networks and Hebbian inspired learning rules.  You present the cue vector and the output stabilises to the previously stored data vector for that cue.  I am mainly interested in hetero-associative networks where the cue is different from the data. For that, the papers suggest using bidirectional associative memory (BAM) as a variant of Hopfield networks, which are used for auto-associative networks, where the cue is a degraded or noisy version of the data.

Some challenges include:

The memory capacity for a given sized network
Support for single-shot learning
How long the network takes to settle on recall operations
How to minimise false recalls on spurious energy minima
How to support activation and decay as a basis for short term memories
How to generalise the simpler models from binary to real-valued cues and data

In the brain, there are lots of lateral connections, e.g. around 10,000 lateral connections for each pyramidal neuron in the cortex.  These lateral connections can be inhibitory or excitatory, and are thought to support sparse coding for winner-takes-all operations as well as mimicking hidden Markov models. How can BAM be extended to exploit lateral connections?

For speech and language understanding the recall time needs to be very short.  For Type 2 cognition it can be much longer, and cognitive science experiments on memory provide some valuable insights, including the effects of interference, spreading activation, the spacing effect and forgetting curve.

For long term memory forgetting is now thought to be dominated by interference from more recent memories.  For short term memories, forgetting may be due to chemical changes in synaptic connections. I want to explore this in terms of composing synaptic connections as blend of short and long term components.

One paper of interest: “Associated memory of structured knowledge” https://www.nature.com/articles/s41598-022-25708-y

Anyone interested in helping with this study?

Dave Raggett <dsr@w3.org>