Re: Work on cognitive neural network architectures

On 1 Feb 2024, at 19:14, Aric Whitewood <aric.whitewood@xaimacro.com> wrote:
> 
> Thanks Dave - just briefly on circular convolution, this is achievable using the Fast Fourier transform (FFT) and the inverse FFT. It has been almost 20 years since I’ve had to calculate this though!
> 
> For two signals, call them s1 and s2, the convolution in this case is ifft ( fft(s1)*fft(s2) )


Thanks - I’ve managed to figure out the python code for circular convolution and its inverse using discrete Fourier transforms.  I have also coded a more direct approach involving progressive rotation, but not yet for the inverse case. However, I hear that the FFT approach is supposedly faster for large arrays, so that may be moot.

I next need to figure out how to adapt code examples for using transformers to a cognitive architecture that more closely mimics human language processing.  I plan to use tiktoken for the tokeniser, and to start with the language model datasets provided with PyTorch.  The code for embedding positional information will be interesting in respect to a sliding window over the token sequence.

Once that is working as expected, the next phase of work will be to mimic sequential cognition acting on working memory.  I have a rough idea for how to implement that, but not yet how to train the cognitive rules needed for different tasks.  It may require work on mitigating task inference, e.g. using some form of lateral inhibition.

Best regards,

Dave Raggett <dsr@w3.org>