This New AI Could Be the First Real ARTIFICIAL BRAIN!
AI Summary
Summary of the Topographic Language Model (Topo LM)
- Introduction:
- New AI model, Topo LM, mimics brain-like clustering for language learning.
- Developments from EPFL’s neuroai lab, led by Assistant Professor Martin Cramp.
- Neuroscientific Background:
- Previous research shows specific brain clusters for verbs, nouns, and syntax.
- The model aims to replicate this clustering in a computational framework.
- Model Architecture:
- Based on GPT2 with 12 transformer blocks and a 28x28 grid for neuron organization.
- Introduces a spatial smoothness loss during training to maintain correlation between nearby units.
- Training Process:
- Trained on 10 billion tokens from the fine web educ corpus using NVidia 100 GPUs.
- Achieves a cross-entropy of 3.075 and a spatial loss of 0.108.
- Performance Analysis:
- Shows clustering of language-selective units across layers similar to human brain responses.
- Demonstrates meaningful differences in activation for normal sentences versus scrambled and jabberwocky sentences.
- Compares favorably to human brain’s verb-noun selectivity, achieving substantial clustering.
- Model Limitations and Comparisons:
- Performs slightly behind in syntax accuracy but equals or surpasses in downstream NLP tasks.
- Topo LM provides better interpretability compared to traditional transformers by visually mapping clusters.
- Potential Applications:
- Insights for neuromorphic chips and better understandings in cognitive neuroscience.
- Promising avenues for clinical applications, guiding stimulation for stroke recovery.
- Conclusion:
- Topo LM represents a significant step towards integrating AI and biological realism while maintaining performance. It suggests broader implications for understanding cognitive processes and advancing artificial intelligence.