An enigmatic branch of the human family tree, Neanderthals lived in parts of Europe, Central Asia and Middle East for up to 300,000 years but vanished from the fossil record about 30-40,000 years ago.
Why they disappeared is one of the hottest topics in anthropology.
Theories say they may have been victims of climate change or were massacred by their Homo sapiens cousins.
Now experts from the University of Oxford and the Natural History Museum in London suggest the answer could lie in available brainpower.
Neanderthals were stockier than anatomically modern humans who shared the planet with them at the time of their demise, but their brains were the same size, the team write in the journal Proceedings of the Royal Society B.
As a result, Neanderthals "would have required proportionately more neural matter" to maintain and control their larger bodies, they say.
Comparing the skulls of 32 Homo sapiens and 13 Neanderthals, the researchers also established the hominids had bigger eye sockets, indicating bigger eyes and visual cortices - those areas of the brain that regulate vision.
"More of the Neanderthal brain would have been dedicated to vision and body control, leaving less brain to deal with other functions like social networking," explained Oxford anthropologist and lead author Eiluned Pearce.
Among living primates and humans, the size of an individual's social network is constrained by the size of specific brain areas, she said.
The larger these areas are, the more connections an individual can maintain.
The archaeological record seems to support the theory that Neanderthals were cognitively limited to smaller groups - they transported raw materials over shorter distances and rare finds of symbolic artefacts suggest a limited ability to trade.
The ability to organise a collective response would have been a key to survival when times turned harsh, like during the Ice Age, Pearce told AFP.
"If Neanderthals knew fewer people in fewer neighbouring groups, this would have meant fewer sources of help in the event of, for example, local resource failure," she said.
"Smaller groups are also more liable to demographic fluctuations, meaning a greater chance of a particular group dying out. Smaller groups are less able to maintain cultural knowledge, so innovations may be more likely to be lost.
"Overall, if Neanderthals had smaller groups/social networks, this could have led to their extinction along a variety of pathways."