Oh, if only I had a dollar for everytime I've heard this phrase! I'd be a millionaire, surely?!
Often it comes from mature adults and chances are it is a self-held belief that developed in childhood - where they were laughed at, or told by a teacher that they shouldn't sing, 'just mouth the words'.
It is such a limiting belief that has been carried for a lifetime. A limiting belief that has robbed that child/person from one of the great delights of being alive - the sheer joy of making music.
The reality IS that anyone who can talk, can sing. Singing is a natural gift given to all of us [show me one newborn baby that doesn't exercise its lungs before anything else!]
In some families, some cultures, singing is encouraged, fostered, celebrated. In other homes it is not and so the art can be lost.
Singing in a group is a great way to re-discover that natural gift, to foster it, and to develop it into something lovely. It's also a great opportunity to meet new friends and be part of a like-minded community.
Don't let a limiting belief - a FEAR - limit your life.