Well yes, many of the best slapstick artists came from poor backgrounds themselves and based their characters on a 'poor or stupid' stereotype. You're quite right. Also physical humour can be fairly claimed to be base and unsophisticated while verbal humour claims to be higher born and sophisticated by degrees.
The higher the class of verbal wit the greater the case of comedy snobbery. For me, Norman Wisdom is worth at least five Stephen Frys.