I am not sure if it matters that some people struggle to understand you. Have you seen The Trailer Park Boys? Some Canadians sound very strange too
I have read that quote about so many British comedians, and it always comes from Americans. Here is a fact: Some Americans struggle to understand other Americans from different states
Be proud of your background and accent. For me its not just your jokes that I like, its your turn of phrase and use of regional dialect. It has worked for many comedians to stay truthful to their backgrounds. You ain't winning these gong-shows for nowt