At the same time, if you overdo the colloquial language you create a structural barrier between your reader and your story; they can't understand what the characters are saying and they have no idea what's going on.
It's really not a sacrifice of verisimilitude to avoid excluding readers with confusing speech patterns. All dialog and narration is essentially adulterated. Although it sounds conversational in your head when you read it, good written dialog is different from the way people actually talk. And anything the reader has to stop and read twice pulls them out of the story. This is true of obscure slang terms, untranslated foreign-language phrases, run-on sentences, big SAT words, and things like references to minor characters that the reader might have to go back and look up
Adding a dialect or accent to a character's speech really doesn't change your goal much; you're still trying to capture speech that feels natural in a way that is clear on the page and doesn't present an obstacle for readers.
You don't want to add foreign language phrases that have to be translated. You don't want to use a lot of slang when the meaning isn't clear from context. You certainly don't want to represent somebody's accent by using a bunch of phonetic spelling. In the same way you don't need to describe every part of a person's face and body to give a reader a sense of the character, you don't need to lay slang and accent too thickly onto your prose to give an impression of how people speak. You can take a look at things like non-standard verb conjugations, peculiar pronoun use for non-native speakers and a few deliberately chosen foreign or slang words to really elicit a lot of voice without hanging up the reader.
Learning to utilize colloquial speech with a light touch is also useful to suggest the speaking patterns of people from backgrounds different than your own; if you apply a very thick coat of somebody else's accent or slang to a character, readers may find the portrayal offensive, especially if you get it a little bit wrong.
Of course, some writers incorporate unexplained slang and colloquial constructions into their writing with the express intention of excluding certain readers. It's an expression of identity as a point of pride, and makes no concessions to readers from outside the author's subculture. The narrative is not intended as an invitation for outsiders to peek into the author's world, so it doesn't matter if some readers don't understand what's going on; nobody invited them anyway.