You'll sometimes hear writers talking about whether their work is character-driven or plot-driven. A writer may come up with characters first and then see what they do, or a writer may come up with a plot and populate it with characters. But to a large extent, fiction is really character-driven because the plot is what happens when characters act and react. You've probably encountered stories that either felt flat or that made no sense because the plot had the characters acting and reacting in ways that didn't fit who they were supposed to be and what they were experiencing. It was clear that the writers had events that they wanted to happen, and the characters were forced to go along with it. That leaves readers going "no one would ever do that" or "there's no way he would have done that."
The thing is, within the world of your story, characters are people, and they need to act like people. You start off with what I think of as the "reasonable man" assumption. When you're on a jury, they talk a lot about a "reasonable man" -- would a reasonable man, given certain information and circumstances, interpret things in such a way or behave in such a way? Readers expect something similar of characters. They put themselves in their shoes (especially viewpoint characters) and think about how they'd react. If someone threatens or harms their child, that someone is probably in trouble. A "reasonable" person wouldn't then go have drinks with that person and listen to their sad life story. If someone is being friendly, a "reasonable" person isn't going to start attacking them.
If your character has reasons for not acting like we'd generally expect people to act, that needs to be established and set up properly. Is there some kind of psychological or emotional trigger at work here? Does the other person remind him of someone in his life? Do they have a history? Has the character been given information about this person (that may or may not be accurate)? In a sense, by changing the context, you're keeping the character reasonable, because readers will think "yeah, under those circumstances, I might do the same thing."
Or maybe your character isn't meant to be reasonable, but that also needs to be established, and it's tricky to pull off if you expect readers to relate and care for the character. This is more likely to be in the villain realm, where the character can be a total psychopath and do things that no one else would do. Even here, though, there needs to be some pattern and method to the madness rather than the character being a total wild card who can just do anything at random.
A related issue is what I refer to as "Idiot Plotting," where the plot only works if the characters act like total idiots -- like the horror movie cliche of going alone into the dark basement without a flashlight or weapon when the characters know there's a killer on the loose.
Beyond the general "reasonable" standard, there's what you establish for your character as normal -- likes, dislikes, fears, hot-button issues. Once you've established these, you can't make your character go against them without a good reason. If you've established that your hero was abandoned as a child and therefore has a huge soft spot for abandoned kids, he's probably not going to leave the orphan lying in the gutter so he can chase after the bad guy's minions unless his reason for chasing the minions is stronger than his abandonment issues sympathy. Your wary, untrusting heroine can't suddenly trust what a shady character tells her just because you need her to so she can get herself in trouble and advance the plot.
But what if the plot you have planned won't work if your characters behave reasonably or according to the way you've developed your characters? You could try changing your plot plans. Think about what these people really would do in that situation and see where it takes you. You can also adjust the circumstances to make the choice you need more likely. In the example above about the abandonment issues hero, someone could step up to tend to the orphan child, allowing him to feel less like he's abandoning the child. You can provide more set-up -- the shady character can provide lower-level reliable information earlier in the story that gives the wary heroine a reason to trust him. You can provide additional motivation to show that this situation is an exception. You can change events earlier in the story to make things more reasonable -- if you want your heroes to team up with a former villain against another villain later in the story and to kind of become friendly, you can't have that villain murdering their families first. Let your future good-guy be less directly villainous, especially to your protagonists.
The bottom line is that you never want to throw someone out of your story with the "no one would ever do that" thought. You also lose a lot of emotion if your characters aren't allowed to react reasonably to the things that happen to them, and a reasonable reaction is going to affect the actions they take. All of this is key to encouraging your readers to relate to and care about your characters.