I see it very often that people who say 'yes' = good people, and people who say 'no' = bad peoples. I've certainly worked in places where turning down drinks after hours isolates you socially. I have also worked in places where those that say 'yes', typically are quite insufferable. I think for many years I have tried to be a 'yes' person but undercover. For example if I agree to drinks and I am aware there is another group that is also forming - with people I am less inclined to socialise with - I will steer the two groups apart 'Oh but I wanted to go somewhere I could eat'. I know this makes me a pretty bad person. More recently I have been able to be more upfront, and tell people I will 'see how it pans out'. If the group gets too big - which typically I can't deal with - then I will bow out.

TLDR - It's a minefield peoples.