I've been struggling with something similar. Where I've landed (for now at least) is to ask, "What does this game give me? Will I be happier/better when I'm done? Is there nothing I'd like to do instead?"
If I have good answers, I'll let myself play the game. "I've been programming all day and my brain needs a break," is fine. But if it comes out that I'm playing because I just want to fill time or am procrastinating taking the next step on a project, I put it down and do something else. Doesn't have to be the thing I was procrastinating, just something that won't automatically expand to fill all available time.
If I have good answers, I'll let myself play the game. "I've been programming all day and my brain needs a break," is fine. But if it comes out that I'm playing because I just want to fill time or am procrastinating taking the next step on a project, I put it down and do something else. Doesn't have to be the thing I was procrastinating, just something that won't automatically expand to fill all available time.