The fact that you leave early is not rude, but the way you do it might be. He should find the lead developer (or whoever was in the lead of interviewing him) and tell him that he feels he won't fit into the culture, thank him for the interview and then leave.

Leaving without a word is in fact rude (“I'm done, show me the elevator” doesn't count, especially if lead developer had to jump the elevator to ask what's wrong and why he's leaving).