For relatively straightforward groups whose tasks are self explanatory and/or isolated, sure. For any kind of invention, however, I don't think this line of thought works. Alan Kay has said in a number of places that Xerox PARC (likely) couldn't have worked if they had been remote, nor could it have worked if they hadn't had the time they did to goof off, argue, share hypotheticals, etc. Developing a shared intuition and connection with others in your group is an immeasurable quality that doesn't get enough love.

In other words, if your organization is made up of a bunch of geese that lay golden eggs, by all means let them lay eggs, but don't ignore the fact that your organization could itself be a goose that lays golden eggs together.