I guess you could compare the outputs for different engines with that of a "suspicious" player - if their moves match exactly, flag them, if it happens too many times, ban them. Something like that.
Someone who plays a lot against a computer will often pick up the computer's style.
But sometimes a computer will make odd moves no human would ever make. I've been playing against an ios stockfish app to relearn how to play, and when it gets behind it starts throwing material away to delay the inevitable; a human would more likely keep the material and hope the opponent doesn't see the path to victory.
One way to detect use of an engine is to look for moves like this, though if it could be done algorithmicly then that same algorithm could be used to make the engine play more like a human.
