Referencing that paper is a red herring. It's a flaw in one specific dataset, not with BERT or any model. Everything GP posted about about BERT models answering questions, performing coreference resolution, etc, is still true and is not at all affected by the flaw in this one dataset (out of many that these models have been tested on). Heck, you can just try out any of these models yourself on completely novel questions you come up with, and see that they work.