You can't always swap. For example, you may want to use a movable but non-copyable type to keep un-aliased access to a single resource (see linear/affine types) and the compiler treating the move source as dead is useful there. Or maybe you don't want to allocate anything for the move target because the resource you're managing is expensive.
Keeping track of when a destructor gets called is not a huge problem at all. From the user perspective it's airtight, proper uninitialized value analysis in the compiler make it impossible to screw up. From the implementation perspective all you need is the occasional flag on the stack (whose value is already calculated) for situations like `if ... { use_by_move(obj) }`, and a rule against moving things out of fields that you're otherwise not responsible for destructing (you can still swap there).
Keeping track of when a destructor gets called is not a huge problem at all. From the user perspective it's airtight, proper uninitialized value analysis in the compiler make it impossible to screw up. From the implementation perspective all you need is the occasional flag on the stack (whose value is already calculated) for situations like `if ... { use_by_move(obj) }`, and a rule against moving things out of fields that you're otherwise not responsible for destructing (you can still swap there).