In your definition if you introduce or remove a bug your not refactoring. More basically for things like simulations or graphics you may trade accuracy for speed so your code may behave slightly differently, but be much faster. The important bit with refactoring is the goal is not fixing a specific bug, however realistically for any sufficiently large change bugs can and will be both added and removed.

Edit: "Typically, refactoring applies a series of standardised basic micro-refactorings, each of which is (usually) a tiny change in a computer program's source code that either preserves the behaviour of the software, or at least does not modify its conformance to functional requirements." http://en.wikipedia.org/wiki/Code_refactoring

I feel we may be getting into bike-shedding territory here. When I'm refactoring, as part of the TDD Red/Green/Refactor cycle, I'm changing how code is implemented, usually to improve the design and readability, such that the functionality stays the same e.g. all of my tests pass without touching them. Anything I gain from the refactoring is bonus on top.

Refactoring without introducing bugs is the #1 reason people do TDD vs Test After. But if you don't trust the test suite to catch bugs introduced when refactoring then I'd argue the test suite itself probably isn't that useful and needs work.

On a side note I believe the term "refactoring" has been way overused these days. You can change code without it being "refactoring". When I've seen people use the term "refactoring" (and I've caught myself multiple times on this) they mean rewriting, for in most cases this rewriting means changing the code and changing the tests. If you have to change the tests because of changes to the code, that's not refactoring, that's just changing code.