Then you're not refactoring. Refactoring is defined as changing how the code is implemented without affecting the functionality / behavior of said code.
Also, Unit tests are where you ensure the non-happy-paths are functional (error handling, input robustness, etc).
That's a ridiculous definition, that's like suggesting coding is turning requirements into bug free computer code. Refracting is attempting to replace working code with code better suited to your long term goals.
The standard definition of refactoring is that it doesn't change behaviour. As Martin Fowler put it:
Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure.
You correctly identified a consequence of this in your later comment:
> In your definition if you introduce or remove a bug your not refactoring.
That is indeed the case.
And also correctly observed that:
> realistically for any sufficiently large change bugs can and will be both added and removed.
If a change adds or removes a bug, it was not a pure refactoring. It may have been an attempt at a pure refactoring, and it may have been otherwise successful, but the introduction of a change in behaviour means that it was not purely a refactoring. This isn't necessarily a bad thing - i'd rather bugs be removed than not! - but it's possible to distinguish the refactoring and bug-fixing aspects of the change. Ideally, i would like to see those aspects formally separated, for example into separate commits in source control. But even if not, we can at least use the terminology correctly and precisely.
The term refactoring dates back at least to the 80's well before TDD showed up and it was used as a shorthand for cleaning up code without the focus on tiny changes. More importantly bugs are often related to side effects such as how long a method takes as such using even one more or one less cycle any any code path prevents code from being 'pure' in your definition as it may add or remove a bug as would changing it's memory footprint etc.
Granted, that may seem pedantic but if you look closely you realise nobody uses your 'ideal' definition in practice.
PS: Feel free to use / introduce new terms such as 'Pure refactoring' but understand they don't change what refactoring actually means. As to popularity wikipedia's "does not modify its conformance to functional requirements" suggests that's the commonly understood definition.
In your definition if you introduce or remove a bug your not refactoring. More basically for things like simulations or graphics you may trade accuracy for speed so your code may behave slightly differently, but be much faster. The important bit with refactoring is the goal is not fixing a specific bug, however realistically for any sufficiently large change bugs can and will be both added and removed.
Edit: "Typically, refactoring applies a series of standardised basic micro-refactorings, each of which is (usually) a tiny change in a computer program's source code that either preserves the behaviour of the software, or at least does not modify its conformance to functional requirements." http://en.wikipedia.org/wiki/Code_refactoring
I feel we may be getting into bike-shedding territory here. When I'm refactoring, as part of the TDD Red/Green/Refactor cycle, I'm changing how code is implemented, usually to improve the design and readability, such that the functionality stays the same e.g. all of my tests pass without touching them. Anything I gain from the refactoring is bonus on top.
Refactoring without introducing bugs is the #1 reason people do TDD vs Test After. But if you don't trust the test suite to catch bugs introduced when refactoring then I'd argue the test suite itself probably isn't that useful and needs work.
On a side note I believe the term "refactoring" has been way overused these days. You can change code without it being "refactoring". When I've seen people use the term "refactoring" (and I've caught myself multiple times on this) they mean rewriting, for in most cases this rewriting means changing the code and changing the tests. If you have to change the tests because of changes to the code, that's not refactoring, that's just changing code.
Well, then by that definition you are not refactoring either, and in fact nobody is refactoring. Because any change can unintentionally change the behavior. That's what we call a bug, and them bugs don't care about your definition.
And Unit tests alone cannot ensure that the non-happy-paths are functional.
