Right, but most deviations one can think of (like, changes over time to physical "constants") would have an observable effect, so ancient galaxies would look much more different from modern galaxies and spectra wouldn't look the same other than a red shift, which moves all the lines in a uniform way.