Of course your daughter's feelings matter. And of course it's OK that you don't want your daughter to feel uncomfortable.

That said, it's also your job as a parent to help your daughter learn to step outside her comfort zone, especially when it will likely reap benefits for her (from her perspective) in the future.

Transitioning between two households is always difficult for a child of divorce - but it is the transition, not the father, that is the source of her angst. Unhappiness will always happen whenever a child is forced out of a routine.

===

I think the point you may be missing is - if she were living regularly with her father, and only spending weekends with you, she would likely have the identical behavior when it's time for her stay with you.

You also have to take into consideration how she will feel about this when she's older. If she (in later years) expresses regret that she doesn't really know her father, and you explain it's because "You let her (a 5 year old) make the decision" ...

... she may well be furious at you. Honestly, many adult children are, after identical circumstances.