That would be the case if you don't have a separate expansion (accumulator) tank. The water heater tank acts as an accumulator tank.
That's the reason why, in some cases, when taking an onboard shower you'll get a blast of scalding hot water when the pressure pump comes on - the hot water side and the cold water side are out of balance. Theoretically, you should have an accumulator tank on the cold water side that is the same size in gallons as the water heater.
This doesn't happen at home because the piping is way longer than onboard a boat, where you have a small, tight system that can easily become "unbalanced" when the pressure on one side is higher than that on the other side.
So getting back to the original topic, if you're going to take the water heater out of the system I'd suggest you either add an accumulator tank on the cold water side or get one of those variable-speed pumps that maintain the same pressure regardless of how much water you are drawing. Otherwise, you'll have a situation where the pump will come on as soon as you open the water tap and will shut off as soon as you turn it off - and the pressure will be all over the place.