Moving the aft edge of the anti-steering torque ("trim")tab on the anti-cavitation plate a bit to the port side should reduce the tendency to pull the wheel to port.
Also, remember trimming the engine/drive up or down effectively changes the prop pitch relative to the flow of water so when the prop is not level the effective pitch on one side of the prop will be greater and less on the other, and will tend to change steering torque. Trimming an O/B or I/O with RH prop down from level-shaft should tend to make the boat easier to steer to stbd,... trimming up easier to the right. Obviously this would be much more apparent with feedback cable steering.