There are many reasons the chart can be off, or you think it may be off. First of all, buoys (and any type of floating aid) are not to be relied upon as a means of position fixing, as their anchor chains allow for some swinging, meaning that the actual position of the buoy at any given time, may be off from its charted position. Strong storms or waves can cause the buoy to drag its snker, as can collisions with vessels. You will often see a note on a paper chart regarding this. Second, your GPS receiver must be selected on the correct datum for the chart, most likely WGS84. There should be an option to select this somewhere in the unit's menus. Third, GPS accuracy is based on the number of satellites it is receiving. An obstruction can reduce the antenna's vision of the sky considerably, resulting in lost satellites. Fourth, the U.S. Government reserves the right to scramble the GPS signal at any time, without warning. The offset is not constant and can jump around. While I doubt that this is the case here, every user should be aware of this. Finally, GPS even when operating at peak capability, is really only accurate to about 8.5 meters, depending largely on atmospheric (not necessarily meteorological) conditions. Differential GPS (DGPS) can get this down to about 3 meters. WAAS systems can reduce it even further to 1-2 meters. In addition, as the accuracy gets down to this small of a distance, you have to take antenna location on the boat into account.
The last thing to consider is the accuracy of the chart itself. Many charts are based on surveys done long ago and it's hard to keep every chart up to date with the latest surveys that are using the most modern and accurate techniques.