When I look at the colour calibration charts while taking the photos, depending on the lighting conditions and even time of day (e.g. if it's very late at night and I'm feeling tired), I often see the white square not as a neutral white colour, and the black square as being not as dark as the dark grey square next to it, so I've learnt not to trust just my eyes as to whether something is black or grey, white or off-white, red or magenta.
I actually think the $100 or so I spent on the commercially available colour calibration charts are a good investment for me in this hobby; those I trust to remain pretty much consistent all the time -- and I do keep them in opaque sleeves or in a drawer when not in use, to slow down any fading or discolouration due to exposure.
The color calibration chart really helps.
Different light sources will make colors look different in real life as will un-calibrated monitors for digital images.
Some colors will not photograph or scan as we see them.
When I photograph for interior designers or copy artwork having an original sample is sometimes essential for color matching the digital file even when I include a color calibration chart in the image.