Crenation is the process where red blood cells shrink and acquire a scalloped surface due to the loss of water when placed in a hypertonic solution. This phenomenon is a result of osmosis, where water moves out of the cell to balance solute concentrations.