Decreasing returns to scale occurs when a proportional increase in all inputs results in a less than proportional increase in output. This means that as a firm or production process scales up, the efficiency of converting inputs into outputs diminishes, leading to higher average costs per unit of output. This phenomenon is crucial for understanding the limitations of expanding production and the optimal size of firms.