A transfer function is a mathematical representation that defines the relationship between the input and output of a linear time-invariant (LTI) system in the frequency domain. It captures how a system responds to various frequencies, providing insights into system behavior, stability, and dynamics.