Internal nodes are the points in a tree diagram that represent non-terminal symbols, which help to show the hierarchical structure of phrases. These nodes are crucial because they connect terminal nodes, or actual words, to each other and illustrate how phrases are constructed according to phrase structure rules. Understanding internal nodes is key to visualizing the relationships between different components of a sentence.