If he is serving as the imam, any congregant who realizes that an extra act has been performed must alert him—the men by saying “Subḥān Allāh,” and the women by clapping—whereupon the imam is obliged to heed their signal and return to what is correct, unless he is certain he himself is right, for their alert guides him back to correctness. In the same way, they must draw his attention to any omission that occurs.
As for additional utterances—such as reciting while in bowing or prostration, or reciting a surah in the last two rakʿahs of a four-rakʿah prayer or in the third rakʿah of Maghrib—if one does so out of forgetfulness it is recommended that he perform the prostration of forgetfulness.
The second situation is the opposite: omitting something from the prayer unintentionally. If what is left out is a pillar, and that pillar is the opening takbīr, the prayer is not valid at all and the prostration of forgetfulness cannot remedy it.
If the omitted pillar is something other than the opening takbīr—such as a bowing or a prostration—and he remembers it before beginning the recitation of another rakʿah, he must at once return, perform the missing pillar and everything that follows it. But if he remembers only after starting the recitation of the next rakʿah, the rakʿah from which the pillar was omitted is nullified and the rakʿah he is now in takes its place, because a pillar that cannot be made up has been abandoned once he engaged in the following rakʿah.
If he does not notice the omitted pillar until after giving salām, he treats the matter as though he had left an entire rakʿah. Provided the interval is not long and he is still in a state of purity, he should perform a complete rakʿah, then make the prostration of forgetfulness and conclude with salām.