“except in the mosque.” This wording is reported as a Companion-level narration from ʿAlī ibn Abī Ṭālib (may Allah be pleased with him) and is also transmitted—attributed to the Prophet ﷺ—as a Prophetic ḥadīth. In either case, it indicates that the neighbour of the mosque is required to perform the prayer in the mosque.

The “neighbour of the mosque,” as the scholars explain, is anyone whose house lies within forty houses of the mosque—that is, the zone of neighbourhood extends to forty houses in every direction around it.

The controlling criterion here is whether one can hear the adhān. If you can hear the regular call to prayer without the aid of loudspeakers, it becomes obligatory for you to pray in the mosque and to answer the caller—unless a recognised sharʿī excuse prevents you, such as illness. Likewise, the situation you mentioned—that when you leave for the mosque at night your wife becomes frightened and feels unsafe—is a valid legal excuse that allows you to pray at home.
***
A Muslim prays according to his circumstances

Question: I travelled to a European country during a period of severe cold. Because I did not possess an entry visa, the authorities detained me in the airport for six days. During those six days I did not perform the prayers, owing to the extreme cold, the intense chill of the water, and the absence of clean earth with which to perform dry ablution. I remained in that state until I returned to my home country, after which I made up all the prayers I had missed during those days. Am I sinful for what I did? If there is something further I must do, please advise me. May Allah reward you.