people simply as a gesture of politeness, and he does not really mind whether you attend or not. However, if you know that your presence genuinely matters to him because he is a relative or a friend, then do accept the invitation.