Shaykh Ubayd bin Abdillāh Al-Jābirī: This is from the kinds of forbidden khawf (fear). So if you enter into an act of worship, then do not cut it off! The Scholars say that this is from Minor Shirk. i.e. leaving an act of worship due to fear of the people criticizing or mocking him; the Scholars say that this is from Minor Shirk. So if you enter into an act of worship, then complete it.
But you may delay it a little, for example when visitors visit you and you want to perform two rak‘ahs (prayer units) of Duhaa, then receive them and prepare for them that which is prepared for guests, then you pray Duhaa in the place where they are or in another place – no problem. Na‘am.
Likewise, if you leave your voluntary acts of worship (nawaafil) due to being busy with your guests, and not due to fear of them, then there is no problem in this. And with this, it can be said that leaving an act of worship… And what is intented here is the naafilah (voluntary act), like the two rak‘ahs of Duhaa or the recitation of the Qur’aan – you have a specific amount that you recite -. So leaving it has two situations:
One of them: that he leaves it due to fear of the people, meaning: he fears their criticism, mockery and their harm. This is not permissible! This is forbidden, as we have preceded.
The second: that he leaves it so that he is able to receive them (i.e. the guests) and to prepare for them that which is prepared for visitors, then he continues after that (with his act of worship), even if he delays it until the guests leave – there is no problem in this inshaa-al-Laah. Because some acts of worship do not have a fixed time (to perform them), like the recitation of the Qur’aan. But the two rak‘ahs of Duhaa can be missed (since they have a fixed time), so if you don’t like to leave them off, then you prepare for your visitors that which is prepared for guests at the moment they enter (your house), then you pray the two rak‘ahs of Duhaa. Na‘am. So there is no problem in this inshaa-al-Laah. Even if you left it, but [only] due to being busy with the guests and not due to fear of them, then there is no problem in this inshaa-al-Laah.
Source: http://ar.miraath.net/fatwah/6468
Translation: Yâsîn Abû Ibrâhîm al-Mutanabbî