No! That is a completely wrong approach to Social Media though process. Your consumers do not need your permission to indulge in a conversation related to your brand or company. If you do not take part in discussions or conversations, they will still occur regardless of your absence. But a setback would be you would not be able to voice your opinion if you have not been actively taking part in the discussions.
Another reason would be, it is human nature to whine and comment about someone who is not in the room, but it takes a lot more guts to talk or comment about someone, when they are listening. This applies to Social Media as well, if your audience knows that you are listening and watching their moves, this can create a psychological trigger and limit the number of negative comments in your social media profile.
If you are inactive in Social Media, this means that you are wasting a precious platform which will give you valuable feedback about the likes and dislikes of your target consumers.