Many people lived in the castle itself, nobles, the nobles servants, officials, and guest of the noble. But most of the time the servants and officials were worrying about the castles protection from enemies. So they weren't in the really around the nobles a lot.