What happens is a Contact Label owner is deleted?

If the owner of a contact label that is being shared is deleted you will need to transfer label and contact ownership to another user. This can be performed automatically via the Patronum deprovisioning policies.