To be able to keep any Fealty Building you need to put a Permanent Talent point in it. A Permanent Talent point can only be obtained during a Reincarnation, not a Restart, and cannot be changed after it is placed in a Talent. Reincarnation is only available if you have finished all the Quests in Volume 1.
Prior to our April 14th, 2014 game build, some players were able to incorrectly keep the Fealty Buildings without putting a Permanent point in them. This bug has been fixed, and Fealty Buildings may not be acquired without putting Permanent Talent points in those buildings after Reincarnating.