You cannot enter the Prison 2 hours prior to a siege. Access will be granted once the siege ends. Also, you can't access the underground dungeon until a castle lord announces the siege after the battle.