It seems to make little sense to me that adventurers would be tied to a specific lord, but it does make sense for them to be tied to the ruler. Perhaps some nominal fee to the ruler for depriving them of a subject? Or maybe have it go wherever it is that fines go?