If the compensation claimed and disputed shall not exceed fifty pounds, unless both parties agree to refer such compensation to arbitration, the same shall be settled by the sheriff.