E. Cobham Brewer 18101897. Dictionary of Phrase and Fable. 1898.
Fancy.
Lovei.e. the passion of the fantasy or imagination. A fancyman is a man (not your husband) whom you fancy or select for chaperon.
1
Tell me where is fancy bred,
Or in the heart or in the head.
Shakespeare: Merchant of Venice, iii. 2.
The fancy. Pugilists. So called because boxing is the chief of sports, and fancy means sports, pets, or fancies. Hence dog-fanciers, pigeon-fanciers, etc.