E. Cobham Brewer 18101897. Dictionary of Phrase and Fable. 1898.
Riff-raff.
The offscouring of society, or rather, refuse and sweepings. Rief is Anglo-Saxon, and means a rag; Raff is also Anglo-Saxon, and means sweepings. (Danish, rips-raps.) The French have the expression Avoir rifle et rafle, meaning to have everything; whence radoux (one who has everything), and the phrase Il na laissé ni rif ni raf (he has left nothing behind him).
1
I have neither ryff nor ruff [rag to cover me nor roof over my head].Sharp: Coventry Myst., p. 224.