E. Cobham Brewer 18101897. Dictionary of Phrase and Fable. 1898.
A private room into which no one uninvited enters. The reference is to the Holy of Holies in the Jewish Temple, a small chamber into which none but the high priest might enter, and that only on the Great Day of Atonement. A mans private house is his sanctuary; his own special private room in that house is the sanctuary of the sanctuary, or the sanctum sanctorum.