A peel tower was built on the site during the 14th century. It passed into the hands of the Sandford family and in 1575 Thomas Sandford had it substantially enlarged. It became a rectory in 1828 and then became a residence of the Lowther family in the 1830s. The 7th Earl of Lonsdale used it as his home after Lowther Castle was dismantled and closed in 1937. Askham Hall became a Grade I listed house in 1968.
Following the death of the 7th Earl in May 2006 the house has been owned by Caroline, Countess of Lonsdale. In 2012 the Countess of Lonsdale and her son, Charles Lowther, arranged the conversion of Askham Hall into a boutique hotel.