In 1520, it was owned by Margaret Boleyn, grandmother of Anne Boleyn, the second wife of Henry VIII of England. In the early twentieth century the important Gothic revival architect Thomas Garner owned the manor.
The house now may have some 16th-century elements, but was mainly built for George Yorke in 1619. 'The Celebrated Captain Barclay' (Robert Barclay Allardice), Regency sportsman, lived there briefly 1815-16. It was restored in the late nineteenth century and underwent further restoration and enlargement in the early twentieth century.
It is a grade II* listed building.
- Loades, David (2011). The Boleyns: The Rise & Fall of a Tudor Family. Amberley Publishing. p. 38. ISBN 978-1445603049.
- Historic England. "Fritwell Manor (Grade II*) (1266393)". National Heritage List for England. Retrieved 23 October 2013.
- "'Manor' class details, 7800 - 7829". Great Western Archive. Retrieved 18 January 2013.