Baldwin de Meulles built an earth-and-timber motte-and-bailey castle here between 1068 and 1086; a stone keep followed. In the early 14th century, Hugh Courtenay II reshaped it as a residence and hunting retreat, enlarging the keep and adding a hall, kitchen, lodgings and gatehouse works. The spread of buildings reflects that later expansion. Henry Courtenay, the last of the family to own the castle, was accused of involvement in the Exeter Conspiracy and executed in 1538. The Crown seized his lands and the castle was abandoned. J. M. W. Turner sketched the ruin in 1814.