Kaltenstein Castle – probably originally a fortress with a curtain wall and mighty walls of ashlar stonework that have survived up to the present day – was first documented as far back as 1096 as "castrum vehingen". First owned by the counts of Vaihingen, it was acquired in 1339 by the counts of Württemberg, who appointed a High Steward to live there. In the 16th century, and again in 1734, the castle was converted and its fortifications strengthened. It served as a garrison and military hospital up to the end of the 18th century, and from 1843 onwards as a workhouse. In 1949 it was taken over by the Christliches Jugenddorfwerk Deutschlands, a Christian youth organisation.