Kaldadarnes was and still is a large estate just east of River Olfusa in the Floi County.  One of the oldest sources relating to it is the ferry deed granting the estate the ferry rights between Kaldadarnes and Arnarbaeli, and Kotferja and Kirkjuferja further upriver.  The Haukdaelir dynasty owned the property for a while and the chieftain Gissur Thorvaldsson lived there in 1252 and again 1257-58, after he had been constituted earl by the Norwegian king.

The estate became a church site rather soon after the acceptance of Christianity (1000) and the Catholic churches were dedicated to the Holy Cross.  The church possessed a crucifix much revered by the public and attracted a great number of people every day of the year.  The pilgrims brought with them gifts, which they pledged to the crucifix for the solution of their problems.  According to a legend, a crowd of 40-50 people once filled the ferry across the river.  It capsized on the way and everybody drowned.  Pilgrimages were organized from the different parts of the country, and for some of the participants it was sufficient just to be able to see the church from a distance to solve their problems.

The first Lutheran bishop of the southern see, Gissur Einarsson, had the Holy Cross removed right after the reformation.  It was taken to the bishopric Skalholt, where a later bishop, Gisli Jonsson, had it cut and burnt.  Many people tried to find splinters from it or the ashes to keep as holy relics.  The fact, that bishop Gissur was taken ill and died soon after the removal of the cross, rather increased its reverence.  Bishop Brynjolfur Sveinsson donated another crucifix to the church to re-reimburse for the one, which was destroyed.  Nowadays this artefact is on display in the National Museum.  The last church was built at Kaldadarnes in 1863.  It was deconsecrated in 1903 and demolished a year later.

The leper asylum at Klausturholar in the Grimsnes County was moved to Kaldadarnes in 1754 and operated there until 1846.

During the Second World War the British built a large base and an airport on the premises.  The base was abandoned after severe floodings of River Olfusa, but was claimed to have served a very useful purpose in sinking German vessels and u-boats around the country.  Ruins of a few constructions still remain.  After the war, in 1945-48, a rehabilitation centre for alcoholics was operated at Kaldadarnes.  Among the benefits of the estate is the salmon catch and the increasing trout and char catch in River Olfusa.  The eider colonies on the islets of the river have also been growing bigger.