The so-called Cabinet House (Stjornarrad) has carried a few names since it was built as a prison during the period 1765-70.  The original idea was to accommodate vagabonds who were fit enough and able to work for the pioneer woolen industies of the country.  The governor of Iceland at the time, Skuli Magnusson, was an ideologist, who wanted to create employment possibilities for the agricultural population of the country by building those factories.  He managed to see his dreem come true, but workers were hard to come by.
The Danish government of Iceland at the time, however, decided that the house should serve as a prison, which it did until 1816.  It probably stood empty for three years, until the Danish governor, count Moltke, arrived and found his lodgings uninhabitable and his office space unusable.  He managed to have the government fund the reconstruction of the house and moved in with his family and officials.

In 1874, the Icelanders celebrated the millenium of the inhabitancy of the country, and the Danish king, Christian IX, participated.  He also brought along the country's first constitution and handed it over in the ancient Parliamentary Plains (Thingvellir).

In 1904, Iceland was granted home rule by the Danish government.  This announcement was read publicly on the steps of the Cabinet House and the official flag of the country was hoisted in front of it for the first time.  Consequently the house was occupied by the three ministers of the Icelandic government and got its present name.

In the late 19th century, vegetables were grown in the garden, and the first trees were planted there in 1834.

Now the house is occupied by the offices of the prime minister.  The offices of the president of the republic were moved there in 1973 and remained there until a political opponent of the prime minister was elected president in 1996.  The prime minister did not waste any time and evicted the president promptly.

The statues in front of the house are the work of the sculptor Einar Jonsson from the years 1915 and 1931.  One depicts King Christian IX handing over the constitution in 1874, and the other Hannes Hafstein, who became the first minister of the country in 1904.