At the time of the Christian Reconquest of the Iberian Peninsula, the primitive streak Limits Portucalense county with the lion kingdom was developing along the left bank of the river Sabor to its confluence with the creek Angueira. This streak was guarded by four main sentinels: Castle Million, Castle Santulhão (both have since disappeared), the Castle of Miranda Outeiro (in ruins) and the Castle Algoso. Complemented the main defense of the northeast sector of the castles Roias Feathers, Mogadouro and although more distant, the Bragança.

The most recent studies indicate that the building of the original structure of the castle dates back to sometime in the end of the reign of King Afonso Henriques (1112-1185), when his son Sancho had exercised the royal power. As recorded in the inquiries of 1258, its builder was a local lord, Mendo Bofino (or Mendo Rufino), which in turn received the lordship of Vimioso village.

From 1224 its structure was radically changed, at which time the castle was donated by Sancho II (1223-1248) to the Knights Hospitaller (called, from the sixteenth century, as the Order of Malta), after a period relatively long war with the kingdom of Leon. At the time, this castle was made up on the pinnacle of local government, residence of the royal representative in charge of the collection of real rights in land and Miranda Penas Roias.

The ancient town and Ylgoso municipality, Ulgoso or San Sebastian Algoso, was part of the bishopric and the district of Bragança. In the year 1230 the Temple and of St John’s Hospital made bankruptcy, in Coimbra, on several land dispute between which entered Ylgoso (Algoso), Vila Cha, Atenor, Roias Feathers, Paradela and others outside the bishopric Bragança.

On June 22, 1239 Pedro Costem, Temple Order of the Commander in Portugal, made a composition between the commendations of Mogadouro and Penas Roias, belonging to their order, and the Algoso.

Subsequently, February 13, 1291, Denis (1279-1325) made new composition, in Coimbra, with the Grand Commander of St. John’s Hospital, D. Fernão Peres (the Mossejo) concerning the commendation Algoso and other nearby lands. The importance of this stemmed income earned annually by its Commander: seven to eight thousand crusaders, including in this amount the two parts of the fruit of the abbeys of Travanca, Sendim, Dry Vilar, Two Churches and Guide. This sovereign will still determined to carry out repair work and improvements in the castle.

Held by the Hospitallers, the castle was transformed into a fortress in the Gothic style, especially the construction of the solid keep, with the commander of the residence function, and a turret on the wall to the south.