From 1466 to 1639 the castle was owned by the Pallandt family. Because of the strategic location of the castle as a vantage point to Maastricht, the castle was plagued by siege from the mid 1500s onwards in wars like the 80-year war and the Dutch war.
On October 25th in 1520, Emperor Charles V stayed the night at the castle during his journey to Aachen to be crowned king.
In 1568 the Spanish-Liègeois army takes hold of the castle. After this, William of Orange, a good friend of Floris II of Pallandt, recaptures the castle and stays for about three weeks, before heading to Maastricht.