Precisely when the building fell in and was abandoned is not known, but it was already in ruins by 1815, most probably as a result of the terrible earthquake on 5^th December 1810, which levelled two thirds of the town. It seems to have been destroyed beyond repair in the 1856 earthquake.

After the population exchange, the National Bank sold Muslim buildings as "exchangeable estate" to private individuals.

In 1954 the Directorate of Antiquities in the Ministry of Education approved plans to demolish the building and convert the site into a square. For some unknown reason the monument was saved from demolition.

From the 1950s to 1970 the entire west side and part of the southwest flank of the block once occupied by the palace was rebuilt, thus altering one third of the building beyond recognition.

The remaining section comprises old buildings dating back to various different periods in the town's history. The four vaulted ground floor premises on the south side, on Eleftheriou Venizelou Square, are the only parts of the Ducal Palace to have survived intact.