It isn’t hard to see why they sited the castle on this rock. The site is a textbook case of what the Crusaders had learned about engineering. The castle is almost impossible to approach from any angle with land-based siege machines. It must have been the devil to build, too.
The castle began in the early period, and its main structure went up in the 13th century, but many of its buildings were added later. Here is a Time Team (British reality archeology) effort to discover some of its lost earliest walls.
The castle’s approach was a road that wound around and doubled back, allowing it to be exposed to defensive fire from the top. At one point, a gatehouse served as a checkpoint.
The original keep and main buildings were right on the sea cliff, with the bailey walls circling toward the town. Most of the buildings added in Tudor times were placed inside this yard. The oldest parts of the castle are clearly the ones facing the sea. Here is some aerial video that shows the road and gatehouse.
The castle was so constantly updated because it never became irrelevant to England’s defense. Every age needed it to be a working fort. Here are drawings for the 17th century upgrades. Even in World War 2, it had additions to bring it up to date. It’s a major tourist site now.