I know. It's alarming. Castle Grayskull is such a joke in TG's canon. It's just an old fortress that Tytus built for King Grayskull in the bios (which is similar to the current DC Canon). It's merely where the power is held, the place isn't a wonder on it's own. In the comics now, Adam doesn't need Grayskull at all since he is the Power!
But wow...IN MY CANON™, I've always though as Castle Grayskull as this indestructible place of power built by unknown hands, which was capable of housing the power of the universe and all of these secrets. Even the oldest beings on Eternia maintain that it has always existed.