In the listings for castle inhabitants, there are two types of magic-users: wizard (11th level) and necromancer (10th level). A wizard will be attended by dragons, balrogs, wyverns or basilisks; a necromancer by chimaerae, manticores, lycanthropes or gargoyles. Unlike the griffins and rocs of the fighting-men, the magic-users' retainers never have heroic mounts. In their castles, the magic-users will use Geas to send the player characters on some quest after treasure, claiming half and preferring the magic items. This is an obvious and easy way to send player characters on a perilous quest of the referee's choosing, and to make sure that the treasure thus gained does not enrich the PCs too heavily. The alternative of giving up a magic item as a toll is a good way to strip out any excess items from the party. All told, a very convenient and simple encounter type. So let's see what lies under the surface for our setting.
It's hardly a coincidence that the wizards who live in these castles are all able to cast 5th level spells. A necromancer has access to Animate Dead, so they can be true "necromancers" in the classic sense. A 10th level magic-user casting the spell gets 2d6 skeletons or zombies at a time; one can imagine that a necromancer's tower would commonly be stocked with zombie servants who need no food and fight without reservation.
The fifth-level of magic-user spells in OD&D is practically built for mass combat. This is the level of Cloudkill and Wall of Stone, of Transmute Rock to Mud and Pass-Wall and Conjure Elemental. It's also the level where Feeblemind comes in, the ultimate defense against magic-users attacking in such a situation. Someone laying siege to a wizard's tower could well find themselves facing a zombie army backed up by dragons or balrogs, and elementals to boot, not to mention a magic-user with three or four fireballs to throw.
There are all kinds of mini-settings here. Consider a wizard with basilisks; the basilisk-handler is reliant on his master's access to the Stone to Flesh spell, and it may be best to recruit a blind man for the task. The wizard's tower will be surrounded by a statue garden the likes of which would suggest a medusa rather than a spellcaster lives in the tower. A necromancer who keeps gargoyles may have them blend in as if they were architectural elements, and they will prove formidable foes indeed since only magical attacks effect them. The tower will seem almost defenseless until it springs to life. If it has lycanthropes, the mild-mannered residents will appear to have nothing unusual about them, until they show their nature - perhaps a Lawful magic-user will be served by ferocious werebears. Some spells also imply interesting settings; consider the wizard who casts Hallucinatory Terrain on the lands around his tower.
A proper magic-user's tower will also have a rich library; in pre-printing times that could number in the low dozens of books, including both spellbooks and tomes of magical knowledge. They're also like to have magic items, preferring the miscellaneous ones; so a wizard in his tower may well have a Crystal Ball or Medallion of ESP and be able to see his foes coming well in advance. Rare items that could be of tremendous use in defense include the Drums of Panic, scattering an invading army at a stroke.
So a wizard's tower should be a rich environment for the PCs to come upon. And after the wizard dies, there's always the possibility of what dungeons lie beneath it.