Over the years, the companies that have produced D&D have shifted their focus on what D&D is about. In the early years, the game was more about player skill, now it is far more about character abilities and the rules themselves. Up until 3e, game designers touted the rules as more like guidelines, now the rules are the rules, fixed. To tamper with them could mean serious unbalancing in the game.
I think the focus on story, beginning in the Late Golden Age and ending around 3e, molded many gamers into believing the could only play within the confines of the rules. The focus on story took the fate of a character out of the player's hands. To compensate, players began thirsting for more abilities and rules to make the game "balanced" again, especially if these abilities somehow supported "the story." Eventually even the story gave way to the rules themselves.
It's a working thesis, which I support by offering the following timeline.
The Early Golden Age: 1973-1978
Focus on exploration, player abilities, figuring out what D&D "is."
Examples: Gygax's own Castle Greyhawk Campaign, Rythlondar.
The Middle Golden Age: 1978-1983
Still a focus on player abilities, yet the non-core classes are on the rise. Dungeon and Wilderness Exploration. A growing emphasis on "realism."The Golden Age of the Module.
Examples: The AD&D Player's Handbook, S1: The Tomb of Horrors, Against the Giants, various articles in Dragon about making the game more "real."
The Late Golden Age: 1983-1989
A growing focus on story, campaign settings. Character building, exploration, and player abilities are fine as long as it contributes to the story.
Examples: Dragonlance (both books and modules), I6: Ravenloft, TSR's novel line
The Early Dark Age 1989-1997
Campaign settings and the focus on story dominate D&D (and the whole industry it seems). Some focus on character building. Many modules are written so that your character is just along for the ride.
Examples: "Boxed Text" in modules, Campaign settings--especially The Forgotten Realms, Greyhawk put on the back burner, the explosion of TSR's novel line, "splatbooks"--especially Skills and Powers. A non-D&D example: White Wolf's World of Darkness.
The Middle Dark Age: 1997-2008
CHARACTER BUILDING, the shrinkage of boxed text, the rise of stat-blocks, the boom and glut of products--more splatbooks than ever (featuring skills, feats, abilities), "back to the dungeon," standardization of rules--everything, constant revisions for balance, prestige classes, miniatures.
Examples: d20 mechanic, the OGL, The Ranger Class, The Epic Level Handbook, Return to the Temple of Elemental Evil, Kyuss's 3-page stat-block in Dragon #135, and D&D 3.5e--of course! Some focus on story with "Adventure Paths."
The Current Dark Age: 2008-to the Present
Character Building (more like "player options" in 4e, character building in Pathfinder, tactics, World of Warcraft emulation, no emphasis at all on player skill, BALANCE, Edition Wars and the Old School Renaissance
Examples: The Player's Handbook 1, 2, and 3, D&D Essentials, D&D Redbox, Pathfinder, D&D Encounters.