Two recent papers on the difficulty of doing research in education:

Educational Research: The Hardest Science of All, by David Berliner

and

Classroom Research and Cargo Cults, by Ed Hirsch

As both point out, there are so many variables in any classroom situation that designing and carrying out any sort of experiment to determine the effectiveness of a particular method or policy is bound to produce at best inconclusive results. The second paper puts it this way:

The statistical methods of educational research have become highly sophisticated. But the quality of the statistical analysis is much higher than its practical utility. Despite the high claims being made for statistical techniques like regression analysis, or experimental techniques like random assignment of students into experimental and control groups, classroom-based research (as contrasted with laboratory research) has not been able to rid itself of uncontrolled influences called "noise" that have made it impossible to tease out the relative contributions of the various factors that have led to "statistically significant" results. This is a chief reason for the unreliability and fruitlessness of current classroom research. An uncertainty principle subsists at its heart. As a consequence, every partisan in the education wars is able to utter the words "research has shown" in support of almost any position. Thus "research" is invoked as a rhetorical weapon - its main current use.

The papers differ in their recommendations. The first advocates not applying the same scientific standards to educational research as we apply to what it calls the "easy-to-do" subjects (what the rest of the world calls the hard sciences). Instead, it recommends more "argument, discourse and discussion".

The second paper concludes that a more useful approach is to refocus attention on laboratory research rather than classroom research:

On a positive note, there already exists some reliable research on which educational policy could and should be based, found mainly (though not exclusively) in cognitive psychology.

In the section entitled "What are 'reliable general principles'?", the author presents a detailed description of those cognitive principles he considers reliably established:

* prior knowledge as a prerequisite to effective learning

* meaningfulness

* the right mix of generalization and example

* attention determines learning

* rehearsal (repetition) is usually necessary for retention

* automaticity (through rehearsal) is essential to higher skills

* implicit instruction of beginners is usually less effective

I tend to be more sympathetic to the second position. For what I'm trying to do (design online educational materials for mathematics), the results from cognitive science appear to be more useful and applicable than those I've seen from education research, which tend to be inconsistent or designed to prove some theory or other. I'd like it to be otherwise, but as math education is currently a very politicized ballpark to play in, I'll quietly take whatever useful ideas I can find from whichever field provides them and ignore the rest. Particularly the eternal and unproductive "argument, discourse and discussion".

... just not doing much blogging recently. What I am doing is coding a new Flash math learning widget, which tends to absorb my focus and spare time almost completely. My days have stretched out (unintentionally) to 4 am bedtimes - it's hard to react to or even notice the time when you're chasing down a bug in a piece of code that "obviously" should run correctly. It's also hard to stop thinking about that bug once your head hits the pillow, so sleep doesn't come quickly even then. Sigh!

Nothing new to experienced/professional coders, of course, nor the tendency to eat improperly while in such a state of absorption. I guess even amateurs like me aren't immune. Gotta stop this, and will, just as soon as I get the next piece working ...

In the beginning (i.e. sometime less than a decade ago), creating material to put on the web was not an easy task. Producing even a simple text-only webpage required a more than passing knowledge of the guts of HTML; creating and adding graphics or (gasp!) animation required both knowledge and perseverance - extreme perseverance in many cases. Nevertheless, many did persevere, particularly educators hooked on the promise of new ways of communicating their content:

* hyperlinking - information organized non-linearly, navigational choice

* visualization - visual language, graphics, animation, video

* (the biggie) interactivity - students actively manipulating and exploring content rather than passively observing it.

Despite the limitations of the early technology and lack of institutional support, many of the initial educational prototypes were both inspired and inspiring, eventually to the extent that others began to notice their value, and more importantly, notice the problem: scarcity. Distribution of these early works was not a problem; the increasing ubiquity of the web took care of that. But the difficult early technologies meant that, despite the obvious demand for them, few of these "learning objects" were actually being produced.

The to-be-expected response to a scarce but valuable quantity arose almost simultaneously from multiple sources: collect and conserve. The fun started as the various groups sought to define what it was they were collecting and conserving.

The first efforts, lead by the managerially-inclined, were to house learning objects in learning object repositories (LORs), all nicely tagged with suitable descriptive information to enable searching (ordinary search engines being deemed insufficient). A learning object became a chunk of web-based content with metadata attached. But what sort of metadata? There were too few learning objects about as yet to abstract their unique characteristics into any sort of meaningful scheme, so perhaps they could be indexed by content? But content ontologies for subject areas as broad as originally envisioned are notoriously difficult to construct, much less implement. Metadata schemes quickly focussed on providing basic but detailed bean-counting such as author, cost of use, language, version, date of contribution, etc..

The next group to weigh in were the commercial interests - the learning management system builders. Their mantra was reusability - LOs should not merely be useable, they should be useable over and over again, repurposed as required by the course designer and the LMS. Unlike metadata, which only added something to the LO, reusability required removing something: the author/developer was expected to delete from his object any contexts that might inhibit its reuse elsewhere, or any connections with other objects that might prevent it being resequenced. Another worm-can opened, at least for academic subjects: contexts and connections are often a very important and enriching component of these subjects, sometimes more important than content. Removing them risks reducing LOs to meaningless "content bites" and fragments learning.

Thus far, neither group addressed what was still the basic difficulty with LOs: scarcity. This was left to the academics, who, in papers and at conferences, redefined LOs to be essentially anything on the web that could be used for learning. Never mind that that included almost everything on the web; an academic's typical response to any idea is to generalize it into meaninglessness, and this was no exception. There were a few attempts to rein the sucker in, but LO toy concepts such as "web-mediated learning experiences" as LOs (true!) were too much intellectual fun to resist.

Now, of course, almost anything could be put on the web and counted as a LO - old lecture notes, research papers, whatever. And counting is key in academia, as long as what you're counting is peer reviewed. Noting (correctly) that academics producing LOs receive very little credit for their efforts, one well-known repository instituted a peer review system for LOs, ostensibly to maintain quality, but also (it stated) for use in its contributors' individual tenure and promotion portfolios. (Fewer than ten percent of the repository's LOs are actually reviewed and the reviewing standards are essentially ad hoc.) From a quick glance at its contents, it appears that almost anyone can and does contribute almost anything to the repository, but little of what is contributed actually uses the new medium to any extent instead of porting existing print to pixels.

Which brings us back to the original vision of learning objects: innovative web-based materials that actually use web capabilities such as hyperlinking, visualization or interactivity for teaching and learning. This vision has yet to be realized; thus far, very little effort has being expended in finding out how these capabilities work for learning or in producing LOs that use them. Instead, what we have (aside from a great deal of hype) is a mad dash after infrastructure; all sorts of mechanisms for storing, finding, assembling, distributing and evaluating content, but very few for designing or producing it.

It's understandable; the technological hurdles for producing innovative content are lowering only slowly. It's now much easier to produce web pages than a few years back, though enhancements like javascript, DHTML and CSS are still beyond the average page maker. Apple's "digital hub" initiative makes multimedia creation accessible to novices; the animation tools of Macromedia Flash are also very accessible, though Flash actionscripting is not. These examples aside, for the most part, we have very few web authoring tools useable by those who need them: nontechnical academics with an innovative vision of web-based learning to implement.

In retrospect, I think we're targeting the wrong goal when it comes to learning objects - we've been trying to manage them before first ensuring that we do indeed have enough of something real and valuable to manage. It won't do to ignore the need for a clear definition of LOs or the need of developing means for their creation - repositories abhor a vacuum as much as nature does, and trivia and ported print will continue to fill the empty space. At some point, we will have to promote and support the creation and design of learning objects, not just by dangling the carrot of peer review before prospective authors, but by encouraging the development of simple, easy to use authoring tools and by developing rubrics for their use - by finding out just when and how intrinsic web capabilities like visualization and interactivity promote learning, for example.

Conceivably, at some point in the future, creating learning objects could be as easy as jotting down a page of lecture notes, so easy that building elaborate repositories to conserve such objects would be as pointless as hoarding scrap paper. Given the current level of hype and megabuck development of repository schemes, I'm not holding my breath.