The following are criteria that we have come up with so far for evaluating educational technology (and in particular, web-based tools). Feel free to add to as you see fit.

Cost
Navigation / usability
[Effectiveness]
Quality of content; is the content authoritative?
Ability to engage (interest)
Can students learn from it?
Aesthetics
Developmentally appropriate (or what ages / grades
Access from home?
Management
Currency
Support? (corporate, user community, knowledgebase, etc)