The paper examines the task of Word Sense Disambiguation (WSD) criticallyand compares it with Part of Speech (POS) tagging, arguing that the abilityof a writer to create new senses distinguishes the tasks and makes it moreproblematic to test WSD by the mark-up-and-model paradigm, because newsenses cannot be marked up against dictionaries. This serves to set WSDapart and puts limits on its effectiveness as an independent NLP task.Moreover, it is argued that current WSD methods based on very small wordsamples are also potentially misleading because they may or may not scaleup. Since all-word WSD methods are now available and are producing figurescomparable to the smaller scale tasks, it is argued that we shouldconcentrate on the former and find ways of bootstrapping test materialsfor such tests in the future.