A selective tour of annotation in historical corpora begins with extra-linguistic markup: how far can it alert the corpus user to usage which is atypical of the variety being sampled? Several syntactic fossils are discussed, and a playful use of foreign and pseudo-foreign words. Are they a kind of code-switching? In the former case the answer No is given, in the latter a partial Yes. As for grammatical mark-up, with few exceptions a given scheme must privilege one particular analysis for each word, sentence or other unit of analysis. Special tags are available in the CLAWS and Penn Treebank tagsets for cases which remain ambiguous but which are in principle decidable. Grammatical mark-up remains essentially a matter of synchronic analysis, and the guiding principle is to be as specific as possible; tagsets routinely deploy a much finer set of distinctions than traditional word classes. Historical corpora like the Penn family aim also for consistency of analysis. I argue that both principles can be problematic. Consider first the push towards a unique POS tag for every word. I propose that certain kinds of word are vague as to their word class not because of a failure of analysis but because they are genuinely underdetermined. Vagueness is not ambiguity, so ambiguity tags would be inappropriate - at least with their currently intended values. Secondly, the desideratum of consistency does not allow for patterns which arguably have dual analyses synchronically, nor for items which are in transition or which have changed over the time-span of a historical corpus. Among the data discussed are the POS-tagging and parsing of adjectives derived from passive participles (interested, amused), multi-word prepositions (on behalf of), phrasal and prepositional verbs (run over), proper-to-common-noun conversions and noun-to-adjective transitions (BandAid), countable-to-mass conversions (He looked at me across a vast expanse of table) and the converse (two coffees). A brief conclusion argues that while some of the problems considered are statistically unimportant, others demand greater flexibility of mark-up.