You might not need Metadata

When going for Text Mining a collection of resources pretty much the first step is to check for the language. If it is a multi-language collection may be there is at least some metadata. At SUB we often use the METS /MODS standard based on XML.

Within MODS there is a mods:languageTerm that can be checked against an authority file for its named languages (or language codes). In this case it is the iso639-2b (LoC ). Unfortunately the schema is not applied very strict as we can see in the following excerpt.

Well, iso639-2b lists three-letter codes only. pt is an iso639-1 alpha-2 code for Portuguese so the document should be added to por which is the correct notation. At least one can guess the correct language, by cross-checking the alpha-2 code. But they also mixed between iso639-2b and iso639-2t (see fre and fra) what makes another cross-check necessary. For others like de e we have to guess that this means something like German, while the blank field is just unusable as long as we are not going for automatic language detection.

There is quite a good one in the even longer complete list of language codes of this special collection. It is bel , the code for Belarusian. It is unlikely that a resources published in Belgium is in Belarusian.

Six flaws are still in there:

Besides these mistakes, it seems that the most part is specified very well. We just have to correct about 10% of the data which can be done via script. But the question remains: Why there is no validation of these files?

You can look for these values by searching the index . The rather strange values are not in the index but still in the METS files, e.g. here .

Thanks to Michelle, who provided the title for this post.