Data integrity checks

The quality of data is very important to us. Although ultimately the data responsibility lies with editors and project managers we take great care to avoid entering of inconsistent data on a technical level, e.g. with the user interface it is not possible to enter begin dates which are later than end dates.

Nevertheless mistakes can happen, not only on the application level but also e.g. when importing data from other projects or deleting files outside of the application. Because data integrity is important for the quality of research we implemented functions to check possible inconsistencies which are described in detail below.

Orphans

Orphans

In this tab entries like dates which are not linked are shown. They could be artifacts from imports or bugs and can be deleted. If they seem to appear regularly again (without imports or known bugs) please report that issue.

Type

These types were created but have no sub types or associated data. Maybe they originate from the first install or were never used.

Missing files

Here are listed file entities which have no corresponding file, most likely because the file itself doesn’t exist anymore.

Orphaned files

Files that have no corresponding entity are listed here.

Orphaned IIIF files

IIIF files that have no corresponding entity are listed here.

Orphaned annotations

Annotations that are linked to an entity, but where the file and the entity doesn’t have a connection are listed here. There are three options to proceed:

  • Relink entity: Add a link between file and entity

  • Remove entity: Removes the entity from the annotation

  • Delete annotation: Deletes the whole annotation

Orphaned subunits

Subunits that are missing a connection to the level above. E.g. a feature without a connection to a place.

Circular dependencies

A check if an entity is linked to itself. This could happen e.g. if a person is married to herself or a type has itself as super. It shouldn’t be possible to create circular dependencies within the application. Nevertheless it’s a useful check for e.g. if data is imported from other systems.

Dates

In this view various results of invalid or inconsistent dates are shown.

Invalid dates

Invalid date combinations are shown, e.g. begin dates which are later than end dates. These issues should be fixed because otherwise the user interface won’t allow to update these entities.

Invalid involvement dates

Incompatible dates at involvement are shown. E.g. a person participated at an event longer than the event took place.

Invalid preceding dates

Incompatible dates for chained events are shown. E.g. a preceding event started after the succeeding event.

Invalid sub dates

Incompatible dates for hierarchical events are shown. E.g. a sub event began before the super event began.

Check similar names

Here you can search for similar names. Depending on selection and data volume this might take some time.

  • Classes - select the class which you want to search for similar names

  • Ratio - select how similar the names should be, 100 is the default and means absolute identical

The function uses the fuzzywuzzy package is used which in turn uses the Levenshtein Distance.