checkdata : High-level integrity tests

The lino.modlib.checkdata plugin adds support for defining application-level data integrity tests. For the application developer it provides a method to define data checkers for their models. For the site maintainer it adds the checkdata django-admin command. For the end user it adds a set of automatic actions.

This is a tested document. The following instructions are used for initialization:

>>> from lino import startup
>>> startup('lino_book.projects.roger.settings.demo')
>>> from lino.api.doctest import *
>>> from django.core.management import call_command
>>> from atelier.sheller import Sheller
>>> shell = Sheller(settings.SITE.project_dir.parent)

Overview

This plugin provides a framework for managing and handling data problems.

data problem

A problem in the database that cannot be detected by the DBMS because finding it requires business intelligence.

Some data problems can be fixed automatically, others need human interaction.

The application developer defines the rules for detecting data problems by writing data checkers.

data checker

A piece of code that tests for data problems.

unbound data checker

Data checkers are usually attached to a given database model. If they are not attached to a model, they are called unbound. Data problems reported by an unbound data checker have an empty owner field.

Lino has different ways to run these checkers. When a data checker finds a problem, Lino creates a problem message.

problem message

A message that describes one or several data problems detected in a given database object.

Problem messages are themselves database objects, but considered temporary data and may be updated automatically without user confirmation.

Each problem message is assigned to a responsible user. This user can see their problems in the MyProblems table, which generates a welcome message if it contains data.

The application developer can decide to add a ProblemsByOwner table to the detail layout of any model. This enables the end users to focus on the data problems related to a given database object.

Data checkers

In the web interface you can select Explorer ‣ System ‣ Data checkers to see a table of all available checkers.

>>> rt.show(checkdata.Checkers, language="en")
... 
=================================== ========================================================
 value                               text
----------------------------------- --------------------------------------------------------
 beid.SSINChecker                    Check for invalid SSINs
 cal.ConflictingEventsChecker        Check for conflicting calendar entries
 cal.EventGuestChecker               Entries without participants
 cal.LongEntryChecker                Too long-lasting calendar entries
 cal.ObsoleteEventTypeChecker        Obsolete generated calendar entries
 countries.PlaceChecker              Check data of geographical places.
 courses.MemberChecker               Check membership payments
 finan.FinancialVoucherItemChecker   Check for invalid account/partner combination
 ledger.VoucherChecker               Check integrity of ledger vouchers
 memo.PreviewableChecker             Check for previewables needing update
 phones.ContactDetailsOwnerChecker   Check for mismatches between contact details and owner
 printing.CachedPrintableChecker     Check for missing target files
 sepa.BankAccountChecker             Check for partner mismatches in bank accounts
 system.BleachChecker                Find unbleached html content
 uploads.UploadChecker               Check metadata of upload files
 uploads.UploadsFolderChecker        Find orphaned files in uploads folder
 vat.VatColumnsChecker               Check VAT columns configuration.
=================================== ========================================================

The lino_xl.lib.countries.PlaceChecker class is a simple example of how to write a data checker:

from lino.api import _
from lino.modlib.checkdata.choicelists import Checker

class PlaceChecker(Checker):
    model = 'countries.Place'
    verbose_name = _("Check data of geographical places.")

    def get_checkdata_problems(self, obj, fix=False):
        if obj.name.isdigit():
            yield (False, _("Name contains only digits."))

PlaceChecker.activate()

More examples of data checkers we recommend to explore:

Showing all data problems

In the web interface you can select Explorer ‣ System ‣ Data problems to see all problems.

The demo database deliberately contains some data problems.

>>> rt.show(checkdata.AllProblems, language="en")
... 
============= =========================================== ============================================================== ========================================
 Responsible   Database object                             Message                                                        Checker
------------- ------------------------------------------- -------------------------------------------------------------- ----------------------------------------
 Robin Rood                                                File uploads/2015/05/foo.pdf has no upload entry.              Find orphaned files in uploads folder
 Robin Rood    *Recurring event #4 Assumption of Mary*     Event conflicts with Activity #1 001  1.                       Check for conflicting calendar entries
 Robin Rood    *Recurring event #11 Ascension of Jesus*    Event conflicts with Mittagessen (14.05.2015 11:10).           Check for conflicting calendar entries
 Robin Rood    *Recurring event #12 Pentecost*             Event conflicts with 4 other events.                           Check for conflicting calendar entries
 Rolf Rompen   *Mittagessen (14.05.2015 11:10)*            Event conflicts with Recurring event #11 Ascension of Jesus.   Check for conflicting calendar entries
 Robin Rood    *First meeting (25.05.2015 13:30)*          Event conflicts with Recurring event #12 Pentecost.            Check for conflicting calendar entries
 Robin Rood    *Absent for private reasons (25.05.2015)*   Event conflicts with Recurring event #12 Pentecost.            Check for conflicting calendar entries
 Robin Rood    *Karl Kaivers (ME)*                         Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Laura Laschet (ME)*                        Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Josefine Leffin (MEL)*                     Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Marie-Louise Meier (ME)*                   Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Alfons Radermacher (ME)*                   Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Christian Radermacher (MEL)*               Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Edgard Radermacher (ME)*                   Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Guido Radermacher (ME)*                    Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Hedi Radermacher (ME)*                     Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Jean Radermacher (ME)*                     Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Erna Ärgerlich (ME)*                       Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Jean Dupont (ME)*                          Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Marie-Louise Vandenmeulenbos (MEC)*        Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Bernd Brecht (ME)*                         Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Jérôme Jeanémart (ME)*                     Member until 2015-12-31, but no payment.                       Check membership payments
 Robin Rood    *Source document PRC_29_2015.pdf*           Upload entry uploads/2015/05/PRC_29_2015.pdf has no file       Check metadata of upload files
============= =========================================== ============================================================== ========================================

Filtering data problem messages

The user can set the table parameters e.g. to see only messages of a given type ("checker"). The following snippet simulates the situation of selecting the ConflictingEventsChecker.

>>> chk = checkdata.Checkers.get_by_value('cal.ConflictingEventsChecker')
>>> rt.show(checkdata.ProblemsByChecker, chk)
... 
============= =========================================== ==============================================================
 Responsible   Database object                             Message
------------- ------------------------------------------- --------------------------------------------------------------
 Robin Rood    *Recurring event #4 Assumption of Mary*     Event conflicts with Activity #1 001  1.
 Robin Rood    *Recurring event #11 Ascension of Jesus*    Event conflicts with Mittagessen (14.05.2015 11:10).
 Robin Rood    *Recurring event #12 Pentecost*             Event conflicts with 4 other events.
 Rolf Rompen   *Mittagessen (14.05.2015 11:10)*            Event conflicts with Recurring event #11 Ascension of Jesus.
 Robin Rood    *First meeting (25.05.2015 13:30)*          Event conflicts with Recurring event #12 Pentecost.
 Robin Rood    *Absent for private reasons (25.05.2015)*   Event conflicts with Recurring event #12 Pentecost.
============= =========================================== ==============================================================

See also cal : Calendar functionality and Defining holidays.

Running the checkdata command

>>> call_command('checkdata')
Found 6 and fixed 0 data problems in Calendar entries.
Found 15 and fixed 0 data problems in Participants.
Found 1 and fixed 0 data problems in Upload files.
Found 1 and fixed 0 data problems in unbound data.
Done 33 checks, found 23 and fixed 0 problems.

You can see the list of all available checkers also from the command line using:

$ python manage.py checkdata --list
>>> call_command('checkdata', list=True)
... 
=================================== ========================================================
 value                               text
----------------------------------- --------------------------------------------------------
 beid.SSINChecker                    Check for invalid SSINs
 cal.ConflictingEventsChecker        Check for conflicting calendar entries
 cal.EventGuestChecker               Entries without participants
 cal.LongEntryChecker                Too long-lasting calendar entries
 cal.ObsoleteEventTypeChecker        Obsolete generated calendar entries
 countries.PlaceChecker              Check data of geographical places.
 courses.MemberChecker               Check membership payments
 finan.FinancialVoucherItemChecker   Check for invalid account/partner combination
 ledger.VoucherChecker               Check integrity of ledger vouchers
 memo.PreviewableChecker             Check for previewables needing update
 phones.ContactDetailsOwnerChecker   Check for mismatches between contact details and owner
 printing.CachedPrintableChecker     Check for missing target files
 sepa.BankAccountChecker             Check for partner mismatches in bank accounts
 system.BleachChecker                Find unbleached html content
 uploads.UploadChecker               Check metadata of upload files
 uploads.UploadsFolderChecker        Find orphaned files in uploads folder
 vat.VatColumnsChecker               Check VAT columns configuration.
=================================== ========================================================
>>> call_command('checkdata', 'cal.')
Found 6 and fixed 0 data problems in Calendar entries.
Done 1 check, found 6 and fixed 0 problems.
>>> call_command('checkdata', 'foo')
Traceback (most recent call last):
...
Exception: No checker matches ('foo',)

The --prune option instructs checkdata to remove all existing error messages before running the tests. This makes the operation quicker on sites with many existing data problem messages. Don't use this in combination with a filter because --prune removes all messages, not only those that you ask to rebuild.

>>> shell("python manage.py checkdata --prune")
Prune 23 existing messages...
Found 6 and fixed 0 data problems in Calendar entries.
Found 15 and fixed 0 data problems in Participants.
Found 1 and fixed 0 data problems in Upload files.
Found 1 and fixed 0 data problems in unbound data.
Done 33 checks, found 23 and fixed 0 problems.

NB the above example uses atelier.sheller instead of call_command. Both methods are functionally equivalent.

Language of checkdata messages

Every detected checkdata problem is stored in the database in the language of the responsible user. A possible pitfall with this is the following example.

The checkdata message "Similar clients" appeared in English and not in the language of the responsible user. That was because the checker did this:

msg = _("Similar clients: {clients}").format(
    clients=', '.join([str(i) for i in lst]))
yield (False, msg)

The correct way is like this:

msg = format_lazy(_("Similar clients: {clients}"),
    clients=', '.join([str(i) for i in lst]))
yield (False, msg)

See Internationalization for details.

Database models

class lino.modlib.checkdata.Problem

Django model used to store a problem message.

checker

The Checker that reported this problem.

message

The message text. This is a concatenation of all messages that were yielded by the checker.

user

The user responsible for fixing this problem.

This field is being filled by the get_responsible_user method of the checker.

class lino.modlib.checkdata.Problems

The base table for problem messages.

class lino.modlib.checkdata.MyProblems

Shows the problem messages assigned to me.

class lino.modlib.checkdata.Checkers

The list of data checkers known by this application.

This was the first use case of a ChoiceList with a detail_layout.

class lino.modlib.checkdata.Checker

Base class for all data checkers.

model

The model to be checked. If this is a string, Lino will resolve it at startup.

If this is an abstract model, get_checkable_models() will yield all models that inherit from it.

If this is None, the checker is unbound, i.e. the problem messages will not be bound to a particular database object. This is used to detect missing database objects. For example vat.VatColumnsChecker is unbound.

Instead of setting model you might want to define your own get_checkable_models() method. For example, ledger.VoucherChecker does this because it wants to get all MTI children (not only top-level models). See Lino core utilities for more explanations.

classmethod check_instance(cls, *args, **kwargs)

Run get_checkdata_problems() on this checker for the given database object.

get_checkable_models(self)

Return a list of the models to check.

The default implementation returns all top-level models that inherit from model (or [None] when model is None).

classmethod activate(cls)

Creates an instance of this class and adds it as a choice to the Checkers choicelist.

The application developer must call this on their subclass in order to "register" or "activate" the checker.

update_problems(self, obj=None, delete=True, fix=False)

Update the problem messages of this checker for the specified object.

obj is None on unbound checkers.

When delete is False, the caller is responsible for deleting any existing objects.

get_checkdata_problems(self, obj, fix=False)

Return or yield a series of (fixable, message) tuples, each describing a data problem. fixable is a boolean saying whether this problem can be automatically fixed. And if fix is True, this method is also responsible for fixing it.

get_responsible_user(self, obj)

The site user to be considered responsible for problems detected by this checker on the given database object obj. This will be stored in user.

The given obj is an instance of model, unless for unbound checkers (i.e. whose model is None).

The default implementation returns the main checkdata responsible defined for this site (see responsible_user).

Automatic actions

This plugin automatically installs two actions on every model for which there is at least one active data checker:

class lino.core.model.Model
fix_problems

Update data problem messages and repair those which are automatically fixable.

Implementing class: lino.modlib.checkdata.FixProblemsByController.

check_data

Update data problem messages for this database object, also removing messages that no longer exist. This action does not change anything else in the database.

Implementing class: lino.modlib.checkdata.UpdateProblemsByController.

Internal utilities

lino.modlib.checkdata.get_checkable_models(*args)

Return an OrderedDict mapping each model which has at least one checker to a list of these checkers.

The dict is ordered to avoid that checkers run in a random order.