Herodotus

(It's a work in progress.)

Herodotus is a project for tracking and linking analytic and synthetic facts about (the releases of) a package. The tracking is to be done independently per package. (Sisyphus is an example of a repository of packages where this can be applied.)

(Herodotus is inspired by, and partly based in the implementation, on herodotos tool. Note the different spelling of the name of this tool and of our project. Named after Herodotus.)

Which computed or external meta-information for a package is tracked

• Analytic facts (computed from the "internal" content of package releases):

• Static analysis of the C/C++ code (warnings):

• by coccinelle
• by cppcheck
• ...

• Discovery of source files which are not used during the build of the package (by means of strace or by the access time)
• ...

• Synthetic facts (added "externally" by maintainers)

• Resolutions for the warnings from the static analysis (a reason why they are invalid or a fix).
• ...

Representation

• Each fact is linked to the corresponding Git (Gear) commit or tag.
• (The facts can be stored in the same Git repository in a separate branch.)
• If the "same" fact appears for several releases, all its occurrences are linked together, so that a maintainer can view them as a single fact. Only when the facts change between releases, it should need attention.

User interfaces

• Files (obtained via Git), org-mode editor (Emacs; org-mode is like a personal wiki)
• ...

Implementation details

The core: herodotos tool

herodotos tool runs the analyzers for different releases and then links identical facts (modulo the diff, i.e., the changes of the source code).

Description of herodotos tool

• http://coccinelle.lip6.fr/papers/aosd10.pdf
• https://kernelnewbies.org/NicolasPalix (its developer; this page has a short description of the tool)

herodotos in ALT repos

herodotos in p8

•  p8, test-only task #243195 herodotos

requires:

•  p8, task #243203 ocaml-bolt (Upstream: downloads; darcs get http://bolt.x9c.fr/)
•  p8, task #243208 ocaml-parmap
•  p8 (works, but not quite ready) ocaml-postgresql

needed mainly for reproducing the author's experiments with Linux sources as a way of testing herodotos:

•  p8, task #243245 coccinelle with support for embedded Python

needed optionally for better correlation:

•  p8 gumtree

requires:

•  p8, task #243201 cgum

herodotos in p9

•  p9 herodotos

requires:

•  p9, test-only task #243204 ocaml-bolt (Upstream: downloads; darcs get http://bolt.x9c.fr/)
•  p9 ocaml-parmap
•  p9 (another version, ok?) ocaml-postgresql

needed mainly for reproducing the author's experiments with Linux sources as a way of testing herodotos:

•  p9 coccinelle with support for embedded Python

needed optionally for better correlation:

•  p9 gumtree

requires:

•  p9 cgum

herodotos in Sisyphus

•  sisyphus herodotos

requires:

•  sisyphus, test-only task #243205 ocaml-bolt (Upstream: downloads; darcs get http://bolt.x9c.fr/)
•  sisyphus ocaml-parmap
•  sisyphus (another version, ok?) ocaml-postgresql

needed mainly for reproducing the author's experiments with Linux sources as a way of testing herodotos:

•  sisyphus, test-only task #243259 coccinelle with support for embedded Python

needed optionally for better correlation:

•  sisyphus gumtree

requires:

•  sisyphus cgum

How to try herodotos

One can install herodotos (for p8) from task 214330.

If you want to try herodotos, try to reproduce the authors' work https://github.com/coccinelle/faults-in-linux . (It is more recent; the older work http://coccinelle.lip6.fr/papers/aosd10.pdf with their data and configuration is not suitable for the current herodotos 0.8+ version.)

I've adapted their herodotos config files and made it a Gear repo: http://git.altlinux.org/people/imz/public/faults-in-Linux.git , so that one can easily pass it to hasher and do the processing in an isolated, easily reproducible hasher environment.

• First, prepare: clone my repo and and set up the sources for APT:

$ git clone --depth=20 git://git.altlinux.org/people/imz/public/faults-in-Linux.git
$ cd faults-in-Linux
$ apt-repo  --hsh-apt-config=/home/imz/.hasher/p8/apt.conf add 214330

Here is what the APT sources config for the hasher should be like (and our current working dir):

$ apt-repo  --hsh-apt-config=/home/imz/.hasher/p8/apt.conf
rpm [updates] file:/ALT/p8 x86_64 classic
rpm [updates] file:/ALT/p8 noarch classic
rpm http://git.altlinux.org repo/214330/x86_64 task
$ pwd
/space/home/imz/wip/2018-10-herodotos-cppcheck/faults-in-Linux

• Then, we execute the authors' processing rules (under the control of my .gear/faults-in-Linux.spec-file from the master branch; it automatically gets and checks out various revisions of the linux sources (so, you must have enough space to hold it):

$ export share_network=1
$ gear-hsh  --apt-config=/home/imz/.hasher/p8/apt.conf --without-stuff 2>&1 | tee hsh.log.1

It stops after the step of applying the static analyzer (coccinelle) to each version of the sources (linux). The results are saved at /usr/src/HERODOTOS/ (inside hasher). I've copied them and saved in commit ad458b0c2 in the EXPERI/imz2/apply-analyzer-results branch, so that you can look and get an idea what they look like:

• the individual per-version *.orig.org files.

/usr/src/HERODOTOS/ is used as the place to cache the analyzed sources and to save the (intermediate and final) results, so it won't be cleaned if you run gear --hasher | hsh-rebuild again (after editing the Git repo with the Makefiles, configs etc). (TODO: Unfortunately, the automatically filled faults/.projects_study.hc file is not relocatable in a similar manner.)

• The next step (correlation of the warnings between versions by herodotos) is to be run by us manually (because I wanted to have a possibility to first commit the results of the previous step):

hsh-shell --mount=/proc,/dev/pts
cd /usr/src/RPM/BUILD/faults-in-Linux-20181023/faults/
make correl

or as a single command:

hsh-run --mount=/proc -- sh -c 'cd /usr/src/RPM/BUILD/faults-in-Linux-20181023/faults/ && make correl'

I saved the results in commit c3f5e56dd7e in the EXPERI/imz2/correl-gnudiff-results branch, so that you can look and get an idea what they look like:

• some non-empty *.correl.org files with undecided possible correlations (marked as TODO);
• the *.new.org files with merged warnings from all versions. It is to be decided whether each of them (marked as TODO initially) is a real error or a false warning.

(In this example, I made herodotos use the --diff gnudiff option, because the default better --diff hybrid requires gumtree and doesn't work correctly if it is absent.)

A follow-up scenario would be to first mark some warnings as checked and then add another version of the project into consideration (by editing the pattern in faults/study.hc.base) and see how the warnings concerning the new version are merged with the marks for the old versions. Let's explore this.

coccinelle support

coccinelle is natively supported by herodotos tool.

Actually, herodotos tool can work with any analyzer which gives output in the org-mode format.

coccinelle in Sisyphus

•  coccinelle

cppcheck support

• cppcheck is supported by flycheck (an Emacs package)
• flycheck can be hacked to output the information in the format suitable for herodotos tool (org-mode)

So, we could easily get the support for any analyzer known to flycheck.

cppcheck in Sisyphus

•  cppcheck
•  emacs-mode-flycheck
•  flycheck output in org-mode format

Discovery of source files which are not used during the build of the package

Either builds under strace can be used to discover files which are not used, or the access time (an idea by boyarsh@, which has already been probably implemented by him).

Extensions to be implemented

Ad hoc sources for herodotos

Ad hoc ways to feed herodotos some specific sources (which are not covered by the configuration "*SCM" and "versions" parameters):

 herodotos preinit-add git REPO TAG
 herodotos preinit-add rpm-bp FILE
 herodotos preinit-add srpm FILE

More; easy to implement; but not really needed much (as for now):

 herodotos preinit-add rpm-bp+gear REPO TAG
 herodotos preinit-add srpm+gear REPO TAG

Here, the way the git option is processed is similar to how the git: sources from the configuration are treated. (An exercise in implementing preinit-add on the base of the existing code.)

The rpm-bp option would invoke rpm to prepare the source tree (with all the patches applied etc. by performing the prep stage with rpmbuild -bp, optionally under hasher); the srpm option is about a stupid unpacking of an .src.rpm and of the archives it contains. The +gear options are about getting the srpm from a Gear repo.

More methods for herodotos to get sources

In the spirit of the current way to write the configuration file, in addition to git: (combined with versions to select the tags), one could implement more methods for herodotos to get sources from some other kinds of repositories:

•  rpm-bp+gear: (or srpm+gear:)
•  the Sisyphus (and branches) archive (whereby the repo index might help to learn the releases and their place in the archive).

This could be useful for a more automated study of packages from Sisyphus and branches.

Usage

(to be written)