Lua script for checking the health of Devuan Linux package mirrors. This is currently under development, not everything has been written yet. Some of this document mentions some of the things that are not written yet. apt-panopticon is a Lua script used by the Devuan mirror admins (maybe, if they like it) to check the health of Devuan Linux package mirrors. Originally there was bash scripts for this job, then Evilham wrote some Python scripts, now onefang has written it in Lua. We all have different tastes in languages. lol The main difference is that this Lua version tries to do everything, and will be maintained. Currently the shell scripts and Python scripts are actually being used I think. Evilham asked me to write this, after I badgered him about his Python scripts. It should also be much easier to use, the previous scripts needed some work before you could run them, this one you just download and run. The source code is at [https://sledjhamr.org/cgit/apt-panopticon/](https://sledjhamr.org/cgit/apt-panopticon/) The issue tracker is at [https://sledjhamr.org/mantisbt/project_page.php?project_id=13](https://sledjhamr.org/mantisbt/project_page.php?project_id=13) Installation. ------------- Download the source. You may want to put the apt-panopticon.lua script in someplace like `/usr/local/bin` and make sure it is executable. It should run on any recent Linux, you'll need to have the following installed - * curl * dig, part of BIND. On Debian based systems it'll be in the dnsutils package. * flock, on Debian based systems it'll be in the util-linux package. * gpgv * gzip * ionice, on Debian based systems it'll be in the util-linux package. * luajit * LuaSocket, on Debian based systems it'll be in the lua-socket package. * md5sum and sha256, on Debian based systems they'll be in the coreutils package. * rrdtool, may need to include lua-rrd. * xz, on Debian based systems it'll be in the xz-utils package. If you want to have lots of graphs, also install [https://sledjhamr.org/cgit/apt-panopticon_cgp/](https://sledjhamr.org/cgit/apt-panopticon_cgp/). Web installation. ----------------- This is a suggestion for installation on a Devuan based web server. Create - /var/www/html/apt-panopticon Install apt-panopticon and apt-panopticon_cgp there, so you end up with - /var/www/html/apt-panopticon/apt-panopticon /var/www/html/apt-panopticon/apt-panopticon_cgp The script update_apt-panopticon is an example script for updating everything, including commented out commands to update the source code. The file apt-panopticron is an example crontab file for updating everything once every ten minutes. They assume your web server user is www-data with a group of www-data, and you have a mirror user called mirrors. For mirror operators, that mirrors user would be the owner of the mirror files. You can change these to suite yourself. Once everything is updated, /var/www/html/apt-panopticon/results/Report-web.html will point to the main web page, and there will be a link at the bottom of that pointing to the detailed graphs. Using it. --------- These examples assume you are running it from the source code directory. A directory will be created called `results`, it'll be full of log files and any files that get downloaded. There will also be `results/email` and `results/web` directories, with the notification emails and web pages (once I write that bit). Note that unlike typical commands, you can't run single character options together, so this is wrong - $ ./apt-panopticon.lua -vvv Instead do this - $ ./apt-panopticon.lua -v -v -v Just run the script to do all of the tests - $ ./apt-panopticon.lua Which will print any errors. If you don't want to see errors - $ ./apt-panopticon.lua -q If you want to see warnings as well (as usual, the more `-v` options, the more details) - $ ./apt-panopticon.lua -v Or use the usual options for the help and version number (not written yet) - $ ./apt-panopticon.lua -h $ ./apt-panopticon.lua --help $ ./apt-panopticon.lua --version To run the tests on a specific mirror, for example pkgmaster.devuan.org - $ ./apt-panopticon.lua pkgmaster.devuan.org You can use the `--tests` option to tune which tests are run, for example to stop IPv6 tests, coz you don't have IPv6 - $ ./apt-panopticon.lua --tests=-IPv6 To do the same, but not run the HTTPS tests either - $ ./apt-panopticon.lua --tests=-IPv6,-https To only run the HTTP integrity tests, only on IPv6 - $ ./apt-panopticon.lua --tests=http,Integrity,IPv6 The tests. ---------- The basic test is to find all the IPs for a mirror, including any CNAMES, then send HTTP HEAD requests to those IPs, with HOST headers for that mirror, and follow any redirections, doing the same for those redirections. Unless a specific mirror is given on the command line, the mirror_list.txt file from pkgmaster.devuan.org is used to select mirrors to test. The --tests= option can be used to adjust the list of tests performed. * IPv4, perform the tests with IPv4 addresses (A records) * IPv6, perform the tests with IPv6 addresses (AAAA records) * ftp, test FTP protocol access, check for the existence of the file instead of a HTTP HEAD. * http, test HTTP protocol access. * https, test HTTPS protocol access. * rsync, test RSYNC protocol access. * DNS-RR, ??????????????? * Protocol, warn if the protocol changed during a redirect. * URL-Sanity, add gratuitous multiple slashes to the URLs. * Integrity, check PGP signatures. * Updated, check Release dates. The old tests include a "DNS-RR" test, I'm not sure what that is. I guess it checks if the mirror responds properly if it's accessed via it's DNS RR (round robin) IP, and a HOST header of deb.devuan.org. If no other mirror is listed on the command line, we start with deb.devuan.org and check all of it's IPs, which are the DNS RR mirrors anyway. The mirror_list.txt file also used to select which protocols to test for each mirror, it will only test those protocols the mirror lists as supporting. Options. -------- --help Print the help text. --version Print the version. --tests Select which tests to run. The arguments are comma separated. A negative argument deselects a test. Examples are given above. --referenceSite The mirror to use as a reference for the tests, the default is pkgmaster.devuan.org. --roundRobin The name of the DNS round robin domain, the default is deb.devuan.org. -v Print more verbose output. Normally only CRITICAL and ERROR message sare printed. -v will print WARNING messages as well, -v -v INFO messages, and -v -v -v DEBUG messages. All messages are logged regardless. -q Only print CRITICAL messages. -k Keep any results from the previous runs, instead of deleting them before running the tests. -n Don't fork the tests. Normally each mirror is run in its own fork. Theory of operation. -------------------- Typically you would call it without any specific mirror mentioned on the command line. I'll start the discussion from there. Create the results directory. If -k is not given, delete results/*.log. Delete results/*.check. touch results/stamp Open results/apt-panopticon.log for message logging. Download mirror_list.txt from the reference site. Build a table of Active mirrors keyed by the FDQN, include the listed Protocols as a sub table. Add the round robin domain name. Resolve all the IPs and add them to this table. Write this table to results/mirrors.lua so that the forked tests can read it. checkHost() the reference site first. Loop through the mirrors table, and checkHost() each one, skipping the reference site. Wait for all forked tests to finish. Delete results/*.check. The checkHost() function does this - If there is no second argument, then the host is set to the first argument, otherwise the host is the second argument. Gather the IPs for the host name with the following command - dig +keepopen +noall +nottlid +answer example.com A example.com AAAA example.com CNAME example.com SRV | sort -r | uniq So it should end up with all the IPV4, IPV6, CNAME, and SRV records for that host. For each IPv4 and IPv6 address, fork a copy of the script something like this (including any arguments originally provided to the script) - ionice -c3 ./apt-panopticon.lua example.com/path x.x.x.x & ionice -c3 ./apt-panopticon.lua example.com/path [x:x:x:x:x:x] & For each CNAME, it checkHost() the host, but with the CNAME as a second argument. SRV reconds don't do anything yet, coz I have yet to see one from my test environment, so can't test it. Each forked call of the script from above does this - Open results/example.com_x.x.x.x.log for message logging. Loads the mirrors table from results/mirrors.lua. If performing the Integrity or Updated testes, delete results/example.com directory, downloads the reference files using wget. While it should actually perform the Integrity and Updated tests now, those haven't been written yet. Note that currently this downloads 4GB per mirror. Calls checkHost() with the host as first and second arguments, and includes the IP this time. The inclusion of the IP causes checkHost() to call checkFiles(). checkFiles() will call checkHEAD() for each of the reference files. checkHEAD() uses LuaSocket (or LuaSec for HTTPS) to send a HEAD request to the IP, with a Host header set to the original host name. Redirects will not be followed by that request. If the request returns a redirect, then checkHEAD() is called recursively. If the redirect is to some host we are not already checking, we call checkHost() on it, with an IP of "redir". This causes checkHost() to bypass the test that would otherwise call checkFiles(), instead gathering the IPs and fork as usual.