From 3f20527faadcfcdb0ce20b1a9dc54cfea8137282 Mon Sep 17 00:00:00 2001 From: onefang Date: Thu, 26 Sep 2019 03:23:54 +1000 Subject: More documentation. Briefly describe the tests. Document the options. Theory of operation. --help now says to read the README.md. --- README.md | 167 ++++++++++++++++++++++++++++++++++++++++++++++++++++- mirror-checker.lua | 2 +- 2 files changed, 167 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3dc0609..c4363ed 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ Lua script for checking the health of Devuan Linux package mirrors. -This is currently under development, not everything has been written yet. +This is currently under development, not everything has been written yet. +Some of this document mentions some of the things that are not written +yet. mirror-checker-lua is a Lua script used by the Devuan mirror admins (maybe, if they like it) to check the health of Devuan Linux package @@ -92,3 +94,166 @@ To only run the HTTP integrity tests, only on IPv6 - $ ./mirror-checker.lua --tests=http,Integrity,IPv6 + +The tests. +---------- + +The basic test is to find all the IPs for a mirror, including any CNAMES, +then send HTTP HEAD requests to those IPs, with HOST headers for that mirror, +and follow any redirections, doing the same for those redirections. +Unless a specific mirror is given on the command line, the mirror_list.txt +file from pkgmaster.devuan.org is used to select mirrors to test. + +The --tests= option can be used to adjust the list of tests performed. + +* IPv4, perform the tests with IPv4 addresses (A records) +* IPv6, perform the tests with IPv6 addresses (AAAA records) +* ftp, test FTP protocol access, check for the existence of the file instead of a HTTP HEAD. +* http, test HTTP protocol access. +* https, test HTTPS protocol access. +* rsync, test RSYNC protocol access. +* DNS-RR, ??????????????? +* Protocol, warn if the protocol changed during a test. +* URL-Sanity, add gratuitous multiple slashes to the URLs. +* Integrity, check PGP signatures. +* Updated, check Release dates. + +The old tests include a "DNS-RR" test, I'm not sure what that is. I +guess it checks if the mirror responds properly if it's accessed via it's +DNS RR (round robin) IP, and a HOST header of deb.devuan.org. If no +other mirror is listed on the command line, we start with deb.devuan.org +and check all of it's IPs, which are the DNS RR mirrors anyway. + +The mirror_list.txt file also used to select which protocols to test for +each mirror, it will only test those protocols the mirror lists as +supporting. + + +Options. +-------- + +--help + +Print the help text. + +--version + +Print the version. + +--tests + +Select which tests to run. The arguments are comma separated. A +negative argument deselects a test. Examples are given above. + +--referenceSite + +The mirror to use as a reference for the tests, the default is pkgmaster.devuan.org. + +-v + +Print more verbose output. Normally only CRITICAL and ERROR message sare +printed. -v will print WARNING messages as well, -v -v INFO messages, +and -v -v -v DEBUG messages. All messages are logged regardless. + +-q + +Only print CRITICAL messages. + +-k + +Keep any results from the previous runs, instead of deleting them before +running the tests. + +-n + +Don't fork the tests. Normally each mirror is run in its own fork. + + +Theory of operation. +-------------------- + +Typically you would call it without any specific mirror mentioned on the +command line. I'll start the discussion from there. + +Create the results directory. + +If -k is not given, delete results/*.log. + +Delete results/*.check. + +touch results/stamp + +Open results/mirror-checker-lua.log for message logging. + +Download mirror_list.txt from the reference site. Build a table of +Active mirrors keyed by the FDQN, include the listed Protocols as a sub +table. Write this table to results/mirrors.lua so that the forked tests +can read it. + +Remove the mirror site from the mirrors table, then CheckRedirects() it +first. + +CheckRedirects() deb.devuan.org, the DNS RR. + +Loop through the mirrors table, and CheckRedirects() each one. + +Wait for all forked tests to finish. + +Delete results/*.check. + + +The CheckRedirects() function does this - + +If there is no second argument, then the host is set to the first +argument, otherwise the host is the second argument. + +Gather the IPs for the host name with the following command - + +dig +keepopen +noall +nottlid +answer example.com A example.com AAAA +example.com CNAME example.com SRV | sort -r | uniq + +So it should end up with all the IPV4, IPV6, CNAME, and SRV records for +that host. + +For each IPv4 and IPv6 address, fork a copy of the script something like +this (including any arguments originally provided to the script) - + +ionice -c3 ./mirror-checker.lua example.com/path x.x.x.x & + +ionice -c3 ./mirror-checker.lua example.com/path [x:x:x:x:x:x] & + +For each CNAME, it CheckRedirects() the host, but with the CNAME as a +second argument. + +SRV reconds don't do anything yet, coz I have yet to see one from my test +environment, so can't test it. + + +Each forked call of the script from above does this - + +Open results/mirror-checker-lua_example.com_x.x.x.x.log for message +logging. + +Loads the mirrors table from results/mirrors.lua. + +If performing the Integrity or Updated testes, delete results/example.com +directory, downloads the reference files using wget. While it should +actually perform the Integrity and Updated tests now, those haven't been +written yet. Note that currently this downloads 4GB per mirror. + +Calls CheckRedirects() with the host as first and second arguments, and +includes the IP this time. The inclusion of the IP causes +CheckRedirects() to call checkPaths(). + + +checkPaths() will call checkURL() for each of the reference files. + + +checkURL() uses LuaSocket (or LuaSec for HTTPS) to send a HEAD request to +the IP, with a Host header set to the original host name. Redirects will +not be followed by that request. If the request returns a redirect, then +checkURL() is called recursively. If the redirect is to some host we are +not already checking, we call CheckRedirects() on it, with an IP of +"redir". This causes CheckRedirects() to bypass the IP gathering +process, and call checkPaths() with the new host. + diff --git a/mirror-checker.lua b/mirror-checker.lua index 6675a49..b2d1cc7 100755 --- a/mirror-checker.lua +++ b/mirror-checker.lua @@ -411,7 +411,7 @@ if 0 ~= #args then local option = "" for i, a in pairs(args) do if ("--help" == a) or ("-h" == a) then - print("I should write some docs, huh?") + print("I should write some docs, huh? Read README.md for instructions.") elseif "--version" == a then print("mirror-checker-lua version 0.1 WIP development version") elseif "-v" == a then -- cgit v1.1