1 files changed, 166 insertions, 1 deletions

diff --git a/README.md b/README.md
index 3dc0609..c4363ed 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,8 @@
 Lua script for checking the health of Devuan Linux package mirrors.
 
-This is currently under development, not everything has been written yet.
+This is currently under development, not everything has been written yet. 
+Some of this document mentions some of the things that are not written
+yet.
 
 mirror-checker-lua is a Lua script used by the Devuan mirror admins
 (maybe, if they like it) to check the health of Devuan Linux package
@@ -92,3 +94,166 @@ To only run the HTTP integrity tests, only on IPv6 -
 
     $ ./mirror-checker.lua --tests=http,Integrity,IPv6
 
+
+The tests.
+----------
+
+The basic test is to find all the IPs for a mirror, including any CNAMES,
+then send HTTP HEAD requests to those IPs, with HOST headers for that mirror,
+and follow any redirections, doing the same for those redirections. 
+Unless a specific mirror is given on the command line, the mirror_list.txt
+file from pkgmaster.devuan.org is used to select mirrors to test.
+
+The --tests= option can be used to adjust the list of tests performed.
+
+* IPv4, perform the tests with IPv4 addresses (A records)
+* IPv6, perform the tests with IPv6 addresses (AAAA records)
+* ftp, test FTP protocol access, check for the existence of the file instead of a HTTP HEAD.
+* http, test HTTP protocol access.
+* https, test HTTPS protocol access.
+* rsync, test RSYNC protocol access.
+* DNS-RR, ???????????????
+* Protocol, warn if the protocol changed during a test.
+* URL-Sanity, add gratuitous multiple slashes to the URLs.
+* Integrity, check PGP signatures.
+* Updated, check Release dates.
+
+The old tests include a "DNS-RR" test, I'm not sure what that is.  I
+guess it checks if the mirror responds properly if it's accessed via it's
+DNS RR (round robin) IP, and a HOST header of deb.devuan.org.  If no
+other mirror is listed on the command line, we start with deb.devuan.org
+and check all of it's IPs, which are the DNS RR mirrors anyway.
+
+The mirror_list.txt file also used to select which protocols to test for
+each mirror, it will only test those protocols the mirror lists as
+supporting.
+
+
+Options.
+--------
+
+--help
+
+Print the help text.
+
+--version
+
+Print the version.
+
+--tests
+
+Select which tests to run.  The arguments are comma separated.  A
+negative argument deselects a test.  Examples are given above.
+
+--referenceSite
+
+The mirror to use as a reference for the tests, the default is pkgmaster.devuan.org.
+
+-v
+
+Print more verbose output.  Normally only CRITICAL and ERROR message sare
+printed.  -v will print WARNING messages as well, -v -v INFO messages,
+and -v -v -v DEBUG messages.  All messages are logged regardless.
+
+-q
+
+Only print CRITICAL messages.
+
+-k
+
+Keep any results from the previous runs, instead of deleting them before
+running the tests.
+
+-n
+
+Don't fork the tests.  Normally each mirror is run in its own fork.
+
+
+Theory of operation.
+--------------------
+
+Typically you would call it without any specific mirror mentioned on the
+command line.  I'll start the discussion from there.
+
+Create the results directory.
+
+If -k is not given, delete results/*.log.
+
+Delete results/*.check.
+
+touch results/stamp
+
+Open results/mirror-checker-lua.log for message logging.
+
+Download mirror_list.txt from the reference site.  Build a table of
+Active mirrors keyed by the FDQN, include the listed Protocols as a sub
+table.  Write this table to results/mirrors.lua so that the forked tests
+can read it.
+
+Remove the mirror site from the mirrors table, then CheckRedirects() it
+first.
+
+CheckRedirects() deb.devuan.org, the DNS RR.
+
+Loop through the mirrors table, and CheckRedirects() each one.
+
+Wait for all forked tests to finish.
+
+Delete results/*.check.
+
+
+The CheckRedirects() function does this -
+
+If there is no second argument, then the host is set to the first
+argument, otherwise the host is the second argument.
+
+Gather the IPs for the host name with the following command -
+
+dig +keepopen +noall +nottlid +answer example.com A example.com AAAA
+example.com CNAME example.com SRV | sort -r | uniq
+
+So it should end up with all the IPV4, IPV6, CNAME, and SRV records for
+that host.
+
+For each IPv4 and IPv6 address, fork a copy of the script something like
+this (including any arguments originally provided to the script) -
+
+ionice -c3 ./mirror-checker.lua example.com/path x.x.x.x &
+
+ionice -c3 ./mirror-checker.lua example.com/path [x:x:x:x:x:x] &
+
+For each CNAME, it CheckRedirects() the host, but with the CNAME as a
+second argument.
+
+SRV reconds don't do anything yet, coz I have yet to see one from my test
+environment, so can't test it.
+
+
+Each forked call of the script from above does this -
+
+Open results/mirror-checker-lua_example.com_x.x.x.x.log for message
+logging.
+
+Loads the mirrors table from results/mirrors.lua.
+
+If performing the Integrity or Updated testes, delete results/example.com
+directory, downloads the reference files using wget.  While it should
+actually perform the Integrity and Updated tests now, those haven't been
+written yet.  Note that currently this downloads 4GB per mirror.
+
+Calls CheckRedirects() with the host as first and second arguments, and
+includes the IP this time.  The inclusion of the IP causes
+CheckRedirects() to call checkPaths().
+
+
+checkPaths() will call checkURL() for each of the reference files.
+
+
+checkURL() uses LuaSocket (or LuaSec for HTTPS) to send a HEAD request to
+the IP, with a Host header set to the original host name.  Redirects will
+not be followed by that request.  If the request returns a redirect, then
+checkURL() is called recursively.  If the redirect is to some host we are
+not already checking, we call CheckRedirects() on it, with an IP of
+"redir".  This causes CheckRedirects() to bypass the IP gathering
+process, and call checkPaths() with the new host.
+