aboutsummaryrefslogtreecommitdiffstatshomepage
diff options
context:
space:
mode:
authoronefang2019-09-26 03:23:54 +1000
committeronefang2019-09-26 03:23:54 +1000
commit3f20527faadcfcdb0ce20b1a9dc54cfea8137282 (patch)
treee7057c03314e0145768bf11f3e897ab210801f4f
parentWhitespace fix. (diff)
downloadapt-panopticon-3f20527faadcfcdb0ce20b1a9dc54cfea8137282.zip
apt-panopticon-3f20527faadcfcdb0ce20b1a9dc54cfea8137282.tar.gz
apt-panopticon-3f20527faadcfcdb0ce20b1a9dc54cfea8137282.tar.bz2
apt-panopticon-3f20527faadcfcdb0ce20b1a9dc54cfea8137282.tar.xz
More documentation.
Briefly describe the tests. Document the options. Theory of operation. --help now says to read the README.md.
Diffstat (limited to '')
-rw-r--r--README.md167
-rwxr-xr-xmirror-checker.lua2
2 files changed, 167 insertions, 2 deletions
diff --git a/README.md b/README.md
index 3dc0609..c4363ed 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,8 @@
1Lua script for checking the health of Devuan Linux package mirrors. 1Lua script for checking the health of Devuan Linux package mirrors.
2 2
3This is currently under development, not everything has been written yet. 3This is currently under development, not everything has been written yet.
4Some of this document mentions some of the things that are not written
5yet.
4 6
5mirror-checker-lua is a Lua script used by the Devuan mirror admins 7mirror-checker-lua is a Lua script used by the Devuan mirror admins
6(maybe, if they like it) to check the health of Devuan Linux package 8(maybe, if they like it) to check the health of Devuan Linux package
@@ -92,3 +94,166 @@ To only run the HTTP integrity tests, only on IPv6 -
92 94
93 $ ./mirror-checker.lua --tests=http,Integrity,IPv6 95 $ ./mirror-checker.lua --tests=http,Integrity,IPv6
94 96
97
98The tests.
99----------
100
101The basic test is to find all the IPs for a mirror, including any CNAMES,
102then send HTTP HEAD requests to those IPs, with HOST headers for that mirror,
103and follow any redirections, doing the same for those redirections.
104Unless a specific mirror is given on the command line, the mirror_list.txt
105file from pkgmaster.devuan.org is used to select mirrors to test.
106
107The --tests= option can be used to adjust the list of tests performed.
108
109* IPv4, perform the tests with IPv4 addresses (A records)
110* IPv6, perform the tests with IPv6 addresses (AAAA records)
111* ftp, test FTP protocol access, check for the existence of the file instead of a HTTP HEAD.
112* http, test HTTP protocol access.
113* https, test HTTPS protocol access.
114* rsync, test RSYNC protocol access.
115* DNS-RR, ???????????????
116* Protocol, warn if the protocol changed during a test.
117* URL-Sanity, add gratuitous multiple slashes to the URLs.
118* Integrity, check PGP signatures.
119* Updated, check Release dates.
120
121The old tests include a "DNS-RR" test, I'm not sure what that is. I
122guess it checks if the mirror responds properly if it's accessed via it's
123DNS RR (round robin) IP, and a HOST header of deb.devuan.org. If no
124other mirror is listed on the command line, we start with deb.devuan.org
125and check all of it's IPs, which are the DNS RR mirrors anyway.
126
127The mirror_list.txt file also used to select which protocols to test for
128each mirror, it will only test those protocols the mirror lists as
129supporting.
130
131
132Options.
133--------
134
135--help
136
137Print the help text.
138
139--version
140
141Print the version.
142
143--tests
144
145Select which tests to run. The arguments are comma separated. A
146negative argument deselects a test. Examples are given above.
147
148--referenceSite
149
150The mirror to use as a reference for the tests, the default is pkgmaster.devuan.org.
151
152-v
153
154Print more verbose output. Normally only CRITICAL and ERROR message sare
155printed. -v will print WARNING messages as well, -v -v INFO messages,
156and -v -v -v DEBUG messages. All messages are logged regardless.
157
158-q
159
160Only print CRITICAL messages.
161
162-k
163
164Keep any results from the previous runs, instead of deleting them before
165running the tests.
166
167-n
168
169Don't fork the tests. Normally each mirror is run in its own fork.
170
171
172Theory of operation.
173--------------------
174
175Typically you would call it without any specific mirror mentioned on the
176command line. I'll start the discussion from there.
177
178Create the results directory.
179
180If -k is not given, delete results/*.log.
181
182Delete results/*.check.
183
184touch results/stamp
185
186Open results/mirror-checker-lua.log for message logging.
187
188Download mirror_list.txt from the reference site. Build a table of
189Active mirrors keyed by the FDQN, include the listed Protocols as a sub
190table. Write this table to results/mirrors.lua so that the forked tests
191can read it.
192
193Remove the mirror site from the mirrors table, then CheckRedirects() it
194first.
195
196CheckRedirects() deb.devuan.org, the DNS RR.
197
198Loop through the mirrors table, and CheckRedirects() each one.
199
200Wait for all forked tests to finish.
201
202Delete results/*.check.
203
204
205The CheckRedirects() function does this -
206
207If there is no second argument, then the host is set to the first
208argument, otherwise the host is the second argument.
209
210Gather the IPs for the host name with the following command -
211
212dig +keepopen +noall +nottlid +answer example.com A example.com AAAA
213example.com CNAME example.com SRV | sort -r | uniq
214
215So it should end up with all the IPV4, IPV6, CNAME, and SRV records for
216that host.
217
218For each IPv4 and IPv6 address, fork a copy of the script something like
219this (including any arguments originally provided to the script) -
220
221ionice -c3 ./mirror-checker.lua example.com/path x.x.x.x &
222
223ionice -c3 ./mirror-checker.lua example.com/path [x:x:x:x:x:x] &
224
225For each CNAME, it CheckRedirects() the host, but with the CNAME as a
226second argument.
227
228SRV reconds don't do anything yet, coz I have yet to see one from my test
229environment, so can't test it.
230
231
232Each forked call of the script from above does this -
233
234Open results/mirror-checker-lua_example.com_x.x.x.x.log for message
235logging.
236
237Loads the mirrors table from results/mirrors.lua.
238
239If performing the Integrity or Updated testes, delete results/example.com
240directory, downloads the reference files using wget. While it should
241actually perform the Integrity and Updated tests now, those haven't been
242written yet. Note that currently this downloads 4GB per mirror.
243
244Calls CheckRedirects() with the host as first and second arguments, and
245includes the IP this time. The inclusion of the IP causes
246CheckRedirects() to call checkPaths().
247
248
249checkPaths() will call checkURL() for each of the reference files.
250
251
252checkURL() uses LuaSocket (or LuaSec for HTTPS) to send a HEAD request to
253the IP, with a Host header set to the original host name. Redirects will
254not be followed by that request. If the request returns a redirect, then
255checkURL() is called recursively. If the redirect is to some host we are
256not already checking, we call CheckRedirects() on it, with an IP of
257"redir". This causes CheckRedirects() to bypass the IP gathering
258process, and call checkPaths() with the new host.
259
diff --git a/mirror-checker.lua b/mirror-checker.lua
index 6675a49..b2d1cc7 100755
--- a/mirror-checker.lua
+++ b/mirror-checker.lua
@@ -411,7 +411,7 @@ if 0 ~= #args then
411 local option = "" 411 local option = ""
412 for i, a in pairs(args) do 412 for i, a in pairs(args) do
413 if ("--help" == a) or ("-h" == a) then 413 if ("--help" == a) or ("-h" == a) then
414 print("I should write some docs, huh?") 414 print("I should write some docs, huh? Read README.md for instructions.")
415 elseif "--version" == a then 415 elseif "--version" == a then
416 print("mirror-checker-lua version 0.1 WIP development version") 416 print("mirror-checker-lua version 0.1 WIP development version")
417 elseif "-v" == a then 417 elseif "-v" == a then