diff options
Diffstat (limited to '')
-rw-r--r-- | README.md | 81 |
1 files changed, 41 insertions, 40 deletions
@@ -4,11 +4,11 @@ This is currently under development, not everything has been written yet. | |||
4 | Some of this document mentions some of the things that are not written | 4 | Some of this document mentions some of the things that are not written |
5 | yet. | 5 | yet. |
6 | 6 | ||
7 | mirror-checker-lua is a Lua script used by the Devuan mirror admins | 7 | apt-panopticon is a Lua script used by the Devuan mirror admins (maybe, |
8 | (maybe, if they like it) to check the health of Devuan Linux package | 8 | if they like it) to check the health of Devuan Linux package mirrors. |
9 | mirrors. Originally there was bash scripts for this job, then Evilham | 9 | Originally there was bash scripts for this job, then Evilham wrote some |
10 | wrote some Python scripts, now onefang has written it in Lua. We all | 10 | Python scripts, now onefang has written it in Lua. We all have different |
11 | have different tastes in languages. lol | 11 | tastes in languages. lol |
12 | 12 | ||
13 | The main difference is that this Lua version tries to do everything, and | 13 | The main difference is that this Lua version tries to do everything, and |
14 | will be maintained. Currently the shell scripts and Python scripts are | 14 | will be maintained. Currently the shell scripts and Python scripts are |
@@ -17,7 +17,7 @@ badgered him about his Python scripts. It should also be much easier to | |||
17 | use, the previous scripts needed some work before you could run them, | 17 | use, the previous scripts needed some work before you could run them, |
18 | this one you just download and run. | 18 | this one you just download and run. |
19 | 19 | ||
20 | The source code is at [https://sledjhamr.org/cgit/mirror-checker-lua/](https://sledjhamr.org/cgit/mirror-checker-lua/) | 20 | The source code is at [https://sledjhamr.org/cgit/apt-panopticon/](https://sledjhamr.org/cgit/apt-panopticon/) |
21 | 21 | ||
22 | The issue tracker is at [https://sledjhamr.org/mantisbt/project_page.php?project_id=13](https://sledjhamr.org/mantisbt/project_page.php?project_id=13) | 22 | The issue tracker is at [https://sledjhamr.org/mantisbt/project_page.php?project_id=13](https://sledjhamr.org/mantisbt/project_page.php?project_id=13) |
23 | 23 | ||
@@ -25,7 +25,7 @@ The issue tracker is at [https://sledjhamr.org/mantisbt/project_page.php?project | |||
25 | Installation. | 25 | Installation. |
26 | ------------- | 26 | ------------- |
27 | 27 | ||
28 | Download the source. You may want to put the mirror-checker.lua script | 28 | Download the source. You may want to put the apt-panopticon.lua script |
29 | in someplace like `/usr/local/bin` and make sure it is executable. | 29 | in someplace like `/usr/local/bin` and make sure it is executable. |
30 | 30 | ||
31 | It should run on any recent Linux, you'll need to have the following | 31 | It should run on any recent Linux, you'll need to have the following |
@@ -52,57 +52,58 @@ and `results/web` directories, with the notification emails and web pages | |||
52 | Note that unlike typical commands, you can't run single character options | 52 | Note that unlike typical commands, you can't run single character options |
53 | together, so this is wrong - | 53 | together, so this is wrong - |
54 | 54 | ||
55 | $ ./mirror-checker.lua -vvv | 55 | $ ./apt-panopticon.lua -vvv |
56 | 56 | ||
57 | Instead do this - | 57 | Instead do this - |
58 | 58 | ||
59 | $ ./mirror-checker.lua -v -v -v | 59 | $ ./apt-panopticon.lua -v -v -v |
60 | 60 | ||
61 | Just run the script to do all of the tests - | 61 | Just run the script to do all of the tests - |
62 | 62 | ||
63 | $ ./mirror-checker.lua | 63 | $ ./apt-panopticon.lua |
64 | 64 | ||
65 | Which will print any errors. If you don't want to see errors - | 65 | Which will print any errors. If you don't want to see errors - |
66 | 66 | ||
67 | $ ./mirror-checker.lua -q | 67 | $ ./apt-panopticon.lua -q |
68 | 68 | ||
69 | If you want to see warnings as well (as usual, the more `-v` options, the more | 69 | If you want to see warnings as well (as usual, the more `-v` options, the more |
70 | details) - | 70 | details) - |
71 | 71 | ||
72 | $ ./mirror-checker.lua -v | 72 | $ ./apt-panopticon.lua -v |
73 | 73 | ||
74 | Or use the usual options for the help and version number (not written yet) - | 74 | Or use the usual options for the help and version number (not written yet) - |
75 | 75 | ||
76 | $ ./mirror-checker.lua -h | 76 | $ ./apt-panopticon.lua -h |
77 | $ ./mirror-checker.lua --help | 77 | $ ./apt-panopticon.lua --help |
78 | $ ./mirror-checker.lua --version | 78 | $ ./apt-panopticon.lua --version |
79 | 79 | ||
80 | To run the tests on a specific mirror, for example pkgmaster.devuan.org - | 80 | To run the tests on a specific mirror, for example pkgmaster.devuan.org - |
81 | 81 | ||
82 | $ ./mirror-checker.lua pkgmaster.devuan.org | 82 | $ ./apt-panopticon.lua pkgmaster.devuan.org |
83 | 83 | ||
84 | You can use the `--tests` option to tune which tests are run, for example | 84 | You can use the `--tests` option to tune which tests are run, for example |
85 | to stop IPv6 tests, coz you don't have IPv6 - | 85 | to stop IPv6 tests, coz you don't have IPv6 - |
86 | 86 | ||
87 | $ ./mirror-checker.lua --tests=-IPv6 | 87 | $ ./apt-panopticon.lua --tests=-IPv6 |
88 | 88 | ||
89 | To do the same, but not run the HTTPS tests either - | 89 | To do the same, but not run the HTTPS tests either - |
90 | 90 | ||
91 | $ ./mirror-checker.lua --tests=-IPv6,-https | 91 | $ ./apt-panopticon.lua --tests=-IPv6,-https |
92 | 92 | ||
93 | To only run the HTTP integrity tests, only on IPv6 - | 93 | To only run the HTTP integrity tests, only on IPv6 - |
94 | 94 | ||
95 | $ ./mirror-checker.lua --tests=http,Integrity,IPv6 | 95 | $ ./apt-panopticon.lua --tests=http,Integrity,IPv6 |
96 | 96 | ||
97 | 97 | ||
98 | The tests. | 98 | The tests. |
99 | ---------- | 99 | ---------- |
100 | 100 | ||
101 | The basic test is to find all the IPs for a mirror, including any CNAMES, | 101 | The basic test is to find all the IPs for a mirror, including any CNAMES, |
102 | then send HTTP HEAD requests to those IPs, with HOST headers for that mirror, | 102 | then send HTTP HEAD requests to those IPs, with HOST headers for that |
103 | and follow any redirections, doing the same for those redirections. | 103 | mirror, and follow any redirections, doing the same for those |
104 | Unless a specific mirror is given on the command line, the mirror_list.txt | 104 | redirections. Unless a specific mirror is given on the command line, the |
105 | file from pkgmaster.devuan.org is used to select mirrors to test. | 105 | mirror_list.txt file from pkgmaster.devuan.org is used to select mirrors |
106 | to test. | ||
106 | 107 | ||
107 | The --tests= option can be used to adjust the list of tests performed. | 108 | The --tests= option can be used to adjust the list of tests performed. |
108 | 109 | ||
@@ -147,7 +148,8 @@ negative argument deselects a test. Examples are given above. | |||
147 | 148 | ||
148 | --referenceSite | 149 | --referenceSite |
149 | 150 | ||
150 | The mirror to use as a reference for the tests, the default is pkgmaster.devuan.org. | 151 | The mirror to use as a reference for the tests, the default is |
152 | pkgmaster.devuan.org. | ||
151 | 153 | ||
152 | --roundRobin | 154 | --roundRobin |
153 | 155 | ||
@@ -187,7 +189,7 @@ Delete results/*.check. | |||
187 | 189 | ||
188 | touch results/stamp | 190 | touch results/stamp |
189 | 191 | ||
190 | Open results/mirror-checker-lua.log for message logging. | 192 | Open results/apt-panopticon.log for message logging. |
191 | 193 | ||
192 | Download mirror_list.txt from the reference site. Build a table of | 194 | Download mirror_list.txt from the reference site. Build a table of |
193 | Active mirrors keyed by the FDQN, include the listed Protocols as a sub | 195 | Active mirrors keyed by the FDQN, include the listed Protocols as a sub |
@@ -221,12 +223,12 @@ that host. | |||
221 | For each IPv4 and IPv6 address, fork a copy of the script something like | 223 | For each IPv4 and IPv6 address, fork a copy of the script something like |
222 | this (including any arguments originally provided to the script) - | 224 | this (including any arguments originally provided to the script) - |
223 | 225 | ||
224 | ionice -c3 ./mirror-checker.lua example.com/path x.x.x.x & | 226 | ionice -c3 ./apt-panopticon.lua example.com/path x.x.x.x & |
225 | 227 | ||
226 | ionice -c3 ./mirror-checker.lua example.com/path [x:x:x:x:x:x] & | 228 | ionice -c3 ./apt-panopticon.lua example.com/path [x:x:x:x:x:x] & |
227 | 229 | ||
228 | For each CNAME, it checkHost() the host, but with the CNAME as a | 230 | For each CNAME, it checkHost() the host, but with the CNAME as a second |
229 | second argument. | 231 | argument. |
230 | 232 | ||
231 | SRV reconds don't do anything yet, coz I have yet to see one from my test | 233 | SRV reconds don't do anything yet, coz I have yet to see one from my test |
232 | environment, so can't test it. | 234 | environment, so can't test it. |
@@ -234,8 +236,7 @@ environment, so can't test it. | |||
234 | 236 | ||
235 | Each forked call of the script from above does this - | 237 | Each forked call of the script from above does this - |
236 | 238 | ||
237 | Open results/mirror-checker-lua_example.com_x.x.x.x.log for message | 239 | Open results/example.com_x.x.x.x.log for message logging. |
238 | logging. | ||
239 | 240 | ||
240 | Loads the mirrors table from results/mirrors.lua. | 241 | Loads the mirrors table from results/mirrors.lua. |
241 | 242 | ||
@@ -245,18 +246,18 @@ actually perform the Integrity and Updated tests now, those haven't been | |||
245 | written yet. Note that currently this downloads 4GB per mirror. | 246 | written yet. Note that currently this downloads 4GB per mirror. |
246 | 247 | ||
247 | Calls checkHost() with the host as first and second arguments, and | 248 | Calls checkHost() with the host as first and second arguments, and |
248 | includes the IP this time. The inclusion of the IP causes | 249 | includes the IP this time. The inclusion of the IP causes checkHost() to |
249 | checkHost() to call checkFiles(). | 250 | call checkFiles(). |
250 | 251 | ||
251 | 252 | ||
252 | checkFiles() will call checkHEAD() for each of the reference files. | 253 | checkFiles() will call checkHEAD() for each of the reference files. |
253 | 254 | ||
254 | 255 | ||
255 | checkHEAD() uses LuaSocket (or LuaSec for HTTPS) to send a HEAD request to | 256 | checkHEAD() uses LuaSocket (or LuaSec for HTTPS) to send a HEAD request |
256 | the IP, with a Host header set to the original host name. Redirects will | 257 | to the IP, with a Host header set to the original host name. Redirects |
257 | not be followed by that request. If the request returns a redirect, then | 258 | will not be followed by that request. If the request returns a redirect, |
258 | checkHEAD() is called recursively. If the redirect is to some host we are | 259 | then checkHEAD() is called recursively. If the redirect is to some host |
259 | not already checking, we call checkHost() on it, with an IP of | 260 | we are not already checking, we call checkHost() on it, with an IP of |
260 | "redir". This causes checkHost() to bypass the test that would | 261 | "redir". This causes checkHost() to bypass the test that would otherwise |
261 | otherwise call checkFiles(), instead gathering the IPs and fork as usual. | 262 | call checkFiles(), instead gathering the IPs and fork as usual. |
262 | 263 | ||