diff options
-rw-r--r-- | README.md | 132 |
1 files changed, 26 insertions, 106 deletions
@@ -23,7 +23,15 @@ badgered him about his Python scripts. It should also be much easier to | |||
23 | use, the previous scripts needed some work before you could run them, | 23 | use, the previous scripts needed some work before you could run them, |
24 | this one you just download and run. | 24 | this one you just download and run. |
25 | 25 | ||
26 | The source code is at [https://sledjhamr.org/cgit/apt-panopticon/](https://sledjhamr.org/cgit/apt-panopticon/) | 26 | After a discussion, it is likely that apt-panopticon will be used to |
27 | automatically decide which Devuan mirrors are in the DNS round robin. | ||
28 | The details of that are yet to be determined. | ||
29 | |||
30 | |||
31 | You can get the source code from [https://sledjhamr.org/cgit/apt-panopticon/](https://sledjhamr.org/cgit/apt-panopticon/) (main repo) | ||
32 | and [https://git.devuan.org/onefang/apt-panopticon](https://git.devuan.org/onefang/apt-panopticon) (Devuan repo). | ||
33 | You can get the cgp graphing source code from [https://sledjhamr.org/cgit/apt-panopticon_cgp/]() (main repo) | ||
34 | and [https://git.devuan.org/onefang/apt-panopticon_cgp](https://git.devuan.org/onefang/apt-panopticon_cgp) (Devuan repo). | ||
27 | 35 | ||
28 | The issue tracker is at [https://sledjhamr.org/mantisbt/project_page.php?project_id=13](https://sledjhamr.org/mantisbt/project_page.php?project_id=13) | 36 | The issue tracker is at [https://sledjhamr.org/mantisbt/project_page.php?project_id=13](https://sledjhamr.org/mantisbt/project_page.php?project_id=13) |
29 | 37 | ||
@@ -56,13 +64,14 @@ If you want to have lots of graphs, also install | |||
56 | For the apt-panopticon_cgp package, which is used to show the detailed | 64 | For the apt-panopticon_cgp package, which is used to show the detailed |
57 | graphs, you'll need a web server that supports PHP. apt-panopticon_cgp | 65 | graphs, you'll need a web server that supports PHP. apt-panopticon_cgp |
58 | includes some support files for running PHP via CGI, which more web | 66 | includes some support files for running PHP via CGI, which more web |
59 | servers support. You'll need php-cgi for that. | 67 | servers support. You'll need php-cgi for that. It's been tested with |
68 | recent versions of Apache 2 and Lightttpd. | ||
60 | 69 | ||
61 | 70 | ||
62 | Web installation. | 71 | Web installation. |
63 | ----------------- | 72 | ----------------- |
64 | 73 | ||
65 | This is a suggestion for installation on a Devuan based web server. | 74 | This is a suggestion for installation on a Devuan based web server. |
66 | 75 | ||
67 | Create - | 76 | Create - |
68 | 77 | ||
@@ -74,7 +83,7 @@ Install apt-panopticon and apt-panopticon_cgp there, so you end up with - | |||
74 | /var/www/html/apt-panopticon/apt-panopticon_cgp | 83 | /var/www/html/apt-panopticon/apt-panopticon_cgp |
75 | 84 | ||
76 | The script update_apt-panopticon is an example script for updating | 85 | The script update_apt-panopticon is an example script for updating |
77 | everything, including commented out commands to update the source code. | 86 | everything, including commented out commands to update the source code. |
78 | The file apt-panopticron is an example crontab file for updating | 87 | The file apt-panopticron is an example crontab file for updating |
79 | everything once every ten minutes. They assume your web server user is | 88 | everything once every ten minutes. They assume your web server user is |
80 | www-data with a group of www-data, and you have a mirror user called | 89 | www-data with a group of www-data, and you have a mirror user called |
@@ -82,9 +91,8 @@ mirrors. For mirror operators, that mirrors user would be the owner of | |||
82 | the mirror files. You can change these to suite yourself. | 91 | the mirror files. You can change these to suite yourself. |
83 | 92 | ||
84 | Once everything is updated, | 93 | Once everything is updated, |
85 | /var/www/html/apt-panopticon/results/Report-web.html | 94 | /var/www/html/apt-panopticon/results/Report-web.html will point to the |
86 | will point to the main web page, and there will be a link at the bottom of | 95 | main web page, and there will be links that point to the detailed graphs. |
87 | that pointing to the detailed graphs. | ||
88 | 96 | ||
89 | Note that two runs of apt-panopticon have to happen ten minutes apart at | 97 | Note that two runs of apt-panopticon have to happen ten minutes apart at |
90 | least in order to see any data on the graphs. | 98 | least in order to see any data on the graphs. |
@@ -98,9 +106,7 @@ Using it. | |||
98 | 106 | ||
99 | These examples assume you are running it from the source code directory. | 107 | These examples assume you are running it from the source code directory. |
100 | A directory will be created called `results`, it'll be full of log files | 108 | A directory will be created called `results`, it'll be full of log files |
101 | and any files that get downloaded. There will also be `results/email` | 109 | and any files that get downloaded. |
102 | and `results/web` directories, with the notification emails and web pages | ||
103 | (once I write that bit). | ||
104 | 110 | ||
105 | Note that unlike typical commands, you can't run single character options | 111 | Note that unlike typical commands, you can't run single character options |
106 | together, so this is wrong - | 112 | together, so this is wrong - |
@@ -115,12 +121,12 @@ Just run the script to do all of the tests - | |||
115 | 121 | ||
116 | $ ./apt-panopticon.lua | 122 | $ ./apt-panopticon.lua |
117 | 123 | ||
118 | Which will print any errors. If you don't want to see errors - | 124 | Which will print any CRITICAL errors. If you don't want to see errors - |
119 | 125 | ||
120 | $ ./apt-panopticon.lua -q | 126 | $ ./apt-panopticon.lua -q |
121 | 127 | ||
122 | If you want to see warnings as well (as usual, the more `-v` options, the more | 128 | If you want to see other errors, warnings, and even more details as well |
123 | details) - | 129 | (as usual, the more `-v` options, the more details) - |
124 | 130 | ||
125 | $ ./apt-panopticon.lua -v | 131 | $ ./apt-panopticon.lua -v |
126 | 132 | ||
@@ -184,7 +190,8 @@ and check all of it's IPs, which are the DNS RR mirrors anyway. | |||
184 | 190 | ||
185 | The mirror_list.txt file also used to select which protocols to test for | 191 | The mirror_list.txt file also used to select which protocols to test for |
186 | each mirror, it will only test those protocols the mirror lists as | 192 | each mirror, it will only test those protocols the mirror lists as |
187 | supporting. | 193 | supporting. Actually it'll test the ones not supported as well, but mark |
194 | them differently in the results. | ||
188 | 195 | ||
189 | 196 | ||
190 | Options. | 197 | Options. |
@@ -200,13 +207,14 @@ Print the version. | |||
200 | 207 | ||
201 | -v | 208 | -v |
202 | 209 | ||
203 | Print more verbose output. Normally only CRITICAL and ERROR message sare | 210 | Print more verbose output. Normally only CRITICAL messages are printed. |
204 | printed. -v will print WARNING messages as well, -v -v INFO messages, | 211 | -v will print ERROR messages as well, -v -v WARNING messages, -v -v -v |
205 | and -v -v -v DEBUG messages. All messages are logged regardless. | 212 | INFO messages, and -v -v -v -v DEBUG messages. All messages are logged |
213 | regardless. | ||
206 | 214 | ||
207 | -q | 215 | -q |
208 | 216 | ||
209 | Only print CRITICAL messages. | 217 | Print no output messages. |
210 | 218 | ||
211 | -k | 219 | -k |
212 | 220 | ||
@@ -257,91 +265,3 @@ negative argument deselects a report. | |||
257 | 265 | ||
258 | Select which tests to run. The arguments are comma separated. A | 266 | Select which tests to run. The arguments are comma separated. A |
259 | negative argument deselects a test. Examples are given above. | 267 | negative argument deselects a test. Examples are given above. |
260 | |||
261 | |||
262 | Theory of operation. | ||
263 | -------------------- | ||
264 | |||
265 | Typically you would call it without any specific mirror mentioned on the | ||
266 | command line. I'll start the discussion from there. | ||
267 | |||
268 | Create the results directory. | ||
269 | |||
270 | If -k is not given, delete results/*.log. | ||
271 | |||
272 | Delete results/*.check. | ||
273 | |||
274 | touch results/stamp | ||
275 | |||
276 | Open results/apt-panopticon.log for message logging. | ||
277 | |||
278 | Download mirror_list.txt from the reference site. Build a table of | ||
279 | Active mirrors keyed by the FDQN, include the listed Protocols as a sub | ||
280 | table. Add the round robin domain name. Resolve all the IPs and add | ||
281 | them to this table. Write this table to results/mirrors.lua so that the | ||
282 | forked tests can read it. | ||
283 | |||
284 | checkHost() the reference site first. | ||
285 | |||
286 | Loop through the mirrors table, and checkHost() each one, skipping the | ||
287 | reference site. | ||
288 | |||
289 | Wait for all forked tests to finish. | ||
290 | |||
291 | Delete results/*.check. | ||
292 | |||
293 | |||
294 | The checkHost() function does this - | ||
295 | |||
296 | If there is no second argument, then the host is set to the first | ||
297 | argument, otherwise the host is the second argument. | ||
298 | |||
299 | Gather the IPs for the host name with the following command - | ||
300 | |||
301 | dig +keepopen +noall +nottlid +answer example.com A example.com AAAA | ||
302 | example.com CNAME example.com SRV | sort -r | uniq | ||
303 | |||
304 | So it should end up with all the IPV4, IPV6, CNAME, and SRV records for | ||
305 | that host. | ||
306 | |||
307 | For each IPv4 and IPv6 address, fork a copy of the script something like | ||
308 | this (including any arguments originally provided to the script) - | ||
309 | |||
310 | ionice -c3 ./apt-panopticon.lua example.com/path x.x.x.x & | ||
311 | |||
312 | ionice -c3 ./apt-panopticon.lua example.com/path [x:x:x:x:x:x] & | ||
313 | |||
314 | For each CNAME, it checkHost() the host, but with the CNAME as a second | ||
315 | argument. | ||
316 | |||
317 | SRV reconds don't do anything yet, coz I have yet to see one from my test | ||
318 | environment, so can't test it. | ||
319 | |||
320 | |||
321 | Each forked call of the script from above does this - | ||
322 | |||
323 | Open results/example.com_x.x.x.x.log for message logging. | ||
324 | |||
325 | Loads the mirrors table from results/mirrors.lua. | ||
326 | |||
327 | If performing the Integrity or Updated testes, delete results/example.com | ||
328 | directory, downloads the reference files using wget. While it should | ||
329 | actually perform the Integrity and Updated tests now, those haven't been | ||
330 | written yet. Note that currently this downloads 4GB per mirror. | ||
331 | |||
332 | Calls checkHost() with the host as first and second arguments, and | ||
333 | includes the IP this time. The inclusion of the IP causes checkHost() to | ||
334 | call checkFiles(). | ||
335 | |||
336 | |||
337 | checkFiles() will call checkHEAD() for each of the reference files. | ||
338 | |||
339 | |||
340 | checkHEAD() uses LuaSocket (or LuaSec for HTTPS) to send a HEAD request | ||
341 | to the IP, with a Host header set to the original host name. Redirects | ||
342 | will not be followed by that request. If the request returns a redirect, | ||
343 | then checkHEAD() is called recursively. If the redirect is to some host | ||
344 | we are not already checking, we call checkHost() on it, with an IP of | ||
345 | "redir". This causes checkHost() to bypass the test that would otherwise | ||
346 | call checkFiles(), instead gathering the IPs and fork as usual. | ||
347 | |||