From c3b5b574bfd81f2093c4d795a929e27ed34708f1 Mon Sep 17 00:00:00 2001 From: onefang Date: Thu, 6 Feb 2020 14:13:24 +1000 Subject: Rewrite some of the output. Move explanations into tooltips. Invent a crappy HTML template "system". General clean up of text. --- README.md | 2 - Report-web_0.html | 4 + Report-web_1.html | 38 +++++++++ Report-web_2.html | 5 ++ Report-web_3.html | 11 +++ Report-web_TABLE.html | 158 ++++++++++++++++++++++++++++++++++++ apt-panopticon-report-email-web.lua | 97 +++++++++------------- apt-panopticon.lua | 1 - explanations.html | 96 ---------------------- 9 files changed, 255 insertions(+), 157 deletions(-) create mode 100644 Report-web_0.html create mode 100644 Report-web_1.html create mode 100644 Report-web_2.html create mode 100644 Report-web_3.html create mode 100644 Report-web_TABLE.html delete mode 100644 explanations.html diff --git a/README.md b/README.md index 982a108..9ff79f2 100644 --- a/README.md +++ b/README.md @@ -158,8 +158,6 @@ redirections. Unless a specific mirror is given on the command line, the mirror_list.txt file from pkgmaster.devuan.org is used to select mirrors to test. -explanations.html explains the tests in more detail. - The --tests= option can be used to adjust the list of tests performed. * IPv4, perform the tests with IPv4 addresses (A records) diff --git a/Report-web_0.html b/Report-web_0.html new file mode 100644 index 0000000..5f56c3f --- /dev/null +++ b/Report-web_0.html @@ -0,0 +1,4 @@ + + + + apt-panopticon results diff --git a/Report-web_1.html b/Report-web_1.html new file mode 100644 index 0000000..a27b076 --- /dev/null +++ b/Report-web_1.html @@ -0,0 +1,38 @@ + + + +

Welcome to the apt-panopticon results page.

+

Here the apt-panopticon system probes into every nook and cranny of the Devuan apt package mirror system, trying to live up to it's name.

+

This is the status of the mirror servers in the Devuan package mirror network.

diff --git a/Report-web_2.html b/Report-web_2.html new file mode 100644 index 0000000..592e8c6 --- /dev/null +++ b/Report-web_2.html @@ -0,0 +1,5 @@ +

EXPERIMENTAL CODE - double check all results you see here, and read the logs if it's important.

+

The full list of Devuan package mirrors is available at the URL: https://pkgmaster.devuan.org/mirror_list.txt

+

Due to the nature of the tests, some errors or warnings will be counted several times.   The links in the table and DNS list go to the detailed testing logs.

+ +
diff --git a/Report-web_3.html b/Report-web_3.html new file mode 100644 index 0000000..915bd56 --- /dev/null +++ b/Report-web_3.html @@ -0,0 +1,11 @@ +

Tooltips are things that have an explanation tooltip, hover over them.

+

FAILED or OK means the tested thing is supported for that mirror.

+

FAILED* or OK* means the tested thing is unsupported for that mirror, but might have been tested anyway.

+

timeout or timeout means the mirror had some timeouts, and tests where not yet aborted. The darker colour means unsupported by the mirror, but tested anyway.

+

TIMEOUT or TIMEOUT means the mirror had too many timeouts, and tests where aborted, so there is no result for this test.

+

NOTE: timeouts may be due to a problem on the testing computer, it might be busy with other things, or be having it's own network problems.

+

NOTE: the speed of the server this checking script is running on might be wildly exaggerated, it doesn't have to go through the Internet to download from itself.

+

The DNS round robin (DNS-RR) column shows the IPs for that mirror, or no if it isn't part of the DNS-RR.   The IPs link to the testing log for that IP accessed via the DNS-RR.   deb.devuan.org is the DNS-RR itself, so it doesn't get tested directly.

+

The time in the Updated column is how often the mirror updates itself.

+

Mirrors with a grey background are not active (though may be usable as part of the DNS-RR).

+

skip means that the test hasn't been written yet.

diff --git a/Report-web_TABLE.html b/Report-web_TABLE.html new file mode 100644 index 0000000..bad7148 --- /dev/null +++ b/Report-web_TABLE.html @@ -0,0 +1,158 @@ + + + + + + + + + + + + + + + + diff --git a/apt-panopticon-report-email-web.lua b/apt-panopticon-report-email-web.lua index e59f543..1cb74fd 100755 --- a/apt-panopticon-report-email-web.lua +++ b/apt-panopticon-report-email-web.lua @@ -15,11 +15,6 @@ APT.mirrors = loadfile("results/mirrors.lua")() APT.debians = loadfile("results/debians.lua")() -local lnk = function(name, link) - if nil == link then link = name end - return name .. " *" -end - local revDNS = function(hosts, dom, IP) if APT.options.roundRobin.value ~= dom then if nil ~= hosts[APT.options.roundRobin.value] then @@ -210,12 +205,18 @@ local DNSrrTest = function(hosts, k) return dns end + +local copyHTMLbit = function(web, file) + local rfile, e = io.open(file, "r") + if nil == rfile then W("opening " .. file .. " file - " .. e) else + for line in rfile:lines("*l") do + web:write(line .. '\n') + end + end +end + local makeTable = function(web, hosts) - web:write("
+
+ FTP +
+

FTP tests

+

The FTP tests have not been written yet.

+
+
+ 		+
+ HTTP +
+

HTTP tests

+

There are two styles of HTTP tests - actual HTTP downloads and HTTP HEAD tests.   + Actual downloads happen when other tests need the files to be downloaded.   + HEAD tests are where apt-panopticon probes things in detail.   + For each mirror (including the DNS round robin domain), and for each IP of that mirror - +

+
    +
  • Chose a small collection of package files to test, and the Release files for each release.
    • +
  • Send a HTTP HEAD request for each of those files.
    • +
  • Carefully inspect and log the response.
    • +
  • Retry the request if needed, and log that.
    • +
  • If the mirror replies with a redirect to the same mirror, then try that and keep checking.
    • +
  • If the mirror replies with a redirect to a different server, then probe that mirror the same way.
    • +
  • If a redirect loop is detected, log that and give up.
    • +
+
+
+ 		+
+ HTTPS +
+

HTTPS tests

+

The HTTPS tests are very similar to the HTTP tests detailed above, though obviously they are tried with HTTPS requests instead of HTTP requests.   + The validity of the HTTPS certificate for each server is tested as well. +

+
+
+ 		+
+ RSYNC +
+

RSYNC tests

+

The RSYNC tests have not been written yet.

+
+
+ 		+
+ DNS round robin +
+

DNS round robin

+

The "DNS round robin" column lists the IP addresses for each mirror that is part of the DNS round robin, or DNS-RR.   + The IPs are linked to the log for that specific IP when used via the DNS round robin, and is followed by the number of ERRORs, WARNINGs, or TIMEOUTs if any. +

+
+
+ 		+
+ Protocol +
+

Protocol tests

+

The Protocol test will give a warning if the protocol is changed during a redirect, HTTP -> HTTPS for example.   + While apt HTTPS transport is now the default in Beowulf / Buster, not everyone with an older release will have that installed, + so redirecting HTTP to HTTPS will break apt for those people.   + An ERROR is given instead if that happens for mirrors in the DNS round robin.   + Servers in the DNS round robin will not have the HTTPS certificate for the round robin domain, so redirecting to HTTPS for that is an ERROR. +

+
+
+ 		+
+ Redirects +
+

Redirect tests

+

Mirrors that redirect /DEVUAN/ back out to deb.devuan.org is an ERROR.   + /DEBIAN-SECURITY/ packages must be redirected to a Debian mirror, specifically a mirror hosting Debian security updates.   + However, some Devuan mirrors might also be Debian mirrors, so this is just a WARNING. +

+
+
+ 		+
+ URL sanity +
+

URL sanity tests

+

The URL sanity test replaces "/" in URLS with "///", to see if the mirror can cope with that.   + This might happen due to a minor mis-configuration by the apt user, but decent web servers should cope with that.   + The result for a mirror that does not cope is a failed download for that user, so this is an ERROR. +

+
+
+ 		+
+ Integrity +
+

Integrity tests

+

Actually download files, then check things like PGP keys, SHA256 check sums, and file size.   + For packages, pick the smallest one that has been recently updated. +

+
+
+ 		+
+ Updated +
+

Updated tests

+

Make sure the Release files are up to date by checking their internal "Date" field.   + If they are up to date, download and check updated Packages.xz files, and actual packages.   + For actual packages, pick the smallest one that has been recently updated. +

+

Also shown is the mirrors scheduled time between updates, with "m" meaning minutes and "h" meaning hours.   + If this is showing a WARNING, it's likely that the mirror hasn't updated yet, but is due to update after this test was run. +

+
+
+ 		+
+ Speed range +
+

Speed range

+

The speed test tries to guess at a minimum and maximum speed range for each mirror.   + It does this by measuring the reported speeds from the curl commands that actually download files.   + Since apt-panopticon is trying hard to download everything from all mirrors all at the same time, this guess will be low.   + Also, the computer running the apt-panopticon might have a network connection that is busy with other things.   + Not to mention that obviously if the computer this is running on is one of the mirrors, it might get a wildly exaggerated speed, + not having to go through the rest of the Internet tends to help.   + Finally, the tested mirror may have a bigger network connection than the computer running the test, so wont show it's true maximum.   + So take this speed measurement with a grain of salt, it's more of an indication, the full graphs might be more useful. +

+
+
+ 		+
+ Weekly statistics +
+

Weekly statistics

+

This is the percentage of time, over the last week, that the mirror was up, and the percentage of time the mirror was up to date.   + Note that if the mirror has a low uptime, then there wasn't much chance to check if it was up to date. +

+
+
+
\n" .. - "" .. - "" .. - "\n") + copyHTMLbit(web, "Report-web_TABLE.html") local bg = '' for k, v in APT.orderedPairs(hosts) do if '' == bg then bg = " style='background-color:#111111'" else bg = '' end @@ -344,20 +345,21 @@ APT.html = false local email, e = io.open("results/Report-email.txt", "w+") if nil == email then C("opening mirrors file - " .. e) else email:write( "Dear Mirror Admins,\n\n" .. - "This is the status of the mirror servers in the Devuan package mirror network.\n\n" .. - "EXPERIMENTAL CODE - double check all results you see here, and read the logs if it's important." .. + "This is a summary of the status of the mirror servers in the \nDevuan package mirror network.\n\n" .. + "EXPERIMENTAL CODE - double check all results you see here, \nand read the logs if it's important.\n\n" .. "The full list of Devuan package mirrors is available at the URL:\n\n" .. " https://pkgmaster.devuan.org/mirror_list.txt\n\n" .. - 'Please contact "mirrors@devuan.org" if any of the information \nin the file above needs to be amended. \n\n' .. - "The full results of the mirror checking is available at the URL:\n\n" .. - " https://sledjhamr.org/apt-panopticon/results/Report-web.html\n\n" .. + 'Please contact "mirrors@devuan.org" if any of the information \nin the file above needs to be amended.\n\n' .. + "The full results of the mirror checking is available at the URLs:\n\n" .. + " https://borta.devuan.dev/apt-panopticon/results/Report-web.html\n (updated once every hour)\n" .. + " https://sledjhamr.org/apt-panopticon/results/Report-web.html\n (updated once every ten minutes)\n\n" .. "Due to the nature of the tests, some errors or warnings will be \ncounted several times. " .. "Refer to the logs on the web page for details.\n\n" .. "Please see below the current status of the Devuan Package Mirror \nnetwork:\n\n" .. "==== package mirror status " .. os.date("!%F %H:%M") .. " GMT ====\n" .. "[skip] means that the test hasn't been written yet.\n\n") for k, v in APT.orderedPairs(APT.mirrors) do - email:write(k .. "....\n") + email:write(k .. "..\n") local results = APT.collateAll(APT.mirrors, 'results', k) local ftp = "[skip]" local http = status(APT.mirrors, k, results, "http") @@ -372,13 +374,19 @@ if nil == email then C("opening mirrors file - " .. e) else -- DNS-RR test. if (APT.options.roundRobin.value ~= k) and (nil ~= APT.mirrors[APT.options.roundRobin.value]) then - dns = " DNS-RR: " .. dns + dns = " DNS-RR: " .. dns .. "\n" end - email:write( " ftp: " .. ftp .. " http: " .. http .. " https: " .. https .." rsync: " .. rsync .. "\n" .. - " " .. dns .. "\n" .. - " Protocol: " .. protocol .. " URL-sanity: " .. sanity .. " Integrity: " .. integrity .. "\n" .. - " Updated: " .. updated .. "\n") + email:write( " ftp: " .. ftp .. "\n" .. + " http: " .. http .. "\n" .. + " https: " .. https .. "\n" .. + " rsync: " .. rsync .. "\n" .. + dns .. + " Protocol: " .. protocol .. "\n" .. + " Redirects: " .. redirects .. "\n" .. + " URL-sanity: " .. sanity .. "\n" .. + " Integrity: " .. integrity .. "\n" .. + " Updated: " .. updated .. "\n") end email:write( "\n==== faulty mirrors: ====\n" .. faulty) email:write( "\n-------------------------\n\n" .. @@ -387,7 +395,7 @@ if nil == email then C("opening mirrors file - " .. e) else "You can get the source code from https://sledjhamr.org/cgit/apt-panopticon/about/ (main repo)\n" .. "and from https://git.devuan.org/onefang/apt-panopticon' (Devuan repo).\n" .. "You can get the cgp graphing source code from https://sledjhamr.org/cgit/apt-panopticon_cgp/about/ (main repo)\n" .. - "and https://git.devuan.org/onefang/apt-panopticon_cgp (Devuan repo)\n" .. + "and https://git.devuan.org/onefang/apt-panopticon_cgp (Devuan repo)\n\n" .. "Love\n\n" .. "The Dev1Devs\n\n") email:close() @@ -454,60 +462,33 @@ APT.rrd.graph('results/speed.png', '--start', 'now-2w', '--end', 'now', '-t', 'S unpack(g)) - results = {} m = {} faulty = "" APT.html = true local web, e = io.open("results/Report-web.html", "w+") if nil == web then C("opening mirrors file - " .. e) else - web:write( "apt-panopticon results\n") + copyHTMLbit(web, "Report-web_0.html") if 0 < tonumber(APT.options.refresh.value) then web:write('\n') end - web:write( '' .. - "

Welcome to the apt-panopticon results page.

\n" .. - "

Here the apt-panopticon system probes into every nook and cranny of the Devuan apt package mirror system, trying to live up to it's name.

" .. - "

This is the status of the mirror servers in the Devuan package mirror network.

\n") + copyHTMLbit(web, "Report-web_1.html") if 0 < tonumber(APT.options.refresh.value) then web:write( '

This page will refresh every ' .. (APT.options.refresh.value / 60) .. ' minutes.

') end - web:write( "

EXPERIMENTAL CODE - double check all results you see here, and read the logs if it's important.

" .. - "

The full list of Devuan package mirrors is available at the URL: " .. - "https://pkgmaster.devuan.org/mirror_list.txt

\n" .. - "

Due to the nature of the tests, some errors or warnings will be counted several times.   " .. - "The links in the table and DNS list go to the detailed testing logs.

\n\n" .. - "
\n

==== package mirror status " .. os.date("!%F %H:%M") .. " GMT ====

\n" .. - - "

FAILED or OK" .. - " means the tested thing is supported for that mirror.

\n" .. - "

FAILED* or OK*" .. - " means the tested thing is unsupported for that mirror, but might have been tested anyway.

\n" .. - "

timeout or timeout" .. - " means the mirror had some timeouts, and tests where not yet aborted. The darker colour means unsupported by the mirror, but tested anyway.

" .. - "

TIMEOUT or TIMEOUT" .. - " means the mirror had too many timeouts, and tests where aborted, so there is no result for this test.

" .. - "

NOTE: timeouts may be due to a problem on the testing computer, it might be busy with other things, or be having it's own network problems..

" .. - "

NOTE: the speed of the server this checking script is running on might be wildly exaggerated, it doesn't have to go through the Internet to download from itself.

" .. - "

The DNS round robin (DNS-RR) column shows the IPs for that mirror, or no if it isn't part of the DNS-RR.   " .. - "The IPs link to the testing log for that IP accessed via the DNS-RR.   " .. - APT.options.roundRobin.value .. " is the DNS-RR itself, so it doesn't get tested directly.

\n" .. - "

The time in the Updated column is how often the mirror updates itself.

" .. - "

Mirrors with a grey background are not active (though may be usable as part of the DNS-RR).

\n" .. - "

skip means that the test hasn't been written yet.

\n" - ) - + copyHTMLbit(web, "Report-web_2.html") + web:write("\n

==== package mirror status " .. os.date("!%F %H:%M") .. " GMT ====

\n") + copyHTMLbit(web, "Report-web_3.html") makeTable(web, APT.mirrors) web:write( "

==== faulty mirrors: ====

\n" .. faulty) - web:write( "
\n
\n

==== DNS, graphs *, and logs: ====

\n") - + web:write( "
\n
\n

==== DNS, links to graphs, and links to logs: ====

\n") m = makeIPlist(APT.mirrors) web:write( "

This lists each mirror, and the DNS entries for that mirror.   " .. - "The links point to the testing log files (" .. logCount("apt-panopticon") .. ") for each domain name / IP combination that was tested.   " .. + "The IP links point to the testing log files (the overall log is " .. logCount("apt-panopticon") .. ") for each domain name / IP combination that was tested.   " .. "If a mirror has a CNAME, that CNAME is listed along with that CNAMEs DNS entries.   " .. APT.options.roundRobin.value .. " is the DNS round robin, which points to the mirrors that are part of the DNS-RR.   " .. "If an IP is part of the DNS-RR, it is marked with 'DNS-RR'   " .. - APT.options.referenceSite.value .. " is the master mirror, all the others sync to it.   " .. + APT.options.referenceSite.value .. " is the master mirror, all the others copy files from it.   " .. "

\n" ) web:write(APT.dumpTableHTML(m, "")) @@ -524,7 +505,7 @@ if nil == web then C("opening mirrors file - " .. e) else ) makeTable(web, APT.debians) - web:write( "
\n
\n

==== Debian DNS, graphs *, and logs: ====

\n") + web:write( "
\n
\n

==== Debian DNS, links to graphs, and links to logs: ====

\n") m = makeIPlist(APT.debians) web:write(APT.dumpTableHTML(m, "")) diff --git a/apt-panopticon.lua b/apt-panopticon.lua index a36b229..60e73f4 100755 --- a/apt-panopticon.lua +++ b/apt-panopticon.lua @@ -1129,7 +1129,6 @@ else os.execute("rm -f results/*.html 2>/dev/null") os.execute("rm -f results/*.txt 2>/dev/null") end - os.execute('ln -s ../explanations.html results/explanations.html 2>/dev/null') if not APT.logOpen('apt-panopticon') then return end I("Starting tests " .. table.concat(APT.options.tests.value, ", ")) diff --git a/explanations.html b/explanations.html deleted file mode 100644 index 73820c8..0000000 --- a/explanations.html +++ /dev/null @@ -1,96 +0,0 @@ -apt-panopticon explanations -

Welcome to the apt-panopticon explanations page.

-

The purpose of this page is to try to explain everything shown on the results page, -so that the results page can link to the various parts for the curious.

- -

IPV4 tests

-

Tests are tested over IPv4 if the mirror has IPv4 addresses.

- -

IPv6

-

Tests are tested over IPv6 if the mirror has IPv6 addresses.

- -

FTP tests

-

The FTP tests have not been written yet.

- -

HTTP tests

-

There are two styles of HTTP tests - actual HTTP downloads and HTTP HEAD tests.   -Actual downloads happen when other tests need the files to be downloaded.   -HEAD tests are where apt-panopticon probes things in detail.   -For each mirror (including the DNS round robin domain), and for each IP of that mirror - -

- - -

HTTPS tests

-

The HTTPS tests are very similar to the HTTP tests detailed above, though obviously they are tried with HTTPS requests intead of HTTP requests.   -The validity of the HTTPS certificate for each server is tested as well. -

- -

rsync tests

-

The RSYNC tests have not been written yet.

- -

DNS round robin tests

-

The "DNS round robin" column lists the IP addresses for each mirror that is part of the DNS round robin, or DNS-RR.   -The IPs are linked to the log for that specific IP when used via the DNS round robin, and is followed by the number of errors, warnings, or timeouts of any. -

- -

Redirect tests

-

Mirrors that redirect /DEVUAN/ back out to deb.devuan.org is an ERROR.   -/DEBIAN-SECURITY/ packages must be redirected to a Debian mirror, specifically a mirror hosting Debian security updates.   -However, some Devuan mirrors might also be Debian mirrors, so this is just a WARNING. -

- -

Protocol tests

-

The Protocol test will give a warning if the protocol is changed during a redirect, HTTP -> HTTPS for example.   -While apt HTTPS transport is now the default in Beowulf / Buster, not everyone with an older release will have that installed, -so redirecting HTTP to HTTPS will break apt for those people.   -An error is given instead if that happens for mirrors in the DNS round robin.   -Servers in the DNS round robin will not have the HTTPS certificate for the round robin domain, so redirecting to HTTPS for that is a mistake. -

- -

URL sanity tests

-

The URL sanity test replaces "/" in URLS with "///", to see if the mirror can cope with that.   -This might happen due to a minor mis-configuration by the apt user, but decent web servers should cope with that.   -The result for a mirror that does not cope is a failed download for that use, so this is an error. -

- -

Integrity tests

-

Actually download files, then check things like PGP keys, SHA256 check sums, and file size.   -For actual packages, pick the smallest one that has been recently updated. -

- -

Updated tests

-

Make sure the Release files are up to date by checking their internal "Date" field.   -If they are up to date, download and check updated Packages.xz files, and actual packages.   -For actual packages, pick the smallest one that has been recently updated. -

-

Also shown is the mirrors scheduled time between updates, with "m" meaning minutes and "h" meaning hours. -

- -

Speed test

-

The speed test tries to guess at a minimum and maximum speed range for each mirror.   -It does this by measuring the reported speeds from the curl commands that actually download files.   -Since apt-panopticon is trying hard to download everything from all mirrors all at the same time, this guess will be low.   -Also, the computer running the apt-panopticon might have a network connection that is busy with other things.   -Not to mention that obviously if the computer this is running on is one of the mirrors, it might get a wildly exaggerated speed, -not haaving to go through the rest of the Internet tends to help.   -Finally, the tested mirror may have a bigger network connection than the computer running the test, so wont show it's true maximum.   -So take this speed measurement with a grain of salt, it's more of an indication, the full graphs might be useful. -

- -

Weekly averages

-

This is the percentage of time the mirror was up, and the percentage of time the mirror was up to date.   -Note that if the mirror has a low uptime, then there wasn't much chance to check if it was up to date. -

- -

Graphs

-

A link to the graphs for this mirror.

- - -- cgit v1.1
" .. lnk('FTP') .. "" .. lnk('HTTP') .. "" .. lnk('HTTPS') .. "" .. lnk('RSYNC') .. "" .. lnk('DNS round robin', 'DNS-RR') .. "" .. lnk('Protocol') .. "" .. lnk('Redirects') .. - "" .. lnk('URL sanity', 'URL-Sanity') .. "" .. lnk('Integrity') .. "" .. lnk('Updated') .. "" .. lnk('Speed range', 'Speed') .. "" .. lnk('Weekly statistics', 'Weekly') .. "