View Issue Details

IDProjectCategoryView StatusLast Update
0000142apt-panopticonFeaturepublic2024-03-03 17:28
Reporteronefang Assigned Toonefang  
PriorityimmediateSeveritymajorReproducibilityN/A
Status assignedResolutionopen 
Summary0000142: Have it automatically drive decisions of what mirror is in or out of the DNS-RR including CC.
Description<Evilham> anyway, if we delegate roundr.devuan.org to some devuany thing, which should be separated from the main DNS, I did a thing that is easily modifiable to fit the needs: https://weneeda.name/
<Evilham> so the general devuan.org zone: boring, static, manually managed
<Evilham> roundr.devuan.org: prone to automating, deb.devuan.org as a CNAME to deb.roundr.devuan.org gets both things in a way
<Evilham> so, for devuan, we'd have to do something that deals with info from apt-panopticon and the prometheus instance
<Evilham> what I'd ask you to add is RSA / SHA512 signing of a list of hosts you consider valid for the RR
<Evilham> with a timestamp :-)
<Evilham> and a random 32bit thing
<Evilham> basically, I think if we get the apt-panopticon data into the prometheus thing, we can send alerts to the DNS server and when that happens, have it trigger instantly an otherwise periodic check on your source file
<Evilham> and for that, we need to make sure that your source file is not tampered with
<Evilham> so, signing + timestamp + nonce are kinda a must
<onefang> I've already written a Nagios / Icinga plugin for alerts, though I'm happy to do your way to.
<Evilham> I don't mind either way tbh, as long as it's not noisy and robust
<onefang> Though that's more for running on the same computer, not sending to some other computer.
<Evilham> just a note that: if we do deploy something with this general line, it means that misbehaving / outdated mirrrors get removed from the pool automatically, that new mirrors get added after we update mirror-list.txt, ...
<onefang> Which is why I keep insisting we should discuss policy soon. B-)
<Evilham> policy about the RR? if it doesn't work it shouldn't be there :-)
<Evilham> ASAP
<Evilham> I think that's all that matters
<onefang> Policy about "how many errors / warnings over how long a time qualifies / disqualifies for the RR". B-)
<Evilham> say, 60 seconds of being "bad" to go out and 300 of being "good" to go in and make it tunable, it's not an issue that we temporarily remove things from the RR if they are back in after they behave properly
<onefang> Oh, that's a different take on what I was thinking. B-)
<Evilham> :-p it's kind of simple really, everything on deb.devuan.org should work fine, if it doesn't it shouldn't be there
<Evilham> or there is a 1/N chance of hitting it and that's a bug
<onefang> Though now I have to get the tests to under 60 seconds, I've got it consistently just under one on mine, rrq's looks to be just over.
<Evilham> I just said 60 seconds, it doesn't have to be that :-D
<Evilham> it can be 2m of being bad, or 5, something that can be changed later
<onefang> I have tunable bandwidth arguments, the default is "medium", there's a "low" that's a lot quicker. 22 seconds on rrq's.
<Evilham> we'd have to check how it behaves since mirrors actually sync every 30m
<Evilham> so, worst case every once in a while pkgmaster would be alone in the RR for a few mins
<Evilham> this can be tested before deployment without delegating roundr.devuan.org
TagsNo tags attached.

Relationships

parent of 0000446 assignedonefang Test CC.deb.devuan.org 
Not all the children of this issue are yet resolved or closed.

Activities

onefang

onefang

2020-01-03 10:21

administrator   ~0000262

In a nutshell -

Move the deb.roundr.devuan.org name server to something that can be automated, so apt-panopticon and Evilhams Prometheus instance can decide to add and remove IPs very quickly. It remains as a CNAME for deb.devuan.org.

Somehow alerts are sent to this thing that removes DNS records when mirrors drop below "good enough for the RR", or adds them when they rise above that limit. The alerts could be some new output module I write for apt-panopticon, Nagios / Icinga using the existing plugin, or some alert thingy from Evilhams Prometheus instance.

What "good enough" is can be tunable.

To avoid nasty people trying to screw with us, the alert has to be cryptographically secure if being sent between computers. (The VPS owned by rrq I just setup apt-panopticon on to avoid "server running it gets ludicrous speed" I doubt is secure enough for this.)
onefang

onefang

2020-06-01 10:48

administrator   ~0000286

Last edited: 2020-07-16 17:01

Think we will need to add a "DNS-RR" field to https://pkgmaster.devuan.org/mirror_list.txt, so we know which mirrors should be part of this automated to and fro.

Done.

onefang

onefang

2022-01-20 16:36

administrator   ~0000535

[03:40:30] <bb|hcb> Next step (maybe) is to make that script to also generate more proper CC.deb.devuan.org resolvings, at least to keep the traffic at the same continent and spread the traffic on the mirrors instead on pkgmaster (where resources are quite limited at 250mbit/s)
[03:43:10] <onefang> Luckily I already added the CountryCode field.
[03:48:52] <bb|hcb> I was thinking about including the ipv4/ipv6 addresses in mirror_list.txt, so that resolving happens only at apt-panopticon, else with misconfigured resolvers both sides may use different views of the internet (not highly probable, though)
[03:52:18] <bb|hcb> Also some clarification on the semantics of Active and DNSRR fields - Active should mean that the mirror is good to use and DNSRR should mean that the mirror is configured and properly serving deb.devuan.org AND that the admin agrees to include it there
[03:52:23] <onefang> Then we would have to keep an eye on those and change them if needed.
[03:53:26] <onefang> That was already my idea for the DNSRR field.
[03:54:16] <onefang> So you are saying apt-panopticon updates https://pkgmaster.devuan.org/mirror_list.txt ?
[03:55:12] <bb|hcb> Isn't it? That was my undestanding
[03:56:41] <onefang> There's a master file on pkgmaster that I update manually, and a cron job that runs once an hour that does that currently.
[03:57:57] <onefang> /home/mirror-admin/mirror_list.txt
[04:39:11] <bb|hcb> Oops, I was thinking that this is auto updated...
[04:42:51] <bb|hcb> And about CC RR, one more field will be needed - DNSRRCC, because not all mirrors may want to be included or configured to serve the content...
[10:53:27] <bb|hcb> onefang: What are your thoughts about automatic update of the Active field in mirror_list.txt? Do you need help for that?
[10:55:10] <bb|hcb> Obviously maintaining some form of the mirror admin intent in DNSRR and/or DNSRRCC should happen manually
[14:38:35] <onefang> bb|hcb: That's one way to deal with things. I have long planned to automatically update the DNS-RR based on the three apt-panopticons voting on which mirrors are up.
[14:39:18] <onefang> If you are now using / planning on using mirror_list.txt to change DNS-RR, that'll work.
onefang

onefang

2024-03-03 17:28

administrator   ~0000634

IMPORTANT - if something like the version number of base-files that we are testing for goes away, coz it got updated, than we shouldn't disable ALL the mirrors coz they all failed that test.

Issue History

Date Modified Username Field Change
2020-01-03 09:43 onefang New Issue
2020-01-03 09:43 onefang Status new => assigned
2020-01-03 09:43 onefang Assigned To => onefang
2020-01-03 10:04 onefang Description Updated
2020-01-03 10:10 onefang Description Updated
2020-01-03 10:21 onefang Note Added: 0000262
2020-06-01 10:48 onefang Note Added: 0000286
2020-07-16 17:01 onefang Note Edited: 0000286
2020-07-16 21:50 onefang Priority normal => urgent
2021-06-15 11:32 onefang Priority urgent => high
2022-01-20 16:36 onefang Note Added: 0000535
2022-10-14 00:50 onefang Summary Have it automatically drive decisions of what mirror is in or out of the DNS-RR. => Have it automatically drive decisions of what mirror is in or out of the DNS-RR including CC.
2022-10-14 01:08 onefang Relationship added parent of 0000446
2022-10-14 01:09 onefang Priority high => immediate
2024-03-03 17:28 onefang Note Added: 0000634