All Projects apt-panopticon  apt-panopticon_cgp boxes meta-impy  ImpyReleaseBuilder NPC-tool SledjHamr  IGnoble  NGIW  opensim-SC toyboxInstaller website - cgit website - CGraphz website - Mantis

View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000124apt-panopticonFeaturepublic2019-10-01 12:252019-12-28 10:06
Reporteronefang Assigned Toonefang  
PrioritynormalSeverityfeatureReproducibilityN/A
Status resolvedResolutionfixed 
Summary0000124: Add a quick option, or perhaps a slow one.
DescriptionQuick - don't check for all the default files in the HEAD tests, just enough to be sure the mirror is mostly doing the correct thing.

Slow - check all the default files for all tests.
TagsNo tags attached.

Relationships

Relationship GraphDependency Graph
related to 0000047 resolvedonefang Implement the Updated and Integrity tests. 
related to 0000062 assignedonefang Track new and updated packages, and check all mirrors have them. 

Activities

onefang

onefang

2019-10-02 05:28

administrator   ~0000210

Last edited: 2019-10-02 06:18

 What we need -

  • Quick test.
  • Low bandwidth test.
  • Medium bandwidth test.
  • High bandwidth test.
  • Exhaustive test.

So --quick, --low, --medium, --high, --all. Default to --quick.

--quick
Connection tests for all protocols on all IPs.
--low
HTTP(S) redirect tests.
--medium
Actually download Packages, and In/Release files.
Check the SHA512 of the Packages files.
Pick a few small packages, download them, check their SHA512.
--high
Same as --medium, but pick more packages, some for each arch.
--all
Do absolutely everything.
Actually download some Contents files and some package files.


What we got -

Contents files are by far the largest. One per release/architecture that lists filename / package for all packages.

Packages are the next largest. One per release/section/architecture that lists details of each package. Includes version, MD5sum, and SHA256, as well as the path/file to it in the mirror.

In/Release are tiny, at the top level of all releases is a list of SHA256 sums of the Contents, Packages, and Release files for that release.
onefang

onefang

2019-10-02 05:47

administrator   ~0000211

Last edited: 2019-11-26 05:55

 InRelease, Release, and Release.pgp files

merged/dists/ascii/InRelease is the PGP signed copy of merged/dists/ascii/Release, and merged/dists/ascii/Release.pgp is the binary PGP signature of merged/dists/ascii/Release.

InRelease is what "apt update" downloads. On the other hand the source code for apt uses both, so we might as well check both during Integrity tests.

It has the release (suite and codename), timestamp of when it was created, architectures, sections (components). The list is sha512, file length, and path/file of all the other Release, Sources*, Contents-*, and Packages.*.

The other Release files are just a tiny bit of info about each release/section/architecture.

Useful for checking the SHA512 of the other metadata files, and the latest timestamp of the latest changes.
onefang

onefang

2019-10-02 05:49

administrator   ~0000212

Last edited: 2019-11-26 06:23

 Contents files

Lists of "file package" so we know which packages include which files.

Are probably useless, except if we want to actually download AND actually unpack a package. Just to double check nothing went wrong with the process. So this is an --all thing, which will need code to be written. Though maybe for --high we can at least check their SHA512.
onefang

onefang

2019-10-02 05:54

administrator   ~0000213

Last edited: 2019-11-26 06:27

 Packages files

Package: astrometry-data-2mass
Version: 1.1
Installed-Size: 13
Maintainer: Debian Astronomy Team <debian-astro-maintainers@lists.alioth.debian.org>
Architecture: all
Depends: astrometry-data-2mass-00, astrometry-data-2mass-01, astrometry-data-2mass-02, astrometry-data-2mass-03, astrometry-data-2mass-04, astrometry-data-2mass-05, astrometry-data-2mass-06, astrometry-data-2mass-07,
astrometry-data-2mass-08-19
Homepage: http://data.astrometry.net/4200
Enhances: astrometry.net
Description: Astrometry.net 2MASS index files downloader
Description-md5: b23dcaa90c3e25f65e6fabb403e38f31
Section: contrib/science
Priority: optional
Filename: pool/DEBIAN/contrib/a/astrometry-data-2mass/astrometry-data-2mass_1.1_all.deb
Size: 2904
MD5sum: ad083db829e2d7837b56912c21ff76b3
SHA256: bfe35f99be23b1e3e6be95b58a51285a025e39bb612e00328bfdba0025ff8ed4

A blank line is between each block.

When we are testing the integrity of a package, this gives us version, package file size, MD5sum, and SHA256, as well as the path/file to it in the mirror. NOTE timestamp of last update isn't included.
onefang

onefang

2019-10-02 06:11

administrator   ~0000214

Last edited: 2019-10-02 06:56

 Timestamp for updated tests

No where in any of these metadata files do I see a timestamp for actual packages. So I'll have to compare versions.

Though perhaps I can get tricky with modified since headers in my HEAD check? Find out the modified since time on pkgmaster, compare that to the rest of the mirrors?
onefang

onefang

2019-10-02 07:44

administrator   ~0000216

Last edited: 2019-11-24 08:47

 Keep the top level Release files from pkgmaster around. Do an if-modified check on them from pkgmaster. If they changed, download the new one, diff against the old one, now we have "what Packages files changed since last time".

Keep the Packages files from pkgmaster around. Download the new ones from pkgmaster, diff against the old ones, now we have "what packages changed since last time".

Do if-modified check on the mirrors for the changed Release and Packages files (using the times from the old pkgmaster files). If they have been updated, then do more update checks on updated packages. If not, mark them as out of date.
onefang

onefang

2019-12-28 10:06

administrator   ~0000255

The basics of this is done, though still need to write and test the highest bandwidth options, which I just don't have bandwidth for at home. I'll get around to it eventually. In the mean time, closing this.

Issue History

Date Modified Username Field Change
2019-10-01 12:25 onefang New Issue
2019-10-01 12:25 onefang Status new => assigned
2019-10-01 12:25 onefang Assigned To => onefang
2019-10-02 05:28 onefang Note Added: 0000210
2019-10-02 05:47 onefang Note Added: 0000211
2019-10-02 05:49 onefang Note Added: 0000212
2019-10-02 05:50 onefang Note Edited: 0000212
2019-10-02 05:51 onefang Note Edited: 0000210
2019-10-02 05:54 onefang Note Added: 0000213
2019-10-02 06:06 onefang Note Edited: 0000210
2019-10-02 06:09 onefang Priority none => normal
2019-10-02 06:11 onefang Note Added: 0000214
2019-10-02 06:12 onefang Relationship added related to 0000047
2019-10-02 06:18 onefang Note Edited: 0000214
2019-10-02 06:18 onefang Note Edited: 0000210
2019-10-02 06:56 onefang Note Edited: 0000214
2019-10-02 07:44 onefang Note Added: 0000216
2019-10-02 08:18 onefang Note Edited: 0000216
2019-10-02 08:26 onefang Note Edited: 0000216
2019-10-02 08:28 onefang Relationship added related to 0000062
2019-11-24 05:32 onefang Note Edited: 0000211
2019-11-24 08:47 onefang Note Edited: 0000216
2019-11-26 03:59 onefang Note Edited: 0000211
2019-11-26 05:55 onefang Note Edited: 0000211
2019-11-26 06:23 onefang Note Edited: 0000212
2019-11-26 06:27 onefang Note Edited: 0000213
2019-12-28 10:06 onefang Status assigned => resolved
2019-12-28 10:06 onefang Resolution open => fixed
2019-12-28 10:06 onefang Note Added: 0000255