30 Dec 2010 robbat2   » (Master)

gentoo mirror stats: master distfiles distribution.

Now for the second set of statistics. These aren't directly useful to mirrors in estimating their traffic, but instead gives a good overview of how our mirroring setup works internally, and now much traffic is involved in the fan-out stage. Distfiles are the main content moved around by this system, but it is also used for the other directories for releases, experimental and snapshots.

A very quick overview of the existing setup:

  1. Developer uploads new distfile directly to dev.gentoo.org.
  2. The master-distfiles box pulls from dev.gentoo.org hourly.
  3. The master-distfiles box checks every ebuild, and downloads missing distfiles from their primary URI if they do not exist. The daily distfile report is also created at this point.
  4. Every hour, the cluster master of ftp.osuosl.org pulls the latest content from master-distfiles. (Averages 240MB/day of traffic).
  5. The OSL FTP cluster master (in Oregon) pushes to it's slave locations in Atlanta and Chicago.
  6. All distfiles mirrors pick up their content from one of the FTP nodes - Internet2-connected hosts are directed via DNS to an Internet2-connected slave for performance.

Each of the distfiles mirrors has about 140-160MB of upstream traffic every day (including both the new files and the rsync overhead for scanning). If there are no files changed, the rsync traffic for a directory scan is 1-2MB. While this isn't a lot of traffic, it's very spiky, as mirrors tend to be on fast links.

The new weekly builds from the Release Engineering team will probably be adding another 1.3GB per week, staggered as one arch per day.

I got a small subset of the logs from the OSU FTP cluster for processing some of these statistics. They cover the 24 hour period of 2008/08/07 UTC. It does not have data of which traffic went via Internet2, and I've grouped the sources by country code (using IP::Country::Fast from CPAN).

CC OutBytesCount, [Notes]
South America
AR 1498379141
BR 1498405221
== 299678436 2
AT 3202290562
BA 1498404221
BE 1464739661
BG 2199886072
CH 1496743121
CZ 8062803705
DE 149092997310
DK 2295154041
EE 1360037741
ES 4493037003
FI 1387115261
FR 7996356615
GB 3960190613
GR 4172227743 [1]
IS 1360037741
LV 1499118641
NL 4519136003
NO 1499088261
PL 6957242141
PT 2840207112
RO 3668540933
SE 4496643343
SK 1498405681
== 8683670590 55
AU 2974020902
JP 4493696853
KR 4509289423
RU 1972457562
SG 1356810941
TH 1358357761
TW 4927311704
== 2159194513 16
North America:
CA 7429692847
US 317491485824
== 3917884142 31
Middle East:
IL 1935272832
KW 1497725501
== 343299833 3

Grand Total:
== 15403727514 bytes 107

[1] One Greek mirror was excluded from the traffic and counts, as this was their catchup sync with 7Gb of traffic after some hardware-related downtime.

As a bit of analysis, I think that more than half of our mirrors (Europe, Middle East, RU) would benefit from having a box to sync against in Europe.

Syndicated 2008-12-16 22:50:31 from Move along, nothing to read

Latest blog entries     Older blog entries

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!