gentoo mirror stats: master distfiles distribution.
Now for the second set of statistics. These aren't directly useful to mirrors in estimating their traffic, but instead gives a good overview of how our mirroring setup works internally, and now much traffic is involved in the fan-out stage. Distfiles are the main content moved around by this system, but it is also used for the other directories for releases, experimental and snapshots.
A very quick overview of the existing setup:
- Developer uploads new distfile directly to dev.gentoo.org.
- The master-distfiles box pulls from dev.gentoo.org hourly.
- The master-distfiles box checks every ebuild, and downloads missing distfiles from their primary URI if they do not exist. The daily distfile report is also created at this point.
- Every hour, the cluster master of ftp.osuosl.org pulls the latest content from master-distfiles. (Averages 240MB/day of traffic).
- The OSL FTP cluster master (in Oregon) pushes to it's slave locations in Atlanta and Chicago.
- All distfiles mirrors pick up their content from one of the FTP nodes - Internet2-connected hosts are directed via DNS to an Internet2-connected slave for performance.
Each of the distfiles mirrors has about 140-160MB of upstream traffic every day (including both the new files and the rsync overhead for scanning). If there are no files changed, the rsync traffic for a directory scan is 1-2MB. While this isn't a lot of traffic, it's very spiky, as mirrors tend to be on fast links.
The new weekly builds from the Release Engineering team will probably be adding another 1.3GB per week, staggered as one arch per day.
I got a small subset of the logs from the OSU FTP cluster for processing some of these statistics. They cover the 24 hour period of 2008/08/07 UTC. It does not have data of which traffic went via Internet2, and I've grouped the sources by country code (using IP::Country::Fast from CPAN).
| CC | OutBytes | Count, [Notes] |
|---|---|---|
| South America | ||
| AR | 149837914 | 1 |
| BR | 149840522 | 1 |
| == | 299678436 | 2 |
| Europe | ||
| AT | 320229056 | 2 |
| BA | 149840422 | 1 |
| BE | 146473966 | 1 |
| BG | 219988607 | 2 |
| CH | 149674312 | 1 |
| CZ | 806280370 | 5 |
| DE | 1490929973 | 10 |
| DK | 229515404 | 1 |
| EE | 136003774 | 1 |
| ES | 449303700 | 3 |
| FI | 138711526 | 1 |
| FR | 799635661 | 5 |
| GB | 396019061 | 3 |
| GR | 417222774 | 3 [1] |
| IS | 136003774 | 1 |
| LV | 149911864 | 1 |
| NL | 451913600 | 3 |
| NO | 149908826 | 1 |
| PL | 695724214 | 1 |
| PT | 284020711 | 2 |
| RO | 366854093 | 3 |
| SE | 449664334 | 3 |
| SK | 149840568 | 1 |
| == | 8683670590 | 55 |
| Asia/Oceania | ||
| AU | 297402090 | 2 |
| JP | 449369685 | 3 |
| KR | 450928942 | 3 |
| RU | 197245756 | 2 |
| SG | 135681094 | 1 |
| TH | 135835776 | 1 |
| TW | 492731170 | 4 |
| == | 2159194513 | 16 |
| North America: | ||
| CA | 742969284 | 7 |
| US | 3174914858 | 24 |
| == | 3917884142 | 31 |
| Middle East: | ||
| IL | 193527283 | 2 |
| KW | 149772550 | 1 |
| == | 343299833 | 3 |
|
|
||
| Grand Total: | ||
| == | 15403727514 bytes | 107 |
[1] One Greek mirror was excluded from the traffic and counts, as this was their catchup sync with 7Gb of traffic after some hardware-related downtime.
As a bit of analysis, I think that more than half of our mirrors (Europe, Middle East, RU) would benefit from having a box to sync against in Europe.
Syndicated 2008-12-16 22:50:31 from Move along, nothing to read