freebsd portsnap - what (it is), why (it was written), and ... · freebsd portsnap what (it is),...
TRANSCRIPT
![Page 1: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/1.jpg)
FreeBSD Portsnap
What (it is), Why (it was written), and How (it works)
Colin PercivalThe FreeBSD Project
May 19, 2007
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 2: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/2.jpg)
FreeBSD Portsnap
A Case Study in Black Magic
Colin PercivalThe FreeBSD Project
May 19, 2007
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 3: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/3.jpg)
Introduction to Portsnap
Portsnap is a system for securely and efficiently distributingthe FreeBSD Ports tree.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 4: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/4.jpg)
Introduction to Portsnap
Portsnap is a system for securely and efficiently distributingthe FreeBSD Ports tree.
Introduced in October 2004, added to the base system inAugust 2005.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 5: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/5.jpg)
Introduction to Portsnap
Portsnap is a system for securely and efficiently distributingthe FreeBSD Ports tree.
Introduced in October 2004, added to the base system inAugust 2005.
Present in all releases since FreeBSD 6.0-RELEASE,5.5-RELEASE.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 6: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/6.jpg)
Introduction to Portsnap
Portsnap is a system for securely and efficiently distributingthe FreeBSD Ports tree.
Introduced in October 2004, added to the base system inAugust 2005.
Present in all releases since FreeBSD 6.0-RELEASE,5.5-RELEASE.
Now used on approximately 30,000 systems.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 7: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/7.jpg)
Introduction to Portsnap
Portsnap is a system for securely and efficiently distributingthe FreeBSD Ports tree.
Introduced in October 2004, added to the base system inAugust 2005.
Present in all releases since FreeBSD 6.0-RELEASE,5.5-RELEASE.
Now used on approximately 30,000 systems.
Yes, I will have some pretty graphs later.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 8: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/8.jpg)
A bird’s-eye view of Portsnap
Portsnap build code runs on hardware “owned” by theFreeBSD Security Team.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 9: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/9.jpg)
A bird’s-eye view of Portsnap
Portsnap build code runs on hardware “owned” by theFreeBSD Security Team.
Builds are uploaded via ssh to portsnap-master.freebsd.org.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 10: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/10.jpg)
A bird’s-eye view of Portsnap
Portsnap build code runs on hardware “owned” by theFreeBSD Security Team.
Builds are uploaded via ssh to portsnap-master.freebsd.org.
Mirrors (3 of them, so far) update fromportsnap-master.freebsd.org.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 11: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/11.jpg)
A bird’s-eye view of Portsnap
Portsnap build code runs on hardware “owned” by theFreeBSD Security Team.
Builds are uploaded via ssh to portsnap-master.freebsd.org.
Mirrors (3 of them, so far) update fromportsnap-master.freebsd.org.
Individual client systems update /var/db/portsnap from arandomly selected mirror.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 12: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/12.jpg)
A bird’s-eye view of Portsnap
Portsnap build code runs on hardware “owned” by theFreeBSD Security Team.
Builds are uploaded via ssh to portsnap-master.freebsd.org.
Mirrors (3 of them, so far) update fromportsnap-master.freebsd.org.
Individual client systems update /var/db/portsnap from arandomly selected mirror.
The ports tree can be extracted or updated from/var/db/portsnap.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 13: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/13.jpg)
Black Magic #1: DNS SRV records
DNS SRV records (RFC 2782) provide a mechanism formapping a type of service to host name(s).
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 14: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/14.jpg)
Black Magic #1: DNS SRV records
DNS SRV records (RFC 2782) provide a mechanism formapping a type of service to host name(s).
Approximately a generalization of MX records.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 15: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/15.jpg)
Black Magic #1: DNS SRV records
DNS SRV records (RFC 2782) provide a mechanism formapping a type of service to host name(s).
Approximately a generalization of MX records.Clients are expected to pick a server randomly based on thespecified priorities and weights.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 16: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/16.jpg)
Black Magic #1: DNS SRV records
DNS SRV records (RFC 2782) provide a mechanism formapping a type of service to host name(s).
Approximately a generalization of MX records.Clients are expected to pick a server randomly based on thespecified priorities and weights.
http. tcp.portsnap.freebsd.org IN SRV 1 10 80
portsnap1
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 17: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/17.jpg)
Black Magic #1: DNS SRV records
DNS SRV records (RFC 2782) provide a mechanism formapping a type of service to host name(s).
Approximately a generalization of MX records.Clients are expected to pick a server randomly based on thespecified priorities and weights.
http. tcp.portsnap.freebsd.org IN SRV 1 10 80
portsnap1
Portsnap runs over HTTP, and obeys the HTTP PROXYenvironment variable.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 18: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/18.jpg)
Black Magic #1: DNS SRV records
DNS SRV records (RFC 2782) provide a mechanism formapping a type of service to host name(s).
Approximately a generalization of MX records.Clients are expected to pick a server randomly based on thespecified priorities and weights.
http. tcp.portsnap.freebsd.org IN SRV 1 10 80
portsnap1
Portsnap runs over HTTP, and obeys the HTTP PROXYenvironment variable.
If HTTP PROXY is set, Portsnap usesSHA256(HTTP PROXY) as a random number seed whenselecting a random mirror.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 19: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/19.jpg)
FreeBSD Update
FreeBSD Update is a system for building, distributing, andapplying binary security updates to the FreeBSD base system.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 20: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/20.jpg)
FreeBSD Update
FreeBSD Update is a system for building, distributing, andapplying binary security updates to the FreeBSD base system.
Introduced in April 2003, presented at BSDCon’03.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 21: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/21.jpg)
FreeBSD Update
FreeBSD Update is a system for building, distributing, andapplying binary security updates to the FreeBSD base system.
Introduced in April 2003, presented at BSDCon’03.
Updates are signed to prove that they are authentic.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 22: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/22.jpg)
FreeBSD Update
FreeBSD Update is a system for building, distributing, andapplying binary security updates to the FreeBSD base system.
Introduced in April 2003, presented at BSDCon’03.
Updates are signed to prove that they are authentic.
No need to trust CVSup mirrors!
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 23: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/23.jpg)
FreeBSD Update
FreeBSD Update is a system for building, distributing, andapplying binary security updates to the FreeBSD base system.
Introduced in April 2003, presented at BSDCon’03.
Updates are signed to prove that they are authentic.
No need to trust CVSup mirrors!
Until August 2006, FreeBSD Update was in the Ports tree.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 24: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/24.jpg)
FreeBSD Update
FreeBSD Update is a system for building, distributing, andapplying binary security updates to the FreeBSD base system.
Introduced in April 2003, presented at BSDCon’03.
Updates are signed to prove that they are authentic.
No need to trust CVSup mirrors!
Until August 2006, FreeBSD Update was in the Ports tree.... which most people downloaded via CVSup.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 25: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/25.jpg)
FreeBSD Update
FreeBSD Update is a system for building, distributing, andapplying binary security updates to the FreeBSD base system.
Introduced in April 2003, presented at BSDCon’03.
Updates are signed to prove that they are authentic.
No need to trust CVSup mirrors!
Until August 2006, FreeBSD Update was in the Ports tree.... which most people downloaded via CVSup.
... Oops.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 26: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/26.jpg)
Secure CVS trees
Add a checksum file to each directory in the tree, containing
... the hashes of all the other files in the directory.
... the hashes of the checksum files in any (immediate)subdirectories.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 27: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/27.jpg)
Secure CVS trees
Add a checksum file to each directory in the tree, containing
... the hashes of all the other files in the directory.
... the hashes of the checksum files in any (immediate)subdirectories.
Sign the checksum file in the root directory.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 28: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/28.jpg)
Secure CVS trees
Add a checksum file to each directory in the tree, containing
... the hashes of all the other files in the directory.
... the hashes of the checksum files in any (immediate)subdirectories.
Sign the checksum file in the root directory.
Each time a commit is done, automatically rebuild checksumfiles going up to the root, and re-sign the root checksum file.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 29: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/29.jpg)
Secure CVS trees
Add a checksum file to each directory in the tree, containing
... the hashes of all the other files in the directory.
... the hashes of the checksum files in any (immediate)subdirectories.
Sign the checksum file in the root directory.
Each time a commit is done, automatically rebuild checksumfiles going up to the root, and re-sign the root checksum file.
I hope someone builds this some day. I didn’t have time.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 30: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/30.jpg)
A simpler approach
Instead of making the tree self-authenticating and usingexisting mechanisms to distribute it, keep authentication outof the tree and have a new utility which downloads andverifies.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 31: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/31.jpg)
A simpler approach
Instead of making the tree self-authenticating and usingexisting mechanisms to distribute it, keep authentication outof the tree and have a new utility which downloads andverifies.
Divide the tree into N independent pieces, and generate anN-line index file containing the hashes of all the pieces.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 32: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/32.jpg)
A simpler approach
Instead of making the tree self-authenticating and usingexisting mechanisms to distribute it, keep authentication outof the tree and have a new utility which downloads andverifies.
Divide the tree into N independent pieces, and generate anN-line index file containing the hashes of all the pieces.
Distribute the N pieces, the index, and a signed hash of theindex as static files over HTTP.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 33: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/33.jpg)
A simpler approach
Instead of making the tree self-authenticating and usingexisting mechanisms to distribute it, keep authentication outof the tree and have a new utility which downloads andverifies.
Divide the tree into N independent pieces, and generate anN-line index file containing the hashes of all the pieces.
Distribute the N pieces, the index, and a signed hash of theindex as static files over HTTP.
We don’t really need to invent a new protocol after all...
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 34: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/34.jpg)
Black Magic #2: Static files
Serving static files is easy – choose your favourite HTTPserver.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 35: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/35.jpg)
Black Magic #2: Static files
Serving static files is easy – choose your favourite HTTPserver.
HTTP servers are light-weight compared to more complicatedprotocols like CVSup and rsync.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 36: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/36.jpg)
Black Magic #2: Static files
Serving static files is easy – choose your favourite HTTPserver.
HTTP servers are light-weight compared to more complicatedprotocols like CVSup and rsync.
Using static files over HTTP makes firewall/proxy traversaleasy.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 37: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/37.jpg)
Black Magic #2: Static files
Serving static files is easy – choose your favourite HTTPserver.
HTTP servers are light-weight compared to more complicatedprotocols like CVSup and rsync.
Using static files over HTTP makes firewall/proxy traversaleasy.
Actually, squid manages to cause problems by not supportingHTTP/1.1, but I think that can be worked around.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 38: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/38.jpg)
Black Magic #2: Static files
Serving static files is easy – choose your favourite HTTPserver.
HTTP servers are light-weight compared to more complicatedprotocols like CVSup and rsync.
Using static files over HTTP makes firewall/proxy traversaleasy.
Actually, squid manages to cause problems by not supportingHTTP/1.1, but I think that can be worked around.
Using static files (and a signature) provides end to endsecurity.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 39: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/39.jpg)
Black Magic #2: Static files
Serving static files is easy – choose your favourite HTTPserver.
HTTP servers are light-weight compared to more complicatedprotocols like CVSup and rsync.
Using static files over HTTP makes firewall/proxy traversaleasy.
Actually, squid manages to cause problems by not supportingHTTP/1.1, but I think that can be worked around.
Using static files (and a signature) provides end to endsecurity.
We don’t need to worry about the possibility of mirrors beingcompromised.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 40: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/40.jpg)
Black Magic #2: Static files
Serving static files is easy – choose your favourite HTTPserver.
HTTP servers are light-weight compared to more complicatedprotocols like CVSup and rsync.
Using static files over HTTP makes firewall/proxy traversaleasy.
Actually, squid manages to cause problems by not supportingHTTP/1.1, but I think that can be worked around.
Using static files (and a signature) provides end to endsecurity.
We don’t need to worry about the possibility of mirrors beingcompromised.We don’t need to worry about the possibility of an SSLcertificate being compromised.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 41: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/41.jpg)
Dividing up the ports tree
We want to divide the ports tree into N pieces.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 42: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/42.jpg)
Dividing up the ports tree
We want to divide the ports tree into N pieces.
The larger N is, the larger the overhead costs (TCP, HTTP,inodes, etc.) of handling many small files.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 43: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/43.jpg)
Dividing up the ports tree
We want to divide the ports tree into N pieces.
The larger N is, the larger the overhead costs (TCP, HTTP,inodes, etc.) of handling many small files.
The smaller N is, the larger the cost (bandwidth, CPU time)of updating each piece.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 44: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/44.jpg)
Dividing up the ports tree
We want to divide the ports tree into N pieces.
The larger N is, the larger the overhead costs (TCP, HTTP,inodes, etc.) of handling many small files.
The smaller N is, the larger the cost (bandwidth, CPU time)of updating each piece.
Asymptotically, we probably want N = O(√
[size of tree]).
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 45: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/45.jpg)
Dividing up the ports tree
We want to divide the ports tree into N pieces.
The larger N is, the larger the overhead costs (TCP, HTTP,inodes, etc.) of handling many small files.
The smaller N is, the larger the cost (bandwidth, CPU time)of updating each piece.
Asymptotically, we probably want N = O(√
[size of tree]).
For a tree of ≈ 100 MB it’s reasonable for N to be a fewthousand.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 46: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/46.jpg)
Dividing up the ports tree
We want to divide the ports tree into N pieces.
The larger N is, the larger the overhead costs (TCP, HTTP,inodes, etc.) of handling many small files.
The smaller N is, the larger the cost (bandwidth, CPU time)of updating each piece.
Asymptotically, we probably want N = O(√
[size of tree]).
For a tree of ≈ 100 MB it’s reasonable for N to be a fewthousand.
In Portsnap, the pieces are
/usr/ports/category/port/usr/ports/category/file/usr/ports/file
and each piece is stored as a tarball.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 47: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/47.jpg)
Black Magic #3: Understand how things change
The central problem of efficient data compression is to modelfiles.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 48: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/48.jpg)
Black Magic #3: Understand how things change
The central problem of efficient data compression is to modelfiles.
Most compressors explicitly use the first n bytes to predict thevalue of the n + 1th byte.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 49: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/49.jpg)
Black Magic #3: Understand how things change
The central problem of efficient data compression is to modelfiles.
Most compressors explicitly use the first n bytes to predict thevalue of the n + 1th byte.
The central problem of efficient delta compression is to modelhow files change.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 50: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/50.jpg)
Black Magic #3: Understand how things change
The central problem of efficient data compression is to modelfiles.
Most compressors explicitly use the first n bytes to predict thevalue of the n + 1th byte.
The central problem of efficient delta compression is to modelhow files change.
Side note: Part of the reason bsdiff is so efficient is that it isthe first delta compressor designed with an awareness of bytesubstitutions.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 51: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/51.jpg)
Black Magic #3: Understand how things change
The central problem of efficient data compression is to modelfiles.
Most compressors explicitly use the first n bytes to predict thevalue of the n + 1th byte.
The central problem of efficient delta compression is to modelhow files change.
Side note: Part of the reason bsdiff is so efficient is that it isthe first delta compressor designed with an awareness of bytesubstitutions.
Commits to the ports tree often modify several files, butusually they are part of the same port.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 52: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/52.jpg)
Black Magic #3: Understand how things change
The central problem of efficient data compression is to modelfiles.
Most compressors explicitly use the first n bytes to predict thevalue of the n + 1th byte.
The central problem of efficient delta compression is to modelhow files change.
Side note: Part of the reason bsdiff is so efficient is that it isthe first delta compressor designed with an awareness of bytesubstitutions.
Commits to the ports tree often modify several files, butusually they are part of the same port.
Dividing the tree into individual ports is a natural granualitybased on how the tree changes.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 53: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/53.jpg)
Black Magic #4: Reference by hash
Traditional approach: ”ports/misc/bsdiff is stored inmisc bsdiff 123.tar and has SHA256 hash01234567...89ABCDEF”.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 54: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/54.jpg)
Black Magic #4: Reference by hash
Traditional approach: ”ports/misc/bsdiff is stored inmisc bsdiff 123.tar and has SHA256 hash01234567...89ABCDEF”.
Reference by hash: ”ports/misc/bsdiff is stored in01234567...89ABCDEF.tar and has SHA256 hash01234567...89ABCDEF”.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 55: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/55.jpg)
Black Magic #4: Reference by hash
Traditional approach: ”ports/misc/bsdiff is stored inmisc bsdiff 123.tar and has SHA256 hash01234567...89ABCDEF”.
Reference by hash: ”ports/misc/bsdiff is stored in01234567...89ABCDEF.tar and has SHA256 hash01234567...89ABCDEF”.
Don’t need to worry about naming collisions, since a stronghash will never collide.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 56: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/56.jpg)
Black Magic #4: Reference by hash
Traditional approach: ”ports/misc/bsdiff is stored inmisc bsdiff 123.tar and has SHA256 hash01234567...89ABCDEF”.
Reference by hash: ”ports/misc/bsdiff is stored in01234567...89ABCDEF.tar and has SHA256 hash01234567...89ABCDEF”.
Don’t need to worry about naming collisions, since a stronghash will never collide.
Well, hopefully, at least.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 57: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/57.jpg)
Black Magic #4: Reference by hash
Traditional approach: ”ports/misc/bsdiff is stored inmisc bsdiff 123.tar and has SHA256 hash01234567...89ABCDEF”.
Reference by hash: ”ports/misc/bsdiff is stored in01234567...89ABCDEF.tar and has SHA256 hash01234567...89ABCDEF”.
Don’t need to worry about naming collisions, since a stronghash will never collide.
Well, hopefully, at least.
Each part of the tree is self-authenticating.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 58: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/58.jpg)
Black Magic #4: Reference by hash
Traditional approach: ”ports/misc/bsdiff is stored inmisc bsdiff 123.tar and has SHA256 hash01234567...89ABCDEF”.
Reference by hash: ”ports/misc/bsdiff is stored in01234567...89ABCDEF.tar and has SHA256 hash01234567...89ABCDEF”.
Don’t need to worry about naming collisions, since a stronghash will never collide.
Well, hopefully, at least.
Each part of the tree is self-authenticating.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 59: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/59.jpg)
Things get ugly: Distributing INDEX files
FreeBSD package tools use an INDEX file which summarizesthe ports tree.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 60: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/60.jpg)
Things get ugly: Distributing INDEX files
FreeBSD package tools use an INDEX file which summarizesthe ports tree.
Package name, version, directory, dependencies...
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 61: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/61.jpg)
Things get ugly: Distributing INDEX files
FreeBSD package tools use an INDEX file which summarizesthe ports tree.
Package name, version, directory, dependencies...
The INDEX file is generated by recursing into every Makefilein the tree.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 62: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/62.jpg)
Things get ugly: Distributing INDEX files
FreeBSD package tools use an INDEX file which summarizesthe ports tree.
Package name, version, directory, dependencies...
The INDEX file is generated by recursing into every Makefilein the tree.
This takes 10–30 minutes.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 63: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/63.jpg)
Things get ugly: Distributing INDEX files
FreeBSD package tools use an INDEX file which summarizesthe ports tree.
Package name, version, directory, dependencies...
The INDEX file is generated by recursing into every Makefilein the tree.
This takes 10–30 minutes.If someone can insert a trojan into misc/nobody-uses-this,they can execute arbitrary code on any system which builds anINDEX.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 64: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/64.jpg)
Things get ugly: Distributing INDEX files
FreeBSD package tools use an INDEX file which summarizesthe ports tree.
Package name, version, directory, dependencies...
The INDEX file is generated by recursing into every Makefilein the tree.
This takes 10–30 minutes.If someone can insert a trojan into misc/nobody-uses-this,they can execute arbitrary code on any system which builds anINDEX.
INDEX is built on the Portsnap buildbox and distributed toclient systems.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 65: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/65.jpg)
Things get ugly: Distributing INDEX files
FreeBSD package tools use an INDEX file which summarizesthe ports tree.
Package name, version, directory, dependencies...
The INDEX file is generated by recursing into every Makefilein the tree.
This takes 10–30 minutes.If someone can insert a trojan into misc/nobody-uses-this,they can execute arbitrary code on any system which builds anINDEX.
INDEX is built on the Portsnap buildbox and distributed toclient systems.
For security reasons, INDEX is built as a non-privileged userinside a jail which contains a minimal FreeBSD world where allfilesystems are mounted either readonly or noexec.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 66: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/66.jpg)
Things get ugly: Distributing INDEX files
FreeBSD package tools use an INDEX file which summarizesthe ports tree.
Package name, version, directory, dependencies...
The INDEX file is generated by recursing into every Makefilein the tree.
This takes 10–30 minutes.If someone can insert a trojan into misc/nobody-uses-this,they can execute arbitrary code on any system which builds anINDEX.
INDEX is built on the Portsnap buildbox and distributed toclient systems.
For security reasons, INDEX is built as a non-privileged userinside a jail which contains a minimal FreeBSD world where allfilesystems are mounted either readonly or noexec.
Hopefully this is good enough...
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 67: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/67.jpg)
Saving bandwidth
Instead of downloading complete files, Portsnap downloadspatches against older versions whenever possible.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 68: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/68.jpg)
Saving bandwidth
Instead of downloading complete files, Portsnap downloadspatches against older versions whenever possible.
Binary patches are used for the component tarballs.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 69: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/69.jpg)
Saving bandwidth
Instead of downloading complete files, Portsnap downloadspatches against older versions whenever possible.
Binary patches are used for the component tarballs.A hacked-up textual patch format is used for the index ofcomponents and for the ports INDEX file.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 70: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/70.jpg)
Saving bandwidth
Instead of downloading complete files, Portsnap downloadspatches against older versions whenever possible.
Binary patches are used for the component tarballs.A hacked-up textual patch format is used for the index ofcomponents and for the ports INDEX file.
For a typical 58 hour window of updates in 2005, CVSup used6388kB of bandwidth, while portsnap only used 370kB.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 71: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/71.jpg)
Saving bandwidth
Instead of downloading complete files, Portsnap downloadspatches against older versions whenever possible.
Binary patches are used for the component tarballs.A hacked-up textual patch format is used for the index ofcomponents and for the ports INDEX file.
For a typical 58 hour window of updates in 2005, CVSup used6388kB of bandwidth, while portsnap only used 370kB.
When very little has changed in the tree, CVSup spends mostof its time/bandwidth listing files and deciding that theyhaven’t changed.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 72: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/72.jpg)
Black Magic #5: Opportunistic patching
Problem: If you have N versions of a file, there are O(N2)pairs between which to build patches.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 73: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/73.jpg)
Black Magic #5: Opportunistic patching
Problem: If you have N versions of a file, there are O(N2)pairs between which to build patches.
Building O(N2) patches takes a long time.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 74: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/74.jpg)
Black Magic #5: Opportunistic patching
Problem: If you have N versions of a file, there are O(N2)pairs between which to build patches.
Building O(N2) patches takes a long time.Applying a series of N patches, one by one, is bothcomplicated and slow.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 75: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/75.jpg)
Black Magic #5: Opportunistic patching
Problem: If you have N versions of a file, there are O(N2)pairs between which to build patches.
Building O(N2) patches takes a long time.Applying a series of N patches, one by one, is bothcomplicated and slow.
Opportunistic patching: Build some patches, but not all ofthem.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 76: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/76.jpg)
Black Magic #5: Opportunistic patching
Problem: If you have N versions of a file, there are O(N2)pairs between which to build patches.
Building O(N2) patches takes a long time.Applying a series of N patches, one by one, is bothcomplicated and slow.
Opportunistic patching: Build some patches, but not all ofthem.
Client systems try to fetch a patch, but fall back to fetching acomplete file if the patch isn’t available.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 77: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/77.jpg)
Black Magic #5: Opportunistic patching
Problem: If you have N versions of a file, there are O(N2)pairs between which to build patches.
Building O(N2) patches takes a long time.Applying a series of N patches, one by one, is bothcomplicated and slow.
Opportunistic patching: Build some patches, but not all ofthem.
Client systems try to fetch a patch, but fall back to fetching acomplete file if the patch isn’t available.By building a small number of patches, we can ensure thatmost systems will be using patches most of the time.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 78: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/78.jpg)
Black Magic #5: Opportunistic patching
Problem: If you have N versions of a file, there are O(N2)pairs between which to build patches.
Building O(N2) patches takes a long time.Applying a series of N patches, one by one, is bothcomplicated and slow.
Opportunistic patching: Build some patches, but not all ofthem.
Client systems try to fetch a patch, but fall back to fetching acomplete file if the patch isn’t available.By building a small number of patches, we can ensure thatmost systems will be using patches most of the time.Right now, patches are always for Portsnap on systems whichupdate at least once a week.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 79: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/79.jpg)
Portsnap updating statistics
0
2×104
4×104
6×104
8×104
105
hour day week month
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 80: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/80.jpg)
Black Magic #6: Pipelined HTTP
Pipelined HTTP can easily speed up fetching small files by anorder of magnitude.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 81: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/81.jpg)
Black Magic #6: Pipelined HTTP
Pipelined HTTP can easily speed up fetching small files by anorder of magnitude.
When Portsnap is fetching patches (typical size 500 bytes) thespeedup can be over a factor of 100.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 82: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/82.jpg)
Black Magic #6: Pipelined HTTP
Pipelined HTTP can easily speed up fetching small files by anorder of magnitude.
When Portsnap is fetching patches (typical size 500 bytes) thespeedup can be over a factor of 100.
Not really black magic at all — pipelined HTTP is somethingwhich everybody should be using.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 83: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/83.jpg)
Black Magic #6: Pipelined HTTP
Pipelined HTTP can easily speed up fetching small files by anorder of magnitude.
When Portsnap is fetching patches (typical size 500 bytes) thespeedup can be over a factor of 100.
Not really black magic at all — pipelined HTTP is somethingwhich everybody should be using.
Unfortunately, shockingly few people do.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 84: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/84.jpg)
Black Magic #6: Pipelined HTTP
Pipelined HTTP can easily speed up fetching small files by anorder of magnitude.
When Portsnap is fetching patches (typical size 500 bytes) thespeedup can be over a factor of 100.
Not really black magic at all — pipelined HTTP is somethingwhich everybody should be using.
Unfortunately, shockingly few people do.I had to write my own command-line pipelined HTTP client aspart of Portsnap because I couldn’t find one anywhere.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 85: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/85.jpg)
Spyware!
PRIVACY NOTICE
As an unavoidable part of its operation, a machine runningportsnap will make its public IP address and the list of files itfetches available to the server from which it fetches updates. Usingthese it may be possible to recognize a machine over an extendedperiod of time, determine when it is updated, and identify whichportions of the FreeBSD ports tree, if any, are being ignored using”REFUSE” directives in portsnap.conf. In addition, the FreeBSDrelease level is transmitted to the server.
Statistical data generated from information collected in thismanner may be published, but only in aggregate and afteranonymizing the individual systems.
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 86: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/86.jpg)
Portsnap usage
7.0
6.2
6.1
6.05.55.41.11.00.9.50.9.40.9.30.9.20.9.10.9
01/04/05 01/10/05 01/04/06 01/10/06 01/04/07
0
10
20
30
Weekly portsnap usage by version
Weeks ofupdatesfetched(×103)
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap
![Page 87: FreeBSD Portsnap - What (it is), Why (it was written), and ... · FreeBSD Portsnap What (it is), Why (it was written), and How (it works) Colin Percival The FreeBSD Project cperciva@FreeBSD.org](https://reader033.vdocuments.mx/reader033/viewer/2022060411/5f1098d07e708231d449e38f/html5/thumbnails/87.jpg)
Portsnap usage on FreeBSD 6.2
6.2.STABLE
6.2.R.p46.2.R.p36.2.R.p26.2.R.p1
6.2.R
6.2.RC2.p16.2.RC26.2.RC1.p26.2.RC1.p16.2.RC16.2.BETA3.p16.2.BETA36.2.BETA26.2.BETA1.p46.2.BETA1.p36.2.BETA1.p26.2.BETA1.p16.2.BETA16.2.PRE
01/10/06 01/12/06 01/02/07 01/04/07
0
2
4
6
8
10
12
Weekly portsnap usage by version
Weeks ofupdatesfetched(×103)
Colin Percival The FreeBSD Project [email protected] FreeBSD Portsnap