Automating OpenBSD snapshot downloads with a .netrc file

[J65nko]

While in http://www.daemonforums.org/showthread.php?t=670, another approach is used, this script uses a .netrc generated on the fly to do the complete download in a single login session.

The .netrc syntax is described in the ftp(1) man page.

Some examples

$ snapget
snapget
Destination directory: Snapshots
./snapget: Please specify one or more sets
The sets are: base x mini pxe floppy man misc iso

Selecting the 'mini' install sets as defined in the snapget script

$ snapget mini
Destination directory: Snapshots
-------------- Created .netrc-----------------------
     1  
     2  machine ftp.nluug.nl login anonymous password thanks@puffy.org
     3  
     4  macdef init
     5  prompt off
     6  epsv4 off
     7  preserve on
     8  get /pub/OpenBSD/snapshots/i386/SHA256 Snapshots/SHA256
     9  get /pub/OpenBSD/snapshots/i386/base46.tgz Snapshots/base46.tgz
    10  get /pub/OpenBSD/snapshots/i386/bsd Snapshots/bsd
    11  get /pub/OpenBSD/snapshots/i386/bsd.rd Snapshots/bsd.rd
    12  get /pub/OpenBSD/snapshots/i386/etc46.tgz Snapshots/etc46.tgz
    13  quit
    14  
------------------------------------------------------
Do you want to start 'ftp' with this '.netrc' ? (Y/N)
n
./snapget: I take this for a 'NO', aborting ...

The second line defines the ftp site, login name and password.

After an empty line a macro definition named 'init' is created. It contains some ftp commands. See the ftp man page for the details. Line 8 to 12 perform a ftp 'get' of the minimal number of filesets to install OpenBSD. By leaving out 'bsd.rd' it could be still be made smaller.

The 'init' macro will be executed immediately after the ftp server has authorized the ftp login. Because the macro ends with a 'quit' command the ftp session will terminate automatically.

$  snapget pxe flopp
Destination directory: Snapshots
./snapget : undefined set flopp
./snapget : aborting

The name of the file sets will be checked, and if an unknown file set is detected, the script will terminate with a message informing the user.

Normally a ".netrc' file is located in the home directory. Actually ftp uses the environment variable HOME to decide in which directory to look.

$ env | grep HOME
HOME=/home/j65nko

In order not to overwrite a possible existing .netrc the script stores the .netrc file in the directory defined in the ${DESTDIR{ variable.

By using env(1) utility, the value of HOME is changed into $DESTDIR}, and thus coaches ftp to locate and execute the right .netrc:

env HOME=${DESTDIR} ftp -4 ${SITE} 2>&1 | tee ${DESTDIR}/Logfile

A very simple illustration of using 'env', is to print the current date and time from another time zone by setting the TZ environment variable

# env  TZ=CET date    
Thu Jan  7 03:24:55 CET 2010

$env  TZ=EST date   
Wed Jan  6 21:25:09 EST 2010

A last example :

$snapget mini floppy
Destination directory: Snapshots
-------------- Created .netrc-----------------------
     1  
     2  machine ftp.nluug.nl login anonymous password thanks@puffy.org
     3  
     4  macdef init
     5  prompt off
     6  epsv4 off
     7  preserve on
     8  get /pub/OpenBSD/snapshots/i386/SHA256 Snapshots/SHA256
     9  get /pub/OpenBSD/snapshots/i386/base46.tgz Snapshots/base46.tgz
    10  get /pub/OpenBSD/snapshots/i386/bsd Snapshots/bsd
    11  get /pub/OpenBSD/snapshots/i386/bsd.rd Snapshots/bsd.rd
    12  get /pub/OpenBSD/snapshots/i386/etc46.tgz Snapshots/etc46.tgz
    13  get /pub/OpenBSD/snapshots/i386/floppy46.fs Snapshots/floppy46.fs
    14  get /pub/OpenBSD/snapshots/i386/floppyB46.fs Snapshots/floppyB46.fs
    15  get /pub/OpenBSD/snapshots/i386/floppyC46.fs Snapshots/floppyC46.fs
    16  quit
    17  
------------------------------------------------------
Do you want to start 'ftp' with this '.netrc' ? (Y/N)
y
Trying 192.87.102.43...
Connected to ftp.nluug.nl.
220-Welcome to the FTP archive of SURFnet BV and
220-The Netherlands Unix Users Group (NLUUG).
[snip]
331 Please specify the password.
230 Login successful.
prompt off
Interactive mode off.
epsv4 off
EPSV/EPRT on IPv4 off.
preserve on
Preserve modification times on.
get /pub/OpenBSD/snapshots/i386/SHA256 Snapshots/SHA256
local: Snapshots/SHA256 remote: /pub/OpenBSD/snapshots/i386/SHA256
227 Entering Passive Mode (192,87,102,43,199,230)
150 Opening BINARY mode data connection for /pub/OpenBSD/snapshots/i386/SHA256 (2162 bytes).
226 File send OK.
2162 bytes received in 0.19 seconds (10.90 KB/s)
get /pub/OpenBSD/snapshots/i386/base46.tgz Snapshots/base46.tgz
local: Snapshots/base46.tgz remote: /pub/OpenBSD/snapshots/i386/base46.tgz
227 Entering Passive Mode (192,87,102,43,239,73)
150 Opening BINARY mode data connection for /pub/OpenBSD/snapshots/i386/base46.tgz (50281346 bytes).
226 File send OK.
50281346 bytes received in 64.90 seconds (756.64 KB/s)
[snip]
get /pub/OpenBSD/snapshots/i386/floppyC46.fs Snapshots/floppyC46.fs
local: Snapshots/floppyC46.fs remote: /pub/OpenBSD/snapshots/i386/floppyC46.fs
227 Entering Passive Mode (192,87,102,43,222,187)
150 Opening BINARY mode data connection for /pub/OpenBSD/snapshots/i386/floppyC46.fs (1474560 bytes).
226 File send OK.
1474560 bytes received in 1.99 seconds (723.88 KB/s)
quit
221 Goodbye.

The retrieved files:

$ ls -l Snapshots/ 
total 134992
-rw-r--r--  1 j65nko  j65nko      3599 Jan  7 03:42 Logfile
-rw-r--r--  1 j65nko  j65nko      2162 Jan  5 22:59 SHA256
-rw-r--r--  1 j65nko  j65nko  50281346 Jan  5 22:58 base46.tgz
-rw-r--r--  1 j65nko  j65nko   7505333 Jan  5 22:58 bsd
-rw-r--r--  1 j65nko  j65nko   6239716 Jan  5 22:58 bsd.rd
-rw-r--r--  1 j65nko  j65nko    522035 Jan  5 22:58 etc46.tgz
-rw-r--r--  1 j65nko  j65nko   1474560 Jan  5 22:58 floppy46.fs
-rw-r--r--  1 j65nko  j65nko   1474560 Jan  5 22:58 floppyB46.fs
-rw-r--r--  1 j65nko  j65nko   1474560 Jan  5 22:58 floppyC46.fs

[J65nko]

#!/bin/sh
# $Id: snapget,v 1.4 2010/01/07 01:42:05 j65nko Exp $
# get snapshot files with a on the fly generated '.netrc' for ftp(1)


DESTDIR="Snapshots"
if [ -d ${DESTDIR} ] ; then
   rm -f ${DESTDIR}/*
else
   mkdir -p ${DESTDIR} || exit 1
fi

# -- destination directory 
# DESTDIR="SNAP$(date -u "+%Y-%m-%d_%H:%M_UTC")"
# mkdir ${DESTDIR} || exit 1

echo "Destination directory: ${DESTDIR}"

# variables  used in 'netrc' template
# ------------------------------------------------------------------
ARCH='i386'		# architecture name used as ftp dir
#ARCH='amd64'		# architecture name used as ftp dir

RN=46			# used in "base${RN}.tgz" etc
#VERSION='4.6'		# ftp dir for releases
VERSION='snapshots'	# ftp dir for snapshots

DIR="/pub/OpenBSD/${VERSION}/${ARCH}"  

SITE='ftp.calyx.nl'
SITE='ftp.eu.openbsd.org'
SITE='ftp.esat.net'
SITE='ftp.openbsd.org'
SITE='ftp.wu-wien.ac.at'
SITE='ftp.eu.openbsd.org'
SITE='ftp.nluug.nl'

# customizable file sets 
# if you add or remove sets, don't forget the update the ALLSETS variable 
# ------------------------------------------------------------------

#SETS_START
# ------------------------------------------------------------------
base="
INSTALL.${ARCH}
SHA256
base${RN}.tgz
bsd
bsd.mp
bsd.rd
comp${RN}.tgz
etc${RN}.tgz
man${RN}.tgz
misc${RN}.tgz
"

x="
xbase${RN}.tgz
xetc${RN}.tgz
xfont${RN}.tgz
xserv${RN}.tgz
xshare${RN}.tgz
"

mini="
SHA256
base${RN}.tgz
bsd
bsd.rd
etc${RN}.tgz
"
pxe="
pxeboot
"

floppy="
floppy${RN}.fs
floppyB${RN}.fs
floppyC${RN}.fs
"
man="
man${RN}.tgz
"

misc="
misc${RN}.tgz
"
iso="
install${RN}.iso
"
# ------------------------------------------------------------------
#SETS_END

# ------------------------------------------------------------------
ALLSETS="base x mini pxe floppy man misc iso"
# ------------------------------------------------------------------

if [ $# -eq 0 ] ; then
    echo "$0: Please specify one or more sets"
    echo "The sets are: ${ALLSETS}"
    exit 1
fi

# -- accumulate all sets in FILES
for SET in "$@" ; do
     eval CONTENTS=\"\$${SET}\"
     if [ "x${CONTENTS}" = "x" ] ; then
         echo "$0 : undefined set ${SET}"
         echo "$0 : aborting"
         exit 1
     fi 
    FILES="${FILES} ${CONTENTS} "
done

# echo ${FILES} | tr '[:blank:]' '\n'

# --- 'here document' as '.netrc' template

cat <<-TEMPLATE >${DESTDIR}/.netrc

machine ${SITE} login anonymous password thanks@puffy.org

macdef init
prompt off
epsv4 off
preserve on
$( for THIS in ${FILES} ; do 
     echo "get ${DIR}/${THIS} ${DESTDIR}/${THIS}" 
   done
)
quit

TEMPLATE

# -- An empty line is the terminator for a macdef macro
# -- So KEEP EMPTY LINE between 'quit' and here document marker 'TEMPLATE'
# -- to prevent 'Macro definition missing null line terminator' messages
# -- from 'ftp'

echo -------------- Created '.netrc'-----------------------
cat -n  $DESTDIR/.netrc
echo ------------------------------------------------------
echo "Do you want to start 'ftp' with this '.netrc' ? (Y/N)" ; read answer

case "$answer" in
[Yy] )		;;

*    )		echo "$0: I take this for a 'NO', aborting ..."
    		exit 2
		;;
esac

# -- ftp uses HOME to locate '.netrc'
env HOME=${DESTDIR} ftp -4 ${SITE} 2>&1 | tee ${DESTDIR}/Logfile

# -- EOF

At this moment the script does not prompt for the ftp site, or architecture. You will have to edit the variables in the script itself.

Because the forum software insists on a file extension, you will have to rename the downloaded script to remove the '.txt'

$ mv snapget.txt snapget

$ chmod u+x snapget