Customizeable FTP auto-fetch script (OpenBSD snapshot download as example)

[J65nko]

The native ftp programs of FreeBSD and OpenBSD have a feature called auto-fetching. Using this to get a single file is doable. You type something like thisftp ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/base35.tgzand it will log you in as an anonymous user, change to the directory and get the the file for you. For multiple files it quickly becomes a lot of typingftp ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/MD5 \
ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/base35.tgz \
ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/bsd \
ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/etc35.tgz \
ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/floppy35.fsYou can do that in a script, but it still a lot of work, to change the ftp site for example. The following script should make that easier.

In its current state. the script has been configured to download by ftp an OpenBSD snapshot. After reading the script, you will see that it is simple to adapt to your needs or wishes.
Some tips The shell continuation character "\" only works as advertised, if it is the last character on the line. You can use "cat --e" to check for that. The "-e" option will show the "$" as (Do you see the error here?)xwin="xbase35.tgz \$
xfont35.tgz \ $
xserv35.tgz \$
xshare35.tgz"$
Run the script with "sh -nv" to check for syntax errors
Use "sh -xv" to see the expansion of the variables
"#!/bin/sh
# --- FTP auto-fetch script
# --- get OpenBSD snapshot

server=ftp.openbsd.org
server=openbsd.bay13.net

dir=pub/OpenBSD/snapshots/i386

# --- customizeable file sets

base="MD5 \
base35.tgz \
bsd \
bsd.rd \
comp35.tgz \
etc35.tgz \
floppy35.fs \
man35.tgz \
misc35.tgz "

xwin="xbase35.tgz \
xetc35.tgz \
xfont35.tgz \
xserv35.tgz \
xshare35.tgz"

misc="INSTALL.i386 \
INSTALL.linux \
index.txt"

# --- main function

get() {
ftplist="" # Set to null in case for second invocation
for item in $@ ; do
full_name=ftp://$server/$dir/$item
ftplist="$ftplist $full_name"
done

echo "About to get the following:"
echo "$ftplist"

ftp $ftplist
}

# --- main script

get $base
#get $xwin
#get $misc
For FreeBSD replacing ftp by the fetch program could be an alternative.

And yes, I know there are more feature rich alternatives like curl and wget.
The advantage of this script is that is will run on a basic FreeBSD or OpenBSD install. So you could use it to download the curl or wget package ;)

[J65nko]

get () {

ftplist="" # Set to null in case for second invocation

for item in $@ ; do
full_name=ftp://$server/$dir/$item
ftplist="$ftplist $full_name"
done

echo "About to get the following:"
echo $ftplist | tr ' ' '\n'
echo "Press Y to continue, any other to quit"
read answer
case $answer in
[Yy] ) ftp $ftplist
;;
* ) echo "Not doing ftp:"
echo $ftplist | tr ' ' '\n'
;;
esac
}
An slighty interactive version of the auto-fetch script. The changes are in blue ;)

[J65nko]

A more user-friendly version with the following features: Interactive ftp site selection
Interactive multiple install sets selection
Automatic OS detection (FreeBSD and OpenBSD) to select either the FBSD 'fetch' or OBSD's 'ftp'.


Technical notes For shell programmers

The script uses 'eval' to create shell variables in the format "server_1", "server_2" etc.
In an effort to keep the global variable namespace as lean and mean as possible, these variables have been defined "local" within their function.

Yes, bash and ksh support arrays, but I have tried to stick to a portable standard "sh" implementation.


The functions "select_server()" and "select_sets ()" are passed a string. Again using "eval" the function creates a shell variable, having this string as it's name.
This way the function is not "tied" to a single global sh variable to return a value.

#!/bin/sh
# --- get OpenBSD snapshot
# j65nko bsdforums.org
# 2005-10-15 : interactive ftp site selection
# 2005-10-17 : interactive fileset selection
# 2006-04-18 : show contents of file sets

# --- customizeable ftp sites

sites="ftp.openbsd.org \
ftp.calyx.nl \
ftp.nluug.nl \
ftp.stacken.kth.se \
ftp.plig.org \
sunsite.cnlab-switch.ch"

# ----- standard ftp directory
dir=pub/OpenBSD/snapshots/i386

# --- customizeable file sets

VERSION=39

base="MD5 \
base${VERSION}.tgz \
bsd \
bsd.rd \
bsd.mp \
comp${VERSION}.tgz \
etc${VERSION}.tgz \
floppy${VERSION}.fs \
man${VERSION}.tgz \
misc${VERSION}.tgz"

xwin="xbase${VERSION}.tgz \
xfont${VERSION}.tgz \
xserv${VERSION}.tgz \
xetc${VERSION}.tgz \
xshare${VERSION}.tgz"

iso="cd${VERSION}.iso"

scsi="floppyB${VERSION}.fs"

mini="MD5 \
base${VERSION}.tgz \
bsd \
bsd.rd \
etc${VERSION}.tgz \
floppy${VERSION}.fs \
"

man="man${VERSION}.tgz"

comp="comp${VERSION}.tgz"

mp="bsd.mp"

pxeboot="pxeboot"

allsets="base xwin mini iso scsi man mp comp pxeboot"

# ========================== end of customizeable section ===========
# --- FTP command
FTPCMD=''

if sysctl kern.ostype | grep -i openbsd >/dev/null 2>&1 ; then
FTPCMD="ftp -4"
elif sysctl kern.ostype | grep -i freebsd >/dev/null 2>&1 ; then
FTPCMD="fetch -4"
else
echo "$0: Sorry, unknown OS. Exiting ..."
exit 10
fi

echo $FTPCMD
#exit

# ---- function to print 50 dashes

dashes() {
echo "--------------------------------------------------"
}

# --- function to select ftpserver

select_server() {
local count name choice
local server_1 server_2 server_3 server_4
local server_5 server_6 server_7 server_8

count=0 # number of servers to choose from

echo ; dashes
for name in ${sites} ; do
count=$((count+1))
echo "${count}) ${name}"
# create shell variable server_1, server_2, server_3 etc
eval "server_${count}=${name}"
done
dashes

echo -n "Select the ftp site by number : "
read choice

# --- check validity of chosen number

if [ ${choice} -gt 0 -a ${choice} -le ${count} ]; then
# $1 (first paramater passed to the function = string "server"
# server=$server_1
eval $1=\$server_${choice}
else
# error code 1
return 1
fi
} # --- end of function


# --- select sets
# parameter $1 is name of variable
# the function will create a shell variable with this name

select_sets () {

local count name nr all
local set_1 set_2 set_3 set_4 set_5
local set_6 set_7 set_8 set_9 set_10
local sets_chosen

count=0

dashes
for name in ${allsets} ; do
count=$((count+1))
#echo -n "${count}) ${name} -["
printf "(%2d) %-15s : " "${count}" "${name}"
eval echo \$${name}
eval "set_${count}=${name}" # create variable: set_1, set_2, set_3 etc
done
dashes

echo -n 'Please enter the number(s) of the filesets separated by a space :'
read all

for nr in ${all} ; do
eval sets_chosen=\"\${sets_chosen} \$set_${nr}\"
#echo ${sets_chosen}
done

# create variable on the fly
eval $1=\${sets_chosen}
#echo "RETURNING these filesets: ${sets_chosen}==="

} # --- end of function

# --- main function

get () {

local ftplist filename answer full_name

for filename in $@ ; do
full_name=ftp://${server}/${dir}/${filename}
ftplist="${ftplist} ${full_name}"
done

echo
echo About to get the following:
dashes
echo ${ftplist} | tr ' ' '\n'
dashes

echo -n "Press Y to fetch, any other to quit "
read answer

case ${answer} in
[Yy] ) ${FTPCMD} ${ftplist}
;;
* ) echo "Skipping ftp of:"
dashes
echo ${ftplist} | tr ' ' '\n'
;;
esac
}

# --- main script

if select_server "server" ; then
echo ; dashes
echo "Selected FTP SERVER : $server"
dashes ; echo
else
echo "$0: No valid selection of ftp server"
exit 1
fi

select_sets "chosen"

echo ; dashes ; echo "Selected fileset\(s\): $chosen" ; dashes ; echo
# exit 1

for fileset in ${chosen} ; do
#echo "====${fileset}======"
eval get \$${fileset}
done

# --- debugging local variables
#set

dashes ; echo "$0: done!"
A transcript of a "get_snapshot" session on a FreeBSD box
Script started on Tue Apr 18 21:43:29 2006
j65nko@plato[~/snap]./get_snapshot

--------------------------------------------------
1) ftp.openbsd.org
2) ftp.calyx.nl
3) ftp.nluug.nl
4) ftp.stacken.kth.se
5) ftp.plig.org
6) sunsite.cnlab-switch.ch
--------------------------------------------------
Select the ftp site by number : 5

--------------------------------------------------
Selected FTP SERVER : ftp.plig.org
--------------------------------------------------

--------------------------------------------------
( 1) base : MD5 base39.tgz bsd bsd.rd bsd.mp comp39.tgz etc39.tgz floppy39.fs man39.tgz misc39.tgz
( 2) xwin : xbase39.tgz xfont39.tgz xserv39.tgz xetc39.tgz xshare39.tgz
( 3) mini : MD5 base39.tgz bsd bsd.rd etc39.tgz floppy39.fs
( 4) iso : cd39.iso
( 5) scsi : floppyB39.fs
( 6) man : man39.tgz
( 7) mp : bsd.mp
( 8) comp : comp39.tgz
( 9) pxeboot : pxeboot
--------------------------------------------------
Please enter the number(s) of the filesets separated by a space :3 7

--------------------------------------------------
Selected fileset\(s\): mini mp
--------------------------------------------------

About to get the following:
--------------------------------------------------
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/MD5
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/base39.tgz
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/bsd
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/bsd.rd
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/etc39.tgz
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/floppy39.fs
--------------------------------------------------
Press Y to fetch, any other to quit y
MD5 100% of 1019 B 48 kBps
base39.tgz 100% of 39 MB 317 kBps 00m00s
bsd 100% of 5348 kB 316 kBps 00m00s
bsd.rd 100% of 4630 kB 317 kBps 00m00s
etc39.tgz 100% of 1126 kB 303 kBps
floppy39.fs 100% of 1440 kB 315 kBps

About to get the following:
--------------------------------------------------
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/bsd.mp
--------------------------------------------------
Press Y to fetch, any other to quit y
bsd.mp 100% of 5391 kB 314 kBps 00m00s
--------------------------------------------------
./get_snapshot: done!

j65nko@plato[~/snap]exit
exit
Script done on Tue Apr 18 21:49:41 2006