Customizeable FTP auto-fetch script (OpenBSD snapshot download as example)

[J65nko]

The native ftp programs of FreeBSD and OpenBSD have a feature called auto-fetching. Using this to get a single file is doable. You type something like this

ftp ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/base35.tgz

and it will log you in as an anonymous user, change to the directory and get the the file for you. For multiple files it quickly becomes a lot of typing

ftp  ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/MD5 \
     ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/base35.tgz \
     ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/bsd \
     ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/etc35.tgz \
     ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/i386/floppy35.fs

You can do that in a script, but it still a lot of work, to change the ftp site for example. The following script should make that easier.

In its current state. the script has been configured to download by ftp an OpenBSD snapshot. After reading the script, you will see that it is simple to adapt to your needs or wishes.
Some tips

• The shell continuation character "\" only works as advertised, if it is the last character on the line. You can use "cat --e" to check for that. The "-e" option will show the "$" as (Do you see the error here?)

  xwin="xbase35.tgz \$
        xfont35.tgz \ $
        xserv35.tgz \$
        xshare35.tgz"$

• Run the script with "sh -nv" to check for syntax errors
• Use "sh -xv" to see the expansion of the variables

"

#!/bin/sh
# --- FTP auto-fetch script
# --- get OpenBSD snapshot

server=ftp.openbsd.org
server=openbsd.bay13.net

dir=pub/OpenBSD/snapshots/i386

# --- customizeable file sets

base="MD5 \
        base35.tgz \
        bsd \
        bsd.rd \
        comp35.tgz \
        etc35.tgz \
        floppy35.fs \
        man35.tgz \
        misc35.tgz "

xwin="xbase35.tgz \
      xetc35.tgz \
      xfont35.tgz \
      xserv35.tgz \
      xshare35.tgz"

misc="INSTALL.i386 \
      INSTALL.linux \
      index.txt"

# --- main function

get() {
    ftplist=""  # Set to null in case for second invocation
    for item in $@ ; do
        full_name=ftp://$server/$dir/$item
        ftplist="$ftplist $full_name"
    done

    echo "About to get the following:"
    echo "$ftplist"

    ftp $ftplist
}

# --- main script

get $base
#get $xwin
#get $misc

For FreeBSD replacing ftp by the fetch program could be an alternative.

And yes, I know there are more feature rich alternatives like curl and wget.
The advantage of this script is that is will run on a basic FreeBSD or OpenBSD install. So you could use it to download the curl or wget package

[J65nko]

get () {

    ftplist=""  # Set to null in case for second invocation

    for item in $@ ; do
        full_name=ftp://$server/$dir/$item
        ftplist="$ftplist $full_name"
    done

    echo "About to get the following:"
    echo $ftplist | tr ' ' '\n'
    echo "Press Y to continue, any other to quit"
    read answer
    case $answer in
    [Yy] )      ftp $ftplist
                ;;
    * )         echo "Not doing ftp:"
                echo $ftplist |  tr ' ' '\n'
                ;;
    esac
}

An slighty interactive version of the auto-fetch script. The changes are in blue

[J65nko]

A more user-friendly version with the following features:

• Interactive ftp site selection
• Interactive multiple install sets selection
• Automatic OS detection (FreeBSD and OpenBSD) to select either the FBSD 'fetch' or OBSD's 'ftp'.

Technical notes For shell programmers

• The script uses 'eval' to create shell variables in the format "server_1", "server_2" etc.
  In an effort to keep the global variable namespace as lean and mean as possible, these variables have been defined "local" within their function.
  
  Yes, bash and ksh support arrays, but I have tried to stick to a portable standard "sh" implementation.
  
  
• The functions "select_server()" and "select_sets ()" are passed a string. Again using "eval" the function creates a shell variable, having this string as it's name.
  This way the function is not "tied" to a single global sh variable to return a value.

#!/bin/sh
# --- get OpenBSD snapshot
# j65nko bsdforums.org
# 2005-10-15 : interactive ftp site selection 
# 2005-10-17 : interactive fileset selection
# 2006-04-18 : show contents of file sets

# --- customizeable ftp sites 

sites="ftp.openbsd.org \
	ftp.calyx.nl \
	ftp.nluug.nl \
	ftp.stacken.kth.se \
	ftp.plig.org \
	sunsite.cnlab-switch.ch"

#  ----- standard ftp directory
dir=pub/OpenBSD/snapshots/i386

# --- customizeable file sets

VERSION=39

base="MD5 \
	base${VERSION}.tgz \
	bsd \
	bsd.rd \
	bsd.mp \
	comp${VERSION}.tgz \
	etc${VERSION}.tgz \
	floppy${VERSION}.fs \
	man${VERSION}.tgz \
	misc${VERSION}.tgz"

xwin="xbase${VERSION}.tgz \
      xfont${VERSION}.tgz \
      xserv${VERSION}.tgz \
      xetc${VERSION}.tgz \
      xshare${VERSION}.tgz"

iso="cd${VERSION}.iso"

scsi="floppyB${VERSION}.fs"

mini="MD5 \
	base${VERSION}.tgz \
	bsd \
	bsd.rd \
	etc${VERSION}.tgz \
	floppy${VERSION}.fs \
"

man="man${VERSION}.tgz"

comp="comp${VERSION}.tgz"

mp="bsd.mp"

pxeboot="pxeboot"

allsets="base xwin mini iso scsi man mp comp pxeboot"

# ========================== end of customizeable section ===========
# --- FTP command
FTPCMD=''

if sysctl kern.ostype | grep -i openbsd >/dev/null 2>&1 ; then
    FTPCMD="ftp -4"
elif sysctl kern.ostype | grep -i freebsd >/dev/null 2>&1 ; then
    FTPCMD="fetch -4"
else
    echo "$0: Sorry, unknown OS. Exiting ..." 
    exit 10
fi
 
echo $FTPCMD
#exit

# ---- function to print 50 dashes 

dashes() {
    echo "--------------------------------------------------"
}

# --- function to select ftpserver
 
select_server() {
    local count name choice
    local server_1 server_2 server_3 server_4
    local server_5 server_6 server_7 server_8
    
    count=0		# number of servers to choose from

    echo  ; dashes
    for name in ${sites} ; do
        count=$((count+1))  
        echo "${count})  ${name}"
        # create shell variable server_1, server_2, server_3 etc
        eval "server_${count}=${name}" 
    done 
    dashes
    
    echo  -n "Select the ftp site by number : "
    read choice 
    
    # --- check validity of chosen number
    
    if [ ${choice} -gt 0 -a ${choice} -le ${count}  ]; then
        # $1 (first paramater passed to the function = string "server"
        # server=$server_1 
        eval $1=\$server_${choice}
    else
        # error code 1
        return 1
    fi 
} # --- end of function


# --- select sets
# parameter $1 is name of variable 
# the function will create a shell variable with this name 

select_sets () {

    local count name nr all
    local set_1 set_2 set_3 set_4 set_5
    local set_6 set_7 set_8 set_9 set_10
    local sets_chosen

    count=0

    dashes
    for name in ${allsets} ; do
        count=$((count+1))
        #echo -n "${count}) ${name}    -["
        printf "(%2d) %-15s : " "${count}" "${name}"
        eval echo  \$${name}
        eval "set_${count}=${name}" # create variable: set_1, set_2, set_3 etc
    done
    dashes

    echo -n 'Please enter the number(s) of the filesets separated by a space :'
    read all

    for nr in ${all} ; do
        eval sets_chosen=\"\${sets_chosen} \$set_${nr}\"
        #echo ${sets_chosen}
    done

    # create variable on the fly
    eval $1=\${sets_chosen}
    #echo "RETURNING these filesets: ${sets_chosen}==="

} # --- end of function

# --- main function

get () {

    local ftplist filename answer full_name

    for filename in $@ ; do
        full_name=ftp://${server}/${dir}/${filename}
        ftplist="${ftplist} ${full_name}"
    done

    echo
    echo About to get the following:
    dashes
    echo ${ftplist} | tr ' ' '\n'
    dashes 
    
    echo -n "Press Y to fetch, any other to quit "
    read answer

    case ${answer} in
    [Yy] )	${FTPCMD} ${ftplist}
		;;
    * )		echo "Skipping ftp of:"
                dashes 
		echo ${ftplist} |  tr ' ' '\n'
		;;
    esac 
}

# --- main script

if select_server "server" ; then
    echo ; dashes
    echo "Selected FTP SERVER : $server"
    dashes ; echo
else
    echo "$0: No valid selection of ftp server"
    exit 1
fi

select_sets "chosen"

echo ; dashes ; echo "Selected fileset\(s\): $chosen" ; dashes ; echo
# exit 1

for fileset in ${chosen} ; do
    #echo "====${fileset}======" 
    eval get \$${fileset}
done

# --- debugging local variables
#set

dashes ; echo "$0: done!"

A transcript of a "get_snapshot" session on a FreeBSD box

Script started on Tue Apr 18 21:43:29 2006
j65nko@plato[~/snap]./get_snapshot

--------------------------------------------------
1)  ftp.openbsd.org
2)  ftp.calyx.nl
3)  ftp.nluug.nl
4)  ftp.stacken.kth.se
5)  ftp.plig.org
6)  sunsite.cnlab-switch.ch
--------------------------------------------------
Select the ftp site by number : 5

--------------------------------------------------
Selected FTP SERVER : ftp.plig.org
--------------------------------------------------

--------------------------------------------------
( 1) base            : MD5 base39.tgz bsd bsd.rd bsd.mp comp39.tgz etc39.tgz floppy39.fs man39.tgz misc39.tgz
( 2) xwin            : xbase39.tgz xfont39.tgz xserv39.tgz xetc39.tgz xshare39.tgz
( 3) mini            : MD5 base39.tgz bsd bsd.rd etc39.tgz floppy39.fs
( 4) iso             : cd39.iso
( 5) scsi            : floppyB39.fs
( 6) man             : man39.tgz
( 7) mp              : bsd.mp
( 8) comp            : comp39.tgz
( 9) pxeboot         : pxeboot
--------------------------------------------------
Please enter the number(s) of the filesets separated by a space :3 7

--------------------------------------------------
Selected fileset\(s\):  mini mp
--------------------------------------------------

About to get the following:
--------------------------------------------------
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/MD5
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/base39.tgz
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/bsd
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/bsd.rd
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/etc39.tgz
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/floppy39.fs
--------------------------------------------------
Press Y to fetch, any other to quit y
MD5                                           100% of 1019  B   48 kBps
base39.tgz                                    100% of   39 MB  317 kBps 00m00s
bsd                                           100% of 5348 kB  316 kBps 00m00s
bsd.rd                                        100% of 4630 kB  317 kBps 00m00s
etc39.tgz                                     100% of 1126 kB  303 kBps
floppy39.fs                                   100% of 1440 kB  315 kBps

About to get the following:
--------------------------------------------------
ftp://ftp.plig.org/pub/OpenBSD/snapshots/i386/bsd.mp
--------------------------------------------------
Press Y to fetch, any other to quit y
bsd.mp                                        100% of 5391 kB  314 kBps 00m00s
--------------------------------------------------
./get_snapshot: done!

j65nko@plato[~/snap]exit
exit
Script done on Tue Apr 18 21:49:41 2006