Find command (Linux options?)

[deadeyes]

Hi all,

I am trying to do the following:
I want to have a list of all files in a directory and its subdirectories with a md5 hash in its first column.

This is to get a list off all mp3s which I have double.

I though to use
find /musiclocation -printf %f
or something similar (which does not show the path)

But it seems that the command syntax is different.

Is there another way I can accomplish this? or is it possible to install a port that has the linux find command?

Maybe someone knows another approach to this.

Thanks in advance.

[jggimi]

Perhaps this will be sufficient?

$ mkdir test
$ cd test
$ touch a b c
$ find . -type f -exec md5 {} > /tmp/results \;
$ cat /tmp/results
MD5 (./a) = d41d8cd98f00b204e9800998ecf8427e
MD5 (./b) = d41d8cd98f00b204e9800998ecf8427e
MD5 (./c) = d41d8cd98f00b204e9800998ecf8427e
$

[deadeyes]

Originally Posted by jggimi

Perhaps this will be sufficient?

$ mkdir test
$ cd test
$ touch a b c
$ find . -type f -exec md5 {} > /tmp/results \;
$ cat /tmp/results
MD5 (./a) = d41d8cd98f00b204e9800998ecf8427e
MD5 (./b) = d41d8cd98f00b204e9800998ecf8427e
MD5 (./c) = d41d8cd98f00b204e9800998ecf8427e
$

This shows the (relative) path... and that is not what I want.

Thanks for your time though.

[jggimi]

Originally Posted by deadeyes

This shows the (relative) path..

Because that particular find(1) command used a relative directory: "find .".

If you need a full path, use find with /path/to/your/files instead. If you don't want any path information, run the output through awk or perl or python and remove it.

[vermaden]

@deadeyes

~ % cat find.sh
#! /bin/sh

find ${1} -type f \
  | while read LINE
    do
      MD5=$( md5 -q ${LINE} )
      FILE=$( basename ${LINE} )
      echo "${MD5} ${FILE}"
    done

~ % ./find.sh misc/sys/FreeBSD/
3d3991e7f828003c3abe0c7f1e48011a loader.conf
66ac143dc9e53b22df4fd77ae6fa01fa make.conf
01963f0ccdd17822c5ee210bba5df10f xorg.conf
085616e4fc95e7af3771deceab5171c3 sysctl.conf
(...)
~ % 

[deadeyes]

Originally Posted by jggimi

Because that particular find(1) command used a relative directory: "find .".

If you need a full path, use find with /path/to/your/files instead. If you don't want any path information, run the output through awk or perl or python and remove it.

It is just that the linux find has this option available.
So scripting is not necessary.
Python, Perl does make more dependencies.

vermaden: nice piece of shell script (and purely bash ) didn't know the command basename

Thanks for your efforts guys!

[BSDfan666]

Originally Posted by deadeyes

It is just that the linux find has this option available.
So scripting is not necessary.
Python, Perl does make more dependencies.

vermaden: nice piece of shell script (and purely bash ) didn't know the command basename

Thanks for your efforts guys!

No.. it's purely bourne-compatible.

[vermaden]

Originally Posted by deadeyes

It is just that the linux find has this option available.
So scripting is not necessary.

# pkg_add -r findutils
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.2-release/Latest/findutils.tbz... Done.
~ % pkg_info -L -x findutils | grep bin
/usr/local/bin/gfind
/usr/local/bin/goldfind
/usr/local/bin/glocate
/usr/local/bin/gupdatedb
/usr/local/bin/gxargs
~ % rehash
~ % gfind --version
find (GNU findutils) 4.4.0
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Eric B. Decker, James Youngman, and Kevin Dalley.
Built using GNU gnulib version e5573b1bad88bfabcda181b9e0125fb0c52b7d3b
Features enabled: D_TYPE O_NOFOLLOW(enabled) LEAF_OPTIMISATION FTS() CBO(level=0) 
~ %

Originally Posted by deadeyes

(and purely bash )

Which part of it works only in bash(1) while it works without any problem on POSIX sh(1) (first line indicates parser as /bin/sh ... not bash(1))

[s0xxx]

Hm, I don't know what's wrong with suggestion jggimy made? Because the name doesn't come before the hash? Then just:

$ find . -type f -exec md5 {} + | awk '{ print $4, $2 }'

IMHO, path to file is usefull if you got several directories containing mp3's.
Also, using awk you could filter the duplicates, using something like this:

$ find . -type f -exec md5 {} + | awk -f find.awk


find.awk:

a[$4]++ { if(match($2, /[^\/]*\)/)) {
           pattern=substr($2, RSTART, RLENGTH-1);
               printf("%s\t%s\n", $4, pattern);
           }
}

I'm sorry for not providing more command output or examples, I'm doing this on a Windows machine.

[s0xxx]

But if you have spaces in names of songs that may disturb the regex, so maybe this is better:

find.awk:


a[$4]++ { 
        regex="\\/[^\\/]*\\)"
        if(match($2, regex)) {
           pattern=substr($2, RSTART+1, RLENGTH-2);
             printf("%s\t%s\n", $4, pattern);
          }
}

[BSDKaffee]

Originally Posted by deadeyes

This is to get a list off all mp3s which I have double.

If you just want to check and see if you have exact duplicate files, I would suggest using something like sysutils/duff. It is very fast, easy to use, and written entirely in C with no other dependencies.

[deadeyes]

I knew there were some tools already for doing this but I wanted to write a script myself (as an exercise)

Thanks for you help guys. Eventually the basename command would do the trick but I kept it out of the script as it would not very usefull if you don't really know where those duplicate files are.