rsync - exclude files in directories with specific conditions
I am setting up an rsync based backup from our FreeNAS server to a QNAP server.
The structure looks something like this:
Sequence
├── SHOT01
│ ├── comp
│ │ ├── work
│ | │ └── images
│ | │ ├── SHOT01
│ | │ | └── SHOT01_v001
│ | │ | ├── SHOT01_comp_v001.0001.exr
│ | │ | └── SHOT01_comp_v001.0002.exr
│ | │ └── precomp01
│ | │ └── precomp01.0001.exr
│ | ├── publish
│ | │ └── elements
│ | │ └── SHOT01
│ | │ └── SHOT01_v001
│ | │ ├── SHOT01_comp_v001.0001.exr
│ | │ └── SHOT01_comp_v001.0002.exr
│ ├── cleanup
│ │ ├── work
│ | │ └── images
│ | │ ├── SHOT01
│ | │ | └── SHOT01_v001
│ | │ | ├── SHOT01_cleanup_v001.0001.exr
│ | │ | └── SHOT01_cleanup_v001.0002.exrI would like to back up everything, excluding .exr image-sequences in the comp/work folder only if they are contained within a sub-folder matching the shot name.
In this example rsync should only exclude:
.../comp/work/images/SHOT01/SHOT01_v001/SHOT01_comp_v001.0001.exr
.../comp/work/images/SHOT01/SHOT01_v001/SHOT01_comp_v001.0002.exrMaybe worth noting that I am setting up the rsync task in FreeNAS. In the configuration it has a field called "extra options" where I'm guessing the exclude argument should be pasted.
Not really sure how to do this with rsync, or if it's even possible.
1 Answer
I don't know FreeNAS, but it seems to have FreeBSD below it, and googling seems to suggest you can create cron jobs and write shell or possibly bash scripts, and even perl scripts. So there is hope. However, GUIs are terrible places to do things dynamically, so I suggest you start small as follows.
rsync is not capable of handling complex patterns, so the list of excluded files will have to be created outside it.
In the rsync GUI add the extra option -F. Then, manually, in the directory where you wish to suppress files create a file .rsync-filter containing the exclude option (-), a space, and the glob pattern *.exr if you mean to suppress all files there:
- *.exrrsync -F makes it look for the file .rsync-filter in every directory, and applies any rules there to all sub-directories. If you only mean to suppress some files you can list them line by line:
- SHOT01_comp_v001.0001.exr
- SHOT01_comp_v001.0002.exrThis is sufficient as a first step if you do not have many directories matching your criteria.
The next step is to setup a cron job to run a script that scans your directories for your criteria and creates the .rsync-filter file appropriately. Here a lot depends on what your system provides, so I will just give an example. If you have the find command that supports regular expressions you can use
find comp/work -regextype posix-extended \ -regex '.*/(SHOT[^/]*)/\1_([^/]*)/\1_.*\2\..*\.exr' -printto get a list of all files where the name is <1>...<2>.exr and the directory path is <1>/<1>...<2>/, and <1> begins SHOT. It isn't too clear what pattern you really want to match, so that is just an example. And obviously it needs to be wrapped into a script to create the .rsync-filter files, and put in a cron job.
