Examples of async exclude/include patterns

IN THIS ARTICLE:

Description

Aspera Sync allows you to synchronize files between a local and remote system by performing the necessary transfers, deletions, renames or moves to match up the endpoints.

In some cases you may want to synchronize certain files within a directory but not others, in which case you can use include and exclude patterns to target the files you care about.

These patterns follow the standard globbing rules used in Unix-like environments (not regular expression rules). The globbing rules are used to match paths on your system so async knows which paths to synchronize and which to skip. For details on these rules, see the product documentation for Aspera Sync.

The following examples assume you are familiar with basic async commands and globbing rules.

Usage

In your async command, include and exclude patterns are specified with the following options:

  • --include "pattern"
  • --exclude "pattern"
  • --include_from=file
  • --exclude_from=file


You can either specify the patterns directly in the command, as with the first two options, or you can reference a file with a list of patterns to apply as with the last two options. Quotes are required around the pattern when using a wildcard (such as * or ?).

Note: Order is important! Patterns specified earlier trump patterns specified later. For example, if you define an exclude pattern for the path /Documents/Project/ before an include pattern for the path /Documents/Project/Data, the Data folder will not be included because you’ve told async to skip anything in the /Documents/Project/ path.

Examples

The following will only demonstrate the exclude and include options and values of an async command. These options come at the end of async commands, as such:

async -N My_Files -d /Docs/ -r user@server:/My_Docs -w passwd --include pattern --exclude pattern

1. A simple and common example is the need to exclude any files that end with a certain extension.

Say you have several folders filled with images that you want to sync to a remote server, but you don’t want any JPEG files to be considered.

You can easily accomplish this with the following exclude:

--exclude "*.jpg"

If you decide you also don’t care for PNG files, you can simply add another exclude:

--exclude "*.jpg" --exclude "*.png"

This will match all files ending in .jpg and .png in the sync directory and any of its sub-directories.

2. Another common example is the need to exclude files in a specific directory or path.

Following with the previous example, say you actually decide to sync all your image files, and you only want to exclude JPEGs in Holiday_Party_2014, which is a sub-directory of your sync directory My_Files. To do this you need to specify the path of Holiday_Party_2014 in relation to My_Files, by including a forward slash (/) in the beginning.

You can do this with:

--exclude "/Holiday_Party_2014/*.jpg" 

What if Holiday_Party_2014 contains sub-directories that may also contain JPEG files? To skip any JPEG file that may exist within Holiday_Party_2014 or any of its sub-directories, you can use the following:

--exclude "/Holiday_Party_2014/**/*.jpg"
Tip: The globstar (**) works as a recursive matching mechanism that matches all sub-directories at any level.

In this case the globstar allows us to match /Holiday_Party_2014/games/pic1.jpg, /Holiday_Party_2014/games/winners/jane.jpg and so on.

3. Sometimes you need to exclude a particular directory altogether.

Deciding you would rather skip the directory Holiday_Party_2014 completely, you use the following command:

--exclude "/Holiday_Party_2014/"
Note: To exclude a directory, you must have a forward slash (/) at the end of its name. Patterns ending in the * wildcard match both directories and files. Any pattern not ending in a slash / or * matches files only.

4. At times you may need to include part of a specific directory, but exclude everything else.

Say you have a directory named All_Film_Projects and you want to sync files at any level ending with the word script, but exclude all other files.

You will need a combination of an include and an exclude:

--include "/All_Film_Projects/**/*script" --exclude "/**"
Tip: The pattern /** works to exclude all files and directories at all levels

Notice the order here, in which we specify the paths to include first, and then all other remaining paths are excluded. If we put the exclude rule first here, all files would be skipped despite the presence of the include rule.

5. It may be the case that you need to target files only within a certain level of your directory structure.

For example, say you have a structure like the following:

/
-read.txt #level 1

-A/
---file1 #level 2
---AA/
-----file.mp3 #level 3
---AB/
-----file.txt #level 3

-B/
---file.mp3 #level 2
---BB/
-----file.gif #level 3

For each folder (A, B, C and so on…) you want to sync only the files that are on “level 3.” In other words, you only want files immediately under the second sub-directory.

You can do this with the following:

exclude "/*/*/*/" --include "/*/*/*" --exclude "/**"

The first exclude pattern skips all folders on level 3. The include pattern includes all files on level 3. Finally, the last exclude pattern skips everything else.

6. When you have many exclude and include patterns, it is a good idea to use a file to define them so they're easier to edit and manage.

The patterns defined in files are applied in order line by line. Here is an example of an exclude file:

*.jpg
*.png
/Photos/**

You can also define includes within an exclude file, by starting the pattern with a +. For example, we can add an include rule into our exclude file as such:

*.jpg
*.png
+ /Photos/*.txt
/Photos/**

In contrast, in an include file the include patterns are defined line by line as normal. To add an exclude pattern to it, precede the pattern with a -:

- *.jpg
- *.png
/Photos/*.txt
- /Photos/**
Note: Line ending are important in exclude/include files. Unix-like systems should have line feed endings (\n) and Windows systems should have carriage returns (\r\n).
Have more questions? Submit a request

0 Comments

Article is closed for comments.
Powered by Zendesk