Using file manifests for automatic distribution

by Serban Simu
 

This is a robust solution for distributing single files (or directories) to a medium/large population of remote locations (called "partners"). It is initiated by a "script", in this example, but can of course be connected to the end of an automated process. The files to be distributed are called "titles".

sol-manifest-distro.png

The solution involves the following components:

  • A distribution server
  • The Aspera Console for monitoring and notification
  • Optionally a "pick-up" server, in case not all remote locations are set up to be pushed files to
  • At the remote locations ("partners"), Aspera Point to Point or Enterprise Server for those locations we push files to, and Aspera Client or Aspera Connect for those locations that pick up their servers from the "pick up" server (of course Aspera Enterprise Server or Point to Point will work as well).

This solution uses Unix systems (for example Linux, Solaris, BSD), but it can be equally applied to Windows systems.

How it works

The source files must be placed in a directory accessible to the distribution computer (for example a local or a network mounted file system). The distribution script is invoked with the following arguments: an ID of the transaction (used later for reporting and auditing), the path to the file to be distributed and a list of distribution end-points ("partners"). The script simply creates a manifest file for each remote "partner", and places it in the "manifest directory". The Aspera Forwarder runs every minute as a cron job, and looks in the manifest directories for new jobs. It then transfers the files listed in the manifest to the "partner" locations. Upon completion, it moves the manifest files to a "done" directory and optionally removes the file itself.

Setting up the distribution server

  • Install the following Aspera software packages:
    • Aspera Enterprise Server (or in some simpler cases Aspera Point to Point)
    • Aspera forwarder (part of the automation tools toolkit)
  • Set up the source directory: the files must be accessible to the distribution server. In this example, we are using a directory called /titles.
    • Create a directory called /titles and give write permissions to the user running the distribution script
  • Set up a local user called asperamanifest and set its docroot to the top of the source directory (for example /titles)
  • Set up a new "partner" to push files to
  • Create a manifest directory for this partner:/partners/partner_1/todo and/manifests/partner_1/done
  • Create an Aspera Forwarder configuration file, and place it under /partners/partner_1/partner_1.conf. This is where we specify the remote location by IP address or name and login (username and path to private key, or password)
  • Example partner_1.conffile:
    manifest_forward="/partners/partner_1/todo"
    manifest_archive="/partners/partner_1/done" forward_action='/opt/aspera/asperaforward/ascp-manifest.pl -m MANIFESTFILE -- -k2 -QTq -l 100M -i /partner/partner1.ppk MANIFEST acmefilm@partner_1.com:'
  • Create a crontab entry for this forwarder:
    * * * * * /opt/aspera/asperaforward/asperaforward -f /partners/partner_1/partner_1.conf
  • Set up a new "partner" to pick up files from the pick up server
    • Same as above, except instead of specifying the remote endpoint, for example acmefilm@partner_1.com, specify the local pick-up server, for examplepartner_1@pickup.acmefilm.com

Sample script to initiate a distribution operation

The script takes the following arguments:

  • an ID (which can be an internal tracking number, or a ticket number if using a ticketing system)
  • the path to the title (file to be transferred)
  • a list of remote partners, by name (for example partner_1 partner_3)

Example invocation:

aspera-distrib A1234 /titles/sisterhood2.final partner_1 partner_3

Example script:

ID=$1 
COOKIE=acmefilm:distro:$1
TITLE=$2
shift
shift
PARTNER_DIR=/tmp
SESSION_ID=`uuidgen`
echo "Distribution title: $TITLE ($ID)"
for i in $*; do
echo " ## Transfer manifest ## Name: $TITLE ## UUID: $SESSION_ID ## Client: ## Server: ## Recipient: asperamanifest@localhost ## Checksum: NONE ## Cookie: $COOKIE ## Token: NONE ## UserStr: NONE ## Start: \"$TITLE\" 0B 0B completed" > $PARTNER_DIR/$i/aspera-transfer-$SESSION_ID.manifest.txt
done

Notification and monitoring in Aspera Console

For monitoring, create a console account for each partner. For example "partner_1". Then give this user the right to monitor the following transfer path:

ANY -> acmefilm@partner_1.com

For partners who pick up their files from the pick up server, grant monitoring access to the following transfer path:

partner_1@acmefilm.com -> ANY

To set up email notifications, the partners can edit their console preferences and choose to be notified for start, success or failed transfers.

In order to notify your team, give your users rights to the transfer paths specified above, and they can set their notification preferences themselves.

Integrating the automated distribution system with a ticketing system

Most ticketing systems work off email notifications. You can create a console email template with the subject line required by the ticketing system, for example [rt.acmefilm.com #COOKIE], where COOKIE will be replaced with the ID# given by the user as argument to the distribution script.

Have more questions? Submit a request

0 Comments

Article is closed for comments.
Powered by Zendesk