Release Notes: IBM Aspera Enterprise Server and Connect Server 3.7.3 for Isilon

Product Release: June 6, 2017
Release Notes Updated: June 6, 2017
 

This release of IBM Aspera Enterprise Server and Connect Server for Isilon provides the new features, fixes, and other changes listed below. In particular, the Breaking Changes section provides important information about modifications to the product that may require you to adjust your workflow, configuration, or usage. Additional sections cover system requirements and known problems.

NEW FEATURES

General

  • When using a local URI docroot or destination (such as file:////data), for example when server-side encryption-at-rest is enabled, temporary partial file names can now be created during ascp transfers by configuring <partial_file_suffix> in aspera.conf. (CIM-85)
  • FASP transfers are more secure with upgraded OpenSSL (from 1.0.2i to 1.0.2j).
  • Special file permissions (setuid and setgid) are now supported in ascp and the Node API when the permissions are configured in aspera.conf. (CIM-201)
  • File checksums SHA-256, SHA-384, and SHA-512 can now be set in aspera.conf. (CIM-269)
  • Improved detection of potential Distributed Denial of Service attempts involving missing or slow SSL negotiation requests. The HTTP fallback daemon now automatically times out (after 20 seconds) connections that do not send a request or are too slow.

ascp

  • The performance of the FASP protocol is enhanced. Compared to prior releases, unencrypted CPU-bound transfers can see an increase of up to 100% in overall throughput; and encrypted CPU-bound transfers may see an increase in overall throughput of up to 20%.
  • Resume mode checks in ascp now use a SHA-2-based checksum algorithm by default; to configure alternative algorithms, there is a new aspera.conf option.
  • Ascp now has include and exclude filter options to support glob and regular expression matching, equivalent to include/exclude options of async.
  • A new FASP Manager API feature can be used to send an event when individual arguments, such as directories in persistent sessions, are completed.
  • Ascp now includes a new asynchronous “in transfer” post-processing engine with Lua scripts. A Lua interpreter engine is packaged with Enterprise Server, allowing for post-processing, validation, and authorization functions through embedded Lua scripts. Scripts execute asynchronously, and their progress is reported through a new validating state in the transfer session, without delaying completion of the file transfer or slowing the transfer pipeline.
  • Parallel (multi-session) transfers initiated from the command line can now use a URI for the filepath.
  • The <file_create_mode> configuration in aspera.conf is now respected when the docroot is a file URI.
  • A new transfer scheme, faux://, enables testing a transfer without reading from disk or writing to disk, eliminating the need to generate large test sets.
  • Users can now specify multiple SSH private keys (both DSA and RSA keys) in the same ascpcommand.
  • SSH private key strings can now be used to authenticate ascp transfers by setting a new environment variable, ASPERA_SCP_KEY.
  • Two new ascp options extend support for base64 encoding: --dest64 can be used to indicate that the destination path is base64 encoded, and --source-prefix64=source_prefix can be used to indicate that the specified source prefix is base64 encoded.
  • Persistent ascp sessions (run with the --keepalive option) can now accept management control messages sent from a server to the remote machine.
  • The ascp management fields "User" and "ClientUser" now reflect the authorization and authentication methods.

Node API

  • A new case-insensitive filter can return file name matches regardless of case to Aspera Files.
  • The /ops/transfers API is improved, including new support for setting transfer rates, bandwidth priorities, and pausing/resuming/canceling active transfers.
  • Asperanoded includes improvements in the /info and / API response, such as support for content protection settings.
  • /ops token authorization now uses SHA-2 as the default checksum, rather than SHA-1.
  • The status of files and transfers can now be reported by iteration token from the Node API. For example, if the user specifies an iteration token tohttps://node:9092/ops/transfers?iteration_token=1234, files and transfers from iteration_token=1234 to the present are returned.
  • File modification events, including delete and rename, are now logged to redis/scalekv.
  • Transfer and bandwidth statistics are self-cleaning for large numbers of sessions.
  • Default and allowed SSL ciphers are updated to eliminate support for 3DES and to align the defaults with hardened versions.
  • Users can now run asnodeadmin --db-update to migrate data from Enterprise Server 3.5.6 and earlier versions.
  • Asperacentral has a new raw options capability that allows users to authorize the use of raw format ascp options in raw format by configuring aspera.conf.
  • The Node API default symbolic link policy is now follow.
  • A new Node API call returns usage data (bytes transferred in, transferred out, and total) listed by usage ID.
  • The Node API now returns events for the creation, rename, and deletion of files by using the Node API, in addition to file transfer events. (CIM-52)
  • A new Node API call returns verbatim the content of a file within the file size restriction set in aspera.conf.
  • The access key and token secret of a Node API user are now passed on to Aspera Central, such that ascp is run with the environment variables that are associated with that user.
  • SSH private key strings are now are now supported in the Node API through a new JSON element, ssh_private_key.
  • RESTful /files calls now return full filepaths for admin users.
  • Specific events can now be requested by ID by using the new /events/{event_id} call.
  • The Node API now supports file locking. ascp transfers respect the lock status while the file is checked out. File locking can be enabled on the server by setting the<files_filelock_enabled> option in the <server> section of aspera.conf to true.
  • The Node API now supports access keys for IBM Bluemix S3 storage and Google Cloud Storage.

Watchfolder and Aspera Watch Service

Watchfolders and the Aspera Watch Service are now supported on Isilon.

The Aspera Watch Service (asperawatchd) is a file system change detection and snapshot service that is optimized for speed, scale, and distributed sources. Changes in source file systems (new files and directories, deleted items, and renames) are detected immediately, eliminating the need to scan the file system. It is automatically installed and started with the installation of Enterprise Server, Connect Server, Point-to-Point Client, and Desktop Client.

The Aspera Watchfolder service (asperawatchfolderd) enables large-scale, automated file and directory transfers, including ultra-large directories with over 10 million items and directories with "growing" files. Watchfolders uses input from asperawatchd to automate file transfers from a source folder to a destination system. Watchfolders is run by the client and the Aspera server endpoint does not need additional software to transfer content. Clients can also specify local or remote post-transfer processing actions. A valid, watchfolder-enabled license is required to use this feature.

Other Changes

  • The pre/post variable TOKEN is no longer valid.
  • Activity bandwidth logging is no longer enabled by default.

BREAKING CHANGES

If you are upgrading from a previous release, the following changes in this release may require you to adjust your workflow, configuration, or usage.

  • With the implementation of new inline validation options, existing users with inline validation enabled must set the validation provider to uri or lua_script. The command is the same regardless of platform, but for Linux you must preface the command with its path:/opt/aspera/sbin/asconfigurator.
    > asconfigurator -x "set_node_data;validation_threshold,uri"
  • Recursive counts are now disabled by default, but must be enabled in order for Aspera Files to use this feature. Workaround: Edit the <server> section in aspera.conf such that<files_recursive_counts_workers> is set to 5:
    <server>
        <files_recursive_counts_workers>5</files_recursive_counts_workers>
    </server>
  • Multi-session (parallel) transfers from 3.5 clients are no longer compatible with 3.6 and newer servers. This is due to a change in how files are split and how the splitting of individual files is configured on both versions. Workaround: Upgrade the client side to 3.6 or newer.

  • For RESTful operations (such as the Node API /files call and ascp run with file IDs), file events as returned by the Node API /events call, no longer show file paths. Instead, they return values for parent_file_id, file_id, and the file name. File events for RPC-style calls to /files and non-file-ID ascp transfers remain unchanged (reporting file_id and file path).

ISSUES FIXED IN THIS RELEASE

Note: This release contains tickets that were created from different issue-tracking systems. For this reason, the list below uses two different formats for issue numbers.
ES-161 (#35255) - When Japanese characters that include the voiced sound mark are used to name a directory in Connect Server, the following issues may occur:
  • The upload fails without an error message.
  • The download fails with the error code 43: The specified path cannot be found.

ES-112 - [Isilon 8.0.1] Node API transfers fail with the error "Failed to Authenticate."

#35453 - Parallel transfers started with the Aspera ascp Client on ES 3.5—transferring to ES 3.6 with the -C option—fail with the error Session Stop (Error: Session initiation failed).

#32080 - An error message spams the log file used by asperanoded each time Console to check the node for the status. A second error can occur when the end user fills in a docroot but doesn't have a valid system user; this results in an error that also spams the log file used by asperanoded.

#29372 - In a Connect Server installation using an Isilon cluster for storage, some icons and checkboxes expected in the directory display are missing.

#28288 - When the FIPS mode in aspera.conf is set to true, ascp fails with the error message, "ascp.exe: failed to authenticate, exiting.", without prompting the user for a password.

#13450 - The -C option does not work for special storage (docroot or direct).

SYSTEM REQUIREMENTS

Server

  • Isilon OneFS: 8.0. 8.1

Client Browsers for Connect Server Web UI

  • Windows: Microsoft Edge, Internet Explorer 8-11, Firefox 27-53, Chrome 32-56
  • Mac OS X: Safari 6-9, Firefox 27-53, Chrome 32-56
  • Linux: Firefox 27-53, Google Chrome 32-56

PACKAGE INFORMATION

Isilon 8: aspera-entsrv-3.7.3.143964-isilon-8.0-64-release.tar
md5: b21903e38bceaa47d719c7749d1f57f3
sha1: 2a3a59cf5bea7464afc2296af1d8e7fcb94e9c90
Isilon8: aspera-entsrv-3.7.3.143964-isilon-8.0-64-release.tar
md5: b21903e38bceaa47d719c7749d1f57f3
sha1: 2a3a59cf5bea7464afc2296af1d8e7fcb94e9c90

KNOWN ISSUES

Note: This release contains tickets that were created from different issue-tracking systems. For this reason, the list below uses two different formats for issue numbers.

General

ATT-245 (#22726) - Successful transfers might log the error, Failure Event: -34 - libssh2_channel_wait_closed() invoked when channel is not in EOF state, particularly downloads in FIPS mode. The error can be safely ignored. (CIM-329)

ATT-98 - If inline validation is configured on the server side, the server does not honor a session timeout if a transfer includes a skipped file.

ES-304 - [Isilon] On a fresh installation, the directory that Aspera uses to contain the Redis database is not created. Workaround: See the Admin Guide for how to configure the database on your Isilon cluster.

ES-249 - The aggressiveness setting is being applied to Vlinks, rather than only the network rate controller. (CIM-399)

ES-216 - If the Aspera Connect Plug-in is unable to connect to the server by SSH, a misleading error message, "Failed to authenticate," is reported rather than indicating that it is a connection problem. (CIM-72)

ES-215 - If the Aspera Connect Plug-in is unable to connect to the server by SSH, no fallback is attempted. (CIM-320)

ES-188 - Transfers through Aspera Forward Proxy are rejected if the node user password contains an @ symbol. (CIM-290)

ES-118 (#21517) - Folders that are created in the Connect Server web GUI can have permissions different from the permissions that are specified in aspera.conf.

ES-116 - [Isilon] asunprotect is not available for Isilon. (CIM-136)

#35952 - asunprotect cannot decrypt a reprotected file.

#35592 - [Isilon] The log rotation for ES for Isilon 7 does not accept J as a zipper; J as bzip2 is not recognized by Isilon and therefore the logs are not compressed. Workaround: Use Z (gzip).

#34811 - You are unable to download encrypted files with an incorrect decryption passphrase when you are using HTTP fallback.

#32934 - If the Internet accountability software Covenant Eyes is installed, some HTTP fallback transfers appear to complete but then lose connection with the server and then attempt to retransfer. Covenant Eyes captures the entire HTTP transmission before forwarding it to the server. If the file is so large that this process takes longer than about 20 seconds, the server times out and cancels the session. Workaround: Reduce the probability of timeout by increasing the server timeout length. Set Session Activity Timeout in aspera.conf by running the following command:
$> asconfigurator -x "http_server;session_activity_timeout,time_in_seconds"

#32517 - Retransfer requests are unencrypted when transfers are encrypted. This change in encryption can cause transfer failures in some scenarios, such as when a network device drops the retransfer request because it detects a bit sequence it considers malicious.

#31791 - Files with the file extension .aspx are not transferred. Workaround: Edit the resume_suffix setting in aspera.conf on the client.

#30690 - ascp fails with an inaccurate message—Error: failed to authenticate—when the server is configured to accept only unsupported ciphers.

#30616 - [Isilon]asconfigurator fails with an error that there is no space left on the device. If this occurs, restart the server so that handles are released.

#28679 - In some cases, the fallback server cannot accept additional connections, possibly due to too many 'incomplete' requests.

#27056 - ascmd does not respect server-side symlink configuration.

#23246 - Warnings are not generated about files skipped due to the source base setting.

#23070 - If a transfer of several files is interrupted, the retries generate a no such file error for files that transferred.

#22998 - If the overwrite setting in the server's aspera.conf is deny, a destination file with the same name as the transfer file is still overwritten.

#21629 - Connect Server aspera-dirlist.pl does not accurately reflect file permissions for user actions.

ascp

ATT-366 - An ascp transfer to object storage does not fail immediately if chunk size is configured incorrectly.

ATT-365 - An ascp transfer that is initiated with persistent session (from one of the FaspManager SDKs) might crash while freeing memory for argument stop management messages.

ATT-226 - If a URL docroot is configured, ascp reports incorrect bytes for the sessions that are involved in a multi-session transfer.

ATT-205 - ascp transfer fails with an internal memory error if <network_rc><module> is set to air in the <bandwidth> section of aspera.conf.

ATT-189 - In rare cases, ascp keeps running after it encounters a disk read error. (CIM-233)

ATT-185 - ascp does not reconnect to Redis database when asperanoded is restarted.

ES-267 - Under rare conditions, ascp transfers to cloud object storage may be reported as successful even though Trapd reports an error and the content is not in the storage. (CIM-475)

ES-177 - The range_low value of a -@ argument is not respected.

#35010 - If the source path in an ascp transfer is a file that is named \ (which is not supported by Aspera), the file is not transferred and an error is generated, but the folder then contains the file and all other files in that folder are transferred.

#33094 - The ascp option delete-before-transfer is not supported for URI storage.

#32890 - During an ascp transfer that uses the --preserve-xattrs= metafile --remote-preserve-xattrs=metafile options, the metafile is not transferred.

#32680 - The option to create a directory (ascp -d) may create a directory at a destination before an expected session failure.

#32553 - When the FASP Session log source file list exceeds 500 bytes and contains multibyte UTF-8 characters, the output is truncated in a manner that creates an invalid UTF-8 sequence.

#31423 - It is possible for an ascp transfer of a file on a full disk to be reported as successful by both the sender and the receiver.

#30324 - During an ascp upload to cloud storage, if a mid-file read failure occurs on the sending computer (which is rare) it can cause the server-side ascp to crash and possibly fail to report transfer completion. This read failure can be caused when a source file is truncated during transfer, a drive or file system fails, or a transfer is canceled with Ctrl+C or other means.

#29255 - Download from SoftLayer of a file larger than 62 GB is unsuccessful. Workaround: Do not use time-stamp preservation with SoftLayer.

#28939 - If command line ascp neglects to specify a destination host, then the failed transfer (error: "no remote host specified") gets recorded in SQLite with client_node_id NULL, instead of being populated with the uuid of the node. This database error causes an issue with Console.

#26281 - If you run approximately 100 (or a similarly high number) concurrent uploads to S3, intermittent transfer session failures can occur.

#26185 - During an upload to S3 storage, an error may result if ascp reports a successful file transfer before the transfer to S3 completes.

#25865 - Allowing symbolic links to be copied also allows access to locations outside the docroot.

#23503 - Uploads of zero-byte files to Akamai appear to be successful, but no file is present at the destination.

#22905 - When you copy a file in S3 storage with ascp, if a slash is appended to the destination -- for example, /path/ -- the file is renamed path/. Because of the trailing slash, it appears to be a directory, but is actually a file.

A4

A4 is not supported for Isilon.

Node API

ES-309 - [Unix-based OS] When a group ID (setgid) is set on a parent directory (chmod g+s parent_dir), subdirectories that are created by a call to the /files/create endpoint have the primary group ID of the user rather than the group ID of the parent directory. (CIM-541)

ES-248 - While a transfer of many files is in process, Node API reports skipped files as complete. The counters are correct once the transfer is complete. (CIM-398)

NODE-345 - A RESTful Node API POST to /ops/transfers can trigger two transfer sessions for the same file and result in a corrupted file at the destination and a slower final transfer rate.

NODE-257 - Reports sometimes fail if the Node API temporarily reports an impossibly large value for bytes_transferred.

NODE-244 - A POST request that contains an invalid value for "storage_class" returns the wrong error message, "Invalid value for server_side_encryption".

NODE-236 - Transfers with a status of "waiting" cannot be canceled.

NODE-177 - [Unix-based OS] ascp transfers and asperanoded might fail when you tryi to transfer many (millions) of small files because the Redis database exceeds available number of file descriptors. Workaround: Increase the maximum number of file descriptors from the default of 1024 to a larger value, such as 1,000,000, by running the following command:

$ ulimit -Sn 10000000

NODE-139 - The --token-key-length option in asnodeadmin allows invalid token key lengths.

NODE-137 - A Node API /ops/transfers call reports the incorrect values for files_completed and files_failed.

#33374 - Symbolic link capability is only available on local storage but an unimplemented function error does not appear when the user attempts to create a symbolic link to a file on cloud storage (S3) from the Node API.

#33229 - Users cannot browse a file on cloud storage by using a /files/browse API request.

#33206 - /ops/transfers erroneously shows some queued transfers (which are farther down in the queue) as failed before they complete.

#32669 - When a directory is linked from a subdirectory, it does not appear in the search result for a /files/search request in the Node API.

#32627 - When a file name is just a dot and an extension, (for example, .pdf), then it is reported as a file with "content_type"=>"application/pdf" or a hidden file named PDF; for example:
{"id"=>"27", "name"=>".pdf", "size"=>12, "content_type"=>"application/pdf", "type"=>"file", "modified_time"=>"2015-09-10T15:24:01Z", "access_level"=>"edit", "permission_count"=>0}

#31712 - For both S3 on AWS and SoftLayer Swift storage, /files returns modified_time for files but not for folders.

#30542 -/files PUT (file rename) should be fixed to involve only one PVCL operation but still return the proper 409 code when there is a destination conflict, and PVCL needs to return proper error codes stating that the move operation failed because of a destination conflict.

#29848 - When <write_allowed>, <read_allowed>, and <dir_allowed> are all set to false inaspera.conf, Node API calls to URLs such as /files/browse are returning response code 500 Internal Server Error: instead of another code that better indicates that access to the resource is denied.

#29787 - When the docroot is not configured, the HTTP error code 500 ("Internal Server Error") is returned.

#29187 - For content in cloud storage, the Node API does not display all the files in the docroot directory. Workaround: Use the /files/info request to browse the docroot directory when content is in cloud-based storage.

#29138 - For files in S3 storage, the Node API does not return the correct file modification time.

#29078 - When an access key is created with the standard node user authorization, the access key inherits that node user and its associated system user. Afterward, asnodeadmin can be used to associate a new system user to the node user, but the new system user is not updated for the access key.

#28219 - [Unix-based OS] asperanoded fails to delete a directory name containing backslashes. Workaround: Increase the number of system processes that are allowed by the operating system to run concurrently. In a Terminal window, take the following steps (note that the limit is shell-specific, so all commands must be run in the same Terminal):
  1. Stop asperanoded.
  2. Determine the current limit by running:
    $ ulimit -a | grep "open files"
  3. Change the limit. For example, to set a value of 2048, run the following:
    $ ulimit -S -n 2048
  4. Restart asperanoded.
  5. Resend the /files/delete request; the directory and all subdirectories are removed.

#25127 - HTTP fallback temporary files (*.haspx) are not excluded by the Node API.

#23434 - Files that start with "._" are not returned by the Node API browse action.

#22619 - In the Node API, /files/search follows symbolic links.

#20002 - The Node API is inconsistent in how it handles symbolic links. /files/browse does not follow the links and reports links and their target (final type and next name), while /files/info reports symbolic links as files or directories.

#18659 - Searches with very long path names (over 520 characters) report an "insufficient buffer space" error.

#18368 - Files with a backslash in the file name are not displayed in the list when the user browses the remote source on the new package page.

Watchfolder and Aspera Watch Service

WAT-501 - Some ascp sessions started by a Watchfolder may not stop running after synchronization is complete when many (50) large (1000 files of 2 KB to 1 MB) Watchfolders are started at the same time.

WAT-314 - asperawatchfolderd must be running in order to delete a Watchfolder.

WAT-200 - Recently finished Watchfolder drops are not stored and are lost if asrund is restarted.

WAT-174 - Watchfolder uses excessive memory when it watches 10 million files.

WAT-169 - If top_level_dirs drop detection is used with x top-level directories in Watchfolder, 7(x)+ drops are created. The drop count continually increases.

WAT-159 - If one file in a Watchfolder transfer fails or a drop is aborted, the other files in the package are reported as aborted but ascp is not stopped and the transfer continues.

#33877 - Aspera Enterprise licenses that are watchfolder-enabled don't display watchfolders as an enabled component.

Sync

Sync is not supported on Isilon.

Object Storage Support

TRAP-71 - Multi-session transfers to object storage can stall if the number of files open for write in multi-session mode exceeds the default number of starting threads (64). Workaround: Open /opt/aspera/etc/trapd/trap.properties and set aspera.session.max-starter-threads to a larger value. If this setting is not in the file, add the following line with an appropriate value:
aspera.session.max-start-threads=1280

TRAP-59 - If an incorrect DNS nameserver is set in /etc/resolve.conf and then corrected, TrapD must be restarted for the correct nameserver to be used by TrapD. If TrapD is not restarted, TrapD fails to connect and retries indefinitely. (CIM-469)

TRAP-57 - If a very large file (several TB) upload to AWS S3 is interrupted after more than 1 TB is transferred, resuming the transfer may take hours and the session may close before any data is transferred. (CIM-476)

TRAP-28 - When downloading from cloud or object storage, ascp always takes the equivalent of 1 GB of buffers from Trapd. This can lock buffers in ascp queues for hours and may prevent other ascp transfers from transferring normally.

TRAP-27 - In some cases, stopping Trapd while an ascp transfer is still running may cause a restart of Trapd to fail.

TRAP-26 - Sometimes when Trapd is being heavily loaded by many ascp transfers, Trap may return a 'No such file or directory' error.

#36067 - Deleting folders from a Limelight directory is slow.

#33214 - Transfers to and from cloud storage using authorization tokens with URIs that do not have a docroot specified are not supported.

#25636 - To use a larger chunk size to transfer large files to AWS S3 storage, some users modify the memory settings in the Trapd initialization script, asperatrapd_init.sh. If you do so, be sure to preserve the script manually during upgrades to prevent it from being overwritten.

PRODUCT SUPPORT

For online support resources for Aspera products, including raising new support tickets, please visit the Aspera Support Portal. Note that you may have an existing account if you contacted the Aspera support team in the past. Before creating a new account, first try setting a password for the email that you use to interact with us. You may also call one of our regional support centers.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk