This integration allows users to monitor a file on the Hadoop Distributed File System (HDFS), then have the option to trigger an action. If the file is found, the file monitor can either go to success or launch a task in the Stonebranch Universal Automation Center. It's also possible to run the task monitor in auto-restart mode, which means it can restart itself automatically after finding a file. Key Features: Uses the Python HDFS module, which calls the Hadoop WebHDFS REST API. Supports Universal agent for both Linux/Unix and Windows. However, it has been currently only tested against a Linux agent.  Select different log-levels, e.g., info and debug. All passwords are encrypted using Universal Controller credentials. Currently only the InsecureClient (the default) is implemented. The TokenClient can be implemented upon request. Currently only direct file matches are implemented, as this is the standard functionality supported by the Python HDFS module. A scan for files using wildcards (*) or regular expressions can be implemented upon request.

Stream data from one object store to another without intermediate storage.The Inter-Cloud Data Transfer integration allows users to transfer data to, from, and between any of the major private and public cloud providers like AWS, Google Cloud, and Microsoft Azure. It also supports the transfer of data to and from a Hadoop Distributed File System (HDFS) and to major cloud applications like OneDrive and SharePoint. Integrations within this solution package include: AWS S3. Google Cloud. SharePoint. Dropbox. OneDrive. Hadoop HDFS. Key Features: Transfer data to, from, and between any cloud provider. Transfer between any major storage applications like SharePoint or Dropbox. Transfer data to and from a Hadoop HDFS. Stream data from one object store to another – no intermediate storage. Executes quickly if the object stores are in the same region. Preserves always timestamps and verifies checksums. Supports encryption, caching, compression, and chunking. Supports regular expressions based on include/exclude filter rules. Supported actions are: List objects, list directory. Copy/move. Remove object/object store. Perform dry-runs. Monitor object. Copy URL.