Technical Brief: Linux Automation

Analysis, Optimization and Automation of Cron executed scripts

By Thorsten Müller

Summary

Universal Controller is a modern, simple to use solution that is re-inventing the way IT can automate the business that matters most to their customers. With its simple installation and highly visual management capabilities, Stonebranch customers are finding it quite easy to provide other areas of IT, as well as the business areas, with the level of automation that can give them a competitive edge. 

With its modern architecture and intuitive capabilities, Universal Controller could be the last scheduling or workload automation solution you will ever need.

Tagged as:

Summary: Analysis, Optimization and Automation of Cron executed scripts

This technical brief describes how Stonebranch would analyze, optimize and further automate and improve performance, security and control of Cron-based scripts via Universal Automation Center.

Introduction

Universal Automation Center and Stonebranch’s Xpress Conversion Tool (XCT) enables you to efficiently manage, integrate and automate your Cron jobs (Cron triggered scripts) with your IT landscape, without the need to redesign your current business process logic. Stonebranch’s experience from many customer automation projects showed that one or even all of the following limitations and problems apply to those Cron triggered scripts:

  • Many scripts are obsolete but not replaced because of unknown behavior.
  • Very limited error handling – if a script fails it needs to be re-started from the beginning.
  • Low performance – Cron executes the scripts using a time-based rather than event-based interval.
  • Complex script dependencies – If scripts depend on each other, then over time these dependencies become more and more complex and can no longer be followed easily.
  • Scripts are not centrally managed
  • No script versioning is applied
  • Scripts are distributed over several servers
  • Scripts are executed via different Cron types (system Cron, user Cron, Cron.d, etc.)
  • Scripts are not documented
  • No Audit Log is available “Who changed the script, when was the change and what was changed”?

Stonebranch will help your organization overcome the above limitations and further improve security, performance and central control using our Universal Automation Center. The following provides some of the key benefits, which have been achieved by our customers after introduction of Universal Automation Center. We assume that most of those benefits would also apply to you.

Reduction of the mean time to detect (MTTD) and the mean time to restore the service (MTTR) Key Benefits

By using our central web-based real-time Controller, all IT operators and application owners can view and interact with the end-to-end business process no matter how many (virtual) servers, Docker containers or OS types are involved. Via our role-based fully configurable self-care dashboard, users can customize their views, according to their job profiles and assigned access rights.

Process errors will be immediately detected and acted upon. A failed process can be restarted automatically or after fixing the problem, in any step of the process.

Customer Example:
A major worldwide company for household products could heavily reduce its operational costs through an offshore-based operations model from India. This was only possible by introducing the complete web-based automation solution from Stonebranch.

Reduction of the entire process duration by introducing event based processing

With Universal Automation Center, time-based processes would be replaced with event-based processes, wherever possible. This is possible because with Universal Automation Center you can build dependencies between the different servers and start a process step in real-time if the prerequisite conditions are met and if a certain time is not reached. Some examples for possible conditions are:

  • A file arrival/change/removal (a file can also be an event or message)
  • HTTP, SOAP, JMS and IBM WebSphere MQ (Message Queue) Event or request
  • Status changes of a task (success, fail, late finish, late start, early finish, etc.)
  • An application status change (e.g. Unix process started)
  • A time definition (e.g. every last business day of the month)
  • A combination of the items mentioned above (e.g. start a process, if two file exists on two different servers and the DWH job from last night was successful)

Customer Example:
A major Danish supermarket chain with more than 1200 locations has reduced their average SAP process duration by 20% due to the introduction of event-based scheduling using Universal Automation Center.

Customer Example:
A large European bank has reduced their number of data transfer scripts from several hundred to one central script with parameters.

Reduction of operational costs and improvement of quality

Through central storage and maintenance of all scripts, including file transfer scripts, the Maintenance and Meantime to Repair (MTTR) can be significantly reduced. In addition, the central version control of all scripts, including full audit capabilities, with tracking of any change with before and after images, increases the operational quality. Everybody knows which scripts are executed when and where and who has last updated them.

Auditability and KPI-based Performance Analysis

  • The Universal Controller Audit function maintains a detailed record of all user interactions with the Controller, including before and after images related to any change and a description of the differences.
  • KPI reports on all executed Jobs are provided including runtimes, errors, re-tries etc.
  • Historical Data and predictive analysis allows you to compare your KPI’s with past results and predict future performance figures.

Highest Security Standards
Our security concept is constantly validated by external BSI certified security companies to ensure the highest security standards. The latest security audits were performed in August 2016 by detack and in 2015 by Secuvera GmbH. We are using the latest OpenSSL toolkit for Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols e.g. AES256-GCM-SHA384 - 256-bit AES encryption. All User Authentication can be done using SSL LDAP/AD

Stonebranch’s automation solutions are used in many data centers requiring the highest security standard. This includes American Hospital Corporation, AXA worldwide Insurances, BMW and one of Europe’s biggest and most secure data centers for banks, Fiducia/GAD.

Our Approach

Our approach consists of three work packages. Each of the work packages brings its own benefits and improvements. Organizations do not have to perform all three work packages simultaneously. They could start with WP1 and then decide for the next step to further improve their IT landscape for automation.

Depending on the availability of your organization’s IT personal, the work packages WP2 and WP3 can be done by Stonebranch experts with limited support by your IT. Optionally, your IT can perform all tasks in WP2 and WP3 by themselves. Stonebranch would provide consulting efforts based on T&M in such a case.

Work Package 1

Cron selection & structuring

Description: Work Package 1

  1. We securely extract all Cron definitions and scripts from all servers and users in scope via the Stonebranch Cron Finder utility.
  2. We import all Cron definitions into the Universal Automation Center and automatically convert them into Universal Automation Center tasks.
  3. We will run a Universal Automation Center report, showing all the details about the Cron jobs configured at the servers in scope.
  4. We will provide the Universal Automation center Cron report including a zip file with all scripts started via Cron to your organization/IT.
  5. After this work package all Cron jobs in scope have been converted to tasks in Universal Automation Center. As a result, you could schedule all or only a selected set of legacy Cron jobs via Universal Automation Center. The scripts could be executed centrally from the Universal Automation Center script library or from the current location on the servers of your company.

Note: It is not required to install Universal Automation Center on-site. The results can be attained via Universal Automation Center in the Cloud, which can be accessed via the web (secured by latest Open VPN SSL AES256 incl. TLS 1.2).

Work Package 2

Script Analysis

Description: Work Package 2

  • Categorize scripts in order to start with the most important ones.
  • Identify obsolete scripts for clean-up.
  • Identify scripts which are not meeting standards anymore, according to your IT requirements, in order to increase performance and security.
  • Check which scripts can be transformed into Universal Automation Center workflows. In many cases scripts contain processes with multiple steps. If one step in a script fails, usually the whole script has to be restarted manually from the beginning. These scripts can be replaced with Universal Automation Center workflows, allowing restarts, error processing and central control and monitoring.  As result, error handling and control is improved.
  • Check which scripts can be consolidated via Universal Automation Center workflows. If a process is realized by dependent scripts, we can replace those “script sequences” via a workflow. The result will be increased processing speed, error handling and control.

Work Package 3

Process optimization

Description: Work Package 3

  • Work on the scripts in the order of the categorization defined in WP2.
  • Clean up obsolete scripts.
  • Adjust or even rewrite scripts, which are not meeting standards anymore, according to your IT requirements.
  • Transform scripts with multiple steps to Universal Automation Center workflows. A good example is a script which calls other scripts.
  • Consolidate processes realized by dependent scripts via Universal Automation Center workflows. Example: Start script b only after script a has finished and the output data from script a is available on the server.

Solution Description

The following provides a more detailed description of the three work packages.

WP1: Cron selection & structuring

In Cron_WP1 all Cron definitions currently available at your organization will be collected from the server systems in scope and will be retrieved into central “*.TXT” files. In addition to the Cron definitions, the Cron finder program will also extract all scripts used in any of the identified Cron definitions. The scripts are saved in a configurable directory, which has the server and related user name in the directory name in order to reference to which Cron tab the script belongs to.

To provide structured reporting on the identified Cron definitions, the associated Cron commands and related scripts are converted into a standard XML format and then uploaded to a structured database. This database is the Universal Automation Center database, located in a central Stonebranch, secured and protected environment.

After finishing WP1: Cron selection & structuring, the following milestone will be achieved:

Milestone 1: All Cron definitions in place at your company have been identified, collected and stored in a structured data format for future reference. All scrips being executed by any Cron definition have been identified, collected and stored in a central location. References have been created to connect the Cron definition and the script belonging to the definition. Script dependencies have been identified and documented.

As a result of WP1, Cron selection & structuring will be provided a report identifying all Cron definitions on the server systems in scope and the related scripts called within each Cron definition in a format like this:

The final format of the report will be discussed at the start of WP1.

The ultimate result of this phase is a structured overview of all Cron definitions in place, the scripts connected to each Cron definition, and the scheduling criteria for each Cron definition.

Once the work package 1 has been completed, there are different options available for pursuing a Cron structuring related project.

1.     WP1a: transition Cron execution to a workload automation solution

a.     Pursuing this route will implement Stonebranch’s Universal Automation Center into your infrastructure and transition all current Cron definitions, including the associated scripts, into the central Universal Automation Center database. This will provide:

                                               i.     Versioning

                                             ii.     Auditing

                                            iii.     Centralization

For all Cron definitions and their associated scripts, it also allows for a GUI driven workload automation approach where dependencies between execution become visible as workflows and planning gets simplified based on calendar and event capabilities.

2.     WP2: Analyze the scripts related to the Cron definitions for potential risks and optimization potential.

a.     Pursuing this route will leave the Cron execution as is for the time being and concentrate on the analysis of the underlying scripts to achieve the categorization of scripts based on:

                                               i.     Criticality

                                             ii.     Priority

                                            iii.     Risk potential

                                            iv.     Responsibility

This will establish a matrix for all scripts identified in WP1: Cron selection & structuring.

WP2: Script Analysis

The following section concentrates on WP2 and details the necessary tasks to be performed to achieve the following milestone.

Milestone2:

All scripts have been analyzed and categorized according to their criticality, priority and risk potential and documented in a single matrix alongside the responsible person.

The final format of the report will be discussed at the start of WP1.

The ultimate result of this phase is a structured overview of all Cron definitions in place, the scripts connected to each Cron definition, and the scheduling criteria for each Cron definition.

Once the work package 1 has been completed, there are different options available for pursuing a Cron structuring related project.

1.     WP1a: transition Cron execution to a workload automation solution

a.     Pursuing this route will implement Stonebranch’s Universal Automation Center into your infrastructure and transition all current Cron definitions, including the associated scripts, into the central Universal Automation Center database. This will provide:

                                               i.     Versioning

                                             ii.     Auditing

                                            iii.     Centralization

For all Cron definitions and their associated scripts, it also allows for a GUI driven workload automation approach where dependencies between execution become visible as workflows and planning gets simplified based on calendar and event capabilities.

2.     WP2: Analyze the scripts related to the Cron definitions for potential risks and optimization potential.

a.     Pursuing this route will leave the Cron execution as is for the time being and concentrate on the analysis of the underlying scripts to achieve the categorization of scripts based on:

                                               i.     Criticality

                                             ii.     Priority

                                            iii.     Risk potential

                                            iv.     Responsibility

This will establish a matrix for all scripts identified in WP1: Cron selection & structuring.

After finishing WP2, the client will have access to complete documentation of their current Cron processing, including the documentation of dependencies within processes, the risk factor involved, and the priority and criticality of individual Cron processes. Furthermore, a proposal will be presented on how to optimize the current processing based on available software technology or process knowledge. The next phase is concentrated on resolving the critical issues identified in WP2 and making the necessary changes to scripts in order to establish a Cron based execution based on the defined operating standard of WP2.

WP3:  Process optimization

This work package is a combination of implementing state-of-the-art, 21st century workload automation engine technology in order to orchestrate the process orchestration in an optimized, secure, auditable and automated manner. Being an independent software vendor of such technology, the definitions made are based on Universal Automation Center technology.

Any estimations made herein are, thus, also based on the usage of the Universal Automation Center technology and only valid based on that fact.

After finishing WP2, the client will have access to complete documentation of their current Cron processing, including the documentation of dependencies within processes, the risk factor involved, and the priority and criticality of individual Cron processes. Furthermore, a proposal will be presented on how to optimize the current processing based on available software technology or process knowledge. The next phase is concentrated on resolving the critical issues identified in WP2 and making the necessary changes to scripts in order to establish a Cron based execution based on the defined operating standard of WP2.

Example: Dependent Scripts

The following provides an example as to how a Universal Automation Center workflow optimizes the execution of dependent scripts. Let’s assume a master shell script is starting several other scripts. The Master script is started via a Cron job.

Current situation using a master script:

  • The master shell script starts in a sequence Script A -> B -> D
  • Each script waits for its successor (B starts after A is finished)
  • Script B starts in parallel script C1 – C6
  • In order not to overload the server, script B only starts two scripts in parallel at a time

Situation after conversion of the master script towards a Universal Automation Center workflow:

In Universal Automation Center, a single workflow is set-up via the web GUI containing all dependencies between all scripts. Via the Universal Automation Center virtual resource feature the number of tasks started automatically in parallel is controlled.

Benefits of the Workflow compared to the script solution:

  • Central “real-time” web based monitoring of the entire flow.
  • If an error occurs in a task in the workflow, you can rerun the tasks that fail (after correction) and continue with the workflow where it failed. Optionally, you can run a task before or after the failed task before you continue or you can continue with correcting the error.
  • Automatic re-start of a failed task with many options e.g. re-start interval, number of re-starts.
  • Variables can be passed to the entire workflow or the individual tasks. These variables can be used in the script code of the task. A scripting language in a task is supported (shell, Perl, python, java, etc.)
  • Dependencies between workflows e.g. the workflow should be started every day, but only after the workflow from the day before has been finished successfully.

Let’s recap the areas, which Universal Automation Center provides superior capabilities compared to Cron.