Real-Time Hybrid IT Automation: Simplifying Your Hybrid IT Automation Strategy

Well, welcome everybody to the second session in our online education forum. Today's topic is real time automation for hybrid IT. As IT infrastructures are becoming more varied and dynamic, and therefore managing your workloads is becoming more complex, today we're going to explore a sample of universal automation centre features that will help you in managing this paradigm. Firstly, we're all going to get on the same page regarding terminology for hybrid IT, then quickly cover some of the drivers that are moving us in the direction of hybrid IT, and finally, review some of the more common hybrid IT automation scenarios, highlighting which product features come into play when you're joining the dots between the various platforms and applications in this hybrid IT world. As Marisa said, we will allow time for questions at the end, so don't forget you can use the questions tab or raise your hand in the GoToWebinar control panel. Firstly, as simply as we can, some terminology. Hybrid IT means that some of your applications run on premise and some in the cloud. So unless you're one hundred percent cloud or zero percent cloud, you have a hybrid IT environment. And within hybrid IT, and specifically when talking about the cloud, there are several subtly different use cases. Hybrid cloud uses a mixture of public and private cloud. Multi cloud is where you hedge your bets between more than one cloud provider. Additionally, some implementations use the distributed cloud to take advantage of different geographical locations, and you'll also hear the term cloud native, which refers to applications that were specifically created to leverage cloud capabilities, such as a microservices application architecture. Hopefully I haven't confused anyone too much yet. And despite how it sometimes seems within some organisations, there are some very compelling reasons for cloud adoption. The attraction of deploying cloud resources is that they lend themselves to a pay as you go pricing model, deploying and paying for resources only as you need them, and in some cases, passing the management overhead on to the cloud or SaaS providers. And inherent into the cloud model is the ability to dynamically scale your IT infrastructure, both up and down, based upon demand and throughput. This is much more attractive than the on premise model, where you need capacity to match your maximum peak demands. The capabilities of the cloud computing model are revolutionising IT's ability to deliver business value, with a significant increase in the speed of getting new features to market. This is really at the heart of the digital revolution that we're seeing today. For those of you who also attended our opening session, you'll recall that one of the two major shifts affecting workload automation is infrastructure. And from a workload automation perspective, hybrid IT is just more infrastructure. Now, I appreciate this is a simplification, but the point I'm making is that it's not as alien as it might first appear. There's just going to be more of it. However, it is more dynamic in nature than the on premise infrastructure that we're used to, and we're going to explore some of the tactics we can use to manage this. So as the infrastructure where our workloads execute evolves, Stonebranch's platform approach will support you in automating your hybrid IT infrastructure. So let's step back and review the architecture of the Universal Automation Centre and see how we can manage a hybrid IT environment. Here, we see your on premise cloud and external partner landscape. You can deploy the Universal Controller on premise in your cloud, or take advantage of our software as a service offering. Universal Automation Centre provides everything you need to manage workloads in a traditional infrastructure environment. And this extends to the cloud, allowing you to run workloads in the cloud, along with managing the provisioning and orchestration of cloud services. And with the Universal Data Mover Gateway, you can manage the transfer of data to and from the cloud, your trading partners, and with file sharing applications. And of course, our components are engineered to work with other workload automation tools. So if you need to manage your hybrid IT landscape, the Universal Agents, Connectors, and Data Mover components can still provide you with these capabilities. So what I'm going to focus on today is how the Universal Automation Center architecture and features supports your hybrid IT automation. Specifically, we're going to cover some of the more ubiquitous hybrid IT automation scenarios. So starting with cloud migration. With a little forethought, you can design workloads that can dynamically be migrated between on premise and cloud infrastructures. So what I have here is a workflow that we're going be going through today, going through a number of different scenarios. And the first thing I'm going to do is we'll just go ahead and launch that workflow. We'll kind of take a look at where we're going to start. So I have a task here that's currently set up to use an agent cluster. If we look into that agent cluster, what we see here is it's an agent. And this is one of my on premise agents. Okay, it happens to be running in a container, but it's running in an on premise environment there. So quite simply, by running this, it'll select the agent that it needs to. And if we go ahead and close this out, actually open up, Get into my instances, and we look at the one that's running today, we see this one that's held. Let's get into the workflow view, so we can see what's going on. And if I release this job, let's release the workflow. There we go. Excuse me. We'll get into the flow in just a second. We see that that's running. And if we drill into this task, what we can see is that this task has selected an agent to run based on that distribution. Okay, so it's done that dynamically. There could have been multiple agents in there, and depending on the rules of the agent cluster, it would have selected one to run. So now let's go ahead and provision some cloud servers. And if we kind of take a look at the tasks that we have set up to do this, this is kind of where we get into some orchestration of some of the cloud based components, in this case, containers. And this could just as easily be a task that provisions a server in a cloud environment, depending on any one of a number of different cloud providers that you have. But one of the things to look at on this here is that within this task, which is managing the Docker, and I'm using the Docker Compose environment, and we have tasks that manage a number of the different sort of Docker flavors for starting and stopping servers. I like the Compose. But this basically has, if you like, is infrastructure as code. So within here, I have a base image, and it's the same image I use for all the agents. And basically, we're just passing parameters into that same image to create different servers. And in this case, I'm going to create two servers, each one of which is running a universal agent. And we're just passing in parameters to tell it which controller to connect to, which agent clusters to register with, etcetera, etcetera. And the other thing to bear in mind here, is that what I'm also doing in this task is a little bit of automation with an action, which is going to take care of just suspending the cluster membership of the on premise agents. So now that that cluster will contain just cloud agents, and not on premise agents, so you'll be able to see in a minute that we can run the same workload, and it'll run somewhere else, and we haven't had to update that workload. So if we go ahead and release this. Okay, we'll see that run. Nope, and it failed. That's always the unfortunate thing with live demos, something will go wrong. If we go and look at the cluster here, it should have created at least one agent, which would be good enough for our purposes. So if we go and look at the agent cluster now, we should see a difference in the agents that we have in there. Okay, so it's not done the suspend that I want it to, because that action was based on the success of the task, so I will do that manually. So I apologize, not quite as smooth as I wanted it to be on there, but good enough. And we can go ahead and release this job, and we'll see that while it's running, that it will have been assigned a different agent once it gets into starting. And what's it Oh, it's waiting for, so we have to Force finish that task to clear its predecessor. So now if we look, we'll see that the agent that's assigned is the agent from the cloud environment, so we've been able to switch that task without too much manual intervention, despite the hiccup in my setup here. So very simply, we can see that by using the agent cluster and variables, we remove the need to recalibrate our workloads when we're migrating to the cloud. One of the other scenarios that we're going to run into is transient IT infrastructure. So particularly when we're talking about cloud environments, what happens is that we don't necessarily know in advance where workload is going to be executed, and we certainly don't know that when we define the tasks. And additionally, some of the things that we run into, whether it's containers or whether it's cloud servers, is that the decision about where the workload executes is probably not made best by the universal automation centre. It's really something that the cloud infrastructure itself, particularly when you're scaling up dynamically in the cloud based on definitions that you have in those environments, or if you're using container management services like Red Hat's OpenShift, that's going to be handled outside of our control. So we have a subtly different technique which handles that within the product. So if we take a look at a very similar task that we have here, we can see that this actually has a different agent cluster. And this one uses a technique called network alias. And if we drill into this, really what this is doing, is this is basically pointing at an IP address. So in this case, it's pointing at a load balancer, and the load balancer is looking at the service behind it, and is going to direct the work to one of those servers based on that rule. And based on that criteria, what's happening behind the scenes is that we're determining if there's an agent on that server, and which agent it is. So if we go ahead and do the resolve on this, we can see now that it's resolved, it's resolved to my cloud agent. If I'd had both of them active, it would have resolved to one or the other, depending on the rules and which one was most eligible for the workload at that point of view. But now if we go ahead and release this task, we go ahead and see that that has, when it gets there, selected the agent based on the network alias rules. So this has given us the capability to really go ahead and manage workload based on an external event or an external process that's deciding where the workload should run. We don't have to know where that workload is going to run at the point that we're making definitions within the environment. And the last thing here, we can go ahead and we can deprovision these. So one of the things that we have within the product is that these agents, and if we look at the agents here, and I just refresh this, we can see that the additional agent has come on. And this agent now has registered within the product. And we can see it there. But particularly with cloud infrastructure environments, where those servers are transient, we don't necessarily want to have agents lying around that are inactive, that are not being used, when those containers or cloud servers are not available. So in the last release, we introduced a new feature, which was a transient agent definition. And if we go back into our workflow here, and I just deprovisioned the cloud, and this is just stopping the Docker containers that we have. This may well fail because one of them isn't up, but let's go ahead and release it, is that agent will now, when it's deprovisioned, will dynamically remove itself. So this is the whole point of paying for what you're using when you're using it within the product. So we're not using up definitions, we're not using up licenses in terms of that when it goes. So when that's finished, and we've gone through, we go back to the agents and refresh that, we can see that that has been removed from the environment. That was a transient agent. And that's really designed using agents in container environments, where you're spinning those up as you need them, and then the definitions are just removing themselves as they go through. And using the agent clusters there enables you to have workflow that can take advantage of those agents when they're available, but but doesn't have to worry about them when they're not. So again, different kind of cluster definition with the network alias. And the network alias cluster feature supports the cloud auto scaling effectively. And the transient agent feature means that short lived containers don't need to hold an agent definition when they're not actively in use. So far, we've seen a simple use case for container management using Docker natively, and there are plenty of container management solutions in use, such as Amazon's ECS, Red Hat's OpenShift, Google's GKE, Azure's AKS, Kubernetes and others. Universal Innovation Centre really plays well with them all. We do document and maintain a public container image in the Docker Hub, but depending on your requirements, you may want to build your own. At last year's TechEd and Innovation events, I presented a session covering what you need to consider when doing this. And since then, we've introduced a number of new universal agent features that simplify this process. So feel free to reach out if you need updated information. And again, information is in the documentation of the product, if you want to leverage the images that we build and maintain. But my contact information will be displayed at the end of the session, so feel free to reach out, and we can provide updated information if you need that. Okay. So the Universal Data Mover simplifies the process of moving and managing your data in the cloud. Effectively what it does is it allows you to simply map cloud storage to standard file transfer accounts. So let's take a look at that. Okay, so if we go into the Universal Data Mover Gateway, one of the modules and features that's provided there is network storage. And the network storage allows me to effectively map this. So I have a definition here that's mapped to some Amazon S3 storage, a specific bucket. But if we take a look at this definition, we can see that there are a number of different protocols that we can map using the network storage feature. And you'll see that Amazon's in there, Google's in there, Microsoft, IBM Cloud. This also takes care of some of the file sharing systems out there as well. So also, you can integrate that when you've got peer to peer file sharing going on, into the environment where you can pull data from that, or push data to that that's available for the users in the environment. And once you've done that, effectively you can just set up a user that has access to that. So if we take a look at that, and I've labeled it so that I remember who it is, we just open that definition, is we can take a look at then the path. So effectively what we've done is we've just mapped the root path for that definition to the Amazon S3. So now I can use whichever file transfer protocol I've enabled for this user, to simply access that storage. So what that effectively means is that within the controller, we can very simply have standard file transfer tasks, standard file transfer monitors, that go ahead and can access, transfer to and from cloud storage, or to and from the file sharing systems. So before I go ahead and move some data in there, if we move into the Amazon console and look at the S3 in the specific bucket, and we'll just refresh that just to show this bucket is empty. So I've got no data in the bucket. So we're going go ahead and we're just going to release a standard file transfer task to transfer some data into that bucket. And we'll take a quick look at the definition here, while we're waiting for that to run. And we can see that this is just using SFTP, it's connecting to my local system, which I have set up with the UDM gateway, and is using those FTP credentials using that user, therefore has access to the S3 storage bucket. So if we go ahead and refresh that now, we can see that now we've got data in there. So again, integration directly into the cloud file storage networks. So obviously, if we can do that with file transfer, we can do that with file monitoring. And if we release that, the file's there, the monitor should go to success fairly quickly. And we'll just go ahead and release file transfer task, and move that data off the cloud. So effectively, the scenario here is that I just go ahead, I can have monitors set up, monitoring cloud storage, and as data is created in that, I can move that data to the internal network. Or vice versa, we can move that data to the cloud as it's created internally, so it can become very dynamic in that environment. And as we did a move, we should see then that that file's disappeared, we've moved that to an internal server, we've emptied the bucket, we can go ahead and process that data. So we also offer a number of universal templates in the Stonebrake's marketplace, in fact, built in templates that use the cloud provider sorry, in the marketplace, that use the cloud provider APIs to facilitate file transfer activity. So there's a number of different options. You can use gateway to have a very simple mapping, or we have specific tasks available that will provide similar capabilities for some of the cloud providers out there. So a common example of a microservice integration is calling an API, extracting part of the response, and using that response in other automation tasks. And there are a number of integrations we do with microservice, microservice applications, but let's take a look at that fairly simple scenario there. So I have a web services task, and if we take a look at the web services task, we can open that up. Very simple, just doing a get to an external application. And if we go ahead and release that, we'll see what kind of data we get back from that, and that should run fairly quickly. So it's gone to success, can pick up the output, which will show us the result set from that. And we can see here, it's a fairly simple information that we've got back. Let's say we want to pull the title out and use that within another workload. So effectively, what we've done within the definition of that task is set up an action, and there are a number of functions that we provide within the tool that allow us to get at this data. But I've just set an action, which is just setting a global variable. So we're setting a variable called the web service result, and we're using a built in function that's pulling specific information from that JSON. And we provide similar ones for the other formats that you might get, such as XML. So we can go ahead and we can just release a subsequent task that's going to go ahead and use that data. And this is just a simple Linux task, it's going to echo out that data. And we should see then that the command there is just echoing out the result that we had from that text. So it's fairly easy to use that, access your microservices, and get information, specific information out that you can use in subsequent tasks within the environment. So integration with microservices and applications in general using APIs can be two way. So there's also inbound listening. We can use the universal agent SOA connector, or you can use the UC APIs as being called from external services. Or we can expose certain integrations, or specific integrations, via Python modules, which integrate nicely with the universal task infrastructure. So there are a lot of services out there that provide, and again, you don't have, with the universal task implementation, we have taken the decision to use Python as a standard when we're doing it at the back end, but it supports any scripting language that you have in terms of how you want to do that. So if there are others that you're more familiar with, you can build tasks that can do these things, or we can provide these through the marketplace as we go forward. So as discussed in last Thursday's session, the growth of the democratization of IT means that your citizen automators may prefer to use native tools for defining their automation on different parts of the infrastructure. What that means is, and we've seen this before, is that the guys using, let's say, an example I have, Amazon, may prefer to use AWS Batch in that environment, okay? They're not your worker automation guys, they don't necessarily want to come in and use the tool. Well that's fine, they can define their workloads there, and basically, you can use the automation center to orchestrate that. So it's kind of a good strategy to let them define their work in the tools that they're most comfortable with, and provide an orchestration capability to encapsulate these tasks into your automation portfolio. So if we take a look at the last part of the demo here, I have a simple universal task, That provides information into the AWS environment. So I'm signing on to AWS, I've got a specific region that I'm going to, we have a job name that we're gonna run based on a specific job definition, and we want it to run-in a certain job queue from that. So we can go ahead and release that guy. And if we refresh it after it goes, and retrieve the output, we'll see some things going on in terms of what's happening. So effectively here, it submitted the job, and it's given us a job ID, and we can go in and we can view that job now, should be running in the AWS console. So if we go to the AWS batch, A couple of things we can look at, we can see the job definition that we have here, so that's my job definition, the example job, but if we actually look at the active jobs for the queue that we're running these things on, and let's see what status it's in. Okay, we can see that it's basically in a runnable status, hasn't been selected to run, but we can see, and if you're paying attention, you'll see that the job ID matches, but this is the job that we created at ten twenty eight this morning. And that will go through its process and be monitored and managed in the environment. It may take a little while to run. Still running, so we're still monitoring the status there. And we can kind of see as it goes through that it's polling the job, it's telling us what status it's in, and at some point, hopefully in the near future, we'll see that run to success, and we'll get the output for that. So we'll come back in a moment and take a quick look and see the output for that. But effectively, have universal tasks built for AWS Batch and Azure Logic apps, and we'll be making those available shortly. But based on the need for different integrations, the infrastructure we have really allows us to deliver that quickly, with the universal templates and the architecture there. So it is something that, as we run into use cases where people are using these tools, or wanting to integrate these tools, we really have the ability to provide that capability fairly quickly. And if we come back, and hopefully Oh, it's going to take a while to run today. Sometimes I found with the Amazon stuff, it takes a long time before it selects the job, based on the criteria I have and the priorities I have there. But that's something that we'll see in, know, oh, it's actually gone to success, so there we go. So now we see it, so if we actually go ahead and retrieve the output, we can see that we get the results back from that job. So these are all the messages that we're generating, these are optional. We're basically telling you what's going on as it runs, giving you the job definition, or we can see here the standard out, we're just saying hello to everyone at Stone Branch Online, it's the output coming back from that task, in terms of that. So we have, usual, when we're monitoring tasks, we get the exit code, and we get the output, so it's just another task within our workflow. Okay, so we have some minutes open for questions, so don't forget you can use the questions tab, or raise your hand in the GoToWebinar control panel. And this session was just a taster of how you can manage your hybrid IT infrastructure today with the Universal Automation Centre. In the world of cloud, you'll come across many different vendors and flavours of solution, and what we're hoping to have done here is instill the sense of confidence that these new infrastructure options are really just business as usual for Universal Automation Center and your standard workload automation routines and so on. It's just, you know, more subtly different task types that you use and just a couple of different ways of handling the infrastructure. So let's open up, and if there are any questions. Yeah, it looks like we have three. We'll start with Lakshmi. She asks, Can you show the agent setup in the cloud? Well, as I've shown them down, no, but you would be able to see these would be the agents that are set up in your EC2, I guess. And as they're defined, Amazon status running instances, what have I got? I don't think it'll be any of ones that I created and destroyed today, but yeah, okay, we can go ahead, and we've got a number of running in that environment, so we can see that. Yeah, mean, sort of yes and no, right? So in the demo, I didn't delve into that, but yeah, they're just agents from that point of view, and they come in and register to that controller. Cool. So, okay. Oh, you can see these questions too, right? Yeah, that's okay, yeah. So the next one there is, do I need an agent license to use this container management solution? So in terms of, I guess there's two parts to the answer to that question. The first is, obviously, if you're running an agent in the container, you need a license for that agent. If you want to just manage containers that don't necessarily contain agents, then the solution does run on an agent. You'll need an agent to control this, and that's the agent that will talk to your container management solution. So where that lives depends on what's going on there. So from that perspective, the tasks to manage the containers are effectively part of the universal controller license, and you do need an agent to run to execute those, but you don't necessarily need an agent license for everyone that you're running on. Hopefully that answered the question. Is Universal Data Mover required to monitor files in the cloud? So there are a couple of different solutions that we offer here. In the marketplace, there are tasks that we'll integrate with a couple of the different vendors for the cloud storage solutions. The solution I demoed today was the Universal Data Mover Gateway, and that provides the very slick integration of allowing you just to map that to different file transfer accounts, which gives you that. So depending on what you need to do, there are kind of, if you like, multiple options, multiple speeds of solution available from that side. But there are universal tasks that are available in the marketplace that provide some of that capability, that don't require the universal data mover licenses. Next question, we already have universal agents running for our product, can we use that license to test them on the cloud? The answer is yes. So typically the way that the agent licences work, it's you buy a number of agents, and you can use those agents wherever, up until you've installed the number that meet your licence. So if you need to do some testing in the cloud, there shouldn't be any reason why not. You may want to just double check-in with your sales rep, some contracts are different, but in general, yeah, certainly if you want to do some testing in the cloud, there shouldn't be an issue there, as long as you're still within your total number of licenses for using agents. Thank you. Yeah, yeah, so I guess that answered that question adequately. Any other questions? Oh, we have one more question. Two more come in. Oh, two more. Yeah, so how difficult is to move an agent from one domain to another? If we're talking network domain. Mean, shouldn't be. So when you're looking at domains, usually the issue is that they tend to be separated by firewalls, so you need to be cognizant of who's communicating in what direction through the firewall. And without going into huge detail, a diagram to explain it, effectively, this is controlled by where you place your if you're talking about with the controller, about where you place your OMS servers, and you can have multiple OMS servers. So effectively, the controller initiates communication to OMS, and the agents can initiate communication to OMS. So in other words, the controller and agents are clients of the OMS from that site. So depending on, again, which way you want to open the firewall, you only have to open it one way, and you can usually cater for that. I do have a diagram somewhere that explains that, but not handy for this presentation. How does Stonebranch or does Stonebranch provide dedicated UAC trainings for this specific subject? The answer is yes, reach out to your contact, and we could get something arranged to do some sort of more formal discussions on that if we need to. All the questions are flooding in. So how can we see a listing of the future sessions? Marissa, you want to tell them where they can get the next Yeah, actually. I sent two links in the chat that everyone should be able to see. The first link is to register for the session this upcoming Thursday. That will be a customer speaker, Reed Elsemer Technology Services. They'll be explaining their experience with StoneBranch, and they're always a crowd favorite. And then the second link I sent is a link to a register for all webinars. So if you put your information in there, we will make sure that you're registered for every session, and you'll get invitations and reminders for all future sessions. So, please take advantage of those links. They are there for you to use. Okay, how do we see a listing? You've covered that. Will we automatically be communicated for future sessions? I think that's more on your side as well. Yeah, we will definitely be emailing a follow-up email after this session is over with information about the recording. So yes, sessions will be recorded. I see that as a question as well. And you will get information about how to register for upcoming sessions if you haven't already, as well as what the coming sessions are about and who will be speaking. So every time we have a session and you attend, we will send you a follow-up email. And you're also welcome to visit our StoneBench online sort of general page. I will also put a link to that in the description box or in the chat box. Let me just grab it for you all. And that has all the information you need about the entire event. So here you are. Megan is asking if, again, asking if we're recording all the sessions, and can I share these with others on my staff? The answer is yes, so as you get the information with the follow ups on that. So one other question, and we'll make this the last question, because this will take me a minute or three to answer. So Mike has asked, Do all the features talked about today require UAC, in other words, the controller, I think he's referring to, or can we use CA-seven? So I kind of mentioned upfront that the capabilities there that the agents and the data mover components provide, and the connectors, etcetera, are available for other scheduling systems. So pretty much, when we looked at the demo, it was very controller centric, right? So we had a number of tasks that I used within there. Now, the universal tasks that we provide, So that would be the Docker integration, the AWS integration, those kind of things. The way that we design those is that, effectively, what there is behind that is Python script. Now, the universal agent comes with a Python implementation, so they can run on a universal agent. And the way those have been designed is that we front them with a form in the controller that provides all the input, but they're designed and written to be able to run from a command line, too. I'm not sure that we distribute them in the script format, but that's something I need to look at and double check, because typically the universal tasks are kind of integrated and through the marketplace, and it's a bundle package that you import in to do that, but the way that the scripts are designed is they can be run either way, which means that you can basically, with the agent, with the Python, or with the thing from the marketplace, yes, you can control that from a different schedule, a CA seven was mentioned in this case. So obviously, the input then wouldn't be through a form. Would be through, CA7 might be JCL, or it would be a script that you're providing the parameters in to do that. So basically, yes is the short answer to that for, I'd say, pretty much ninety to ninety five percent of the things that we talked about today, the Universal Data Mover Gateway, which does the integration into the cloud stuff, that's a separate component module product, if you like, is not dependent in any way on having agents or having the universal controller in place. So again, that can operate quite nicely with, and if your CA7 is able to do standard file transfer protocol jobs, then it can do the same things that I was showing in terms of being able to push data to and pull data from the cloud storage systems. I think that was all our questions. So I want to thank everyone once again for attending. We look forward to seeing you at our next session. That'll be on Thursday at the same time. As I said, it will be a customer speaker, so I think it'll be great. And otherwise, we hope you have a wonderful day and see you Thursday. Thanks, Thanks, everybody.