DevOps Strategies: Jobs-as-Code or GUI-Based Lifecycle Management?

Hello everybody and welcome back. I'm Nadia Davis and I will spend the next hour here with you moderating the session. Onto today's session. So far, over the last four weeks, we covered a lot. We've talked about enabling IT ops, cloud ops, data ops, platform ops with a centralized platform that allows teams to collaborate while enjoying the single pane of glass view into their hybrid IT environment. The one area of IT that we have not really covered so far, but we'll make sure we fix it today, is DevOps. So the topics of today's session is DevOps strategies and do you use jobs as code or would you rather go with a GUI based lifecycle management? So let me introduce today's speakers. I'm very excited to present to you Tim Holmberg of Donegal Insurance. Tim will be presenting something today that no other guest speaker has showcased so far on this series. Tim will be doing a live demo of the Stonebridge Universal Automation Center and how Donegal Insurance uses it to maximize the output and to achieve the best results when promoting jobs. Additionally, we have Colin Coxage, the Stonebridge Director of Product Management. Colin will kick off this session, once this intro part is done, and he will show a live demo as well. So, thank you so much for joining us today. And Colin, I will turn this over to you. Thank you, Nadia, and welcome everybody. So today's session, a lot of different things to sit and discuss, but I think when we talk about DevOps, it's definitely not a one size fits all scenario, So really what we'd like to do is kind of start with getting a feel for how your environments are. So we have a poll question up here, and please answer the poll: Do you currently develop and test UC updates in a lower environment, or do you do that directly in production? So are you moving the definitions between the environments? So A, select we have one or more data for test environments, or B, no, all updates are made directly in whichever environment they're needed in. Sorry, I got answers rolling in. I'll give it another maybe two seconds, and I will close this out. And Colin, I'm sure you'll be happy to hear this, but everybody on the session said, yes, we have one or more dev test environments. So you've got a crowd that knows what we're talking about. Right. So this should be relevant to the whole audience, which is great. And to be honest with you, not unexpected. So, okay, so let's kind of move in with a little bit of scene setting here. So when we talk about DevOps, it covers a couple of different things from, the UAC perspective, the Stonebranch perspective, right? So one part of it which we're not really going to be covering today, we're going to do the DevOps integration rather than the orchestration, is, you know, orchestrating DevOps toolchains. You know, we have a workload automation orchestration tool well suited for specific environments in helping you sort of bring all those different tools together under one umbrella for visibility and, you know, auditing and all of those governance and all of those things there. But what the focus here today is the other part of the strategy, which is the integration. So basically, you know, the definitions you have for automation orchestration are part of a development life cycle, they're part of the applications that you manage to support your business. So, you know, we need a strategy and you need a strategy as to how you're going to roll those updates through as you roll out updates to your applications. So basically, this session is really looking at the tools we have in place and the different strategies you can adopt, or consider adopting for automating those updates in concert with your CICD processes. So, you know, we have the built in DevOps, or the built in lifecycle management, the bundle and promotion feature, which Tim is going to spend some time talking about towards the end of this session. We have then the kind of jobs as code which involves, you know, definitions in, JSON format, which can be, you know, moved between environments in that format, stored externally in that format, updated in that format, and so on. And then we have a series of tools that we provide that help you do integration with the DevOps tools that you're using such as GitHub, so you know a command line, an API or a task integration to GitHub, and we'll discuss those as we kind of go through this and I'll be demonstrating some of those capabilities and talking about others as we go through. So for the built in DevOps or the Lifecycle Management App Bundler Promotion feature, you know this provides a number of features to allow you to select which workloads or which definitions are going to be moved between the different environments, whether it's a single object, a single object and all the things that it refers to, what we call follow references within the product, things that have been modified since a certain date, so you may have had a cut off since you did the last updates, want to pick up all the updates since you last promoted to a higher environment, you want to maybe want to do that by business service, or maybe you want to, you know, have something external come in and select what needs to be done, and we can do that by the APIs that we provide with the product. And then you've got the promotion itself, so the first one, the Workplace selection is, you know, it could be an immediate promotion, it could be creating what we call a bundle, which is effectively a list of objects that we want to move between the environments as part of our development update, and then the promotion itself, which can be immediate. You can do that through the UI, you can schedule it for a future date, you know, whether it's a single object or, you know, a bundle, a group list of objects that you want to do. You can trigger all of these through the APIs, and you also have the capability to take a snapshot and say, okay, I want to, when I do the scheduled promotion, I want to promote something in the future, but I want to continue updating it, so I want to take a snapshot of it is now even though I'm going promote it you know at five am tomorrow because that's when we're going to roll the updates out there. And then of course there's all of the protection around that, one of the things that we need to have is governance in terms of that, so there's all of the security, who's allowed to do this, you know, which the roles that cover the various, you know, whether I can create bundles based on my permissions on the objects within the system, whether I have authority to be the person that's doing the promotion, which environments I can do that through. And we've got all the auditing, the governance around that, who did what, what did they do, the versioning which is built into all of the definitions, which is automatically taken care of in all of the things that we're going to talk about today, but specifically in the built in lifecycle management, and the ability to back out changes. So if we push something into a higher environment and we decide that there are issues with it, we need to do some more adjustments in the lower environment, we can back out those changes and so on. So this is really part of what Tim is going to be covering, but that's the native capabilities built into the product there. Okay, so in the jobs as code world, it's, you know, the bundled promotion is really there, you know, through the UI, through the APIs, and, you know, automation through the scheduling, the jobs as code is we want to take that code outside of the product, make adjustments to it, version it separately, and then push it back into a controller. And there is also a hybrid approach, which I'll talk about on the next slide, but the things that we have within the product is really a set of tools and capabilities that support the different ways that people might want to do, you know, jobs as code. So the APIs are an integral part of this. This is what provides us the ability to get the definitions in and out of the product via, you know, in a JSON format, making sure that when we're pushing updates into a product that it still takes care of all the versioning, the security, the audit trail, and all of those important things that we still need to have our governance around this process. And you know, the APIs are there for all the definition types, including the bundle of promotion process, I'll kind of get to why that's important, because if you get into the hybrid approach that allows you to maintain that also as jobs as code, but you know to create, read, update, delete, list all the definitions. We also provide a number of tools, some of which are available now, some of which will be available shortly, some of the stuff I'm going be demonstrating is not quite out there, we still have a review process, some documentation to do, but we figured we'd introduce the concepts there to you. And that's a script integration of a local repository, the ability to go and select definitions to pull down, to put into your, you know, GitHub repository, so that you can work on those as a unit, and a number of ways that it does that, know, by business service, by workflow, you could even create a bundle and use that as the list of definitions that the script will pull out, but it also provides the ability to, you know, upload, push them back as well, and you could automate that as you push commits, and that's thing that I'm going to demonstrate today and we'll get into that in more detail, but as you push commits into your repository, you made updates into GitHub, you can then automate pushing those into whichever controller environment that you're developing into. And there's also a number of universal templates with universal tasks that we provide. There's a GitHub push pull task, it's a direct integration to and from controllers in the GitHub environment that's available in the marketplace today. There's also a webhook listener that allows you to provide automation based on, you know, webhooks that you set within your GitHub repository environment. Again, as you maybe push commits, you may want to automate that push and pull directly between GitHub and the controller environment. But one of the things that I think, you know, you've got that then the tools that you can do upon the promotion, the tools that you can integrate this into your CICD process, but one of the things that we've definitely seen is a hybrid approach, and this is an example from a customer I've been working with on this is, you know, they have a development environment and that's all their CICD development process, that's all the developers touch, they in their case use IBM's Open CodeDeploy to do this, GitHub, they use Microsoft's Visual Studio Code to actually, you know, code their updates and so on, and that's just nicely integrated through the API, that as they put commits, they're pushed into that development environment, and as they've done their testing, as they go through their processing, they can stage into their test environment and move that into production, but they do that via the inbuilt lifecycle management, the bundle promotion. But part of the code that they maintain then is that bundle definition, which they push up into development when they promote that bundle definition gets promoted, so it's specific for each deployment they have, the list of artifacts that they're moving between the environments is part of what they maintain as part of their project. So you certainly have a lot of capabilities and one thing that you know, we've learned in talking to customers about the DevOps process is that there are a lot of different flavors out there, a lot of different tools, even sometimes within the same company, different development groups working on different projects will have subtly different practices and use in some cases different tools to do effectively the same thing. So our approach and strategy is really to look at, okay, what are the building blocks that we can provide to the community that allow you to integrate with your process, your tool sets in a way that you know everyone can do it and you can do it based on the way that you want to do it rather than us say hey here's a process, you use our process, no it's more of a collaborative approach, so it will take some thought as you get into looking at moving forward with this, but the process that we have is that we look at providing those building blocks that you need to do that. So I'm going to ask a few more questions, we've got another poll before I get into my side of the demo, and that is who maintains your controller definitions? You know, in some cases we essentially a central scheduling group, that's more of the traditional approach that we have, you know, experts in the tool that are there providing, the updates for all the different groups there. Is it the development teams that do it directly or is it a hybrid approach where, you know, in some cases it's done centrally, you still have that central scheduling group, but some of the developer groups will do updates or they do some of their own updates in certain environments. So we'd like to get a feel for where the people here are at, so please provide us some information here. So right now I have the answers coming in. I'll give the folks probably another few seconds, and I'll be closing out this interesting. So forty seven percent, almost half, said that the central scheduling group maintains the UC definitions. Both central and developers, that's fifty percent, so half of you share that responsibility between central group and developers, and only six percent said that it's within the realm of the development team. So how does that come to you, Colin? Well, that's very interesting and I can certainly read things that we need more background information, but one of the things I'm potentially thinking is that you know bringing workload into that DevOps CICD process is a relatively new concept and I think that's something that you know a lot of companies haven't got there or started that project yet, they're intending to. It seems that there's a large number, know, with fifty percent, that are kind of doing the hybrid approach, so they dip their toe in that water, they've got that working maybe with some projects or certain applications, you know, and there's a small number who've gone one hundred percent it's all the development. I suspect, you know, if we come back next year and run this same poll, we'll see a different spread, we'll see a shift there, that would be my best guess. Okay, so I have a follow-up question there, that how are the definitions updated, you know, are they all done through the user interface, which is you know the traditional way you do and interact with the product, are you already doing the jobs as code, is there a hybrid approach there, or you know are you currently doing the user interface but want to move to a jobs as code approach. So this is kind of a follow on question, think will give us a little more information to see, you know, what's behind the results from the last poll, but give us the feedback. All right, so let's read the poll results. Fifty percent said that user interface is the way how they do it. Then we have the hybrid crowd at thirty six percent that said both. And then fourteen percent said current user interface, but wanting to move to jobs as code approach. And interestingly enough, nobody said jobs as code through API JSON format at all. So nobody does that. Okay, interesting. We have those that do that exclusively, So no one's there one hundred percent, right? So there's a mix of a hybrid approach there and obviously a small number that want to move to that. So, okay, I'm going to get through and actually kind of go through a little bit of a demo here, so we're going to switch to screen sharing. Okay, so the scenario I have here is I have a workflow that, I am maintaining externally to the product. I'm I have a GitHub repository where I store this workflow and all the different artifacts or definitions that relate to it, and I have a little task that I want to do, which is we go through the workflow and edit it and show it up here, I actually have an additional task that I want to to run on the end. So this is a little workflow that I use to update some of the environments that I use to to look at the development build. As the developers of the controller push new builds for the new development projects, I have an automated process that downloads those builds and runs them in a Docker container environment. I'd like to put a notification on the end of this so that when this process is run I get a nice little email that tells me that the new build has been deployed and a link that will take me directly to that. Okay, so that's my objective here. So if we look, I don't have any email tasks for the for the particular business service that this runs in. We can see in the workflow editor, I don't have the task on the end there. So let's go and look into my local, GitHub environment, so I'm running, GitHub desktop here. So I've got two changes queued up ready to go, I've already done the coding. The first one here is we can see the the email task, we can see type email task, it just contains the basic information I need to create that task, it's very simple. The body is just going to contain the link I need to that environment in that side, and it's going to send myself just that email. And then we have the workflow here, we can see the two bits of code that have been added, the bottom one here is adding the task into the workflow, and this one is creating the dependency for the new tasks that I've added to the tasks that I want it to be dependent on, so it sits there in there. So we can go ahead and push these updates directly, commit them to to my GitHub, and I'll do that through Visual Code Studio. So I need to stage them first, okay, to change his stage, and then we can go ahead and we can commit the changes. And demo and put today's date, so I have a reference, so that'll be the comment we'll see in the the GitHub repository there. So the changes are pushed. I will just sync them here so that it pushes up to my actual GitHub repository from the local repository now. On the activity side, the way I'm automating pushing the updates is that I have a file monitor running that is monitor or monitoring the GitHub log for, that specific repository. So I'm just looking for commits. And then in my environment, I'm kicking off a script that does the upload directly from the local repository to the controller, and we can see that that was kicked off, that ran, it ran successfully, we could go ahead and we can see the results of that, we can see it's pushed up the changes, tells me which versions are now at for all the different things that have been updated there from that site. So if we go, we should now see that we have a new email task, so with the bottom half of the screen we can see the very simple email task filled in with the information I had in the code that I created to do that and if we reopen the workflow we should see very nicely, if I make this full screen, that we have the new task that's been added in the right place nicely at the end of that, that definition. So it gives you an example idea that, you know, the tools are there, whether you use the the script implementation, which I did, or whether you use the direct GitHub repository implementation, and you automate via the file monitor as I do, suits my purposes, or whether you use the webhook to do that, you have that capability there that as you push commits into your projects, you could get that information into your dev environment, there. You can also use these tools to push those into the other environments directly from your GitHub repositories if you so wish, or you could take the hybrid approach, which would be then once you've got them into the development environment using the integrations with GitHub, you can then move on, and have a process where you promote those and you know, put that into notifications or approvals through external tools you may use such as Slack or Teams, those kind of things. We have universal tasks available in the marketplace that provide you levels of integrations to do those kind of things as well. And now I'd like to introduce to you Tim Holmberg. Tim comes from the Donegal Insurance Group and he's very kindly joining us today to walk through his process and his setup for the built in lifecycle management, the bundle of promotions. Tim, I'm going to hand this over to you, thank you very much. Thank you Colin. Good morning everyone, my name is Tim and I am an agency automation or excuse me, a automation specialist here at Donegal Insurance Group. And we've been using UAC now for about eight to ten years here. It started out as a scheduling system to replace CA7 on our ZOS mainframe system. Our mainframe processing was becoming more and more dependent on processes taking place outside of ZOS and so we needed a new solution to automate those processes from both within ZOS as well as outside of ZOS and UAC has fit that role perfectly. Soon our application developers heard about the new automation system that we were using and their request for creating more and more automation for other applications came pouring in. There was only two or three of us doing the automation at the time and so we became overwhelmed pretty quickly with all the different requests. It was decided that the best option to handle all the requests here for automation was to provide the developers with use some UAC training and let them automate their own processes. And then we also created at the time we were running a single controller environment because again it was just mostly for mainframe and that we had some different definitions for testing and production inside that one controller but for the most part it was just a single controller. So we broke our systems out into, let's see if I can share my screen here, out into multiple environments. So we have a development now, a test environment, QA pre prod environment and then we have our production environment. Each one the developers are less, more and more restricted of what they can and can't do in those environments. Development's kind of a free for all, they can pretty much do whatever they want and that's you know, because that's where they do their development at and such. Then as they can promote their own tasks using bundles up to test and start doing, you know, any kind of testing they need to do and such. And then again, it goes a little more restrictive when it gets to QA And in production they can't really do anything in production. Everything is, it's automated or they need to have a UAC administrator help them with something if they need something done in there. We do allow them a few control functions in there. If something fails in production, they have the ability to go in and restart it themselves after they've correct whatever the issue was or force finish it and let processing continue. But that's not about the only thing they can do, there's not much else they can do in that. So one of the things we did also is, as a lot of you know, you can promote by individual tasks if you want to or you can promote by bundle. We didn't want the developers to promoting individual tasks too often although I think we do allow them on development to do that. Mostly everything we force them via permissions to promote via bundle only. And then also, we also set up restrictions, everybody has their own business service group, So all the warehouse people are in one group and they can only work with the tasks that are designated as warehouse, they can't work with any other ones. There are a few overlapping systems and in those cases we just identify those tasks for both business services groups such as enterprise systems and warehouse and then that way then either one can work with those products and stuff. So we first, you know, had people to show them the differences in how to promote up through the different controllers. One of the first things you have to do when you're going to promote and a lot of you know this, essentially a lot of you are already promoting is you have to define promotion targets for each environment. So on development, we target the promotions to the test. On test, promote, you know, target the QA environment but we also always give them a path back to development. So if maybe they got up to QA and found that something wasn't working right with their task, they couldn't make a change if they needed to, if they didn't wanna come back quickly to promote it back up to the QA, they could make the change there and then promote that change back to development and that's the only direction they can go. Everything is always a forward promotion and but you can always promote back to development. And that way then, you know, if they need to make an on the fly change, something failed and some of these systems are only available in certain ones. We don't have some of our production, we have fifty something agents in production but we don't have all of those available in development test in QA. So some of these things, only time you can actually find out how they work is once they get to production and see how they work and if there is an issue that's encountered, then administrators would usually help them if it was a production environment, make the change they need to get the task complete successfully and then we would promote that change back to development so that it actually has a starting point that they can, you know, again redevelop that and bring that back up through the different environments. So the other problem is, the other thing that we had to stress with our developers is that when they start building their workflows and such, they have to be careful as to what they put on their workflows. Lot of them decided that they wanted to start stacking workflows one inside the other, the other and we kind of warned them about the consequences of doing that but they wanted to keep doing that anyway because that was for them, that was an easiest way to maintain it. So they had some of these mega workflows, they had, like you guys said, workflows embedded inside of workflows, embedded inside of workflows and the problem with that is is when anybody makes a change to a high end workflow, it will grab everything in there. So like for instance, if we go over here, I just have a simple workflow here. If I add the workflow for well, let's try this one. Just one workflow added and if we actually come out here now and use the bundle, we can see that actually grabs all that just by adding the one workflow and actually most of you know that but the developers, we had kind of stress. You can see this is the one that I actually added and everything else was added by association. They were childs of this workflow. So we try to encourage the developers to add all the tasks involved in their workflow just so that they, it's a lot easier to reference than what is getting pulled in by, So if they add, you know, the this this, it certainly makes it a lot easier if they put those in there then that way then when they actually look at their bundle promotion, they can see and then say, oh, hey, this is getting pulled in and I didn't make that change. So then they might need to take a look at why, what that effect is gonna be. Was there something on there that wasn't ready to go to test or QA or production and that happens a lot. Somebody's working on this part of a workflow, somebody else is working on another part of a workflow. One person's ready to promote their part but the other person's not but because of the changes the one person is doing, the other person gets pulled up to the next level unintentionally and when it wasn't ready. So one of the things that they did there and I always, we always encourage the developers to always look at the bundle promotions report before they actually do promotions because that'll definitely tell you everything you need to know about what's gonna happen if you promote that bundle. Also another thing we had to do is get people to not use the promote members of business service. You know, if you go in there and set that to demo and that it will actually pull everything in that's associated with the business service called demo and there's been a few of them that has done that and so luckily there's a way to revert that change back because that is obviously something they didn't intend to do, they just accidentally changed instead of this, they set that one. And so you definitely don't wanna do that if you don't intend to do that. Most of the time in our case, that is not an intentional thing. We try to keep people from using that at all and I think some people have requested their ability to disable that field and we certainly are one of them that would like to do that because there's no reason for them to use that. So here we have our single task and then we look again and it's back to its normal there and we want to go ahead and promote that up to production. So we just promote, most of the time we never have to, for developers, they don't have access to the templates for universal tasks and as long as the versions are identical on those and we usually promote those up if we ever make a change, there's no reason for them to have to check that box. It'll probably fail for them anyway because they don't have the authority to, the permissions to promote universal templates. If they want to schedule it, they can and, but they can just go ahead and just that fast that, well actually Power BI, as you can see there is currently not on the other environment. That was a one we've been only working with on development. So in this case, it's telling me that I need to actually go in and when I promote this is to check that box because there is a, and in there the promotion completed successfully, there was a new template update to the Power BI template that was not on the test yet. So if we go over here on test and look out here now and there's our new test we just promoted out there. One of the other things that we had a problem with is that some of the developers, they didn't understand about the need to promote things on each environment as opposed to just creating it new, especially for things like a database connection. Here's one we have that's just a demo database. It's a demo D, you know, for the database it's referencing. And out here on test, I created this individually and it looks identical to the other one other than it's a demo T database but the problem is, is I just created it so if anybody who's ever worked with UAC and I think everyone here probably has, behind the scenes there's a UUID, universal unique identifier. And this identifier is on every item that's in there. This is right here, there's some other ones that reference different things but this is one right here and this is going to be your key to how everything promotes in UAC. This one's E8FF, if I go out and actually look at this one, it is, it's not the same. So there was a task here that I had, this was the one right here that is actually referencing that database but when I promoted that up here on if I go to that same one again, you're going to see a problem here is that that database, even though the database is actually out there, it's not the same database, even though it looks like it's the same database, it's not because it was not promoted from development to test, somebody just created it standalone and that's gonna obviously break your promotion and this isn't gonna work. So the preferred method is to, we'll get rid of this one and over here on our database, we'll just promote this one up. And so our database appears, what we do then is go out there and just modify the task but then if we go over to look at our task and reference it, it's now resolving the database because it sees it there and that's the same with QA and production. You always wanna keep those UUIDs in sync all the way across because that can cause you all kinds of problems if you don't, it just isn't gonna work. That's the same with credentials, databases, even tasks themselves. You can actually promote a task that have different UUIDs. As long as the task name matches, the promotion will match it up and promote that task. For instance, if I go out here to like, well, just take the robocopy one here and make a copy of it, and we actually found a couple of developers were doing this, is they would come out here, this would be a the new task that they created or and everything looks the same as the other one. They would just come out here, make a new copy, change that. They didn't wanna change the original until they knew they had the new task working correctly. Then they'd go out here, delete the original task and that one's on the workflow so you can't, gotta temporarily take it off the workflow. So yeah, here's our demo workflow. So just take that off there. So we'll add the new task out here. Where's our demo one? There's our copy, copy one. So we'll take the old one off, put the new one in its place, attach it. Looks like it's all gonna work fine. What they would do then is go over here to let's save that first and go over here. Don't need that anymore. They would go out here delete this one and then they would just rename their new copy to be the copy one and that used in a single control environment that actually works fine. But in here it's not so much. You look at it out on here, everything looks normal, it's actually changed the name on the workflow but when you get to go promoting this task, it will actually even though it's, you know, this one's got, let's put the task new, new in here and we'll pull up the other one over here on here for robocopy which is just our other task. If you can actually promote this, I'm gonna promote it individually. And it'll promote fine everything it says it promoted and you come up here, you refresh your task and look at it and there's our new new. The only problem is that it has a different UUID now than the other one. So if I go over here and I create a task monitor, this is one of the other things that used to get the developers in trouble, is somebody later on, this could go on for a year or two that nobody knew this was done or none of the UAC administrators would know this was done until somebody comes along. They want to create a new task and we'll just give it to demo, demo. And we'll go down here, do the Robocopy one, we'll save it. Forgot to specify the field here. I had to give it some things to watch for finishing success. Save it, there's our new task and it'll work fine over here on development as long as you run it over here. You'll see here's your demo task, I mean I could run it, launch the task, it'll run, it'll watch for the actually got two incidents running because I launched it twice. But it'll run and it'll be just fine. You go to promote this task up because of the difference in UUIDs, I promote the task, come up here, you take a look at it over here, and as you can see now, down here, that task doesn't exist on there because even though the task did promote successfully and there it is right there, it doesn't have the same UUID, this is the UUID of the task as it was on development but that is not what the UUID is on test. The problem is also is that you can launch this task with that missing and you'll see that it'll run just fine and watch a task that currently doesn't exist which that's caught us before too because then we're sitting around wondering why is this process not completing, why is this task monitored? We know the task completed and yet on this environment but yet it's the task monitor is still monitoring and then of course when we open it up and take a look at it, we realize we see the task it's monitoring is blank. It's just, it's not there, the task doesn't exist. So that task is never going to complete as it is right now and then you have to go back and fix that by getting the same, the UUIDs back in synchronization. So one of the biggest keys is maintain UUID synchronization. It can cause a lot of problems. We stress that all the time to the developers but again we always get somebody every now and then that doesn't understand the difference. And with that also then let's say in the future decide you want to change this one to copy ninety nine and promote it. And so there's our copy, we're gonna promote it up to test again. So now this one now, not only is it got a different UUID but it also has a different task name. So what's gonna happen is is that UAC is gonna interpret it as a new task and so when we come out here instead of updating the task we wanted it to update, it So it would have actually created a new task. I'm not sure why it's giving me an error here or something about a lock. Do I have it open over here somewhere? I don't think so. But it would have created a new task because it doesn't match up with this task name and it doesn't match the UUID so it would have created a new task altogether. So if you just maintain the UUID synchronization across all your development test, QA and production, these kind of issues never pop up because they're hard to find once they do happen and they're hard to notice. Again, like I said, if until somebody came along and did something like create a task monitor, you wouldn't have any idea that this issue even happened until maybe it could be a couple of years, two, three years later which has actually happened for us. So that's pretty much all I wanted to show there is just to kind of give a quick demonstration there and show you that process, the credentials and such. We usually try to recommend don't, you know, database and such, don't put them as part of a bundle just because we'd like them to instead be on the on an individual or a separate bundle. Just because if you have your task and you need to re promote them multiple times, you could override, you know, because you have to change your details on each environment, you're gonna override that with development if you promote like a database connection up to test, you're gonna override it with what's on development. So that's pretty much all I had there for that. Colin, I can return things back to you. Well, you, Tim. So I have one comment. Some of the things you mentioned regarding, you know, the things like credentials or database connections and so on, one strategy to handle that is to use variables so that you basically have each environment has effectively a set of global variables that control, you know, which credentials are being used, which connections are being used, and so on. That way when you promote, there's no references to follow, it's just a matter of, you know, so it's one strategy, I know that some customers use that as a strategy. Right, and we do that right now actually with the agents because our promotions, again, because we don't have agents on every controller that match up with each other, in the promotions target, I didn't show that, but you can actually map promotion targets by, so if it's ZOSD on your development, it would automatically transition the agent to ZOST on there. But we use variables for just that exact reason. So we kinda did, you know, probably should use variables on some of that but some things we did use the variables and some things we did not. So each one works fine. Task monitors, that's one other thing too. You can actually switch it over instead of a specific task, you can go after a task name and that's a way of solution around that too. It just depends on what the developers are doing. It's just things that have kind of got been a gotcha for us. Okay, no, useful information, think good to share with the audience, you know, a, they may already be using this and weren't aware of some of the potential gotchas, as you say, or people that are considering what their strategy is gives them good food for thought. So, you know, we've covered a lot of information here, but, you know, the basic thing is that there is a lot of different options that we provide that hopefully will suit whatever the development process, the DevOps process that different organizations have, and you can basically, you know, develop a strategy that suits your methodologies, that suits your processes there, you know, using the different capabilities options that we provide there. So I think we're open for questions, Nadia. Yeah, let me go ahead and open the Q and A. So you guys can submit the questions through the Q and A tab, and I already see some people asking, so go ahead and do that. And while we're waiting, I want to thank Tim for a very in-depth demo. And one thing that you mentioned, which I think in a lot of cases kind of comes unnoticed, is the fact that the rules of engagement have to be socialized across the entire business. They have to be communicated to different team members because about six months ago, we did an in-depth research on the evolution of DevOps. And one of the prominent problems that came up on multiple occasions, and we heard that from the analysts as well is the multiple acts of automation, or we called it random acts of automation. So duplicate automation that happens where developers use it as a means to an end within their environment without realizing how it impacts the entire environment for everybody else, right? If you have multiple teams collaborating. So you stressing on that, I think it pretty much validated that point that that's a real case, something that you guys see. So thank you for that. And we have Ron who actually got a comment for you Tim. He said, hi Tim, you can set exclude on existence for the credentials within the bundle to prevent the problem that you mentioned. So that's a suggestion that putting up on the So yeah, there is a feature within the product and you can set a default for that though. Yeah, good point. You can prevent certain types of definitions being included in a bundle, you know, obvious candidates for that are things like credentials and connections, because they tend to be environment specific and not what you want to promote between environments. But of course, then you have to have consistent naming across the different environments, which often isn't the case, because sometimes the name will contain something related to the environment or a different server if it's a credential potentially, so you know the variable approach is one, the exclude on existence really depends on how you've chosen to set the product up or what your requirements are there, so yeah, a lot of options to, a lot of ways to skim the same rabbit. All right, actually today we have a more, actually a less vocal crowd than last time. I know last time we did not even get through all of the questions because we had an avalanche. So looks like this is pretty much it for today. Gentlemen, thank you so much for your time. Thank you for today's participation. I'll close out today's session. This is StoneBranch Online. Thank you.