A CTIO’s POV: Automating Operations at Makedonski Telekom

Hello, everyone. Thank you for joining and welcome to the fifth session of Stone Branch Online twenty twenty one, a global IT automation and orchestration forum to get you ready for whatever comes next. I'm Lauren Tanzini, marketing manager at StoneBranch, and I will be moderating today. So let me introduce today's session presented from a CIO's point of view on how automating operations at Macadonsky Telecom improved efficiencies, reduced costs, and strain on internal teams. Please help me welcome today's speakers, Miroslav Yovanovitch, the CIO of Makodansky Telecom at T Mobile Company and Sebastiano Mian, VP of Technology Services at Stonebranch EMEA. With all that said, let's move on to today's session. Siva, over to you. Right. Thank you. Thank you, Lauren. Hello, everybody. So, okay, today we're going to be speaking, we're going to be touching a few topics together with Miro's Lab. We're to talk, of course, about the managed operations. We're going to talk about automation, of course. And we're going to touch on the very nice use case that we've got with Macedonsky Telco, which is a subsidiary of Deutsche Telekom. So, okay, let's just start with a little icebreaker, because we noticed that you guys like polls. So here you go, guys. So if you'd like to answer this apparently simple question, right? So what's your company's biggest driver for automating IT operations? Is about speed, velocity? Is it about accuracy, response and robustness? Is it about scalability? Or is it all of the above? All right, we'll go ahead and give it a few more seconds and I will go ahead and close the poll. So we actually have an overwhelming number with eighty nine percent said all of the above. How does that sound to you, Siva? Yeah, I think it sounds about right, think, which lead us neatly to the next slide, which is actually what we consider, right, the drivers to automate operations. And of course, are the three of them, right? So the first one, and this is not in terms of importance, it's just an order, right, an arbitrary order. So speed, of course, right, we need through IT, we need to enable the organization to deliver faster, both internal and outside customers. You know, the thing was a time to market. We need robustness, we need accuracy. IT operations require strict KPIs that are very demanding SLAs and you cannot make mistakes, because sometimes they're also subject to penalty as well, right? And third but not last but not least is, you know, the IT landscape is becoming more complex. There are a lot of applications, servers are on cloud, on hybrid, on premises, but also we are dealing with some resources like time and people, which are by definition scarce. And that's another very important reason why we need something to automate everything and make it more scalable. Why UAC then? Of course, right, when we, for us, of course, was a no brainer to implement UAC in such a project and it helps on addressing all the challenges that we saw before, right? First of all, there is the GUI that allows us to focal point for a single view across all the servers, all the applications that are running. We have all the alerts and analytics centralized in a console. Another very important point, is we are building more and more is integration rise and the capacity to easily integrate with other applications. As you know, the IT tool chain is becoming more and more complex and more and more every day there are new applications are being introduced and you need to have a system which is flexible enough to enable integrating with whatever comes up fast, in a faster, robust way. And now, especially UAC now is very easy through the universal extensions, right? Of course, real time automation, again, and this is more or less real time and automation, generally speaking, we need a scheduling based on events, on time or anything. And that of course will allow to reduce all kind of, or at least all kind of manual activities that are by definition prone to error, right? And let's not forget also the fact that we minimize repetitive tasks, that's also a good factor in terms of, for people, right, for people satisfaction, for employee satisfaction. And finally, the fact that we have the central console enables actually the connecting different teams, different applications, which in turn offers an end to end value chain to the whole company. So now we are the main topic here, of course, is introducing the specific use case, which is our experience in Macedonsky Telecom, where well, first of all, a few words on Macedonianki Telecom as an operator. It is a subsidiary of Deutsche Telekom Group. It is the dominant operator in North Macedonia with across all segments, as you can see, half a million postpaid subscribers, half a million prepaid and two hundred ks fixed. Makdoski Telecom is following the blueprint of Deutsche Telekom, which was to outsource the legacy IT operations while insourcing some DevOps innovative digital services. So what was the scope that was demanded from us? Had, Macronovski Telecom had eighteen different systems that they had to outsource. Always, we had to deliver a regular operational maintenance, server requests, instant management. We had to, of course, deal with the 3PP for line three, right? And so to vendor management on that and everything again under strict SLAs and KPIs. What we did as part of the outsourcing was also to take over, at least on paper twenty six FTEs and twenty eight mandates of recurring CRs. The challenge from Macedosky Telecom side was that what we said, right, follow the blueprint. That means to outsource legacy and focus on digital transformation. Also another requirement for us was to improve operational quality, right, through DevOps and automation, which is something that as a legacy organization, they weren't really focusing on, right? And of course, important nowadays is the reduction of attrition, right, which is a key point right now in all IT landscapes, right? As you all know, IT people is very much in demand these days. And again, selected Stonebrands. And why did they selected us? Well, of course, Miroslav will confirm, I hope later. Stormbranch was a candidate vendor which, of course, a very, very automation centric strategy, of course, right? Because we are a software builder which builds an automation controller, right? We of course have established DevOps tools and expertise that would allow this rapid deployment of any new software and application. We had long expertise in orchestration, in automation, and generally speaking, in digital transformations. And finally, of course, important, we cannot deny that the relationship with the customer is also very important and that helped in successfully closing the deal and in the overall success of the first year of experience. So okay, now let me hand over to Miroslav, who is the CTO of Makenovski Telecom. And Miroslav, this is over to you then. Thank you. Thank you, Seba. Practically, you have put it a very good opening on the part what we have done in this managed service part. We had, as Seva mentioned, quite high attrition rate. I mean, even we are discussing about the pre COVID time. So people were leaving the company, very difficult from the market to fill up the positions, you know, you have to retrain the people and then put them in production, particularly where they will deliver the things which are expected to be delivered. And very difficult, I mean, for us as IT to deliver the things and to make the business happy with the new products on time and to follow the market changes which are happening. So that's why we have decided also to go with managed services or outsource of the legacy. But at the same time, Deutsche Telekom was introducing a new strategy where we are focusing on internal development of micro services. So the stack should be practically built on micro services to follow-up the new technologies and the digitalization of the portfolio and the markets all over the Deutsche Telekom footprint, especially in Europe, because we are belonging to the European segment of Deutsche Telekom as such. So we started preparing the whole thing and discussing with possibilities how we are going to enter the whole deal. The idea was also to discuss and introduce some of the, let's say, well known world big companies who are dealing with managed services or outsourced part, but also we have discussed how to introduce some of the partners with whom we have worked quite long time with experience. And we knew that actually as Makilonsky Telecom, I mean, you see the subscriber base, are having a clue that this is a small country and a small operation. Yes, it's a part of Deutsche Telekom, but still not so, how to say interesting for the bigger companies to enter the market because we have also some specific things to put into the tender, which was also related to the local labor law, and also some specifics which are, how to say, for North Macedonia actually as a non new member country. So we have introduced, by the way, not only the big companies, but also some companies we thought that actually can enter in the field of outsourcing part and splitting the whole IT into four categories actually into four domains, which were possible to be outsourced. Could be either whole deal or could be part by part. On the end, we end up as a part by part. Fortunately, we have done the billing domain with Stonebranch. It's one of the reliable partner with which we have a very positive experience in the past. And as Seva mentioned, I can confirm that actually it was a very good, how to say, deal for us, but also for the Stone branch in the whole period of the entering into the deal by today, and I hope that this will continue in the following years as well, because this is a long term contract and partnership with Storm Branch in this area. So what was so specific also for us, I mean, so, you know, when you're making an outsource or managed services, you also have to take care about the people. Mean, so this is something which you have to find a sourcing partner who will be reliable, who will be attractive also for our employees and willing to come to the new company. And also, one of the prerequisite that we put it, as I said, I mean, this is also connected with our labor law, is actually that the company should have presence in the country. So this was something which was very important for us. And also, it was a good opportunity for Stone Branch to make a local office in Macedonia, which not only will give services for us, I mean, Macedonsky Telecom, it will be a much easier, let's say, entrance into the Deutsche Telekom footprint of Europe, but also due to the takeover of the people who have deep telecom knowledge also to have to extend their capabilities on the area of billing domain, also for other operators where there is opportunity for such things. Because even in the past, I mean, we at Zamakenosky Telecom, we were giving some additional services to our sister companies who are in the group. So this was also transferred to StepStone branch to continue as such. What is also important for us, although what's important and is also important for Stonebridge, is actually that we have a dedicated contract, which means that we have spoke or, let's say, manager who is dedicated to Macedonoski Telecom and we have a common governance model. And this is very important for us, but also I think for Stonebridge to have a clear communication and clear possibility to practically run the deal and also to have a delivery which will be satisfactory for both sides. Also, what was important, I mean, we have discussed even in the past with Stonebridge, this was the part of automation, which is also always connected with DevOps principles, which will be introduced in the legacy stack, but also to reconnect it with the new stack that we were internally developed because, you know, somehow when we speak about legacy, we are not speaking this in a negative tone, but actually it's a very crucial part of the operation what we are having. And the connection between the new and old is also something or the legacy is something which you have to take care to be practically fully operated and extended, I mean, not only for us but also for the capabilities in the Deutsche Telekom Group because there is a lot of other our sister companies who are actually having a similar product portfolio in the billing domain. Also, I mean, when we are discussing about the target digital stack, which is actually what we are building, I mean, now for two or two and two and a half years, we have to also take over the and overcome the challenges that we are having in the bimodal IT. I mean, this is the legacy part, but also the new thing. So you have to make a good connection between the operations of the old and the new one. And also, I mean, when you're having these things, I mean, some of the things will be removed from the legacy operation and built into the new stack. This is mainly in the CRM domain, but still this have to be connected with the billing domain, which is already given to StoneBranch for operation and developments. As I said, I mean, the big value of StoneBranch is actually the automation tools, which practically were introduced and improved significantly during this year and here in something, kind of year and a half, almost two years, will be practically in end of this year to improve their operation, but also to integrate it with our also DevOps principles that we are doing on new stack. So automation can be easily also extended to the new stack as well. Having this in mind, we also during the time, I mean, we are having also discussion how to make the improvements, I guess, how, because the idea or the, let's say, benefit of the both companies is actually how to increase the automation, decrease the number, let's say, of errors which can be because of the manual in other parts, let's say, either configuration or, I don't know, operation which is practically making some possibility for errors. And also to practically increase the automation in a way to make the skillful people more open for the new things to be done, but also to eventually jump in into the some other projects which will be in the area of our managed services because, as I mentioned, it's not only just operation, but we have outsourced also the development part as well, but also to build the people for eventually additional possibilities on the market or operators who have the similar or let's say same problem that have to be solved. That's why, I mean, a lot of processes were automated and workflows with the UAC. So this was proven that it's a very good way how to how to be done. I think later on, will see also what how this was done and what are the benefits for the STONG branch, but also for us as well. I mean, because we have practically automated many of the things which were previously done manually with a lot of, let's say, overtime hours, with a lot of, let's say, possibilities for errors which can occur during the process because you have to, you know, it's a interactive process in a way that you are waiting for the results of the previous operation, etcetera, and you have to continue. It took a lot of, let's say, hours of hours of our people in the past. Now, it's most of it is practically fully operated with script, with platform, and then practically we can also predict and quickly recover if something unexpected happens. So not only that, I mean, this was also put it into tools what we are jointly using, like JIT, I mean, for example, this is version control. So everything what we are making at the configuration management, whether this is a source release or change release lifecycle, this is practically put into version control, fully controlled and automated deployment for the Ansible playbook. Of course, I mean, it's a very important part of the things is actually to take care about the incident management. I mean, this is one of the things how you are going to make management of the trouble ticketing part and SLA monitoring, because this is in a common benefit for the both parties to be fulfilled according to what we have agreed and what we are making a review on. Because, you know, these things are something which you are not putting only once, mean, you are making yearly reviews, for example. Something will be changed, something will be improved, something will be discussed depending also focuses what we are having on in our, let's say, operations and our company that what we are developing or what we are trying to achieve, what kind of goals and targets will be put in there, because it's a changeable category. So this also goes very well in coordination with the SoundBench part. Automatization on the fault management is also one of the things which brought a big benefit for us. And of course, I mean, as I said, I mean, the story about the continuous integration and between the stack of what we are having in the new portfolio and also what is put it in this legacy part. So if we go to the next slide, I mean, this is practically an overview what have been done. I mean, the part where we are negotiating and also putting the documents is not putting on this slide, but we're actually putting when we have put it up, letter of intent actually to sign. This was the ramp up phase and ended somewhere end of November. We had transition period of phase one in December twenty nineteen, so this is still pre COVID time. So you can imagine that everything was done in a relatively peaceful time, but still in a very time constrained period where actually everything was practically pushed into one year to be done. This month was actually where we were working more or less on the very old way, but it was practical introduction of the transition of the people to the new company, but also to Stonebridge to make, let's say, a little bit more to be acquainted with our processes, how we are dealing also with the business. Nevertheless, we are having very good cooperation in the past with the Stone branch, this is also continuing the following days as well. I mean, we put it also in transition period phase, which was practically planned for six months, and then the COVID came in. So this is something which also make a little bit more difficult to make the transition. And also, the additional part is actually that we have split our managed services into, let's say, multiple service providers. And the third one, the CRM part, was also done end of February and March. That's why we have extended a little bit this initial period of live operation and transition phase a little bit longer. And practically starting from June, I would say July twenty twenty, full managed operation with the SLAs in place. So this means that practically all the systems are fully operated and developed by Stone Branch, all 3PP contracts, all DSLAs that we have agreed now, which have been also measured during the transition period of phase two, are now put it in place and practically fully engaged as should be. So practically all the information system and IT in this domain was fully transferred to and operated by Stonebridge. And this is continuing today. We are very happy with what we have achieved here. It's a lot of automation, a lot of, let's say, on time delivery on everything came in. People are quite skillful because we also gave a lot of, let's say, personnel which was actually knowledgeable about processes, about the systems we are having. But the value add, what Stormbranch put it in the automation, also on their experience in the area of the billing, and it was something which also significantly improved our delivery. What is also important that not only us as IT and technology are happy with this, but also our business is happy. And you know, it's very difficult to make the business happy, I mean, so this makes me additionally, how to say, satisfied that at least one headache is less in whole story because it's not so easy to achieve this kind of things. Today, are also operating in this area with Stormbridge. Nevertheless, that we had difficulties in a way that it was actually something in the COVID period, you know, we had to change. The people were not in the offices. They were working remotely. But all the results what we have achieved in twenty twenty was something which we think that was very successful, short term from the company results, and we are happy to continue in the next period as well. All right. We have our next poll. I'll go ahead and start it. What main reasons would compel you to use managed services for workload automation and orchestration as opposed to doing it in house? Is it cost reduction, improve operational quality from the vendor's best practices and industry experience, improve IT staff job satisfaction while driving down staff attrition, reduce errors while driving efficiency, or all of the above. We see some responses coming in so we'll take a few more seconds. I'm going go ahead and close the poll. It looks like we have sixty percent say all of the above. Siva, how does that sound to you? Yeah, yeah, yeah. I mean, yes, I understand why there wasn't such a, know, downright my job like the other one, right? It's a bit more complex scenario, this one here. But yeah, I mean, all of those are applicable, right? So now what are and thank you Miraslov for your presentation, for your nice words as well. So what we are gonna look at now, you saw from Miroslav's slide before, right? I mean, it is a relatively short time that we are doing this, right? So it's one year and a half that we took over the managed of the operation, the IT operations of a very complex domain in Macedonsky Telecom, which like I said, right, is almost twenty different applications. So of course we haven't managed to automate everything, but we've started and we've automated a lot, many processes. And I just wanted to give you a taste of what we've done, right, without getting into much details, but just to have an idea of what we are looking at. So I'm going to present the three little cases. So one that was done was actually on the online charging workflow. I hope there will be there is people familiar with telco terminology here, but anyway, so on the online charging workflow, what we're doing is monitoring the backup logs and transferring over in order to see that if the backup was correct. So we're parsing logs to see if everything was okay. And then if the backup was completed successfully, then we are transferring this information through, of course, we are pushing notification right to the StormRash portal, to Marketosky Telecom, and of course moving and backing up the log files that are relevant to the activity. So this one was automated pretty early in our experience. Another workflow, again, specific is pre billing. So, you know, the billing execution, of course, as you can understand in a telco operator is a crucial process. And normally before that, there are many, many checks that take place and most of them are done manually, right? And you can see here just by looking at the left side of the slide, how many of these checks we are talking about, right? So there are a variety of SQL Unix scripts that need to be run one depending on the other, and all of this was automated, right? And all of these allowed to reduce the execution time by two thirds, right? Of course, every task is also associated and the successful execution of this task is associated with a notification and the overall workflow success or unsuccessful execution has got a notification as well. Another example is also another billing example. This is actually the bill printing, so after the bill process, before it was a pre billing, now we're talking about the post billing process. Again, the complexity of this one was because there are a variety of processes and application that need to be started depending on a variety of constraints, right? So as you can see here, so there are first some SQL procedures that need to be run depending on some data checks and some other billing checks, which in trick in afterwards, need to run some Unix script. And then in turn we need to run the Talend process. You may know about Talend, it's an open source ETL, which depends on all the above, right, until we reach again the execution of the actual bill printing process, right? So this is again, without going too much details, it is just to show you how this is actually in progress right now, we're about to finish. This is just to show you how USC can be used, and it is used to automate and coordinate a variety of processes, applications, and so forth, right? So again, these are just some examples of what we've been doing and we are continuing regularly trying to understand the processes and put some automation in place. Okay, now just, this is the last section of the presentation. What we've been discussing so far was really how, and I hope it is of interest on anybody of you who is dealing with managed operations and understanding how USC can help in IT operations and in a managed operations context. Now, the next slide is actually a proposition that we put together now in order to leverage on what we have learned, right, with MacKinowski Telegram, with our experience, and trying to basically offer managed services for UAC. That means we are taking over the operations of UAC. So what we're looking here is as, like I said, right, basically Stormrage will take over the workload automation with UAC. So we'll take over, it's gonna be a turnkey offering, right? With the all in software subscription, managed operations, again, course, KPI and SLA based. And if anybody needs that, we can take over also the staff of the client or the customers that is actually dealing with workload automation at the moment. Of course, we're going to deliver what we are delivering right now, right? Which is operations, maintenance, health checks, service requests, change requests, and so on. And we're going to keep be sticking to our best practices, which is DevOps, we are following ITIL, and of course, we're gonna be leveraging on our very experienced center in Macedonia. What we're proposing here is actually two models. So one is basically we cover, and now of course, I'm talking about a case in which we are converting into UBC, right, from a different platform, or for greenfield if necessary, right? So we will basically build the UAC, right, as a project. We're going to operate it on the customer behalf for a period, and then we can just transfer it back to the customer. Another option that we are putting on the table is for mostly for customers that are a bit defensive or concerned about a transformation project, will be to actually hand out the legacy WLA team with the legacy scheduler or application they are using. And then StormBrends then will start operating on the legacy scheduler and then taking over also the conversion to UAC, right? Until then, of course, then we continue moving forwards in the same way operating once we are live and then transfer it back if necessary. So just to visualize how high level a project plan like that would work, it will be a moment in which there will be so you can see here in blue is what is expected from Stoneware side and in, what is this color, some purplish stuff from customer. So so of course, that will be right. The requirements collection and the conversion is required on both sides. The implementation of SIT is of course taking over from Stormbranch and the 3PP part, of course, we need the full customer on board on that. The operations again is taken over by Stonebrands and together with the customer as well, right? Because we will need the support on that. Until then, we to switch it to operations, which is completely taken over by Stonebrands. And then in case it is opted that way, it is transferred back to the operator while Storm Brands maintains the support and maintenance. So yeah, so this is a proposition that we're putting in place to leverage on the success and the experience that we gained from our experience in North Macedonia and the excellent team that we found there and we keep empowering. And this is it, I think. Thank you very much. All right, we'll go ahead and start with Q and A. We have quite a few questions. All right, the first question: Would it be possible to share with us the biggest challenges you were facing during the implementation of UAC? Okay, I'll take this one, right? The biggest challenge, the biggest theoretical challenge was that, of course, the team that we took over didn't know about UAC, right? So it was just a matter of knowledge. But then, like I said, the team was very skillful and it was, and of course, with our support internally, it was very easy in quote to get them up to speed. So the whole team was trained and was certified on UAC. And then it was really, asked them, the team, to suggest what to automate. And so I still have a big document of like twenty pages, which are all kinds of processes that could be automated. And this is what we are implementing right now. Great. All right, next question. It would be unusual for one technology to solve all the complexities of automating end to end processes in IT services as it is in a business process. I know using Ansible, what other technologies, what other tech was required? Well, look, again, UIC is just one of the automation tools, like I said, right? But, and together with it, we are using Jenkins, of course, we are using internal tool for test automations. Of course, we have also, Makenosky Telecom themselves, they have a business process automation. So yes, it is true. It is not only UAC. There are layers of application tools, but what we see is that UAC is like a big umbrella, right? Across the application and across tools, you can execute and monitor all of them. So that's the way we are trying to use it. You see, for example, in the one of the latest use cases you saw, you saw we are calling Talend, right? Which again, right, I mean, it is not a workflow automation, but it is a framework to extract, transform and load, right? And that is orchestrated through UAC, for example. UAC is a meta orchestrator that helps automate all solutions used in these workflows. It doesn't replace tools, rather it centralizes solutions to build the workflows and automate the tools. All right, we'll go ahead with the next question. Can the level of automation achieved by Macadonsky Telecom be achieved with internal resources instead of StoneBranch, or is StoneBranch skills and knowledge key to their achievement? I think that in theory, yes, we can achieve it. But you know, the problem is actually when you have the organization as such, I mean, it's focused, I mean, because most of the time it's eighty percent of operation, twenty percent of development, and the people are not focused, you know, how to optimize the things. And the things should be seen from the outside and with someone who have more knowledge and I would say courageous bravery, to come and automate the things. So I think that this was a good move that you give this task to someone else from the outset who can make the automation and be, how to say, quite successful in this. Okay. All right. Miro, this question is for you. In terms of cost efficiencies, do you recommend for larger enterprises like yourself to go with a managed service option for their IT operation automation? Well, I would say yes. I mean, of course you can build your business case. I mean, you can put everything into the plate and make your choice whether you should go or not. In our case, I think also for the others, when you're discussing about legacy where you can optimize, I think that this is a good choice. Of course, I mean, this contract is also not short term, I mean, so you have to take this into account and to see how the development will be. But as we can see, the times are not becoming easier, but it's becoming even more tougher and difficult. I mean, we see that the IT workforce is now booming again, especially in the part after the COVID. And I think that this is the thing that we have done well, I mean. So it was a good choice and I can recommend it for the rest of it to to to make their their choice in a way with the, of course, business case and to see whether this flies for them or not, most probably it will. Okay, great. All we have another question. Was DevOps automation the key use case for Macadonsky or did they want to bring end to end automation across the entire IT, including IT ops and other cross functional IT departments in addition to DevOps? Well, actually we didn't have this kind of, let's say, this is a long term view, but on the very beginning, we didn't have this in mind in general. What we had with the deal is actually to secure operation, improve the whole thing, and also to make integration between the new IT model of the DevOps with the legacy one. Of course, I mean, it would be interesting and on a longer term practical to make a full automation end to end. Of course, this is a holy grail, but it needs time to be done. Okay. We have one more. Can any component developed by or for Macadonsky Telecom be reused for other DT Group members? Are they transferable or reusable? Well, somehow, I mean, nevertheless, that we are in the same group, it's not one to one, I mean. So there is a specific to every Natco, what kind of, let's say, tools they have implemented and the environment such. Most of the things or the benefits what is practically done here can be reused later on, but not directly. And it's not the idea to practically how to say bring it directly to the other, let's say, sister company, but actually it's something that we can give it to the Natcos and the experience and possibility how they can automate it in their, let's say, environment. Because like I said, it's not one to one, unfortunately. Yeah, I think it's mostly philosophy and the expertise can be reusable, right? But of course, right, every IT landscape is very different. Absolutely. Different maturity, different tools, different things, but the idea and the philosophy definitely can be done. Okay, perfect. Okay, this concludes today's session. Thank you for joining us today. Have a great day.