Servoy Monitoring – Pettrie de Bondt, Gerard Immeker
Pettrie de Bondt and Gerard Immeker from Autoflex explain how they monitor their Servoy servers
Monitoring
- The speaker asks the audience about their monitoring practices and why they use monitoring.
- The speaker explains the benefits of monitoring and why it is important to monitor your infrastructure.
- They use monitoring to identify and resolve problems before users are affected and to track performance metrics over time.
Hosting
- The speaker discusses different hosting options, including on-premise, Azure, CVO Cloud, and Digital Ocean.
- The speaker explains why they chose Digital Ocean for their hosting needs and why they avoid AWS and Azure.
Kubernetes Environment
- The company uses Kubernetes for their production environment and has four staging environments: def, work, VIP, and EU.
- They use a Caddy proxy, Ingress, and Docker containers in their Kubernetes environment.
Database
- They use a multi-tenant PostgreSQL database with a hot swappable backup for redundancy.
Tools
- They use a tool called car8s to manage their Kubernetes cluster and provide an easy-to-use interface for accessing and managing servers.
- Grafana is used as the display engine to visualize metrics and logs collected by Prometheus and P drill.
- Prometheus scrapes metrics and stores them in a database, while P drill scrapes logs and stores them in a database.
- Loki queries the database for the collected log data.
- N8n is used for automating tasks and can be integrated with Grafana for monitoring.
- PG STS collects performance information from a PostgreSQL environment and stores it in a database for historical analysis.
- They use a custom tool for generating custom reports, managing client information, and monitoring system usage.
- The tool integrates with various services and platforms through plugins, such as the “Client Manager” plugin for managing client sessions.
- The tool collects information from different nodes through the “Admin API” that runs on each machine.
Yeah, yeah, welcome, welcome. I hope everybody is still awake. Still awake for our zone. A couple of lessons. So I’m back again here at this here also. So let’s start monitoring. What are we going to talk today a little bit about us? Our environment, why do we be monitoring some samples, some information, a lot of try to do a little bit of hands-off, a hands-on stuff. And our thing really, really cool too. We use it for a lot of things with also monitoring and questions now. So let’s keep it into a question in between. No problem, just ask them. So yes, of course we want to show you a lot. But we didn’t make it. That’s the guy who did several of our monitoring. So we can’t really go real deep into the specifics, but we will tell you a lot and try to explain the most of it. Yeah, and we have really difficult questions over here. We are passing through to have it. Yeah. Yeah, it’s my problem. It’s my little problem. So this is me. You can read what I do. I am. Yeah, you built Altiplex 9. Yeah, I could load them all to this. Which was in the Fox Pro and started out in Visual Fox Pro now, but it started out in Fox Pro. So it was in the MS-DOS and we still left two customers. Yeah, we really don’t want to support them, but they’re so stubborn. They won’t switch over. And once a year, you visit them. Yeah. Yeah. A little old lady. A little old lady, too. Yeah, accounting of it. Yeah. And then myself already had an introduction in the previous slides, had minutes out of Fox technical background with last year. I do a lot of talking, a lot of minutes in and building a really nice solution for our characters in the netherns, totally based in the netherns, about 1900 customers. I think about 600 still on Visual Fox Pro, really in the power progress of moving them to Altiplex and on Altiplex, 30, 1400 customers. So going to show that a little bit. And this is our slogan for today. For this year, relax with Altiplex. You don’t have to worry. We take care of everything. So we also take care of all the monitoring and all keeping up the servers. Yeah. Bit about our team, 11 persons, there are two managers, me and my kind, two testers, one full time and one part time. Six developers, three viewers, several, and three guys do some other stuff. And one death of which is my little brother. Yeah. Yeah. Originally planted, it was a half time job, a part time job, for half of the time, and the rest of the time he could do some programming of other stuff. But gradually we got bigger and bigger and sort of more things out. He’s now doing it more full time. So he spends a lot of time on that, but also on the other environments we have some Altiplex, Ne online with a lot of window systems on it and remote desktop and stuff like that. We have more and more internal stuff we want to do. And we also do a lot of monitoring on our other systems with maybe we have some time we will show you that also. So some questions. Please join again. The results of the previous slides I did get some screenshots. I also put them into the discord. So what OS are you using for enrolling out your applications in which? Okay. Okay. Yeah. I’m missing the survey file. Oh, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well, well. Well, well, what’s the information with those individuals still the same? Yes, yeah. What thatVa doesn’t include. What that’s actually like is the idea. Right. Okay. So then so now what we’re going to represent a lot of people at a time joining us, right? Just like the few people looking at what we call those girls who are the Radh had to On Windows we first had one server and then we had some more servers. We tried to split it up and then we moved to Linux and I already talked with some of you some hesitation to move from Windows to Linux because then you have to have some knowledge some more than the LS has to go a little bit further than that. So there we can imagine that it can be unheard of but that’s why we also got the government. There’s a lot about all kinds of OS. So really a nice to see this first poll. The next one, do we use monitoring? Yes, no. Why should you use monitoring? Everything is taken care of. You just install it once and it doesn’t all buy itself. Ah, yeah, that’s true. We go to a space room. More from a high level look at all your infrastructure. Is that what you mean? Yeah, our issue is doing database doing software. Which queries are running? Yeah. Yeah, so some people are so good. Is it going to be interesting? But hopefully we can show you some stuff why we do it. And a lot of people are doing it. Can imagine that. It’s a different kind of breed of person. You can in its now called more DevOps. So the older system operators are really struggling IC because the modern DevOps is also a lot of scripting and you have to use Git and do scripts. And not only just click through the Windows Wizard and everything will run fine. And you want to have it especially if you beneath it, you want to have it repeatable. So you create these nice scripts and then everything runs kind of automatically. You have to monitor, of course, but we’ll get to that. The next and the last question. Why is it already in full in there also? We need to reset it or what is it? I don’t know. So first one, well, where and how do we do the hosting with more like the where? So on premise Azure server cloud, it’s the ocean that we use RVS and RIR. So, we miss you. We miss you. Oh, what do you think? Just a lot of cloud server. Yeah. Oh, no. We can’t edit it. Yeah. Sorry about that. Yeah. Yeah, yeah. Do this. Yeah. Yeah. Excite. Yeah. How many won’t go into there? Thank you. Thank you for all your input. Really nice to see a lot of on premise. So does anyone do anybody want to comment in it? Do you want to have control? Why on premise? Yes. I don’t want to check on premise, but it’s very good. They’re much of so. Yeah. You want to have control or just cost? Well, where the data is? Yeah. Basically, it’s easy for us to answer what you’re hosting and I say, yeah, what if they can have the firms that can basically use our data? Yeah. Yeah. For regulations, yeah. For regulations, call us again easily over-promising my system. Yeah. But too many calls, so I want to run into problems. And if I really want to, I think, have the so very little eyes, little Kubernetes, and other slices, and it’s not the system next to me. Yeah. Yeah. So I think that a lot of hosting is of course a shared hosting or shared networking. So we can run into troubles there. Yeah. Anybody else want to comment on it? Or do I have a specific need for on premise? Security-wise or regulations or other things? Because we also started with our own requitators for AF9. It was more cost-based. We just bought it for five years. It’s right. We will also buy all the windows licenses just for five years. And that was the most cost-effective. It really is saved as a lot. But then you have to put up the money on front. And you’re not that flexible. And there are new technologies or new ideas. So that’s what we did in ourselves also. But now all our Kubernetes and modern stuff is running on this solution. We still ocean. We didn’t choose AVS or Azure on purpose. Because AVS we think they do a lot of locking. They have these special kind of databases like Mongo. We can put on their own little sauce. And you can never change provider. You’re only allowed to run on their AVS server. So you can switch to Azure or your own Kubernetes because you cannot run their version of MongoDB. Because they put some special stuff in there. So that’s one reason why we don’t didn’t choose AVS. And Azure will in my opinion pretty okay. But a little bit more expensive. And so far we really like it. It is still ocean a little bit smaller provider. But they have a big farm in Amsterdam. So that’s really near to it. So that’s good. So a little bit of a tradeoff. Sometimes they are a little bit slower to implement stuff. And their team is a little bit smaller. Also better support, I think. A little bit about our environment. Our Kubernetes K8S is short for Kubernetes. We have a Kati proxy in front. Then the Kubernetes. The standard on there. Then you have the different nodes. And the nodes are actually the machines. And we have for our product environment. We have five nodes. And on there there are different dockers. But in the dockers there is the top cut. And then there is the server solution. A little bit more expanded than this. But a nice simple overview. And we have Postgres database. Multi tenant Postgres database. We put in all the stuff. But there’s a hot, suppleable backup running behind it. So if one goes down, it’s switch, switch, switch, switch automatically the other one. And we have. It will. It will. It will. OK. And then we are moving to the, we already have this running. Postgres read. The other one is read, right? Because we want to offload a lot of load from the main database. And let everybody hit the post there’s read for all the reads. And then there are off-post a lot less writes. And then move there to the right database. So there are a lot of optimization. You can easily duplicate Postgres read part. And then that everybody hit that. Yes, person. Just for clarification you have Postgres not as a computer contact. No. No. No. You can run it for not production environments or smaller systems. You can run Postgres inside Kubernetes. But it’s really not recommended. And for a lot of performance things and scalability. And yeah. Yeah. But if you have just a small system just like your test system or a few users. And then it’s easy to set up your just inside Kubernetes or on your local machine. If you don’t want to have Postgres or on a database running all the time, just put it in Kubernetes. That’s no problem. But for real production environments, don’t put it inside the Kubernetes. No. That’s really not recommended. With the IOPS and all kinds of other things. Yeah. And then the next one. Yeah. We have at the moment four staging environments. Death. Death. That’s when you commit your code in Git. It’s automatically run the CICP script from Git. And we’re moving from Git. Let to create help. So a lot of different differences there. But it’s running automatically with code. And then the dev server gets automatically rebooted our domain. And then you know, because the test is targeting a web. So we do that manually. We could do that automatically. Yeah. Just more like a preference. And then from death, we have this extra environment. It’s called work. And that’s for debugging or website extra fixes. Stuff like that. Just in some weird stuff, you don’t want to directly put it on the dev server so that we will affect everybody who’s working on there. So that’s where the virus work is forward. Yeah. Yeah. And sometimes it’s easy to just test an inversion of server or other big things. And then everybody, the normal developer’s can go on and work on the dev environment. And then we can do some larger tests on our work environment. But those are just minimal environments with two nodes. Because you do want to test all the interaction between the networking and the Rebit and Q and stuff like that and the broadcast. But those are the smaller environments. And then we have the VIP. We have a lot of small customers like I said, almost 19, more than 1900. And they are small businesses. And we had a lot of problem finding customers who want to be the first one to try out new software. So we instead of all kinds of terms, we made them VIP customers. And they’re really happy. I, as a viewer, really important special customers. They come by four times a year at our office. We have some nice barbecues. They have a dedicated Slack channel. So they can discuss with the chat and his team not really about the real problems and fixes they have to call the search desk, but just have a better interaction and see what things are working. And a lot of discussions. So that’s really nice. So we want to really be in touch with our customers. And the whole customer base is a little bit too large. And also a lot of customers will say, oh, we’re really busy with our own work. Don’t call us. So that’s why we create the VIP group. So that’s a little bit of a larger environment also in Kubernetes. And then we have the real environment. And it was an L. But we’re thinking big. So we renamed it to EU. Still only the customer. But yeah, we’re now in the future. Yeah. And all the Windows environment was called NL. So we could use EU and everything started out of Flex 10. And then all these are also the domain names. So out of Flex 10. There’s 10. There, EU. So all the other parts that are part of the system. We also run inside Kubernetes, not only SharePoint, because I don’t think, and I don’t believe in an holistic approach, I want the best tool for every job. And SharePoint is really great too. We do a lot in SharePoint, but it’s not the tool for everything. So we have a lot of other different tools and a lot of different tools which we create in different environments like now the Angular and stuff. So why do you want to use monitoring? I will use it because we want to see problems before the day occur. We want to know the problem before the user sees it. That’s why we use it. To view it, general performance, which is important. It needs to perform. And finding the source of the problem is the query or is it something else? This is CPU, very high, and stuff like that. But it’s very high. It’s very low. But no, no. When we release a new version, we can easily go back two weeks ago and see all the metrics and see why it was performing then, okay, and why is it not performing today? And then see from there, okay, it doesn’t have anything to do with what you call the queries or the V ad, some other functionality, which may, essentially, work correct, but have a negative impact on performance so we can compare all these data points which we store for pretty long time, I think. And so we can see that and also we add some new customers every week, I think we add now about two or three customers every week, but we want to see what is the impact on that in the apps of small customers, and some bigger customers, so try to have you set it, see into the future, okay, what do we need, how do we need to scale, what do we need to add, do we need to grow on the disk site on the post-grid site, on the CPU’s. So, yeah. And we’d scale out when we need more servers, because, or less. That would be strange. That would be good. That would be new. It means something now. Maybe there’s a lot of optimisations, so we can use less resources, yes, that would be great. Yeah, it would be like a nice, but nobody’s working. I only need one server, maybe. Yeah. Yeah. Maybe when it’s busy, I need two or three. Yeah, there’s a lot of thinking about it and a lot of preparation, but at the moment that’s why we put many more insight break, we do not scale up and down automatically, especially the downscaling. There’s always more users than you think online at night, because normally I just say we don’t, we’re not active 24-7, the servers are running, of course. But our customers work from 7 to 7, our mechanics start really early, and the shield goes on a little bit later, so at that time, but if you restart the server at 2 o’clock, there’s still some users there. So, and get rid of draining all the sessions, that’s a really problematic still. So preferably we would do that more in a smart way, so we can drain it, put all the users on one server, a great server every day, preferably, and then start as soon as the load increases, but yeah, we’re not dead for. And also we want to control everything, that’s why we get a lot of the Kubernetes ourselves. Yeah, so keep the logs on a Kubernetes small, and I have a single interface, that’s why, and display it on a big screen. And that is how it looks at our office, the monitoring. So it rotates, it has different views, but this is a sample. Yeah, we try to use it to grab all the information, it’s a different tool, so which we will discuss later on, to keep all the systems clean, and put all the logging information, also the server, log, and all kinds of other logs, put it into a separate post-gres database, and then use that and analyze that for all the monitoring. So just keep the systems really clean and speedy, and don’t accumulate the logs on the system itself. We’ll get it all out, it was a pretty system. How does it look? Yeah, now this is a tool we use, I don’t know if I probably know, no idea about it, it’s a guy, eight S, it’s actually an interface around the API of Kubernetes. So it gets all the nodes for us, and we can shell into a specific server, or a seed log for me to buy it into, that’s all manual, manually. Yeah, you have a lot of command line commands for a Kubernetes, and this is a really nice interface, it’s still, of course, in the terminal, but you can switch environments really easily, you see all the different Docker containers running on there, and also have a live shot, I think. But this is at the moment currently running live, so you see all the different parts of our Kubernetes cluster of EU, and it gives some performance information that we were using monitor and for that. But this is just live, and you can shell into it, and… Yeah, port, port, port. Yeah, that’s really a really, really good one, because sometimes you, because we have five server, Tomcat’s instance is running, and if you want to go to the server, I admit page, how do you do this? Because there’s a nice round robin load balancer in front of it, so how do we know that you want to end on the DX7, a K8? So that’s really difficult, normally, really annoying. So what we do is you say, do a port forward, and then you just have a local, a port number, and type in local host, and then you’ll know that you’re actually on that server, and see all the server admin of that server, so, and for all kinds of other tools, it can also be really handy. You have this support forward. So no rocket science with really handy too. Yeah, yeah, we use it a lot for all kinds of analysis. So this is a part of our monitoring, monitoring, of course, a lot of information, which we can show here, and we show it in some gauges, and we show how many sessions per server, right there, there are four sessions. A lot of information. The basic memory, if you have memory, how’s the CPU doing, are there any block threads? Yeah, so we should have block threads, so we still have some problems. Yeah, lot’s here. They occur, and then we can see them here, and actually immediately immediately, if needed. Yeah. So another sample, like some crash, you can make a show, memory basic, how many are used, what is free, and a lot of possibilities. Yeah, a lot of notes, so this is information for one note, and we should scroll down there’s more, but… Maybe you can show it for life, see how many users we have today. Just refreshing it. I don’t know, shoot it, don’t that? Shoot, boy. So now you see at the moment, we have 1, 1, I didn’t just start my video, but… That’s why we need four notes. We really want to grow, because we have, at the end of the year, we have probably 2,000 customers, and there are 6,000, 6,000, 7,000 guaranteed in the Netherlands, so at least one to double this, so we have to… I still see someone from nine version, and they should go to the end. Yeah, yeah. We need to grow. Yeah. So you see here the different… Normally it should just load balance between all the notes, I don’t know why this one has a lot more user on it. Sometimes it happens, because it not only does the rounds rubble, but also tries to look at the load on the different notes, and it’s routed the users to the least used note. So that’s why sometimes they have a little bit of a disbalance. And then some more technical information you go. You can go really deep from Q&A to straight all kinds of tools. You can really see all the details of your server environment, and see the different spikes, and these tools, Afana, also lets you add alerts, if you think, OK, this is like the lock tables you see here. And you think, OK, we shouldn’t have that much. The locks are locked on the table. You can put an alert on that, and then you can send it to Slack, or an email, or SMS, and investigate, and see why it is happening. What can you do about it? If you really try to be proactive, of course, it’s always better. But sometimes you have to do some analysis after some problems occur. And that’s why it’s really nice to have all this information, because we really want to pinpoint fast if it’s in the survey, if it’s in the networking, if it’s in the firewall, if it’s in the routing, if it’s in the poster server. But you want to know, where do I have to look for these problems, performance, or other issues? Yeah. So that’s why we have a lot of these graphs, and also try to categorize them in a handy manner. And also the plain locks, if you want to, all the locks are grabbed out of the Q&A T cluster. So if you really want to look into the locks, you can, you can just specify a time frame, and in this sample, you see the HTTP errors, strange why we have five. Yeah, five or four errors shouldn’t happen. But we can investigate more here in the, in the log from the KDAU, which client was it, where it’s coming from, and then we can get a lot of information from. Yeah. So you can really drill down into all the information, and the nice thing is on top here, I’ll show it in the live on a little bit clearer, I think. It’s also not rocket science, but you can really fine tune which moment you want to see. You can see the last hour, but you can also see the last year, and then only a week in the last year, if you store all these data, of course. But I don’t see, we go that far. But you can really pinpoint and see different relations when our users are logging in, because you can also see when there are activity in half that, I really like that one that that. Yeah, it’s not the dashboard. It’s not on this dashboard, but Google Analytics, you also grab that, and then you see everybody has coffee around to port of PARS 10, and everybody goes to lunch at 12, fell 30. So then you see large dips, and then immediately after that you see all the users are working out again. So that’s really funny to see. And then we had, we loved this one. It’s like a lot of financial information we sent from our system to account CPAC dish. So you see all, and that’s a big process of ours, so you can see that between 9 and 10, they were most active, and most of the time this is at Wednesday in the afternoon. And they do a lot of financial stuff, apparently. I don’t know why they’re using Wednesday, but you see some strange graphs, it’s become wild. They’re doing this, and then we can anticipate it and keep this all into sync. So how did we do this? We installed some tooling on our Kubernetes system, which is called Prometheus for metrics and a palm trail for the logs. So they basically scraped the metrics in the logs and stored in the database, OCRES database, which is great by Grafana, to show the details. Yeah, we’ve got the Grafana, it’s only the display engine. And all the nice graphics. So which is the export, which we use, like a J-mix, an artist, which gets the info from the Tomcat and JVM, and the current. And also from a little bit info from server, like the… Yeah, the user sessions. We really, really like to see the different solutions in there also. I think there’s already an issue for that, is it possible? Because we have, like you saw, we only see the total number of server users, but you want to have a solution. Finally, if you have that. Well, it’s not in the area. No. But, okay, it’s… Ah, it just… just a quick question. Yes, it’s not. And another export, which gets the info from the… Yeah, the machine selects the server. How is the CPU doing, and I was remembering this, but like that. And, uh, Prometheus, as I said, scripts, the files. Um… And the result is written in a database. Yeah. If it was… we already set a separate post-grift database, and you can do a lot of analysis on that, but mostly it’s done visually with the Hrapan. It makes it somewhat easier to see things over time. The prompt here. Yeah. That works with the log files, and, uh, scrapes them. And, again, it’s written into the database. And, uh, the lucky queries, the database for the click data. Uh, and then we have an N8N, yeah, which basically is a small part of our, uh, monitoring that can do a lot more, which we go a bit… Yeah, we’ll show, uh, yeah, after this, yeah. So, yeah, graphan is basically the interface. There you, uh, can create the queries, uh, with a lot of variables, and a lot of stuff you can specify, and a lot of things you can attach to it, like, uh, plugins and stuff like that. There’s a lot available for the… Yeah. Really great community. A lot of these dashboards are shared, and if you want to have some information from the dashboards you want to use and see how we get this. No problem. You can share this easily. It’s really a… It’s just a simple JSON file, and you can share it and have all the different metrics and all the things you want to show, show, I mean, one graph or a couple of graphs, and you can also, a lot of information just to, uh, relate all the data, and, uh, we have a running on the big screen, and you can have a couple of dashboards running after each other. Yeah, and you can connect to your live database, like the financial transaction you’ve actually shared, or from the live database of our customers. And then the NSTAN you already talked a little bit about that. Really, really great all-purpose tool for a lot of things. We use it for the monitoring to get some information out of systems, but we also use it for customers if you want to send them special reports or send them some, some things you don’t want to program, uh, or show you that in a minute. It’s really handy just to do some stuff in NSTAN instead of programming it. I don’t know if everybody can see that can zoom in, not that far. Zoom in a little bit. So this is used for the, for the graphon, I think. Yep. Yep. Can you explain this, but it does? Yeah, it starts with a trigger, which you may have already say how many times I’m going to do this, then it gets some data from a database and several stuff it gets from there. And then basically, it gets from activity from one database and starts with in another database. What it does. Yeah, this is a pretty easy one, but I don’t know, uh, the PT starts. It collects a lot of performance information of your postgres environment. And then we just want that this is the live information that anyone to have it historically available. So we’re getting that from one postgres database to the next step, postgres database. So that’s just an easy, easy script, but you can do a lot of, a lot of nice things, like, uh, it’s a really, really easy thing to do. Yeah, there’s one customer wanted to have a specific report based, which nearly only was useful for him. So didn’t want to make a report for it, which would just make a query. And we sent him every night, we sent a report by email. And then the customer’s happy. Yeah. We didn’t have to program anything, just make a query and then which creates a CSV. And then. Yeah, this was a little bit more complex query, but that was, yeah, easily put into here. And then we run it, uh, save it to a file and sent the file via email. So the customer’s happy. Just, uh, the, the, the, the query was, uh, took some time, but setting up this, this work flow was really easy. It’s just a couple of minutes. Yeah. And then it’s working. So you can use it for a lot of, a lot of different things. We also use it for the year that tickets and we use it for, uh, for, uh, the, the phone system and move just data from one source to the other one. So that’s just a general purpose. And I think we hosted ourselves and nothing. Yeah. All the source tool. You can just install it. Yeah. We put it on one of our systems. So it’s also all the data is from us. So no security concerns there. It’s not shared. All this, this post-gres information. We just keep it in our own environment. There’s a lot of integrations. Can you, uh, love it against? Can you, when you add a plus and then you see here, a lot of, a lot of tools like, like, email, like, uh, all kinds of CRM systems. Absolutely. Yeah. There’s a lot. Yeah. People made a lot of integration. So that’s really handy just for, uh, we have a lot of like these. Well, one of the kinds of stuff things or just, uh, you don’t want to program a patch process for that. Most of the things you can do in here. All right. Yeah. Sit down and show this because Mark had a really nice, nice introduction about the, what is this, the client manager. And we want to be have this cluster of, uh, for a, uh, for a, uh, a, a, a system and we sometimes want to check if sort of users need to be kicked or check it or so we have here all the different clusters. You can filter that. You just made that simple interface around it. You can see here the different nodes or pods. And then you can search on it and see if our favorite customer, uh, Vita is online. Yes. She’s online with five users and you can see it’s on different pods. And then we can say, okay, let’s, uh, send them a message or kill their clients or just make an export in the users for other analysis. So if I add death also, so we can also specify all our environments and see who’s online in. See if I’m online. Yes, I’m online. I’m on death. Here it’s running and I can say, uh, kill. Yes, do you want to continue? Yes, we want. And then. That’s it’s going to die. Yes. So that’s a bit, bit in practice of the client management. And then you can do a lot of it. Yeah, we use it for a lot of information, get a lot of details information from the customers to keep track of things. But this was a pretty easy to develop interface, but the problem is you need to get information from the different nodes. So we run this, we call this the admin API. We run this small piece of code on the different machines. Yep. Because. You get the information from the nodes also from the nodes. Yeah. Yeah, fire the nodes because it’s a small tool that runs on the Kubernetes systems. So it can talk to all the nodes because going from the outside, you can talk to the different nodes. And that’s actually calls an API code doesn’t know the API call and get the clients from that machine. And then it collects everything and is by there. Yes. So just you ask your to provide something for better information. Yeah. Which would you use what this you could pretty just say, you say that to your customers to have a query this data. Yeah. Yeah. That’s true. You can do something. Yeah, we were doing that thing in a way, but in the analytics, I really want to see because. Sorry. Yeah. Yeah. Oh, the way. And now we really want to test this through, but this is a bit slow. So if you want to do this every couple of seconds, don’t do that. It takes a lot of resources. That’s why it doesn’t update real time. You have to click here or we have a timer here like this is a bit too, too fast. Because it takes some really does take some resources. And in the AMX, it’s just a counter because I want to see the users in a different solutions. They have a cloud solution with the health distribution. We have a API solution actually to API solutions and I want to see how many sessions are used for all the solutions. More. Yeah. But they could do this with it. But it is a bit intensive. So don’t run this every minute. That’s not good for performance. There’s for a lot of other things. This is really handy just to get an overview. See if they’re not double users or lock users or sometimes the users reuse stuck. So then we just kick it like we have shown and everybody can use this interface. If they have access to it. Yeah. So that’s really nice that we can provide this information. I think, yeah, or the talk a little bit about this also. But we’re using all the monitoring also for other systems like our cluster of window systems. If we remote desktop, we use it for our phone system and for year or tickets. So have everything in one place. So that’s really a really handy way to set of tools. It goes to some time to implement it and to get the familiar with all these tools. But it’s really, really powerful. And it gives us a lot of insight into our business. So are there any questions? Not all at once. No questions? Yeah. So the monitoring panel that we showed us where you can kick the users off from set messages. Is that an application written in server? Is that or is that written in something else? Yeah. But it’s written in something else. Yeah. Yeah. So the admin app that we’re using to get this information if it was written in server. And now it’s not. Well, it could be done, I think. Yeah. It’s not the deepest part. It’s not the complicated part. It’s just that we have to see that it took some time for us to have the admin API on all the different machines. And have the connection through there and also the authentication. Because which is already it’s checking all the different clusters and then all the different nodes spots on those clusters. So you can check it and see all the all the user from all the different systems. Because sometimes because on the main system, the main login, which is also not in server, we check the main database for all the credentials and then like the CPU users, they have the same login. I’ll show you. You so everybody uses this login based on their credentials. We sent them to a different cluster. Most of the time to the fifth was can also be to the work. So. Yeah. It’s basically is a different domain. And no user is in where he says should go to them. If you’re on your own, you send you to you. Otherwise to fit or death or work or whatever. Yeah. Yeah. It’s the way we set this up. More questions. And I guess thank you. Yes. Thank you. Thank you.