Servoy tech webinar series part 3: data broadcasting module

Hi, welcome to today’s third session of the Tech webinar series. My name is Jan Ilman and I’m the founder of SIRVOY and I will be moderating this session. If you have any questions, feel free to post them on the questions channel. You’ll be here to help you out, whether it’s with technical issues with go to meeting or with specific questions that you would like to ask. And for the third time, I’m very proud to introduce you to Sean Deflin, our one of our senior engineers at SIRVOY, who is presenting today’s session about data broadcasting. And so, Sean, we have quite some content to cover. Let me try to keep these sessions short. So, let’s get started. Welcome to the floor. Thank you. Is my audio clear? Yes, it sounds very good. And my screen is visible. Yes, I’m seeing the first slide here. Okay, perfect. Let’s get going then. Today’s the third installment in our Tech webinar series that we intend to keep going every two weeks. So, in the past, we’ve covered some announcements of new modules or new web components. We also promised to deliver some topics about best practices and, you know, a whole mix of topics related to building business applications with SIRVOY. Today’s topic is about data broadcasting between applications. And what I want to do is take a look at sort of the use cases around this topic. Talk a bit about best practices. And then introduce yet another free open source module in this case to help with one of the use cases. So, I think I should start off by maybe clarifying the term data broadcasting, especially for those who are new to SIRVOY, or maybe just never noticed that when you’re using an application that’s running in SIRVOY, you get this concept of data broadcasting for free automatically without having to write any code or set up any configurations. And what this means is that any time multiple clients are looking at the same records and one of those records is changed from inside a SIRVOY client. The data changes broadcast to all the other clients so that they can update their cache of those records. SIRVOY clients run completely off of a cached view of what’s in the database, and this allows the clients to perform much better. It optimizes the number of queries that are sent to the database. There’s a lot of benefit, and this is a very mature part of the product that’s been in there from since the beginning. As with anything related to caching, there is a downside, and that downside is that the cache is not the database. And although data broadcasting within SIRVOY applications takes care of most scenarios, there is the scenario that a database can be accessed by non-SIRVOY applications. And if records are modified outside of the realm of a SIRVOY client, then there’s no opportunities to refresh the cache. So you end up having a stale cache as we call it. And it really could be quite a while before a record is refreshed, because as long as it’s in the cache, anytime that record is viewed, it’s going to view it from the cache. So it can be quite a problem. I was, in fact, just recently working with a customer who had migrated a big mission critical system out of a legacy platform, and into SIRVOY. Because it was such a large application, they staged the deployment a little bit, such that they deployed most of their modules in SIRVOY in one go, but they had a remaining module that they wanted to leave in the legacy platform, and then migrate again a few months later. Because of this setup, they had the situation where there was the legacy platform and the SIRVOY applications talking to essentially the same database tables, and so they really had to ensure that both applications were seeing the real true picture of what’s in the database. And I was working with them, and it occurred to me, I’ve seen this dozens of times throughout the years, and there’s ways to deal with it, but I thought why not sort of formalize this into a best practice, and also create a helper module, and just be done with it so that I can just point people into the right location when they have this situation. So a couple of ideas about best practices and update strategies when you have this situation. I feel like there’s really two approaches. One is sort of what I call the opportunistic approach when the data change that happens from outside of the SIRVOY client session, when the data change event is unknown, meaning you can’t really trap it, you can’t really proactively do something about it. And so the one approach that seems to work well here, and kind of the only way to do it, I think, is to do a polling on a schedule. So you set up a headless client, you set up a task that runs every 30 seconds, once a day, whatever it is, and you poll some resource, be it a shared table, a file, maybe you’re querying actual records for modification dates, since the last time you ran the poll. So there’s a lot of different ways to do it, but the strategy is generally the same. You don’t really have an event that you can say, okay, right now I have to refresh my cache, so instead you have to go and look for some signal on a recurring schedule. So that works well when you don’t really know that the changes happen, but if you do know, and you can trap it inside of that other application and do something about it, then you can take a proactive approach, and this is nice because it’s less complicated, and it’s real time. You don’t have to wait for something scheduled event, and it’s a lot less resource intensive, because you’re not running a constant headless client that’s polling some data source. So when the data change event is known from that third party application, what we can do is expose a hook for this third party application to notify the Servoy application server that a data change on a shared table has happened, and this could be through some API like a web service. Just a bit about how to notify the Servoy clients in either of those strategies. There’s two ways to do it. It’s on the raw SQL plugin, which ships and all builds of Servoy. One is to flush all clients cache, and this approach is I call it the brute force approach. It’s when you know that a table’s been modified, you don’t really know which records have been modified, so you say listen, Servoy clients, there’s been a change to this table, update your caches. Then all the clients of the app server will update any found sets that they have, which are based and any records which are based off of that table. So you can imagine that this has a high overhead because all the clients are going to re-queer these found sets and the records that are contained in those found sets at the same time. So this works, it works, but it has a performance overhead. If you know the second approach, if you actually know which records have been changed, then you can do a notified data change, which is essentially a programmatic way to initiate that data broadcast event, and then you say to all the Servoy clients, these five records have been updated, and then you send the data broadcast event, and then that’s a much more nuanced and lightweight way to update the data. So if you can do it, if you know which records have changed, this is really the best approach. So I mentioned this helper module, it’s called the SVY data broadcaster, and really what this does is it takes, all you have to do is include it in your application, and you don’t even have to include it in your application, you just have to import it to your app server and run the data. And it exposes some REST endpoints that can receive data broadcast notifications from these third-party applications so that if you can trap that event in the third-party application, then you can tell the Servable App Server via a simple REST call of the data change. It also provides a client API for initiating the broadcast, because in some circumstances we see that there could be two Servable App Servers that need to broadcast that are not running in a cluster or anything, then need to broadcast one to the other about a data change. So this is also a scenario in some organizations where you have multiple Servable App Servers doing different things, and they might modify the same table in that case they want to tell each other about it. So it does have this client API, and it also has a little UI for doing tests, and that’s all included in one single module all you have to do is import it to your App Server and it’s ready to go. It’s kind of difficult to do a demo of this, but I’ve come up with something that I think explains it, so let’s jump over to a demo here. I have a simple CRM here up that we’re looking at an order record. This is running in a Servable HTML5 client. And what I’m going to do is simulate this third party application that goes into the database and makes a change, and we’ll take a look at what a stale cache looks like. So I also have up my database admin tool, I’m going to go in and edit the record that we’re looking at its order number, 10248. You saw the ship city was Paris, I’m going to just change this to say Leon, another city in France, and that data, once I left that field, that’s been committed to the database. The change has been committed from outside of a Servable Client Session, and if I go back to my Servable Client, you’ll see that the city for this order still shows Paris, and it’s going to stay that way for a while because this record’s been cached. The record had not been cached, then when the record loaded, it would show the right city because the query would be run. But because Servoyis optimizing not to issue more queries than it needs, then you can see that if I move around, come off that record, come back to it, it still says Paris, even if I ran a search and found the record, it would still say it finds a by pk, and it still just loads the record data from the cache. So it’s going to take a while before this cache, the stale cache becomes refreshed. So if we knew about the data change outside of the Servoian in the other application, and presumably that application can do simple things like send a HTTP request, then what I can do is tell the app server to update the cache. And so that’s what I’m going to simulate here, and this is where we enter the other module that I was talking about. I mentioned that this data broadcaster, helper module, does have a client-side API. Just in case you want to initiate the data broadcast also from a Servoian application. So for my test purpose, there are demo purposes here. I’m using another Servoian client, a smart client in this case, and what I’m showing here is a little form, which ships with this helper module that allows you to just send the notification. Now remembering I said there’s two different ways you can update the cache. One is a flush all clients cache, that sort of brute force approach. The other one is a notify data change, which is you actually know the record pks. And that’s why you see that for the flush, these additional fields are great out. But for this example, we know the order number that’s been modified, and we want to notify the data change. So I’m going to select notify, and it’s actually the orders table. It wasn’t a SQL update, and the pk is that 10 to 4a record. So I’m going to go ahead and push the go button here and pay attention to the ng client to see the data update. So you can see that I get a little notification here at Toaster Pop-Up. You also notice that the city change from Paris to Lyon. So what just happened is that the smart client that I have sent an HTTP request to my app server and notify to the data change, and then regular old automatic Servoydata broadcasting kicked in and notify all the clients of the data change. And I got the data broadcast event down in this client. I showed a little dialogue and the city was updated automatically. I think it’s worth taking a quick look at the module itself. It is this SQLite data broadcaster module, and it has two scopes. One is the actual SQLite data broadcaster itself. There’s not too much you need to do because it exposes those endpoints automatically. So this is just where that code lives. It’s not something that you would typically program against. And then it has the scy data broadcaster client. This would be the client side. If you happen to want to initiate broadcast events from a servoic client, more typically it’s probably coming from a third party application. And you can see those two public methods, the flush clients cache, and the notify data change that we called. So in my little test form, I was calling notified data change, and passing in the host and the server name, the table name, and the primary keys, also the SQL action to initiate that update. So that’s really all there is to this demo. It’s a little bit abstract, but there’s, if you want to take a look at, if you want to take a look at that module, everything’s released on GitHub. It’s called SQLite data broadcaster. There’s complete wiki documentation. Also, if you’re kind of new to this whole subject, I recommend that you go to the servoic wiki page, and you look at the wiki documentation for the raw SQL plugin, which has those two methods for refreshing the cache. If you have any questions about this webinar, or you have any ideas for future topics or any feedback about the format or anything, there’s my email, please send it to me. And, Yond, do we have any questions? Yeah, there’s a question if you can explain a bit more in detail how the code works, that is receiving the calls, and also maybe more detail example on how to connect to that code through web service. Okay, sure. So, I mentioned that there’s these two scopes. One is the sort of the server side scope. The other one is that, like a client if you need it. And, a bit of this also goes to just how web services work in servoi in general. This is also a feature that’s been in the platform for years, and what it essentially does is allows you to expose any method, simple method as a web service endpoint. So, there isn’t much you have to code against really to send the web service request from the servoi side. It’s really just rest. It’s sending an HTTP call. Server side that gets picked up in one of these forms, and this is how web services is done. So, there’s two endpoints. There’s a flush and a notify. So, the request that I had sent hits the update, which is an HTTP put. And, so this method gets called automatically. In fact, automatically a headless client is launched on the app server, and then this method gets called, and the HTTP body of that request gets de-serialized into a JavaScript object and passed into this method, and I’ve called that content. So, here I’m just handling that, and I’m pooling out what the SQL action is, and what the PK arrays are, and then I’m initiating the data broadcast. There also does some logging. So, when you run this module, you’ll be able to see a log of all the web service requests that are coming in and what’s happened. In fact, it should be in the developer as well. Here, my go to meetings blocking it. Just restore this. You can see that when I initiated that, that it does this logging, in this case, it just does the developer console, but in production it’ll actually log to your server log that a notify request was received and tells you the data source and the PKs and what the action was. So, you get that bit of logging. Now, if you’re initiating the request from outside of Servoy, then you have to format your web service call and do it in whatever language of the third party application you’re using. As I said, there is complete wiki documentation. And if you go to the home page, you’ll see how to, you know, what the endpoint is going to look like. And some examples of how to format your HTTP request body. So, that happens in whatever language that a third party application would support. Are there any other questions? Yes, there’s a question on best practices on using notify versus flashes. Is it better to do a flush when you change like 100 records or 1000 or a large part of the table? Or simply when you don’t know what is what is changed? Right. Yeah, so there’s more overhead on the flush than the notify. But, you know, if you’ve changed 1000 records, at some point there’s going to be, and I don’t know what that point is. I’m sure it varies. I think there’s going to be an intersection between the, you know, which, which approaches, performs the best. Because at some point, you’re doing a discrete data broadcast on so many records that it’s better to just have the the data. So, that’s, you know, do a complete, complete flush. But, more often than not, the flush is never going to perform as well as the notify data change. But, the use cases are different because you can’t really call a notified data change unless you know specifically which records were modified. There’s a question from Rafi, will this work with Servoy seven? And six and five. This has nothing to do with the NG client or Servoy eight. I happen to have had one of the demo clients up in Servoy eight. But this is, as I said, the data broadcasting feature and the web services and point feature have been in the platform for years. Since at least five, I think, or six. And yeah, I aware of the, the, the rest calls that you’re doing are going to work all the way back to version five. That seems a bit older me. Yeah, I don’t know. I, I have to go back. I think five to was when the rest was introduced. So, at least six for sure. Nothing is really changed in, in the web services architecture or the data broadcasting architecture since six. I know they’re going back to five. The data broadcasting’s the same. Okay, so we think it should work, but it’s always going to run a test if you don’t serve with seven, six or five. Yes, and if you have, if you have any issues running this, the best, the best thing to do is to go to the, the GitHub project home and go to the issues page and log an issue. Because when you log an issue, we get a notification and then it’s publicly known about the issue and these are open source to so the maintenance on these modules can be pretty quick turnaround. It’s not tied to the Servoyplatform necessarily. Okay, cool. So the, the answer to Rafi is, please test it and let us know. All right, well, it looks like there are no other questions. Bob has a comment that it is very clear demo. I guess that’s why it’s done within the 30 minutes. Okay, well, in that case, thanks very much for attending. We will be posting this recording on the main web and our landing page and on the Servoy forum. Next week, next, this in two weeks actually isn’t it. In two weeks time, there will be a second updates of web components December 21st, just in time before Christmas. If you do have any additional suggestions for topics and please head to the forum and there’s some in thread there about this series and feel free to post any suggestions. I think officially December 21st was the last scheduled webinar. Is that right, John? Yes, that’s the last scheduled one, but we plan to continue the tech series in the next year. Yes. So we’ll be announcing new topics coming up soon. Okay, and yeah, so we definitely drew input on the topic so that we are talking about the topics that keep you awake at night. So for can help you to be more successful with servos. It’s a combination of new things like the components, but also in making it easier to use existing and Servoy applications. All right, so thanks all for attending and Sean. Thanks for the excellent presentation and demo.

Close