So far in this "TMS WEB Core and More" blog series, we've covered how to both develop and deploy TMS WEB Core applications in various environments, particularly of interest to anyone looking for solutions that don't involve Windows. This time out, we're going to move on from there and take a look at a key requirement for nearly every business-class web application - getting data in and getting data out. One way to approach this issue is by connecting a web application to a remote REST API server. And it just so happens that the developers of TMS WEB Core have also developed such a product: TMS XData. 

To better illustrate how a TMS WEB Core project might interact with XData, we're going to create a working example. In fact, here is the final application, embedded right here in this post - a survey app. It's been configured to present a survey for you, the TMS Blog reader, to help us gain a little more insight about what you like or don't like about the posts that have been coming out so far. Please take a few minutes and complete the survey. Both because we're all quite honestly curious about what you think, and because it will greatly help in understanding the rest of these posts, where you can see how the process of developing this app unfolded.

If you'd like to access the survey directly, rather than using the <iframe> embedded in this post, please use this link. If you encounter anything noteworthy or otherwise problematic at any point during the survey, please submit feedback using the icon in the top-right corner. Note that the survey should work equally well on a mobile device.

1. Introduction
2. Bargain: Three Apps For The Price of One!
3. Survey Client Initial Layout
4. Survey Server Initial Setup
5. Swagger
6. First Contact - Getting Data from XData

7. Survey Admin Client Initial Setup
8. Serviceability - Getting Logged In
9. More Service Endpoints
10. Survey Admin Client: Login
11. Survey Admin Client: Managing Surveys
12. Connecting SQL to Tabulator
13. Global Save and Cancel
14. Editing HTML
15. QR Codes
16. Custom Dropdown

17. Survey Client Questions
18. Saving Responses
19. Storing Data - JSON versus Structure
20. Displaying Responses
21. To-Do List
22. Roadblocks Overcome
23. All Done!

Introduction

Alright. Hopefully, you filled out the survey (thanks very much if you did!) and you now have a crystal-clear picture of our end goal. But before we get any further, let's quickly touch on what a REST API is and why we might want to set up our own server to host such a thing. 

The term "REST API" is an acronym for "representational state transfer application programming interface". A bit of a mouthful. The general idea is to have a place to store and retrieve "state" information - basically any data you like.  As we're very likely all aware, a typical basic web server (Apache, NGINX, etc.) is stateless - meaning that whenever a browser requests a page from the server via a simple URL, the server doesn't keep track of any previous pages that have been requested. It just serves up the same, identical page to every browser that makes a request via the same URL. And nothing about the request that has been made is stored in any fashion that might impact subsequent requests. Hence, stateless. 

There are many perfectly great reasons why web servers were designed to behave this way. Just as one example, it means that you can split up the work of serving web pages across multiple web server threads, multiple physical machines, or even multiple physical locations, and there's no concern about keeping state information synchronized. This greatly simplifies scaling up to insane numbers of users. 

As web technology has evolved, web servers and web browsers have become considerably more complex, and more efficient, about what data is passed back and forth. Aggressive caching, and many more communications protocols with an increasing number of obscure rules, make for a very fast browsing experience today, but the underlying web servers are still considered stateless most of the time.

In order to move data back and forth, and not just serve up web pages, some kind of application server is needed. It might accept connections from browsers directly or sometimes connections might be passed through libraries or other interfaces that are tacked on to extensible web servers, or it might be entirely separate. 

Application servers have also evolved over the years, and one popular class of application servers is the REST API server. Its popularity is largely due to its simplicity, much like the original HTTP protocol, and also because it builds on that very same HTTP protocol to carry out its business. Many variations of REST API servers are available that are largely accessible by many variations of REST API clients. So many in fact that this has become the most prevalent web service API approach in use today.  

So what's a web service? Well, we could be at this topic all day, but for our purposes, just think of making a call to a REST API as something like making a call using a Delphi function - you pass some parameters, and you get back some data. Just that the function is executed entirely in another process, likely in another machine, maybe on another continent.

A REST API server might offer a collection of such functions, combined into something that is referred to as a service but is generally just a set of these functions where each is individually referred to as a service endpoint.  There may be certain rules or rate limits or API keys involved in accessing some endpoints, or there may be none at all, depending on the capacity made available by the owners of the web service and the type of web service that is offered. 

A frequent use of REST APIs is as gateways or middle tiers between web applications and databases. The complexity, security, and performance aspects of dealing with a database can be tucked away safely behind the welcoming face of a REST API server, likely lowering the stress level (and workload) of countless developers!

For our project today, what we're after is a REST API service that can provide our TMS WEB Core application with data in the form of a set of survey questions, and also be a repository for data that the application generates, in the form of a set of survey responses.  We'll want to also be able to do a few other things along the way, like create and manage more than one survey, and also look at the survey results.  So we'll need at least a handful of endpoints to work with, along with some extra bits to deal with ensuring that everything is properly locked down. We don't want just anyone to be able to view the results or edit the surveys.

Bargain: Three Apps For The Price of One!

And for those visiting us from the far future, we're currently at TMS WEB Core 2.0.4.0 and XData 5.8.0.0. While Delphi 11.2 has recently become available, this was all put together with Delphi 10.3. Any supported version of Delphi should work the same for the purposes of TMS WEB Core and XData, however. Here's what we're going to be creating.

• Survey Client - a relatively simple web application built with TMS WEB Core and not much else. This could be developed on any platform that is supported by TMS WEB Core, and as a web application, should be usable on any platform that has a JS/HTML5/CSS3-compliant browser. On its own, this application is mostly an empty shell. It doesn't have any survey data (questions) included in it directly. What it does have is the logic to download and process a list of questions, and to generate the UI showing those questions. For example, there might be "Pick One" questions, "Pick Many" questions, or questions that request a free-form text response.  This app will also know how to upload the responses that it receives. The main design of this application is conceptually a bit like a finite state machine - guiding a user through a list of questions, potentially with branching logic, navigating back and forth between questions, and so on.

• Survey Server - a little bit more complex, we'll set up an XData server application. This is a traditional Delphi VCL app, with really nothing about it that has anything to do with the web, browsers, or UI elements. It is intended to run similarly to (or sometimes exactly) the way that you would run a Windows service application.  It will need to be installed on a system connected to the internet if the survey is to be publicly accessible.  There is the possibility of creating an XData server that runs on Linux, but that's a bit beyond what we're looking to do here today. The main design is focused on the implementation of a number of service endpoints.  Like "please give me survey #TMSBlog1" or "record that user {ABC} responded to survey {XYZ}, question #14 with the response "IBM Model M Keyboard". It will need to provide admin-level users with access to their surveys and accept from admin-level users any requested changes. It will also need to provide something to help with reporting what data has been collected. Much of this section will include a rehash of the XData steps followed in this post about getting data into Tabulator. We're doing something very similar here, so if this section is of interest, be sure to have a look at that post for another example.

• Survey Admin Client - this is also a TMS WEB Core project, with the same development/deployment options as the Survey Client, albeit with a considerably greater level of application complexity. The idea with this app is to have the ability to create surveys and edit the questions, as well as view the resulting responses. This is the front end to everything that we'll need to do with XData. This app might not necessarily need to be made publicly accessible. Structurally, the Survey Admin Client will no doubt end up looking a lot like this Labels Example from another recent TMS Blog post. If this section is of particular interest, be sure to read that post first as it explains in more detail the kinds of elements that are being used here.

If this sounds overly complicated, it will all make sense as we put all of the pieces together. The biggest question, really, is where to start?!

Survey Client Initial Layout.

Well, we could start on any of the three, but let's start at the end result - the web application that we want visitors to use. The rationale here is that it will quickly help us identify what kinds of things we'll need in the other two applications, but we'll be moving back and forth among the three, testing out various features along the way. 

For the client app, what we're after is a responsive web page that will serve as the bin into which we can pour survey definitions. These definitions might include information about the survey itself, as well as the questions, some theme settings, and so on. It would be handy if the application looked nice on both mobile and desktop browsers, and we're after as clean and crisp a UI as we can manage, so that it does its business subtly, leaving the survey itself to take center stage.

We're not really going to go into much detail for this initial part as there are more interesting things to cover, but by all means, have a look at the project source code or post a comment if there's anything here that might need further discussion. For right now, though, this is just a stub app of sorts. Some of the highlights include the following.

1. Start from a TMS WEB PWA Application template.
2. The main form is a constrained rectangle that will stretch to fill smaller screens, but not larger screens.
3. Properties in the IDE are set to match the condition where no survey data has been provided (see screenshots below).
4. There's an 'About' <div> and 'Feedback' <div> that are mostly separate from the survey itself.
5. A simple transition effect is implemented between them using CSS and adjusting <div> classes.
6. An "Activity Log" is maintained, and output to the console in certain conditions, or included with any feedback.
   
7. Plenty of Bootstrap classes and CSS styling are used throughout to get things to look as best we can.
8. Font Awesome icons are used, minimally, to spruce things up a little bit.
   
9. Resize and Show events are used to keep app sizing in sync with the current page size/orientation.
10. Minimal coding at this stage, and nothing that is really survey-specific as yet.
    

Survey Client Stub.

Survey Server Initial Setup.

The next step is to set up the Survey Server XData app. This is going to be set up as a traditional VCL Win64 app.  Here are the first few steps in getting this setup.

1. Using the Project Manager window, add Windows 64-bit under the target platforms and double-click to make it the current selection.
2. Create a new Delphi project.  Select New/Other... and then select the TMS XData/TMS XData VCL Server template.
3. Save the Project, which will prompt you for the names of all the units in the template.  Defaults are fine.
4. Run the project.

This should get you the following. 

Default XData Server Template.

Before we get any further, let's take a look at what this actually is. As we mentioned at the outset, a REST API server, like XData, is built on top of an HTTP server (a traditional web server). In this case, the XData template automatically included an HTTP server in the project, which comes by way of TMS Sparkle. There are a few interesting things that come from this, but the first and most relevant is that you can point your web browser at the URL that is shown, and you'll get a response. Not much of a response, but a response all the same.  

XData Default Response.

All this is telling us at the moment is that the XData server is running. Which isn't nothing. In fact, when it comes to troubleshooting connections later, this is the first thing to check - that the XData server is running, and it is running on the port that you expect it to be running on.

Depending on how your development machine is configured, you may even be able to connect to this XData server from another computer on your network, using the URL http://<devcomputerip>:2001/tms/xdata and get the same result.

If you encounter an error, the most likely cause is that Windows Firewall or something similar is blocking access.  Doesn't really matter as we don't ever need to connect to this outside of our development environment anyway, and even the port number doesn't matter much, so long as it isn't conflicting with something else already running there.  Note that the development web server is intended for just that - development. Please don't use it for anything else!

The next step is to add an actual service and service endpoint to this XData server. Whether you have one service with many endpoints or multiple services with fewer endpoints is entirely up to you. Similarly, you can set up multiple XData servers running on different ports on the same computer, if you want to have completely separate functionality.

For our purposes, one XData server is perfectly fine, but we'll separate out the endpoints into different sets. One service for the Client functions and one for the Admin functions. This makes things a little easier to manage, particularly if you have different people working on different parts of the application at the same time. So let's create the first service. This can be done using a template as well. Here are the basic steps involved.

1. File / New... and select TMS XData/TMS XData Service 
   
2. Enter a name for the service and leave the defaults as they are

This looks something like the following.

Creating an XData Service.

We'll include the sample methods initially, primarily because it is easier to copy and paste them to create our own methods rather than try and remember where they go and what the syntax is. After the service is added, we have two new units in our project, one for the interface and one for the implementation, which is the recommended and preferred arrangement here. 

For the SurveyClientService, the first endpoint we're going to want is to provide a survey. Recall that an endpoint is like a Delphi function call. So we're going to name our endpoint GetSurvey. It's going to have two parameters, SurveyID (a string) and ClientID (a string) and it's going to return information about a survey, which is going to be an arbitrarily large block of JSON. It may be short and contain just text, or it may contain dozens of embedded images.

NOTE: If you're not already familiar with using JSON as a data type, check out this post to see a few examples.

To get our endpoint setup, we'll need to update our interface and implementation units. After cleaning out the sample methods, here's what we're left with.

SurveyClientService Interface:

unit SurveyClientService;

interface

uses
  System.Classes,
  XData.Service.Common;

type
  [ServiceContract]
  ISurveyClientService = interface(IInvokable)
    ['{6269A461-E393-4B03-ADF9-7749DECB3E7D}']

    [HttpGet] function GetSurvey(SurveyID: String; ClientID: String): TStream;

  end;

implementation

initialization
  RegisterServiceType(TypeInfo(ISurveyClientService));

end.

SurveyClientService Implementation:

unit SurveyClientServiceImplementation;

interface

uses
  XData.Server.Module,
  XData.Service.Common,

  System.Classes,

  SurveyClientService;

type
  [ServiceImplementation]
  TSurveyClientService = class(TInterfacedObject, ISurveyClientService)
  private

    function GetSurvey(SurveyID: String; ClientID: String): TStream;

  end;

implementation

function TSurveyClientService.GetSurvey(SurveyID: String; ClientID:String): TStream;
var
  SurveySample: String;
begin
  TXDataOperationContext.Current.Response.Headers.SetValue('content-type', 'application/json');

  SurveySample := '{"SurveyID":"1234",'+
                   '"SurveyName":"Test Survey",'+
                   '"Header":"Welcome to the Test Survey",'+
                   '"Footer":"Get started!",'+
                   '"Cover":"TMS WEB Core Blog Reader Survey"}';

  Result := TStringStream.Create(SurveySample);
end;

initialization
  RegisterServiceType(TSurveyClientService);

end.

As can be seen from the above, we've just got one service endpoint setup for testing at the moment - GetSurvey - and it just returns the same bit of JSON regardless of what SurveyID or ClientID is passed. In order to use the TStream datatype, we needed to add the System.Classes unit (to both the interface and implementation). JSON could be passed around using the string datatype, just as we're doing when we create it here. Later, however, we'll be generating it via other means where it arrives as a TStream, so we'll just get in the habit of using that. The TXDataOperationContext line is used to indicate what is contained within the TStream. We're not doing anything yet with the parameters passed to this endpoint.

When we run the project, however, we get something that looks the same, even when looking at the browser page we looked at previously. So while our new XData server has a perfectly functional SurveyClientService, we don't really have an easy way to test it.

Swagger

Fortunately, we've got just the thing for that - Swagger UI. This is an interactive tool for documenting and testing REST APIs, and it is integrated into XData. And the best part is that much of it's interface is generated automatically. We just need to turn it on. To do that, we've got a bunch of quick steps to follow. There are a handful of other items on this list that are entirely unrelated to Swagger, but as we're here we might as well get them off our to-do list at the same time.

1. Find the XDataServer component on the form, right-click on it, and select "Manage middleware list"
2. Add "CORS" to this middleware list. The "Origin" property (in the Object Inspector) should be set to a single asterisk.
3. Add "Compression" to this middleware list. It doesn't need any further configuration.
4. Add "JWT" to this middleware list. Set the "Secret" property to some random string (something fun!), but at least 32 characters long.
   
5. Close the middleware list and select the XDataServer component so we can adjust its properties in the Object Inspector.
6. Under SwaggerOptions, set "Enabled" to "true" and set "AuthMode" to "Jwt".
   
7. Under SwaggerUIOptions, set "Enabled", "ShowFilter", and "TryItOutEnabled" to "True".
8. In the code for Unit1.pas, add 'XData.Aurelius.ModelBuilder' to the Unit1.pas uses clause.
9. Create the DataModuleCreate procedure, (click on the Form, then double-click on the Create event).
10. Fill out the event with something like the following.
    
    

    procedure TServerContainer.DataModuleCreate(Sender: TObject);  
    begin  
      TXDataModelBuilder.LoadXMLDoc(XDataServer.Model);  
      XDataServer.Model.Title := 'Survey API';  
      XDataServer.Model.Version := '1.0';  
      XDataServer.Model.Description :=  
        '### Overview'#13#10 +  
        'This is the REST API for interacting with the Survey project.';  
    end;

Naturally, the above text can be updated to reflect whatever you want other developers or users of your REST API to know, including contact information, restrictions or limitations, and so on.  Now, let's move on to Unit2.pas.

1. Open Unit2.pas and add a new button at the top, with a caption of Swagger.  
   
2. Add the following code to the onClick event for the new button.
   
   

   procedure TMainForm.Button1Click(Sender: TObject);
   var
     url: String;
   const
     cHttp = 'http://+';
     cHttpLocalhost = 'http://localhost';
   begin
     url := StringReplace(
         ServerContainer.XDataServer.BaseUrl,
         cHttp, cHttpLocalhost, [rfIgnoreCase])+'/swaggerui';
     ShellExecute(0, 'open', PChar(url), nil, nil, SW_SHOWNORMAL);
   end;

   
   

3. Add WinApi.ShellApi to the uses clause of Unit2.pas.

All we're doing there is setting up a button to automatically launch a browser and load up the Swagger page. 

Finally, we've got a couple of Delphi project-level options to set.

1. Go to the Delphi Project/Options menu, and then find the Building/Delphi Compiler/Compiling option.
2. Enable the Generate XML documentation option.
3. From the same menu, set the XML output directory to your application's debug folder.

Lots of steps! But when you run the app and click on that shiny new Swagger button, you'll be taken to the SwaggerUI page that has been generated automatically for the project. Fill in anything for the SurveyID and ClientID values, and then click the "Execute" button. If all goes according to plan, you should get the following, complete with a handsomely formatted JSON response. Swagger picks up the "content-type" from the header that we set using the TXDataOperationContext call, and as it knows what JSON is, we get a better display. All kinds of things can be set in the response header using this mechanism.

XData and SwaggerUI.

And this is exactly the kind of thing that we're after with XData. The Request URL shown in the image above shows how we're calling the GetSurvey endpoint (function) within the SurveyClientService that is part of the REST API on this particular XData server.  Here we are passing it the parameters for SurveyID and ClientID.

If you were to just copy and paste that Request URL into your browser, you'd get the same response back.  Depending on the browser, it may just be unformatted JSON (the default with Google Chrome) or displayed within a JSON viewer of sorts, where you can work with it a little (the default with Firefox).

The end result is that we've got our first functioning XData service endpoint which is returning survey data as JSON.  Note also that we've not done anything here in terms of access controls - we're assuming that anyone can request a survey using the GetSurvey function. And what about those parameters, SurveyID and ClientID? We'll use those a bit later. But let's leave the SurveyServer running and go back to our Survey Client and see if we can get it to talk to the Survey Server.

First Contact - Getting Data from XData.

Back at the Survey Client, we're going to add the code to go and get a survey from our newly minted XData Survey Server, to test that the communications aspect is working. This example is going to use RawInvokeAsync to make the connection, explicitly identifying the server, the service endpoint, and the necessary parameters.

The overall concept here is that the Survey Client is going to be launched with some value for SurveyID as a URL parameter, and ClientID will be something generated that is unique to that client and survey. When the survey data comes back, it will update the page with the data it has received. If anything goes amiss prior to that point, nothing is changed in the UI and we'll end up with the same Survey Client view as we had originally, where we had no survey data. Any errors will be logged to the web console.

Passing parameters to a TMS WEB Core app is done by adding parameters to the URL. This takes the form of  ?Param1=Value1&Param2=Value2 and so on. We're just going to use the parameter name of "S" for our surveys.  You can specify parameters to be passed from the IDE during development, similar to the "run" parameters in a traditional VCL project, by entering values into Project/Options/TMS Web/Compile/URL params. 

The GetQueryParams function can be used to test for the existence of a particular parameter. We'll need to add the WEBLib.WebTools unit to our uses clause to access the GetQueryParams function. Note that URL parameters are subject to various restrictions. The first of which is that you can't have any spaces or special characters in the parameter names or values. There are supporting functions for encoding and decoding URLs if you run into a situation where more complex data needs to be passed in this way.

For the connection, we'll need to come up with a URL - the same one reported by the Survey Server when it first starts. This is complicated by the idea that you typically want to develop code pointing at a development XData server, but you want to have your production deployment pointing at a production XData server, which will most likely have a different URL, almost definitely HTTPS instead of HTTP, and also a different port number.

We'll have to update the Survey Server app to do something similar - running with one configuration in development and another in production. We'll use a 'Dev' parameter to make the distinction here. You could also pass a parameter containing the important bits of the URL for the XData server, or have it pick up the value from a configuration file. Nothing but options here.

To set up the actual connection, we'll need two more components - a TXDataWebConnection and a TXDataWebClient (referred to as ServerConn and ClientConn, respectively, in the code). Also note that there are several calls that are asynchronous, which are made using "await", so we'll need to add the [async] attribute to the interface declaration for any such methods. 

There are other ways to implement this type of communication, using other components, making calls to XData directly using HTTP requests, or implementing it directly within JavaScript, but this works pretty well for any kind of service endpoints we might encounter. Putting it all together, here's what it looks like.

procedure TForm1.GetSurveyData;
var
  Response: TXDataClientResponse;
  Data: JSValue;
  Blob: JSValue;
begin

  // An indicator that something is going on.  Likely happens to fast to
  // ever be seen, but maybe for a really large survey download...
  btnNext.Caption := '<i class="fa-solid fa-spinner fa-spin fa-2x" style="font-size:42px; margin-top:-4px; margin-left:-8px;"></i>';

  // Development server or Production server?
  if GetQueryParam('Dev') <> '' then
  begin
    ServerName := 'http://localhost:2001/tms/xdata';
    LogActivity('Development Mode Specified');
    LogActivity('Connecting to '+ServerName);
  end
  else
  begin
    ServerName := 'https://carnival.500foods.com:10101/tms/xdata';
    LogActivity('Connecting to '+ServerName);
  end;

  // See if we've got a SurveyID as a parameter?
  if GetQueryParam('S') = '' then
  begin
    LogActivity('No Survey Specified');
    exit;
  end
  else
  begin
    SurveyID := GetQueryParam('S');
    ClientID := 'tbd';
    LogActivity('SurveyID: '+SurveyID);
    LogActivity('ClientID: '+ClientID);
  end;

  // Try and establish a connection to the server
  if not(ServerConn.Connected) then
  begin
    ServerConn.URL := ServerName;
    try
      await(ServerConn.OpenAsync);
    except on E: Exception do
      begin
        LogActivity('Connnection Error: ['+E.ClassName+'] '+E.Message);
        console.log('Connnection Error: ['+E.ClassName+'] '+E.Message);
        tmrRetry.Enabled := True;
      end;
    end;
  end;

  // We've got a connection, let's make the request
  if (ServerConn.Connected) then
  begin
    try
      Response := await(ClientConn.RawInvokeAsync('ISurveyClientService.GetSurvey', [
        SurveyID,
        ClientID
      ]));
      Blob := Response.Result;
      asm
        Data = await Blob.text();
      end;
    except on E: Exception do
      begin
        LogActivity('Survey Download Error: ['+E.ClassName+'] '+E.Message);
        console.log('Survey Download Error: ['+E.ClassName+'] '+E.Message);
        tmrRetry.Enabled := True;
      end;
    end;
  end;

  // Do we have any data?
  if (Length(String(Data)) > 0) then
  begin
    // Yes we do!
    SurveyData := TJSJSON.parseObject(String(Data));

    // Let's Update the UI
    divTitle.HTML.Text := String(SurveyData['Header']);
    divSubTitle.HTML.Text := String(SurveyData['Footer']);
    divMiddle.HTML.Text := String(SurveyData['Cover']);

    // And we're off and running
    SurveyState := 'Loaded';
  end;
end;

Once the GetSurveyData method has completed, the form variable SurveyData will contain the JSON for the survey information and the main form's front page will have been updated with whatever could be gleaned from SurveyData. We can then enable the "next" button which kicks off the survey itself. Here's what the Survey Client looks like after updating it with the contents of SurveyData. Naturally, we'll be wanting to spruce things up a bit with an actual survey.

Test Data Loaded.

With this part of the communications between the SurveyClient and SurveyServer apps sorted out, we can now concentrate on what we want to go into the SurveyData JSON. This brings us to our third and final application for this project - the Survey Admin Client.

Related Posts
TMS XData: An Extended Example Part 1 of 3
TMS XData: An Extended Example Part 2 of 3
TMS XData: An Extended Example Part 3 of 3

Follow Andrew on 𝕏 at @WebCoreAndMore or join our 𝕏 Web Core and More Community.