Blog

All Blog Posts  |  Next Post  |  Previous Post

Firestore async dataset methods in TMS WEB Core v1.7

Monday, April 26, 2021

TMS Software Delphi  Components

In this last blog article of 4 blog posts, we introduce a last major new feature in the Google Firestore dataset, i.e. new async features. Before providing details about these new features, let's first look at the classic VCL TClientDataSet, how it can work how it works and what the implications are of inherently asynchronous implementations for use with a REST API based communication between a client dataset and the Google Firestore backend.

Classic VCL TClientDataSet operations

The editing and navigation interface for a TClientDataSet is best provided by using data-aware controls such as TDBGrid, TDBNavigator and more. Still, many times you need to process some records in code according to specific business logic. When it comes to populating and modifying a TClientDataSet in classic VCL code, you would use code similar to the following code snippets.

Inserting a record in the dataset

ClientDataSet1.Insert;
ClientDataSet1.FieldByName('Description').AsString := 'Call up a meeting with Sales group.';
ClientDataSet1.FieldByName('Status').AsString := 'New';
ClientDataSet1.Post;

Modifying a record in the dataset

ClientDataSet1.Edit; 
ClientDataSet1.FieldByName('Status').AsString := 'Done';
ClientDataSet1.Post;

Deleting a record in the dataset

ClientDataSet1.Delete;

 Processing all the records in the dataset

ClientDataSet1.First;
while not ClientDataSet1.EOF do
begin
  //do some processing based on one or more fields of the record

  ClientDataSet1.Next;
end;

Equivalents with Firestore?

Question is, will the above code work in Firestore ClientDataSet too? Yes the above code will work as standalone. But the code that follows the above code will fail if it depends on the success of the database operation in the previous code. This is important to understand so let's look at 2 examples of how the code that follows can fail in Firestore TClientDataSet.

Failing code example 1
Consider the following variation of the last example:

if not ClientDataSet1.Active 
  then ClientDataSet1.Open;
ClientDataSet1.First;
while not ClientDataSet1.EOF do
begin
  // do some processing based on one or more fields of the record ..

  ClientDataSet1.Next;
end;

This will fail on TClientDataSet1.First. Why? Because an open is an asynchronous operation that takes time to open the cloud database. There is no guarantee that the open will be complete by the time you reach the next statement. What will happen is unpredictable. In fact, this is true of all the ClientDataSets in TMS Web Core dealing with cloud databases.

Failing code example 2

Consider another example where we want to first insert a record in a Firestore collection, then get the record's id that Firestore generates and stuff it as a foreign key value in another dataset's new record.


// adding a new customer's record
Customers.Insert;
Customers.FieldByName('FirstName').AsString := 'John';
... more data for the customer record as needed ...
Customers.Post;

// getting the generated id from Firestore
newid := Customers.FieldByName('id').AsString; 

// adding the invoice for the customer
Invoices.Insert;
... more data for the invoice record as needed ...

// setting the foreign key value as the id generated earlier
Invoices.FieldByName('customer_id').AsString := newid;
Invoices.Post;
Do you see what will cause a problem here? Customers.Post is an async operation as it will add a new record in the Firestore collection. There is no guarantee that it will finish by the time you reacth the next statement that remembers the generated id in newid. 

To rescue with async

Fortunately, we have a solution for the above coding problems in the new version of TWebFirestoreClientDataSet. They consist of the following methods that allow you to wait for the outcome of the previous async operation before proceeding with code that depends on the outcome.

OpenAsync and SignInAsync

As shown in the following code sample, the OpenAsync method accepts an anonymous procedure as a parameter to which it returns a success or failure result after the open completes.

fireStoreClientDataSet.OpenAsync(
    procedure(success: Boolean; errorName, errorMsg: String)
    begin
      if not success then
      begin
        .. handle error case by using errorMsg ..
      end
      else
      begin
        .. further processing on success ..
        .. First, Insert, Edit, etc ..
      end;
    end);

The method SignInAsync discussed in Part 1 of this article is another async method to use when you also need to login with the Email and Password of the end user.

    fireStoreClientDataSet.SignInAsync(
        aUserEmail,  
        aPassword, 
        False, // IsSignup flag

        procedure(success: Boolean; errorName, errorMsg: String)
        begin
          .. do something on success or failure ..
        end
    );

PostAsync after Insert

Similarly, if you were to do an Insert and obtain the generated ID for the record in the Firestore collection, you will use this kind of code.

fireStoreClientDataSet.Insert;
fireStoreClientDataSet.FieldByName('FirstName').AsString := 'John';

.. set other field values as required in the new record ..

fireStoreClientDataSet.PostAsync(
    procedure(success: Boolean; data: JSValue; errorName, errorMsg: String)
    begin
      if not success then
      begin
        .. handle error case ..
      end
      else
      begin
        .. data parameter coming above is the id generated by Firestore for the new record ..
        .. do something with the id if required ..
        .. further processing on success ..
      end;
    end);


PostAsync after Edit

Here is an example of modifying a record and waiting for the outcome.

fireStoreClientDataSet.Edit;

fireStoreClientDataSet.FieldByName('FirstName').AsString := 'John';

.. set other field values as required in the new record ..

fireStoreClientDataSet.PostAsync(
    procedure(success: Boolean; data: JSValue; errorName, errorMsg: String)
    begin
      if not success then
      begin
        .. handle error case ..
      end
      else
      begin
        .. data parameter is the JSON data object of the record that was updated ..
      end;
    end);

The data parameter is not useful in this example. But it can be useful to identify which update finished when the response procedure is a common object method of the form that receives response for many update operations.

DeleteAsync and CloseAsync

Similarly, there are DeleteAsync and CloseAsync methods that return a success or failure to the passed response procedure with a signature similar to OpenAsync.

Processing Loops

It might be tricky to make processing loops this way that process all the records till EOF using Next but it's certainly possible. Several designs can be made by either using anonymous response functions with recursion or by using an object method instead of an anonymous response procedure. 

Batch Inserts with AddServiceObjects

If you need to insert a large number of records in the Firestore collection, you could write a processing loop as described above. But that is complicated and would be slow if you waited for previous insert to finish before inserting the next record. On the other hand, if you decided to fire many inserts at once, the speed might improve but there are complications of finding when they finish and whether there were any errors.

To deal with such use cases, we have added a Class Method AddServiceObjects that you can use to insert upto 500 records from a JSON Array at once, directly to the Firestore collection. This is also an async method by its very nature that sends the result to a response procedure passed to it.

Since this is a class method, you are supposed to use it by prefixing with the class name TFirestoreClientDataset. You don't need to open any dataset locally as it directly inserts at the server end.

For an advanced code sample on how this method is used, please see the ImportTestData project of the Firestore Filters Demo.

Summary

  • First, we discussed some issues with the classic VCL code used to process the records of a Firestore ClientDataSet in code. This is due to the async nature of the cloud database operations.
  • Next, we showed how the new async methods help you wait for the outcome of previous database operation before proceeding with code that depends on it.

We hope that you have enjoyed discovering the new features in the Firestore ClientDataSet component in this 4 part series.
If you have missed the previous articles, here are the links:

We will look forward to any additional ideas and feedback from you!



Bruno Fierens




This blog post has not received any comments yet.



Add a new comment

You will receive a confirmation mail with a link to validate your comment, please use a valid email address.
All fields are required.



All Blog Posts  |  Next Post  |  Previous Post