Conference App Secrets #3: JSON vs XML

December 2, 2013 — Leave a comment

This is part #3 of my series ‘Conference App Secrets’.

This time I want to discuss some aspects of HowTo persist data to guarantee a fast-responding app.

I was asked why my conference apps are performing so incredible fast while getting Sessions or Speaker or searching. (Got many 5 star reviews because of this)

You should download the BBJam Asia Conference App to see it there in action.

The answer isn’t easy and will cover some more articles. Let me start to explain…

BBJam Asia uses a JSON DataModel

I’m using JSON to persist the data of my conference apps and there are many reasons. Using REST interfaces you still get many data from responses in XML format and in most cases you have to consume such data and cannot change the backend.

The first thing I’m always doing is to convert those XML data into JSON data before going on.

You’ll find all used datafiles of this sample app and sources at gihub: https://github.com/ekke/cascades_json_vs_xml

I cannot tell you details about the BBJam Asia Conference API, so I generated some mockup data:

  • 10’000 random addresses to have a large file to experiment with
  • 119 public available speaker data to have some conference-relevant data

The addresses I’m using are generated by a free service from fakenamegenerator.com I can highly recommend this service to developers if you have to provide prototype apps with data or if you want to test using large data sets. They even generate different data for the Country you selected and also GeoCoordinates.

The speaker data is a wordpress export file from another conference APP and contains real data from public available sites.

Addresses and Speaker data is in XML format. I’ll show you HowTo convert these files into JSON and explain WHY you should do this.

Both datasets can be found here:

assets_data

While developing and testing in many cases you have to take a look at those data and JSON is much easier to read as of XML.

Here’s one address in XML:

    <Address>
        <City>Lincoln</City>
        <Company>Record Bar</Company>
        <Country>US</Country>
        <CountryFull>United States</CountryFull>
        <Domain>AffordableShow.com</Domain>
        <EmailAddress>SharonSMcCall@cuvox.de</EmailAddress>
        <GivenName>Sharon</GivenName>
        <Latitude>40.722909</Latitude>
        <Longitude>-96.678577</Longitude>
        <Number>1</Number>
        <State>NE</State>
        <StreetAddress>3002 Poling Farm Road</StreetAddress>
        <Surname>McCall</Surname>
        <TelephoneNumber>402-310-0424</TelephoneNumber>
        <Title>Mrs.</Title>
        <ZipCode>68501</ZipCode>
    </Address>

and here the same address in JSON:

      {
         "City" : "Lincoln",
         "Company" : "Record Bar",
         "Country" : "US",
         "CountryFull" : "United States",
         "Domain" : "AffordableShow.com",
         "EmailAddress" : "SharonSMcCall@cuvox.de",
         "GivenName" : "Sharon",
         "Latitude" : "40.722909",
         "Longitude" : "-96.678577",
         "Number" : "1",
         "State" : "NE",
         "StreetAddress" : "3002 Poling Farm Road",
         "Surname" : "McCall",
         "TelephoneNumber" : "402-310-0424",
         "Title" : "Mrs.",
         "ZipCode" : "68501"
      }

So better readability is one reason.

If you’re using QML then you can easy copy / paste objects from JSON directly into your QML code like

var myAddress = {  …. the JSON object … } … myDataModel.insert(myAddress) ….

Sometimes this is useful for mockup data where you simulate inserting objects into List DataModels.

In most cases JSON datafiles are smaller then XML:

  • 10’000 addresses in XML: 6.7 MB
  • 10’000 addresses in JSON: 5.9 MB

If the XML is using many long property names the file size can be reduced much more, but sometimes if the XML structure is very complicated it may happen that a JSON file is larger then XML. (Take a look at the Speaker data)

Perhaps you already know that Cascades supports JSON very well, per ex. if using QVariants from C++ in QML where QVariantMap was mapped automatically to JavaScript Objects (== JSON). Perhaps this deep integration of mapping is the reason for ….

Speed Speed Speed🙂

speed

© David Castillo Dominici | Dreamstime.com

Inside a conference app most times I’m reading data from cache and this is where JSON clearly wins over XML. Here are some values I measured using the 10’000 addresses:

  • READ JSON:  2’392 ms
  • READ XML:  71’342 ms

Rule #1: if caching data, always use JSON – this can be up to 30 times faster then XML

It’s curious: writing data to file JSON is something slower then XML. Here’s what I measured for the 10’000 addresses:

  • WRITE JSON: 5’255 ms
  • WRITE XML:   3’567 ms

The WRITE speed differences are marginal and you can do the WRITE async and work on while you’re probably waiting to get the data from READ.

In some cases you’re getting more data as you need. Take a look at the Speaker data exported from WordPress as RSS in XML format. Both file formats: XML and JSON aren’t easy to read and contain too much data in a complex structure. But it’s easy to workaround: as soon as you get the data from HTTP Response transform it and remove unneeded properties, which will give you even more speed. Here’s the measurement using the 119 Speakers:

  • READ XML:     378 ms
  • READ JSON:     76 ms
  • READ JSON v2: 23 ms

JSON v2 contains only the attributes you need in a less complex structure.

Rule #2: if getting complex data remove unused properties to get even more speed

It’s boring only to compare numbers, so let’s use Cascades to demonstrate the speed of JSON – based persistence or caching and see how easy it is to convert data between XML and JSON:

UI (QML)

The sample application is only using two QML files:

  • main.qml
  • MeasureSpeakerPage.qml

main.qml contains a NavigationPane displaying a Page to visualize the measurements of reading / writing the 10’000 addresses.

MeasureSpeakerPage.qml will be pushed on top to visualize the measurements of reading Speaker data from XML, JSON and JSON less properties.

Let’s take a look at the main.qml first. There are some attachedObjects:

  • SystemToast
  • ComponentDefinition
  • QTimer
attachedObjects: [
    SystemToast {
        id: infoToast
        ....
    },
    ComponentDefinition {
        id: measureSpeakerPageComponent
        source: "MeasureSpeakerPage.qml"
    },
    QTimer {
        id: myTimer
        property int usecase
        interval: 100
        singleShot: true
        onTimeout: {
            switch (myTimer.usecase) {
                case 0:
                    app.compareJSONandXMLaddresses()
                     eturn
                case 1:
                    app.compareJSONandXMLspeaker()
                    return
                ....
            }
        }
    }
]

We’re using the SystemToast as an easy way to let the user know if a process was started or finished.

While the first Page is defined directly inside the main.qml, the MeasureSpeakerPage will only be pushed on demand if the users starts an Action. To avoid unnecessary creation of UI Controls, we only define the component and create it dynamically before pushing on top.

The QTimer is used as a workaround to have a small delay between hitting the Action item and calling the process in C++. As we’ve seen above it takes a long time to read the 10’000 Addresses from XML: more then 1 Minute ! While executing this task we want to disable the Action items what was done async and needs some ms.

Without using the QTimer there will be not enough time to disable the ActionItems. Why ? Because we’re doing something you never never never should do in a real application: executing a long-running task on the UI Thread. While Network requests are async operations out of the box, reading a file isn’t and will block the UI.

The goal of this demo application is to let you feel the long execution time😉

So I’m using the QTimer – workaround. Now disabling the ActionItems works as expected. Here’s how I have done this:

NavigationPane {
	id: navPane
	property bool running: false
	.....
	Page {
	    id: measureAddressesPage
		....
		actions: [
            ActionItem {
                title: "Measure Addresses"
                enabled: ! navPane.running
                ...
                onTriggered: {
                    navPane.running = true
                    ...
                    myTimer.usecase = 0
                    activityIndicator.start()
                    myTimer.start()
                }
            },
			... more ActionItems
		]

	}
}

All ActionItems will be disabled if a Task is running: I’m using a property from NavigationPane. As soon as an ActionItem was triggered, the property ‘running’ was set to true. Then the QTimer was started with a delay of 100 ms. Before the QTimer starts, the System has enough time to disable all Action Items. Because I only want to use one QTimer Object I added a property ‘usecase’ to know which task from C++ should be started.

How to get the information, that the task was finished ? We’re using Qt Signals / Slots: from C++ a signal was emitted and in QML we connect a function (Slot) to this Signal. The connection itself was done in onCreationCompleted{}.

You’ll find different Slots (functions) connected in the sample APP. Here’s a simple function showing a Toast, stopping the ActivityIndicator and setting the property ‘running’ to false:

function processFinished() {
    activityIndicator.stop()
    navPane.running = false
    switch (myTimer.usecase) {
        case 0:
            infoToast.body = "10'000 Addresses processed.\nread JSON ..."
            infoToast.button.label = "OK"
            infoToast.icon = "asset:///images/ca_done.png"
            infoToast.exec()
            return
        case 1:
             ........
            default:
                return
    }
}
onCreationCompleted: {
    app.conversionDone.connect(processFinished)
}

There are more complex Slots getting result values from C++ if measuring Addresses or Speakers.

Here are some screenshots to see how the results from measuring Addresses are displayed.

We start with an empty Screen:

IMG_00000016

Starting the Action “Measure Addresses” the ActionItems will be disabled, a Toast will be shown and an ActivityIndicator is running:

IMG_00000017

This is a long running task and can take up to 2 Minutes !

Getting the Signal that the task is finished, the Toast changes the text and icon, the ActivityIndicator is stopped, a bar chart was drawn and the ActionItems are enabled:

IMG_00000018

The BarChart should also work in Landscape:

IMG_00000019

…and also on the Q10:

IMG_00000077

From this BarChart you easy understand how fast reading a JSON file is directly compared with XML.

You also see that the Z30 is much faster then Q10 (Z10 similar): 81 seconds vs 117 seconds to read a XML file with 10’000 Addresses.

Perhaps you know that Cascades provides no BarChart diagram as UIControl out of the box, but it can be done using plain Containers.

To get all of this smooth working there are some Cascades – layouting – tricks, so let’s take a look at the code step-by-step.

I’m using a StackLayout, but you can also use a DockLayout – it’s up to you. For me in this case a StackLayout seems to fit better. Here’s how the Containers are layed out:

barchartLayouting

All is contained into an outer Container with Top->Bottom orientation:

  • the ActivityIndicator: only visible and consuming space if running
  • two Containers using StackLayout Left->Right for the Header Labels using spaceQuota to adjust them correct
  • one Container using StackLayout Left->Right for the Footer Labels
  • in the middle the most difficult Container: the valueContainer – also using a StackLayout Left->Right containing 4 Containers: one for each bar

The valueContainer should consume all available space, so the footer Labels will be always positioned at the bottom. The available space will change if changing the orientation or starting/stopping the ActivityIndicator.

The 4 Containers inside the valueContainer represent the bars and are filled with a background color. The height of the bars must correspond to the values we’re getting from C++ where the largest value should fill out the complete height – all other bars will get a height relative to their values. This means every time the height changes we have to recalculate the height of these Containers.

The trick to get it done is using LayoutHandlers: a LayoutHandler will be notified if the Layout changes and you can ask the handler about the height, which is the value we need in this case. We attach a LayoutHandler to the valueContainer and if the height changes, we recalculate the values.

Page {
    id: measureAddressesPage
    property int barHeight: 1200
    onBarHeightChanged: {
        calculateBarChart()
    }
    property int max: -1
    property int readJson: 0
    property int writeJson: 0
    property int readXml: 0
    property int writeXml: 0
    ......
    Container {
        id: valueContainer
        topPadding: 10
        preferredHeight: measureAddressesPage.barHeight
        layout: StackLayout {
            orientation: LayoutOrientation.LeftToRight
        }
        attachedObjects: [
            LayoutUpdateHandler {
                id: valueContainerLayoutHandler
                onLayoutFrameChanged: {
                    measureAddressesPage.barHeight = layoutFrame.height
                }
            }
        ]
	    ......
	 }
}

We store the values as properties at the Page (max, readJson, writeJson, readXml, writeXml) and there’s also a property ‘barHeight’.

The ‘barHeight’ property is bound to the preferredHeight of the valueContainer and initially set to a high value: 1200. Cascades tries to set this value as preferredHeight, but is intelligent only to use a height which fits into the available space. As we already discussed the available space can change – per ex. starting the ActivityIndicator will reduce the height and stopping will enlarge the height. In this case the LayoutHandler will get a notification and we’ll use the actual assigned height to recalculate the values.

There are some special situations where the LayoutHandler from valueContainer won’t be notified about Layout changes, but the outerContainer was notified. So as a little trick we also add a LayoutHandler to the outerContainer and if the Layout changes we simply set the preferredHeight of the valueContainer to a high value (1200) causing Cascades to grab all the available space and so on….

Another situation is if coming back from Landscape to Portrait where we also set this hight value to let Cascades do the calculating of the available space.

The TitleBar was set as sticky – this is a trick to get the correct height on Q10. Otherwise the Q10 will use a floating TitleBar, which means the height of the TitleBar will be ‘available’ and then your footer Labels will be only visible if scrolling the TitleBar away. Setting the TitleBar as sticky solves this.

Please take a deeper look at the sample code …

Business Logic (C++)

Now let’s see what happens at C++ side.

we need the ‘app’ context property to get access to the app. Also all methods must be invokable from QML. Take a look at the sources or the previous articles of this series.

We now want to concentrate an JSON and XML. You must include:

    #include <bb/data/JsonDataAccess>
    #include <bb/data/XmlDataAccess>

Now we can convert. Here’s an example HowTo read the content of a File, convert from XML to JSON and save the File:

void ApplicationUI::convertXMLtoJSONAddresses() {
	XmlDataAccess xda;
	QVariant data;
	QString filenameXML;
	filenameXML = "addresses_us.xml";
	QFile dataFile(assetsXMLPath(filenameXML));
	if (!dataFile.exists()) {
		// do something
		return;
	}
	bool ok = dataFile.open(QIODevice::ReadOnly);
	if (ok) {
		data = xda.loadFromBuffer(dataFile.readAll());
		dataFile.close();
	} else {
		// do something
	}
	QString filenameJSON;
	filenameJSON = "addresses_us.json";
	JsonDataAccess jda;
	jda.save(data, dataPath(filenameJSON));
	emit conversionDone();
}

Removing the File-Open-Read-Write code converting is as simple as:

XmlDataAccess xda;
QVariant data;
...
data = xda.loadFromBuffer(dataFile.readAll());
...
JsonDataAccess jda;
jda.save(data, dataPath(filenameJSON));

You can load from Files or from a ByteArray you’re getting from a Network request. JsonDataAccess or XmlDataAccess will give you a QVariant. This QVariant can be a QVariantMap if the content is a single Object or a QVariantList if it’s an Array.

In the sample app you’ll find samples HowTo convert from XML to JSON or from JSON to XML, you’ll see how Signals are emitted and HowTo convert the Speakers to a much smaller JSON file with really fast READ access. Here’s the MeasureSpeakerPage:

IMG_00000020

Don’t forget

Please don’t forget to add the libraries to your .pro file

LIBS +=  -lbbsystem -lbbdata

Summary

from this sample you have learned

  • HowTo create a Custom BarChart diagram
  • Enable / Disable ActionItems
  • Use of SystemToast in QML
  • Use of ActivityIndicator
  • Use of LayoutHandler to provide flexible Layouts for all Devices
  • Connect QML functions to C++ Signals
  • HowTo read and write JSON and XML files
  • HowTo convert JSON to XML or XML to JSON

The main goal of this article was to demonstrate the speed of JSON and the easy way to deal with JSON.

The next part of this series will explain

  • How I’m using JSON to persist the data in a conference app
  • HowTo attach JSON data to ListView DataModels
  • HowTo search and filter JSON data.

We’ll use the same data as used in this sample.

Download and Discuss

The Sample APP is available at GitHub Open Source (Apache 2 License): https://github.com/ekke/cascades_json_vs_xml

I also created a Thread in the Forums, where you can ask or discuss this.

Have fun with the sample app and copy/paste what you need to implement Custom BarCharts into your own apps or to use JSON for your data files.

No Comments

Be the first to start the conversation!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s