Tuesday, December 23, 2014

Mobile Development Platform Performance (Native, Cordova, Classic Xamarin, Xamarin.Forms)

Last month I published my Mobile Technology Decision Making White Paper and I feel that goes a long way to helping choose what technology to use in the mobile space.  One question I get asked frequently is how do the different mobile development platforms compare to each other from a performance perspective. While I've heard a lot of anecdotal information I have not seen a lot of formal comparisons.  At Magenic we normally work with native (iOS, Android, WP), Cordova and Xamarin so I wanted to compare these.  I plan on this being the first post on this topic.

For some background on testing methodology using Android and iOS.

The Development Platforms
- Native (Objective-C 64 bit and Java)
- Cordova (Multi-Device Hybrid Apps) using Intel's App Framework for the UI
- Classic Xamarin (64 bit unified beta for iOS)
- Xamarin.Forms (64 bit unified beta for iOS with beta version of Xamarin.Forms, note latest version of the unified API in the beta/alpha channels could not be used as it is not supported by Xamarin.Forms Note: 1.3.1 pre 1 was released Dec 24th so Xamarin.Forms may now work with the version of the unified iOS API in the alpha and beta channel)

The Devices
- iPad Mini (non Retina) running iOS 8.1.1 (12B435)
- ASUS K00F running Android 4.2.2

The Test Apps
Applications were made for each of the development platforms that are functionally similar. There was little (if no) effort to make them look exactly the same or even look "good".  But they looked about the same.  There were some differences such as Java, Classic Xamarin and Xamarin.Forms rendered the tabs on the top in Android as expected while the JavaScript library showed them on the bottom.

The Timing Methodology
Due to difficulties in knowing when things are "done", particularly with JavaScript, timings were handled via stopwatch.  Each timing was taken ten times and the results were averaged.  It should noted that hand timings have an accuracy of about 2/10 of a second so that does give us an approximate margin of error.

Test 1: Test App Size
The size of the application can impact how much bandwidth it takes to deploy and also have some impact on load times.  For Android the size of the APKs was examined.  For iOS I looked at Settings to find out how much space the apps took up on disk.


Development PlatformSize
Android
Java166kb
Cordova433kb
Classic Xamarin3.5mb
Xamarin.Forms4.7mb
iOS
Objective-C (64 bit)644kb
Cordova2.7mb
Classic Xamarin12.1mb
Xamarin.Forms16.9mb

When it comes to application size Xamarin shows the extra size involved in the overhead of the .Net framework.  There was an attempt to reduce the size of the deployed Xamarin application by using the "Link SDK assemblies only" setting.  I am surprised in a very small application how large the difference is.  However, from experience in "real" applications the difference is much less consequential as graphics and frameworks get added to the projects.

Test 2: Load Times
I wanted to see how long it took the application to load into memory.  While the initial load time is important, many mobile applications tend to stay in memory so it tends to have a limited impact.  For this test I made sure to close all applications before each timing.

Development PlatformTest Avg.
Android
Java1.085
Cordova3.978
Classic Xamarin1.704
Xamarin.Forms2.764
iOS
Objective-C1.221
Cordova1.715
Classic Xamarin1.28
Xamarin.Forms1.813

In all cases the vendor native technologies loaded the fastest.  Classic Xamarin loaded nearly as fast as the native languages.  Xamarin.Forms and Cordova had the slowest load times.  The Cordova load time on Android was particularly bad while on iOS the load times were close enough to not be a huge factor.

Test 3: Loading a List from Azure Mobile Services
In this test I wanted to look at getting data from an external service so I loaded 1000 records from Azure Mobile Services.  For Xamarin iOS 64 bit I had to modify the Azure Mobile Services to be compatible with the unified API.  The timings were taken from pushing the button to load the list until the results visibly came back and were displayed on a list on the screen.

Java:
public  void addRecord(String firstName, String lastName, int index, String misc) throws Exception {
    if (dbConn == null) {
        openConnection();
    }

    ContentValues values = new ContentValues();
    values.put("firstName", firstName);
    values.put("lastName", lastName + index);
    values.put("misc", misc);
    dbConn.insertOrThrow(TABLE_NAME, null, values);
}

Objective-C:
- (void)addRecord:(NSString*)firstName withLastName:(NSString*)lastName withIndex:(int)index withMisc:(NSString*)misc withError:(NSError**)error {
    NSString *sqlStatement = NULL;
    char *errInfo;
    *error = nil;

    if (dbConn == nil) {
        [self openConnection:error];
        return;
    }
    
    sqlStatement = [NSString stringWithFormat:@"%@%@%@%@%d%@%@%@", @"INSERT INTO testTable (firstName, lastName, misc) VALUES ('", firstName, @"', '", lastName, index, @"', '", misc, @"')"];
    
    int result = sqlite3_exec(dbConn, [sqlStatement UTF8String], nil, nil, &errInfo);
    
    if (result != SQLITE_OK) {
        NSMutableDictionary *errorDetail = [NSMutableDictionary dictionary];
        [errorDetail setValue:[NSString stringWithFormat:@"%@%s", @"Error writing record to database: ", errInfo] forKey:NSLocalizedDescriptionKey];
        *error = [NSError errorWithDomain:@"testDomain" code:101 userInfo:errorDetail];
    }
}

JavaScript:
db.executeSql("INSERT INTO testTable (firstName, lastName, misc) VALUES (?,?,?)", ["test", lastName, "12345678901234567890123456789012345678901234567890"], function (res) {
    successCount++;
    if (successCount === maxValue) {
        $.ui.popup({
            title: "Success",
            message: "All records written to database",
            doneText: "OK",
            cancelOnly: false
        });
        $.ui.unblockUI();
    }
}, function (e) {
    $.ui.popup({
        title: "Error",
        message: "An error has occurred adding records: " + e.toString(),
        doneText: "OK",
        cancelOnly: false
    });
    $.ui.unblockUI();
    return;
});

Xamarin (All Versions):
public void AddRecord(string fName, string lName, int i, string m)
{
    if (dbConn == null)
    {
        OpenConnection();
    }

    var testRecord = new TestTable {firstName = fName, id = 0, lastName = lName + i, misc = m};

    dbConn.Insert(testRecord);
}

Xamarin Classic Android Alternate:
public  void AddRecord(string firstName, string lastName, int index, string misc) 
{
    if (dbConn == null) 
    {
        OpenConnection();
    }

    ContentValues values = new ContentValues();
    values.Put("firstName", firstName);
    values.Put("lastName", lastName + index);
    values.Put("misc", misc);
    dbConn.InsertOrThrow(TABLE_NAME, null, values);
}


Development PlatformTest 1Test 2Test 3Test 4Test 5Test 6Test 7Test 8Test 9Test 10Test Avg.
Android
Java22.7117.518.0417.718.6320.332.682.422.162.342.369
Cordova25.9924.7627.0523.324.0622.862.122.021.942.482.149
Classic Xamarin34.0727.3832.0538.7729.2734.631.611.631.841.851.738
Xamarin.Forms1.991.762.321.911.91.581.932.022.031.641.908
iOS
Objective-C2.382.442.242.32.342.322.322.352.22.272.316
Cordova3.572.182.071.951.972.052.041.932.21.962.192
Classic Xamarin21.871.882.061.741.91.811.941.751.961.891
Xamarin.Forms2.112.012.231.961.952.072.122.162.082.12.079
*results in seconds

In many ways this test is showing how well the Azure Mobile Services libraries perform on the different platforms.  Unsurprisingly Xamarin, with it's underpinnings of a .Net implementation, performs the best in this test.  I was surprised to see the libraries for the native technologies perform the worst, both on Android and iOS.

I have heard that the Xamarin.Forms lists can perform poorly with large data sets.  These results did not show that, at least with lists of up to 1000 records.

Test 4: Prime Number Calculation
In the final of my first series of tests I wanted to try out a CPU intensive operation.  I created a Sieve of Eratosthenes on each of the platforms.  My first plan was to calculate all prime numbers up to 50,000,000.  This required some special handling of the method's array for both Objective-C and JavaScript.  In the case of Objective-C I had to malloc memory to support arrays that large.  Also for Objective-C and JavaScript I had to initialize the array items to 0.  To keep the timings the same I did the array item initialization to 0 on all platforms even though it could have been left out for .Net (and Java I believe).  It that was done, the .Net timings would have been even better.

I did end up having to settle for only calculating primes up to 5,000,000.  The reason for this is that the JavaScript performed so poorly that I was unwilling to wait for it to complete 10 times.

Java:
private int getPrimesFromSieve(int maxValue)
{
    byte[] primes = new byte[maxValue + 1];
    for (int i = 0; i <=maxValue; i++)
    {
        primes[i] = 0;
    }
    int largestPrimeFound = 1;

    for (int i = 2; i <=maxValue; i++)
    {
        if (primes[i - 1] == 0)
        {
            primes[i - 1] = 1;
            largestPrimeFound = i;
        }

        int c = 2;
        int mul = i*c;
        for (; mul <= maxValue;)
        {
            primes[mul - 1] = 1;
            c++;
            mul = i*c;
        }
    }
    return largestPrimeFound;
}

Objective-C:
- (int) getPrimesFromSieve: (int) maxValue {
    Byte *primes;
    primes = (Byte *) malloc(maxValue * sizeof(Byte));
    for (int i=1; i<=maxValue; i++)
    {
        primes[i-1] = 0;
    }    

    int largestPrimeFound;
    largestPrimeFound = 1;

    for (int i=2; i<=maxValue; i++)
    {
        if(primes[i-1] == 0)
        {
            primes[i-1] = 1;
            largestPrimeFound = i;
        }

        int c=2;
        int mul = i*c;
        for(; mul <= maxValue;)
        {
            primes[mul-1] = 1;
            c++;
            mul = i*c;
        }
    }
    return largestPrimeFound;
}

JavaScript:
function getPrimesFromSieve(maxValue) {
    var primes = new Uint8Array(new ArrayBuffer(Number(maxValue)));
    for (var i = 0; i <=maxValue; i++) {
        primes[i] = 0;
    }
    var largestPrimeFound = 1;

    for (i = 2; i <= maxValue; i++) {
        if (primes[i - 1] == 0) {
            primes[i - 1] = 1;
            largestPrimeFound = i;
        }

        var c = 2;
        var mul = i * c;
        for (; mul <= maxValue;) {
            primes[mul - 1] = 1;
            c++;
            mul = i * c;
        }
    }
    return largestPrimeFound;
}

Xamarin (All Versions):
public static int GetPrimesFromSieve(int maxValue)
{
    var primes = new byte[maxValue + 1];
    for (var i = 0; i <=maxValue; i++)
    {
        primes[i] = 0;
    }
    var largestPrimeFound = 1;

    for (var i = 2; i <=maxValue; i++)
    {
        if (primes[i - 1] == 0)
        {
            primes[i - 1] = 1;
            largestPrimeFound = i;
        }

        var c = 2;
        var mul = i*c;
        for (; mul <= maxValue;)
        {
            primes[mul - 1] = 1;
            c++;
            mul = i*c;
        }
    }
    return largestPrimeFound;
}

Development PlatformTest 1Test 2Test 3Test 4Test 5Test 6Test 7Test 8Test 9Test 10Test Avg.
Android
Java4.314.314.24.334.394.374.324.454.344.44.342
Cordova91.699594.3194.494.7394.194.191.893.6397.7594.151
Classic Xamarin4.274.254.154.324.514.414.224.124.144.194.258
Xamarin.Forms4.214.174.314.34.24.344.294.364.224.194.259
iOS
Objective-C5.045.495.384.864.85.025.034.834.844.855.014
Cordova66.9667.3667.2267.367.1767.4467.1367.1167.5867.6467.291
Classic Xamarin4.414.424.354.344.494.374.174.274.394.284.349
Xamarin.Forms4.514.334.314.314.334.44.414.44.334.464.379
*results in seconds

While I was expecting that JavaScript would be slower, I was unprepared for how much worse it was for this type of operation.  This would make Cordova problematic for highly CPU bound work were performance is important (I wonder what that means for server side Node.js...).  CPU bound work is not as important in many of today's mobile applications but given the history of increased performance for mobile CPUs and what happened in the PC market in the 1990's, it is likely that in the future CPU bound work will be more prevalent in mobile applications.

Having said that, using native HTML commands can be very fast.  In the past I've testing loading JSON using HTML commands vs. the JSON.Net library on Xamarin and found them nearly comparable.

I was also surprised that Xamarin performed better than Objective-C by a noticeable amount.  It also performed better than Java on Android but by a very marginal amount, well within the .2 second margin of error that manual timings give us.

That's it for my first installment of performance tests.  Much of this code was made more difficult for Xamarin due to the flux around the 64 bit Unified iOS API.  For next month I'll take a look at loading large JSON strings and perhaps something else.

I hope this is useful or you.  If you have any ideas for performance tests I can perform, I'd love to hear them.

Source code for the tests can be found here: Performance Tests Source

16 comments:

  1. Thanks for taking the time to discuss this, I feel strongly about it and love learning more on this topic. If possible, as you gain expertise, would you mind updating your blog with more information? It is extremely helpful for me.
    mobile app development company usa

    ReplyDelete
  2. Peter,
    I am working putting together a set of test apps for the second post on this subject. I'll be focusing on saving/loading from a custom text file and saving/loading from SQLite.

    ReplyDelete
  3. THANKS! Great post, love when people actually do some real benchmarking instead of just spouting opinions.

    I think all can be the best route to go depending on the team/situation.

    ReplyDelete
  4. Daniel,
    I completely agree, they all have situations where they are a good fit. Companies looking to standardize on a particular mobile development technology have to weigh several different factors and performance is just one of them.

    ReplyDelete
  5. Great Post Kevin,
    The tests where exactly what I wanted to do myself, but couldn't find the time.

    The only thing I tested where loading times for the MVVM systems for Xamarin; MVVMCross (2.68), CrossLight (1.73), .Forms (2.08) and without any MVVM (1.24) systems.

    Thanks again for the test results and hope to see more in future.
    Toine db

    ReplyDelete
    Replies
    1. Toine,
      I'm glad you found it useful.

      I'm interested in what you found with the load times for the different MVVM frameworks?

      Delete
    2. My conclusion was Xamarin.Forms best suited to my needs, good loading time and has the most promising future.

      I added the results to your xls, I can send them if you want.
      PS: its a very short test with 3 runs on a real android device

      Delete
    3. Toine,
      I'd love to see the results. The test source code would be cool too.

      If you want I could run it on an iOS and Android device and put the results here (crediting you).

      Delete
    4. Hi Kevin,
      Where can I send it to?

      PS: I have the XLS, but I'm not sure if I canfind the sources anymore and if they are still workable. but I will look (no credits needed)

      Delete
    5. Toline,
      My email is my twitter handle, bowman74 and then at hotmail dot com.

      Delete
  6. Kevin, I'm fascinated by your result that the Xamarin compiler can beat Apple in generating machine code, since it's pretty tough to beat the company that actually designed the CPU itself.

    I’d like to try and replicate your results. Can you say which version of XCode you used along with the optimization settings?

    Regards,
    Lee

    ReplyDelete
    Replies
    1. I didn't capture the exact version but it was the stable version that was available at the time of this post. These findings match what I had heard as anecdotal tales for quite some time. The project and all settings can be found in Github.

      When you think about what is happening it shouldn't be overly surprising that in certain cases one particular native compilation will outperform another. This has been true for every other platform as well and no reason why iOS would be an exception. If you are watching the evolution of Swift you can watch Apple themselves slowly improving performance over time to get it on par with the Objective-C compilations. At the end of the day the compiler is just creating machine code that may run differently on different HW configurations. There are choices being made by the different compiler teams and those choices will result in certain operations performing better than others in different situations. As far as beating a company in the interoperation of the OS with the CPU, as I said in certain situations depending on the choices made by the compiler team, this is likely to happen, as has happened on every other OS before. No magic involved here.

      Delete
    2. I'm sorry I must not be seeing the GitHub link. Is it on this page?

      Thanks,
      Lee

      Delete
    3. No problem, it's the last line of the main post.

      Delete
  7. This comment has been removed by the author.

    ReplyDelete
  8. How to Choose Hybrid Apps Framework for your App

    The Next-Generation of Apps for the Mobile Technologies is the Hybrid Apps. Why you have to use the Hybrid Apps for the Business nor going to use Native Apps Development? Some of the Technologies, Frameworks and library benefits to do fast development and get the lead for your Business with low budget Apps.

    ReplyDelete