Small Gains: Big Impact
In the transit-planning universe, planners and economists often get excited about 5 minutes shaved off of a 30-minute bus ride. The individual rider might shrug at a mere 50 minutes saved per week, but the planners and economists multiply that 50 minutes by the number of people who ride that route during a typical week and see something much bigger.
There’s a compounding effect: it doesn’t take long before these small gains in efficiency add up to a huge economic impact overall.
What’s this have to do with FileMaker, you ask? Well, the new FileMaker 15 is now available, and the inevitable question comes up “should we upgrade?” Sometimes the best discoveries when a new version of FileMaker hits the shelves are the “sleeper” features and under-the-hood improvements. In the most extreme cases, these go entirely undocumented by FileMaker Inc.; such is the case with the improvements in portal performance that accompany the FileMaker 15 release.
Portals being a very important part of the FileMaker user interface, anything affecting their performance, good or bad, is going to have a major impact on the user’s experience. Users of a database application often spend large chunks of their workday in the application, so, like that 5 minutes shaved off the bus ride, small efficiencies add up through the same compounding effect. And, as you’ll see, this one is a “big” small efficiency!
No, I’m not talking here about the new, threaded portal rendering (aka “inline portal progress bar”) that is a key public feature point for FileMaker 15; I’m talking raw performance optimization. So impressive is this optimization, in fact, that I never even saw the new inline progress bar during my testing!
We didn’t set out to measure differences between FileMaker 14 and FileMaker 15, and the marketing materials gave us no reason to expect them. We had developed a file to assess performance of several different techniques (possible future blog posts!) over the wide-area network (WAN). At Beezwax, we often build large database applications that are hosted in one place and then accessed by clients around the world, so testing and optimizing performance over the WAN is always critical. We hosted this particular file on a server in London, so that we could measure the new techniques’ performance over the WAN from our home office in Oakland, California. It happened that the file included a portal that shows 10,000 records.
Noticing that this 10K-row portal loaded so rapidly and scrolled so smoothly in a FileMaker Pro 15 client prompted us to compare its performance using a FileMaker Pro 14 client. The result was a huge “wow”!
Performance Testing Methods
Our setup was an FMP file hosted in London, on FileMaker Server 15. Our testing was performed in Oakland, California using both FileMaker Pro 14 and FileMaker Pro 15 clients. Server and client machines are both connected to a corporate network via VPN. Additional hardware, software, and network details are provided in the footnote, for technically curious readers.
In order to test portal performance itself, independent of the techniques we initially set out to test, I added a single parent record with a UUID primary key, and created a standard equijoin to 10,002 child records, all with the same UUID value in a foreign key field; that is, one parent record with 10K related records displaying in a portal. For each test, I quit FileMaker to clear cached data, relaunched, opened the hosted file to a no-record Home layout, and navigated directly to a layout showing the single parent record and either an unsorted or sorted portal.
For the scrolling tests, I didn’t drag the scrollbar’s thumb, which skips records, but scrolled as quickly as I could (without locking up the layout in the FileMaker Pro 14 tests) by swiping on the trackpad, measuring the time it took to scroll the entire length of the portal. The results presented here for each set of tests are the mean of 3 replicate tests for each client (FileMaker Pro 14 or FileMaker Pro 15). All tests were highly reproducible: coefficients of variation were 5% or less for the portal-loading tests, and less than 20% for the scrolling tests.
Time to initially load and render the portal
FileMaker 15 consistently loaded and rendered the portal faster than FileMaker 14 (Figure 1).
FIGURE 1. Time to Load and Render Portal
For unsorted portals, both clients loaded reasonably fast, although there was a highly reproducible 2-fold difference in loading time favoring FileMaker 15. The 2 seconds that version 14 required to render the loaded portal may not seem like much, but remember that each time you navigate to a different parent record, the portal needs to be loaded with fresh data (and FileMaker 14 lacks the inline progress bar feature, so you have to wait for the portal to completely draw and load before you can interact with the layout or move on to the next record): it’s that 5 minutes on the bus again.
For sorted portals, the difference was more dramatic, with FileMaker 15 showing only a negligible (0.1 second) penalty for sorting the portal, and a 5-fold improvement over FileMaker 14, which consistently threw up a “records remaining to sort” dialog.
Time to scroll length of portal
Here’s where my socks started to really get knocked off!
FIGURE 2. Time to Scroll to End of Portal
Both the sorted and unsorted versions of the 10,000-row portal scrolled quickly and smoothly in FileMaker 15. This was so contrary to our extensive experience with previous FileMaker versions that it is what first prompted me to open the file in FileMaker Pro 14, just to remind myself that I’ve never seen a large portal like this perform so responsively in a hosted file. As expected, in FileMaker 14, attempting to quickly scroll through the portal was almost painfully choppy and halting.
Time to put some metrics to it (Figure 2). Your eyes are not deceiving you: FileMaker 15 performed 31-fold faster for unsorted portals, and 12-fold better for sorted portals. Scrolling through this long portal in FileMaker 14 was an exercise in patience. The tests were actually a bit hard to conduct; pushing too hard brought the scrolling performance to its knees and the portal would almost freeze up periodically. Interestingly, the sorted portal performed a bit better (133 seconds vs. 154 seconds), presumably due to the sorting operation resulting in some additional caching, but was still unacceptably slow. FileMaker 15, in contrast, raced along with nary a stumble, completing the task in a small fraction of the FileMaker 14 time; there was a small, reproducible penalty for sorting (11.4 seconds vs. 4.9 seconds), but scrolling was extremely responsive nonetheless.
Conclusion
FileMaker engineers have obviously done some serious optimization of portal performance in FileMaker 15, resulting in much faster loading (particularly notable for sorted portals, which almost seem not to need additional time to sort the records) and a remarkably improved scrolling performance with large portals.
What’s this mean for you?
If you’re an end user, it’s back to the bus analogy, where small efficiencies add up. You’ll immediately appreciate the more responsive UI and spend less time waiting for your database to catch up to you. While these improvements will be most noticeable in solutions hosted over the WAN, they will be there even in solutions hosted on a local network, and will add up over time through the compounding effect.
If you’re a developer, this opens up the possibility of using portals, even portals displaying a large number of records, as more central components of the UI. In the past we might have avoided UI patterns that are highly portal-dependent, but the results presented here suggest that we have turned a major corner in this regard. Of course, this is really an end-user advantage, because we can provide our users with ever more flexible options in UI design.
For everyone, this means that one of the more compelling reasons to upgrade to FileMaker 15 is one that didn’t even make the marketeers’ bullet-point lists, but is big enough to bring a smile to all of our faces.
Technical Details
Round-trip ping times: 171.5 ± 1.24 ms [avg ± SD, n=10]
Server machine: Mac Mini with 2.8 GHz chip and 8 GB RAM.
Server memory cache set to default (512 MB)
Client cache set to default (256 MB, for both FMP14 and FMP15 clients)
Thanks for sharing, Mark!
In addition to Daniel’s (weetbicks) question about cacheing improvements in FM 15, I’d love to see further testing to narrow down which side(s?) of the equation give you the most benefit.
You mention that your testing was done with FMS 15, using FM 14 & 15 clients. I’d love to see the same tests run with FMS 14, comparing FM 14 & 15 clients in that situation.
From what we have observed, there are noticeable performance gains to be had just by upgrading your client machines to 15, even if the server is still on 14. Are you seeing further gains by having 15 on both ends?
There are other areas where having 15 on both ends makes a difference. For instance, the new Truncate Table script step requires 15 on both ends. Handy if you need to clear out a temp table frequently
Thanks!
-Shawn
Thanks, Shawn, for your comments about performance improvements to be gained just by upgrading the client machines.
Our collaborator who hosted this particular file for us, so that I could test from the comfort of our Oakland office, had upgraded his server installation to FMS15. This setup made sense as the best test platform, as it’s likely to be the most representative of our users’ experience going forward. (And our initial goal with this particular hosted file wasn’t a 14 vs. 15 comparison at all.) My guess is that the scrolling performance is all client-dependent, since we’re talking about locally cached data (but I’d certainly like to confirm, if we manage to arrange to host the same file on FMS14 abroad and repeat the tests here). As for cache reuse and it’s potential contribution to initial portal load times, we’ve confirmed that that is a client feature available in FMP15, irrespective of whether the server is 14 or 15.
BTW, speaking of Truncate Table, I presume you’ve seen our blog post by fellow-Bee, Jud Wolfskill (https://blogbeezwax.wpcomstaging.com/2016/06/08/speeding-filemaker-performance-with-truncate-table/) — another sign that performance was a major focus for FMI in this release cycle!
Mark, I hadn’t seen that article, but just checked it out now. Great info there… especially the bit about multi-users in a mixed 14/15 environment. All the more reason to push clients to upgrade quickly, rather than a lazy roll-out. Thanks again!
hi Mark. Great article thanks for sharing! I wanted to make sure your findings are not the results of caching optimizations in 15 and are actually to do with the portals themselves.
During a normal session in 14/15, data is cached as it is downloaded from server to client. Once records in a portal are downloaded, going back to that portal during the session will show the records load fast (unless you flush the cache in the mean time).
With FileMaker 14, the local client cache was cleared after every logout of the database, and so each login you had to redownload data.
In FileMaker 15, the local cache is kept between sessions, and is instead cleared upon a machine restart. This means if you ran your sorted tests in 15 after your unsorted scroll test then this could explain your very large performance gains here (even if you closed and re-opened the solution it would still be fast due to the cache).
The true way to test would be to restart your machine after each test. The caching however does give very marked improvement too, but it would be great to be sure that these gains are purely portal optimization realted and not the caching 🙂
cheers!
Thanks, Daniel, for the kind thoughts and additional insights! While we were putting the blog post together, we became aware of the cache-reuse changes in 15 and agree that these could, in whole or in part, explain at least the portal-load time improvements. It would seem less likely to explain the scrolling performance improvements, which are the biggest part of what makes us hopeful that portal-centric design patterns have taken a huge leap in 15: even after you’ve scrolled the length of a portal in FM14 and the portal’s data is all cached on client, further scrolling remains as painful as ever! That, arguably more than anything, is what has historically steered us away from large portals in our designs.
We had thought we would hold off on expanding our testing, and commenting on this new cache-rescue behavior, until after DevCon, where we’re all likely to learn a lot more about the under-the-hood performance improvements. I think that, now that you’ve brought it up, it makes sense to repeat our tests and see how much contribution cache reuse makes to initial portal load times. (Mind you, even if that is the only factor contributing to the improved load times, we’ll happily take it. 🙂 Anything that helps us use portals with less trepidation is welcome, and it does seem like we’ve turned a corner here!)
One small correction to your comments is that even machine restarts don’t clear the saved caches; they are still saved and reused. (Stale portions are, of course, refetched from server, restart or not.) This potentially complicates performance testing where one wants to clear all caches to get a “clean baseline” between replicate tests. Or does it? It could be argued that the new “clean baseline” is one in which the reusable caches are not cleared, but rather just accepted as a given. That said, you’ve piqued my curiosity to see what effect they have, and I hope to repeat my tests over the coming days. When I do, I’ll post up my results here.