Posts in Category "Data Management"

Large Data Sets (Part 2 of 3?)

Our second topic for dealing with large data sets is sorting. The problem with sorting is you need to have all of the data available in order to produce an accurate sort. However, as we saw in the last article bringing down all of the data from server to client is incredibly expensive and therefore unrealistic. Even normal web applications rarely load all of their data from the database server to the app server when dealing with a large amount.

Instead of downloading all of the data to the client we’ll let the database take care of sorting for us (which is one of the things a database is very good at). Let’s build on our examples from the previous article and add sorting capability to the implicit paging solution. The examples can be found here and are a superset of what we did in article 1 (meaning that the old examples are there too).

Continue reading…

Large Data Sets (Part 1 of Many)

This is the first of what will most likely be many postings related to data management and Flex. Our opening topic is a brief discussion on large data sets and how to efficiently transfer them from your application server to the client. I’m going to assume you are familiar with MXML and the components and services that are a part of the Flex runtime.

This post will refer to many examples that are found here. Instructions for installing the samples can be found in a README included in the zip.

The Problem

Our problem is simple. We need to display a lot of data, and the user does not want to wait too long for it to appear. The more data a user requests, the longer it takes for the data to display. What we need to do is a find a way to download data in pieces so that the user can view a smaller amount of data immediately and then only wait a brief amount of time as they request additional data. To keep the problem domain simple we are only going to discuss improving the actual retrieval of data. We’ll leave discussions on sorting, modification, filtering, etc. for another time.

Example 1 can help you visualize this problem. The data is census information that I got from The UCI KDD Archive and reduced to 20,000 records. Start by downloading a few records, perhaps 10. This doesn’t take long at all. If we increase that number to 100 the response is usually a little slower (these numbers can vary depending on your system setup, for example whether the database and application servers are on the same machine in addition to where you run the Flex app). Increase the number to 1000 and you’ll see a bigger jump. Finally 5000 and the time is becoming painful. Painful enough that an end-user might give up.

Continue reading…