We understand that FDS supports lazy associations, but the documentation is unclear about whether it's necessary and in what manner to partially initialize an array which is not empty but in the "lazy" state, as opposed to the fully initialized state, and the empty state.
We are using the Java adapter, so our application is responsible for how data comes out of the DAO layer.
This is a good question. We will support the ability for your assembler to return referenced objects which only have the id of the referenced object populated for the final release but unfortunately the code changes required to make this work did not make it into beta2 or beta3. I think it will work in previous releases but only when you have cache-items set to false and auto-sync-enabled also set to false.
To give you more detail, in Beta2 and in Beta3 the system expects that you return your items so that even lazy referenced items have their complete state populated. Partly this is so that we can cache those items and avoid calling your "get" method when the client requests a lazily reference instance. You can turn that off by adding <cache-items>false</cache-items> to your destination's properties. The other reason we rely on that is so that we can store the referenced object graph to detect changes when auto-sync-enable is true. For example, client A may be caching the entire graph but client B may only be caching a single leaf item. We need to have that reference graph so we know to update client A when client B changes the leaf object. Both of these will be fixed in the final release so we just won't cache values of lazy associations (unless you set a special flag) and we lazily build out the referenced graph as needed.
We deal with many tables with thousands, tens or even hundreds of thousands of records in which case means returning all of the results is not possible, even back to FDS in response to a fill request - let alone to a client over the network. How does the DAO layer need to respond to a fill request for a destination which is paged? Can lazy associations still be paged?
When you mention caching, it reminds me of another aspect of concern to us - stale data. Sometimes FDS is not the application which updates the database. Instantly the cache can be rendered obsolete. Has this problem been addressed in any way?
We are adding version columns to all tables to prevent problems getting into the database. However, this kind of column could be of interest to FDS if it is available. I have not thought it through completely, but should a configurable interval of time pass it might be useful for FDS to query the DB to get the latest version numbers to see if it needs to refresh the cache and resend to all subscribers a refresh.
I realize there is quite a bit here to contend with, but all of these issues come up in about the first hour of development in our environment....
The final aspects of the associated data set is filtering and sortation (client defined but server performed).
I see support within the Flash AS component library for these concepts but they are problems that really need to be solved on the server side. Client side sorting and filtering only really work with small datasets (not found often in the enterprise).
Do you know if FDS will conquer client-defined, server performed sorting and filtering?
If you are returning collections where the set of ids is too large to be returned all at once, you should implement that relationship using a fill method, not using an association property. In this case, usually the id of the related item is at least one of the fill parameters. For example, I have reimplemented the crm demo in beta 2 so that it returns the list of employees for a given company as a fill method which takes the company id as a parameter. This allows you to either use fill parameters to demarcate pages (i.e. you might have a page number parameter), or to use flex's built in paging support.
Currently in FDMS, the fill method must return all of the items to be returned in the fill so if the fill is too large to be returned as a single result set even for the server, you need to split it up using fill parameters to control the paging (in other words, you have to do paging yourself and can't use fdms' builtin paging support to do this automatically). Though we wanted to support complete 'client controlled' paging all the way through to the database, it is fairly challenging to keep a list sync'd between two clients when you do not even know the complete contents of the list, how big it is, the ways the client can access pages etc. We might be able to support a variant of the fill method which implements paging only for the auto-sync-enabled=false case (where you are not sync'ing changes between clients) but it sounds like you need clients to be kept in sync even on these large lists as they change?
I think that in the general case of client controlled but server performed filtering and sorting, you pass in the sorting and filtering criteria (including ranges) via fill parameters. The server performs the operation and returns that set of items so you get an ArrayCollection on the client which contains just a page of data at a time. If you want to implement a virtual array collection on top of that which represents the entire data set that will then make fill calls as needed to fetch pages you'd have to do that yourself in action script code. Another strategy would be to make the paging controls explicit in the UI (ie. have a next page button which fetches the next page instead of scrolling through the list) and that would make it easier.
In terms of caching, when you have auto-sync-enabled set to true, FDMS must at least cache the graph of ids of all of the managed items by all active managed clients. By default, it caches the items too but you can turn this off with <cache-items>false</cache-items>. If you need to keep your clients up-to-date with changes performed by code that accesses the DB directly or is just not going through flex, you can use the 'server push api' to tell flex about these changes, or just use this api to periodically refresh your clients. If your clients do not need to be made aware of these changes, you can just let them drift out of sync and use the conflict detection facilities to deal with stale updates. If you are using timestamps for your version numbers, you could probably implement a "get me a list of everything that has changed" query in a scheduled task which then uses the server push api to keep clients in sync.