On Aug 16, 2011, at 9:06 , Patrick Ohly wrote:
> On Aug 3, 2011, at 16:31 , Patrick Ohly wrote:
>> [...] Is there some way to force UID comparison for added items even in a
>> normal, incremental sync?
> Something like a slow sync match on UIDs for adds, to convert them to updates?
> That's something that could be added to the engine if we add UID-based
> matching support natively (would make sense, but also quite some work
> to get it right).
Would it be simpler if it was done without native UID-based matching?
Perhaps by specifying that a comparescript also needs to be run for
It would be a run trough the entire sync set. Which is never available in a searchable
form in the client side (and I don't have a reasonably quick and safe idea how to
change that), and in the server it is only loaded in slow syncs - during normal syncs only
changes are loaded. Of course that could be changed, but I doubt it's worth the effort
- because getting it right without breaking some of the more complicated features like
filtering will be hard work...
> However I once had a similar case, where a backend would do an
> implicit merge instead of an add based on the contents (wasn't the
> UID, but that doesn't matter) of the to-be-added entry.
> For that we added the possibility for the plugin to return
> DB_DataMerged (207) when adding the item caused some sort of merge in
> the backend.
SyncEvolution already does that. But because it can't do a real merge of
the data, some information is lost.
Why can't it do a real merge? If I understood the scenario correctly, at the time the
"other" item arrives from the peer the backend still has it's own version
unmodified, and should be able to merge both semantically correctly behind the scene, and
then mark one deleted and the other one changed for the next sync?
I can imagine imagine practical reasons which make this difficult/not feasible to do in
your backend without duplicating a lot of functionality for doing the merge (hello,
libvxxx...) but otherwise I see no obstacle.
> This causes the engine to read the item back from the backend
> having added it, assuming to get back the merged data, which is then
> sent back to the client in the same session.
> This only works server-side, because on the client there is no way to
> send updates after having received server updates (must wait for the
> next session).
> Your backend would need to issue a delete for the other item in a later session to
SyncEvolution is not doing that. Can you elaborate?
What happens is this, from the backend's perspective:
* Item with local ID FOO exists.
* Backend asked to add new item.
* Backend detects that the new item is the same as the existing
item, returns 207 and local ID FOO.
* In the next sync, the item with ID FOO is reported as
"unchanged". Nothing is said about the other item because as far
as the backend is concerned, it doesn't exist and never has.
But yes, when the sync started, the backend should have reported the original item as a
new one (before the merge occurred). So for the engine, the situation would be that
* item with localID FOO exists
* item with remotedID BAR is coming in as an add
* backend merges it with existing item and returns localID FOO with status 207
Now in the server case, the merge occurs BEFORE changes are sent to the client. So the
server should probably
* read back the item with localID FOO to get the result of
the merge (that's what actually happens already)
* search the list of items to-be-sent to the client for
another item with the same localID. That would be the
item reported as an add to the engine BEFORE the merge
* if one is found, and it is an add, remove that item from
the to-be-sent list to avoid to ever create a duplicate
at the client's end
* if one is found, and it is a replace for a already mapped
remoteID, convert it to a delete. After all, it could
well be that the sequence of events is:
* invitation added to server
* sync -> new item gets added to client
* invitation added to client (dumb, not detecting
* sync -> client side version gets added to server,
detected as duplicate.
All but the first bullet point are not implemented so far.
> Overall, I'm not sure where these types of cross-item merges
> conceptually. I guess with series and exceptions such a merge could
> easily get more complicated and involve multiple items, so I wonder if
> that should better be implemented outside the SyncML scope, like a
> "robot user" who is editing the database between syncs on one end or
> the other.
I think it fits into the engine. The Funambol server has worked in this
mode (always do mandatory duplicate checking on adds) for a long time. I
have my doubts whether it is always the right choice (for example, one
cannot add "similar" contacts that the server deems identical), but for
iCalendar 2.0 UID it would be.
I see that the actions to be taken after a backend-detected merge make sense to be added
to the engine, as outline above.
However, actually detecting and merging a duplicate belongs into the backend, as the
search usually must extend beyond what the engine sees as "current sync set"
(imagine a date-range filtered sync, and an invitation added on both sides which is to far
in the future to be in the date range window. The candidate for a merge could not be found
in the sync set!).
The problematic part is indeed the client side, because a duplicate
detected there will temporarily leave client and server out of sync. But
when assuming that the server does the same duplicate checking, it can
only happen when a new item was added on the client during the sync
(otherwise the server would have detected the duplicate).
Agreed. Technically, modifications happening *during* sync in real time are considered
having happened *after* sync (except if the backend cannot pinpoint the modification dates
of all entries modified by sync excactly to the sync reference token sampled at the
beginning of the sync).