11:32 p.m.
Hi Patrick,
I have a few questions aimed at trying to understand what I see in the backend code
samples. I've been looking at SyncEvolution 0.8, because I assumed it was more stable
at this point in time. Anyway, my questions are a mix of my possibly false assumptions and
actual questions:
1) In the backend, are all the uid parameters local UID's? Need we have any concern
about mapping to remote ID? Is the engine handling that?
2) listAllItems(): the backend itemizes the entire local store and set
revisions[<localUID>] with last revision stamp or modification time.
3) a) createItem(): generate a sync item data blob to ship to the server, which could
be for add or update and we don't need to know?
b) insertItem(): either update or add item to the local store depending on whether UID
already exists locally?
c) deleteitem(): delete item from the local store
Does the sync engine handle deleting items on server based on the results of
listAllItems (UID's detected missing)?
4) Our contacts records do not have a last mod time
a) we are able to get a list of records that changed and another list of records
deleted since last time we checked
b) if my assumption that items not appearing in the listAllItems will be deleted
remotely by the engine, we can ignore this deleted list
c) with the changed records list, we could say last mod time is NOW (say, the start of
sync), and set a custom field in the record to act as last mod time. (Will have to re-save
the record locally.)
d) with unchanged records, do nothing but extract that last mod time from the custom
field
5) exportData walks through all records and create item data blobs - presumably this is
called in the slow sync scenario?
I appreciate your help - maybe these questions have been answered somewhere already. :)
Bob Eggers
________________________________________
From: Patrick Ohly [patrick.ohly(a)gmx.de]
Sent: Sunday, August 02, 2009 7:26 AM
To: Bob Eggers
Subject: Re: SyncEvolution
On Fri, 2009-07-31 at 08:51 -0700, Bob Eggers wrote:
I am looking at using the Posix build of SyncEvolution to support a
contacts client on the Palm webOS. I've been getting familiar with
SyncML concepts and have looked at a few engines. I like SyncEvolution
because of the WBXML support in particular.
You've mentioned Funambol and WBXML support in the same email - just to
be sure that there are no misunderstandings, you are aware of
SyncEvolution 0.9 using the Synthesis SyncML Engine? It no longer
depends on Funambol source code or libraries at all.
One thing I'm struggling with is "getting it". I know
I'm not the
first because I saw a just such a comment today in a Funambol slide
slow. The docs are generally sketchy, as you know, and the naming
conventions they used were not always the clearest.
I might have inherited some of that in SyncEvolution :-/
I have been studying backends that last couple of days trying to
size
up our task. I wonder if you might field a handful of questions from
me? If so, let me know.
Sure, just go ahead. However, if you don't mind, can we have that
discussion on the SyncEvolution mailing list
(syncevolution(a)syncevolution.org)? Don't worry about subscribing (unless
you want to, of course), just mail me with that list on CC and I'll make
sure that all your emails get through.
That way answers will be available for others later on; the other
SyncEvolution developers might also be able to help.
--
Bye, Patrick Ohly
--
Patrick.Ohly(a)gmx.de
http://www.estamos.de/
1:39 a.m.
New subject: [SyncEvolution] SyncEvolution
On Mon, 2009-08-03 at 06:32 -0700, Bob Eggers wrote:
Hi Patrick,
I have a few questions aimed at trying to understand what I see in the
backend code samples. I've been looking at SyncEvolution 0.8, because
I assumed it was more stable at this point in time.
I'd recommend 0.9 beta 3 at this point. From a user's perspective it has
one bug fix which never made it into a 0.8.x maintenance release
(regarding notes with only a summary and no description). For a
developer it simply is the code base which is under active development.
You've mentioned WBXML. That is only supported in 0.9, thanks to the
Synthesis engine.
Anyway, my questions are a mix of my possibly false assumptions and
actual questions:
1) In the backend, are all the uid parameters local UID's?
Yes.
Need we have any concern about mapping to remote ID?
No.
Is the engine handling that?
Mostly the server does it. There's only one case where the client sees a
GUID at all, that's when it is asked by the server to add a new item.
The server only has the GUID, so it sends that to the client together
with the data. The engine remembers the GUID, combines it with the LUID
created by the backend, then tells the server about the mapping. So even
in that case, the backend never has to deal with anything other than its
own LUID.
2) listAllItems(): the backend itemizes the entire local store and
set
revisions[<localUID>] with last revision stamp or modification time.
Correct. This is the concept used by backends derived from
TrackingSyncSource, which is the easiest to implement *if* there is a
suitable revision/last-mod property. A more flexible backend would
derive from EvolutionSyncSource and implement the change tracking
itself.
3) a) createItem(): generate a sync item data blob to ship to
the
server, which could be for add or update and we don't need to know?
Correct.
However, note that the "blob" is treated differently in 0.8.1 and 0.9:
in Funambol clients (0.8), the backend is responsible for converting to
and from the format expected by the SyncML server.
In 0.9, the "blob" is processed by the Synthesis engine and converted
into the format supported by the SyncML server. The complexity of that
conversion can range from very simple (using different X- extensions) to
very complex (iCalendar 2.0 locally, vCalendar 1.0 remotely).
This article introduces the conversion:
http://moblin.org/documentation/syncevolution/pim-data-synchronization-wh...
Here's more about the different layers of data conversion in
SyncEvolution 0.9:
http://moblin.org/documentation/syncevolution/architecture-testing
If your local datastore doesn't have a native serialization of an item,
then the SyncEvolution API should be changed so that you can exchange
individual properties directly with the internal Synthesis format
("field list"). The internal format can already be configured so that it
matches exactly what your client supports (important for accurate Device
Information).
b) insertItem(): either update or add item to the local store
depending on whether UID already exists locally?
That, and whether the caller provided a UID. Without a UID, do an add
first. The reason for making this a single call is that in some cases an
add must be turned into an update (iCalendar 2.0 meeting was imported
into server and locally, backend detects the match based on iCalender
2.0 UID) or an update into an add (only on the server side).
c) deleteitem(): delete item from the local store
Does the sync engine handle deleting items on server based on the
results of listAllItems (UID's detected missing)?
Yes.
4) Our contacts records do not have a last mod time
Then it might be more suitable for implementing the EvolutionSyncSource
base class. In that class you initialize your data store, then report
added, updated, deleted items yourself.
The Evolution backends used to work like that based on a similar "change
list" kept by the Evolution Data Server. I found that a bit limited, for
example, reading that list was destructive (it wasn't possible to just
read the list without erasing it). It might be better with your
implementation.
a) we are able to get a list of records that changed and another
list of records deleted since last time we checked
b) if my assumption that items not appearing in the listAllItems
will be deleted remotely by the engine, we can ignore this deleted
list
c) with the changed records list, we could say last mod time is NOW
(say, the start of sync), and set a custom field in the record to act
as last mod time. (Will have to re-save the record locally.)
d) with unchanged records, do nothing but extract that last mod time
from the custom field
Instead of faking a last mod field it might be easier to implement the
EvolutionSyncSource class.
5) exportData walks through all records and create item data blobs -
presumably this is called in the slow sync scenario?
No, slow sync goes through the normal createItem() call to get item data
as needed. exportData() (renamed to backupData() in 0.9, to match the
new restoreData()), bypasses the conversion into the Synthesis format.
This way it can provide a true local backup/restore with as little
transformations as possible. Besides backup/restore, it is also used by
the client-test test program. You don't have to implement it for sync.
I appreciate your help - maybe these questions have been answered
somewhere already. :)
No, not in detail. So it's worth doing this on the list :-)
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
4:35 a.m.
New subject: [SyncEvolution] SyncEvolution
Hi Patrick,
This is a huge help! It appears with 0.9 beta 3 we can abandon developing a vCard
reader/writer to deserialize/serialize the data. One thing I have not seen is a
replacement for Funambol's "Push Module". Is it there, or should we expect
to build our own?
Thanks,
Bob
________________________________________
From: Patrick Ohly [patrick.ohly(a)intel.com]
Sent: Monday, August 03, 2009 11:39 AM
To: Bob Eggers
Cc: syncevolution(a)syncevolution.org
Subject: RE: SyncEvolution
On Mon, 2009-08-03 at 06:32 -0700, Bob Eggers wrote:
Hi Patrick,
I have a few questions aimed at trying to understand what I see in the
backend code samples. I've been looking at SyncEvolution 0.8, because
I assumed it was more stable at this point in time.
I'd recommend 0.9 beta 3 at this point. From a user's perspective it has
one bug fix which never made it into a 0.8.x maintenance release
(regarding notes with only a summary and no description). For a
developer it simply is the code base which is under active development.
You've mentioned WBXML. That is only supported in 0.9, thanks to the
Synthesis engine.
Anyway, my questions are a mix of my possibly false assumptions and
actual questions:
1) In the backend, are all the uid parameters local UID's?
Yes.
Need we have any concern about mapping to remote ID?
No.
Is the engine handling that?
Mostly the server does it. There's only one case where the client sees a
GUID at all, that's when it is asked by the server to add a new item.
The server only has the GUID, so it sends that to the client together
with the data. The engine remembers the GUID, combines it with the LUID
created by the backend, then tells the server about the mapping. So even
in that case, the backend never has to deal with anything other than its
own LUID.
2) listAllItems(): the backend itemizes the entire local store and
set
revisions[<localUID>] with last revision stamp or modification time.
Correct. This is the concept used by backends derived from
TrackingSyncSource, which is the easiest to implement *if* there is a
suitable revision/last-mod property. A more flexible backend would
derive from EvolutionSyncSource and implement the change tracking
itself.
3) a) createItem(): generate a sync item data blob to ship to
the
server, which could be for add or update and we don't need to know?
Correct.
However, note that the "blob" is treated differently in 0.8.1 and 0.9:
in Funambol clients (0.8), the backend is responsible for converting to
and from the format expected by the SyncML server.
In 0.9, the "blob" is processed by the Synthesis engine and converted
into the format supported by the SyncML server. The complexity of that
conversion can range from very simple (using different X- extensions) to
very complex (iCalendar 2.0 locally, vCalendar 1.0 remotely).
This article introduces the conversion:
http://moblin.org/documentation/syncevolution/pim-data-synchronization-wh...
Here's more about the different layers of data conversion in
SyncEvolution 0.9:
http://moblin.org/documentation/syncevolution/architecture-testing
If your local datastore doesn't have a native serialization of an item,
then the SyncEvolution API should be changed so that you can exchange
individual properties directly with the internal Synthesis format
("field list"). The internal format can already be configured so that it
matches exactly what your client supports (important for accurate Device
Information).
b) insertItem(): either update or add item to the local store
depending on whether UID already exists locally?
That, and whether the caller provided a UID. Without a UID, do an add
first. The reason for making this a single call is that in some cases an
add must be turned into an update (iCalendar 2.0 meeting was imported
into server and locally, backend detects the match based on iCalender
2.0 UID) or an update into an add (only on the server side).
c) deleteitem(): delete item from the local store
Does the sync engine handle deleting items on server based on the
results of listAllItems (UID's detected missing)?
Yes.
4) Our contacts records do not have a last mod time
Then it might be more suitable for implementing the EvolutionSyncSource
base class. In that class you initialize your data store, then report
added, updated, deleted items yourself.
The Evolution backends used to work like that based on a similar "change
list" kept by the Evolution Data Server. I found that a bit limited, for
example, reading that list was destructive (it wasn't possible to just
read the list without erasing it). It might be better with your
implementation.
a) we are able to get a list of records that changed
and another
list of records deleted since last time we checked
b) if my assumption that items not appearing in the listAllItems
will be deleted remotely by the engine, we can ignore this deleted
list
c) with the changed records list, we could say last mod time is NOW
(say, the start of sync), and set a custom field in the record to act
as last mod time. (Will have to re-save the record locally.)
d) with unchanged records, do nothing but extract that last mod time
from the custom field
Instead of faking a last mod field it might be easier to implement the
EvolutionSyncSource class.
5) exportData walks through all records and create item data blobs -
presumably this is called in the slow sync scenario?
No, slow sync goes through the normal createItem() call to get item data
as needed. exportData() (renamed to backupData() in 0.9, to match the
new restoreData()), bypasses the conversion into the Synthesis format.
This way it can provide a true local backup/restore with as little
transformations as possible. Besides backup/restore, it is also used by
the client-test test program. You don't have to implement it for sync.
I appreciate your help - maybe these questions have been answered
somewhere already. :)
No, not in detail. So it's worth doing this on the list :-)
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
5:07 a.m.
New subject: [SyncEvolution] SyncEvolution
On Mon, 2009-08-03 at 19:35 +0100, Bob Eggers wrote:
This is a huge help! It appears with 0.9 beta 3 we can abandon
developing a vCard reader/writer to deserialize/serialize the data.
Yes. Have a look at the data handling in Synthesis, then let's talk
about a suitable API which exposes the Synthesis field list to
SyncEvolution backends. I expect that we'll have to break up
EvolutionSyncSource and TrackingSyncSource into its different interfaces
(integration into SyncEvolution, Synthesis DB API) and then allow
backend developers to mix and match.
One thing I have not seen is a replacement for Funambol's
"Push
Module". Is it there, or should we expect to build our own?
There is none. If you want to build one which is compatible with the
Funambol protocol, then this certainly would be a worthwhile extension.
Of course, I would hope that you contribute it back to the open source
community. For core SyncEvolution code I'm trying to keep the copyright
ownership consistent, but for independent modules, LGPL (or compatible)
with other copyrights is fine (see [1]).
[1] http://lists.syncevolution.org/pipermail/syncevolution/2009-June/000000.html
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
5:49 p.m.
New subject: [SyncEvolution] SyncEvolution
Hi!
I kept thinking some more about this problem, additional comments
below...
On Mon, 2009-08-03 at 16:42 +0100, Patrick Ohly wrote:
On Mon, 2009-08-03 at 06:32 -0700, Bob Eggers wrote:
> 4) Our contacts records do not have a last mod time
Then it might be more suitable for implementing the EvolutionSyncSource
base class. In that class you initialize your data store, then report
added, updated, deleted items yourself.
The Evolution backends used to work like that based on a similar "change
list" kept by the Evolution Data Server. I found that a bit limited, for
example, reading that list was destructive (it wasn't possible to just
read the list without erasing it). It might be better with your
implementation.
Other limitations (which may or may not apply to your implementation):
* changes made by the backend on behalf of the server must not be
included in the journal maintained by the data store
* changes have to be tracked per peer, not per data store
(required to sync with multiple servers and for unit testing of
the backend in client-test)
> a) we are able to get a list of records that changed and
another
> list of records deleted since last time we checked
> b) if my assumption that items not appearing in the listAllItems
> will be deleted remotely by the engine, we can ignore this deleted
> list
> c) with the changed records list, we could say last mod time is NOW
> (say, the start of sync), and set a custom field in the record to act
> as last mod time. (Will have to re-save the record locally.)
> d) with unchanged records, do nothing but extract that last mod time
> from the custom field
Instead of faking a last mod field it might be easier to implement the
EvolutionSyncSource class.
... or go to the Synthesis DB Interface directly. It's not possible with
the current SyncEvolution API, but as I said, it would be useful and I
think I know now how to expose it. What's your timeline for implementing
something?
The Synthesis DB Interface contains some support code ("binfile") which
is not used by EvolutionSyncSource. Therefore the functionality which
has to be provided by a DB backend is less than what is expected by
SyncEvolution:
* listing all items, deleted and updated items is enough (added
items are calculated automatically)
* works with data stores which have only a single "item changed
flag": this information can be passed to the Synthesis engine
once, then the flag can be reset. The engine remembers when the
item was changed and can sync the change against multiple peers.
At least that's my understanding, I haven't used the feature...
* no need to serialize items (as mentioned before)
On the other hand, because relevant features are missing (expected APIs,
native serialization of items), the SyncEvolution client-test test
framework and the backup/restore/diff feature can't be used with such a
"native Synthesis backend".
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
1:20 a.m.
New subject: [SyncEvolution] SyncEvolution
Hi Patrick,
We're still in design phase at this point, trying to get all the ducks in a row.
It's likely we'll implement the EvolutionSyncSource class. It's possible there
is a private, backdoor interface to access revision/modtime, which would be ideal. My
modtime hack is the last line of defense; I always like to have one of those. We'll
begin implementation shortly, and we're aiming for as few unknowns/dependencies as
possible in that phase.
Cheers,
Bob
________________________________________
From: Patrick Ohly [patrick.ohly(a)intel.com]
Sent: Tuesday, August 04, 2009 3:49 AM
To: Bob Eggers
Cc: syncevolution(a)syncevolution.org
Subject: RE: SyncEvolution
Hi!
I kept thinking some more about this problem, additional comments
below...
On Mon, 2009-08-03 at 16:42 +0100, Patrick Ohly wrote:
On Mon, 2009-08-03 at 06:32 -0700, Bob Eggers wrote:
> 4) Our contacts records do not have a last mod time
Then it might be more suitable for implementing the EvolutionSyncSource
base class. In that class you initialize your data store, then report
added, updated, deleted items yourself.
The Evolution backends used to work like that based on a similar "change
list" kept by the Evolution Data Server. I found that a bit limited, for
example, reading that list was destructive (it wasn't possible to just
read the list without erasing it). It might be better with your
implementation.
Other limitations (which may or may not apply to your implementation):
* changes made by the backend on behalf of the server must not be
included in the journal maintained by the data store
* changes have to be tracked per peer, not per data store
(required to sync with multiple servers and for unit testing of
the backend in client-test)
I'll find a
> a) we are able to get a list of records that changed
and another
> list of records deleted since last time we checked
> b) if my assumption that items not appearing in the listAllItems
> will be deleted remotely by the engine, we can ignore this deleted
> list
> c) with the changed records list, we could say last mod time is NOW
> (say, the start of sync), and set a custom field in the record to act
> as last mod time. (Will have to re-save the record locally.)
> d) with unchanged records, do nothing but extract that last mod time
> from the custom field
Instead of faking a last mod field it might be easier to implement the
EvolutionSyncSource class.
... or go to the Synthesis DB Interface directly. It's not possible with
the current SyncEvolution API, but as I said, it would be useful and I
think I know now how to expose it. What's your timeline for implementing
something?
The Synthesis DB Interface contains some support code ("binfile") which
is not used by EvolutionSyncSource. Therefore the functionality which
has to be provided by a DB backend is less than what is expected by
SyncEvolution:
* listing all items, deleted and updated items is enough (added
items are calculated automatically)
* works with data stores which have only a single "item changed
flag": this information can be passed to the Synthesis engine
once, then the flag can be reset. The engine remembers when the
item was changed and can sync the change against multiple peers.
At least that's my understanding, I haven't used the feature...
* no need to serialize items (as mentioned before)
On the other hand, because relevant features are missing (expected APIs,
native serialization of items), the SyncEvolution client-test test
framework and the backup/restore/diff feature can't be used with such a
"native Synthesis backend".
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
2:11 a.m.
New subject: [SyncEvolution] SyncEvolution
On Tue, 2009-08-04 at 16:20 +0100, Bob Eggers wrote:
Hi Patrick,
We're still in design phase at this point, trying to get all the ducks
in a row. It's likely we'll implement the EvolutionSyncSource class.
It's possible there is a private, backdoor interface to access
revision/modtime, which would be ideal. My modtime hack is the last
line of defense; I always like to have one of those. We'll begin
implementation shortly, and we're aiming for as few
unknowns/dependencies as possible in that phase.
"unknowns/dependencies" as in "unimplemented features in
SyncEvolution"?
Makes sense. Would it be acceptable for you to use the development
branch? I'd like to get at least some API changes/cleanups implemented
before people seriously write new code against the current API.
This API change is unlikely to break the code at runtime, it's more of
the "let's break all source code and put the pieces back together"
variety. Besides, the master trunk will be tested again against all
servers once 0.9 is out.
Can you explain a bit more about what you are trying to achieve? That
would help to understand what you need and how it could interact with
the rest of SyncEvolution.
For example, how do you intend to provide a GUI? Have you seen the
D-Bus-based design article [1] and the discussion here on the list about
the syncevo-dbus-server<->GUI API? The push listener would fit into such
an architecture well.
[1]
http://moblin.org/documentation/syncevolution/direct-synchronization-aka-...
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
4793
days inactive
4794
days old
syncevolution@synevolution.org
6 comments
2 participants
participants (2)
-
Bob Eggers -
Patrick Ohly