[SyncEvolution] A rather general question about synchronisation (as in syncevolution) compared with what rsync (for example) does
1:22 a.m.
I have been using synchronisation tools for quite a while (started with
the Linux 'pilot' family with an early Palm) and also use rsync and
similar tools.
Is there possibly a case for simply synchronising the data on the
Phone/PDA with the PC/Cloud using something like rsync rather than
trying to translate between different representations of the data as
syncevolution does? I realise that syncevolution tries to put the data
conversion in the interfaces at the 'ends' as it were but I believe many
of the difficulties of getting syncevolution (and similar) to work in
the modern environment are down to the difficulty of getting from "data
format A on machine A" to "data format B on machine B".
If, instead, there was a common data format used by both systems a
simple synchronisation would be all that's needed.
OK, it's probably not going to happen for the mainstream but for someone
developing an application for the Linux (and related) desktop for
example it could be a viable approach because all you'd need to do is to
write something that would allow viewing and updating of data in the
format used by the particular phone/PDA (or family thereof).
In fact this is presumably what the jpilot (and the underlying
libraries) did for the Palm PDAs and it's why they work so reliably and
consistently. While they did their own data transfer they had no need
to transform the data at all, they simply kept it in the same/similar
format on the PC that it was kept on the Palm device.
These are just ramblings really, brought on by the recent more
'philosophical' threads about where syncevolution is going.
--
Chris Green
11:56 p.m.
On Sa, 2011-01-08 at 14:22 +0000, Chris G wrote:
I have been using synchronisation tools for quite a while (started
with
the Linux 'pilot' family with an early Palm) and also use rsync and
similar tools.
Is there possibly a case for simply synchronising the data on the
Phone/PDA with the PC/Cloud using something like rsync rather than
trying to translate between different representations of the data as
syncevolution does?
[...]
If, instead, there was a common data format used by both systems a
simple synchronisation would be all that's needed.
You give the main reason why this isn't very attractive in the real
world for complex data like PIM: there simply isn't one common format
which can be copied around without transformations.
Even if there was, how would such a system handle conflicts? To resolve
conflicts without simply duplicating items and putting the burden on the
user, you will end up parsing the conflicting items and merging them,
which defeats much of the advantage of "moving around blobs of data" as
rsync does.
CouchDB perhaps comes closest to what you want. To my knowledge, no-one
has tried to use it for complex data like calendar, and I'm skeptical
whether that would work - not because I think that CouchDB is bad, but
because I have seen too many attempts at modeling iCalendar 2.0 semantic
fail.
These are just ramblings really, brought on by the recent more
'philosophical' threads about where syncevolution is going.
I agree, this is worthwhile discussing.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12:04 a.m.
On Sun, Jan 09, 2011 at 01:56:53PM +0100, Patrick Ohly wrote:
On Sa, 2011-01-08 at 14:22 +0000, Chris G wrote:
> I have been using synchronisation tools for quite a while (started with
> the Linux 'pilot' family with an early Palm) and also use rsync and
> similar tools.
>
> Is there possibly a case for simply synchronising the data on the
> Phone/PDA with the PC/Cloud using something like rsync rather than
> trying to translate between different representations of the data as
> syncevolution does?
[...]
> If, instead, there was a common data format used by both systems a
> simple synchronisation would be all that's needed.
You give the main reason why this isn't very attractive in the real
world for complex data like PIM: there simply isn't one common format
which can be copied around without transformations.
I was suggesting that one might write a dedicated application for the
desktop end that would simply handle the format of a spcific phone/PDA.
Even if there was, how would such a system handle conflicts? To
resolve
conflicts without simply duplicating items and putting the burden on the
user, you will end up parsing the conflicting items and merging them,
which defeats much of the advantage of "moving around blobs of data" as
rsync does.
Rsync (or at least wrappers which use it) can decide which of two files
to keep by looking to see which is the newer. I guess you hit a problem
when one file contains many separate items of data though.
CouchDB perhaps comes closest to what you want. To my knowledge,
no-one
has tried to use it for complex data like calendar, and I'm skeptical
whether that would work - not because I think that CouchDB is bad, but
because I have seen too many attempts at modeling iCalendar 2.0 semantic
fail.
> These are just ramblings really, brought on by the recent more
> 'philosophical' threads about where syncevolution is going.
I agree, this is worthwhile discussing.
:-)
--
Chris Green
1:58 a.m.
On Jan 9, 2011, at 14:04 , Chris G wrote:
> Even if there was, how would such a system handle conflicts? To
resolve
> conflicts without simply duplicating items and putting the burden on the
> user, you will end up parsing the conflicting items and merging them,
> which defeats much of the advantage of "moving around blobs of data" as
> rsync does.
>
Rsync (or at least wrappers which use it) can decide which of two files
to keep by looking to see which is the newer.
If it was that easy :-)
Sync peers all have slightly differening ideas of the concept "calendar item" or
"contact", so the real challenge is not just to find which side has most recent
data, but especially which data is available at one side at all. If you just choose
"most recent", you'll eventually loose all data but what the dumbest (most
limited) sync participant can store.
Just imagine a mobile which stores name and tel numbers, but no postal address. If the
most recent update (say, to a telephone number) comes from such a device, there's a
whole lot of field by field merging magic needed to avoid loosing the postal address a
more capable device might have.
Lukas
4:17 a.m.
On Sun, Jan 09, 2011 at 03:58:24PM +0100, Lukas Zeller wrote:
On Jan 9, 2011, at 14:04 , Chris G wrote:
>> Even if there was, how would such a system handle conflicts? To resolve
>> conflicts without simply duplicating items and putting the burden on the
>> user, you will end up parsing the conflicting items and merging them,
>> which defeats much of the advantage of "moving around blobs of data"
as
>> rsync does.
>>
> Rsync (or at least wrappers which use it) can decide which of two files
> to keep by looking to see which is the newer.
If it was that easy :-)
Sync peers all have slightly differening ideas of the concept "calendar item"
or "contact", so the real challenge is not just to find which side has most
recent data, but especially which data is available at one side at all. If you just choose
"most recent", you'll eventually loose all data but what the dumbest (most
limited) sync participant can store.
Just imagine a mobile which stores name and tel numbers, but no postal address. If the
most recent update (say, to a telephone number) comes from such a device, there's a
whole lot of field by field merging magic needed to avoid loosing the postal address a
more capable device might have.
I'm designing (in my mind) the way the data is stored as well as the
synchronisation system.
If the data is stored as one file per record - e.g. each address/contacts
entry is a separate file then I see no major issue. *Both* ends store
all the data, name, address, phone number, E-Mail and whatever. Then
an rsync type approach of always keeping the latest one will work pretty
well - the only case it won't handle is if both ends are changed between
synchronisation runs.
By the way, no, I can't imagine a mobile which stores only names and
telephone numbers! :-) Part of the point of synchronising two devices
(for me anyway) is to have the same data on both, by which I mean *all*
of the same data for a given person.
If I add some data to my desktop contacts list and then synchronise with
my PDA/phone I would be most upset if the synchronisation software
silently ignored that change because the phone/PDA doesn't have a
matching field to put it in. It's exactly this issue which prompted my
initial comments about the difficulty of handling the differences
between what different devices store. I want to force them to be the
same, otherwise it's almost inevitably broken.
--
Chris Green
5:30 a.m.
On Jan 9, 2011, at 18:17 , Chris G wrote:
> Sync peers all have slightly differening ideas of the concept
"calendar item" or "contact", so the real challenge is not just to
find which side has most recent data, but especially which data is available at one side
at all. If you just choose "most recent", you'll eventually loose all data
but what the dumbest (most limited) sync participant can store.
>
> Just imagine a mobile which stores name and tel numbers, but no postal address. If
the most recent update (say, to a telephone number) comes from such a device, there's
a whole lot of field by field merging magic needed to avoid loosing the postal address a
more capable device might have.
I'm designing (in my mind) the way the data is stored as well as the
synchronisation system.
If the data is stored as one file per record - e.g. each address/contacts
entry is a separate file then I see no major issue. *Both* ends store
all the data, name, address, phone number, E-Mail and whatever. Then
an rsync type approach of always keeping the latest one will work pretty
well - the only case it won't handle is if both ends are changed between
synchronisation runs.
I see. However this is more a file based approach to database replication than to PDA
sync.
You postulate that all peers need to understand one common format for each data type. This
is something that I've never seen happen in reality. Not even plain text is always
exchangeable without conversions (line ends, character set), yet alone more complex
structured data.
So while it is certainly desirable to reduce the conversion needs as much as possible by
using data exchange formats that are as universally usable as possible, a sync
architecture IMHO always needs to deal with conversion.
By the way, no, I can't imagine a mobile which stores only names
and
telephone numbers! :-)
You're probably too young, then :-) These things have existed and I have some lying
around here.
Seriously, you're probably right regarding what we today consider PDA data, i.e.
contacts and calendar entries. We can more or less assume that modern back-ends for PDA
data are able to store everything for contacts and calendars (oops, Outlook IMHO still has
a fixed number of fields for telephone numbers...).
Part of the point of synchronising two devices
(for me anyway) is to have the same data on both, by which I mean *all*
of the same data for a given person.
And that *all* will continue to expand. Today it's tel+email+postal+im. But what's
with the most recent blog entry, geo coordinate, twitter status, hires photo, github
commit? And the relation between all this? Even simple meeting invitations is something
today's backend often have a hard time to get right.
All of this will eventually become (if it hasn't already) part of what we today call a
"contact". But capabilities will be added to some systems sooner than to others.
With the result that we'll have quite similar situations as my (admittedly a bit
outdated) device-with-no-postal-address scenario, just on the next level.
I see no chance for a sync solutions that relies on *the* format that can represent
everything.
Lukas
Lukas Zeller, plan44.ch
luz(a)plan44.ch - www.plan44.ch
8:50 p.m.
On Sun, Jan 09, 2011 at 07:30:11PM +0100, Lukas Zeller wrote:
On Jan 9, 2011, at 18:17 , Chris G wrote:
>> Sync peers all have slightly differening ideas of the concept "calendar
item" or "contact", so the real challenge is not just to find which side
has most recent data, but especially which data is available at one side at all. If you
just choose "most recent", you'll eventually loose all data but what the
dumbest (most limited) sync participant can store.
>>
>> Just imagine a mobile which stores name and tel numbers, but no postal address.
If the most recent update (say, to a telephone number) comes from such a device,
there's a whole lot of field by field merging magic needed to avoid loosing the postal
address a more capable device might have.
>
> I'm designing (in my mind) the way the data is stored as well as the
> synchronisation system.
>
> If the data is stored as one file per record - e.g. each address/contacts
> entry is a separate file then I see no major issue. *Both* ends store
> all the data, name, address, phone number, E-Mail and whatever. Then
> an rsync type approach of always keeping the latest one will work pretty
> well - the only case it won't handle is if both ends are changed between
> synchronisation runs.
I see. However this is more a file based approach to database replication than to PDA
sync.
You postulate that all peers need to understand one common format for each data type.
This is something that I've never seen happen in reality. Not even plain text is
always exchangeable without conversions (line ends, character set), yet alone more complex
structured data.
So while it is certainly desirable to reduce the conversion needs as much as possible by
using data exchange formats that are as universally usable as possible, a sync
architecture IMHO always needs to deal with conversion.
It's trivial surely (I just thought about this a little) you have a
'dictionary' type format for the data, each file/record simply contains
fields which consist of a type (i.e. a field name) and the data for that
type. Thus every device that uses the data can extract any or all of
the data as it needs. If some device decides to add a new field then it
does so, that new data gets passed around with all the rest and if other
devices need it or display it they can but if they don't need it they
just ignore it.
The only downside is wasted space on devices that don't use all the data
but in this day and age of cheap memory even 'phones have a silly amount
of memory so I don't see this as an issue at all.
> By the way, no, I can't imagine a mobile which stores only names and
> telephone numbers! :-)
You're probably too young, then :-) These things have existed and I have some lying
around here.
I'm 64, been programming since the 1970s! :-)
Seriously, you're probably right regarding what we today consider
PDA data, i.e. contacts and calendar entries. We can more or less assume that modern
back-ends for PDA data are able to store everything for contacts and calendars (oops,
Outlook IMHO still has a fixed number of fields for telephone numbers...).
> Part of the point of synchronising two devices
> (for me anyway) is to have the same data on both, by which I mean *all*
> of the same data for a given person.
And that *all* will continue to expand. Today it's tel+email+postal+im. But
what's with the most recent blog entry, geo coordinate, twitter status, hires photo,
github commit? And the relation between all this? Even simple meeting invitations is
something today's backend often have a hard time to get right.
Yes, absolutely agree and if you make the format inherently flexible and
able to add any of these fields without breaking existing devices that's
what we need isn't it? The continuous chase to keep up with changing
data is almost bound to fail.
All of this will eventually become (if it hasn't already) part of
what we today call a "contact". But capabilities will be added to some systems
sooner than to others. With the result that we'll have quite similar situations as my
(admittedly a bit outdated) device-with-no-postal-address scenario, just on the next
level.
I see no chance for a sync solutions that relies on *the* format that can represent
everything.
The more I think about it the more I believe that a format which doesn't
define the fields but does define how to represent fields of any type
seems to me to be the right way to go.
I *fear* that XML is the obvious candidate, I just don't like it
personally. :-)
--
Chris Green
4268
days inactive
4271
days old
syncevolution@synevolution.org
6 comments
3 participants
participants (3)
-
Chris G -
Lukas Zeller -
Patrick Ohly