[SyncEvolution] configuration + multiple peers
by Patrick Ohly
Hello!
Let's think about our configuration handling and how we are going to
deal with multiple peers.
Right now, we have one set of configuration files per server. Each
server synchronizes against its own set of local data sources.
In the future, we are not necessarily syncing against an HTTP server.
"Peer" is a better word, because it covers clients and servers.
Multiple peers want to synchronize against the same local data. That's
where the current configuration system becomes awkward. It forces us to
duplicate certain configuration settings (local data sources, logging).
When looking for local database dumps for restore, it would be necessary
to check multiple configurations.
Therefore I suggest that we change our configuration layout. Instead of
a "server configuration" directory, we start with a "host
configuration" (better names welcome!). A host configuration has global
settings pertaining to all syncs done with that configuration (logging).
It has a set of local source configs (backend and database selection).
Then it has a set of peers using this particular host configuration.
Each peer has its own peer-specific settings (credentials, URL and
transport) and source-specific settings (URI, sync mode). It might even
have more than one way of contacting it (Bluetooth, USB), although how
to handle this is a bit uncertain right now. My intention is to deal
with this inside the per-server config file instead of creating more
config files.
Because there hardly ever is a need to have more than one host
configuration, we could create one named "default" and use that unless
specified otherwise.
The layout would look like this:
.config/syncevolution/
default/
config.ini host config
.synthesis shared (!) Synthesis binfiles
sources/
addressbook|calendar|.../
config.ini local source config
peers/
scheduleworld/
config.ini peer configuration
sources/
addressbook/
config.ini peer-specific source config
.other.ini per peer and per source storage for backend
For each config.ini there's also an .internal.ini which is not exposed
via command line or D-Bus API. The exact location of each property needs
to be defined.
The advantage of this layout is that when a SyncML server contacts us
via OBEX, we can select the "default" host config, check whether the
server is known already, otherwise create a new peer config using the
"default" set of local data sources. In the current scheme it is unclear
where the host configuration is supposed to come from.
In the new scheme the Synthesis binfiles are shared between peers, as
intended by Synthesis. This is not necessary for us at the moment, but
some of the binfile features (simplified change tracking via a single
"updated" flag in the local database) might be useful for other
backends.
The drawback of this revised config layout is the increased complexity.
Instead of two sets of properties as in the current config (sync
properties and source properties) users of the command line and the
D-Bus API would have to deal with four sets, even in the simple use case
of a single peer.
Here's an idea how this could be avoided: when selecting a configuration
for a server, create a "view" on the more complex configuration layout
which looks exactly like it did so far. The "sync properties" include
the ones that are identical for all peers and the peer specific ones.
Likewise for each source. Setting the properties puts the properties
into the right config.ini automatically.
The D-Bus API as discussed so far and the command line interface don't
have to be changed. Configs are selected via the peer name, which is
typically unique. If not, then appending a @<host config name> makes it
unique and can be used to create new host configs.
The empty string as peer name selects only the host configuration, which
is useful for local operations (restore, status) or accessing all sync
reports.
In that scheme it may be a bit unexpected that, for example, updating
the logdir setting of the "scheduleworld" server affects the config of
the "funambol" server. When using the SetConfig(update=false) D-Bus API
to remove a config, clients must not remove the host's source
properties, because they might still be needed by other peers. I think
this is acceptable, considering the advantages of the otherwise simpler
API.
When asking for a peer template, that template should already be
populated with existing host properties, so that it can be written back
without unintentionally overwriting those.
Speaking of templates, the list of available templates might also
contain paired Bluetooth devices which are known to support SyncML. Then
(almost) the same UI as for adding an HTTP server could also be used for
devices.
Another drawback of changing the configuration layout is the question of
migrating old configs and changing back and forth between new and old
SyncEvolution releases. Ideally migration should be automatic and still
allow going back to an older SyncEvolution seamlessly.
With two tricks this might be feasible:
1. move existing properties into the new layout and *duplicate* the
host properties inside the per-server config.ini
2. recreate the traditional layout with symlinks
So we would end up with:
.config/syncevolution/scheduleworld/
.synthesis/ -> ~/.config/default/.synthesis
config.ini -> .../peers/scheduleworld/config.ini
sources/ -> .../peers/scheduleworld/sources/
config.ini files are updated by creating a new file and then renaming it
to config.ini. This replaces the config.ini symlink. Also, if a host's
source property is changed using an old SyncEvolution, the change won't
be noticed by a new SyncEvolution because although the changed
config.ini is read, the newer SyncEvolution doesn't take the host
properties from there. I think we don't have to support this.
The automatic migration also doesn't work if the user has multiple
server configs because we cannot "join" different .synthesis binfile
dirs. In such a situation the user has to force a migration and be
prepared for slow syncs. Without such a forced migration the new
SyncEvolution should continue using the old layout. This is easily
implemented by using the same config node twice.
Finally, the instructions for synchronizing multiple ScheduleWorld
calendars have to be updated. When setting up the second scheduleworld
config, the name of the "calendar" source is already in use and
something else must be used to avoid overwriting the "evolutionsource"
property in the existing source.
How does this sound so far?
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 3 months
[SyncEvolution] gdbus-cxx and methods in base classes
by Patrick Ohly
Hello!
Yongsheng found a problem in the C++ D-Bus binding, triggered by this
code:
void Session::activate()
{
static GDBusMethodTable methods[] = {
...
makeMethodEntry<ReadOperations,
bool,
ReadOperations::Config_t &,
typeof(&ReadOperations::getConfig), &ReadOperations::getConfig>
("GetConfig"),
ReadOperations is a private base class of Session, which was meant to
inherit the implementation without writing any glue code.
The problem is that the "void *data" pointer is a "Session *" for all
interface methods, but the methodFunction template for getConfig casts
from void * to ReadOperations, which is incorrect.
I have been unhappy with the need to repeat method parameter types as
template parameters of makeMethodEntry. It would be much more natural to
write:
makeMethodEntry(this,
&ReadOperations::getConfig>
"GetConfig")
A traits class which hinges on the type of the second parameter then
could provide the result of the call. The "this" pointer is passed so
that its type is known and the cast can be generated correctly.
I gave this a try and think it should work - except that g++ 4.3 and 4.4
crash when the type case is enabled in methodFunction2. Without that
line, the code compiles. Darn. If anyone has an idea how to make this
work with existing compilers, feel free to help me out.
Yongsheng, in the meantime please work around the problem by writing
some glue code (Session::getConfig, Session::getReports, ...). I don't
want to make the makeMethodEntry<> calls any more complex while there
still is a chance to make them simpler.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 3 months
Re: [SyncEvolution] D-Bus API: availability of local sources (was: syncevo-dbus-server implementation)
by Patrick Ohly
On Fri, 2009-09-18 at 02:19 +0100, Zhu, Yongsheng wrote:
> > What we need is a D-Bus call which actively checks whether the source is available. For a not-yet written configuration this could work like
> > this:
> > * create a Session
> > * get template
> > * set template *temporarily*
> > * ListSources(source_name) and/or CheckSource(source_name)
>
> > ListSources() should return a list of valid values for the
> > evolutionSource property, much like "syncevolution" without
> > parameters does.
> What is the source name?
Something like "addressbook". "ListDatabases()" is perhaps a better
name: it lists all local databases which can be reached through the
source and its associated backend.
> > Then GUI can pick one. CheckSource() should do the
> > syncSource->open() test to verify that the source really works.
> CheckSource(source_name) should be used to check the specified 'evolutionsource' named 'source_name'
No. In this case, "CheckSource()" really means check the source settings
currently in place (permanently or temporarily) for a source like
"addressbook".
> So we define 2 new APIs for this issue instead of doing them in getConfig/setConfig?
Yes.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 3 months
[SyncEvolution] schedule SyncEvolution 0.9.1 + 1.0
by Patrick Ohly
Hello everyone!
Here's some background information about when we might release updates
for SyncEvolution. This isn't set in stone; think of it more like the
grand master plan that we strive for... and then change after impact
with the real world ;-)
Scheduling is driven by "release when ready" and "we want regular
updates at a deterministic point in time", so let's look at the feature
list first:
The high-level goals for 0.9.1 are:
* bug fixing, including string changes that we had to postpone in
0.9
* include message resend (merged, found problems in testing)
* revised backend API (merged, no regressions so far)
* mobical.net interoperability
* Memotoo interoperability (because of user demand)?
* stretch goal: revised D-Bus API
Message resending was merged, but then a thorough testing against
Funambol and ScheduleWorld showed that there is one particular case
where the sync hangs when a certain message reply gets lost. This looks
like a server bug. With the Synthesis server it works.
The high-level goals for 1.0 are:
* implement revised D-Bus API and architecture (support Genesis
GUI; syncevolution as D-Bus client)
* performance and reliability improvements (EDS API)
* minimal support in sync-ui for SyncML server mode
* stretch goal: implement the intended SyncML client design in
sync-ui (current one is lacking recovery features, progress spinner,
etc.)
* stretch goal: OBEX support for SyncML client and server
* SyncML server mode (peers to be decided)
In particular the last item has a huge tail of detailed testing work
associated with each peer. For 1.0, our goal has to be to get the
infrastructure in place and prove that it works with a few phones. We
probably could work this forever. At the same time we want to get
another stable release out the door so that we can ship the other
improvements.
The biggest unknown factor for 1.0 are the necessary changes in
libsynthesis. Lukas has reassured me that he has made good progress
towards those, but they have a business to run and therefore might have
more important work to deal with first. September 4th in the schedule
below is way too early - more realistic proposals welcome ;-}
With that in mind, here are some milestones that could get us towards 1.0,
with names attached to the major tasks and until when they need to be done
to stay on track:
* Jussi, Patrick, August 26: decide about D-Bus server-side
technology - done, let's use libdbus + gdbus + custom C++ binding
* Patrick, August 28: core server engine and complete C++ binding -
much of that is done.
* Jussi, August 28: D-Bus GUI API redefined, with feedback by others -
still a bit of work needed
* Patrick, Yongsheng, September 11: new D-Bus API implemented (must
work incrementally, so that Jussi can adapt the GUI in parallel)
* Synthesis, September 4 (?): SyncML server API discussed and usable
* Jussi + Patrick + Nick, September 9+10: face-to-face in London,
discuss design issues
* Congwu, September 4: EDS API enhancements + usage of those in
SyncEvolution backends - patches ready for review, Ross will do
that for EDS, Patrick for SyncEvolution
* Congwu, September 11: outgoing OBEX connections with transport
inside SyncEvolution
* Patrick, September 18: Synthesis server feature integrated into
SyncEvolution
* all, end of September: first alpha
* all, end of October: final release
We have to be aware that we have to leave enough gaps in the schedule
for unexpected problems or requests and the "optional" open issues (see
attachment). I consider some of these important that I don't want to
push them out forever:
* code cleanup
* revive SQLite demo backend and perhaps Mac OS X and Maemo
support (although I really hope that the later two will be
covered by external developers)
* split data transformation part out of libsynthesis
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 3 months
[SyncEvolution] notes from design meetings last week
by Jussi Kukkonen
[I have a bunch of email just sitting in my outbox, sorry I'm a it late
with this -- meant to send it on Monday]
Hi guys,
Here are my notes from the design meetings we had in London, Sep
9th-11th. More or less present were Nick (designer), Patrick and me.
These aren't complete minutes, so Patrick will probably want to add
something.
- Jussi
Summary for things related to the user interface:
Short term:
* Go through some minor bugs that are still open
* add "Recovery" mode
* don't show sync type in the main window,
put it in service settings
* Change the setting screen so there is no "service dialog":
service settings will open in an expander in the settings
window
Mid to long term
* fix the main window to better support automated syncs, emphasizing
more sync "history".
* notifications
* time based synchronization
* EDS event based synchronization
* Any settings that involve "the cloud" will be integrated in Moblin.
This includes Sync settings. This may lead to some problems with the
generic GTK+ UI...
* we'll need to support device to device sync in the ui, which means
we want to have multiple servers/peers "active" at the same time.
This will impact the presentation of the syncs, sync history and
settings.
In addition to UI things we discussed the D-Bus api. We talked about how
the server should request info from the user and how we can accomodate
the multi-peer needs within the API we have defined. Patrick will
hopefully talk about this in more detail.
Some more specific notes:
*** Recovery:
There are two situation where we want to give the user the opportunity
to do potentially destructive SyncML things:
1. user notices that data is corrupt (duplicate items etc).
There will be a button in main window for this case.
2. SyncML server has decided it needs to do a slow sync
on next sync and SyncEvolution is not sure what to do.
We should open the Recovery view when the user opens sync-ui.
The recovery options will be
* Slow sync
* Delete all local data and refresh from server
* Delete all data on server and refresh from local
* Restore a backup
Nick has an action item here:
* define a UI for the "restore backup" case with these req's:
- we have data on at most ten last syncs
(number of changed items, total number of items, etc)
- we can restore before or after each sync
(it might still make sense to only offer restore before a sync,
just to keep things simpler)
- preferably simple to implement...
We need write the explanations for the recovery options as well as a
overall description for both "error cases" mentioned above. Nick will
probably need some assistance from the SyncML-speaking people on that...
*** Unattended syncs
I believe we all agreed that some sort of automatic synchronization is a
"must have" feature -- we cannot expect users to open a synchronization
program. In the long term synchronizing based on EDS events (new
contact, changed event, etc) would be great, but for a start periodic
syncs will do.
However, the UI side will need some changes before we can do that: The
main window needs to be more about syncs and possible problems in recent
history.
We touched the issue of what/who actually drives the unattended syncs,
but this wasn't really solved.
My thoughts on this: the unattended-sync-client needs IMO to be able to
sync on these cases:
1. X minutes has passed since the last sync
2. user logs in or resumes from suspend and more than X minutes have
passed since last sync (this is a specia lcase of #1).
3. do a sync some time after local data has changed
It also needs to use notifications to inform the user about syncs.
Implementing #2 (approximately) might be very easy by having a small
process running all the time that subscribes to network events and
initiates syncs a minute or two after a new connection is made... The
footprint would be small so I don't think this would be a problem.
*** Editing configurations "in-place"
There really should not be a "Edit service" -dialog, there should just
be a "Service list" where the services have an expander that includes
the settings.
The "manual services" list will be removed and the "Add a service
manually" button will just add a new service into the main list
12 years, 3 months
[SyncEvolution] SyncEvolution using Synthesis server engine: works
by Patrick Ohly
Hello!
I'm happy to announce that I just ran a two-way sync between
SyncEvolution (client) and SyncEvolution (server), using the new
Synthesis engine on both sides :-)
I still need to commit the code (SyncEvolution "dbus-api" branch,
libsynthesis @moblin.org "server"). Once I've done that, the following
commands will configure a client and server config using localhost and
the same test databases as client-test, then run a sync:
./syncevolution --configure \
--sync-property syncURL=http://localhost:9000/syncevolution \
--sync-property username=test \
--sync-property password=test \
--template scheduleworld \
syncevolution_client
./syncevolution --configure \
--source-property evolutionSource=SyncEvolution_Test_vcard30_1 \
syncevolution_client addressbook
./syncevolution --configure \
--source-property evolutionSource=SyncEvolution_Test_ical20_1 \
syncevolution_client calendar
./syncevolution --configure \
--source-property evolutionSource=SyncEvolution_Test_text_1 \
syncevolution_client memo
./syncevolution --configure \
--source-property evolutionSource=SyncEvolution_Test_itodo20_1 \
syncevolution_client todo
for i in calendar addressbook todo memo; do \
./syncevolution --configure --source-property uri=$i syncevolution_client $i; \
done
./syncevolution --configure \
--sync-property username=test \
--sync-property password=test \
--template scheduleworld \
syncevolution_server
./syncevolution --configure \
--source-property evolutionSource=SyncEvolution_Test_vcard30_2 \
syncevolution_server addressbook
./syncevolution --configure \
--source-property evolutionSource=SyncEvolution_Test_ical20_2 \
syncevolution_server calendar
./syncevolution --configure \
--source-property evolutionSource=SyncEvolution_Test_text_2 \
syncevolution_server memo
./syncevolution --configure \
--source-property evolutionSource=SyncEvolution_Test_itodo20_2 \
syncevolution_server todo
test/syncevo-http-server.py syncevolution_server 9000 &
LD_LIBRARY_PATH=build-synthesis/src/.libs syncevo-dbus-server &
LD_LIBRARY_PATH=build-synthesis/src/.libs syncevolution syncevolution_client
syncevo-http-server.py is very limited. It causes a session to hang
until the client starts the next one. Needs to be rewritten with a
single event loop (twisted looks promising).
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 3 months
[SyncEvolution] Question about connection with syncevolution
by Zhao, Forrest
Hi Patrick,
I have 2 questions about connection with syncevolution to discuss with you. All the questions are in the context of OBEX server/SyncML client binding.
1 If concurrent SyncML requests come from multiple SyncML servers, should obexd stub initiate multiple connections to syncevolution for the requests coming from seperate SyncML server? Or obexd should initiate only one connection with syncevolution and reuse it for the request from all SyncML servers?
2 When should obexd stub destroy the connection with syncevolution? Should obexd stub destroy the connection with syncevolution when the "final" is TRUE in the "Reply" signal of interface "org/syncevolution.Connection"?
Thanks,
Forrest
12 years, 3 months
[SyncEvolution] syncevo-dbus-server implementation: config and report access
by Patrick Ohly
Hello!
I've added some of the new D-Bus API calls to the C++ based
syncevo-dbus-server. See the "dbus-api" branch. The implementation of
Server/Session.GetConfig/GetReports() and Session.SetConfig() is
missing. Yongsheng, if you want to, feel free to have a stab at
implementing the body of these functions.
I'll continue to work on the sync-related methods. Because this is in a
different part of the file, we should be able to merge changes easily.
Storing the temporary configuration is unclear at this point. We can
leave out that part of the API for the time being.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 4 months
[SyncEvolution] dbus api specification
by Jussi Kukkonen
Hi all,
I'm mailing the dbus xml files now so Patrick and Yongsheng can continue...
Changes since last version:
* Changed some arrays to dictionaries. Original reasoning was that
every item in an array that contains sources will always be used so
dictionary brings no improvement. Practical tests show otherwise,
dictionaries are still nicer.
In practice it means source array "a(sssu)" turns into "a{s(ssu)}"
with source name being the dictionary key.
* config dictionaries use "" for the main config and "sources/<source>"
for source configs.
* SetConfig() now has "boolean temporary" parameter
* Server.GetSessions and Server.SessionChanged now have equivalent
content (basically only the object path).
Open issues:
I think the api is in pretty good shape from my point of view, I've
rebuilt most of the client wrapper object with this for testing
purposes. The last item on the list above is still nagging me a bit,
feel free to take a look... I _think_ we don't need the information on
which session(s) are not "queueing" (so, which one is actually syncing
at the moment) in the Server object but I'm not absolutely sure. The
info is available by hooking into Session object anyway.
Handling passwords... Patrick, maybe we can go through this on thursday
so we don't miss any corner cases?
Docs:
The xsl is the counterpart for spec-strip-docs.xsl in the repo, it
creates docbook documentation from the xml files:
%-doc.xml: %-full.xml
$(XSLT) -o $@ $(srcdir)/spec-to-docbook.xsl $<
The docbook can be used by gtkdoc or you can just call docbook2*
converters (which are found in docbook-utils in Debian).
- Jussi
12 years, 4 months
Re: [SyncEvolution] sync - Attributes mapping with servers
by Patrick Ohly
Hello!
Sorry for taking so long, but I finally went through this list to
identify items that we need to work on. I wasn't in a hurry, because I
expected most to be material for 1.0.
On Wed, 2009-09-02 at 02:31 +0100, Zheng, Yanshuang wrote:
[...]
> [Scheduleworld]
[...]
> Others:
>
> 1) “Assistant” on scheduleworld is saved/sent as both
> X-EVOLUTION-ASSISTANT and X-WSS-ASSISTANTNAME
It's sent like that, but I'm fairly sure that in Evolution we only store
the name once, because X-WSS-ASSISTANTNAME is ignored by the Synthesis
engine.
> 2) TZ keeps value “+00:00”, no matter what we set in “Time Zone”
What do you mean here? Where does Evolution display "+00:00"? The
expected outcome is that if you select "Europe/Berlin" for an event or
task, Evolution also shows that as timezone.
> [Memotoo]
[...]
> Missed:
>
> Addressbook– Skype ID, Assistant Phone, Parents, Children, User
> Defined(1~4)
These are properties that we should add to our configuration, even
though Evolution cannot store them. We want them in our config for
different local backends (file) or servers (ODBC backend).
=> http://bugzilla.moblin.org/show_bug.cgi?id=6303 "extend vCard field
list (Memotoo)"
Looking at the Memotoo vCard example, I saw another field which is not
currently stored like that, even though Evolution has a FBURL:
X-SYNCEVOLUTION-FBURL:http://www.memotoo.com/contactsIFB.php?uid=73fc9832...
I've also added that to #6303. In contrast to the other properties, this
is one that we can support locally.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
12 years, 4 months
