http://bugzilla.meego.com/show_bug.cgi?id=685 Summary: join/dejoin different sources Classification: MeeGo Projects Product: SyncEvolution Version: unspecified Platform: Netbook OS/Version: IA Status: ASSIGNED Severity: enhancement Priority: Medium Component: SyncEvolution AssignedTo: syncevolution-bugs(a)meego.bugs ReportedBy: jingke.zhang(a)intel.com QAContact: jingke.zhang(a)intel.com CC: syncevolution-bugs(a)meego.bugs, syncevolution-default-bugs(a)meego.bugs Estimated Hours: 0.0 This is from http://bugzilla.moblin.org/show_bug.cgi?id=4611 Description From pohly 2009-07-21 07:59:27 PST (-) [reply] It is a very common complaint that SyncML in most installations only supports one calendar, address book, task list, memo list per device. This is not a limitation of the protocol, more like the lowest common denominator that is typically supported. Our own sync UI for Moblin follows the same model, although SyncEvolution itself could handle more than one instance of each data category if the peer did (evolutionSource config option). For those cases where the peer doesn't, it would be nice if SyncEvolution could join data from multiple databases if the peer supports only one database, then when receiving updates back correctly sorted them into the the right database. Here's a user giving more background information: http://article.gmane.org/gmane.comp.mobile.funambol.user/11522 ScheduleWorld has such a feature for calendars, based on the CATEGORY property. We could adapt a similar system and perhaps even make it so that n:n calendar synchronization within the same sync session becomes possible. Another use case for such a join/dejoin are devices which exchange events and tasks via one single URI. For our direct-sync work we need to support that. The Synthesis engine has the concept of a <superstore>: it automatically merges and splits based on some configurable property. Currently this is used to put VEVENT into one <datastore>, VTODO into another. Perhaps this can be configured to use CATEGORY, too. For joining items, make sure that a unique CATEGORY is set per database. For receiving items or updates back, check for the presence of those categories and store accordingly, with one database as default for new items which don't have any of these categories. ------- Comment #1 From pohly 2009-10-29 00:47:48 PST (-) [reply] ------- Besides joining/dejoining different data sources of the same content type based on some item property like CATEGORY, a much more urgent (and simpler) use case is accessing events and tasks under one URI. Apparently ovi.com (bug #3182) expects its clients to support this and thus only offers one URI for both kinds of items. It's debatable whether the server shouldn't better support separate URIs (because there are clients which only support that). Our own server mode definitely has to support a joined event/task URI. I'm not creating two issues for this because the implementation of the two use cases clearly is related. For tracking purposes: joined events/tasks are P2 for 1.0, the rest can wait. ------- Comment #2 From Chen Congwu 2009-11-19 21:01:21 PST (-) [reply] ------- (In reply to comment #1) And to sync with Nokia phones, we also need this feature. FileSyncSource can handle both sync types but EvolutionSyncSource not. We need to configure and join Evolution events and todos. My suggestion on the configuration file change: Add a "super calendar" = "super evolution calendar" special backend, which will map to (calendar+todo), the format used is vcard2.1 and ical2.0 by default, but user can change it: super calendar:vcard3.0+ical2.0 ------- Comment #3 From pohly 2009-11-20 00:15:22 PST (-) [reply] ------- (In reply to comment #2) The solution of this problem should be independent of the Evolution backends. The general task is: "combine two (or more) real data sources into one virtual one". I'm not sure whether we really need to write a backend. At least for the "tasks + events" use case, the Synthesis engine already has a solution. We just need to activate it somehow. Why vcard2.1? ------- Comment #4 From Chen Congwu 2009-11-20 00:30:47 PST (-) [reply] ------- (In reply to comment #3) Now comes this solution: The configuration of type properties is defined as: backend1+backend2+...backendN:format1+format2_...formatM (0=<M<=N) The information we lost is: the datastore name is the same as backend name in such joined case which I think is OK. Another dirty point is the "forced" property in format, the original single m_forced property is no longer working. We will maintain a list of all joined datastores, which typically like: superdatastore1: backend1, backend2,..backendN This is used later to generate the "superdatastore" configuration. Now comes the specific part, the generation of "superdatastore" element is different case by case (The filtering criteria). At this time I will always generate a "super calendar" datastore for the calender +event case to work. Because this is the situation in non-joined case, I just don't want to change. ------- Comment #5 From pohly 2009-11-20 01:42:16 PST (-) [reply] ------- (In reply to comment #4) I assume that with "backend" you mean something like "Evolution Contacts". When that backend gets invoked, how does it know which local database it is supposed to access? I suggest to solve this problem by referencing other *source* configurations instead of *backends*. Then these other source configurations are defined normally. A sync where a supersource is active is treated like a sync where each of the involved normal sources is active (database dumps, reporting, ...). The configuration could look like this: [calendar+todo] type = virtual:text/calendar evolutionsource = calendar,todo It's a bit unfortunate that I followed Evolution's example with the "calendar" name. Other products use "events", "tasks" for what we call "calendar" and "todo" and "calendar" for what we have to call "calendar+todo" (just a proposal, other ideas are welcome). Note that only data sources with compatible data formats can be merged, as far as I understand it. Even if the Synthesis engine supports it, I don't think we have use cases which require it, so we can impose this limitation ourselves and thus keep the data format selection via [calendar+todo] working as usual. I think these problems don't occur with the approach above. Agreed. Later we can add additional configuration parameters which control the advanced use cases. ------- Comment #6 From Chen Congwu 2009-11-20 02:08:16 PST (-) [reply] ------- (In reply to comment #5) That's the part I missed. Do you mean add another dummy source folder reference other sources to be joined ? This simplifies my problem but adding duplicated configuration(in this case calendar, todo, and calendar+todo are mostly duplicated). This is not very nice and may confuse user. For example is it meaningful to fire a sync with a Nokia phone asking two-way sync for calendar and slow-sync for todo? Mostly this does not work. I agree with you and actually I already changed my mind during testing. ------- Comment #7 From pohly 2009-11-20 02:23:18 PST (-) [reply] ------- (In reply to comment #6) Yes. [calendar+todo] is indeed mutually exclusive with both [calendar] and [todo]. This is something that users need to know. Depending on how we implement this "virtual" backend, this semantic can be checked by SyncContext. In the sync-UI, we can hide the complexity for the default case. ------- Comment #8 From Chen Congwu 2009-11-22 23:21:22 PST (-) [reply] ------- (In reply to comment #7) Ok, I have pushed the fix to phone branch including several SAN generation fixes to work with Nokia phone. I ignore other properties of the virtual syncsource except "type" and "evolution syncsource" to avoid inconsistency. The format in the "type" field for a virtual syncsource must be presented, because this may otherwise leads to inconsistency for different backends providing different preferred formats. ------- Comment #9 From pohly 2009-11-23 03:15:31 PST (-) [reply] ------- Commit message: At this time, the configuration is fixed to work with "calendar+todo" case. This means "only virtual sync sources which combine one calendar and one todo are supported, in this order. In other words, a valid configuration is evolutionsource=calendar,todo" Is that correct? Why was it necessary to change the ISEVENT check in the XML config? A bug in the original example or in our copy? Please include such information in the commit message. What I tend to do is to go through a diff while writing the commit message; that way I notice everything which might require a comment. + "This field is also used to identify a virtual \n" + "backend which is used to join several sub backends \n" + "together, Nokia phone typlically needs todo+calendar \n" + "virtual backends" Mind the extra white space at line ends. I don't care about those much in source code, but would like to keep the strings clean. I also suggest a different text: => ----------------------------- \n A special "virtual" backend combines several other data sources and presents them as one set of items to the peer. For example, Nokia phones typically exchange tasks and events as part of one set of calendar items.\n \n Right now such a virtual backend is limited to combining one calendar source with events and one task source. They have to be specified in the "evolutionsource" property, typically as "evolutionsource = calendar,todo".\n \n In all cases the format of this configuration is <backend>[:format] Different sources combined in one virtual source must have a common representation. As with other backends, the preferred format can be influenced via the "format" attribute. ------------------------------- virtual:text/x-vcalendar - a virtual backend uses vCalendar 1.0 format\n" => virtual:text/x-vcalendar - a virtual backend using vCalendar 1.0 format\n" ------------------------------- Regarding the new "evolutionsource" parsing: are spaces around a comma allowed? What if a user has chosen a sync source name which includes a comma? We need some kind of escaping for this. std::vector<VirtualSyncSource*> m_virtualDS; Wouldn't it be better to store shared_ptrs in the vector? The code which uses plain pointers is legacy code; the Funambol library expected an array of such pointers. sourceList.m_virtualDS.push_back (new VirtualSyncSource (params)); would leak the VirtualSyncSource instance if push_back throws an out-of-memory exception. + } + else { The normal formatting is "} else {". You are not checking whether the virtual sync source is enabled. This has to be added, because a virtual sync source configured for a SyncML client peer will show up as inactive source for a SyncML server peer. SE_THROW ("superdatastore now only supports events+tasks case"); => the user has no idea what a "superdatastore" is. The error message must refer to the concepts introduced for the "virtual data source" and include the source name => vSource->throwError() ------- Comment #10 From Chen Congwu 2009-11-23 04:13:21 PST (-) [reply] ------- (In reply to comment #9) Correct. That's not a bug. I need to add a field 'KIND' for calendar+todo dispatch, this filed also covers the original 'ISEVENT field'; therefore I changed it to use 'KIND' instead of 'ISEVENT'. ------- Comment #11 From pohly 2009-11-23 06:33:37 PST (-) [reply] ------- (In reply to comment #10) The upstream libsynthesis syncserv_sample_config.xml still looks wrong to me: it uses <dispatchfilter>F.KIND:=EVENT</dispatchfilter> but only defines ISEVENT <subprofile name="VTODO" nummandatory="1" showifselectedonly="yes" field="ISEVENT" value="0"> Your solution solves this and should be reported to Synthesis. For the server config we don't have a process in place to do this automatically, so an email is due. Can you notify them? ------- Comment #12 From Chen Congwu 2009-11-24 19:19:59 PST (-) [reply] ------- (In reply to comment #9) Updated. Right, I am adding "boost::trim" to strip the spaces and using "\" as the escape character to handle comma in source name problem. Note I am not using SafeConfigNode because it is designed to be used for internal properties which will is read and written only by SyncEvolution. The "evolutionsource" is a public property which maybe set by user explicitly. Therefore I am just adding escape capable decoding and update the documentation accordingly. Pushing to phone branch now.. Please have a look. ------- Comment #13 From pohly 2009-12-18 01:10:34 PST (-) [reply] ------- (In reply to comment #12) Merged. I think we have resolved the part we need for SyncEvolution 1.0. For tracking purposes I am going to split this issue into different parts. -- Configure bugmail: http://bugzilla.meego.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug. You are watching someone on the CC list of the bug.