On Tue, 2012-03-13 at 12:57 +0100, Thomas Pequet wrote: > Le 13/03/2012 10:43, Patrick Ohly a écrit : >> On Tue, 2012-03-06 at 17:16 +0100, Thomas Pequet wrote: >>> Le 06/03/2012 13:14, Patrick Ohly a écrit : >>>> Thomas Pequet wrote: >>>> It's a bit curious that in one case the anchor is a time stamp, in the >>>> other a number. Both are strings sent by the Memotoo server. Did you >>>> perhaps change the anchor formatting from "seconds since epoch as time >>>> stamp" to "seconds since epoch as integer"? I haven't checked whether >>>> 20120228T115051Z matches 1330608696000 when interpreted like that. >>> It is strange, Memotoo return only date with seconds (ex: >>> 1330608696000) not timestamp... So why SyncEvolution store the >>> timestamp ???? >> SyncEvolution and libsynthesis treats the sync anchor as string. It >> should never convert to a time stamp. Does your user still have other >> logs? He can do a "grep -l -r 20120228T115051Z ~/.cache/syncevolution" >> to find all relevant log files. I got logs for two sessions from Thomas. In the session from 2012-02-28 there is an entry for the anchor from Memotoo as UTC date/time: [2012-02-28 12:50:51.753] Received<next> Remote Server Anchor='20120228T115051Z' (to be compared with<last> in NEXT session) Then in the next session, Memotoo reverts to the integer format: [2012-03-01 14:31:36.449] Saved Last Remote Server Anchor='20120228T115051Z', received<last> Remote Server Anchor='1330603400000' (must match for normal sync) [2012-03-01 14:31:36.449] Received<next> Remote Server Anchor='1330608696000' (to be compared with<last> in NEXT session) This confirms my theory that the Memotoo server switched between different ways of formatting its internal time stamp. There are no XML message dumps to proof it, but I am nevertheless confident that this really what the Memotoo server sent. There simply isn't any code in libsynthesis which transforms remote server anchors.

I actually work on a new version of Memotoo SyncML server to be more faster, ... Is it possible to start the tests you done for Memotoo ? If Yes, can you start the tests on this address: http://www.memotoo.com/syncmldev (the database is the same, just the scripts are not the same) and give me the results ???

On Wed, 2012-03-21 at 17:22 +0100, Patrick Ohly wrote: > On Tue, 2012-03-20 at 15:59 +0100, Thomas Pequet wrote: >> I actually work on a new version of Memotoo SyncML server to be more >> faster, ... >>> Is it possible to start the tests you done for Memotoo ? >> If Yes, can you start the tests on this address: >> http://www.memotoo.com/syncmldev (the database is the same, just the >> scripts are not the same) and give me the results ??? > I'm currently heads down working on SyncEvolution source code. I'll run > the tests against the new server, but I'll take a few more days before I > get to it. The first test result is in, and shows a failure due to some WBXML encoding issue in one of the later messages sent by Memotoo: http://syncev.meego.com/2012-03-27-11-57_dist/prebuilt-amd64/23-memotoo/C... http://syncev.meego.com/2012-03-27-11-57_dist/prebuilt-amd64/23-memotoo/C... I've increased the loglevel so that future tests will dump the full incoming message (and not just the incomplete translation, as above) and use XML instead of WBXML. But perhaps you see something in the server-side logs already?

On Tue, 2012-03-27 at 18:10 +0200, Thomas Pequet wrote: >> But perhaps you see something in the server-side logs already? > Sorry there was a easy bug to correct and now it is better :) > Can you try again ? I ran the full set of tests: http://syncev.meego.com/latest/testing-amd64/nightly.html#memotoo

The testManyItems for events failed with something that looks like an internal error in EDS and/or SyncEvolution. It is unclear why the following item was asked for in the first place. I'll investigate: [ERROR 00:00:53] error code from SyncEvolution object not found (remote, status 404): eds_event: retrieving item: 008-1234567890!@#$%^-rid http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e...

In a case where<MoreData> was necessary the server rejected the second chunk of the<Add> with a 500 error: http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... First part: http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... Second part: http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... 500 error: http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... Does Memotoo support<MoreData>?

That issue probably also affects some more (no pun intented) of the tests involving large items (testMaxMsg, testLargeObject). Memos don't seem to be affected, but I suspect that this is merely because splitting a plain text into two halves doesn't result in unparsable items.

On Fri, 2012-03-30 at 09:32 +0200, Thomas Pequet wrote: > Le 30/03/2012 09:02, Patrick Ohly a écrit : >> The testManyItems for events failed with something that looks like an >> internal error in EDS and/or SyncEvolution. It is unclear why the >> following item was asked for in the first place. I'll investigate: >>> [ERROR 00:00:53] error code from SyncEvolution object not found (remote, status 404): eds_event: retrieving item: 008-1234567890!@#$%^-rid >> http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... > I have corrected the MoreData bug, Confirmed, that seems to work now (manual tested, still need to run the full suite again). > so may be it will be better for > testManyItems. > I think the problem code from Memotoo with the new version. Give me the > XML and I think I will see the problem It's indeed on the server side. The server seems to truncate the local ID at the& sign. <Add> command sent to server: http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... <Status> received back: http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... I know, I am a nasty bugger for sending such tricky characters in the local ID, but these are the corner cases which have to be tested>:->

On Fri, 2012-03-30 at 15:16 +0200, Thomas Pequet wrote: > Le 30/03/2012 11:31, Patrick Ohly a écrit : >> On Fri, 2012-03-30 at 09:32 +0200, Thomas Pequet wrote: >>> Le 30/03/2012 09:02, Patrick Ohly a écrit : >>>> The testManyItems for events failed with something that looks like an >>>> internal error in EDS and/or SyncEvolution. It is unclear why the >>>> following item was asked for in the first place. I'll investigate: >>>> >>>> [ERROR 00:00:53] error code from SyncEvolution object not found (remote, status 404): eds_event: retrieving item: 008-1234567890!@#$%^-rid >>>> http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... >>> I have corrected the MoreData bug, >> Confirmed, that seems to work now (manual tested, still need to run the >> full suite again). >> >>> so may be it will be better for >>> testManyItems. >>> I think the problem code from Memotoo with the new version. Give me the >>> XML and I think I will see the problem >> It's indeed on the server side. The server seems to truncate the local >> ID at the& sign. >> >> <Add> command sent to server: >> http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... >> >> <Status> received back: >> http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... >> >> I know, I am a nasty bugger for sending such tricky characters in the >> local ID, but these are the corner cases which have to be tested>:-> > You have right to want to test all case ! :) > Theses links do not work ... Can you send me the good link to see the XML ? Replace "/latest/" with "/2012-03-29-17-01_testing_memotoo/" and you'll find the files.

On Fri, 2012-03-30 at 20:06 +0200, Thomas Pequet wrote: > Le 30/03/2012 15:47, Patrick Ohly a écrit : >> On Fri, 2012-03-30 at 15:16 +0200, Thomas Pequet wrote: >>> Le 30/03/2012 11:31, Patrick Ohly a écrit : >>>> On Fri, 2012-03-30 at 09:32 +0200, Thomas Pequet wrote: >>>>> Le 30/03/2012 09:02, Patrick Ohly a écrit : >>>>>> The testManyItems for events failed with something that looks like an >>>>>> internal error in EDS and/or SyncEvolution. It is unclear why the >>>>>> following item was asked for in the first place. I'll investigate: >>>>>> >>>>>> [ERROR 00:00:53] error code from SyncEvolution object not found (remote, status 404): eds_event: retrieving item: 008-1234567890!@#$%^-rid >>>>>> http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... >>>>> I have corrected the MoreData bug, >>>> Confirmed, that seems to work now (manual tested, still need to run the >>>> full suite again). >>>> >>>>> so may be it will be better for >>>>> testManyItems. >>>>> I think the problem code from Memotoo with the new version. Give me the >>>>> XML and I think I will see the problem >>>> It's indeed on the server side. The server seems to truncate the local >>>> ID at the& sign. >>>> >>>> <Add> command sent to server: >>>> http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... >>>> >>>> <Status> received back: >>>> http://syncev.meego.com/latest/testing-amd64/23-memotoo/Client_Sync_eds_e... >>>> >>>> I know, I am a nasty bugger for sending such tricky characters in the >>>> local ID, but these are the corner cases which have to be tested>:-> >>> You have right to want to test all case ! :) >>> Theses links do not work ... Can you send me the good link to see the XML ? >> Replace "/latest/" with "/2012-03-29-17-01_testing_memotoo/" and you'll >> find the files. > Ok thanks ! I have corrected the problem of "&" :) > You can send again the tests. Sorry for the delay, I was away last week. A full test run today shows that ID problem still exists: http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23...

MoreData support is better, but not fully okay yet. ----------------------------- http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... vCard data with lots of "> & > & > &..." looses a space at some point. Does not happen when using WBXML. XML message dumps are here (sending to Memotoo and retrieving again): http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... It is suspicious that the affected item was sent using MoreData.

----------------------------- http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... In that log client A, client B and the server have the same item. Client A and client B both modify that item, then client A updates the server: http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... Then client B tries the same, leading to a conflict on the server: http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... The expectation is that the server somehow resolves that conflict. The tests shows that Memotoo merges the conflicting item data. After client B is done with the server, client A synchronizes again: http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... The expectation is that in that sync, client A is sent the data as it now exists in client B and the server. The server sends "John Doe" with a "X-AIM" property: http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... But client B doesn't have that property. It seems to me that the server should have sent the merged item to client B in the "conflict" sync above, which it didn't do.

----------------------------- Finding twins during a slow sync fails for calendar events. Although client and server have exactly the same data, a duplicate item is created on the client: http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... Populate server: http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... Slow sync: http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23...

------------------------------

On Mi, 2012-04-11 at 11:43 +0200, Thomas Pequet wrote: > Le 10/04/2012 17:30, Patrick Ohly a écrit : >> On Fri, 2012-03-30 at 20:06 +0200, Thomas Pequet wrote: >> A full test run today shows that ID problem still exists: >> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... > Ok now it is corrected :) Confirmed. >> MoreData support is better, but not fully okay yet. >>> ----------------------------- >>> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> vCard data with lots of "> & > & > &..." looses a space at some point. >> Does not happen when using WBXML. XML message dumps are here (sending to >> Memotoo and retrieving again): >>> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> It is suspicious that the affected item was sent using MoreData. > This problem is because I use a "trim" function when I get the data... > So if I receive with "space" at the end or at the begin, it is removed ... > Now it will be better. Confirmed. >> ----------------------------- >>> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> In that log client A, client B and the server have the same item. Client >> A and client B both modify that item, then client A updates the server: >> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> Then client B tries the same, leading to a conflict on the server: >> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> The expectation is that the server somehow resolves that conflict. The >> tests shows that Memotoo merges the conflicting item data. >>> After client B is done with the server, client A synchronizes again: >> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> The expectation is that in that sync, client A is sent the data as it >> now exists in client B and the server. The server sends "John Doe" with >> a "X-AIM" property: >> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> But client B doesn't have that property. It seems to me that the server >> should have sent the merged item to client B in the "conflict" sync >> above, which it didn't do. > I will have difficult to correct this ... Will the updated item on the server be sent in the next sync? Otherwise client and server remain permanently out of sync. >> ----------------------------- >>> Finding twins during a slow sync fails for calendar events. Although >> client and server have exactly the same data, a duplicate item is >> created on the client: >>> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> Populate server: >> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> Slow sync: >> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... > May be it come from the previous ID problem. > Can you try again ? No, still fails. I reran the test manually, so logs are not public yet.

On Sat, 2012-04-14 at 11:28 +0200, Thomas Pequet wrote: > Le 13/04/2012 17:16, Patrick Ohly a écrit : >> On Mi, 2012-04-11 at 11:43 +0200, Thomas Pequet wrote: >>> Le 10/04/2012 17:30, Patrick Ohly a écrit : >>>> ----------------------------- >>>>>>> Finding twins during a slow sync fails for calendar events. Although >>>> client and server have exactly the same data, a duplicate item is >>>> created on the client: >>>>>>> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>>>>>> Populate server: >>>> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>>>>>> Slow sync: >>>> http://syncev.meego.com/2012-04-10-08-05_testing_memotoo/testing-amd64/23... >>> May be it come from the previous ID problem. >>> Can you try again ? >> No, still fails. I reran the test manually, so logs are not public yet. > Ok I think I have corrected this :) Yes, seems fixed: http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/nightly.html#m... But it's still not all rosy. ------------------------------------- Unexpected update sent back to client: http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/23-memotoo/Cli... Could this be related to a change for handling the case where the server resolves a conflict (see earlier discussion around testMerge)? Or perhaps it is simply a timing issue? If it is timing, then the client-side test driver cannot do anything about it, because it happens inside a single sync session. In the testCopy case, client A is sending an item that the server doesn't have (Add). Why is the server sending back an update for it? Send: http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/23-memotoo/Cli... Update (same session): http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/23-memotoo/Cli... testCopy only failed for events, worked for contacts, tasks and notes. Same failure in testAddUpdate for events (Add command) *and* tasks (but at a different part of the test, after the Replace command). I've spot-checked some of other failures, they all seem to be about this unexpected update. -------------------------------------

On Wed, 2012-04-18 at 18:07 +0200, Thomas Pequet wrote: >> ------------------------------------- >> Unexpected update sent back to client: >> >> http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/23-memotoo/Cli... >> >> Could this be related to a change for handling the case where the server >> resolves a conflict (see earlier discussion around testMerge)? Or >> perhaps it is simply a timing issue? If it is timing, then the >> client-side test driver cannot do anything about it, because it happens >> inside a single sync session. >> >> In the testCopy case, client A is sending an item that the server >> doesn't have (Add). Why is the server sending back an update for it? >> >> Send: >> http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/23-memotoo/Cli... >> >> Update (same session): >> http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/23-memotoo/Cli... >> >> testCopy only failed for events, worked for contacts, tasks and notes. >> >> Same failure in testAddUpdate for events (Add command) *and* tasks (but >> at a different part of the test, after the Replace command). >> >> I've spot-checked some of other failures, they all seem to be about this >> unexpected update. >> ------------------------------------- > This problem is strange because I can not reproduce it ... I'm not seeing it either when running the eds_event::testCopy test manually now. It succeeds 5 out of 5 times, with and without valgrind. It still failed in the full test over the weekend: http://syncev.meego.com/2012-04-20-15-16_all/testing-amd64/nightly.html#m... But even that isn't deterministic: some tests passed which failed earlier (green), some failed only now (red).

> Does this problem is also with actual SyncML server > (http://www.memotoo.com/syncml";) ? Not as far as I remember. The logs don't go back that far. I'll switch to the regular server for another full run.

On Wed, 2012-04-18 at 18:07 +0200, Thomas Pequet wrote: > > ------------------------------------- > > Unexpected update sent back to client: > > > > http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/23-memotoo/Cli... > > > > Could this be related to a change for handling the case where the server > > resolves a conflict (see earlier discussion around testMerge)? Or > > perhaps it is simply a timing issue? If it is timing, then the > > client-side test driver cannot do anything about it, because it happens > > inside a single sync session. > > > > In the testCopy case, client A is sending an item that the server > > doesn't have (Add). Why is the server sending back an update for it? > > > > Send: > > http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/23-memotoo/Cli... > > > > Update (same session): > > http://syncev.meego.com/2012-04-13-15-25_all/testing-amd64/23-memotoo/Cli... > > > > testCopy only failed for events, worked for contacts, tasks and notes. > > > > Same failure in testAddUpdate for events (Add command) *and* tasks (but > > at a different part of the test, after the Replace command). > > > > I've spot-checked some of other failures, they all seem to be about this > > unexpected update. > > ------------------------------------- > This problem is strange because I can not reproduce it ... I'm not seeing it either when running the eds_event::testCopy test manually now. It succeeds 5 out of 5 times, with and without valgrind. It still failed in the full test over the weekend: http://syncev.meego.com/2012-04-20-15-16_all/testing-amd64/nightly.html#m... But even that isn't deterministic: some tests passed which failed earlier (green), some failed only now (red). > Does this problem is also with actual SyncML server > (http://www.memotoo.com/syncml";) ? Not as far as I remember. The logs don't go back that far. I'll switch to the regular server for another full run.