Re: [SyncEvolution] server class + fork/exec dbus-server + command line testing

On Fri, 2012-04-13 at 15:28 +0200, Patrick Ohly wrote: Below are the commits, already merged into master after a quick test run: http://syncev.meego.com/2012-04-16-12-37_lucid-amd64_prebuilt-amd64_testi... I started with your initial commit and then included the changes that we were discussing, plus a fix for a failure still visible in that test run. The reason is that your scanFiles() implementation is slightly different from the C++ version. In contrast to the C++ version it includes lines like: sources/addressbook/config.ini:# ActiveSync Address Book = eas-contacts sources/addressbook/config.ini:# ActiveSync Events = eas-events sources/addressbook/config.ini:# ActiveSync Todos = eas-todos sources/addressbook/config.ini:# ActiveSync Memos = eas-memos These need to be excluded by the comparison because these comments can change, depending on how the binaries were compiled or run. Your reference text in the ScheduleWorld config includes one particular variation and therefore it worked for me when testing manually, but the nightly testing differs. A good test for the assertEqualDiff() :-) See: http://syncev.meego.com/2012-04-16-12-37_lucid-amd64_prebuilt-amd64_testi... With everything in place, do you still want to convert some more tests? I think one commit per converted test would be good, with C++ removed and Python added in the same test. I expect that most tests can and should be moved. The only reason for keeping a test in the C++ code would be if it explicitly tests some aspect that cannot be tested when going through D-Bus - not sure whether there is any such case. commit 2925b97bac45ae4dc4dac92b04b91683cad29ddc Author: Krzesimir Nowak <krnowak(a)openismus.com&gt; Date: Thu Apr 12 15:45:02 2012 +0200 D-Bus Testing: add Cmdline tests framework This is a translation of the corresponding C++ TestCmdline from Cmdline.cpp into Python. commit aa562e5ef97eba78b7c45ab095df5071134840dc Author: Patrick Ohly <patrick.ohly(a)intel.com&gt; Date: Mon Apr 16 09:56:50 2012 +0200 D-Bus Testing: added assertEqualDiff() Replaces the diffStrings() from the previous commit. Renamed it because it really is specifying an assertion and moved it to DBusUtil because it may be useful outside of the command line testing. Also made sure that the diff is part of the assertion because then it will be repeated when listing the failure. commit 5389b8eeeddc328e29a86c91aec8606ac3fb9741 Author: Patrick Ohly <patrick.ohly(a)intel.com&gt; Date: Mon Apr 16 10:54:52 2012 +0200 D-Bus Testing: ensure that we have assertRegexpMatches() unittest.assertRegexpMatches() is very useful for command line output testing. But it is not yet available in Ubuntu Lucid, the current base line for SyncEvolution releases. Therefore provide a fallback implementation. commit 0e63bc8e59b53a8821beff9f29d061cae48a66fc Author: Patrick Ohly <patrick.ohly(a)intel.com&gt; Date: Mon Apr 16 17:26:12 2012 +0200 D-Bus Testing: fixed scanFiles() The function must return all property assignments, including those which are commented out. The Python version accidentally included some comment lines which looked like assignments, partly because it ignored leading spaces, partly because it looked anywhere in the line for an equal sign. Fixed by using a regex for the matching. commit 67ce74b57fcb51f57b8cf29e353e13da16d60b8b Author: Patrick Ohly <patrick.ohly(a)intel.com&gt; Date: Mon Apr 16 12:13:14 2012 +0200 D-Bus Testing: eliminate TestCmdline.testdir All D-Bus tests already have their private xdg_root = 'temp-test-dbus' which is cleared before running each test. Use it also for TestCmdline instead of creating our own temporary directory, for the sake of consistency. Another advantage is that the syncevo-dbus-server is running with that same XDG root and reading/writing files in it automatically. The command line client currently *does* send its XDG env variables and the server uses them, but perhaps that feature will have to be removed, because it has a design mistake (*), so don't depend on it. (*) If different clients and the auto-syncing code in the server are allowed to use different config roots, the 'config name' alone no longer is a unique identifier for configs. commit 7c2de4ed5199f9c971d1474595f5a04988c1eac3 Author: Patrick Ohly <patrick.ohly(a)intel.com&gt; Date: Mon Apr 16 12:58:52 2012 +0200 D-Bus Testing: added running of command line tool The 'syncevolution' command line tool is started inside the same D-Bus session as the 'syncevo-dbus-server' that is started before running each test. They use the same XDG env variables. The test-dbus.py script is able to watch and report all D-Bus traffic between tool and server. Because tests are meant to be realistic, SYNCEVOLUTION_DEBUG has to be optional and must be turned off for all tests which test for real output as seen by users. Every single invocation of runCmdline() should check stdout and stderr, to detect unexpected output. The return code is already tested by runCmdline() itself, unless explicitly turned off (may be useful for tests which expect certain kind of failures). Ideally the entire string gets tested, either via an exact string comparison or a full regex match. Note that assertRegexpMatches() only does a search, so use ^ and $ to ensure full match. If (and only if) the command is expected to produce output on both stdout and stderr where the order matters (for example, syncing), then stdout and stderr should be combined into one stream. Otherwise the order cannot be tested. commit 470f0e7d7a56b96adff21e5e918c58ee0a9e3420 Author: Patrick Ohly <patrick.ohly(a)intel.com&gt; Date: Mon Apr 16 13:02:23 2012 +0200 D-Bus Testing: added testSetupScheduleWorld The test is a straight copy from Cmdline.cpp. Because the hard-coded default SSL certificate path is not known in test-dbus.py, it has to extract that string by printing a template once with the command line tool (only the command line exposes the comments, normal D-Bus API doesn't) and finding it in the output. This allows copying the C++ tests without making them less strict by filtering out the SSL certificate path before the result comparisons. commit a27f2af580a00d81107f69bc5802c079cef23766 Author: Patrick Ohly <patrick.ohly(a)intel.com&gt; Date: Mon Apr 16 14:38:34 2012 +0200 D-Bus Testing: work around Python 2.6 subprocess bug Explicitly pass an environment when running the command line tool. Otherwise subprocess.Popen() from Python 2.6 uses not os.environ (which would be okay) but rather the environment passed to a previous call to subprocess.Popen() (which will fail if the previous test ran with an environment which had SYNCEVOLUTION_DEBUG set). -- Best Regards, Patrick Ohly The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel's position on the issue, nor am I authorized to speak on behalf of Intel on this matter.