Re: [SyncEvolution] server class + fork/exec dbus-server + command line testing

2012/4/13 Patrick Ohly <patrick.ohly(a)intel.com&gt;: > On Fr, 2012-04-13 at 14:38 +0200, Patrick Ohly wrote: >> On Do, 2012-04-12 at 15:59 +0200, Krzesimir Nowak wrote: >> > Some of the tests are implemented in following branch: >> > https://meego.gitorious.org/~krnowak/meego-middleware/krnowaks-syncevolut... >> > >> > Commits are not so informative - will do it later. >> >> And they are also incomplete, aren't they? stdout and stderr from >> running the command line tool are captured, but they are not checked >> anywhere. > > The way how they are captured doesn't allow checking the right order of > output via stdout and stderr, because they are stored in separate > strings. But I am hard-pressed to suggest a better solution: one could > write a custom communicate() method which uses two pipes and copies into > one string for each stream and one combined string. > > But that suffers from race conditions. If output is ready in both > streams when the parents starts to collect it, then the information > about which output is older is already lost. > > I'm inclined to forgo testing the stdout/stderr distinction in most > tests: use the same pipe for both (so that we can test the order and > thus how it really looks to users) in most tests and then write a few > tests which check for stdout/stderr usage (and ignore the order). I wonder if there is some crazy platform where stdout is buffered so long, that unbuffered stderr shows up first even if something earlier was printed into stdout. I guess that such tests are hardly reliable as an order of messages is guaranteed for one stream, but not between two streams. My idea would be to print usage messages to different streams depending on situation. That is - if I call "syncevolution --help" then it would print usage to stdout. But if I call "syncevolution --template" then it would print usage to stderr with some "[ERROR] missing parameter for 'template' option" following it. Are there any other cases when we want to have messages in stdout and stderr in some order?

> Actually, any test which expects output only via one pipe can use two > pipes, because order doesn't matter when one pipe never has data. > > Below my patch to make the tests work for me: > 1. I had to set SYNCEVOLUTION_TEMPLATE_DIR before invoking > test-dbus.py, otherwise os.environ["SYNCEVOLUTION_TEMPLATE_DIR"] > fails. This really should be rethought. Yes, because there is a test (e.g. testMatchTemplate) which wants different directory ("$(srcdir)/test/testcases/templates"). Maybe copying some files/directories during build to $(builddir)/src would suffice. That way we can specify SYNCEVOLUTION_TEMPLATE_DIR with relative directories instead (like C++ tests are doing it).

> 3. Output checking only gives "normal" results when > syncevo-dbus-server is run without SYNCEVOLUTION_DEBUG. > @property('debug', False) achieves that (better name welcome). I just added a function stripping all lines having "[DEBUG" prefix. The original output with debugs might be useful (maybe print it when some test fails).

> This is just a quick-and-dirty solution, not committed anywhere. Shall I > do the base work for a few tests and then let you continue with adding > the rest? Ok. I have some tests already written, but I will adapt them to your changes. I still have to rebase them to your ccs-pohly-cmdline branch,