[HPDD-discuss] Lustre's effect on application pthreads

I have a user application that does a write behind of its buffers as they are dirtied. The main application thread posts writes to a linked list. 4 other threads take writes from the linked list and issue them in their thread space. The writes being posted to the linked list are posted and handled in sequential order and are 4MB in size. The stripe size is 1MB. O_DIRECT is not used. When the file is stripped on 1 OST, things work pretty much as expected. The main application thread that posts the writes to the linked list spends very little time posting the writes and the length of the list grows quickly, as it should. It is easy to observe the asynchronous behavior. When the file is stripped across 16 OSTs, things work OK for about the first 20 writes that get posted, but eventually when the main thread posts a write, it ends up waiting until one the aio threads actually completes the write. All asynchronous behavior is lost. Has anyone ever observed this behavior? Is this the expected behavior? I only see this on Lustre when multiple OSTs are used. Other file systems always exhibit the asynchronous behavior. I should also point out that the system time, as reported in /proc/stat, is significantly higher ( doubled ) when the file is 16 way stripped. Could the increased competition for cpus by Lustre impact the scheduling of the application's threads? John