Ah, that makes sense. Malloc is always high on my list of potential performance culprits, so it was one of the first things started thinking about when looking at this code.
One thing that seemed a bit odd is that, since paramv is defined as a **, an EDT has to do two mallocs for each EDT it creates, one for the paramv array, and one for the pointer to the paramv array. Having not been part of the design discussions, it wasn’t clear to me what advantage you get by passing a pointer to a pointer to paramv, but this might be someplace where a custom allocator might speed things up. There are certainly things I could have done to tune that part of my code (re-use the input paramv for each EDT, do one malloc of all the paramv space each EDT needs and hand-generate the sub-pointers, etc.), but this seems like something users aren’t going to be excited about doing themselves.
Figuring out how to implement recursion in EDTs wasn’t a big deal. It took a bit of thinking to figure out how to do it and a bit more playing with the dependencies to get them passing around right, but now that I’ve done it once, it wouldn’t take long to do again. Some sort of template or example in the documentation would be good, though.
The competitive performance was definitely encouraging, particularly since this is a good problem for Pthreads and since the OCR-code overhead was a significant percentage of execution time. Once you’ve had some time to tune it, I’m betting that OCR will turn in some impressive numbers.
This is really helpful. We had to resort to malloc from our in-house memory allocator (heavily influenced by Cilk allocator) because of licensing issues for the SC release. We intend to ship OCR with a scalable allocator for the next release.
There will an allocation for each EDT, as the task has to live in the heap. We also plan to cut the number of allocations down for the next release. We implicitly build a class hierarchy (remnants of our initial object-oriented design) and incur allocations on the way. However the base classes are static interfaces and we will restructure that code to avoid unnecessary allocation/replication.
Regarding recursion, I can cite the usual cop-out and say you have the luxury to build the reduction tree as you like. Nevertheless, this point was also raised by CnC users and probably most papers which criticize data-flow models. We may consider delivering some prepackaged way to build a recursion tree to flatten the learning curve.
I am personally glad the performance is competitive though we did not have the time to tune for performance. This is rather encouraging.
Thanks for your time,
 For those who recall the Chord-CnC effort with Intel's Mayur Naik
On Jan 21, 2013, at 12:11 PM, "Cledat, Romain E" <firstname.lastname@example.org> wrote:
Thanks for the feedback. This is very detailed and useful. I think some take-aways are:
- definitely have to focus on some of the memory overhead and see if some things can be optimized
- there seems to be a scaling issue as the number of cores go up. We should probably investigate this.