In this work we have developed two designs, namely, "Collective I/O" and "Pipelined Collective I/O", of a runtime library for irregular applications. Both models use chunking and compression mechanisms. In the first scheme, all processors participate in compressions and I/O at the same time, making scheduling of I/O requests simpler but creating a possibility of contention at the I/O nodes. In the second approach, processors are grouped into several groups, overlapping communication, compression, and I/O to reduce I/O contention dynamically. Finally, evaluation results are shown that demonstrates that we can obtain significantly high-performance for I/O above what has been possible so far.
This research has been supported by NSF, Intel, and DOE and has supported 2 Ph.D. students, and resulted in 3 conference publications.