Making Good Use of your Parallel File System

There are various parallel file systems available. In (roughly) descending order of popularity

  • GPFS - IBM's General Parallel File System
  • Lustre from Intel
  • FhGFS - Fraunhofer File System
  • Hadoop Distributed File System
  • GFS - Google File System
  • Panasas file system

For a more comprehensive list see, for example, Wikipedia

In many cases, there are several read/write heads and, under the right circumstances, codes can read and write in parallel thus reducing IO bottlenecks.

Typically, to read or write in parallel, a code would make use of

  • MPI-2
  • Hadoop