A standardized workflow to utilise a grid-computing system through advanced message queuing protocols

Abstract

Point-based crop models are frequently used to investigate the interaction of genotype, environment and management (G × E × M) for breeding programs and adaptation research. These studies require the processing of millions of simulations to make assessments in national scales. Here we demonstrate a platform to run crop simulations with HTCondor as implemented across more than 12 000 cores on CSIRO network. The workflow of HTCondor usages was built through a server-client structure with standardized messages (simulations), which were queued by message queue sever. The new workflow maximized the performance of the CSIRO HTCondor service and could simultaneously utilize more than 8000 cores. A case study of 4.8 million APSIM simulations (ca. 2 mins each on model desktop), was completed in 32.5 h. This platform makes it is possible to assess all combinations of G × E × M in a short period. The generic platform has since been used to run other massive serial processing tasks.