q pipeline manager
q pipeline manager Description
The q utility is a platform for creating and managing data analysis pipelines. It expands the value of your existing job scheduler - either Grid Engine or TORQUE PBS - through numerous functions that help you organize, submit, monitor, manage and share your informatics work.
Data processing pipelines require high-level organization and parallelization of work to optimize resource utilization and decrease the time to results. q (from queue) allows complex job sequences to be efficiently assembled and managed, including dependency tracking, parallelization, and pipeline-level monitoring, error recovery, and data protection. Pipelines are constructed from modular script files, with job definitions and results stored in easily retrieved job files. A web interface facilitates job submission and monitoring, with the complete pipeline exportable for full transparency.
q pipeline manager Features
Organization of work into higher order units from existing code pieces in any language.Pipelines are submitted and managed as a unit by calls to a single master file.Robust definition and tracking of job dependencies.Multiple modes of parallelization maximize resource utilization and minimize time to results.Admixture of submission-time and execution-time commands supports dynamic job configuration.Intelligent tracking of queued jobs supports incremental execution of new work.A few q-specific syntax elements create complex pipelines with minimal code.Highly modular structure facilities code reuse and sharing.Various features facilitate cross-platform implementation of pipelines.Extensive submission-time syntax checking helps prevent fractured pip elines.Enhanced job reporting and interfaces into job scheduler functions, including error handling.Write-protection and backup of output files integrated into pipeline definition files.History tracking of all work performed in a pipeline, with rollback.Web-based interface into job submission and management.Fully transparent sharing of work at publication via comprehensive HTML reports.End-user installation with no required management by system administrators. Download:
Click Here
