1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
|
# Cooperative Computing Tools Documentation
## Getting Started
- **[About](about.md)**
- **[Installation](install)**
- **[Getting Help](help)**
## Software Components
- [**Makeflow**](makeflow) is a workflow system for parallel and distributed
computing using either the classic Make syntax or the more advanced
[JX Workflow Language](jx). Using Makeflow, you can
write simple scripts that easily execute on hundreds or thousands of
machines.
- [**Work Queue**](work_queue) is a system and library for creating and
managing scalable master-worker style programs that scale up to thousands of
machines on clusters, clouds, and grids. Work Queue programs are easy to
Python ([example](work_queue/examples/work_queue_example.py)|[api](http://ccl.cse.nd.edu/software/manuals/api/html/namespaceWorkQueuePython.html))
Perl ([example](work_queue/examples/work_queue_example.pl)|[api](http://ccl.cse.nd.edu/software/manuals/api/html/work__queue_8h.html)),
or C ([example](work_queue/examples/work_queue_example.c)|[api](http://ccl.cse.nd.edu/software/manuals/api/html/work__queue_8h.html))
.
- [**Resource Monitor**](resource_monitor) is a tool to monitors the cpu,
memory, io, and disk usage of applications running in distributed systems,
and can optionally enforce limits on each resource. The monitor can be
compiled to a single executable that is easily deployed to track executable
file, or it can be used as a library to track the execution of [Python
functions](http://ccl.cse.nd.edu/software/manuals/api/html/namespaceresource__monitor.html).
- [**Parrot**](parrot) is a transparent user-level virtual filesystem that
allows any ordinary program to be attached to many different remote storage
systems, including HDFS, iRODS, Chirp, and FTP.
- [**Chirp**](chirp) is a personal user-level distributed filesystem that
allows unprivileged users to share space securely, efficiently, and
conveniently. When combined with Parrot, Chirp allows users to create custom
wide-area distributed filesystems.
- [**Catalog Server**](catalog) is a common facility used to monitor
running services, workflows, and tasks. It provides real-time status
and historical data on all components of the CCTools.
## Research Prototypes
- [**Accelerated Weighted Ensemble**](awe) (AWE) is an ensemble
molecular dynamics applications that uses Work Queue to scale
out molecular simulations to thousands of GPUs on multipel clusters.
- [**Confuga**](confuga) is an active storage cluster file system designed for
executing DAG-structured scientific workflows. It is used as a collaborative
distributed file system and as a platform for execution of scientific
workflows with full data locality for all job dependencies.
- [**Umbrella**](umbrella) is a tool for specifying and materializing execution
environments, from the hardware all the way up to software and data. Umbrella
parses a task specification and determines the minimum mechanism necessary to
run it. It downloads missing dependencies, and executes the application
through the available minimal mechanism, which may be direct execution, a
system container, a virtual machine, or submissions to a cloud and cluster environments.
- [**Prune**](prune) Prune is a system for executing and precisely preserving
scientific workflows to ensure reproducibility. Every task to be executed in
a workflow is wrapped in a functional interface and coupled with a strictly
defined environment.
## Reference Information
- [**Man Pages**](man_pages.md)
- [**JX Workflow Language**](jx)
- [**Chirp Protocol Specification**](chirp/chirp_protocol.md)
- [**Networking Configuration**](network)
- [**API**](http://ccl.cse.nd.edu/software/manuals/api/html/index.html)
|