About this Topic
- This topic is a wiki document for the collaborative development of a project charter.
- Let’s evolve it to a point where it answers all essential questions for a wide audience in establishing a need and justifying an approach for a proposed national project
- Please post replies with any questions that need answering or suggestions for improvements of the presentation or the plan itself
- Direct improvements via wiki editing are also welcome
Background
The Code for America Brigade Network consists of ~80 volunteer groups in cities across the United States. A couple have funding for one paid organizer, but most are completely led by volunteers. Some brigades, in addition to one or two “captains” / executive directors, have a second tier of leaders or co-leaders in charge of functional areas. In most cases, leaders turn over on a ~2 year cycle. Leadership turnover is staggered and planned if possible, but often people have to scale back volunteer responsibilities unexpectedly.
The Mission
Give every brigade the ability to institutionalize projects that benefit their community, such that the community can trust that such projects will remain accessible/usable indefinitely as part of the public commons as long as the brigade exists and is sufficiently resourced in aggregate, and such that project teams can focus on development and hand off deployment.
The Vision
Ideal state: maximally localized clusters
The ideal state of the civic cloud is each brigade operating their own completely independent civic cluster. A shared national volunteer team would publish patterns, build tools, and provide a support community for brigades’ local ops teams. Code for America and/or peer brigades would hold copies of keys and backups of other key data to provide for continuity of a brigade’s hosting commitments in the event of a local break in leadership (either of the brigade as a whole or its ops team/person).
Meta-clusters
Additional national clusters may exist for large projects that either have a national scope within one instance or have significant benefits to being hosting under a multi-tenant model, but it is preferable to keep all application swithin local clusters where the cost difference isn’t extreme.
Cluster + git repo + capacity pool
Each civic cluster would consist of a collaboratively-managed git repository tracking the desired state of the entire cluster, and a collection of heterogeneous compute and storage resources divided into classifications of primary or secondary. The local brigade’s ops team would be responsible for reviewing and merging changes to the desired-state git repository to add, update, reconfigure, or remove projects from the brigade’s hosting commitment. A software interface will report publicly on the estimated aggregate committed load, and local organizers’ responsibility will be to aggregate enough primary capacity to exceed the committed load at all times, with secondary capacity ideally kept at 30-200% of primary capacity.
Maintaining capacity
The process of adding capacity to a cluster should be as automated as possible, with the most primitive method being to paste a curl | bash script or deploy a premade system image. Both local metal and public cloud virtual capacity must be supported. Advanced methods should include granting API access and a budget to one or more public clouds.
Self-healing
All capacity added to a civic cloud should be assumed liable to be destroyed at any time. Backups should be frequent and deduplicated, and all storage should be sufficiently redundant. The civic cloud will be optimized for long-term, cheap loads as opposed to highly-available, mission-critical loads. Applications accepted to civic clouds for hosting should be able to tolerate several hours of downtime or data loss without significant impact. The cluster should be fully self-healing, with storage re-replicated and runtimes restored from latest snapshots automatically.
Building vs borrowing
The civic cloud should borrow as much “hard” technology as it can from the enterprise world, but not shy away from developing new tools and glue that adapt adopted technology to the constraints that make the civic cloud distinct from typical enterprise use cases. Longevity, resilience, sovereignty, and ease-of-use for brigade users will be prioritized above high-availability and scalability. Economics will take a back seat to these priorities in the short term, but ultimate success should include delivering vastly superior economics over traditional hosting options through the deprioritization of high-availability and scalability.
Use cases
Case 0: General operations tools
Tools like Mattermost, Bitwarden, and Odoo that are already being maintained for cloud-native deployment by enterprise communities. These tools may be used as part of the operation of a brigade or as a utility to one or more project teams. This does not include services that project deployments would have technical dependencies on.
Case 1: Civic projects with national teams
Projects like CourtBot and Laddr that are used by multiple brigades and can be prepared/maintained for civic cloud deployment by a concentrated national effort.
Case 2: Local civic projects
Any project developed by a local brigade team, where the local brigade will need to be educated and supported in preparing and maintaining the project for civic cloud deployment
Case 3: Aggregate national projects
Collaborations between multiple brigades to aggregate capacity from multiple civic cloud clusters to provide scale or redundancy to projects/data of national or regional importance.
Case 4: Pubic interest sponsorships
Applications defined in cases 0-2, but provisioned to support a specific external organization like a public school, non-profit, or community group
Anchor technologies
These technologies have been identified as best fitting the civic cloud model and spirit among their alternatives. Implementations should proceed on top of them unless a concrete case, benchmarked against the existing anchor technology, is presented to use something else instead.
- Containerization: Docker
- Orchestration: Kubernetes
- Build Automation: Chef Habitat
- Networking: ZeroTier
- Snapshotting: Restic
- Object storage: Minio