The Framework Foundation is a non-profit organization dedicated to putting together, maintaining, and popularizing end-to-end open-source data pipelines and cognitive frameworks, abbreviated the SMACK Stack and the MIND Stack. This document outlines the mission, goals, and work of the foundation.
SMACK Stack The original acronym SMACK stands for the end-to-end data pipeline operationalized on Mesos, consisting of
· API: Akka
· Message Bus: Kafka
· Compute: Spark (also the language of Spark, Kafka, and other distributed systems: Scala)
· Persistence: Cassandra
· Operationalization: Mesos (M also signifies Memory, as in Alluxio, GridGain, etc.)
MIND Stack The MIND stack denotes the Cognitive Frameworks that run on top of the SMACK stack. It encompasses various approaches and OSS for
· Machine Learning
There is a huge gap between AI/Cognitive as generally discussed and Data Engineering/Operationalization of the web-scale data pipelines required to feed that AI with data. Enterprises, generally, do not run Python in production, nor do they run scalable APIs on Python. SMACK is a reference architecture. All of the letters in SMACK are replaceable. You can fathom using YARN instead of Mesos, Flink instead of Spark, HBase instead of Cassandra, Kinesis instead of Kafka, and so on. The key point is that OSS components of data pipelines do n to exist in vacuum. You need to care and feed them, taking inputs from their predecessors (or a million devices for the API) and sending the results back to the next step (or back to the users), often in near-real-time. Dr. Khrabrov, the founding director of the Foundation, had first popularized the SMACK stack — see its history.
Open-Source Integration Challenge Open Source succeeded at every step of the stack, beating proprietary opponents. However, when it comes to building complete backends, extra effort is required to assemble solutions from OSS components. A reference SMACK Stack implementation, available for experimentation and rapid prototyping, would allow best practices to emerge in the solution space where complete backends are being architected and built to power the next generation of cognitive frameworks. Open Source succeeded and won against every individual vendor. E.g. Spark is replacing multiple “big data” vendors of yore. However, when the time comes to build an actual backend for a startup or an enterprise, we see the following rule of thumb apply. It takes a year, a million dollars, and a team of five senior engineers to build a bespoke SMACK stack. Why do we (the OSS community) succeed at the component level, but fail at the integration level? One answer is the freemium model that most data vendors use, including all the AMPLab graduates. A vendor supports a limited set of integrations necessary for their product, e.g. Spark with Kafka. A vendor alliances are discussed but never materialize in the form of a maintained stack on GitHub. The premise of the Framework Foundation is quite simple: we can succeed at the stack level in the same way we succeed at the component level. We need a better OSS integration model.
Non-Profit with Industry-Readiness and Community Education Focus
The Framework Foundation is currently producing educational materials based on the Bay Area meetups and conferences.
With a permanent foundation setup we’ll be able to facilitate SMACK collaborations and hackathons, fostering solutions emerging bottom-up, and also ensure that the solutions are helping industry partners in building cognitive solutions. The following activities are planned for the second half of 2017 and into 2018: Scale By the Bay + Data By the Bay — the fifth year of the flagship conference. Focus on
· Data Pipelines
· Functional Programming
· Reactive Microservices
Rethink.Money — a through technical exploration of the disruption coming to Fintech. A track on November at Scale, and a separate event in Amsterdam in March 2018. Hackathons on SMACK+MIND in San Francisco.
Partners We collaborate with multiple companies building and using components of the SMACK and MIND stacks already. Participants, sponsors, and open-source contributors include
· UC Berkeley
… and many more.
Applications of the Reference SMACK Stack A typical deep-knowledge data-centric startup consults on a specific step in the pipeline. E.g., Skymind develops deeplearning4j that is great at running various deep learning algorithms on data sets brought by a SMACK stack. In an industry setting some kind of a SMACK stack should be in place. If it isn’t, the work to set it up might be too much and out of scope for the consultants, or they have to team up with others and it can delay things. A reference stack makes it easier to plug into.
Shared Core for Community Work If an outsourcing or consulting team uses the reference stack as a foundation of their contract work, contributing back improvements, it makes its easier to reuse and maintain. Several outsourcing companies in our communities are working towards such an approach.
Learning and Education Having a popular reference implementation will accelerate its development in a virtuous cycle similar to Spark, with all the benefits of that process for open-source. As the co-founder of the original Spark Users meetup I’ve seen the dynamics upfront and believe you can reproduce it with other OSS projects through community stewardship.
The Foundation Structure The foundation should be structured as a three-tiered system. The higher tier gets the voting rights to direct the strategy, and the lower tier gets the benefit of the community engagement and support.
COMMUNITY STEWARDSHIP AND ENGINEERING STAFF The Foundation strives to achieve its goals through fostering alliances between the components of the reference stack, with all the work being done as open source. The Foundation may hire its own engineering staff or collaborate with existing organization, for the purpose of advancing a production-ready SMACK Stack.
The Board The Board consists of the voting members (see below). The board meets quarterly and sets the priorities for the projects. The Executive Director of the Foundation takes the priorities and maps them out into the workplace.
FOUNDING MEMBERS Founding members enable creation of the Foundation. They make special contributions and run the Foundation activities jointly. Founding members are also Board Members for the three years since the founding contribution.
Voting Members Voting members contribute $50K annually and get a vote on the board. There’s an ongoing communication between the board meetings to keep track of the priorities.
Non-voting Members Non-voting members contribute $15K annually and get the full Foundation support in its community outreach.
The Foundation Process The foundation works to obtain and maintain the reference SMACK Stack on Github agreed upon and used by the most practitioners in the field. It pursues multiple avenues to achieve it.
Fostering Industry-Focused Alliances Since FF is not a vendor, it crystallized and serves as a common neutral ground for all the letters in SMACK to work together. FF strives to collaborate and secure contributions and relationships with each letter company to work o SMACK directly. It also works with the SMACK users in industry to bring the SMACK components together for given use cases.
Fostering Alliances among the OSS Projects We have many companies in our communities relying on the SMACK Stack for their end products, who are interested in plugging into it. Those companies are most interested in a reliable easily deployable, production-ready SMACK to add their value on top of it. Since they go into the field they have the most relevant feedback.
Working with the Developer Teams and marketplaces There are multiple teams commercially building the SMACK Stack. We posit that having a common core will lift all the boats, and already have companies in the network interested in working on the SMACK Stack. There are new platforms in the works that bring together companies and software providers. They can use SMACK capabilities to certify and connect consultants with customers, and hence are interested in convergence.
Learning and Education We’ll work with L&E capabilities in the letter companies, specialists, and all others, as well as produce our own strong content (FunctionalTV) to popularize SMACK among the developer champions, following the Spark adoption model. FunctionalTV is a developer channel and the Foundation will expand its coverage of the Cognitive Frameworks, their components, and the end-to-end integrations needed to apply them.
Deliverables The key “deliverable” of the Foundation is the strong community of learners and doers, building new companies, solving bold problems at scale with the stacks we promote and support. We also want to see the stacks used successfully by the industry partners, both growing their businesses, and also engaging more engineers and startups from the community, lifting all the boats. We’ll need to define the metrics that work best for the board and the community as we ramp up, but they would generally address the following criteria:
· the reference OSS SMACK Stack, maintained by the community, engagement around it
· the combinations of projects comprising parts of the SMACK stack as best practices
· OSS use cases from partners ready to use as a template in new companies
· Industrial adoption of the Foundation-affiliated solutions and stacks
· Developer TV channel and related media (blogs, newsletter)
Governance The Foundation will be governed by the Board, that will set priorities quarterly. Industry partners will keep the focus of stack work on the enterprise-ready, pragmatic solutions immediately useful by the community.
2017-2018 Events Scale By the Bay + Data By the Bay: Fall 2017 (November) Rethink.Money: November 2017 in the US, Spring 2018 in Amsterdam SMACK+MIND Hackathon: October 2017, then scaling up Spring 2018: AI+Data By the Bay, San Francisco