Do we have any cloud / distributed storage experts?
Hello, Lately I've been working on network automation stuff for HamWAN, and I'm at the point where storage systems have to be dealt with. I'm wondering if anyone in the group has knowledge to provide some design feedback in distributed storage systems. Not from a "how to use it in AWS" perspective, but rather how to design them ground-up. Ideally we want a couple abstractions available network-wide: 1) A global DB (NoSQL should be fine, and Cassandra is looking possible for this) 2) A global FS (GlusterFS is in very early stages of being evaluated for this) Each of these has to have N-way redundancy where N is the # of sites we have, with each instance being capable of continuous operation even if it finds itself on its own network island for a while. Conflict resolution / merging when the network heals is a must. --Bart
I have some experience with distributed file systems. What do you need to know? Subhro Sent from my iPhone
On Feb 28, 2017, at 12:52, Bart Kus <me@bartk.us> wrote:
Hello,
Lately I've been working on network automation stuff for HamWAN, and I'm at the point where storage systems have to be dealt with. I'm wondering if anyone in the group has knowledge to provide some design feedback in distributed storage systems. Not from a "how to use it in AWS" perspective, but rather how to design them ground-up.
Ideally we want a couple abstractions available network-wide:
1) A global DB (NoSQL should be fine, and Cassandra is looking possible for this) 2) A global FS (GlusterFS is in very early stages of being evaluated for this)
Each of these has to have N-way redundancy where N is the # of sites we have, with each instance being capable of continuous operation even if it finds itself on its own network island for a while. Conflict resolution / merging when the network heals is a must.
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
We’re looking for recommendations on what to use, and how to set it up. As Bart mentioned, the critical aspects of this are N-way redundancy so that any given fragment of the network if it gets separated can still operate, and then graceful handling of re-syncing after the network rejoins. Nigel
On Feb 28, 2017, at 13:15, Subhro Kar <subhro@80386.org> wrote:
I have some experience with distributed file systems. What do you need to know? Subhro
Sent from my iPhone
On Feb 28, 2017, at 12:52, Bart Kus <me@bartk.us> wrote:
Hello,
Lately I've been working on network automation stuff for HamWAN, and I'm at the point where storage systems have to be dealt with. I'm wondering if anyone in the group has knowledge to provide some design feedback in distributed storage systems. Not from a "how to use it in AWS" perspective, but rather how to design them ground-up.
Ideally we want a couple abstractions available network-wide:
1) A global DB (NoSQL should be fine, and Cassandra is looking possible for this) 2) A global FS (GlusterFS is in very early stages of being evaluated for this)
Each of these has to have N-way redundancy where N is the # of sites we have, with each instance being capable of continuous operation even if it finds itself on its own network island for a while. Conflict resolution / merging when the network heals is a must.
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
Well, I can follow up with Subhro off-list. I have more detailed questions than that. Was just looking for volunteers to help out with design. --Bart On 2/28/2017 1:18 PM, Nigel Vander Houwen wrote:
We’re looking for recommendations on what to use, and how to set it up. As Bart mentioned, the critical aspects of this are N-way redundancy so that any given fragment of the network if it gets separated can still operate, and then graceful handling of re-syncing after the network rejoins.
Nigel
On Feb 28, 2017, at 13:15, Subhro Kar <subhro@80386.org> wrote:
I have some experience with distributed file systems. What do you need to know? Subhro
Sent from my iPhone
On Feb 28, 2017, at 12:52, Bart Kus <me@bartk.us> wrote:
Hello,
Lately I've been working on network automation stuff for HamWAN, and I'm at the point where storage systems have to be dealt with. I'm wondering if anyone in the group has knowledge to provide some design feedback in distributed storage systems. Not from a "how to use it in AWS" perspective, but rather how to design them ground-up.
Ideally we want a couple abstractions available network-wide:
1) A global DB (NoSQL should be fine, and Cassandra is looking possible for this) 2) A global FS (GlusterFS is in very early stages of being evaluated for this)
Each of these has to have N-way redundancy where N is the # of sites we have, with each instance being capable of continuous operation even if it finds itself on its own network island for a while. Conflict resolution / merging when the network heals is a must.
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
Hey Bart, I'm new here, but I figure now's as good a chance as any to contribute. I have quite a bit of experience with distributed systems, and I'd love to help. Let's talk reconciliation semantics! On Tue, Feb 28, 2017, at 01:19 PM, Bart Kus wrote:
Well, I can follow up with Subhro off-list. I have more detailed questions than that. Was just looking for volunteers to help out with design.
--Bart
On 2/28/2017 1:18 PM, Nigel Vander Houwen wrote:
We’re looking for recommendations on what to use, and how to set it up. As Bart mentioned, the critical aspects of this are N-way redundancy so that any given fragment of the network if it gets separated can still operate, and then graceful handling of re-syncing after the network rejoins.
Nigel
On Feb 28, 2017, at 13:15, Subhro Kar <subhro@80386.org> wrote:
I have some experience with distributed file systems. What do you need to know? Subhro
Sent from my iPhone
On Feb 28, 2017, at 12:52, Bart Kus <me@bartk.us> wrote:
Hello,
Lately I've been working on network automation stuff for HamWAN, and I'm at the point where storage systems have to be dealt with. I'm wondering if anyone in the group has knowledge to provide some design feedback in distributed storage systems. Not from a "how to use it in AWS" perspective, but rather how to design them ground-up.
Ideally we want a couple abstractions available network-wide:
1) A global DB (NoSQL should be fine, and Cassandra is looking possible for this) 2) A global FS (GlusterFS is in very early stages of being evaluated for this)
Each of these has to have N-way redundancy where N is the # of sites we have, with each instance being capable of continuous operation even if it finds itself on its own network island for a while. Conflict resolution / merging when the network heals is a must.
--Bart
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
_______________________________________________ PSDR mailing list PSDR@hamwan.org http://mail.hamwan.net/mailman/listinfo/psdr
participants (4)
-
Bart Kus -
Eric Wolak -
Nigel Vander Houwen -
Subhro Kar