• Forum has been upgraded, all links, images, etc are as they were. Please see Official Announcements for more information

Say no to three backup servers

GrandMasterDash

GrandMasterDash

Well-known member
Masternode Owner/Operator
If I understand correctly, the new Dash Drive will create a decentralized sharded database to help manage user accounts. There is a 5:1 redundancy in the case of failure but to be sure there are to be three large backup servers. It concerns me that someone feels, in the event of disaster, just three servers will be suffice to save the day. In my mind, it demonstrates a lack of faith in the current design and, therefore, a possible weakness in the event of a catastrophic event.

Keep in mind, Evolution is being designed for mass adoption. Any significant loss of data could wipe out the value of dash overnight. We therefore need a system that is incredibly robust. Something we can test and simulate to extremes, and show the system is capable of surviving.
 
I would have to agree. If it is all about anonymity then anyone having a full copy puts people at risk. encrypted or not.

BTW If anyone can host a drive share count me in.
 
I will not be so frightened. We will soon have IPFS, Storj. and more choice. But the choice of redundancy is important question. We should definitely not talk about it in this speculative way. We need to see some mathematical model with statistical probabilities behind it.
 
The thing is that the information will still be encrypted and sharded, totally unreadable. And anyone who wants to, would be able to grab a full copy of everything on the network if they wanted to anyway, They could remove duplicate information and sit down with their massive hard drives and enjoy the gobbledy gook at their leisure. But they won't have anything of interest. So although the 3 backups sound like a weak point, they're just a full version of what is already out there in a reundundancy. It's just 2 types of redundancy for infinitesimally small likelihood that ANY data is ever lost. These would be expensive servers kept in at least 3 different geographical locations around the world, well maintained and eventually probably protected. Not against data theft but against data destruction. Evan has seen Mr. Robot, LOL.
 
TanteStefana said:
The thing is that the information will still be encrypted and sharded, totally unreadable. And anyone who wants to, would be able to grab a full copy of everything on the network if they wanted to anyway, They could remove duplicate information and sit down with their massive hard drives and enjoy the gobbledy gook at their leisure. But they won't have anything of interest. So although the 3 backups sound like a weak point, they're just a full version of what is already out there in a reundundancy. It's just 2 types of redundancy for infinitesimally small likelihood that ANY data is ever lost. These would be expensive servers kept in at least 3 different geographical locations around the world, well maintained and eventually probably protected. Not against data theft but against data destruction. Evan has seen Mr. Robot, LOL.
Click to expand...

I'm sorry but I fundamentally disagree. Having these three backups is a clear statement that they might actually be required one day.

I want to see wargame simulations carried out that demonstrate how dash can survive even when these three backups are wiped (however dispersed they are).

How confident am I meant to be if the core team doesn't have 100% confidence themselves?

Maybe the data redundancy should adapt to the number of active masternodes? - as the masternode count goes down, the redundancy requirements automatically adjust upwards.
 
Yes, I can see that aspect of Dash to be looked upon badly, similar to the reference node. A point of centralization in a decentralized system. There is no place for it, and it will bring us criticism. I for one am willing to wait longer for this to be eliminated in a robust, tested system. With 3500+ Masternodes all storing data, there really should be no need for backups. They should be their own backups.

GrandMasterDash A data redundancy algorithm based on number of Masternodes seems to be a great idea, although no idea how hard it would be to implement. While, to be honest with you, I don't think the number of Masternodes will ever go down significantly, it's better to prepare for the possibility.

I don't like the backup server idea either.
 
Last edited by a moderator:
We just lost 120 masternodes because people wanted to play the market. 120/3500 = 3.4% of our nodes. The question is, what are the chances that all 5 pieces of a single bit of information is stored on one of these stopped nodes? What if for some reason 1/2 of all the nodes are cashed out one day? What are the chances of losing forever a shard of information? How many duplicates do we need? Is taking any chance of losing a shard of information even OK when you're talking about a person's life savings? This isn't a matter of centralization. it's simply a form of redundant back up. In fact, it's actually more decentralized with these full backup storage units because it's decentralizing the type of backup. If we lose all of the full storage, it's not likely to happen the same time that we lose 100 or 1000 Masternodes and vise-versa. In fact, maybe in the future there will be another way to store vast amounts of Data for emergency backup and if we were to add that type, we would have even more decentralization of type of storage.

It's like you all want to throw the baby out with the bath water. It's silly. This particular service also has absolutely nothing to do with the need for trustless mining and processing of transactions. Hence "decentralizing" the storage doesn't mean anything. The only reason for sharded storage is so that our storage system can expand exponentially without forcing all masternodes to run petabytes of storage, which would be extremely expensive and far too redundant and wasteful. You're focusing on the wrong technology for this particular problem. Just because trolls can make this sound bad to ignorant ears, doesn't mean we should abandon good, quality design that can protect our network and make it impenetrable to attack.
 
Last edited by a moderator:
TaoOfSatoshi said:
Yes, I can see that aspect of Dash to be looked upon badly, similar to the reference node. A point of centralization in a decentralized system. There is no place for it, and it will bring us criticism. I for one am willing to wait longer for this to be eliminated in a robust, tested system. With 3500+ Masternodes all storing data, there really should be no need for backups. They should be their own backups.

GrandMasterDash A data redundancy algorithm based on number of Masternodes seems to be a great idea, although no idea how hard it would be to implement. While, to be honest with you, I don't think the number of Masternodes will ever go down significantly, it's better to prepare for the possibility.

I don't like the backup server idea either.
Click to expand...
We may have 3500+ masternodes but there may not be 3500+ copies of the data because there are thin provisioning and deduplication in storage. So, I think it makes sense to require every masternode to have a copy but make it possible to do dedup or share storages behind the masternodes one owns. So, the real minimum number of copies could be the number of MN operators.

EDIT: For example, if the masternodes use Kinetic Open Storage API to access Kinetic HDDs behind them operated by a single operator, the operator can have tens of MNs putting/getting three copies of data stored in tens or hundreds of HDDs. When one drive fails, the operator just pull that drive out and resilver the new drive. No one single MN will come down due to a failing HDD.
 
Last edited by a moderator:
TanteStefana said:
It's like you all want to throw the baby out with the bath water. It's silly. This particular service also has absolutely nothing to do with the need for trustless mining and processing of transactions. Hence "decentralizing" the storage doesn't mean anything. The only reason for sharded storage is so that our storage system can expand exponentially without forcing all masternodes to run petabytes of storage, which would be extremely expensive and far too redundant and wasteful. You're focusing on the wrong technology for this particular problem. Just because trolls can make this sound bad to ignorant ears, doesn't mean we should abandon good, quality design that can protect our network and make it impenetrable to attack.
Click to expand...


You won't be saying that when a small bunch of people lose their money and three backups fail, because a fail to the minority could trigger a mass exodus, causing the price of dash to collapse and MN operators to sell up while they can. Then what?
 
I think you don't understand, we will have two types of backups. One type holds everything, it'll be massive and expensive, but the blockchain will pay to maintain a few of them, number unknown. The second type is the sharded network. The sharded network, I don't have any calculations on how possible it is that all versions of a shard of information might go off line, but if this should happen, the backup servers could be tapped to replace them. The missing files would be reloaded to other Masternodes. This would also happen any time fewer than X number of Masternodes are holding a copy of a shard. Masternodes will pick up the slack either by pulling info from the Masternode network, or if for some reason, all copies have disappeared, they'd pull it off the backup servers.

If what you're saying is that you don't want any of this, no sharded Dash Drive let alone backup servers, then yes, you would never have any lost data. I'm arguing that if we have a sharded Dash Drive, that having a second system in place that holds a full copy of everything, duplicated a couple of times would be prudent.
 
I'd be happy when someone can show just how robust it is. It needs to be bomb proof and it needs to dynamically adjust to the size and efficiency of the network.

I'm sorry, I don't want to sound mean or ungrateful, I just think it needs pressing home how important the loss of data could be. The level of security needs to be something we can be truly proud of... if that means MN operators have to carry a bit more load and responsibility then so be it.
 
I agree, there is something weird about these 3 backup drives. Does that mean that someone has special permissions to upload stored encrypted user data back to the Dash drive, or any MN can do it? How would this backup drive nodes be selected? Are they to be located in the Dash headquarters in Seizemeville?
 
It's not a security issue, it's a "movement" issue. People are free to shut their Masternodes down at any time, so what happens to their share of the storage? The slack has to be picked up by others. But then there is that chance, whatever it is, that so many servers are taken off line at once, that some shards of information go down with them (being that all the redundancies were on those servers) In that case, the backups kick in and provide the backup to "fill" the Masternode network where it's missing information. This is probably not likely to even happen at all, but it's there as a backup.

All the Dash Drive is is a database. It has columns and rows. Lets say row 3706 is your row, then 3706A is your user name, 3706B is your password, 3706C is your deterministic seed, 3706D is your email.....etc All of these cells are encrypted with 256 bit encryption and distributed redundantly among the Masternodes IE "Sharded" The only purpose to sharding is to save hard drive space. Without it, everyone would have to hold a copy of the entire database, wasting hundreds or thousands of petabytes. Which is wasted space and wasted money. This has absolutely NOTHING to do with anonymity or privacy, only storage.

I mean, GrandMasterDash , your first sentence: "If I understand correctly, the new Dash Drive will create a decentralized sharded database to help manage user accounts." is wrong. It's NOT a decentralized database, it's a sharded database, meant to have each masternode only carry a small part of the load. That's it.

Having full backup servers does not create a weak link in any way shape or form, instead it is another layer of security against loss.
 
TanteStefana said:
It's not a security issue, it's a "movement" issue. People are free to shut their Masternodes down at any time, so what happens to their share of the storage? The slack has to be picked up by others. But then there is that chance, whatever it is, that so many servers are taken off line at once, that some shards of information go down with them (being that all the redundancies were on those servers) In that case, the backups kick in and provide the backup to "fill" the Masternode network where it's missing information. This is probably not likely to even happen at all, but it's there as a backup.

All the Dash Drive is is a database. It has columns and rows. Lets say row 3706 is your row, then 3706A is your user name, 3706B is your password, 3706C is your deterministic seed, 3706D is your email.....etc All of these cells are encrypted with 256 bit encryption and distributed redundantly among the Masternodes IE "Sharded" The only purpose to sharding is to save hard drive space. Without it, everyone would have to hold a copy of the entire database, wasting hundreds or thousands of petabytes. Which is wasted space and wasted money. This has absolutely NOTHING to do with anonymity or privacy, only storage.

I mean, GrandMasterDash , your first sentence: "If I understand correctly, the new Dash Drive will create a decentralized sharded database to help manage user accounts." is wrong. It's NOT a decentralized database, it's a sharded database, meant to have each masternode only carry a small part of the load. That's it.

Having full backup servers does not create a weak link in any way shape or form, instead it is another layer of security against loss.
Click to expand...


That's distributed.. may not be complete copies everywhere like the blockchain, but it is indeed distributed i.e. everything is not in one place.

And when I say "security" I mean the "backup" meaning of security, not the hack meaning of security.

Whichever way you look at it, it is felt that three full copies might possibly be needed one day even if "not likely to even happen at all". It's basically the last mile.

Okay, I admit, this is not an easy problem to resolve but I have a thought....

Can we vary the share of redundancy per MN? Each MN declaring storage allocation and other MNs randomly verifying the integrity. We could then vary the MN reward against overall network redundancy. As the network redundancy increases the reward gets smaller. And, of course, as the network redundancy decreases, the reward goes up. At least this way, the last mile is not predetermined.
 
I'm sorry GrandMasterDash , I lost you? I'll keep trying :)

But I did want to point out that the 5 redundancies is just a number Evan chose. We could do 100 redundancies. This would dercease the amount of total storage but then again, we may not need so much. Also, when/if the price goes up really high, it could easily be paid for. What we need is statistical probabilities calculated, and when they're infinitesimally tiny, PLUS the backup servers, I think that would be just fine. Even DarkSend isn't 100%, and probably nothing in life is other than 100% probability that we're all going to die someday :D
 
TanteStefana said:
I'm sorry GrandMasterDash , I lost you? I'll keep trying :)

But I did want to point out that the 5 redundancies is just a number Evan chose. We could do 100 redundancies. This would dercease the amount of total storage but then again, we may not need so much. Also, when/if the price goes up really high, it could easily be paid for. What we need is statistical probabilities calculated, and when they're infinitesimally tiny, PLUS the backup servers, I think that would be just fine. Even DarkSend isn't 100%, and probably nothing in life is other than 100% probability that we're all going to die someday :D
Click to expand...

I agree, some stats (and war games) would be good... I'm just very uncomfortable with a predetermined endgame.
 
I'm obviously missing something. It hasn't clicked yet. What is the predetermined endgame please? I'm not being a butt, hubby often thinks I'm messing with him, but I'm just having a neurological blockage :what:
 
TanteStefana said:
I'm obviously missing something. It hasn't clicked yet. What is the predetermined endgame please? I'm not being a butt, hubby often thinks I'm messing with him, but I'm just having a neurological blockage :what:
Click to expand...

I thought the plan was to use widely spread designated servers. It is decided now and doesn't adapt to rapidly changing circumstances. It just seemed to me, if the load and redundancy was dynamic, the number of servers required and their location would not be known until disaster struck.. at which point the system would automatically adapt.
 
Well, I understand that the masternodes all carry a piece of the database. Each piece of data (which is encrypted and gobbelty gook to anyone who doesn't have the keys) is stored on at least X number of Masternodes. So if a masternode goes down, that info is still available on another one. The only reason for huge full database backup servers is in case of the teeny tiny chance that a whole shard of information is taken offline. These full database servers could then supply the missing shard. It should never be required, and yet it would be there just in case. If we had each shard copied to 100 masternodes, it would be seriously impossible that any shard could have been completely stored on all 100 of those masternodes that went offline. But if that crazy situation should happen, then the backup machines would re-supply it.

Frankly, if the network goes down, the world has entered a new dark age, and it won't matter anymore. We'll run out of bullets before you know it, and will be fighting with the butt end of our rifles :)

And yet, even with a huge number like 100 redundancies, the network would only need 1/35th the allotted Hard Drive space, saving resources. Which is the whole point.

10 would undoubtedly be more than sufficient, allowing for the network to run at 1/350th the size of the data needed to be stored.

So for 1 petabyte of storage, each MN can get away with 3 Terabyte hard drives. And these numbers are for the future when almost everyone uses Dash, and you can see that that number is doable for each masternode.
 
Last edited by a moderator:
I'm obviously too paranoid :p because I think disasters / governments / hackers can strike 10 servers a lot easier than 3500. Personally, I wouldn't take those teeny tiny chance with so much money at stake.

If I had the resources, could I keep my own copy of a full backups?
 
You must log in or register to reply here.
Back
Top