Proposal: Infrastructure - Basic Needs

[Ryan Taylor]

This is a cross post from www.dashwhale.org/p/infra-basic-needs

This proposal is meant to finally address the minimum ongoing infrastructure requirements of the Dash project. Coming out of our recent review of our needs, infrastructure was one area that clearly required some investment. Critical Dash assets are being hosted on private servers, donated servers, etc. In addition, the provider of our Bamboo Cloud – which we use to compile all versions of Dash – announced on May 24th that the product will be discontinued. Needless to say, as the project grows and matures, our infrastructure needs are changing.

In an ideal scenario – and hopefully in the long-run – we would like to host Dash infrastructure on dedicated equipment, but that would be cost-prohibitive at our current scale. However, there is much that can be done to improve our infrastructure without a great deal of investment.

First, we need to replace the functionality of the discontinued cloud Bamboo service. After evaluating our options, we propose purchasing our own hardware. By owning our own server, we can compile new versions much more efficiently, which accelerates our automated testing (continuous integration). This will make all of our development teams more efficient, especially during the testing phases of each release. As the development efforts ramp up the second half of this year, having a dedicated Bamboo server is becoming only more critical, and will address a pain-point for our development efforts.

For other infrastructure needs, we feel that we can continue to operate using cloud-based infrastructure, which has lower upfront costs, more flexibility to add or remove capacity, and yet is still hosted in a professionally managed datacenter. It would also ensure that these assets are under the control of the project, as opposed to individuals who are presently donating capacity.

A special thanks to @moocowmoo for leading this effort and taking on our infrastructure planning and management.

Here are the details on the hardware and services we propose as part of our minimum ongoing infrastructure needs. Please note that when 12.1 is deployed a separate proposal will be needed to fund the ongoing costs. This proposal covers only one-time costs and first month’s expenses.

Compile/Deploy/Backup server
Budget:
$3000 for hardware (one-time cost)
$150 per month for co-location and replacement parts
Server’s roles:
- Continuous Integration (Bamboo slave)
- Private Docker Registry (generate/deploy production images)
- Online Incremental Backup (first tier hourly backup)
Specifications:
(subject to change according to available deals at the time of purchase and USD available based on the Dash exchange rate when paid)
1U enclosure, redundant power supply
1x Motherboard (TBD), 2x 1Gb NIC
2x 8-core AMD Opteron
128G RAM
2x 1TB SSD
2x 6TB HDD

Cloud infrastructure costs
Budget:
$250 per month for AWS and Google services and administration
Roles:
- AWS : Route53 DNS zone management
- AWS : EC2 web server instances
- AWS : S3 for web server instances
- AWS : Backups and auditing
- Google: Gmail for domains

Requested funding is as follows for the July 6th budget cycle:

• 362.67 Dash for Bamboo server ($3,000.00 USD @ $8.272 per Dash based on June 18th average rate at https://bitinfocharts.com/comparison/price-dash.html)
• 18.13 Dash for first month’s Bamboo server colocation hosting and parts ($150 USD)
• 30.22 Dash for first month’s cloud infrastructure ($250 USD)
• 5.00 Dash reimbursement for the proposal cost

Total: 416.02 Dash

Note: Should any funding remain after the project is complete, we will apply those funds toward future infrastructure funding needs

Manually vote YES on this proposal:
dash-cli mnbudget vote-many 6a67175f7a47b09ebd1852e1b6a84ade77e475a818a90e577a074c694a4efaa1 yes
OR from the qt console:
mnbudget vote-many 6a67175f7a47b09ebd1852e1b6a84ade77e475a818a90e577a074c694a4efaa1 yes


Manually vote NO on this proposal:
dash-cli mnbudget vote-many 6a67175f7a47b09ebd1852e1b6a84ade77e475a818a90e577a074c694a4efaa1 no
OR from the qt console:
mnbudget vote-many 6a67175f7a47b09ebd1852e1b6a84ade77e475a818a90e577a074c694a4efaa1 no

 

[fible1]

Where will the server be hosted? Who will be carrying out the build?

Thanks,
Pablo.

 

[Ryan Taylor]

@moocowmoo Can you please answer @fible1's questions?

 

[fible1]

I voted yes, I'm happy knowing that @moocowmoo is doing it.

My main concern is that we should keep our startup spirit and be as frugal as possible, at least until we breach the billion market cap, and perhaps beyond, regardless of available funding. There have been a couple of proposals come forward recently. requesting funding for small stuff the community used to do for free, such as community management, and now server co-allocation. We are only talking a few hundred bucks, but once we stop being frugal and stop thinking like a young startup, we will start making unnecessary expenses in larger ticket items. I would rather we have funding left over every month than spending a chunk of it on stuff we have always donated to the community, be it our time or hosting space.

Hope that makes sense,
Pablo.

 

[Balych]

[offtopic]

- Google: Gmail for domains

Why not Tutanota/Protonmail/etc?
[/offtopic]

 

[balu]

Hey,

While I agree about the topic of the proposal, I'd like to recommend an alternative solution to the colocated server.
Since we're trying to act like a startup, we should totally go with renting a server instead of buying one.This would cost less, is better cash-flow wise (no initial investment), would require no hw maintenance (less burden on our core people), and will enable us to upgrade later way easier.

The things we listed above do not justify a 2-core colocated server with 3k initial investment. I don't want to link exact offers, but compared to a very reputable host like LeaseWeb, what advantage would we get with the solution above? Please compare and evaluate. I think the requirements above could be handled by a 70-80 USD/month server without any upfront cost, and later if required we could upgrade to a dual-cpu one still within 150 usd/month (and without the initial 3k investment)

Thanks for the consideration and answer.

 

[crowning]

- Google: Gmail for domains

I remember the discussion over at Slack, but it's not there any more: what was the reason why we don't run our own mail server?

 

[TroyDASH]

I voted yes, I'm happy knowing that @moocowmoo is doing it.

My main concern is that we should keep our startup spirit and be as frugal as possible, at least until we breach the billion market cap, and perhaps beyond, regardless of available funding. There have been a couple of proposals come forward recently. requesting funding for small stuff the community used to do for free, such as community management, and now server co-allocation. We are only talking a few hundred bucks, but once we stop being frugal and stop thinking like a young startup, we will start making unnecessary expenses in larger ticket items. I would rather we have funding left over every month than spending a chunk of it on stuff we have always donated to the community, be it our time or hosting space.

Hope that makes sense,
Pablo.

You make a good point. It can be difficult to judge whether the time and resources someone is putting into the project should be considered a hobby/volunteer work, vs. when it starts meriting a paid job. In the end I think it should still come down to a cost benefit analysis. What tangible things would happen or not happen depending on whether the proposal gets funded? Will the project receive a marginal benefit that is greater than the marginal cost, and greater than the net marginal benefit of other competing proposals? If the answer is yes, then that's how I vote. Proposal owners would do well to attempt to help answer those questions as well without MNs having to guess or pry it out.

 

[n00bkid]

If a new development team ever gets voted in, what happens with the $3k hardware?

I agree with some others that it would be preferable to keep these operations on a host. That way these operations can be kept flexible in terms of who runs them.

 

[aka5]

If a new development team ever gets voted in, what happens with the $3k hardware?

I agree with some others that it would be preferable to keep these operations on a host. That way these operations can be kept flexible in terms of who runs them.

I totally agree with that, in my eyes renting a server is much better than buying one - for cheap high end ones look hetzner.de for example ... the problem i see with buying one is, who is the owner? it's hard today to let the blockchain own the server, so someone has to be the strawman, but that's not a good solution - renting would be much better !

 

[GrandMasterDash]

I totally don't get this proposal, imo it's ridiculous. We have 3800 servers and you want us to rent out someone else's resources... eeewwww. I think we should be eating our own dog food.

 

[moocowmoo]

Where will the server be hosted? Who will be carrying out the build?

We haven't decided yet, but either I or @flare will be performing the build and co-locating the server near whomever builds it.

I voted yes, I'm happy knowing that @moocowmoo is doing it.

Thank you for saying that, I appreciate it!

My main concern is that we should keep our startup spirit and be as frugal as possible

That is the goal in mind. I've scratched out plans for our billion-cap dream infrastructure, but would like a single, powerful box to handle most of the backend administrative processing in the mean time.

we should totally go with renting a server instead of buying one.This would cost less, is better cash-flow wise (no initial investment), would require no hw maintenance (less burden on our core people), and will enable us to upgrade later way easier.

I considered dedicated packages, but prefer the isolation that purchasing our own hardware affords.
One of my goals in building the new infrastructure is high security with as little exposure to logical and social attack vectors as possible.
I lean a bit extreme on the paranoid side, but it's served me well over the years.

My intent with this first system is to build a secure base that compiles and deploys all artifacts and services to cloud instances using ipfs, docker, and rancher.
With so much control in one system, it will only be accessible over a vpn, probably cjdns and only by a few community-trusted people. @eduffield, @flare, @moocowmoo, and maybe a few others that need it. (For Evolution for example)

As far as upgrades go, these specifications will suffice as a build system for years and years to come. I expect builds to take under five minutes per platform serially, with the capability to run several gitian builds in parallel, probably 3 at a time before any resource contention. This will allow us to go from commit to test suite completion to development version installation (automated mass testnet deploy) within a few minutes.
I've spec'd this box fast and powerful to supercharge development cycles.
I've also spec'd it huge (drive space) to be the first level of proper incremental backups. Not to mention docker images aren't tiny at the repository-mirror level.

Why not Tutanota/Protonmail/etc?

I like protonmail, we may end up there.
But for now we've chosen google to unify administration of all the google accounts for google docs (which we use extensively).

what was the reason why we don't run our own mail server

Because it's easier to farm it out for now.
Running postfix/dovecot/spamassasin is easy enough, but providing a web interface isn't a priority at the moment.
As far as third-party exposure goes, any communications that are sensitive are protected by gpg.
Additionally we'd need several ip's for PTR records to guarantee 100% delivery for our multiple domains. Forwarding it solves this issue for the time being.

If a new development team ever gets voted in, what happens with the $3k hardware?

I consider this server to be a step toward unifying the administration of the entire dash ecosystem.
If a new development team were to be voted in, I'd hope they'd want to write code and leave the administration to someone already doing it. (Assuming I'm doing a good job, hehe.)


---

This path forward is how I've chosen to evolve our administration from a single openvz vps to something dynamic and manageable.
Owning our own hardware guarantees that we always have a stable, secure base to manage all of dashs services and needs.
Everything else will be reproducible, deterministic cloud deploys.

I look forward to building us all a spectacular, reliable system!

 

[fible1]

Awesome, I knew there was some good reasoning behind this choice .

Pablo.

 

[rustycase]

Small disagreement...
The server does not need to be near it's builder, it needs to be on the absolutely best, top of the heap, fiber optic trunk line.

And, even though I favor the red team, (for my level), Intel makes superior processors, yet unfortunately commands the highest prices for them.
(Perhaps someone has done specific detailed comparisons ?) { Xeon E3-1230 v5 }

...Conducting all business from one box does not seem to be optimal security, imo...

just a few thoughts I had... Newbie kinda thoughts...
rc

 

[moocowmoo]

Small disagreement...
The server does not need to be near it's builder, it needs to be on the absolutely best, top of the heap, fiber optic trunk line.

It is important the server is accessible when physical maintenance is needed.
The bandwidth is not as critical as you think, as distributing binaries to the public is not its role. That's what cloud instances are for.

And, even though I favor the red team, (for my level), Intel makes superior processors, yet unfortunately commands the highest prices for them.
(Perhaps someone has done specific detailed comparisons ?)

AMD was chosen just because of the Intel premium you mention.

...Conducting all business from one box does not seem to be optimal security, imo...

Optimal security is having the smallest possible attack footprint. This server will never serve content to the general public.
Optimal redundancy is another matter, and I'm making plans to have a few personal servers as hot-spare temporary failovers should the need arise while we drive to the colo.
When the infrastructure and budget is more established, these will be dedicated co-located failovers.
But, keep in mind, this system does not have to be online for our services to be. It is simply an administrative and content generation backend.

 

[rustycase]

Well... OKay !
Seems you are right on top of things and my thoughts had already been considered !

Best
rc

 

[nmarley]

Having just a single server just screams SPOF to me. Other than that, I think @moocowmoo 's plan is solid.

 

[moocowmoo]

Having just a single server just screams SPOF to me. Other than that, I think @moocowmoo 's plan is solid.

Agreed, but I want to start small. All of the server roles have second-tier personal backups.

Server downtime will only prevent docker instance upgrades and the to-be-built testnet deployment harnessing.
Annoying in the middle of a testnet deploy and test iteration, but not critical.

Compiles will happen here on my 4GHz 8 core at home as a bamboo secondary slave. Maybe other volunteers' as well.
Backups will either be an emergency cloud instance image ready to be spun up. (It won't need to be ginormous, just fill in the gap while the server is repaired) or fail over to the second tier volunteer backup server(s). I've not settled on a backup plan yet.

 

[balu]

@moocowmoo I agree with you that we need such a system, and the use cases above I can totally understand.

My only concern is the costs. Calculating with a 3 year TCO this server would cost 8400 USD, while renting an arguably lower spec, but still good enough server from e.g. Leaseweb is ~3000 USD. That is almost 3* the costs, and even if we added a separate backup server we'd still be at half TCO.
And on top of this we'd have lower risk and time investment because of buying a service instead of hardware.

All the technical details are fine as you wrote them down and since I find such an environment highly beneficial I'm really glad that you took this on you, it's just about the money.

 

[moocowmoo]

My only concern is the costs. Calculating with a 3 year TCO this server would cost 8400 USD, while renting an arguably lower spec, but still good enough server from e.g. Leaseweb is ~3000 USD.

Well, I'm willing to consider other options, but the "good enough" comparison you're making isn't an equivalent comparison.

Speed is the goal. There is no way we can rent a server with the specifications I'm looking for.

I'm still rearranging items to fit under budget (would rather have $6000 for a 4 socket, 48 core machine), but the current specs look like this:

16 cores at 2.8 GHz -- we need massive threading to compile all the PC/Linux/Mac flavors quickly.
64G Ram -- gitian deterministic builds need lots of room. I want more, but am having to compromise.
2TB SSD -- Lots of very fast I/O for above. 1TB OS/Userland (low wear), 1TB compile (high wear) - the Samsung 850 Pro's I'm looking at are rated very well and are a fantastic value for the money. -- I'll probably shrink these in lieu of ram after gitian profiling.
12TB Disk -- Lots of space for backing up every service we have and storing every compiled artifact for ipfs (internal) distribution. Am looking at WD Gold 6TB's.

I do agree we could do it cheaper, but only by compromising attributes I wish to build upon.