6/20 RC3 Post-mortem

[DRKLord]

Hey everyone,

Pleased to be here and be part of the dev team! Been wanting to get on board for a while, and yesterday's network difficulties finally made it official. Like many of you, I'm sitting on a little pile of DRK of my own and want to do anything I can to make our coin flourish and prosper.

IMO, the market wildly over-reacted yesterday. People were panicking and trying to dump DRK. I saw some people exclaiming "Oh no! <insert exchange> disabled deposit so I can't sell my <less than $100 worth> DRK!". You would've thought the sky was falling. And yet what we saw yesterday was actually progress. We experienced a glitch and some minor forking, we know what caused it and we're going to fix it. That being said, I think Red-Shionobi does have a point. We should probably change the way we deal with major updates and rebrand them as something other than an "epic masternode launch". When the public is expecting an "epic masternode launch" and there is a bug, it's viewed as a failure by the dev team rather than progress in developing a brand-new, cutting-edge technology. And then we see panics and flash crashes like yesterday. That being said, I think just renaming it something like a "public network test" would work wonders in the PR and investor relations departments...

Anyway... Preparing myself for a marathon session to dig deep into the Darkcoin source. Got a lot of catching up to do and a lot of old rust to scrape off...

@ Evan:
Sorry I passed out on you yesterday, bro! I had been awake over 24hrs and after I ate dinner it was lights out, haha

Regards,

The DRK Lord

 

[eltito]

Red-Shionobi has a point.
Why not reconsider the way that updates are released.
Why not deliver to mainnet more frequently with just devs and testers up-to-date with the network at first and eventually the secret is out. So what if it's hard on the network.
It's called continuous-integration. If something breaks, then simply revert. It's okay.

Respectfully, no, it's not ok. There is ~$50M of other people's money involved here, not including the business models built around the coin (e.g. exchange fees). It would be grossly irresponsible to be anything other than as clear and upfront as possible about what is going on with the network now and what will be going on with the network in the near future.

 

[eltito]

That being said, I think Red-Shionobi does have a point. We should probably change the way we deal with major updates and rebrand them as something other than an "epic masternode launch". When the public is expecting an "epic masternode launch" and there is a bug, it's viewed as a failure by the dev team rather than progress in developing a brand-new, cutting-edge technology. And then we see panics and flash crashes like yesterday. That being said, I think just renaming it something like a "public network test" would work wonders in the PR and investor relations departments...

To my knowledge, it was never billed (by us) as "epic masternode launch," though I certainly understand that it took on that level of hype on its own. This is exactly what I meant earlier when I said there is room for improvement in how we manage the expectations we set.

That said, a slick rebranding of hard fork launches, IMHO, would be quite meaningless. Call it whatever you want, it's still the same thing and everyone knows it - and the effect will be exactly the same if it fails. Potentially worse, as a matter of fact, because now we've given ammunition to antagonists to allow them to say we were intentionally misleading people about the seriousness of what we were doing by trying to call it something other than what it really was.

The better course of action is to give a more thorough accounting of what should happen, what may happen, and what actions we will take if something goes wrong. Events cause panic when they are unexpected, not necessarily just because they happen. If you tell people "if we notice a problem, here is what we will do" and the DO what you said you will do, nobody panics.

 

[derk]

The better course of action is to give a more thorough accounting of what should happen, what may happen, and what actions we will take if something goes wrong.

Yes please! Will someone draft a few lines doing this?

 

[DRKLord]

@ eltito:
While I'm forced to agree with you in many ways, what I'm suggesting is that we tread more carefully when dealing with the public concerning major changes to the network. Chosing the right words is vital. We're a bunch of geeks and programmers, not PR or investor relations specialists, not CEOs or CFOs and not advertising managers. But we have to start thinking about the way we communicate with the userbase to avoid causing unnecessary turmoil in the markets.

I would never suggest lying to the public and our users or hiding what's really going on. But we should take steps to avoid "hyping up" a release or modification to the network...

EDIT:
I'm not saying anyone intentionally "hyped up" the masternode launch or did any such thing. Just saying we see how wild people can get about this, so let's try to keep things calmer in the future...

 

[DRKnite]

Hi all. I usually lurk here, but I registered to make this post.

Being highly interested in Darkcoin myself and concerned for its future, I agree with many of the points in this thread. It is understandable that software is complicated and sometimes, issues will arise despite extensive testing and every precaution in the world. With cryptocurrency, these become very public and have a real, immediate financial effect on everyone involved. That being said, I think the spotty communication with the general public during yesterday's release had the biggest negative impact on the perception of the coin, allowing anyone with a vested interest in competing crypto to spread copious FUD on every chat and forum.

This has to improve if we are to build up an image of a bulletproof coin that is safe and stable. Further, I also agree that developer testing / troubleshooting talk should be moved out of a public forum.

If the team is at all interested, I'd be happy to contribute to provide timely updates and dispel any misconceptions on public forums. What about putting up a page, say status.darkcoin.io, that lists some key metrics of the network and, more importantly, has fast updates regarding new releases, forking, and any other issues that may come up? With this available to anyone who is concerned with the current status, they can easily see that the development team is on top of maintaining stability and will work to resolve any problems immediately. It would be a single point of communication with John Q. Dark, and he no longer has to venture into IRC or dig through the FUD to try and figure out what the hell is going on. I think it'd give off a more professional image and just maybe prevent the mass hysteria panic dumps that we saw yesterday.

Give it some thought. Let's work together to make Darkcoin the next generation crypto of choice by polishing its image in the view of the general public, and, needless to say, ongoing development to iron out the kinks.

 

[tungfa]

DRKLord
Chosing the right words is vital. We're a bunch of geeks and programmers, not PR or investor relations specialists, not CEOs or CFOs and not advertising managers. But we have to start thinking about the way we communicate with the userbase to avoid causing unnecessary turmoil in the markets.
------
exactly thank you !!!
and the "hype" come from the market (which is a good thing) as it is exciting times, we are in the eye of the market, exactly where we want to be !!!
so we gotta make the best out of it and use it for our advantage !!!
chain of comand ... PR setup .... team work !
....>>>
(you guys can program in your garage, and a team out here is dealing with the real world and media, but there has to be a working connection between that garage and here !!)

 

[TanteStefana]

Hi all. I usually lurk here, but I registered to make this post.

Being highly interested in Darkcoin myself and concerned for its future, I agree with many of the points in this thread. It is understandable that software is complicated and sometimes, issues will arise despite extensive testing and every precaution in the world. With cryptocurrency, these become very public and have a real, immediate financial effect on everyone involved. That being said, I think the spotty communication with the general public during yesterday's release had the biggest negative impact on the perception of the coin, allowing anyone with a vested interest in competing crypto to spread copious FUD on every chat and forum.

This has to improve if we are to build up an image of a bulletproof coin that is safe and stable. Further, I also agree that developer testing / troubleshooting talk should be moved out of a public forum.

If the team is at all interested, I'd be happy to contribute to provide timely updates and dispel any misconceptions on public forums. What about putting up a page, say status.darkcoin.io, that lists some key metrics of the network and, more importantly, has fast updates regarding new releases, forking, and any other issues that may come up? With this available to anyone who is concerned with the current status, they can easily see that the development team is on top of maintaining stability and will work to resolve any problems immediately. It would be a single point of communication with John Q. Dark, and he no longer has to venture into IRC or dig through the FUD to try and figure out what the hell is going on. I think it'd give off a more professional image and just maybe prevent the mass hysteria panic dumps that we saw yesterday.

Give it some thought. Let's work together to make Darkcoin the next generation crypto of choice by polishing its image in the view of the general public, and, needless to say, ongoing development to iron out the kinks.

This is so true, and I feel so guilty! I made an appointment to help my family friends, deep friends, like family, yesterday before the hard fork was announced and I couldn't back out, but then it ended up being an all day affair!

Otherwise, I love "chatting" on the forums and should have been there to answer questions. It was actually the plan, and I was hoping I would be back home within a couple of hours, but that didn't happen. Around 2:30, I took a peek at the forum, saw something went wrong, ugh, and had to get back to work. I don't remember my password, it's written on a card under my keyboard, LOL. So I really should have been there, and I'm sorry I wasn't.

Next time I will be, but next time, this is going to work.

 

[DarkWater]

Just wanted to say despite all the hardfork issues, I still have the most confidence that DRK will come on top and stay there. Crypt was literally destroyed today, and Cloakcoin looks like a rip off of DRK and is doing jack squat. If anything im just buying up more coins now so when the masternode system is live I can jump in! But I do think there should be a softening of the testing phase till we know that everything is live and up and running.

With the competition being eliminated DRK still has a chance.

 

[MykelSIlver]

This may have been already mentioned in the forum but did you notice XMR over on Poloniex?
Another spook to investors could cause them to dump out in another wave.

My opinion, its not all about investors. Those who really believe in the fundamentals and long term possibilities of the coin stay in.

 

[DeathRay1977]

I am a developer with 20 years experience in card based transactions such as POS and ATM. 15 years C++ development experience too. I also have my CISSP and CEH. I would be happy to help the testing in any way. I have two VPS servers and can bung some hash rate at testnet and run a couple of master nodes as well. I can do code reviews too.

 

[fernando]

My opinion, its not all about investors. Those who really believe in the fundamentals and long term possibilities of the coin stay in.

Absolutely. It is too soon to focus on the price. Everyone who's in so early should be ready for some bumps in the road.

 

[kurzwell]

Absolutely. It is too soon to focus on the price. Everyone who's in so early should be ready for some bumps in the road.

I think fernando got it exactly right. I know everyone is frustrated with this last glitch. I'm sure none more than the developers and testers. However, it is important to keep in mind that this is still beta software; not even v1.0 yet. Delivering flawed/buggy technology faster benefits no one but competing coins and day traders looking for more ups and downs. It isn't good for the long-term viability of Darkcoin, so I'm happy to let them hammer on this thing as long as it takes to get it right.

One of the most important parts of the announcement may have been overlooked by some.

Next week we will begin testing code to fix this issue. This will include setting up hundreds of daemons and several more pools on testnet to better simulate mainnet.

A major reason that they missed this timing problem was that their test group was small enough that it was unlikely to generate such a rare condition. Now they are investing time in creating a more robust testnet which will pay dividends in future testing and debugging. I'm glad to hear it.

 

[jimbit]

You are mixing the concept of continous integration with continous delivery.

I am in the process of setting up a continous integration server for darkcoin, which will additionally solve the problem of missing build for Linux 32bit and MacOSX. Below you can have a sneak preview on my ci-server, but i still have to wrap my head around the outdated build system darkcoin is using (bitcoin moved to autotools/autoconf since several month) As soon as i resolve them, Darkcoin is set for producing deterministic software builds, which can be reproduced and signed by independent parties.

Sneak preview at http://bamboo.darkcoin.qa/

Continous delivery is an interesting concept which i agree on, but you still have to plan on how you do it - otherwise in will end in chaos.

You see: things are moving. Not at the speed we all want it to be, but Rome wasn't built in a Day either.

Flare: this is awesome. I thought I was at work for a minute , we have bamboo and a ci.something.com, we also have di.something and stage.something, then production.

If someone has documented the qa use cases, maybe we could have a tasklist on the wiki, or bugzilla or something, then the testers could take a use case(or be given) and test it and report on it.

I have a MN on an EC2 server, also windows client that I am soloing against with about 8Mh

 

[raze]

- Secure communications for the Dev team
the chatter on IRC between evan and others of the Dev team has to move to a "saver" location !
chats and communications from evan and others get copied from IRC and show up all over BItcointalk with positive or negative comments.
i know AlexGR did this to calm the community and bring necessary updates (when it was needed), but this can NOT be and happen as it was totally unaproved by Evan and others copied there ! (that chatter is till now floating around out there)
this is really unprofessional (from a PR standpoint) and can go terrible wrong in the future as more and more people find their way to IRC, use quotes they copy from there, and might even change the content they copy.
PLEASE move to a saver location (non public) as i can see the PR disaster coming from this.

I disagree. Like AlexGR, I believe development should be as transparent as possible and I support him posting the IRC chats on bitcointalk. It's not as if they were PMs or private conversations; #darkcoin-test is open to the public and I think it should stay that way. Of course it could be taken out of context, but so can posts from anywhere else. So many coins try to sugarcoat or hide negative aspects of their development from their users/investors. I don't want Darkcoin to become like that.

 

[flare]

Flare: this is awesome. I thought I was at work for a minute , we have bamboo and a ci.something.com, we also have di.something and stage.something, then production.

If someone has documented the qa use cases, maybe we could have a tasklist on the wiki, or bugzilla or something, then the testers could take a use case(or be given) and test it and report on it.

Yep, you are getting the idea where i want the Darkcoin QA subsection of Darkcoin to evolve to - even dreaming of a small 'integnet', where i can run automated testcases against - time will tell

If we want Darkcoin to enter enterprise level market regions, we will need to apply enterprise level software development, testing and production principles somehow.

 

[flare]

So many coins try to sugarcoat or hide negative aspects of their development from their users/investors. I don't want Darkcoin to become like that.

Me neither - that's why darkcoin.qa will keep its state of being a independent reviewer/auditor.

Just like any other project/product in the cryptographic field, we can only gain acceptance and trust if we are fully transparent regarding development, issue management and quality control.

 

[HashEngineering]

Masternode Forking Issues
The solution to this is straightforward: any changes to the votes must also change the hash of the block, which will prevent the network from thinking these two blocks are the same. Next week we will begin testing code to fix this issue.

I am looking forward to seeing how the votes will be changing the hash of the block. Will the block header be changed to include a hash of the master node votes in the same way the merkle root is a hash of the transactions? As the developer of the android wallet, I hope that the changes don't break the SPV requirements that are part of the android app.

 

[flare]

I am a developer with 20 years experience in card based transactions such as POS and ATM. 15 years C++ development experience too. I also have my CISSP and CEH. I would be happy to help the testing in any way. I have two VPS servers and can bung some hash rate at testnet and run a couple of master nodes as well. I can do code reviews too.

Any chance i can have you working on the unit test suite of darkcoin? Darkcoin inherited a lot of unit tests from litecoin/bitcoin which are now unmaintained and need someone to have a look at ;-)

 

[TanteStefana]

I am a developer with 20 years experience in card based transactions such as POS and ATM. 15 years C++ development experience too. I also have my CISSP and CEH. I would be happy to help the testing in any way. I have two VPS servers and can bung some hash rate at testnet and run a couple of master nodes as well. I can do code reviews too.

Make sure you email Evan! He's building a team and you sound like you'd fit in beautifully!

 

[derk]

I disagree. Like AlexGR, I believe development should be as transparent as possible and I support him posting the IRC chats on bitcointalk. It's not as if they were PMs or private conversations; #darkcoin-test is open to the public and I think it should stay that way. Of course it could be taken out of context, but so can posts from anywhere else. So many coins try to sugarcoat or hide negative aspects of their development from their users/investors. I don't want Darkcoin to become like that.

Yes but it would be good to organise it a little more and for AlexGR to become "official" in some sense (Perhaps he already is and I don't know this )

 

[vertoe]

I am looking forward to seeing how the votes will be changing the hash of the block. Will the block header be changed to include a hash of the master node votes in the same way the merkle root is a hash of the transactions? As the developer of the android wallet, I hope that the changes don't break the SPV requirements that are part of the android app.

Keep an eye on the testing subforum. We will start testing the new block hashs on testnet next week. New source code (testing branches) and binaries are incoming.

 

[Bizmonger]

Respectfully, no, it's not ok. There is ~$50M of other people's money involved here, not including the business models built around the coin (e.g. exchange fees). It would be grossly irresponsible to be anything other than as clear and upfront as possible about what is going on with the network now and what will be going on with the network in the near future.

Thank you for clarifying the reason.

 

[tungfa]

I disagree. Like AlexGR, I believe development should be as transparent as possible and I support him posting the IRC chats on bitcointalk. It's not as if they were PMs or private conversations; #darkcoin-test is open to the public and I think it should stay that way. Of course it could be taken out of context, but so can posts from anywhere else. So many coins try to sugarcoat or hide negative aspects of their development from their users/investors. I don't want Darkcoin to become like that.

i totally hear you
but this is NOT bitcoin in 2007, the market and community moved way forward, sitting in the garage and coding away is great (and i love when it is done that way) but you can not develop a financial product "with the public" , with a 50 Mill market you have to protect your investors and community as much as yourself.
i am not talking about closed doors, but ...
example:
steve jobs would have never openly discussed his development stages and coding issues (that was done in a save and secure matter with his team involved) as otherwise Mr Samsung would have beaten them in the android/iphone market way back already !
know what i mean ? does this makes sense ?

 

[Lariondos]

i totally hear you
but this is NOT bitcoin in 2007, the market and community moved way forward, sitting in the garage and coding away is great (and i love when it is done that way) but you can not develop a financial product "with the public" , with a 50 Mill market you have to protect your investors and community as much as yourself.
i am not talking about closed doors, but ...
example:
steve jobs would have never openly discussed his development stages and coding issues (that was done in a save and secure matter with his team involved) as otherwise Mr Samsung would have beaten them in the android/iphone market way back already !
know what i mean ? does this makes sense ?

I think your comparison is not valid. We are talking about open source here! Your will get competitors being on par in functionality if darksend is released. Only way to be ahead of the race ist to be the first and stay there by innovation. In the meantime, you should be as transparent as possible not trying to hide your intentions.
Just my 2 duffs.

 

[tungfa]

I think your comparison is not valid. We are talking about open source here! Your will get competitors being on par in functionality if darksend is released. Only way to be ahead of the race ist to be the first and stay there by innovation. In the meantime, you should be as transparent as possible not trying to hide your intentions.
Just my 2 duffs.

Sure transparency is good and needed
but you are "releasing open source" when you are done programming (and you know it works and is save, and then everybody can check)
you are NOT programming open source in the public eye for a financial product !

 

[Lariondos]

Sure transparency is good and needed
but you are "releasing open source" when you are done programming (and you know it works and is save, and then everybody can check)
you are NOT programming open source in the public eye for a financial product !

In general you are right, but a vital part of crypto projects is the community. Your success is based on the community supporting it and you should not lock them out. All coins have still a long way to go mainstream and being widely accepted financial assets. I think and hope we will reach this stage sometimes - I'm on your side then.

 

[DeathRay1977]

Any chance i can have you working on the unit test suite of darkcoin? Darkcoin inherited a lot of unit tests from litecoin/bitcoin which are now unmaintained and need someone to have a look at ;-)

Sure. I would be more than happy to look at them. Are you planning automated testing as part of continuous via Jenkins or something similar?

 

[DeathRay1977]

Make sure you email Evan! He's building a team and you sound like you'd fit in beautifully!

Will do!

 

[flare]

Sure. I would be more than happy to look at them. Are you planning automated testing as part of continuous via Jenkins or something similar?

Exact, i have already set up a CI server at http://bamboo.darkcoin.qa - i still need to tweak the Buildsystem/makefile a bit, as it does not support cross compiling yet, but once i have that up and running i want to continue setting up the automated tests and gather some metrics (code coverage, cyclic complexity, ...)

Very welcome to have someone looking at the tests in src/test folder