Welcome to the Dash Forum!

Please sign up to discuss the most innovative cryptocurrency!

6/20 RC3 Post-mortem

Discussion in 'Official Developer Thread' started by eduffield, Jun 21, 2014.

  1. eduffield

    eduffield Core Developer
    Dash Core Team

    Joined:
    Mar 9, 2014
    Messages:
    1,084
    Likes Received:
    5,318
    Trophy Points:
    183
    Masternode Forking Issues

    When we launched the Masternode payment system today, the network exhibited some instability issues similar to those we experienced with the last fork. This new instability was nowhere near as serious as the first implementation, but we erred on the side of caution and disabled Masternode payments for the time being.

    Immediately after the network problems were noticed, many users sent debug information from their clients. After a few hours of analysis, we discovered the root cause of the forks.

    Two blocks are solved at nearly the same moment on the network, and both are propagated and accepted by the network. In the current implementation, both blocks have the same hash, but in these blocks there's some discrepancy about who to vote for.

    In one block the miner votes for 88802 and 88803, in the other the miner abstains from voting. When the next block is solved, it's based on of one of the older blocks, so half of the network believes the miner cheated and rejects the block causing a fork.

    Although the network was “pruning” the bad forks as intended, the amount of time it was taking to do so was beyond the confirmation window. This was untenable, so we decided to revert.

    The solution to this is straightforward: any changes to the votes must also change the hash of the block, which will prevent the network from thinking these two blocks are the same. Next week we will begin testing code to fix this issue. This will include setting up hundreds of daemons and several more pools on testnet to better simulate mainnet. Barring any new issues, we should be ready to launch in 2-3 weeks.

    If interested, here’s the debug information we used to track down the issue : http://pastebin.com/QmbM8dPH

    New Developers

    We are happy to announce the addition of two developers to the Darkcoin team. David (DRKLord) and Fabian (CHAOSiTEC).

    DRKLord brings about 10 years of experience with C, C++, x84/64 ASM and many other skills to the team.

    CHAOSiTEC is a long time Darkcoin supporter with extensive knowledge of cryptocurrency and programming.

    We’re really excited that a core team is coming together and we’re look forward to seeing what we can accomplish.
     
    • Like Like x 17
  2. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,844
    Likes Received:
    1,863
    Trophy Points:
    1,283
    Yaaay for David and Fabian (Now I know your names finally! LOL) Thank you so much for joining and helping out! We all really appreciate it!
     
    • Like Like x 2
  3. RasNetworks

    RasNetworks New Member

    Joined:
    Mar 14, 2014
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    Something so simple as a spanner. Time to ride the ups and downs of the roller coaster!
     
  4. george9178

    george9178 New Member

    Joined:
    May 22, 2014
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    launch in 2-3 weeks?So long!
     
  5. TanteStefana

    TanteStefana Moderator
    Linguistic Foundation Member

    Joined:
    Mar 9, 2014
    Messages:
    2,844
    Likes Received:
    1,863
    Trophy Points:
    1,283
    Yah, but we learned 2 important things, that the rare event (well probably super rare in testnet, as it never happened) where a block is solved by two nodes at the same time can cause a glitch in the voting. And that the system did eventually solve the problem, though it took a little too long. It looks like Evan has a plan on how to fix this. It looks like we have a refinement issue, not a major out of left field issue.

    I'll bet that if Evan get's it fixed to his satisfaction, he'll schedule a re-launch with in a week (of feeling confident the issue is solved) so I know a week seems like a long time, but remember, the 3rd time is the charm :tongue: ;) :D :)
     
    • Like Like x 1
  6. Red-Shinobi

    Red-Shinobi Member

    Joined:
    Apr 9, 2014
    Messages:
    117
    Likes Received:
    76
    Trophy Points:
    78
    No more launches ok, just stop with the launching stuff...you used up all your launches...stop now.
    And no more dates, youre out of dates too..

    From now on, how about you do "Live testing" or maybe "active audits"..."security simulation".... Something so that when you find a bug, its a success and not another one of these Epic Crypto Fails.

    Im only saying this cause i love you, but yeah, dont put us through another one of these launch date face-smashers. At this point we have to assume the next release will also have problems, so basically im suggesting you build that into the plan.
     
  7. derk

    derk New Member

    Joined:
    Jun 4, 2014
    Messages:
    37
    Likes Received:
    20
    Trophy Points:
    8
    Be patient. It's new software. If you keep a cool head it's not hard to see that these are just bumps in the road ; the project stays sound.
    This experience shakes out speculators with no interest in the coin.
    The team is constantly growing. The project is growing. And we are in perfect synch with the upcoming BTC bubble.
    Relax. Enjoy. :)
     
    • Like Like x 2
  8. flare

    flare Administrator
    Dash Core Team Moderator

    Joined:
    May 18, 2014
    Messages:
    2,287
    Likes Received:
    2,426
    Trophy Points:
    1,183
    Thanks for the great analysis!

    I've seen the behaviour of the network, having blocks with identical hash but different blocktemplates (votes) in testnet with 70 clients too - but did not draw the right conclusion that this might be a problem. I assumed that this is part of the normal network operation, gathering concensus for the votes. Mainnet proofed me wrong :)

    And i think you are right: The fix should be quite easy. Remembering the "blockchain in a blockchain" picture we had for the voting system, the state of the inner votechain must influence the hash of the outer blockchain. Quite as easy as that.

    Looking forward to get RC3.1 in testnet.

    Cheers,
    Holger

    EDIT: Even now, with testnet having 5~6 masternodes i get different blocktemplates:

    Node1:

    Code:
        "votes" : [
            "f4590000000000001976a914ecbade2ca5d03f145e3fafc632da3b698053952088ac04000000",
            "f5590000000000001976a91497f4cfe44cd5eb16df9c9d8f01cd22a6c78c8e5888ac03000000",
            "f6590000000000001976a9147f898281b5ca5b7ec9385e04421b4e2a1aa008d488ac02000000",
            "f7590000000000001976a91497f4cfe44cd5eb16df9c9d8f01cd22a6c78c8e5888ac01000000"
        ],
        "payee" : "mxv11kABs5n2aLWHytKgFQvESd2f9vwywL",
        "masternode_payments" : true
    }
    

    Node2:

    Code:
        "votes" : [
            "f4590000000000001976a914ecbade2ca5d03f145e3fafc632da3b698053952088ac04000000",
            "f5590000000000001976a91497f4cfe44cd5eb16df9c9d8f01cd22a6c78c8e5888ac02000000",
            "f6590000000000001976a9147f898281b5ca5b7ec9385e04421b4e2a1aa008d488ac02000000",
            "f7590000000000001976a91497f4cfe44cd5eb16df9c9d8f01cd22a6c78c8e5888ac01000000"
        ],
        "payee" : "mxv11kABs5n2aLWHytKgFQvESd2f9vwywL",
        "masternode_payments" : true
    }
    Notice line "f559000..." node 1 has 3 votes, node 2 has 2 votes. So this behaviour can already reproduced in testnet with small amount of masternodes :)
     
    #8 flare, Jun 21, 2014
    Last edited by a moderator: Jun 21, 2014
    • Like Like x 2
  9. eltito

    eltito Active Member

    Joined:
    Apr 21, 2014
    Messages:
    157
    Likes Received:
    185
    Trophy Points:
    103
    While I agree that there is room for improvement in how we manage the expectations we set, there is simply no way around issuing launch dates for planned hard forks. Pools, exchanges, and users require ample lead time to update their wallets.
     
    • Like Like x 4
  10. derk

    derk New Member

    Joined:
    Jun 4, 2014
    Messages:
    37
    Likes Received:
    20
    Trophy Points:
    8
    +1 Everyone needs to have ample time to update.
    But of course the update must take into account that there will always be problems caused by people not updating. That will never go away! ;)
     
    • Like Like x 1
  11. AlexGR

    AlexGR New Member

    Joined:
    May 24, 2014
    Messages:
    26
    Likes Received:
    21
    Trophy Points:
    3
    It's all about expectations. You announce that you hope to get it right but, as has been seen so far, this is a tricky update and might as well need patching.

    One solution would be to do what Vertcoin does with merge mining of their testbed coin. They apply the changes to the merge-mined test coin and thus the mainnet is free from anomalies. The testcoin also has much more real-life correlation than a testnet coin because it is actually mined by real-life clients/pools etc. The merged mined coin is also traded on poloniex btw.
     
    • Like Like x 2
  12. yidakee

    yidakee Well-known Member
    Foundation Member

    Joined:
    Apr 16, 2014
    Messages:
    1,812
    Likes Received:
    1,168
    Trophy Points:
    283
    Lets roll up our sleeves and get down and dirty back to testnet! Unfortunately, I'll only be able to when I get back Monday to fire up the nodes.
     
  13. wmr1988

    wmr1988 New Member

    Joined:
    May 27, 2014
    Messages:
    10
    Likes Received:
    2
    Trophy Points:
    3
    While this has been yet another disappointment, it is always good to know the there's light at the end of the tunnel. I just wish that we get there sooner than later.
     
  14. Red-Shinobi

    Red-Shinobi Member

    Joined:
    Apr 9, 2014
    Messages:
    117
    Likes Received:
    76
    Trophy Points:
    78
    Seems like maybe 72 hours is enough time? I know i certainly never got any lead time when the wallets updated in the past. And Derk is correct isnt he? That the hard forks have to deal with people/ pools/ exchanges not updating as they should anyway?

    I mean...is the plan to basically just redo this all again in 2-3 weeks?
    Let's keep Evans reputation in mind and make arrangements so that failure isnt even possible. Foresnstance; no more launch, how about a "live debugging session" If theres no bugs then great, success! And if we find a bug, great, successful debugging session!

    If the team is heart-set on "launching" something launch something that's a 100% lock, A new logo or color scheme, a new startup graphic, i dont care, just a win ffs, no matter how trivial.
     
    • Like Like x 1
  15. JGCMiner

    JGCMiner Active Member
    Masternode Owner/Operator

    Joined:
    Jun 8, 2014
    Messages:
    333
    Likes Received:
    199
    Trophy Points:
    113
    Sounds like an excellent idea. Anybody know the difficulties or the downsides of this?
     
    #15 JGCMiner, Jun 21, 2014
    Last edited by a moderator: Jun 21, 2014
    • Like Like x 1
  16. chaeplin

    chaeplin Official Dash Dev
    Core Developer

    Joined:
    Mar 29, 2014
    Messages:
    749
    Likes Received:
    355
    Trophy Points:
    133
    Dash Address:
    XiDWe5fkVcrXBQApmCFQUxpue5iuWcbmcK
    But there was no fork in testnet.

    I have checked 3 nodes in mainet.
    Code:
    [email protected]:~> grep REORGANIZE .darkcoin/debug.log
    2014-06-20 16:18:22 REORGANIZE: Disconnect 1 blocks; 00000000000212dcf55b5f098c73f0821880ccc545f6a90794693aa02794c67a..
    2014-06-20 16:18:22 REORGANIZE: Connect 2 blocks; ..00000000000022dacba046c9bad97104b4ca97226b90fad6f8ad31d0400d692b
    2014-06-20 16:37:04 REORGANIZE: Disconnect 1 blocks; 00000000000022dacba046c9bad97104b4ca97226b90fad6f8ad31d0400d692b..
    2014-06-20 16:37:04 REORGANIZE: Connect 2 blocks; ..00000000000d82a5cd6f1bd0314a9ccd2a2d513d2ef6cecd852a454a6c7e2c5b
    
    Code:
    [email protected]:~$ grep REORGANIZE .darkcoin/debug.log
    2014-06-20 16:31:02 REORGANIZE: Disconnect 1 blocks; 00000000000022dacba046c9bad97104b4ca97226b90fad6f8ad31d0400d692b..
    2014-06-20 16:31:02 REORGANIZE: Connect 2 blocks; ..00000000000d82a5cd6f1bd0314a9ccd2a2d513d2ef6cecd852a454a6c7e2c5b
    2014-06-20 17:03:00 REORGANIZE: Disconnect 1 blocks; 00000000001392f1652e9bf45cd8bc79dc60fe935277cd11538565b4a94fa85f..
    2014-06-20 17:03:00 REORGANIZE: Connect 2 blocks; ..000000000015426875b0125a3d0e4f0bcea496ef77d83137aac407792b85ebb5
    
    Code:
    [email protected]:~$ grep REORGANIZE .darkcoin/debug.log
    2014-06-20 17:03:00 REORGANIZE: Disconnect 1 blocks; 00000000001392f1652e9bf45cd8bc79dc60fe935277cd11538565b4a94fa85f..
    2014-06-20 17:03:00 REORGANIZE: Connect 2 blocks; ..000000000015426875b0125a3d0e4f0bcea496ef77d83137aac407792b85ebb5
    
     
    • Like Like x 1
  17. flare

    flare Administrator
    Dash Core Team Moderator

    Joined:
    May 18, 2014
    Messages:
    2,287
    Likes Received:
    2,426
    Trophy Points:
    1,183
    Yeah, maybe not enough distinct miners with diverging masternode lists in testnet - just me, nomp, nomp2 + p2pool. Maybe not sufficient to produce a fork. We need to distribute the hashing power to more miners.

    I will patch a client to force diverging masternode/vote candidate lists, setup four of them as miners and put some MHs on them.
     
    #17 flare, Jun 21, 2014
    Last edited by a moderator: Jun 21, 2014
  18. vertoe

    vertoe Three of Nine

    Joined:
    Mar 28, 2014
    Messages:
    2,574
    Likes Received:
    1,656
    Trophy Points:
    1,283
    Keep in mind, all pools, all exchanges, all merchants, and whatever need enough time to prepare for forks. Being on the wrong chain is no fun at all. Announcing a fork within only 2-3 days (we had that once accidently) is a huge mess for the network!
     
  19. chaeplin

    chaeplin Official Dash Dev
    Core Developer

    Joined:
    Mar 29, 2014
    Messages:
    749
    Likes Received:
    355
    Trophy Points:
    133
    Dash Address:
    XiDWe5fkVcrXBQApmCFQUxpue5iuWcbmcK
    Ah... we have...

    I have reindexed 3 testnet nodes.
    One of them got this.
    Code:
    [email protected]:~$  grep REORGANIZE .darkcoin/testnet3/debug.log
    2014-06-21 09:24:50 REORGANIZE: Disconnect 1 blocks; 00000000fd93e980730f288f1a640fc524b86f5e2fc99005bdf5b3a1d4d7aeea..
    2014-06-21 09:24:50 REORGANIZE: Connect 2 blocks; ..000000018a59904ad1e02ae1cfdf30b17fb6f740c574ff31c46ab1e0b754b1d6
    
    SetBestChain: new best=00000000fd93e980730f288f1a640fc524b86f5e2fc99005bdf5b3a1d4d7aeea  height=19451  log2_work=44.693617  tx=33603  date=2014-06-14 17:43:42 progress=0.993643
    
    SetBestChain: new best=000000018a59904ad1e02ae1cfdf30b17fb6f740c574ff31c46ab1e0b754b1d6  height=19453  log2_work=44.693717  tx=33608  date=2014-06-14 17:47:22 progress=0.993646
    
     
    • Like Like x 3
  20. Red-Shinobi

    Red-Shinobi Member

    Joined:
    Apr 9, 2014
    Messages:
    117
    Likes Received:
    76
    Trophy Points:
    78
    I remember that quick fork, THAT fork went smoothly actually...smoothest one of them all perhaps. The forks with the long announcement times have failed the worst no?

    Look, Mintpal caused a panic today being not fully prepared and they had plenty of time...like a month! They probably could have done as well with a last minute phone call. So It's not a lead time issue right?

    I'm concerned if the plan from here is actually just "Third times a charm!"
     
  21. flare

    flare Administrator
    Dash Core Team Moderator

    Joined:
    May 18, 2014
    Messages:
    2,287
    Likes Received:
    2,426
    Trophy Points:
    1,183
    So regular reindexing of the testnet clients and grepping for REORGANIZE should now be part of the test plan :)
     
  22. stilgars

    stilgars New Member

    Joined:
    Jun 21, 2014
    Messages:
    11
    Likes Received:
    10
    Trophy Points:
    3
    Exactly, non responsive under-concerned ppl will still be non responsive under-concerned ppl in two weeks time.
    In fact, waiting for so long is underproductive since the migration effort will lose some momentum and fall under the radar in the next coming days, and awareness should be raised *again* from scratch within the last couple of days.

    EDIT: but I missed the part stating that testnet will now reproduce more accurately mainnet. Good news, just a little puzzled since I thought rigorous testing was performed after last time already.
     
    #22 stilgars, Jun 21, 2014
    Last edited by a moderator: Jun 21, 2014
    • Like Like x 1
  23. flare

    flare Administrator
    Dash Core Team Moderator

    Joined:
    May 18, 2014
    Messages:
    2,287
    Likes Received:
    2,426
    Trophy Points:
    1,183
    It was, but obviously we missed a testcase with producing blocks with identical hash but different payee - will be adressed now.

    Any suggestions for further testcases you can think about? Constructive input welcome ;-)
     
  24. JGCMiner

    JGCMiner Active Member
    Masternode Owner/Operator

    Joined:
    Jun 8, 2014
    Messages:
    333
    Likes Received:
    199
    Trophy Points:
    113
    Well is it about testcases -- or testnet not being sufficient to serve as a testbed for mainnet?

    The first time the fork failed a much larger effort was made on testnet (more testcases, purposely trying to break it, etc.). In a sense that effort paid off because the cause of the issue this time was not outdated clients -- an improvement, but in the end the fact is that the fork still failed.

    What makes you think that once again re-doubling efforts will improve the situation? Especially since the issue this time was the exact type of issue that is hard to find on testnet because of the lack of hashpower.

    Just like there was a difference in testing effort between attempt 1 and attempt 2, there needs to be a difference this time as well. Otherwise there is no reason to think that the next payment fork will be any different.

    IMO, the first thing to do is somehow make testnet more similar to mainnet. AlexGR suggested merged mining -- not sure how doable that is, but it would be excellent if possible. That may not be the only method, but just relying on volunteer miners is not likely to improve the current situation by very much.

    tl;dr You can't expect the next fork attempt to be any different if the same level of testing on the same platform is carried out. Testnet needs to be improved. Not just for the masternode payment testing, but for all the anonymity experiments down the road.

    Edit: Sorry for overstepping my bounds as someone who hasn't put in the same amount of time as you and others, but I felt that the above needed to be stated strongly.
     
    #24 JGCMiner, Jun 21, 2014
    Last edited by a moderator: Jun 21, 2014
    • Like Like x 1
  25. Bizmonger

    Bizmonger New Member

    Joined:
    Jun 2, 2014
    Messages:
    28
    Likes Received:
    4
    Trophy Points:
    3
    Red-Shionobi has a point.
    Why not reconsider the way that updates are released.
    Why not deliver to mainnet more frequently with just devs and testers up-to-date with the network at first and eventually the secret is out. So what if it's hard on the network.
    It's called continuous-integration. If something breaks, then simply revert. It's okay.
     
    • Like Like x 1
  26. luke997

    luke997 New Member

    Joined:
    Mar 27, 2014
    Messages:
    10
    Likes Received:
    3
    Trophy Points:
    3
    No.

    Continuous integration is not the same as frequent releases of production software.
    But anyway, frequent released can be applied to not distributed systems, or to distributed system with strict update control.
    If you don't control updating process of the clients, you have a allow a liberal amount of time to update.

    Would you want to have, for example a 3 versions of accounting system clients sending different data structure to the back end?
     
    • Like Like x 2
  27. flare

    flare Administrator
    Dash Core Team Moderator

    Joined:
    May 18, 2014
    Messages:
    2,287
    Likes Received:
    2,426
    Trophy Points:
    1,183
    Both :)
    Any additional effort will improve the situation, as long you follow some kind of plan - that's called continous improvement process: Plan - Do - Check - Act. That's the way you improve situations.
    That's why we add the testcases we missed to the testsuite and improve testnet to match mainnet even closer.
    I fully agree with you here, and started a private discussion with eduffield and AlexGR two hours ago, since i really like the approch of making testnet a clone of mainnet - if it's feasabe to have merged mining for testcoins, that's the way to go.

    I fully agree with you, never said that testing/testbed should not be improved.

    You are welcome, as long as the discussion is focused on moving forward and improving things, i'll take any kind of constructive criticism.

    Thanks for you input. :)
     
    • Like Like x 2
  28. flare

    flare Administrator
    Dash Core Team Moderator

    Joined:
    May 18, 2014
    Messages:
    2,287
    Likes Received:
    2,426
    Trophy Points:
    1,183
    You are mixing the concept of continous integration with continous delivery.

    I am in the process of setting up a continous integration server for darkcoin, which will additionally solve the problem of missing build for Linux 32bit and MacOSX. Below you can have a sneak preview on my ci-server, but i still have to wrap my head around the outdated build system darkcoin is using (bitcoin moved to autotools/autoconf since several month) As soon as i resolve them, Darkcoin is set for producing deterministic software builds, which can be reproduced and signed by independent parties.

    Sneak preview at http://bamboo.darkcoin.qa/

    Continous delivery is an interesting concept which i agree on, but you still have to plan on how you do it - otherwise in will end in chaos.

    You see: things are moving. Not at the speed we all want it to be, but Rome wasn't built in a Day either.
     
    #28 flare, Jun 21, 2014
    Last edited by a moderator: Jun 21, 2014
    • Like Like x 3
  29. JGCMiner

    JGCMiner Active Member
    Masternode Owner/Operator

    Joined:
    Jun 8, 2014
    Messages:
    333
    Likes Received:
    199
    Trophy Points:
    113
    Thanks for the detailed response.
    I misinterpreted your earlier responses as "well let's do the same thing again but get it right this time". I am very happy to see that improvements are already being discussed. :)
     
    • Like Like x 1
  30. tungfa

    tungfa Administrator
    Dash Core Team Foundation Member Masternode Owner/Operator Moderator

    Joined:
    Apr 9, 2014
    Messages:
    8,642
    Likes Received:
    6,630
    Trophy Points:
    1,283
    RC 3 launch yesterday was definitely a bummer !
    i do not know enough about the technical part to participate in this discussion and believe evan and the team know what they are doing and i totally trust them.

    i have a couple of pointers (which in my opinion) have to be discuss,
    hopefully solved before the next release of any kind.
    (i am not sure where else to put them, so i post them here, hope you guys do NOT mind !!)

    - Communication
    there has to be a proper chain of command be set up for updates to the community and public !
    it can NOT be that 2 hours into the release there are NO updates what is going on !!! no updates only fuel the trolls and all the other nonsense on the chats. people get confused and worried and that is NOT necessary or helpful (for confidence in DRK and the price) !
    we are by now talking about a 41 Mill market, this is not just evan with some boys in the garage coding away, DRK has to be treated as a professional product and its community and "customers" have to be updated and maintained as they deserve it.
    even after the 1st updates and change of wallet messages, there was no (until 10 hours after) update if we are on a wrong fork, or maybe it is all good now (as IRC copies suggested evan can fix it fast), or not ... you Dev guys might have it all figured out at that point (or not) but you HAVE to tell the community (treat them as your customers, thats what they are!) where DRK is standing and going, good or bad , failed or running ... there is only good PR in the long run, but NO PR and communication is totally unprofessional and not acceptable in a 41 Mill market. as we (DRK) want to play in the big boys (BTC, LTC,..) league, we have to act like it !!!

    - Secure communications for the Dev team
    the chatter on IRC between evan and others of the Dev team has to move to a "saver" location !
    chats and communications from evan and others get copied from IRC and show up all over BItcointalk with positive or negative comments.
    i know AlexGR did this to calm the community and bring necessary updates (when it was needed), but this can NOT be and happen as it was totally unaproved by Evan and others copied there ! (that chatter is till now floating around out there)
    this is really unprofessional (from a PR standpoint) and can go terrible wrong in the future as more and more people find their way to IRC, use quotes they copy from there, and might even change the content they copy.
    PLEASE move to a saver location (non public) as i can see the PR disaster coming from this.

    - Release Dates
    i am only approaching this from a PR side of things .... if a RC date is set (27th June) and then the date is moved forward (20th) due to tons of testing and confidence of the developer, and then the RC still fails ... well bummer eh !
    we gotta stick to our set dates, test the sh..t out of it before launching, and launch when 100% ! do not get pushed into earlier releases by the market or whatever pushed you, stick to YOUR timeline ! and the market will follow and appreciate it. (sure now we are behind with releases and the pressure is on.... but you EVAN are the MAN, thats why we are all here, so you tell the community and market how long it takes to be ready for whatever launch !!!! )

    - New Developer Team
    i am very happy to see that evan extended his development team !!!! finally !
    extend your team further !! a couple of guys can NOT do this business by them self !! delegate to others !! i know that started already and there was a long communication from eltito (who is very good), regarding communication going on, but this has to go even further. treat it as a company and a product !

    these are only my 2 cent of knowledge and thoughts in all of this
    i am happy to be part of DRK and i know we will launch to further highs, but i believe it is time to change certain parts of the "team" to be as professional as "our" coin DRK is or will be soon !
     
    #30 tungfa, Jun 21, 2014
    Last edited by a moderator: Jun 21, 2014
    • Like Like x 5

Share This Page