Welcome to the Dash Forum!

Please sign up to discuss the most innovative cryptocurrency!

DashRadar development thread

Discussion in 'Projects' started by Antti Kaikkonen, Jun 20, 2017.

  1. UdjinM6

    UdjinM6 Official Dash Dev
    Dash Core Team Moderator

    Joined:
    May 20, 2014
    Messages:
    3,634
    Likes Received:
    3,534
    Trophy Points:
    1,183
    np :) you got hit with an improvised community driven stress test which brought down some nodes in our insight cluster for some time too (not to mention poloniex and shapeshift :D)
     
  2. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    The issue was that determining the transaction type (privatesend, create denomination, mixing, ..) was too slow. I optimized the code and it seems to be working fine now. Just processed a 1.91 MB block without an issue. The block explorer should catch up withing an hour or two.
     
    • Like Like x 1
  3. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    Update
    • Block explorer
      • InstantSend icon is now displayed for InstantSend transactions in the block explorer
      • Added a summary of the last 24 hours to the front page of the block explorer
        • Number of transactions
        • Number of InstantSend transactions
        • Number of PrivateSend transactions
        • Hashrate
      • Changed the recent blocks table to also display date instead of only time, because viewing old blocks is possible.
      • Added PrivateSend analysis page. Link available from the transaction page of a PrivateSend transaction.
        • Example 1, Example 2
        • Currently I'm running the analysis on my own computer. When I'm running it, the analysis should be available a few minutes after the PrivateSend transaction is confirmed, but I won't be able to run it 24/7 currently.
        • Todo: Add explanation, add basic transaction info, create analysis where PrivateSend transactions are tracked back to guesstimated wallets instead of create denominations transactions (group together create denominations transactions belonging in the same wallet)
    • Backend
      • Updated to Dash Core v0.12.3
      • Updated to the latest insight-api. Also wrote a tutorial on how to install it.
      • InstantSend locks are stored in the database
      • Had to do some performance optimizations thanks to the recent stress test by the dash community
        • Now able to handle ~2mb blocks with ease
        • Initial sync time should also be greatly reduced
    Source code of the PrivateSend analysis program: https://github.com/Antti-Kaikkonen/PrivateSendAnalysis.

    As recommended to me by Dash Watch, I also added an analytics software matomo to the site. It's open source and self hosted (unlike something like google analytics), so no data is provided to third parties. I may, however, share some statistics here and with the Dash Watch team. So far only 60 unique visitors since I installed it in July 11th.

    There is currently an issue with the search functionality of the block explorer because of this bug.
     
    • Like Like x 1
  4. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    Search functionality is now again working properly in the block explorer. Also updated the privatesend analysis page with loading indicators, basic transaction details and visual improvements.

    Currently working on changing the privatesend analysis by combining create denominations transactions originating from the same wallet. Soon the privatesend analysis page will show guesstimated wallets (based on the multi-input heuristic) instead of create denominations transactions.

    I also want to inform everyone that I will keep working on the ongoing proposal in August so that I can finish the remaining proposal objectives.
     
    • Like Like x 3
  5. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    Update:
    Changed the PrivateSend analysis page to use guesstimated wallets as mixing sources. New analysis is available in https://dashradar.com/explorer/privatesend/[txid]. Old analysis is still available in https://dashradar.com/explorer/privatesend-old/[txid]. The new analysis typically has less mixing sources and higher percentages because it combines create denominations transactions belonging in the same wallet. To get a list of example analysis pages you can run this query. It fetches privatesend transactions between blocks 850000 and 855000 because the old analysis isn't available for recent transactions.

    I want to emphasize that it's not possible for someone to determine the number of rounds used by just analyzing the blockchain. Different inputs can even use a different number if you interrupt the mixing process. So the results are more like leads rather than something conclusive.
     
    #95 Antti Kaikkonen, Aug 7, 2018
    Last edited: Aug 7, 2018
    • Like Like x 2
  6. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    I found a bug which caused mixing transactions to affect the guesstimated wallets feature although mixing transactions should have been ignored.

    I'm going to remove all of the guesstimated wallets and then recompute them. This will cause a few hours of downtime. Edit: Done

    I will also need to recompute the PrivateSend analysis results which were using the guesstimated wallets feature.
     
    #96 Antti Kaikkonen, Aug 7, 2018
    Last edited: Aug 8, 2018
    • Useful Useful x 2
  7. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    I found another bug where a trasaction included in an orphaned block re-appearing in the mempool caused the transaction to be duplicated. This has so far happened to only 1 transaction but it already scewed the average transaction fee chart for August 8th. Currently reprocessing the data and I will edit this post when fixed. Edit: done
     
    #97 Antti Kaikkonen, Aug 10, 2018
    Last edited: Aug 12, 2018
  8. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    • Like Like x 1
  9. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    • Like Like x 2
  10. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    Here is the number of unique visitors for July, August and September
    dashradar unique visutors.png

    I'm using an open-source self hosted analytics software called Matomo, so that your information is not leaked to Google or other third parties.
     
    • Like Like x 1
  11. jeffh

    jeffh Member

    Joined:
    May 8, 2017
    Messages:
    108
    Likes Received:
    45
    Trophy Points:
    78
    @Antti Kaikkonen If you want to pull more information about the masternodes here, feel free to run a query against this data from Dash Intel:

    https://api.dashintel.org/dash_masternode_insights. We have some additional information that we're pulling from paid API queries and saving in our database that you might find useful. Keep in mind, I ignore non-ENABLED masternodes in most cases.
     
    • Like Like x 1
    • Winner Winner x 1
  12. jeffh

    jeffh Member

    Joined:
    May 8, 2017
    Messages:
    108
    Likes Received:
    45
    Trophy Points:
    78
    @colofi this was some testing done by @DeltaEngine for a stress test a few months ago and then more recently in preparation for a network stress-test (theoretical max tx/s to beat any production blockchain record) scheduled to happen around the 12.4 launch.
     
    • Like Like x 1
  13. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    Nodes page added to dashradar: https://dashradar.com/nodes

    There is still some things that could be added/improved:
    • Make the map size dynamic depending on screen resolution
    • Add an option to filter active/inactive nodes in the "All nodes" tab
    • (Individual node page by clicking the ip address)
    • (Add historical charts for country/version/organization distribution)
    • (Add link to collateral txid/address for masternodes)
    • (Show voting data for masternodes)
     
    • Like Like x 1
  14. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    Changed the default url (https://dashradar.com) from the graph visualization to a simple landing page containing links to explorer, graph, charts and nodes pages. In the future I might add more content to the landing page to make it more like a dashboard.

    Also upgraded from Angular 6 to Angular 7.
     
  15. Gaby_64

    Gaby_64 New Member

    Joined:
    Sep 21, 2016
    Messages:
    16
    Likes Received:
    3
    Trophy Points:
    3
    how long does it take for a new node to appear in the nodes list?

    I have been running a non masternode node for 2 days now.
     
  16. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    It only lists nodes that accept new connections.

    If you are running the node at home then it might be behind a NAT router. In that case you should forward port 9999 from the router to the LAN IP address of the computer running the node.

    It's also possible that there could be something wrong with the configuration of the node (dash.conf).

    You can check if your node is accepting connections with the node checker found in http://nodes.blakeanderson.ca/networks/dash.
     
    • Winner Winner x 1
    • Useful Useful x 1
  17. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,480
    Likes Received:
    713
    Trophy Points:
    183
    Any chance you can include charts for mining hashrate & mining difficulty & mining pools spread?
     
  18. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    I recently updated the website to use angular universal server side rendering. This means that the first page the user is loading is rendered to HTML on the server and after that it will continue to function as a single page application. It can make the initial page load slightly faster but the main benefit is that it makes the website crawlable by search engines.

    Also moved the navigation from the top to the side. This solution should work better for mobile users and allows fitting more navigation elements in the future.

    I think that is a good idea but I'm not sure when I will be able to do that.
     
  19. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,480
    Likes Received:
    713
    Trophy Points:
    183
    No pressure, no hurry.

    Good to hear about your website improvements.
     
  20. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    @qwizzie It's currently possible to get generate those charts using the data explorer but it can be quite slow for long time periods. Adding them to the charts page would require storing a running total of the difficulty so that average difficulty of a day can be computed very quickly. Hashrate by pool would require even more changes.

    Here are some examples using the data explorer:
    You can modify the queries to get more data but if you make the period too long then the query will not finish because the is a time limit (1 minute if I remember correctly).
     
    • Informative Informative x 1
  21. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,480
    Likes Received:
    713
    Trophy Points:
    183
    i like the line chart of the daily hashrate last 3 months, it would fit well with the other line charts i think.
    And maybe think of ways implementing mining difficulty and pools for a later date ? Or skip if its getting to be too much of a hassle
     
  22. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    I have had some issues with the Neo4j database that this project is using to store the blockchain:
    • Poor write performance
    • Issues with the cost planner sometimes generating highly inefficient query execution plans
    • Community edition lacking some useful features
    • Other issues
    I'm probably not going to build any new features on top of Neo4j and I might start replacing some or all of the current features using other databases.

    I'm currently building an improved version of the address clustering a.k.a guesstimated wallets feature. In the new version users will be able to browse an ordered list of the wallet's transactions. Wallet balance after each transaction will also be available as well as a historical balance chart.
     
    • Informative Informative x 1
  23. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    Update

    Added the new 0.001 DASH PrivateSend denomination.
     
    • Like Like x 5
  24. qwizzie

    qwizzie Well-known Member

    Joined:
    Aug 6, 2014
    Messages:
    1,480
    Likes Received:
    713
    Trophy Points:
    183
    Quick question about two charts :

    Transactions per day
    Mixing transactions per day

    Is the "mixing transactions per day" data included or excluded from the "transactions per day" chart ?
    I ask this because i wonder if a lot of mixing could inflate the transactions per day chart.
     
  25. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    Transactions per day includes all transactions including mixing transactions, privatesend transactions and create denominations transactions.
     
    • Like Like x 1
    • Informative Informative x 1
  26. tiger-HZX

    tiger-HZX New Member

    Joined:
    Apr 28, 2019
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    I found that in block 206980,there are two mixing trx, since their trx fee is 0, but dashradar recognize them as create denomination trx, Is it a bug? If I am wrong, is there something about mixing trx I don't know?
     
  27. thephez

    thephez Member
    Dash Core Team

    Joined:
    Jan 23, 2016
    Messages:
    109
    Likes Received:
    46
    Trophy Points:
    78
    In 2015 when that tx happened it was possible to send any tx with a fee of 0. Also, the txs I looked at (mentioned here) cannot be mixing txs. One of them does not have an equal number of inputs/outputs. The other cannot be for multiple reasons including - 1. too many inputs/outputs, and 2. Not all inputs/outputs have the same value (unless the PS rules were different back then (@UdjinM6 ?).
     
  28. tiger-HZX

    tiger-HZX New Member

    Joined:
    Apr 28, 2019
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    Thanks for your reply, but there are still are something I did not understand. why a PrivateSend tx can not tcontains too many inputs/outputs, if there is a rule, what is the limit of inputs, I remember there are at most 200 inputs/outputs in a tx, but I didn't hear about the inputs/outputs limits of PS tx.

    In the follow links(please replace 。 with .) , we can see someone said that inputs/outputs in dash's PrivateSend tx can be different denominations once,
    https://forum。bitrad。io/index。php?thread/953-what-is-darksend-and-how-to-use-it/ , maybe it is wrong, maybe it is right.

    There is another msg, the 5 PS denominations was introduced in protocal version 72013, it means there may be another rules about PrivateSend before 72013, I searchd a lot and did not find any information about it.

    I am still confused, if you know, please tell me a detail about it. @thephez @UdjinM6
     
  29. Antti Kaikkonen

    Antti Kaikkonen Active Member

    Joined:
    Jun 20, 2017
    Messages:
    256
    Likes Received:
    164
    Trophy Points:
    103
    Dash Address:
    XnZdwT1w2kGeH6RujwoyJ7BBNrukdyTBRB
    Dashradar does not currently recognize old mixing transactions where multiple different denominations in the same mixing transaction was allowed.
     
    • Informative Informative x 1
  30. tiger-HZX

    tiger-HZX New Member

    Joined:
    Apr 28, 2019
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    1
    ok, I know. Thx