DashRadar development thread

[UdjinM6]

np you got hit with an improvised community driven stress test which brought down some nodes in our insight cluster for some time too (not to mention poloniex and shapeshift )

 

[Antti Kaikkonen]

The issue was that determining the transaction type (privatesend, create denomination, mixing, ..) was too slow. I optimized the code and it seems to be working fine now. Just processed a 1.91 MB block without an issue. The block explorer should catch up withing an hour or two.

 

[Antti Kaikkonen]

Update

• Block explorer
  • InstantSend icon is now displayed for InstantSend transactions in the block explorer
  • Added a summary of the last 24 hours to the front page of the block explorer
    • Number of transactions
    • Number of InstantSend transactions
    • Number of PrivateSend transactions
    • Hashrate
  • Changed the recent blocks table to also display date instead of only time, because viewing old blocks is possible.
  • Added PrivateSend analysis page. Link available from the transaction page of a PrivateSend transaction.
    • Example 1, Example 2
    • Currently I'm running the analysis on my own computer. When I'm running it, the analysis should be available a few minutes after the PrivateSend transaction is confirmed, but I won't be able to run it 24/7 currently.
    • Todo: Add explanation, add basic transaction info, create analysis where PrivateSend transactions are tracked back to guesstimated wallets instead of create denominations transactions (group together create denominations transactions belonging in the same wallet)
• Backend
  • Updated to Dash Core v0.12.3
  • Updated to the latest insight-api. Also wrote a tutorial on how to install it.
  • InstantSend locks are stored in the database
  • Had to do some performance optimizations thanks to the recent stress test by the dash community
    • Now able to handle ~2mb blocks with ease
    • Initial sync time should also be greatly reduced

Source code of the PrivateSend analysis program: https://github.com/Antti-Kaikkonen/PrivateSendAnalysis.

As recommended to me by Dash Watch, I also added an analytics software matomo to the site. It's open source and self hosted (unlike something like google analytics), so no data is provided to third parties. I may, however, share some statistics here and with the Dash Watch team. So far only 60 unique visitors since I installed it in July 11th.

There is currently an issue with the search functionality of the block explorer because of this bug.

 

[Antti Kaikkonen]

Search functionality is now again working properly in the block explorer. Also updated the privatesend analysis page with loading indicators, basic transaction details and visual improvements.

Currently working on changing the privatesend analysis by combining create denominations transactions originating from the same wallet. Soon the privatesend analysis page will show guesstimated wallets (based on the multi-input heuristic) instead of create denominations transactions.

I also want to inform everyone that I will keep working on the ongoing proposal in August so that I can finish the remaining proposal objectives.

 

[Antti Kaikkonen]

Update:
Changed the PrivateSend analysis page to use guesstimated wallets as mixing sources. New analysis is available in https://dashradar.com/explorer/privatesend/[txid]. Old analysis is still available in https://dashradar.com/explorer/privatesend-old/[txid]. The new analysis typically has less mixing sources and higher percentages because it combines create denominations transactions belonging in the same wallet. To get a list of example analysis pages you can run this query. It fetches privatesend transactions between blocks 850000 and 855000 because the old analysis isn't available for recent transactions.

I want to emphasize that it's not possible for someone to determine the number of rounds used by just analyzing the blockchain. Different inputs can even use a different number if you interrupt the mixing process. So the results are more like leads rather than something conclusive.

 

[Antti Kaikkonen]

I found a bug which caused mixing transactions to affect the guesstimated wallets feature although mixing transactions should have been ignored.

I'm going to remove all of the guesstimated wallets and then recompute them. This will cause a few hours of downtime. Edit: Done

I will also need to recompute the PrivateSend analysis results which were using the guesstimated wallets feature.

 

[Antti Kaikkonen]

I found another bug where a trasaction included in an orphaned block re-appearing in the mempool caused the transaction to be duplicated. This has so far happened to only 1 transaction but it already scewed the average transaction fee chart for August 8th. Currently reprocessing the data and I will edit this post when fixed. Edit: done

 

[Antti Kaikkonen]

Update

• Added two new transaction types "PrivateSend make collateral inputs" and "PrivateSend collateral payment"
  • Added them to the database. Example queries: Number of 'make collateral inputs' transaction withing the last 24 hours and Transaction ids of 'PrivateSend collateral payment' transactions occoured in the last hour
  • Added them to the graph visualization
  • Added them to the block explorer

Here is an image of the new icons used:

 

[Antti Kaikkonen]

Recently I started making a full nodes + masternodes page for DashRadar. Here is a demo of the current progress: https://antti-kaikkonen.github.io/DashradarNodesDemo/

It uses the CoinCrawler full node crawler that was recently developed my be thanks to the sponsor @Technologov .

Any suggestions welcome!

 

[Antti Kaikkonen]

Here is the number of unique visitors for July, August and September

I'm using an open-source self hosted analytics software called Matomo, so that your information is not leaked to Google or other third parties.

 

[jeffh]

@Antti Kaikkonen If you want to pull more information about the masternodes here, feel free to run a query against this data from Dash Intel:

https://api.dashintel.org/dash_masternode_insights. We have some additional information that we're pulling from paid API queries and saving in our database that you might find useful. Keep in mind, I ignore non-ENABLED masternodes in most cases.

 

[jeffh]

@colofi this was some testing done by @DeltaEngine for a stress test a few months ago and then more recently in preparation for a network stress-test (theoretical max tx/s to beat any production blockchain record) scheduled to happen around the 12.4 launch.

 

[Antti Kaikkonen]

Nodes page added to dashradar: https://dashradar.com/nodes

There is still some things that could be added/improved:

• Make the map size dynamic depending on screen resolution
  
• Add an option to filter active/inactive nodes in the "All nodes" tab
  
• (Individual node page by clicking the ip address)
• (Add historical charts for country/version/organization distribution)
• (Add link to collateral txid/address for masternodes)
• (Show voting data for masternodes)

 

[Antti Kaikkonen]

Changed the default url (https://dashradar.com) from the graph visualization to a simple landing page containing links to explorer, graph, charts and nodes pages. In the future I might add more content to the landing page to make it more like a dashboard.

Also upgraded from Angular 6 to Angular 7.

 

[Gaby_64]

how long does it take for a new node to appear in the nodes list?

I have been running a non masternode node for 2 days now.

 

[Antti Kaikkonen]

how long does it take for a new node to appear in the nodes list?

I have been running a non masternode node for 2 days now.

It only lists nodes that accept new connections.

If you are running the node at home then it might be behind a NAT router. In that case you should forward port 9999 from the router to the LAN IP address of the computer running the node.

It's also possible that there could be something wrong with the configuration of the node (dash.conf).

You can check if your node is accepting connections with the node checker found in http://nodes.blakeanderson.ca/networks/dash.

 

[qwizzie]

Any chance you can include charts for mining hashrate & mining difficulty & mining pools spread?

 

[Antti Kaikkonen]

I recently updated the website to use angular universal server side rendering. This means that the first page the user is loading is rendered to HTML on the server and after that it will continue to function as a single page application. It can make the initial page load slightly faster but the main benefit is that it makes the website crawlable by search engines.

Also moved the navigation from the top to the side. This solution should work better for mobile users and allows fitting more navigation elements in the future.

Any chance you can include charts for mining hashrate & mining difficulty & mining pools spread?

I think that is a good idea but I'm not sure when I will be able to do that.

 

[qwizzie]

I recently updated the website to use angular universal server side rendering. This means that the first page the user is loading is rendered to HTML on the server and after that it will continue to function as a single page application. It can make the initial page load slightly faster but the main benefit is that it makes the website crawlable by search engines.

Also moved the navigation from the top to the side. This solution should work better for mobile users and allows fitting more navigation elements in the future.


I think that is a good idea but I'm not sure when I will be able to do that.

No pressure, no hurry.

Good to hear about your website improvements.

 

[Antti Kaikkonen]

@qwizzie It's currently possible to get generate those charts using the data explorer but it can be quite slow for long time periods. Adding them to the charts page would require storing a running total of the difficulty so that average difficulty of a day can be computed very quickly. Hashrate by pool would require even more changes.

Here are some examples using the data explorer:

• Daily hashrate last 3 months
  
• Hourly hashrate last week

You can modify the queries to get more data but if you make the period too long then the query will not finish because the is a time limit (1 minute if I remember correctly).

 

[qwizzie]

@qwizzie It's currently possible to get generate those charts using the data explorer but it can be quite slow for long time periods. Adding them to the charts page would require storing a running total of the difficulty so that average difficulty of a day can be computed very quickly. Hashrate by pool would require even more changes.

Here are some examples using the data explorer:

• Daily hashrate last 3 months
  
• Hourly hashrate last week

You can modify the queries to get more data but if you make the period too long then the query will not finish because the is a time limit (1 minute if I remember correctly).

i like the line chart of the daily hashrate last 3 months, it would fit well with the other line charts i think.
And maybe think of ways implementing mining difficulty and pools for a later date ? Or skip if its getting to be too much of a hassle

 

[Antti Kaikkonen]

I have had some issues with the Neo4j database that this project is using to store the blockchain:

• Poor write performance
• Issues with the cost planner sometimes generating highly inefficient query execution plans
• Community edition lacking some useful features
• Other issues

I'm probably not going to build any new features on top of Neo4j and I might start replacing some or all of the current features using other databases.

I'm currently building an improved version of the address clustering a.k.a guesstimated wallets feature. In the new version users will be able to browse an ordered list of the wallet's transactions. Wallet balance after each transaction will also be available as well as a historical balance chart.

 

[Antti Kaikkonen]

Update

Added the new 0.001 DASH PrivateSend denomination.

 

[qwizzie]

Quick question about two charts :

Transactions per day
Mixing transactions per day

Is the "mixing transactions per day" data included or excluded from the "transactions per day" chart ?
I ask this because i wonder if a lot of mixing could inflate the transactions per day chart.

 

[Antti Kaikkonen]

Quick question about two charts :

Transactions per day
Mixing transactions per day

Is the "mixing transactions per day" data included or excluded from the "transactions per day" chart ?
I ask this because i wonder if a lot of mixing could inflate the transactions per day chart.

Transactions per day includes all transactions including mixing transactions, privatesend transactions and create denominations transactions.

 

[tiger-HZX]

Oh sorry about that. I just moved to a diffent city so I couln't work for 2 days.. I will fix asap.

Edit: It got stuck at block 905186 which has 4785 transactions. I probably need to change the code to save in a single database transaction one transaction at a time instead of one block at a time. Meanwhile I'm trying to give the database more memory to work with to get it rolling again.

I found that in block 206980,there are two mixing trx, since their trx fee is 0, but dashradar recognize them as create denomination trx, Is it a bug? If I am wrong, is there something about mixing trx I don't know?

 

[thephez]

I found that in block 206980,there are two mixing trx, since their trx fee is 0, but dashradar recognize them as create denomination trx, Is it a bug? If I am wrong, is there something about mixing trx I don't know?

In 2015 when that tx happened it was possible to send any tx with a fee of 0. Also, the txs I looked at (mentioned here) cannot be mixing txs. One of them does not have an equal number of inputs/outputs. The other cannot be for multiple reasons including - 1. too many inputs/outputs, and 2. Not all inputs/outputs have the same value (unless the PS rules were different back then (@UdjinM6 ?).

 

[tiger-HZX]

In 2015 when that tx happened it was possible to send any tx with a fee of 0. Also, the txs I looked at (mentioned here) cannot be mixing txs. One of them does not have an equal number of inputs/outputs. The other cannot be for multiple reasons including - 1. too many inputs/outputs, and 2. Not all inputs/outputs have the same value (unless the PS rules were different back then (@UdjinM6 ?).

Thanks for your reply, but there are still are something I did not understand. why a PrivateSend tx can not tcontains too many inputs/outputs, if there is a rule, what is the limit of inputs, I remember there are at most 200 inputs/outputs in a tx, but I didn't hear about the inputs/outputs limits of PS tx.

In the follow links(please replace 。 with .) , we can see someone said that inputs/outputs in dash's PrivateSend tx can be different denominations once,
https://forum。bitrad。io/index。php?thread/953-what-is-darksend-and-how-to-use-it/ , maybe it is wrong, maybe it is right.

There is another msg, the 5 PS denominations was introduced in protocal version 72013, it means there may be another rules about PrivateSend before 72013, I searchd a lot and did not find any information about it.

I am still confused, if you know, please tell me a detail about it. @thephez @UdjinM6

 

[Antti Kaikkonen]

I found that in block 206980,there are two mixing trx, since their trx fee is 0, but dashradar recognize them as create denomination trx, Is it a bug? If I am wrong, is there something about mixing trx I don't know?

Dashradar does not currently recognize old mixing transactions where multiple different denominations in the same mixing transaction was allowed.

 

[tiger-HZX]

Dashradar does not currently recognize old mixing transactions where multiple different denominations in the same mixing transaction was allowed.

ok, I know. Thx