Questions about InstantSend behavior in a flooded network

t0dd

Active Member
Mar 21, 2016
150
132
103
keybase.io
Dash Address
XyxQq4qgp9B53QWQgSqSxJb4xddhzk5Zhh
I'm working on a DashForceNews article that discusses scale from a user/merchant perspective. Capacity, throughput, latency, response times... throughout the phases of an interaction between a payer and a payee...

The stated transaction "speed" of Dash is approximately 48 transactions per second. This is a raw "included in a block" metric with loads of variables at play and ignores a lot of other factors involved in a transaction and the perception to a customer and merchant.

What I am curious about is InstantSend. How does it fare when pushed to scale? Do we have some ideas surrounding this?

The unique thing about InstantSend (IS) is that it combines the world of blockchain (a discrete synchronous transaction heartbeat) with the world of traditional client-server (client-cloud?): "as fast as you can move data"; continuous; asynchronous. Scaling IS is more similar to traditional scaling any other network. You have anticipated load... the ability to form a quorum of 10 masternodes, and the ability to send and receive data with the client.

So, the first 5 confirmation equivalents are asynchronous and operate at masternode-network-speed. But it is that last 6 confirmation post-lock-in that then requires the transaction to settle to the blockchain. Off-chain then on-chain.

But the transaction is locked. At least amongst the masternodes. But what if the network is flooded? Blocks are filled, people are queuing up IS payments that are waiting to be included in a block. Those payments still are guaranteed...

But is there a breaking point? I know there is a 15 second timeout, but that really only addresses response time between client and server. Not a filled pipeline of unconfirmed IS transactions.

Anyone have thoughts on this? The article is not going to be utterly low level, but I want to make sure I am characterizing things appropriately. Plus this is a really interesting topic.

Thanks! -t0dd (in discord, keybase as toddwarner, etc. etc.)

...
PS. For those interested in reading about some of the mechanics of all this, we have outstanding documentation:
 
Last edited:
  • Like
Reactions: Dandy and UdjinM6

thephez

Member
Dash Core Team
Jan 23, 2016
113
49
78
I fell down a rabbit hole looking into this for a few hours. It's a bit off-topic from what you were asking, but I thought it was interesting :).

The amount of bandwidth taken up by an InstantSend transaction compared to a normal transaction is substantial. This is part of why they cost more, right? The number of network messages generated by an IS transaction is:
Code:
1 + (Number of Inputs) * (30)
Broken down:
  • 1 - ix message
  • Numberof inputs * 10 - TxlVote Inventory message
  • Number of inputs * 10 - GetData message (to get TxlVote)
  • Number of inputs * 10 - TxlVote message
Using some rough approximations for the size of those messages (129 bytes for txlvote inventory and getdata, 262 bytes for txlvote, 147 bytes per input/34 bytes per output), the amount of traffic would be more than an order of magnitude greater than the size of the InstantSend transaction (more like 20x-40x if my rough calculations are anywhere close to accurate).

In practice, I sent an InstantSend transaction with 100 inputs (the warning threshold currently defined in Dash Core) on testnet to see what would happen. The transaction was ~14.8 KB and I was expecting around 500-600 KB of traffic (capture with Wireshark). The actual amount of traffic recorded by Wireshark was about 1.4 MB - nearly 3 times what I expected. I repeated it a second time with similar results. Not sure if this is due to very faulty math on my end or errors / retransmissions on the network.

I should mention - neither one of the transactions actually was a successful InstantSend. Both choked with ~700 of 1000 signatures.

Another interesting observation can be seen in the IO graph from Wireshark. InstantSend puts a sudden, heavy load on the network (note the chart is on a log scale - Y-axis is Bytes).

InstantSend_100_inputs.png
The masternodes have to respond quickly because the quorum votes have to be received within 15 seconds of each other or the IS transaction will timeout. I suspect this is why my transactions failed - it is difficult to see here, but there was actually a second burst of txlvote inventories received outside the 15 second window.

I really have no idea how it actually will scale, but I think it is good to contemplate. Given the forward thinking that we see from Dash Core, I am guessing this stuff has been discussed in detail.
 
  • Like
Reactions: UdjinM6

thephez

Member
Dash Core Team
Jan 23, 2016
113
49
78
But what if the network is flooded? Blocks are filled, people are queuing up IS payments that are waiting to be included in a block. Those payments still are guaranteed...

If the network is flooded by quantity of transactions, presumably IS transactions would be the first confirmed due to their higher fee. If fee pressure increased too much, at some point all transactions would potentially be IS because if you have to pay a fee large enough for IS why wouldn't you try to use it. There would be no downside to trying because if it failed, it would just become a standard Tx.

But is there a breaking point? I know there is a 15 second timeout, but that really only addresses response time between client and server. Not a filled pipeline of unconfirmed IS transactions.
Given the amount of network traffic created by IS transactions, my guess is that they would simply start failing to confirm (as InstantSend) due to latency rather than ending up with a pipeline full of unconfirmed IS transactions. I think InstantSend is most latency-sensitive part of the network.

BUT... I am not in any way an expert in these things, so this might all be wrong. Your speculation is as good as mine :)
 

Dandy

Member
Mar 1, 2017
276
99
88
44
Belgrade, Serbia
This is something I was thinking about, too.
I would like to hear some official response from the core team.
Also, how would increased minimal masternode specs impact this?
 

thephez

Member
Dash Core Team
Jan 23, 2016
113
49
78
Also, how would increased minimal masternode specs impact this?
For the scenario I ran, it is not really possible to tell. There are a lot of moving parts, so the issue could have been quite a few things. I am not sure what the breakdown would be for the load: network capacity vs. MN processing capability vs. ??? Also, my scenario was pretty unrealistic. It would be more likely to see a lot of small transactions (1-5 inputs) originating in different places than a few giant ones (100 inputs) concentrated at one location. It would also be interesting to see how doing this on Mainnet would work, but I do not want to pay the fees :).
 

t0dd

Active Member
Mar 21, 2016
150
132
103
keybase.io
Dash Address
XyxQq4qgp9B53QWQgSqSxJb4xddhzk5Zhh
Thanks for all this. A lot to think about. This is why they make simulation languages. This is a particularly tough problem to really understand. Maybe a scheduled stresstest would be beneficial. Doing that against testnet would be subpar though because testnet is so... desolate. It's too artificial. Great for testing functionality, not great for testing real-world use modeling.

I think this requires someone who really understands continuous simulation modeling. A CloudSim expert would be a nice pickup for the community. I bet there are not a ton of these, though. I bet the ASU team could make this part of their scale testing.

Anyway, this is enough for me to not utterly bolo the article I am going to write. It is really just discussing the reality of today and where it could go tomorrow and some of the challenges that need to be addressed. @thephez ... really great work!

I would love to hear from @flare or someone to see what we have done so far to at least understand some of the variables at play that have been tested.
 

Barrett Davis

Member
Nov 28, 2017
41
22
48
31
I fell down a rabbit hole looking into this for a few hours. It's a bit off-topic from what you were asking, but I thought it was interesting :).

The amount of bandwidth taken up by an InstantSend transaction compared to a normal transaction is substantial. This is part of why they cost more, right? The number of network messages generated by an IS transaction is:
Code:
1 + (Number of Inputs) * (30)
Broken down:
  • 1 - ix message
  • Numberof inputs * 10 - TxlVote Inventory message
  • Number of inputs * 10 - GetData message (to get TxlVote)
  • Number of inputs * 10 - TxlVote message
Using some rough approximations for the size of those messages (129 bytes for txlvote inventory and getdata, 262 bytes for txlvote, 147 bytes per input/34 bytes per output), the amount of traffic would be more than an order of magnitude greater than the size of the InstantSend transaction (more like 20x-40x if my rough calculations are anywhere close to accurate).

In practice, I sent an InstantSend transaction with 100 inputs (the warning threshold currently defined in Dash Core) on testnet to see what would happen. The transaction was ~14.8 KB and I was expecting around 500-600 KB of traffic (capture with Wireshark). The actual amount of traffic recorded by Wireshark was about 1.4 MB - nearly 3 times what I expected. I repeated it a second time with similar results. Not sure if this is due to very faulty math on my end or errors / retransmissions on the network.

I should mention - neither one of the transactions actually was a successful InstantSend. Both choked with ~700 of 1000 signatures.

Another interesting observation can be seen in the IO graph from Wireshark. InstantSend puts a sudden, heavy load on the network (note the chart is on a log scale - Y-axis is Bytes).

View attachment 5963
The masternodes have to respond quickly because the quorum votes have to be received within 15 seconds of each other or the IS transaction will timeout. I suspect this is why my transactions failed - it is difficult to see here, but there was actually a second burst of txlvote inventories received outside the 15 second window.

I really have no idea how it actually will scale, but I think it is good to contemplate. Given the forward thinking that we see from Dash Core, I am guessing this stuff has been discussed in detail.
Good job man. This is quite extensive and impressive.
 

t0dd

Active Member
Mar 21, 2016
150
132
103
keybase.io
Dash Address
XyxQq4qgp9B53QWQgSqSxJb4xddhzk5Zhh
# Interesting point: InstantSend funds received are treated no differently than any other funds

Photosynthesis on discord brought this up as a surprise in user experience. The Dash Wallet will recognize an InstantSent payment as having 6 confirmations in its UI. But this is deceiving. It does build confidence to the user that they "for sure" have those funds. That's great. BUT! Those funds can't be InstandSent out again until really 6 "real" confirmations. This is not obvious -- you have 5-pseudo-confirmations and then the one 6th-pseudo-confirmation, but really that is just really the 1st. Confusing to users! I poked HashEngineering to see if there is a way in the UI to make this more recognizable (differently shaded pie pieces or somesuch) but it is a user experience challenge. With the CoPay wallet, there is no indication at all... which kinda loses the point of InstantSend.

So...

My questions:
  1. I assume the reason that 6 full "real" confirmations have to occur on the blockchain is to ensure that there is no chance those funds are in blocks on the wrong branch, right?
  2. Do the outputs (and inputs) involved in an InstantSend contain metadata that says "I was part of an InstantSend!" ... which would be great for UI developmers, but I suppose wallets could track this themselves, since it is only relevant for that transaction. I.e., How does a receiving wallet know it is an instantsent transaction? If there were metadata, someone would have to strip it out since it is relevant only for that transaction. Is there metadata?
As long as I have been using Dash, somehow this nuance escaped me. I suppose I never tried to InstantSend funds that were just InstantSent to me. Folks that have plenty of Dash in their wallet probably don't often run into this because when they InstantSend some funds, it will just pull from other outputs.
 
  • Like
Reactions: GrandMasterDash

solarguy

Active Member
Mar 15, 2017
867
413
133
60
Sending an InstantSend transaction locks the transaction in a second or two. This is the magic that prevents double spends and makes it possible for Dash to work like money in real life.

However, the receiving party can't spend the Dash (in any way, InstantSend or regular) until that transaction gets processed in a regular block. But, that should hardly ever take longer than a minute or two unless Dash has failed in its job to scale properly.

that is my current understanding at any rate.
 

Photosynthesis

New Member
Dec 15, 2017
12
2
3
The problem that manifested itself is that the received IS funds could indeed be regular-spent after being mined into a block but could not be instant-sent until 6 blocks had been mined.
 

thephez

Member
Dash Core Team
Jan 23, 2016
113
49
78
Do the outputs (and inputs) involved in an InstantSend contain metadata that says "I was part of an InstantSend!" ... which would be great for UI developmers, but I suppose wallets could track this themselves, since it is only relevant for that transaction. I.e., How does a receiving wallet know it is an instantsent transaction? If there were metadata, someone would have to strip it out since it is relevant only for that transaction. Is there metadata?
No, there's nothing on the blockchain to indicate a transaction is InstantSend (or PrivateSend for that matter). A transaction is just a transaction regardless of how it was sent. To fully validate an IS transaction, a wallet would have to look for the `ix` message (InstantSend transaction) AND watch for the masternode votes locking each input to the transaction. And really, it would need to also validate that the masternode votes were legitimate (masternode signatures are correct and from only masternodes in the proper quorum for that input).

I suppose data could be added to the transaction to indicate this, but there's no need to include that in the blockchain because it's only relevant for a short period of time.
 

t0dd

Active Member
Mar 21, 2016
150
132
103
keybase.io
Dash Address
XyxQq4qgp9B53QWQgSqSxJb4xddhzk5Zhh
And that short period of time is the killer feature. It solves the confidence gap problem.... which will be in my article.
 

t0dd

Active Member
Mar 21, 2016
150
132
103
keybase.io
Dash Address
XyxQq4qgp9B53QWQgSqSxJb4xddhzk5Zhh
So, this is something that needs to be addressed from a usability point of view for both the CoPay wallet and for the Evolution UI.
People already are, and will increasingly be (as adoption rises), confused in how InstantSend versus a traditional transaction work.

  1. For a traditional transaction, you can spend it upon a single confirmation.
  2. For an InstantSend transaction, you can only spend it after 6 confirmations.
  3. We need to stop calling the InstantSend lockin a 5 confirmation equivalent. It isn't. It really isn't. It's an asynchronous 0-confirmation (pre-confirmation) locked in trade that has cleared but not yet setted. And that settlement is still upon the 1st confirmation.
  4. Received InstantSend payments need to be obvious from a UI perspective. There should be no guessing.
  5. When you try to send an InstantSend payment, and you don't have enough outputs with enough confirmations, it has to be absolutely obvious why this is. AND you have to give the alternative to a traditional payment. And link to contextual help or some documentation or something for further explanation.
Just some ideas I jotted down. Thanks for reading.
Hmm... maybe this should be in the CoPay wallet thread (since it does InstantSend UI wrong) or another thread entirely.
 
Last edited: