Multiple Network Interfaces on T2.medium - A small bounty challenge!

stonehedge

Well-known Member
Foundation Member
Jul 31, 2014
696
333
233
10 DRK to the person who can solve this Linux networking problem for me!

I have six masternodes running on one T2.medium using chaeplin's excellent guide which can be found here

https://darkcointalk.org/threads/ec2-multiple-remote-nothing-mn-max-5.1660/

Both CPU and network resources are underutilised and I have plenty of spare elastic IPs in the same region so I would like to run a few more masternodes on the same instance.

The problem is that you can only bind a maximum of 6 private IPs to a T2.medium network interface. I have tried creating a second network interface and attaching it to the same instance but unfortunately it appears that by default all traffic received by the second interface (eth1) is forwarded to and dropped by eth0.

This is how I want a T2.medium to work:

¦ User Proc ¦
nm01 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm02 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm03 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm04 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm05 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm06 darkcoind <-> eth0 <-> PrivIP <-> EIP
-------------------------------------------------------------- theoretical networking wall!
nm07 darkcoind <-> eth1 <-> PrivIP <-> EIP
nm08 darkcoind <-> eth1 <-> PrivIP <-> EIP
nm09 darkcoind <-> eth1 <-> PrivIP <-> EIP
nm10 darkcoind <-> eth1 <-> PrivIP <-> EIP
nm11 darkcoind <-> eth1 <-> PrivIP <-> EIP

Can anything be done with static routes to allow this to happen? I think need to learn how to enable eth1 in linux and then set up routes to ensure that traffic is handled symmetrically. I.e traffic received in to eth1 is processed by the appropriate user darkcoind instance and routed back out through Eth1 and subsequently the correct IP.

You'll probably want to read chaeplin's guide (linked above).

So...any ideas?

First person to provide info that actually helps me solve it gets the bounty.
 

karisu

Member
Jun 30, 2014
70
26
58

chaeplin

Active Member
Core Developer
Mar 29, 2014
749
356
133
Use 1 interface. With 2 interface, I think you have to use iproute2 utility.
iproute2 is more complicated, so one interface is best practice.

If you can get more EIP, add more private in EC2 console.

At instance creation, add more private



At running instance, using 'Network Interfces' menu - 'Action' - 'Manage Private IP'
 
Last edited by a moderator:

stonehedge

Well-known Member
Foundation Member
Jul 31, 2014
696
333
233
Thanks karisu , I'll have a read. From a best practice standpoint I appreciate your concerns about having too many nodes on the same instance.

chaeplin, if you try to launch a T2.medium instance with more than 6 IPs the launch fails. Six is the maximum per network interface.
 
  • Like
Reactions: karisu

flare

Administrator
Dash Core Team
Moderator
May 18, 2014
2,286
2,404
1,183
Germany
Got it working using two subnets

Code:
[email protected]:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 12:22:57:26:7b:11
          inet addr:10.0.1.30  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::1022:57ff:fe26:7b11/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:760 errors:0 dropped:0 overruns:0 frame:0
          TX packets:646 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:74494 (74.4 KB)  TX bytes:88547 (88.5 KB)

eth1      Link encap:Ethernet  HWaddr 12:0b:e8:fd:d7:dd
          inet addr:10.0.2.201  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::100b:e8ff:fefd:d7dd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:29 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1600 (1.6 KB)  TX bytes:1315 (1.3 KB)

[...]

[email protected]:~$ ip route show
default via 10.0.1.1 dev eth0
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.30
10.0.2.0/24 dev eth1  proto kernel  scope link  src 10.0.2.201

[...]

[email protected]:~$ ip rule show
0:      from all lookup local
1000:   from 10.0.2.0/24 lookup eth1_rt
32766:  from all lookup main
32767:  from all lookup default

[...]


[email protected]:~$ curl --interface 10.0.1.30 icanhazip.com
54.88.128.189
[email protected]:~$ curl --interface 10.0.2.201 icanhazip.com
54.88.41.124
 

stonehedge

Well-known Member
Foundation Member
Jul 31, 2014
696
333
233
Got it working using two subnets

Code:
[email protected]:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 12:22:57:26:7b:11
          inet addr:10.0.1.30  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::1022:57ff:fe26:7b11/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:760 errors:0 dropped:0 overruns:0 frame:0
          TX packets:646 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:74494 (74.4 KB)  TX bytes:88547 (88.5 KB)

eth1      Link encap:Ethernet  HWaddr 12:0b:e8:fd:d7:dd
          inet addr:10.0.2.201  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::100b:e8ff:fefd:d7dd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:29 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1600 (1.6 KB)  TX bytes:1315 (1.3 KB)

[...]

[email protected]:~$ ip route show
default via 10.0.1.1 dev eth0
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.30
10.0.2.0/24 dev eth1  proto kernel  scope link  src 10.0.2.201

[...]

[email protected]:~$ ip rule show
0:      from all lookup local
1000:   from 10.0.2.0/24 lookup eth1_rt
32766:  from all lookup main
32767:  from all lookup default

[...]


[email protected]:~$ curl --interface 10.0.1.30 icanhazip.com
54.88.128.189
[email protected]:~$ curl --interface 10.0.2.201 icanhazip.com
54.88.41.124
So you did get bored then flare? Looks good to me. I think we might have a winner! To be confirmed when/if I get it working...

vertoe I'm happy to pull a wiki together for this (if chaeplin is happy for me to link to his guides) however we really shouldn't be advising this technique as a matter of course. I just want to make full use of the instances that I have running for reasons of value for money.
 
  • Like
Reactions: flare

stonehedge

Well-known Member
Foundation Member
Jul 31, 2014
696
333
233
I want to give up my day job and become a full time darkcoiner (ist?).

This is the only fun I have all day.
 
  • Like
Reactions: flare

stonehedge

Well-known Member
Foundation Member
Jul 31, 2014
696
333
233
Yes me too. I wish someone could pay me for this fulltime.
I have strict rules of engagement at home. My wife is away on business for most of next month so I will have some more time to devote to the cause.

Its just a shame that Darkcoin doesn't need Zenworks, Solarwinds, Cisco Nexus or HP SAN consultancy :D
 

crowning

Well-known Member
May 29, 2014
1,414
1,997
183
Alpha Centauri Bc
Yes me too. I wish someone could pay me for this fulltime.
Up your Masternodes number to 100 and you can (almost) live from the payments.

Of course you would have to ...<cough>...SELL...<cough>... some coins to live...what of course will never happen :tongue:
 

crowning

Well-known Member
May 29, 2014
1,414
1,997
183
Alpha Centauri Bc
Got it working using two subnets
That. Whenever you add more network interfaces, create a new additional subnet and set the default route for this subnet to the corresponding ethx of your choice. After that apply chaeplin's excellent HowTo to this subnet.

Not sure if this possible with those Amazon EC-whatever thingies, though.....
 

stonehedge

Well-known Member
Foundation Member
Jul 31, 2014
696
333
233
Ok so flare has spent hours helping me tonight. He has taught me loads about VPC and subnetting and I have eth1 up and running fine in the instance.

I believe that I have rc.local and iptables set up correctly however when I run the following under account nm07 "curl ipecho.net/plain ; echo" I eventually get a timeout.

If I add -o eth1 in the postrouting in iptables "curl ipecho.net/plain ; echo" returns the primary public IP of eth0.

I am assuming this is a routing issue. How do I configure the instance to receive and send via eth1 via one particular IP?

If I run this command I receive the correct public IP

[email protected]:~$ curl --interface 10.0.1.216 icanhazip.com
I can only assume that all traffic that comes into eth1 is routed out of eth0 which obvs will not do. Can anybody help me set up my routes?
 

stonehedge

Well-known Member
Foundation Member
Jul 31, 2014
696
333
233
Success! flare is the bounty winner as he held my hand and taught me how to solve the problem step by step and even drew me pictures. Thanks to all who helped.
 
  • Like
Reactions: crowning and flare

Propulsion

The buck stops here.
Feb 26, 2014
1,008
468
183
Dash Address
XerHCGryyfZttUc6mnuRY3FNJzU1Jm9u5L
Success! flare is the bounty winner as he held my hand and taught me how to solve the problem step by step and even drew me pictures. Thanks to all who helped.
Wow he drew you pictures?! That is awesome.
 

stonehedge

Well-known Member
Foundation Member
Jul 31, 2014
696
333
233
BTW, as chaeplin has said already it is not recommended to use multiple network interfaces.

AWS routes asymmetrically by default which meant that all outgoing traffic, regardless of the NIC that it arrived on, exits via eth0 (if it exits at all).

To get it working you need to set up static routes for each IP that you have on eth1. Getting eth1 enabled within Linux is not exactly beginners Linux either.