• Forum has been upgraded, all links, images, etc are as they were. Please see Official Announcements for more information

Multiple Network Interfaces on T2.medium - A small bounty challenge!

stonehedge

Well-known member
Foundation Member
10 DRK to the person who can solve this Linux networking problem for me!

I have six masternodes running on one T2.medium using chaeplin's excellent guide which can be found here

https://darkcointalk.org/threads/ec2-multiple-remote-nothing-mn-max-5.1660/

Both CPU and network resources are underutilised and I have plenty of spare elastic IPs in the same region so I would like to run a few more masternodes on the same instance.

The problem is that you can only bind a maximum of 6 private IPs to a T2.medium network interface. I have tried creating a second network interface and attaching it to the same instance but unfortunately it appears that by default all traffic received by the second interface (eth1) is forwarded to and dropped by eth0.

This is how I want a T2.medium to work:

¦ User Proc ¦
nm01 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm02 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm03 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm04 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm05 darkcoind <-> eth0 <-> PrivIP <-> EIP
nm06 darkcoind <-> eth0 <-> PrivIP <-> EIP
-------------------------------------------------------------- theoretical networking wall!
nm07 darkcoind <-> eth1 <-> PrivIP <-> EIP
nm08 darkcoind <-> eth1 <-> PrivIP <-> EIP
nm09 darkcoind <-> eth1 <-> PrivIP <-> EIP
nm10 darkcoind <-> eth1 <-> PrivIP <-> EIP
nm11 darkcoind <-> eth1 <-> PrivIP <-> EIP

Can anything be done with static routes to allow this to happen? I think need to learn how to enable eth1 in linux and then set up routes to ensure that traffic is handled symmetrically. I.e traffic received in to eth1 is processed by the appropriate user darkcoind instance and routed back out through Eth1 and subsequently the correct IP.

You'll probably want to read chaeplin's guide (linked above).

So...any ideas?

First person to provide info that actually helps me solve it gets the bounty.
 
Use 1 interface. With 2 interface, I think you have to use iproute2 utility.
iproute2 is more complicated, so one interface is best practice.

If you can get more EIP, add more private in EC2 console.

At instance creation, add more private
0EDYE93.png



At running instance, using 'Network Interfces' menu - 'Action' - 'Manage Private IP'
Wd1uDbt.png
 
Last edited by a moderator:
Thanks karisu , I'll have a read. From a best practice standpoint I appreciate your concerns about having too many nodes on the same instance.

chaeplin, if you try to launch a T2.medium instance with more than 6 IPs the launch fails. Six is the maximum per network interface.
 
Got it working using two subnets

Code:
ubuntu@ip-10-0-1-30:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 12:22:57:26:7b:11
          inet addr:10.0.1.30  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::1022:57ff:fe26:7b11/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:760 errors:0 dropped:0 overruns:0 frame:0
          TX packets:646 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:74494 (74.4 KB)  TX bytes:88547 (88.5 KB)

eth1      Link encap:Ethernet  HWaddr 12:0b:e8:fd:d7:dd
          inet addr:10.0.2.201  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::100b:e8ff:fefd:d7dd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:29 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1600 (1.6 KB)  TX bytes:1315 (1.3 KB)

[...]

ubuntu@ip-10-0-1-30:~$ ip route show
default via 10.0.1.1 dev eth0
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.30
10.0.2.0/24 dev eth1  proto kernel  scope link  src 10.0.2.201

[...]

ubuntu@ip-10-0-1-30:~$ ip rule show
0:      from all lookup local
1000:   from 10.0.2.0/24 lookup eth1_rt
32766:  from all lookup main
32767:  from all lookup default

[...]


ubuntu@ip-10-0-1-30:~$ curl --interface 10.0.1.30 icanhazip.com
54.88.128.189
ubuntu@ip-10-0-1-30:~$ curl --interface 10.0.2.201 icanhazip.com
54.88.41.124
 
Got it working using two subnets

Code:
ubuntu@ip-10-0-1-30:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 12:22:57:26:7b:11
          inet addr:10.0.1.30  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::1022:57ff:fe26:7b11/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:760 errors:0 dropped:0 overruns:0 frame:0
          TX packets:646 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:74494 (74.4 KB)  TX bytes:88547 (88.5 KB)

eth1      Link encap:Ethernet  HWaddr 12:0b:e8:fd:d7:dd
          inet addr:10.0.2.201  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::100b:e8ff:fefd:d7dd/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:29 errors:0 dropped:0 overruns:0 frame:0
          TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1600 (1.6 KB)  TX bytes:1315 (1.3 KB)

[...]

ubuntu@ip-10-0-1-30:~$ ip route show
default via 10.0.1.1 dev eth0
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.30
10.0.2.0/24 dev eth1  proto kernel  scope link  src 10.0.2.201

[...]

ubuntu@ip-10-0-1-30:~$ ip rule show
0:      from all lookup local
1000:   from 10.0.2.0/24 lookup eth1_rt
32766:  from all lookup main
32767:  from all lookup default

[...]


ubuntu@ip-10-0-1-30:~$ curl --interface 10.0.1.30 icanhazip.com
54.88.128.189
ubuntu@ip-10-0-1-30:~$ curl --interface 10.0.2.201 icanhazip.com
54.88.41.124

So you did get bored then flare? Looks good to me. I think we might have a winner! To be confirmed when/if I get it working...

vertoe I'm happy to pull a wiki together for this (if chaeplin is happy for me to link to his guides) however we really shouldn't be advising this technique as a matter of course. I just want to make full use of the instances that I have running for reasons of value for money.
 
I want to give up my day job and become a full time darkcoiner (ist?).

This is the only fun I have all day.
 
Yes me too. I wish someone could pay me for this fulltime.

I have strict rules of engagement at home. My wife is away on business for most of next month so I will have some more time to devote to the cause.

Its just a shame that Darkcoin doesn't need Zenworks, Solarwinds, Cisco Nexus or HP SAN consultancy :D
 
Yes me too. I wish someone could pay me for this fulltime.

Up your Masternodes number to 100 and you can (almost) live from the payments.

Of course you would have to ...<cough>...SELL...<cough>... some coins to live...what of course will never happen :tongue:
 
Got it working using two subnets

That. Whenever you add more network interfaces, create a new additional subnet and set the default route for this subnet to the corresponding ethx of your choice. After that apply chaeplin's excellent HowTo to this subnet.

Not sure if this possible with those Amazon EC-whatever thingies, though.....
 
Ok so flare has spent hours helping me tonight. He has taught me loads about VPC and subnetting and I have eth1 up and running fine in the instance.

I believe that I have rc.local and iptables set up correctly however when I run the following under account nm07 "curl ipecho.net/plain ; echo" I eventually get a timeout.

If I add -o eth1 in the postrouting in iptables "curl ipecho.net/plain ; echo" returns the primary public IP of eth0.

I am assuming this is a routing issue. How do I configure the instance to receive and send via eth1 via one particular IP?

If I run this command I receive the correct public IP

nm07@ip-10-0-0-227:~$ curl --interface 10.0.1.216 icanhazip.com

I can only assume that all traffic that comes into eth1 is routed out of eth0 which obvs will not do. Can anybody help me set up my routes?
 
Back
Top