CentOS 5上基于源的策略路由和NAT(DNAT / SNAT)或多WAN


4

最初发布在Unix和Linux上,但没有人能够回答,因此在这里迁移问题:

我的问题是关于具有2个WAN和带有负载平衡的LAN(NAT)端口的CentOS 5上基于源的策略路由,但是首先要进行一些说明,然后再开始描述问题。

我知道这个话题已经在堆栈交换中多次出现,并且似乎排名前5位的答案是(从最大到最小)。

  1. 禁用rp_filter
  2. 基于标记/连接标记的策略路由
  3. 基于IP的策略路由(添加更多IP)
  4. 安装pfSense,Shorewall,Ubuntu ?、等等...
  5. 购买昂贵的Cisco / 3com / Juniper / Etc ...路由器

在大多数情况下,某些答案是正确的,但对我来说,解决方案1和2没做运动(我不放弃至少第2点,因为我的设置可能有问题),第3点基本上是隔离问题而不是解决它(这也增加了路由表的复杂性),解决方案4和5超出了范围,因为我没有资源来购买专用硬件,也无法脱机服务器,因为它已经投入生产,因此可以总结一下更换CentOS服务器出现了“更好”的情况。

好了,现在回到问题所在,让我们先描述一下当前的设置...

接口

eth1: IP: 10.0.0.1, GW: 10.0.0.1, NM: 255.255.255.0 (LAN)
eth0: IP: 10.0.1.1, GW: 10.0.1.254, NM: 255.255.255.0 (ISP1 - ADSL Router)
eth2: IP: 10.0.2.1, GW: 10.0.2.254, NM: 255.255.255.0 (ISP2 - ADSL Router)

/etc/sysctl.conf

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

# Controls source route verification
net.ipv4.conf.default.rp_filter = 0

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_messages = 1

的/ etc / iproute2的/ rt_tables:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
2 ISP1
3 ISP2

/ etc / sysconfig / network-scripts / route-eth0

10.0.1.0/24 dev eth0 src 10.0.1.1 table ISP1
default via 10.0.1.254 dev eth0 table ISP1

/ etc / sysconfig / network-scripts / route-eth2

10.0.2.0/24 dev eth2 src 10.0.2.1 table ISP2
default via 10.0.2.254 dev eth2 table ISP2

/ etc / sysconfig / network-scripts / rule-eth0

fwmark 2 table ISP1
from 10.0.1.1 table ISP1

/ etc / sysconfig / network-scripts / rule-eth2

fwmark 3 table ISP2
from 10.0.2.1 table ISP2

/ etc / sysconfig / iptables

*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# Basic Rules
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT

# SSH
-A INPUT -i eth0 -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -i eth2 -m tcp -p tcp --dport 22 -j ACCEPT

# OpenVPN
-A INPUT -i eth0 -m udp -p udp --dport 1194 -j ACCEPT
-A INPUT -i eth2 -m udp -p udp --dport 1194 -j ACCEPT

# Allow everything from LAN
-A INPUT -i eth1 -j ACCEPT

# Allow everything from the VPN
-A INPUT -i tun0 -j ACCEPT

# Default Drop on everything else
-A INPUT -j DROP

# Allow forwarding from LAN and VPN
-A FORWARD -i eth1 -j ACCEPT
-A FORWARD -i tun0 -j ACCEPT

# Allow all outbound traffic
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -o eth1 -j ACCEPT
COMMIT

*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# DNAT to Developer Box (SSH Server)
-A PREROUTING -i eth0 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222
-A PREROUTING -i eth2 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222

# SNAT
-A POSTROUTING -o eth0 -j SNAT --to-source 10.0.1.1
-A POSTROUTING -o eth2 -j SNAT --to-source 10.0.2.1
COMMIT

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
# Mark Based Routing? (based on NerdBoys site)
-A PREROUTING -j CONNMARK --restore-mark
-A PREROUTING --match mark --mark 2 -j ACCEPT
-A PREROUTING -i eth0 -j MARK --set-mark 2
-A PREROUTING --match mark --mark 3 -j ACCEPT
-A PREROUTING -i eth2 -j MARK --set-mark 3
-A PREROUTING -j CONNMARK --save-mark
COMMIT

使用gwping bash脚本可以实现负载平衡,该脚本基本上监视2个WAN(eth0和eth2)并像这样在服务器中设置默认路由和权重(在负载平衡或2个wan正常运行时):

ip route replace default scope global nexthop via 10.0.1.1 dev eth0 weight 1 nexthop via 10.0.2.1 dev eth1 weight 1

我的问题是,即使使用这种设置,很多人似乎都同意这是正确的设置,即使从包访问数据包,仍然存在从外部访问网络内部服务的问题(特别是ssh开发人员盒和OpenVPN)被“标记”并据此进行路由的话,开发人员的回答总是指向错误的路径。我不知道是不是在漏斗区或nat区中遗漏了某些东西,还是根本误解了基于源的路由,无论如何,如果有人知道如何进行相应的工作,将不胜感激。

我的设置来源是:

lartc.org/lartc.html#LARTC.RPDB.MULTIPLE-LINKS
fatalsite.net/?p=90
nerdboys.com/2006/05/05/conning-the-mark-multiwan-connections-using-iptables-mark-connmark-and-iproute2/
policyrouting.org/PolicyRoutingBook/ONLINE/CH08.web.html
unix.stackexchange.com/questions/58635/iptables-set-mark-route-diferent-ports-through-different-interfaces
unix.stackexchange.com/questions/22770/two-interfaces-two-addresses-two-gateways
bulma.net/body.phtml?nIdNoticia=2145

最亲切的问候

PS1:我发现一个网站,其说法是路由表中的标记应与iptables标记相差+ 1(kim.attr.ee/2010/08/source-based-policy-routing-on-centos.html)这是真的?或这个网站是不正确的。


更新15/08/2013 22:15

经过更多的研究和调试后,我发现一个网站,上面说我忘记在路由后表中添加SNAT部分,因此我将此规则添加到iptables配置中:

-A POSTROUTING --match mark --mark 2 -j SNAT --to-source 10.0.1.1
-A POSTROUTING --match mark --mark 3 -j SNAT --to-source 10.0.2.1

但是我仍然无法从网络外部连接到devbox。从好的方面来说,iptables -t nat -nvL POSTROUTING会提示基于connmark的策略路由的工作方式,因此它可能与ISP1和ISP2路由器边缘有关:

Chain POSTROUTING (policy ACCEPT 520 packets, 56738 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK match 0x2 to:10.0.1.1
    6   312 SNAT       all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK match 0x3 to:10.0.2.1
  903 70490 SNAT       all  --  *      eth0    0.0.0.0/0            0.0.0.0/0           to:10.0.1.1
  931 78070 SNAT       all  --  *      eth2    0.0.0.0/0            0.0.0.0/0           to:10.0.2.1

另外,我还从设置中添加了更多信息,因为有人开始用尽想法,所以请有人丢下我的救命笔...>。<

ip路由显示

10.8.0.2 dev tun0  proto kernel  scope link  src 10.8.0.1
10.0.2.0/24 dev eth2  proto kernel  scope link  src 10.0.2.1
10.0.0.0/24 dev eth1  proto kernel  scope link  src 10.0.0.1
10.8.0.0/24 via 10.8.0.2 dev tun0
10.0.1.0/24 dev eth0  proto kernel  scope link  src 10.0.1.1
169.254.0.0/16 dev eth2  scope link
default
        nexthop via 10.0.1.254  dev eth0 weight 1
        nexthop via 10.0.2.254  dev eth2 weight 1

ip规则显示:

0:      from all lookup 255
1024:   from all fwmark 0x2 lookup ISP1
1025:   from all fwmark 0x3 lookup ISP2
2024:   from 10.0.1.1 lookup ISP1
2025:   from 10.0.2.1 lookup ISP2
32766:  from all lookup main
32767:  from all lookup default

新来源:

sarcasmasaservice.com/2013/04/linux-routing-capabilities-my-abuse-thereof/

最亲切的问候


您能否更改标题并删除centos,将其替换为Linux?我已经在Debian / Devuan上使用了确切的配置
Hvisage,

Answers:


5

好...

经过数千小时的调试,尝试了不同的设置以及72个小时的生产测试,我能够找到正确的解决方案/设置,问题出在iptables规则(缠结部分)中,数据包进来时显然被正确标记了但是,当它们出来时,没有任何处理过的数据包,无论如何,这是我解决问题的最终方法:

/ etc / sysconfig / iptables

*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# Basic Rules
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT

# SSH
-A INPUT -i eth0 -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -i eth2 -m tcp -p tcp --dport 22 -j ACCEPT

# OpenVPN
-A INPUT -i eth0 -m udp -p udp --dport 1194 -j ACCEPT
-A INPUT -i eth2 -m udp -p udp --dport 1194 -j ACCEPT

# Allow everything from LAN
-A INPUT -i eth1 -j ACCEPT

# Allow everything from the VPN
-A INPUT -i tun0 -j ACCEPT

# Default Drop on everything else
-A INPUT -j DROP

# Allow forwarding from LAN and VPN
-A FORWARD -i eth1 -j ACCEPT
-A FORWARD -i tun0 -j ACCEPT

# Allow all outbound traffic
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -o eth1 -j ACCEPT
COMMIT

*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# DNAT to Developer Box (SSH Server)
-A PREROUTING -i eth0 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222
-A PREROUTING -i eth2 -p tcp -m tcp --dport 2222 -j DNAT --to-destination 10.0.0.200:2222

# SNAT
-A POSTROUTING -o eth0 -j SNAT --to-source 10.0.1.1
-A POSTROUTING -o eth2 -j SNAT --to-source 10.0.2.1
COMMIT

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
# CONNMARK Source Based Routing
-A PREROUTING -i eth0 -m state --state NEW,RELATED,ESTABLISHED -d 10.0.1.1 -j CONNMARK --set-mark 0x2
-A PREROUTING -i eth2 -m state --state NEW,RELATED,ESTABLISHED -d 10.0.2.1 -j CONNMARK --set-mark 0x3
-A PREROUTING -i eth1 -m connmark --mark 0x2 -j CONNMARK --restore-mark
-A PREROUTING -i eth1 -m connmark --mark 0x3 -j CONNMARK --restore-mark
-A OUTPUT -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark
COMMIT

显然,加上以前与iproute和gwping有关的所有设置(用于链路负载平衡和故障转移),多亏了源[1]和[2],这些解决方案才得以实现,这两个方面都使我指向了不同的部分(Luca Gibelli for解决方案的PREROUTING部分和Karl Bowden的OUTPUT部分),这里还有其他网站的其他资源,这些资源为我指明了正确的浏览方向。希望这对以后的其他系统管理员有所帮助。

最亲切的问候

资料来源:

[1]www.nervous.it/2010/09/dnat-and-ip-source-routing-woes/
[2]blog.khax.net/2009/12/01/multi-gateway-balancing-with-iptables/
[3]home.regit.org/netfilter-en/links-load-balancing/
[4]mailman.ds9a.nl/pipermail/lartc/2006q2/018964.html
[5]web.archive.org/web/20120320115329/http://versa.net.au/index.php?option=com_content&task=view&id=21&Itemid=34

更新10/10/2013

OpenVPN需要一个额外的配置指令才能与多个WAN设置一起使用(如前一个),因此只需在server.conf中添加multihome选项(OpenVPN> = 2.1,对于较低版本,只需将本地指令更改为仅侦听特别是IP),您就可以使用了。


谢谢,-t mangle -A输出... -j CONNMARK --restore-mark是我在我的设置中缺少的内容:)
Hvisage

0

只是给您一个提示,我只是在没有负载平衡和3个wan链接的情况下执行相同类型的操作,而我什至不使用iptables来完成所有操作。我发现使用简单的基于策略的路由和socat更有效

我的rt_tables:

100     lan
102     wireless
103     wan1
104     wan2
105     wan3

在每个wan链接的接口中:

/bin/ip route add <wan network id> dev eth0 src <wan host ip> table wan1
/bin/ip route add default via <wan gateway address> table wan1
/bin/ip rule add from <wan host ip> table wan1

在每个wan链接向下的接口中:

/bin/ip route del default via <wan gateway address> table wan1
/bin/ip rule del from <wan host ip> table wan1

强制局域网客户端断开特定链接:将其放在您的界面脚本中:

/bin/ip rule add from <lan client ip> table <wan table number of link to force it through>

并在下脚本中:

/bin/ip rule del from <lan client ip> table <wan table number of link to force it through>

发送到特定机器(例如Web服务器)的传入连接,向rc.local添加类似的内容(无论链接是否建立,都可以运行)

exec socat -T15 tcp4-listen:80,reuseaddr,fork tcp:<lan host ip to send it to>:80 >> /var/log/socat-web.log 2>&1

然后打开IP转发和伪装以及您需要的其他防火墙规则

在ubuntu上,您甚至可以为socat创建一个新贵的作业,这是我的/etc/init/socat-web.conf:

description "socat web port tunnel"
author      "jacqueline"

start on started mountall
stop on shutdown
respawn
respawn limit 99 5

script
    export HOME="/root"
    exec socat -T15 tcp4-listen:80,reuseaddr,fork tcp:192.168.0.97:80 >> /var/log/socat-web.log 2>&1
end script

post-start script
   # Optionally put a script here that will notifiy you socat has (re)started
end script

0

@CentOS_noob

首先,非常棒,非常感谢您的分享,非常感谢!我努力奋斗了好多年。:)

如果您想通过选定的链接定向指定的服务,则可以在当前MARKING规则之前这样做。

iptables -t mangle -A PREROUTING -i br1 -s 10.1.1.2 -p tcp --dport 80 -m state --state NEW,RELATED,ESTABLISHED -j CONNMARK --set-mark 0x3
iptables -t mangle -A PREROUTING -i br1 -s 10.1.1.2 -p icmp  -j CONNMARK --set-mark 0x4

哪里:

br1 - is LAN interface
10.1.1.2 - is LAN's IP of some host.
By using our site, you acknowledge that you have read and understand our Cookie Policy and Privacy Policy.
Licensed under cc by-sa 3.0 with attribution required.