is Debian retarded, or is it me?

or possibly both? let me tell you my story...

I wanted to give Debian a try on a server, because I'm tired of the version churn with Fedora. so I put Sarge on a machine whose only duty is to serve as a web proxy server. I get Sarge installed and updated, get squid and winbind installed, all that. now, the thing is, this machine is going to have 2 ethernet interfaces. 1 with a public IP address, through which all communication with the outside world will take place, and one with a private IP address, on which the web proxy and the ssh daemon will listen. everything was working fine with one interface. but then when it came time to put the other ethernet card in, well, that's when the fun started!

first thing I realize is that I don't know squat about how Debian sets up networking stuff. a little basic poking about leads me to /etc/network/interfaces, which at first glance seems so simple that even I couldn't screw it up. so I put in another stanza there for the new card; I just cut and pasted the one for eth0, and then changed the name to eth1 and all the IP info to what it should be for the new card. first mistake! on reboot, for some reason, it hangs at "starting MTA" for a few seconds, but then goes on and continues to boot. everything seems cool, until I try to ssh to the private address from my workstation with a public address. doesn't work, just times out. however, I can ssh in there just fine from a machine on the same private subnet.

okay, it's a routing issue. so, I do a `route`, and again there's an odd few second long pause, and when it finally comes back, I see that there are 2 default gateways. yeah, that's probably not right. so I go back and look in /etc/network/interfaces, and take out the "gateway" line on the second interface. after a reboot, that annoying pause at "starting MTA" is gone (looks promising), and a `route` shows the gateway I specified. unfortunately, I still can't ssh to the private address on the box from my public addressed box.

it is at this point that I have the disheartening realization that maybe I don't know as much about networking as I like to believe. my theory here is that while the ssh daemon is listening on the private addressed interface, since the default gateway I've set is the one for the publicly addressed interface, it's trying to reply back to my ssh initiation packets on the *wrong* interface (well, I mean that it's probably replying on the correct interface, but it's trying to send to the wrong gateway from that interface, which will go precisely nowhere). I have yet to verify this with tcpdump but I will do so soon. however, the fact that adding in a static route for my workstation (which has a public address) through the correct gateway for the private network makes ssh magically work supports this theory. anyway as I said I realize that I've never had to really futz with this stuff before, so I actually had to... read the manual! which it turns out is absolutely no help. the online docs, btw, are positively ancient.

at any rate, it seems to me what I need here is just a way to tell the kernel that the gateway for eth0 is X, AND the gateway for eth1 is Y. but I have no idea how to do this. well, that's not entirely true; I could "fake it" by adding routes in for all of my internal subnets, but that's incredibly kludgy and ugly. surely there's a more elegant way of fixing this? but if so, I haven't found it, and none of the people I've asked know off the tops of their heads, either.

the odd thing is, I've done this same setup (proxy servers with 1 private and 1 public interface) many times before using RedHat and Fedora, and in all of those cases, it "just works", without any hoop-jumping at all. and in fact, on those RH and Fedora based proxies, the routing table looks exactly like it does on this Debian box. it just doesn't work on Debian, for whatever reason. can one of you Debian gurus tell me what it is that I am missing?

*edit*
I guess I should mention that there's no NATing involved. sorry I didn't make that clear.

first pass (ie there may be other passes at this from me and/or others)

I do exactly the same thing -- well mostly. My router is running Debian woody (stable). Sarge is about to become stable but is currently the testing branch.

I have two network interfaces, one talks on the public interface and the other talks to the internal private network. I have ssh running so I can ssh into the box from anywhere (inside, outside, whatever). In my case though the only difference I can see is that I have a firewall on the machine for nat and forwarding (among other things).

On a separate note, you don't need to reboot after changing your /etc/network/interfaces, you only need to ifdown/ifup the interface you changed.

I haven't run route on that machine in quite a while, but all my internal machines are dhcp served from the internal interface on the router. The routers' external interface is dhcp from cox.

Maybe some information in there helped you. I have my network running well so I don't get on the router very often (almost never except for apt-get'ing security updates when they come out). I'll post more later when I can get on the box and run a few commands and cat a few config files to show how I have mine setup.

Evilpig might chime in here since she is a Debian wonk also and has better networking skills than mine.

ahhh, but I bet all your internal machines are on the same subnet, right? and any talking they do to outside machines is NATed, so it's hard to make a comparison.

also, I know I shouldn't have to reboot, but I'm going to continue doing that anyway, for a few reasons. first, there's a chance that I'll forget something, like fixing the routing table or something. second, I want to make sure that it's all set up correctly such that it'll come up right on a reboot anyway, so I may as well test that. third, I don't trust Debian's ifdown command. why? because at one point during this testing, I did do just that. a few minutes later, I tried to ssh to that interface... but I had forgotten to ifup the thing. nevertheless, I was able to connect just fine! ergo, I don't trust ifdown to do what I think it should.


*edit*
I guess I should mention that there is no NATing involved in my setup. sorry I didn't make that clear.

ahhh, but I bet all your internal machines are on the same subnet, right? and any talking they do to outside machines is NATed, so it's hard to make a comparison.

True on all accounts.

also, I know I shouldn't have to reboot, but I'm going to continue doing that anyway, for a few reasons. first, there's a chance that I'll forget something, like fixing the routing table or something. second, I want to make sure that it's all set up correctly such that it'll come up right on a reboot anyway, so I may as well test that. third, I don't trust Debian's ifdown command. why? because at one point during this testing, I did do just that. a few minutes later, I tried to ssh to that interface... but I had forgotten to ifup the thing. nevertheless, I was able to connect just fine! ergo, I don't trust ifdown to do what I think it should.

That is odd. I have seen that type of behavior on my laptop from time to time. It's like it gets cached or something. ifdown a "down" interface gives an error that says it's not configured but ifconfig plainly says it is. So ifconfig <interface> down should help, but it still shows it in the list of "up" interfaces. However, I haven't seen this type of behavior recently. I wonder if that is because for all but my router machine, I am on the unstable branch. hmm....



*edit*
I guess I should mention that there is no NATing involved in my setup. sorry I didn't make that clear.

I'm not sure I'm much help then. I'm nat'ed (as indicated) so all my problems are solved that way.

First off: more info would definitely help. You don't give much info about how your network is laid out - what routers/switches are connected to what gear, how your public and private subnets connect, what sort of connection to the outside world you have, etc. Contents of /etc/network/interfaces, /etc/network/options on the debian machine, and the output of /sbin/route -n and /usr/sbin/arp on both machines would be helpful as well.

Without that info, my wild guess is that it's a problem with rp_filter being enabled via "spoofprotect=yes" in /etc/network/options. To verify you can look at that file, or "cat /proc/sys/net/ipv4/conf/*/rp_filter". Having it enabled will cause the stack to not accept traffic on one of your interfaces if the return packets would be routed out one of your other interfaces (yeah, that's a pretty horrible explanation, sorry). This is where more info on your switch/hub/router layout would really come in handy.

The Linux network stack does some things that lead to weird problems like this that may not be intuitively obvious, and this is where specialized tools like "arping" are very nice to use.

Good luck!

- Colleen

PS: IMO the best way to deal with the bringing the network up and down under debian is to use "/etc/init.d/networking [start|stop|restart]". It makes the calls to ifup/ifdown, but also takes into account the /etc/network/options files and such. If you don't make crazy changes the "restart" arg will restart things without you losing your connections.

Colleen, may God bless you and your House. you were right; it was rp_filter that was causing me grief. I thank you!

and yeah, I know I could've posted more info and been more explicit, but this is for my work network. Believe me when I say that our network here does NOT follow best security practices (doesn't seem to matter how many times I complain about it, as long as stuff looks like it's "working fine" to the higher-ups, nothing will change). therefore I was loathe to divulge anything that might help compromise our security.

hehe, too late for that :)

Colleen, may God bless you and your House. you were right; it was rp_filter that was causing me grief. I thank you!

Hot! Maybe now that my house has been blessed, it will clean itself... Glad that worked.

and yeah, I know I could've posted more info and been more explicit, but this is for my work network. Believe me when I say that our network here does NOT follow best security practices (doesn't seem to matter how many times I complain about it, as long as stuff looks like it's "working fine" to the higher-ups, nothing will change). therefore I was loathe to divulge anything that might help compromise our security.

Hahaha, I feel your pain. Sorry, I assumed it was your home network or I wouldn't have asked. :D