The problem seems to be NIS. When I take NIS out of /etc/nsswitch.conf (as suggested Ken Teh earlier in the thread - thanks!), that seems to fix it: --- Begin Diff --- 38c38 < hosts: files nis dns --- > hosts: files dns --- End Diff --- I'm running the Microsoft SFU35 NIS server, so who knows what the hell they're trying to do in there.... The problem is solved - but here's a more precise description of it, just in case anyone wants a brain-teaser and/or runs into something similar. The exercise was to change alexandria-nfs (128.173.191.2) to have the IP of alexandria (128.173.188.43). The machine can only have 2 NICs (a 1U racmount machine with some other hardware), and I wanted to use the second NIC for another purpose. The bizarre output from one of the workstations is as follows: --- Begin Transcript --- [lscharf@hephaistos ~]$ nslookup alexandria-nfs Server: 128.173.188.25 Address: 128.173.188.25#53 alexandria-nfs.aoe.vt.edu canonical name = alexandria.aoe.vt.edu. Name: alexandria.aoe.vt.edu Address: 128.173.188.43 [lscharf@hephaistos ~]$ nslookup 128.173.188.43 Server: 128.173.188.25 Address: 128.173.188.25#53 43.188.173.128.in-addr.arpa name = alexandria.aoe.vt.edu. [lscharf@hephaistos ~]$ nslookup 128.173.191.2 Server: 128.173.188.25 Address: 128.173.188.25#53 ** server can't find 2.191.173.128.in-addr.arpa: NXDOMAIN [lscharf@hephaistos ~]$ dig -x 128.173.188.43 ; <<>> DiG 9.2.4 <<>> -x 128.173.188.43 ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49690 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;43.188.173.128.in-addr.arpa. IN PTR ;; ANSWER SECTION: 43.188.173.128.in-addr.arpa. 3600 IN PTR alexandria.aoe.vt.edu. ;; Query time: 3 msec ;; SERVER: 128.173.188.25#53(128.173.188.25) ;; WHEN: Fri Dec 30 13:51:44 2005 ;; MSG SIZE rcvd: 80 [lscharf@hephaistos ~]$ dig -x 128.173.191.2 ; <<>> DiG 9.2.4 <<>> -x 128.173.191.2 ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4576 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;2.191.173.128.in-addr.arpa. IN PTR ;; AUTHORITY SECTION: 191.173.128.in-addr.arpa. 3600 IN SOA pluto.aoe.vt.edu. hostmaster.aoe.vt.edu. 423 900 600 86400 3600 ;; Query time: 4 msec ;; SERVER: 128.173.188.25#53(128.173.188.25) ;; WHEN: Fri Dec 30 13:51:58 2005 ;; MSG SIZE rcvd: 131 [lscharf@hephaistos ~]$ dig alexandria-nfs ; <<>> DiG 9.2.4 <<>> alexandria-nfs ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 45818 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;alexandria-nfs. IN A ;; Query time: 3 msec ;; SERVER: 128.173.188.25#53(128.173.188.25) ;; WHEN: Fri Dec 30 13:52:02 2005 ;; MSG SIZE rcvd: 32 [lscharf@hephaistos ~]$ ping alexandria-nfs PING alexandria-nfs.aoe.vt.edu (128.173.191.2) 56(84) bytes of data. 64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=0 ttl=64 time=0.742 ms 64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=1 ttl=64 time=0.238 ms --- alexandria-nfs.aoe.vt.edu ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.238/0.490/0.742/0.252 ms, pipe 2 [lscharf@hephaistos ~]$ ping alexandria-nfs.aoe.vt.edu PING alexandria-nfs.aoe.vt.edu (128.173.191.2) 56(84) bytes of data. 64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=0 ttl=64 time=0.329 ms 64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=1 ttl=64 time=0.329 ms 64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=2 ttl=64 time=0.297 ms --- alexandria-nfs.aoe.vt.edu ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.297/0.318/0.329/0.020 ms, pipe 2 [lscharf@hephaistos ~]$ --- End Transcript --- But, alas, I still have to ssh around to every Linux machine and fix this. The Mac OS X and Solaris machines in the NIS domain don't seem to be affected. (The only reason I run something as evil as NIS is because it's supported by all of the OSs that I have mixed together...) Thanks! -Luke Ken Teh wrote: > I second Jon. Try 'dig <your-server-name>' > > At the bottom of the output will be the DNS server that responded. I also > don't know of any other caches that might cache this information other > than the ones already mentioned. > > Another thing you can try to "strace nslookup <your-server-name>". This > should also indicated how the resolver was called. It should open > /etc/resolv.conf, then open a socket to the nameserver to get the > information. If it does anything else, well, that's your answer. > > Ken > > > > On Fri, 30 Dec 2005, Jon Peatfield wrote: > > >>On Thu, 29 Dec 2005, Luke Scharf wrote: >> >> >>>I've run into this problem before: how do I empty the DNS cache in Linux? >>> >>>I've changed a record in my DNS server, and ping still insists that the >>>server is at the old address. However, if I point nslookup directly at >>>any/all of the DNS servers, I get the correct (new) address. >>> >>>I've tried touch'ing /etc/resolv.conf and /etc/nsswitch.conf. I've >>>poked around in /var looking for the file that the DNS client uses to >>>store the database. It has to be a file, because the last time I ran >>>into this problem, a reboot wouldn't encourage ping or any of the other >>>tools I tried to actually query the server and get the new record... >>> >>>Any suggestions? >> >>I know of only 2 caches which might be relevant, nscd (as mentioned >>before), and a caching named. You can flush the nscd hosts table by >>running (obviously as root): >> >> nscd --invalidate=hosts >> >>you can see if there are things cached in there by running: >> >> nscd --statistic >> >>The other is if you are running a local (cache-only) named, but you would >>know that from the resolv.conf entries (e.g. a 127.0.0.1 or similar >>nameserver entry). >> >>If there is one you can cause it to reload/restart to flush the cache. >> >>I'm assuming that you don't have any other sources of hostnames defined in >>nsswitch.conf ... >> >>On this host what does host/dig say for the DNS name you changed? >> >> -- Luke Scharf, Systems Administrator Virginia Tech Aerospace and Ocean Engineering