The problem seems to be NIS. When I take NIS out of /etc/nsswitch.conf
(as suggested Ken Teh earlier in the thread - thanks!), that seems to
fix it:
--- Begin Diff ---
38c38
< hosts: files nis dns
---
> hosts: files dns
--- End Diff ---
I'm running the Microsoft SFU35 NIS server, so who knows what the hell
they're trying to do in there....
The problem is solved - but here's a more precise description of it,
just in case anyone wants a brain-teaser and/or runs into something
similar. The exercise was to change alexandria-nfs (128.173.191.2) to
have the IP of alexandria (128.173.188.43). The machine can only have 2
NICs (a 1U racmount machine with some other hardware), and I wanted to
use the second NIC for another purpose. The bizarre output from one of
the workstations is as follows:
--- Begin Transcript ---
[lscharf@hephaistos ~]$ nslookup alexandria-nfs
Server: 128.173.188.25
Address: 128.173.188.25#53
alexandria-nfs.aoe.vt.edu canonical name = alexandria.aoe.vt.edu.
Name: alexandria.aoe.vt.edu
Address: 128.173.188.43
[lscharf@hephaistos ~]$ nslookup 128.173.188.43
Server: 128.173.188.25
Address: 128.173.188.25#53
43.188.173.128.in-addr.arpa name = alexandria.aoe.vt.edu.
[lscharf@hephaistos ~]$ nslookup 128.173.191.2
Server: 128.173.188.25
Address: 128.173.188.25#53
** server can't find 2.191.173.128.in-addr.arpa: NXDOMAIN
[lscharf@hephaistos ~]$ dig -x 128.173.188.43
; <<>> DiG 9.2.4 <<>> -x 128.173.188.43
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49690
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;43.188.173.128.in-addr.arpa. IN PTR
;; ANSWER SECTION:
43.188.173.128.in-addr.arpa. 3600 IN PTR alexandria.aoe.vt.edu.
;; Query time: 3 msec
;; SERVER: 128.173.188.25#53(128.173.188.25)
;; WHEN: Fri Dec 30 13:51:44 2005
;; MSG SIZE rcvd: 80
[lscharf@hephaistos ~]$ dig -x 128.173.191.2
; <<>> DiG 9.2.4 <<>> -x 128.173.191.2
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4576
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;2.191.173.128.in-addr.arpa. IN PTR
;; AUTHORITY SECTION:
191.173.128.in-addr.arpa. 3600 IN SOA pluto.aoe.vt.edu.
hostmaster.aoe.vt.edu. 423 900 600 86400 3600
;; Query time: 4 msec
;; SERVER: 128.173.188.25#53(128.173.188.25)
;; WHEN: Fri Dec 30 13:51:58 2005
;; MSG SIZE rcvd: 131
[lscharf@hephaistos ~]$ dig alexandria-nfs
; <<>> DiG 9.2.4 <<>> alexandria-nfs
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 45818
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;alexandria-nfs. IN A
;; Query time: 3 msec
;; SERVER: 128.173.188.25#53(128.173.188.25)
;; WHEN: Fri Dec 30 13:52:02 2005
;; MSG SIZE rcvd: 32
[lscharf@hephaistos ~]$ ping alexandria-nfs
PING alexandria-nfs.aoe.vt.edu (128.173.191.2) 56(84) bytes of data.
64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=0
ttl=64 time=0.742 ms
64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=1
ttl=64 time=0.238 ms
--- alexandria-nfs.aoe.vt.edu ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.238/0.490/0.742/0.252 ms, pipe 2
[lscharf@hephaistos ~]$ ping alexandria-nfs.aoe.vt.edu
PING alexandria-nfs.aoe.vt.edu (128.173.191.2) 56(84) bytes of data.
64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=0
ttl=64 time=0.329 ms
64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=1
ttl=64 time=0.329 ms
64 bytes from alexandria-nfs.aoe.vt.edu (128.173.191.2): icmp_seq=2
ttl=64 time=0.297 ms
--- alexandria-nfs.aoe.vt.edu ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.297/0.318/0.329/0.020 ms, pipe 2
[lscharf@hephaistos ~]$
--- End Transcript ---
But, alas, I still have to ssh around to every Linux machine and fix
this. The Mac OS X and Solaris machines in the NIS domain don't seem to
be affected. (The only reason I run something as evil as NIS is because
it's supported by all of the OSs that I have mixed together...)
Thanks!
-Luke
Ken Teh wrote:
> I second Jon. Try 'dig <your-server-name>'
>
> At the bottom of the output will be the DNS server that responded. I also
> don't know of any other caches that might cache this information other
> than the ones already mentioned.
>
> Another thing you can try to "strace nslookup <your-server-name>". This
> should also indicated how the resolver was called. It should open
> /etc/resolv.conf, then open a socket to the nameserver to get the
> information. If it does anything else, well, that's your answer.
>
> Ken
>
>
>
> On Fri, 30 Dec 2005, Jon Peatfield wrote:
>
>
>>On Thu, 29 Dec 2005, Luke Scharf wrote:
>>
>>
>>>I've run into this problem before: how do I empty the DNS cache in Linux?
>>>
>>>I've changed a record in my DNS server, and ping still insists that the
>>>server is at the old address. However, if I point nslookup directly at
>>>any/all of the DNS servers, I get the correct (new) address.
>>>
>>>I've tried touch'ing /etc/resolv.conf and /etc/nsswitch.conf. I've
>>>poked around in /var looking for the file that the DNS client uses to
>>>store the database. It has to be a file, because the last time I ran
>>>into this problem, a reboot wouldn't encourage ping or any of the other
>>>tools I tried to actually query the server and get the new record...
>>>
>>>Any suggestions?
>>
>>I know of only 2 caches which might be relevant, nscd (as mentioned
>>before), and a caching named. You can flush the nscd hosts table by
>>running (obviously as root):
>>
>> nscd --invalidate=hosts
>>
>>you can see if there are things cached in there by running:
>>
>> nscd --statistic
>>
>>The other is if you are running a local (cache-only) named, but you would
>>know that from the resolv.conf entries (e.g. a 127.0.0.1 or similar
>>nameserver entry).
>>
>>If there is one you can cause it to reload/restart to flush the cache.
>>
>>I'm assuming that you don't have any other sources of hostnames defined in
>>nsswitch.conf ...
>>
>>On this host what does host/dig say for the DNS name you changed?
>>
>>
--
Luke Scharf, Systems Administrator
Virginia Tech Aerospace and Ocean Engineering
|
