wviewweather.com

THogland · Joined: 26 Sep 2005 Posts: 33

No idea why this is happening, but after a day or so wvwunderd stops sending data. There's no errors in the system logs (even on the verbose settings), and using the init'd script (or manually using kill-15) won't kill it - just kill -9... Killing it and restarting it doesn't upload any missing data, so the system thinks it's been working.

This is 1.81, fresh install. I've tried using the init.d/wview stop/start, and also manually stopping and starting the wvwunderd manually, but neither makes a difference.

Is there a good way to debug this, or is this a known issue?

THogland · Joined: 26 Sep 2005 Posts: 33

mteel · Joined: 30 Jun 2005 Posts: 435 Location: Collinsville, TX

It is not a known issue, there are only a couple of vaguely described issues that I cannot reproduce. I don't let known issues that I can reproduce "remain" in the system - I fix them. One other guy reported that Wunderground submissions would just "stop" but gave no other details of value (you have).

You apparently have your system log messages of type "critical" being routed somewhere other than the normal log file, that is why they are not showing up in the /var/log/messages file. Check your /etc/syslog.conf file and look for where "*.crit" messages are being routed - add that to your "/var/log/messages" line so they are logged to that file as well.

That log message indicates that there are no system buffers available of the size required for the archive notify message, which should be the smallest or certainly the next smallest size. Very odd considering your CWOP submissions seem to be working and that process receives the same archive notify message just before the wunderground process should receive it...

Did you change the "sysBufferCounts" values in radsysdefs.c prior to building radlib? This is where the numbers of system buffers are defined.

If not, then there would appear to be a buffer leak although my wview has been running for well over a week with no CWOP or Wunderground problems. Other folks' have been running even longer.

I will try to whip up a simple debug utility to attach to the wview system and dump out system buffer counts tonight and send it to you so we can get a better idea what is going on...

THogland · Joined: 26 Sep 2005 Posts: 33

THogland · Joined: 26 Sep 2005 Posts: 33

Rebooted and restarted, looked good again. (That was ~4 hours ago, just before leaving for work.) Just checked, no updates going anyplace - local folder for website, Weather Underground, etc. - nothing. So, I deleted the old source, downloaded a fresh copy, recompiled and reinstalled. Checked the .conf files. verified they're correct. Restarted, all looks good again.

Only issue is, I get the following when FTP'ing (which is the mesonet.txt file):

mteel · Joined: 30 Jun 2005 Posts: 435 Location: Collinsville, TX

OK, if it happens again, before you restart wview (leave it running), download this version of radlib: http://prdownloads.sourceforge.net/radlib/radlib-2.3.0.tar.gz?download

Install it as normal (wview still running):

tar zxvf radlib-2.3.0.tar.gz
cd radlib-2.3.0
./configure
make install
cd debug
make
./raddebug 1

Post the output of that utility...

THogland · Joined: 26 Sep 2005 Posts: 33

Froze up again - 12:05 to 13:10 Sad

mteel · Joined: 30 Jun 2005 Posts: 435 Location: Collinsville, TX

OK, you are not running out of buffers, although there are more allocated than there should be (should be more like 15 or so nominally)...

Makes me think that libcurl is having a problem...

Post results of:
ps aux | grep wv

I bet wvwunderd is hung up waiting on libcurl to finish a transaction - you could do a little research on libcurl - see how to debug it or look for error output (I think it has its own log file), etc. I would, but I don't have much time during the week.

Your log file, anything unusual in it? Are there bufferGet failure messages or queueSend failure messages? If anything in it doesn't look like the example normal output in the User Manual, post it with some context around it.

THogland · Joined: 26 Sep 2005 Posts: 33

I'm guessing you want this when it's hung... I've already restarted it, so it looks fine now. I ran it and left it sitting in a window; if/when it hangs I'll run it again and post.

Any particular version of libcurl you're using? Mine shows libcurl3 7.13.2-2... Also, any sense in upping the 64 buffer to 128? Or is one free normal?

mteel · Joined: 30 Jun 2005 Posts: 435 Location: Collinsville, TX

No 1 free does not appear to be normal (as I said earlier), but it would only mean that radlib would go up to the next bigger size when allocating. My guess is the ones which are allocated (other than the 15 or so that stay allocated) were for the Archive Notify message to wvwunderd, and they are sitting in his FIFO waiting to be read while he waits on a libcurl transaction. Just a guess at this point...

I am using 7.13.1 but I doubt the newer version is the problem.

Also, you could look in your log file for the unusual messages even after restarting, and do the libcurl error log investigation as well...

mhweather · Joined: 07 Aug 2005 Posts: 54

I saw this in Gary's Weathernet Forum (http://www.weatherforum.net), some people are complaining about WU not updating:

THogland · Joined: 26 Sep 2005 Posts: 33

So, I checked all the configs (again), went through dselect and loaded all the libcurl, libgd, etc. libraries (the verious debug, dev, etc. versions, including the -gssapi and -ocaml ones), did a few other system updates, shutdown (to swap UPS connections), and since then there's been *NO* problems. No problems for over 4 days. (I thought that it might be "working until you shut it down and start it manually, when it starts having issues" but I 'wview stop', changed the .conf to turn off verbose logging, then started it again and so far it's looking fine.)

I get an occasional wakeup error, or invalid data packet, this kinda stuff:

Oct 18 00:05:07 fred wviewd[2648]: <46308281> : wakeupConsole: Read ERROR!
Oct 18 00:05:07 fred wviewd[2648]: <46308281> : daemonRunState: WAKEUP failed
Oct 18 00:20:05 fred wviewd[2648]: <47206702> : daemonReceiveArchiveState: WAKEUP2 failed
Oct 18 00:20:05 fred wviewd[2648]: <47206702> : wakeupConsole: Invalid data: 39 20
Oct 18 00:23:07 fred wviewd[2648]: <47388270> : wakeupConsole: Read ERROR!
Oct 18 00:23:07 fred wviewd[2648]: <47388270> : daemonRunState: WAKEUP failed

... but nothing that looks like real errors, and nothing that repeats regularly.

I even got a note from mesonet at NOAA that they love the regularly-updating data feed I'm providing them! I have *no* idea what happened, or didn't happen, or broke-which-what-way-when-where, but apparently it's happily working again, at least for the past hour or two. I'll call it fixed when it's up for another couple days (after the manual stop/start, which was the referenced hour or two ago), but I'm thinking I've got a case of the weird gremlins, not an actual repeatable software bug...

Thanks for the time on this, even though I'm not sure how productive it was Sad

mteel · Joined: 30 Jun 2005 Posts: 435 Location: Collinsville, TX

I suspect the network problems that Wunderground (as reported by MullicaHill above) was having was your problem...

BTW, there is no difference in a system boot wview start and a manual start...

Keep an eye on those console wakeup errors. I NEVER get them. I think it is an indication of a serial port/cable that is on the hairy edge. If you keep seeing them, you may want to try another serial cable. If that doesn't fix it, you may have a shaky serial port... Those are real errors, just not catastrophic errors.

I've added the raddebug utility to the radlib distro, so that was productive anyway Smile

Mark

THogland · Joined: 26 Sep 2005 Posts: 33

Don't know how that last post showed up - I didn't send it...

Since I re-spliced the cable ends, I get those errors fairly regularly, but it's still reading and working properly. They're coming from a pretty new Dell server (less than a year old), so I don't think it's a bad port. I'm thinking I'll try setting the serial port at the correct speed manually and see if that's part of the problem. Otherwise I'm guessing the cable, so I'll have to dig up about 40' of new wire and re-string it...

mteel · Joined: 30 Jun 2005 Posts: 435 Location: Collinsville, TX

It won't be the serial port setup done by wview, that has kinda been tested by a whole bunch of folks (not to mention my initial testing) and is pretty solid Wink

40 foot of serial cable sounds pretty likely, especially if you are wiring it up yourself.