wviewweather.com Forum Index wviewweather.com
wview and Weather Topics
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Alternative to FTP in wview

 
Post new topic   Reply to topic    wviewweather.com Forum Index -> FTP
View previous topic :: View next topic  
Author Message
mdg



Joined: 17 Jan 2006
Posts: 20

PostPosted: Tue Jan 24, 2006 7:38 pm    Post subject: Alternative to FTP in wview Reply with quote

I've been experimenting with an alternative to the FTP system in wview. I went down this path for a couple reasons.

First, most FTP clients send the password in clear text. I'm a little paranoid about such things, so I wanted something that supports SSL because tnftp does not. My FTP server supports it, so why not use it!

I also noticed that the wviewftp.conf uses wildcards. This gives the potential of uploading files that haven't changed and don't need to be sent.

To use FTP over SSL, I found lftp 3.4 (http://lftp.yar.ru/) is excellent for this. It has many of the same functions as tnftp, especially the ability to use a URL from the command line. It also takes a -f option where you can specify a script to transfer files. (The commands being FTP commands.)

First, to set up SSL, I put this in my ~/.lftp/rc file:
Code:

set ftp:ssl-allow on
set ftp:ssl-force on

Then I created a commands file like this:
Code:

open -u USERNAME,PASSWORD ftp://ftpserver.com/weather
mput /var/wview/img/*.png
mput /var/wview/img/*.htm
mput /var/wview/img/*.html
mput /var/wview/img/*.xml
mput /var/wview/img/*.txt
exit


Then from cron, I set up the command:
Code:

/usr/bin/lftp -f ~/ftpcommands.txt

You'll see the mput commands are the same as what is in the wviewftp.conf file. Now to change that.

Using the find command, you can find the most recently changed files like this:
Code:

find /var/wview/img -mmin -10 -print

The -mmin -10 specifies to look for things that are newer than 10 minutes old. (I'm transferring every 10 minutes right now.)

So now I wanted to take that list and make an lftp command file out of it. So I used perl to do this. While I was in there, I might as well FTP the files. So here is how I do it all in one step:

Code:

#!/usr/bin/perl

open(FTPLIST,">/tmp/ftp.list")  || die "can't open file for reading: $!";

print FTPLIST "open -u USERNAME,PASSWORD ftp://ftpserver.com/weather\n";

open(FILELIST,"/usr/bin/find /var/wview/img -mmin -10 -print |");

while (<FILELIST>) { print FTPLIST "mput $_"; }

print FTPLIST "exit\n";

system("/usr/bin/lftp -f /tmp/ftp.list");

unlink </tmp/ftp.list>;


Normally, 66 files would be transferred with the old list. With this script, I only transfer as little as 36.

I hope this is useful. Maybe Mark would like to take some of these ideas and implement them into the program.
Back to top
View user's profile Send private message
mteel



Joined: 30 Jun 2005
Posts: 435
Location: Collinsville, TX

PostPosted: Wed Jan 25, 2006 1:39 pm    Post subject: Reply with quote

Why wouldn't you just use scp via wviewsshd if you are worried about security? FTP over SSL? scp is very fast and secure. That's why wviewsshd exists - as an alternative for security minded users.

Mark
Back to top
View user's profile Send private message Send e-mail Visit poster's website
mteel



Joined: 30 Jun 2005
Posts: 435
Location: Collinsville, TX

PostPosted: Wed Jan 25, 2006 3:59 pm    Post subject: Reply with quote

By the way, scp has all of the "don't copy unchanged files" logic built-in - it is actually a synchronization operation.

Mark
Back to top
View user's profile Send private message Send e-mail Visit poster's website
NorwegianWould



Joined: 07 Jan 2006
Posts: 23

PostPosted: Thu Jan 26, 2006 8:23 am    Post subject: Reply with quote

You could use rsync too. This copies the minumum of files, and for large files will only copy the parts that have changed.

Jeremy
Back to top
View user's profile Send private message
mteel



Joined: 30 Jun 2005
Posts: 435
Location: Collinsville, TX

PostPosted: Thu Jan 26, 2006 8:27 am    Post subject: Reply with quote

In fact that is what wviewsshd uses: rsync. scp and rsync are both utilities built on top of ssh, thus my error in quoting scp earlier...

My point is this: Why roll your own when wview already has a secure solution using ssh/rsync?

Mark
Back to top
View user's profile Send private message Send e-mail Visit poster's website
mdg



Joined: 17 Jan 2006
Posts: 20

PostPosted: Thu Jan 26, 2006 5:46 pm    Post subject: Reply with quote

Because my ISP doesn't give me shell access, so I can't use anything else. I'm familiar with scp, I use it all the time at work and I'd use it here if I could. (I do applaud its inclusion in the software!)
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    wviewweather.com Forum Index -> FTP All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group