get-url with redirection?

Started by zxcel, December 11, 2009, 04:23:13 PM

Previous topic - Next topic
Print
Go Down Pages1
User actions

zxcel

December 11, 2009, 04:23:13 PM Last Edit: December 31, 1969, 04:00:00 PM by Guest
I want to download the file, but get-url can't do it. That's because the url that i have, redirects me to another url. All browsers, Opera,  wget - all of them understand that redirection. How can I do it?

hilti

#1
December 12, 2009, 01:10:35 AM Last Edit: December 31, 1969, 04:00:00 PM by Guest
Try to use CURL instead


Code Select
(exec (string "curl -u "http://www.google.com" --progress-bar"))

CURL handles HTTPS, too.



Cheers

Hilti
--()o Dragonfly web framework for newLISP

http://dragonfly.apptruck.de\">http://dragonfly.apptruck.de

hilti

#2
December 12, 2009, 01:12:04 AM Last Edit: December 31, 1969, 04:00:00 PM by Guest
Sorry.



That's the correct snippet:


Code Select
(exec (string "curl 'http://www.google.com' --progress-bar"))
--()o Dragonfly web framework for newLISP

http://dragonfly.apptruck.de\">http://dragonfly.apptruck.de

Fritz

#3
March 07, 2010, 11:00:58 AM Last Edit: December 31, 1969, 04:00:00 PM by Guest
You can also look into the text "get-url" returns to you. I believe, inside you will find a correct address of the file.



First get-url:
Code Select
(get-url "http://lenta.ru/info")

Result:
Code Select
"<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">n<html><head>n<title>302 Found</title>n</head><body>n<h1>Found</h1>n<p>The document has moved <a href="http://lenta.ru/info/">here</a>.</p>n<hr>n<address>Apache Server at lenta.ru Port 80</address>n</body></html>n"

Parsing correct address:
Code Select
(first (regex {(?<=href=")[^"]+} (get-url "http://lenta.ru/info")))

Result:
Code Select
"http://lenta.ru/info/"

Final get-url:
Code Select
(get-url (first (regex {(?<=href=")[^"]+} (get-url "http://lenta.ru/info"))))

Lutz

#4
March 08, 2010, 07:08:30 AM Last Edit: December 31, 1969, 04:00:00 PM by Guest
In the past 'get-url' only handled re-  "Location:" headers together with error codes 301 and 303. I have now added error code 302 for v.10.2.0 to force a second request also on this error code, and it handles the http://lenta.ru/info">http://lenta.ru/info case correctly.
Print
Go Up Pages1
User actions