PS C:binnewlisp> newlisp -v
newLISP v.10.7.1 64-bit on Windows IPv4/6 UTF-8 libffi.
Running Apache2.4 on a Win10 box.
Apache is choking - once again - on the header sent by newlisp:
Quote
Lutz fixed a similar issue back in 2015, but it seems to have re-surfaced!
This is the code used then and now:
(print "Content-type: text/htmlnn")
(print "<html>")
(print "<h4>CGI by newLISP v" (sys-info -2)" on " ostype "</h4>")
(dolist (e (env)) (println (e 0) " : " (e 1) "</br>"))
(println "</html>")
(exit)
Anybody know how to fix this issue?
Looks suspiciously like the ole windoze versus unix line ending issue/difference. I'd check that before moving on to anything else.
For instance, what about trying this instead? (Sorry, "I don't do windows." :)
(println "")
Or something like that? Good luck!
Quote from: "rickyboy"
Nope! It's a newlisp issue - just like back in 2015.
I tried your suggestion, even though I knew it wouldn't work. And it didn't. Lutz will have to get involved I'm sure. Thanks anyway!
You might want to try the initial line
Quote from: "ralph.ronnquist"
Don't know what you mean!
afair, a CGI response that includes headers should have a very first line being the HTTP response status code, then the header lines, then an empty line, and then the payload.
You may omit the headers part and start with the empty line, and then the CGI runner would insert a standard header component (with a default status line).
But not all CGI runners insert a status line before headers when headers are given. For example, newlisp does insert a default status line and default headers (in its -http mode CGI runner), but possibly apache2 doesn't.
Though, I do this irregularly enough that I typically have to refer to the CGI spec's to remind myself about the nitty-gritty, and apache2 is (like most) a moving target anyhow.
@ralph.ronnquist - I'm sure that you are absolutely correct. However, I've never had to do what you are suggesting. Most of my experience has been on Apache using from CL to Perl to crank out CGI stuff. Newlisp is the only interpreter that has ever given me fits with these type of issues - given that I didn't trash that very important first line of code.
Lutx fixed this issue once. He may not want to deal with it again. I've got the very same script running just fine using CLISP2.49. I don't use -http unless I want to serve an Intranet.
thx for getting back to me.
What ralph said. Apache 2.4 got "strict" with the header validations to mitigate vulnerabilities when used as an http proxy, e.g. this one in 2016 (interesting read too):
https://httpd.apache.org/security/vulnerabilities_24.html#CVE-2016-8743
Notice the switch where you can turn that off (although not recommended):
http://httpd.apache.org/docs/2.4/mod/core.html#httpprotocoloptions
Could that have been the reason that that code worked recently and now it doesn't? (Switch got changed?)
I think keeping that setting to the default strict mode and going down the path that ralph suggested for your troubleshooting sounds good to me -- at least in the meantime before you hear anything from Lutz. Again, good luck.
(If I had the same stack, I could try to recreate the problem; however. along with not being a Windows user, I'm also not an Apache user. :)
@rickyboy All good stuff!
Thanks for the heads-up regarding Apache security stuff!
I think the problem is that the command-line prompt and initial copyright banner output needs to be suppressed.
Change the line:
To:
-- xytroxon
Quote from: "xytroxon"I think the problem is that the command-line prompt and initial copyright banner output needs to be suppressed.
Change the line:
To:
Thx! I'll give that a shot!!
Nope! No cigar! LOL
Try:
@lutz Thx, but that does not work either. Sorry!
Have you verified the byte sequence from the script, eg by running it manually and pipe its output into a file, which you the inspect with a hex code viewer? Or even perhaps like the following:
Note, I don't really use windows myself, so I'm guessing about piping the output from one command (thescript) to another (the unpacking into character codes). In any case, the resulting character codes list should then begin
@ralph.ronnquist
You bet that I have an Hex editor!! I don't see any garbage - which puzzles me. I've included a screen capture of the output. Now I'm wondering what Apache is choking on!! Pain in the Pa-toot!! LOL
Check how NEWLISPDIR and PROGRAMFILES are defined.
See here for an explanation of these directories: http://www.newlisp.org/downloads/newlisp_manual.html#startup
You could use the fowlowing script to check:
Quote
(println (env "NEWLISPDIR"))
(exit)
Both are set. I've installed newlisp in c:binnewlisp to make sure that I did not have future issues with spaces in directory names like "Program Files" etc. So NEWLISPDIR points to c:binnewlisp.
PROGRAMFILES is set to "Program Files". I changed it to c:binnewlisp.
The change did not fix my problem. BTW what did you do in 2015 to fix this very same issue? You modified something in newlisp.exe and asked me to try your modifications. One of them worked!!! :-)
duke,
Hex dump seems ok to me. I'm assuming you did something like:
Then, looked at newlisp-cgi.out in the hex editor. Maybe try also something like:
and maybe the same with the Ruby, CLISP scripts, etc. (i.e., all the CGI scripts that Apache doesn't choke on), and see if you can spot any difference between each output from those and newlisp-cgi.out.
Try the following:
Create a subdirectory "newlisp" in "Program Files" and put the newlisp.exe executable in there. If there are any modules you are going to use, create a "modules" directory inside the "newlisp" directory.
(env "NEWLISPDIR") => C:Program Files (x86)newlisp
Path of newlisp.exe is now: C:Program Files (x86)newlispnewlisp.exe
So both environment variables point to directories and the path in NEWLISPDIR must be in the executable path.
May be Windows finds the newlisp executable when using the command line, but not when called from Apache.
The original fix for Apache in 2015 was putting defaults (the above) for both environment variables.
@Lutz Thanks for all the trouble you went through!! Unfortunately, after doing
Quote
I keep wondering if it is not an Apache issue, but that question quickly fades away because I never get that response from any other interpreter! So it must be something that "print" or "println" is emitting? Maybe UTF8?
Quote from: "dukester"
That's not what your hex dump indicates.
It may be due to a configuration/interface issue — I believe that's the trail Lutz was going down and a good one to pursue.
It would be nice to see what text stream Apache is seeing. It may be something very unlike what we are imagining in our heads. Any way to do that? IOW, does Apache have a better way to diagnose these problems (like a debug level) because that exception message that you get tells us hardly anything at all and it may even be "lying" to us.
Also, if you don't know already, you may want to research into how Apache is calling out to / running your script. There may be some devil in the details there. If they didn't doc it well, you might have to go code spelunking. Ugh.
@rickyboy You know - I'm thinking that Lutz has already wasted enough time looking into this issue. If there were hundreds of folks using newlisp on Apache, it might then be a worthwhile investment of time. I'm sure that the folks at Apache would pretty much feel the same way! How many people are they aware of that use Apache to run newlisp scripts. I'm sure they couldn't be bothered.
How's the song go? "You got to know when to hold 'em, know when to fold 'em, Know when to walk away and know when to run. " Maybe that time is come. I'll pick at it now and then, but let's not waste anymore time. I thought the solution might be simple, and already solved. But there I went
duke, I wasn't saying for Lutz to look at it, but for *you* to look at it. lol!
No worries, I get it. I've been there many times. We do indeed have to make that determination: whether to spelunk or not. :) Sometimes, it's not worth it, especially when we have time constraints. I hope you get some time here and there to mess around with it ... and succeed!
All the best! rickyboy
Yeah! I realized that you meant
Hey!
Quote from: "dukester"
Quote
That error message shows that newlisp has an initial "complaint" about "TMP", which means what it says, and that complaint precedes the script output. It has nothing to do with your script.
So the quest should rather be to get rid of that piece of output. That amounts to making sure that TMP is set before newlisp is invoked, or, you get it compiled without that complaint.
I've got TMP/tmp/temp set in so many places that all hell wont have it. BTW I never get this bleeding tmp issue with any other script.
RIght. And yet it's missing for newlisp.
Apparently Apache has "SetEnv" and "PassEnv" directives that possibly would service your use case. Or else, I suppose you can wrap the script with a shell script for this purpose. Or if you like living on the edge, you could apply a binary editor to your newlisp executable and put a 00 at the beginning of the boring output string ;)
Good catch, Ralph. I read that message wrong (but yet there is the response header name, right there between the two single quotes). Didn't help that duke's OP didn't have the entire message there to begin with. Kinda faked me out. dukester! lol ;)
I configured Apache to run as a reverse proxy on a sub-directory to my cgi-bin containing newlisp scripts. I run newlisp as -http -d 8080 -w <fullpathtocgi-binnewLISP>
In Apache httpd.conf, simply load:
mod_proxy.so
mod_proxy_http.so
mod_proxy_http2.so
and insert the following directive in the same file:
<Location "/cgi-bin/newLISP/">
ProxyPass "http://127.0.0.1:8080/"
</Location>
Cigars and joy all around!!! :-)