SSL is coming - prepare...

[chalsall]

Originally Posted by Madpoo

I'm not sure how that would have worked before though. The only change I made was redirecting http -> https on the website.

OK. I'm just reporting what was observed.

Again, when you announced the migration last month I added a regex to redirect any "http://" requests to be "https://", and it worked. Literally "$URL =~ s/http:/https:/;" was the only change to the proxy script.

This morning I observed the proxy wasn't working; removing this transform resulted in the proxy working again.

If it would help at all, I could provide you (privately) with the transaction logs just before and just after the proxy started seeing the 404 errors.

Edit: Just saw your edit (LOL)... As mentioned in my post above, everything my proxy sees is targeted to "http://v5.mersenne.org/v5server/". The $URL variable is derived from $ENV{"REQUEST_URI"} (in Perl).

[Madpoo]

Originally Posted by chalsall

OK. I'm just reporting what was observed.

Again, when you announced the migration last month I added a regex to redirect any "http://" requests to be "https://", and it worked. Literally "$URL =~ s/http:/https:/;" was the only change to the proxy script.

This morning I observed the proxy wasn't working; removing this transform resulted in the proxy working again.

If it would help at all, I could provide you (privately) with the transaction logs just before and just after the proxy started seeing the 404 errors.

Edit: Just saw your edit (LOL)... As mentioned in my post above, everything my proxy sees is targeted to "http://v5.mersenne.org/v5server/". The $URL variable is derived from $ENV{"REQUEST_URI"} (in Perl).

LOL... well, cool then. I should have been more explicit about this only affecting the website URL and not the API.

I poked around on the website and saw a few hits from the GPU_to_72 submission spider ... they were all POST to /account/default.php on HTTP and they were getting the 301 redirect to HTTPS in response, as expected, however I didn't see a followup POST to the https link... it's not resubmitting after the redirect as far as I could tell.

I also saw Misfit doing insecure POSTs to the manual result, Misfit-specific URL, and although I'm not seeing a follow up POST resubmittal to the secure URL, I do see a GET to the same URL securely. That's odd.

Unfortunately in the case of Misfit. the data it's sending should be encrypted from the start, and while it's great that it's resubmitting securely, the fact that it still tried going to HTTP first means the info in there is still out there bubbling around the intertubes in the clear.

I think he'd said Misfit would be updated to use SSL but I suppose then it's a matter of making sure that change is there and that anyone using it is updating to the latest/greatest?

For now I think I'll have to exclude the misfit specific URL from being redirected. The SSL will still work so the transition on the client side can happen over time.

TL;DR:
--------
I'm searching for any other POSTs to HTTP that don't seem to be handled correctly, but for now the ones I saw are GPU72 submitting something to /account/default.php, and then Misfit. I'll exclude the Misfit hits from redirecting but the GPU72 posts, to me anyway, may seem to be harmless if they're not re-posting? Leaving those alone for now, hoping for a quick fix from your spider?

[chalsall]

Originally Posted by Madpoo

I poked around on the website and saw a few hits from the GPU_to_72 submission spider ... they were all POST to /account/default.php on HTTP and they were getting the 301 redirect to HTTPS in response, as expected, however I didn't see a followup POST to the https link... it's not resubmitting after the redirect as far as I could tell.

That's a bit funny, informative, and enlightening... The "GPU_to_72 Submission Spider" UserAgent string is presented by a Perl script I wrote about three years ago or so and released under the GPL. It was written very quickly largely as a proof of concept.

I had no idea anyone was still using it! Much better tools are now available (MISFIT et al).

So you know, the GPU72 site's observation spiders are using SSL, and all come from the same IP address. There don't appear to be any warnings being issued by them after this morning's transition.

I will continue to have the GPU72 proxy not transform any HTTP requests to be HTTPS unless and until you ask for that to be done.

Are we having fun yet? (I am! Complexity is interesting.)

[Madpoo]

Originally Posted by chalsall

That's a bit funny, informative, and enlightening... The "GPU_to_72 Submission Spider" UserAgent string is presented by a Perl script I wrote about three years ago or so and released under the GPL. It was written very quickly largely as a proof of concept.

I had no idea anyone was still using it! Much better tools are now available (MISFIT et al).

Weird... yeah, someone is using it, but for whatever reason the only hits I saw coming from it so far today were POSTs to that /account/default.php

Meh... well, if it's not you then I'm inclined to ignore it then. People do weird things.

Okay, I think I have the redirect for those misfit hits disabled. I'm just tracking any other peculiar POSTs. I see some script kiddiez (I assume) posting to weird things, looking for holes no doubt. Punks.

[Madpoo]

Originally Posted by Madpoo

Weird... yeah, someone is using it, but for whatever reason the only hits I saw coming from it so far today were POSTs to that /account/default.php

I think the person using that spider is AirSquirrels.

[Dubslow]

Originally Posted by Madpoo

Those were working great, *except* if it was to the root directory. For whatever reason, (.+) does NOT match a path of nothing (although in truth the path is "/".

I ended up changing things around on both rules to do a negative match of cgi-bin/(.+) instead which achieves the same goal and makes that home page redirect as needed.

The + modifier matches 1 or more repetitions of the previous pattern.
The * modifier matches 0 or more repetitions of the previous pattern.

In [1]: import re

In [2]: re.search("(.+)", "thing")
Out[2]: <_sre.SRE_Match object; span=(0, 5), match='thing'>

In [3]: re.search("(.+)", "")

In [4]: re.search("(.*)", "thing")
Out[4]: <_sre.SRE_Match object; span=(0, 5), match='thing'>

In [5]: re.search("(.*)", "")
Out[5]: <_sre.SRE_Match object; span=(0, 0), match=''>

[Madpoo]

Originally Posted by Dubslow

The + modifier matches 1 or more repetitions of the previous pattern.
The * modifier matches 0 or more repetitions of the previous pattern.

Oh geez... I should have thought of that. Send me back to Regex 101...

[Brain]

Originally Posted by chalsall

That's a bit funny, informative, and enlightening... The "GPU_to_72 Submission Spider" UserAgent string is presented by a Perl script I wrote about three years ago or so and released under the GPL. It was written very quickly largely as a proof of concept.

I had no idea anyone was still using it! Much better tools are now available (MISFIT et al).

So you know, the GPU72 site's observation spiders are using SSL, and all come from the same IP address. There don't appear to be any warnings being issued by them after this morning's transition.

I will continue to have the GPU72 proxy not transform any HTTP requests to be HTTPS unless and until you ask for that to be done.

Are we having fun yet? (I am! Complexity is interesting.)

I am still using it and it has broken:

20170223_215245 INFO:  Submission spider starting...
20170223_215245 INFO:  Attempting to log into PrimeNet.  This can take a little while...
20170223_215245 ERR :  Bad response:
301 Moved Permanently

Adding the "s" to http fixed it: my $PrimeNetURL = "https://www.mersenne.org/";

Could you please update version 0.24 on GPU72: http://www.gpu72.com/software/submit_spider

Tnx, Sebastian

[Madpoo]

Originally Posted by Brain

I am still using it and it has broken:

20170223_215245 INFO:  Submission spider starting...
20170223_215245 INFO:  Attempting to log into PrimeNet.  This can take a little while...
20170223_215245 ERR :  Bad response:
301 Moved Permanently

Adding the "s" to http fixed it: my $PrimeNetURL = "https://www.mersenne.org/";

Could you please update version 0.24 on GPU72: http://www.gpu72.com/software/submit_spider

Tnx, Sebastian

Sorry about that Brain.

After I got in touch with AirSquirrels (who was using it to auto-submit results), he switched it to "https" and it worked. I checked again later yesterday and saw someone else was also trying to submit to http and not having any luck but unfortunately I couldn't tell who it was (looking up the reverse IP didn't help at all).

I was hoping the unlucky soul would reach out for help or see this thread, so I'm glad you did. As far as I know, it was probably just the two of you using that, but now I'll check again and see if some other hits come in sporadically.

I think it may take a while for some people who are using it to make themselves known... it only tries when it has something to check in, I guess, but once it does, it'll retry every xx minutes since the POST to http would always fail.

Fortunately it seems like adding the "s" is all it needs and it'll submit whatever it needed to.

[Madpoo]

Originally Posted by Madpoo

I think it may take a while for some people who are using it to make themselves known... it only tries when it has something to check in, I guess, but once it does, it'll retry every xx minutes since the POST to http would always fail.

Fortunately it seems like adding the "s" is all it needs and it'll submit whatever it needed to.

Actually it looks like the other person I noticed using that submission spider is our own "petrw1". I might be wrong but that's the best I could suss out.

I'll have to PM him and mention this thread.

[Brain]

Thanks a lot for all your work and improvements, finder of the great M49, ruler of the DB.