Badger
Joined: Nov 16, 2013
Posts: 4536
Wisconsin
|

Posted on Mar 04 2015 09:16 PM
Brian wrote:
So either we stop hot-linking altogether, or the site automatically downloads non-secure images and uploads them to our Amazon S3 bucket.
Keep in mind we will still allow people to upload their own images (these will go into our Amazon S3 bucket).

If you allow the automatic downloads of non-secure images would it be fair to say there is no sense in going through the effort? You know the threat as well as anyone. In my mind these issues are the same as having unprotected sex based solely on the recommendation of someone you never met. I see this all the time with other sites, even ones that are initially secure. But some internal link of theirs may try to take me to a server that has an invalid site certificate. (and my security software trips all kinds of bells & whistles). These get exploited all the time.
It seems to me that the safer path, if the individual member wants to include an image, is to let that download burden from an unsecure site be on them. As you say, they can still directly upload the image in their post. Am I missing something in the nut you're trying to crack? (besides the bigger hammer part) Seems the cyber-credibility of the site is (or should be) extremely important as well.
— Wes
SoCal ex-pat with a snow shovel
DISCLAIMER: The above is opinion/suggestion only & should not be used for mission planning/navigation, tweaking of instruments, beverage selection, or wardrobe choices.
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 05 2015 08:23 AM
Well the issue is if I disable hot linking, will it change the fundamental "feeling" of the website, and drive people away?
Having the server download the unsecure image and upload it to a secure location we control is much safer than letting a user's browser do it.
I now understand why SSL-using sites like reddit (maybe the largest discussion site on the Internet) does not allow embedding images. At the other extreme Facebook copies every image you hot link to its own servers. They can afford to do that (shockingly).
I'm thinking of doing this:
Phase 1 - disable all hot linking (except to SG101 itself or our secure bucket).
See how that goes... gauge the user backlash....
Phase 2 - after we switch to SSL and see how it goes, let the user hot link, but the server will move the photo to our bucket behind the users back (ala Facebook). Since we don't have that many active users it should scale for now.
Phase 2 increases the cost of running the site over time as we'll have to pay more and more to Amazon.
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
|
Badger
Joined: Nov 16, 2013
Posts: 4536
Wisconsin
|

Posted on Mar 05 2015 09:08 AM
Brian wrote:
Having the server download the unsecure image and upload it to a secure location we control is much safer than letting a user's browser do it.
...
Phase 2 - after we switch to SSL and see how it goes, let the user hot link, but the server will move the photo to our bucket behind the users back (ala Facebook). Since we don't have that many active users it should scale for now.
Phase 2 increases the cost of running the site over time as we'll have to pay more and more to Amazon.
Thanks Brian; first sentence helps understand that better now, got it.
In terms of recurring costs, if there's no provision for compressing & archiving stuff especially the high-payload image stuff, I think the active user corps will bear the freight. This site remains the benchmark for its genre, my heavily-biased opinion only. Also, most other fora, hitting a years-old thread to find a missing image from one of the posts is not unusual at all around the web.
Sounds like a good plan.
— Wes
SoCal ex-pat with a snow shovel
DISCLAIMER: The above is opinion/suggestion only & should not be used for mission planning/navigation, tweaking of instruments, beverage selection, or wardrobe choices.
|
Tuck
Joined: Sep 02, 2006
Posts: 3166
Denver, CO
|

Posted on Mar 05 2015 04:32 PM
I think disabling hot linking will be unpopular. I suppose it applies to all media files? Including audio and video?
It's always awkward when you are looking at a post about an image that isn't there.
|
Badger
Joined: Nov 16, 2013
Posts: 4536
Wisconsin
|

Posted on Mar 05 2015 05:45 PM
Tuck wrote:
I think disabling hot linking will be unpopular. I suppose it applies to all media files? Including audio and video?
That's a good question. If it's not just image files that would eat the site alive in storage in short order I'd think.
— Wes
SoCal ex-pat with a snow shovel
DISCLAIMER: The above is opinion/suggestion only & should not be used for mission planning/navigation, tweaking of instruments, beverage selection, or wardrobe choices.
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 07 2015 02:27 PM
Tuck wrote:
I think disabling hot linking will be unpopular. I suppose it applies to all media files? Including audio and video?
It only applies to hot-linking images, i.e. using the [image](http://example.com/image.jpg) Markdown in forum posts and comments. You will still be able to embed YouTube and Vimeo videos.
It's always awkward when you are looking at a post about an image that isn't there.
That's why I am planning on saving all the hot-linked images to date in an Amazon S3 bucket. They are rapidly rotting even as we speak. Close to 50% of the hot-linked images posted in the first few months of SG101 in 2006 are now bad links.
In the future, once we go SSL, at least at first, you won't be able to hot link. But you can still upload images to our bucket. I will look into restoring hot-linking, but the site is really going to download any hot-linked images and re-upload them to our secure storage behind your back. That's the plan for now, but I'm still thinking about it and am open to suggestions.
This SSL task is much more complicated than I first thought.
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
Last edited: Mar 07, 2015 14:28:34
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 07 2015 02:33 PM
I'm going to put this SSL task on the back burner for just a bit. The first reason is my laptop I do all developing on is ailing and I need to send it off for repair. Secondly, I've had a few people report problems with registering with SG101, including a certain high profile AMA special guest... It's not good when people can't register at your site. I made the sign up process too difficult in order to prevent automated sign-ups from spammers. I'll see if I can make it easier but still make it unusual enough that automated bots can't easily register.
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 07 2015 02:46 PM
So to recap: I'd really like to get SG101 on SSL in order to better protect your privacy and security.
However, there is a trade-off: you won't be able to hot-link images like you can right now, at least for while until I figure something out. You will still be able to upload and embed photos from your computer / phone / device.
Is that a trade off you are willing to make?
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
|
Badger
Joined: Nov 16, 2013
Posts: 4536
Wisconsin
|

Posted on Mar 07 2015 02:53 PM
Brian wrote:
So to recap: I'd really like to get SG101 on SSL in order to better protect your privacy and security.
However, there is a trade-off: you won't be able to hot-link images like you can right now, at least for while until I figure something out. You will still be able to upload and embed photos from your computer / phone / device.
Is that a trade off you are willing to make?
If that's the solicitation for a vote, I'm in & willing to make that tradeoff.

"AMA" = American Motorcycle Association?
— Wes
SoCal ex-pat with a snow shovel
DISCLAIMER: The above is opinion/suggestion only & should not be used for mission planning/navigation, tweaking of instruments, beverage selection, or wardrobe choices.
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 07 2015 03:19 PM
AMA == Ask Me Anything
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
|
Badger
Joined: Nov 16, 2013
Posts: 4536
Wisconsin
|

Posted on Mar 07 2015 05:11 PM
Check, thanks.
— Wes
SoCal ex-pat with a snow shovel
DISCLAIMER: The above is opinion/suggestion only & should not be used for mission planning/navigation, tweaking of instruments, beverage selection, or wardrobe choices.
|
Ariel
Joined: Aug 29, 2009
Posts: 1556
Israel
|

Posted on Mar 08 2015 04:17 AM
Brian wrote:
Is that a trade off you are willing to make?
Absolutely. This is only 1st line of defense anyway. It is important. And whoever's not 'paranoid" now, will be in a couple of years.
Your priority of making the registration smooth first, is correct.
For longevity and historical reference, surely all the pics under the Amazon bucket will serve us better. That one will need to be backup-ed too...
Is there a way maybe, to have a script take the image from a web location, and automatically upload a copy to the bucket? That will enable bypassing the user (disk's) side through the whole process.
And thank you.
|
Fady
Joined: Mar 07, 2010
Posts: 2272
North Carolina
|

Posted on Mar 08 2015 08:13 AM
Little late getting caught up on this thread, B.
Kudo's for your efforts to go SSL 
I like the direction you're going (automagically move non-SSL images behind the scenes).
In the meantime, so long as you have simple error handling or alternatives for users, no harm, no foul. Go for it!
BTW - The more I think about just turning non-SSL image links into hyperlinks (I think that was an initial consideration) defeats the purpose. I'd be willing to bet an overwhelming majority of site users would still click the link to see the image - especially if it's from a 'known' active member, so from the security standpoint, it hasn't really improved things for users. May save us from malicious just registered/first post/bogus content thread, but maybe not much more. Just a thought.
— Fady
El Mirage @ ReverbNation
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 08 2015 10:38 AM
Thanks Ariel and Fady for your thoughts.
DreadInBabylon wrote:
Your priority of making the registration smooth first, is correct.
I worked on this almost all afternoon and it is almost ready to roll out. It should be a much nicer user experience.
Is there a way maybe, to have a script take the image from a web location, and automatically upload a copy to the bucket? That will enable bypassing the user (disk's) side through the whole process.
Yes this is the ultimate goal. After sleeping on it I think I should just go ahead and implement this before we go to SSL, that way no one notices an interruption in (apparent) hot linking service.
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 08 2015 10:47 AM
Onslow_Beach wrote:
I like the direction you're going (automagically move non-SSL images behind the scenes).
Yeah I think it is the way to go. We'll have to watch the performance and Amazon costs. It will probably make posting take a few more seconds than what we have now, especially if there are a lot of images. I'll probably limit the number of images per post too. It will be sufficiently high that it should not be hit often; it is just there to prevent malicious behavior.
BTW - The more I think about just turning non-SSL image links into hyperlinks (I think that was an initial consideration) defeats the purpose. I'd be willing to bet an overwhelming majority of site users would still click the link to see the image - especially if it's from a 'known' active member, so from the security standpoint, it hasn't really improved things for users. May save us from malicious just registered/first post/bogus content thread, but maybe not much more. Just a thought.
Well you are right in the sense that it could still be a malicious link. This issue is that modern browsers will rightly balk at embedding a non-SSL image onto a SSL secured page because someone could be tampering with the connection to the non-SSL image, and could inject javascript or something into your otherwise secure page (the so called 'man in the middle' attack).
(Sorry for geeking out but I'm fascinated by all this).
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 08 2015 11:14 AM
Ok, new registration system is up and running. Any feedback from newly registered members would be appreciated!
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
|
Badger
Joined: Nov 16, 2013
Posts: 4536
Wisconsin
|

Posted on Mar 08 2015 05:10 PM
Brian wrote:
(Sorry for geeking out but I'm fascinated by all this).
No apologies necessary. My step-son now is certified to teach CND and Ethical Hacking and keeps me abreast of the threat. I know not that many years ago when I was doing support for our .mil network at the same base, it is sobering to look through router logs and see that bad guys have been banging on your door several thousand times a day, just looking for an opportunity. To have the site end up as an unwitting platform for such shenanigans is not how we want to be regarded.
Geek on!
— Wes
SoCal ex-pat with a snow shovel
DISCLAIMER: The above is opinion/suggestion only & should not be used for mission planning/navigation, tweaking of instruments, beverage selection, or wardrobe choices.
Last edited: Mar 08, 2015 17:11:41
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 09 2015 10:06 PM
Brian wrote:
Ok, new registration system is up and running. Any feedback from newly registered members would be appreciated!
I broke the registration, but it should be fixed now. I thought it was odd no one had registered in 24 hours... 
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
|
Brian
Joined: Feb 25, 2006
Posts: 19334
Des Moines, Iowa, USA
|

Posted on Mar 21 2015 02:30 PM
As part of operation SSL, I just converted all 5115 video embeds we have to use https. Interesting to note some 10% of the videos posted in almost 3 years are either 404 (gone) or 401 (not authorized; i.e. YouTube pulled it for copyright or the original content owner asked YouTube to yank it).
That leaves only the hundreds of thousands of photos to convert.
— Site dude - S3 Agent #202
Need help with the site? SG101 FAQ - Send me a private message - Email me
"It starts... when it begins" -- Ralf Kilauea
|
Badger
Joined: Nov 16, 2013
Posts: 4536
Wisconsin
|

Posted on Mar 21 2015 03:24 PM
Wow. Holy linkage-replacement.
— Wes
SoCal ex-pat with a snow shovel
DISCLAIMER: The above is opinion/suggestion only & should not be used for mission planning/navigation, tweaking of instruments, beverage selection, or wardrobe choices.
|