Site Maintenance & Downtime (July 16/17, 2013)

Use this space for bug reports or to discuss site functionality and statistics.

Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Tue Jul 09, 2013 8:33 pm

Further investigation into the wonkiness we've been experiencing over the last several days has revealed that we are in fact, awesome. Okay, well it's revealed that the problems were a combination of some work being done with our hosts, a change in the server backup strategy we had in place, and quite a bit more traffic on the boards these days (as opposed to the olden days). :D

Anyhow, the three things combined spelled intermittently terrible performance, and connection timeouts happening a lot for some users. Even after our cloud server overlords finished up their stuff, we've wrestled with performance issues. For the short term we've reverted to the previous server backup strategy, and some behind-the-scenes tweaks (and a good swift kick or two) have helped improve performance a little. For a longer term solution (significantly increasing our capacity), we need to take the board down for a bit.

We're scheduling that down time for the night of July 16. Starting at 10pm 9pm PDT on July 16, and continuing until around 2am PDT on the morning of July 17, ePlaya will be offline for maintenance and improvements. Sorry for any inconvenience that the downtime will cause, but the increased capacity will keep us from experiencing performance problems or outages as we head into the home stretch for this year's burn.

Many many thanks to Cat and Backbone on the IT team for their help and support.
User avatar
trilobyte
Site Admin
 
Posts: 12710
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Mon Jul 15, 2013 10:37 pm

Bump, and update! The site will be down for maintenance and improvements tomorrow night. I will be taking the board offline at 9pm PDT (not 10pm, as originally announced). I apologize in advance for any inconvenience, but look forward to the improved performance we'll see once we finish up in the early hours of Wednesday morning.
User avatar
trilobyte
Site Admin
 
Posts: 12710
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 12:03 am

And...we're back!

MASSIVE thanks to Backbone for all his help and technological greatness!

Thanks everybody for your patience during the downtime, as well as over the last couple weeks with the board's slow performance. We should hopefully be good to go for the foreseeable future.
User avatar
trilobyte
Site Admin
 
Posts: 12710
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 12:10 am

Google's DNS servers still have the old IP address cached, so some users may not be able to access the board for another hour or two.
User avatar
trilobyte
Site Admin
 
Posts: 12710
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Elliot » Wed Jul 17, 2013 7:21 am

If we still believed in gods, you wizards of electric computing machines would be it.
Elliot's Naked Bicycle Service & Piano Bar - a 2015 BRC Theme Camp

---------------Bike come unglued? Take it to the nude dude!---------------


Image
User avatar
Elliot
 
Posts: 6177
Joined: Sun Dec 18, 2005 9:41 pm
Location: Clearlake, Northern California
Burning Since: 2006
Camp Name: Elliot’s Bicycle Service & Bevera...

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby aserendipity » Wed Jul 17, 2013 7:27 am

Thank you Trilo !
aserendipity
 
Posts: 1654
Joined: Mon Aug 20, 2007 8:34 pm

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 11:29 am

Things may be sporadically herky-jerky (yes, that's a technical term!) for the next dozen or so hours. Turns out that Google still hasn't updated all of it's cache, that could end up taking up to 24 hours. Your mileage may vary, but what I've noticed before is that 80% of the time page loads are pretty close to instantaneoous (I'm on a fast connection). The other 20% is still sluggish, or the page may even time out. Once things in DNS land are sorted, we should be back to whatever passes for normal around here :)
User avatar
trilobyte
Site Admin
 
Posts: 12710
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Canoe » Wed Jul 17, 2013 11:31 am

I'm getting response times all over the map. One page will load in < 2 seconds. The next, 45 seconds (most typical long time). The next will timeout with no response.
Just came back to the site and I was listed as still signed in...

ah
your post beat mine

I was wondering about all the bots I was seeing.
Odd. No bears to watch in the dump. Oh well, lets go across the road & pick blueberries.
.
... but don't harm the red dragon that frequents the area from time to time. He and I have an agreement.
User avatar
Canoe
 
Posts: 2090
Joined: Sun Jul 31, 2011 8:01 pm

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 11:59 am

There's been no change in the kinds and types of bots that we allow on the site (we let in the usual suspects...Google, Bing, Yahoo, the Alexa crawler, etc). You may be seeing more of them on the bottom of the page because I've opened up our 'Users Online' setting back to 30 minutes, which is what we had it set to for a long time.
User avatar
trilobyte
Site Admin
 
Posts: 12710
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Elderberry » Wed Jul 17, 2013 2:08 pm

Canoe wrote:I'm getting response times all over the map. One page will load in < 2 seconds. The next, 45 seconds (most typical long time). The next will timeout with no response.
Just came back to the site and I was listed as still signed in...

ah
your post beat mine

I was wondering about all the bots I was seeing.


I'm getting the same here.

Are you sure this is a DNS issue? There could be a problem accessing the site for the first time, but once the requester's ISP DNS server has received the correct information,it is cached and all further requests are then answerd by that server. Doesn't make sense to me.
JK
Image
http://www.mudskippercafe.com
When I was a kid I used to pray every night for a new bicycle.
Then I realised that the Lord doesn't work that way so I stole one and asked Him to forgive me.
User avatar
Elderberry
Moderator
 
Posts: 12618
Joined: Tue Jul 17, 2007 10:00 pm
Location: Los Angeles
Burning Since: 2007
Camp Name: Mudskipper Cafe

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby lemur » Wed Jul 17, 2013 2:36 pm

yeah! or something.

it doesnt seem like a regular DNS propogation issue to me, i.e. the TTL on something needs to expire in a cache before the new IP shows up on a DNS resolve request



the TTL on eplaya.burningman.com seems to be 15 seconds..

if ya sit there and ping eplaya.burningman.com a few times (like in different requests..) after about 15 seconds you might encounter a part where it says it doesnt exist

~: ping eplaya.burningman.com
PING eplaya.burningman.com (54.213.17.175) 56(84) bytes of data.
^C
--- eplaya.burningman.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3505ms

~: ping eplaya.burningman.com
ping: unknown host eplaya.burningman.com
~: ping eplaya.burningman.com
ping: unknown host eplaya.burningman.com


it looks to me like after 15 seconds its hitting up another one of the resolvers listed in the WHOIS:

Domain servers in listed order:
NS3.BURNINGMAN.COM 65.161.206.37
NS.HYPERREAL.ORG
NS1.BURNINGMAN.COM 209.237.226.70

(i dont know if the middle one not having an IP address attached to it may cause problems)

all 3 of the resolvers report 'the correct' ip address for eplaya..

~: dig 65.161.206.37 eplaya.burningman.com
eplaya.burningman.com. 15 IN A 54.213.17.175

~: dig NS.HYPERREAL.ORG eplaya.burningman.com
eplaya.burningman.com. 15 IN A 54.213.17.175

~: dig 209.237.226.70 eplaya.burningman.com
eplaya.burningman.com. 15 IN A 54.213.17.175



my guess is that.. some systems are balking at the 2nd nameserver entry in the WHOIS being blank.

and with such a short TTL.. youre ending up with that 2nd nameserver (without a IP addy) in the WHOIS often..



just a guess!


(in other news: i think the previous issues were unrelated to this)

as for WHOIS records... THOSE may take a while to update as the domain delegater updates them on their own schedule, or not..
Don't link to anything here!
User avatar
lemur
 
Posts: 3599
Joined: Fri Feb 13, 2009 3:07 pm
Location: Madagascar
Camp Name: Plug N Play Camp

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Bless » Wed Jul 17, 2013 7:12 pm

This site isn't working on my desktop computer, only my iPhone. I tried three different browsers and tried clearing my history and caches. Please help!
FUCK YOU I'M A WIZARD. FUCK YOU I'M A SHARK.
User avatar
Bless
 
Posts: 574
Joined: Tue Jul 16, 2013 8:43 pm
Location: District of Columbia
Burning Since: 2012

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 7:29 pm

The problem should work itself out by around midnight PDT tonight, as the the various DNS servers driving the web replace the old info with the new.
User avatar
trilobyte
Site Admin
 
Posts: 12710
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby msj2u » Wed Jul 17, 2013 8:13 pm

Trilobyte,

Should have said this earlier, thanks to you and the team for the work on the site!

MSJ
The older I get, the less patience I have for stupid
msj2u
 
Posts: 310
Joined: Mon Feb 04, 2013 11:17 pm
Location: Inland Empire, CA
Burning Since: 2003

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Elderberry » Wed Jul 17, 2013 9:21 pm

pinging is never a reliable indicator as it is often blocked either on the server or the firewall.
JK
Image
http://www.mudskippercafe.com
When I was a kid I used to pray every night for a new bicycle.
Then I realised that the Lord doesn't work that way so I stole one and asked Him to forgive me.
User avatar
Elderberry
Moderator
 
Posts: 12618
Joined: Tue Jul 17, 2007 10:00 pm
Location: Los Angeles
Burning Since: 2007
Camp Name: Mudskipper Cafe

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby lemur » Thu Jul 18, 2013 12:47 am

the point of that wasnt the ping ..it was the name resolution
Don't link to anything here!
User avatar
lemur
 
Posts: 3599
Joined: Fri Feb 13, 2009 3:07 pm
Location: Madagascar
Camp Name: Plug N Play Camp

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Thu Jul 18, 2013 9:44 am

If anyone's still experiencing issues with connecting to the site, or maintaining that connection, please send me a private message with the details.
User avatar
trilobyte
Site Admin
 
Posts: 12710
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Bless » Thu Jul 18, 2013 10:05 am

I was having sporadic problems this morning, but it seems to be all fixed now. Thank you so much!
FUCK YOU I'M A WIZARD. FUCK YOU I'M A SHARK.
User avatar
Bless
 
Posts: 574
Joined: Tue Jul 16, 2013 8:43 pm
Location: District of Columbia
Burning Since: 2012


Return to Site Function

Who is online

Users browsing this forum: No registered users and 0 guests