STEP has launched... but server crashed immediately

Want to talk about tickets? You've come to the right place

Re: STEP has launched... but server crashed immediately

Postby bradtem » Fri Mar 02, 2012 2:04 pm

I gotta say, I have always felt first-come first-served is not so different from a lottery sometimes.

It is different, of course. It does favour eagerness and spare time, as well as luck. And people do attribute failure to win more to themselves than to luck sometimes. It is much less prone to oversubscription (though as prone, if not more, to various established scalper systems which have been tuned for FCFS ticket systems.) Your bad luck is a bit more apparent -- you see that you didn't get in because you didn't get one of the lucky slots within the ticket server's capacity, as opposed to the opaque "the random number generator did not smile on you."

But neither system is particularly good for this purpose.

It should be noted that Amazon Web Services has immense capacity -- but you have to plan to use it, and pay for it. If your app doesn't know what to do with the thousands of instances it can buy on AWS, it won't make could use of them. Based on data, there were about 70,000 lottery entries (many for 2 tickets) and about 24,000 winners. So there could be at most around 45,000 people who might try to join STEP. Don't know what fraction of them were pounding on servers at noon. Of course, once you get slow, you get even slower as people keep retrying and reloading on you.
See giant panoramas of BRC: http://www.templetons.com/brad/burn
User avatar
bradtem
 
Posts: 440
Joined: Fri Sep 05, 2003 12:27 pm
Location: Silicon Valley
Burning Since: 1998

Re: STEP has launched... but server crashed immediately

Postby lemur » Fri Mar 02, 2012 2:04 pm

I REFRESHED 517 TIMES.. WHY ISNT IT WORKING?!
Don't link to anything here!
User avatar
lemur
 
Posts: 3599
Joined: Fri Feb 13, 2009 3:07 pm
Location: Madagascar
Camp Name: Plug N Play Camp

Re: STEP has launched... but server crashed immediately

Postby vargaso » Fri Mar 02, 2012 2:08 pm

suicyco wrote:
Robishop wrote:The Ticketing for STEP seemed to be hosted by Amazon Web Services, yes that Amazon... Did you notice the URL theat was in the addrwess of the links sent?

No matter, even Amazon is slow at times I guess.

Amazon Web Services
http://www.amazonaws.com/
Amazon Web Services (AWS) delivers a set of services that together form a reliable, scalable, and inexpensive computing platform “in the cloud”.


The way it appeared to me was it was bouncing around several servers. The initial emailed link was to BM's servers. Which redirected to Amazon. The "SAVE" button page looked to be hosted on Amazon. However, hitting save seemed to redirect traffic back to burningman.com. I noticed that eplaya was very unresponsive during all of this. I got to the amazon page several times but it kept timing out when trying to redirect back to BM's servers. Eventually I got the confirmation message, but I think the big issue was the connection to BM's servers, not Amazon. Eplaya is working just fine again, so I would guess they shared the same connection. I doubt BM has as huge a pipe as Amazon.



Hitting save redirected you back to the confirmation screen on burningman.com, but the Save request went to Amazon servers. As others have said, it's not Amazon, it's inTicketing. Amazon servers can handle almost unlimited traffic depending on how you configure it for your purposes. All they provide are the physical boxes, it's up to the client (inTicketing) to configure it.
User avatar
vargaso
 
Posts: 453
Joined: Mon Sep 17, 2007 11:26 pm
Location: Lincoln, CA
Burning Since: 2007
Camp Name: Suburbanoya

Re: STEP has launched... but server crashed immediately

Postby trilobyte » Fri Mar 02, 2012 2:10 pm

Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.
User avatar
trilobyte
Site Admin
 
Posts: 10636
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: STEP has launched... but server crashed immediately

Postby RevDusty » Fri Mar 02, 2012 2:12 pm

trilobyte wrote:Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.


'Cuz I'm making big bucks where I am solving these and similar problems. Seriously.
Rev. Dusty Sportswood
RevDusty
 
Posts: 121
Joined: Thu Jul 14, 2005 9:37 pm
Location: Oakland, CA

Re: STEP has launched... but server crashed immediately

Postby RevDusty » Fri Mar 02, 2012 2:15 pm

trilobyte wrote:Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.


And, Trilo, TicketBastard, as much as I loathe them, don't have this problem. I don't get kicked off the servers. I may not get the ticket ,but I don't get booted off after getting through. Big difference.
Rev. Dusty Sportswood
RevDusty
 
Posts: 121
Joined: Thu Jul 14, 2005 9:37 pm
Location: Oakland, CA

Re: STEP has launched... but server crashed immediately

Postby lemur » Fri Mar 02, 2012 2:15 pm

and as we know! even the best doods in the world end up fucking up their config ....even the people with plenty of capacity end up with bottlenecks and issues that just didnt come up in smaller scale testing.... even if you were able to simulate 50,000 people hitting it at once, odds are you probably also werent accoutning for 25,000 of them constantly hitting refresh over and over..

theres basically no real way to simulate the real thing beforehand, you just gotta hope for the best .. hope you did it right..

bonnaroo got screwed over by their implementation when their tickets went on sale.. and they surely have a bigger budget than burning man.. have 15+ corp sponsors (including DELL and StubHub) who surely allow them to be able to dedicate more cash to the project than burning man can.. and they still had problems..

s0 yeah, it sucks that they had problems, but it wasnt like issues would be unexpected..... and as bradtem mentions, as soon as things start getting slow the problem becomes compounded because you not only have 50,000 people hitting the site at once.... but you have 50,000 people refreshing over and over numerous times.. basically making themselves feel to the servers like double, or triple.. or even more than that.
Last edited by lemur on Fri Mar 02, 2012 2:18 pm, edited 1 time in total.
Don't link to anything here!
User avatar
lemur
 
Posts: 3599
Joined: Fri Feb 13, 2009 3:07 pm
Location: Madagascar
Camp Name: Plug N Play Camp

Re: STEP has launched... but server crashed immediately

Postby vargaso » Fri Mar 02, 2012 2:16 pm

RevDusty wrote:
trilobyte wrote:Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.


'Cuz I'm making big bucks where I am solving these and similar problems. Seriously.


The tools and methodologies exist, they really do (online banking for instance, WAY more complicated and much larger user base), it's just a matter of forking over the dough to get the right people in there. I don't fault BMORG this year, it's all new, even though some of us have been suggesting they find a new vendor for awhile now. If next year the same issues happen, then I don't know. And personally, I don't have the expertise, I'm a front-end developer, but I've worked on top of massively powerful systems (online banking and health care) that handle stupid amounts of traffic.
User avatar
vargaso
 
Posts: 453
Joined: Mon Sep 17, 2007 11:26 pm
Location: Lincoln, CA
Burning Since: 2007
Camp Name: Suburbanoya

Re: STEP has launched... but server crashed immediately

Postby lemur » Fri Mar 02, 2012 2:19 pm

banks and healthcare providers also have billion $ budget.
Don't link to anything here!
User avatar
lemur
 
Posts: 3599
Joined: Fri Feb 13, 2009 3:07 pm
Location: Madagascar
Camp Name: Plug N Play Camp

Re: STEP has launched... but server crashed immediately

Postby theCryptofishist » Fri Mar 02, 2012 2:20 pm

lemur wrote:banks and healthcare providers also have billion $ budget.

That's what you "99%ers" claim, anyway.
Simon's real sig line?

Embrace the Sock

Winners never quilt, quilters never win...
User avatar
theCryptofishist
 
Posts: 37425
Joined: Mon Feb 23, 2004 10:28 am
Location: In Exile
Burning Since: 2017

Re: STEP has launched... but server crashed immediately

Postby RevDusty » Fri Mar 02, 2012 2:21 pm

lemur wrote:banks and healthcare providers also have billion $ budget.


Yes, they do.

But Amazon (and others) greatly equalize out those capabilities, especially when we are discussing renting machines for 30 or so minutes.
Rev. Dusty Sportswood
RevDusty
 
Posts: 121
Joined: Thu Jul 14, 2005 9:37 pm
Location: Oakland, CA

Re: STEP has launched... but server crashed immediately

Postby vargaso » Fri Mar 02, 2012 2:22 pm

lemur wrote:banks and healthcare providers also have billion $ budget.


That is true. Even so, this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.
User avatar
vargaso
 
Posts: 453
Joined: Mon Sep 17, 2007 11:26 pm
Location: Lincoln, CA
Burning Since: 2007
Camp Name: Suburbanoya

Re: STEP has launched... but server crashed immediately

Postby suicyco » Fri Mar 02, 2012 2:22 pm

trilobyte wrote:Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.


I don't think it was any sort of huge disaster. It just failed ungracefully. The 2000 who got in, did so in a relatively short amount of time. It could have been designed to fail in a more friendly way, however it did its purpose.

I work in IT and design systems that handle orders of magnitude larger datasets and network traffic. However, these systems are for much deeper pockets than BM. I'm not complaining though, it is what it is. I mean it only really lasted for what, 30 minutes? Frustrated some folks but hey, its over, you got in or you didn't, and thats that. I personally have very slim hopes of receiving tickets via the STEP program. I am working other avenues as best I can. I already have a ticket for myself if I cannot get one through STEP, however our camp needs two more.
suicyco
 
Posts: 13
Joined: Sun Jan 15, 2012 9:18 pm
Location: Orange County, CA
Burning Since: 2012

Re: STEP has launched... but server crashed immediately

Postby lemur » Fri Mar 02, 2012 2:24 pm

vargaso wrote:
lemur wrote:banks and healthcare providers also have billion $ budget.


That is true. Even so, this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.


how do you know ?
Don't link to anything here!
User avatar
lemur
 
Posts: 3599
Joined: Fri Feb 13, 2009 3:07 pm
Location: Madagascar
Camp Name: Plug N Play Camp

Re: STEP has launched... but server crashed immediately

Postby hellolinds » Fri Mar 02, 2012 2:25 pm

Really Inticketing doesn't give a fuck! and they don't have to. The Borg keeps using them again and again. Fail after fail. Failed last year? Here, have a go at it again. oopsie, cheaped out a bit on that 30 minute rush.

and still, there are 2,000 peeps that got through (happy I wouldn't say, just hanging onto small pathetic hopes...at least that's where I'm at) And so BMORG doesn't care either. It's been a total shitshow, how much worse could it get for them?
hellolinds
 
Posts: 21
Joined: Mon Nov 28, 2011 11:51 pm

Re: STEP has launched... but server crashed immediately

Postby lemur » Fri Mar 02, 2012 2:26 pm

RevDusty wrote:
lemur wrote:banks and healthcare providers also have billion $ budget.


Yes, they do.

But Amazon (and others) greatly equalize out those capabilities, especially when we are discussing renting machines for 30 or so minutes.


they still need all of the resources to design a system using the amazon infrastructure.... it doesnt build itself!
Don't link to anything here!
User avatar
lemur
 
Posts: 3599
Joined: Fri Feb 13, 2009 3:07 pm
Location: Madagascar
Camp Name: Plug N Play Camp

Re: STEP has launched... but server crashed immediately

Postby vargaso » Fri Mar 02, 2012 2:26 pm

RevDusty wrote:
lemur wrote:banks and healthcare providers also have billion $ budget.


Yes, they do.

But Amazon (and others) greatly equalize out those capabilities, especially when we are discussing renting machines for 30 or so minutes.


This too. That's the genius of Amazon's cloud service, you don't have to buy and maintain anything, just rent. And has RevDusty says, it can literally be in 30 minute increments. You can rent a tiered solution for these kinds of things where your capacity is at maximum the first 30 minutes, then gradually scales down, as does the pricing.

With that said, hats off to the ticket dept., this has to have been the most stressful month of their BM lives, dealing with unprecedented demand and scrutiny. I don't envy them (unless they get a ticket out of the deal, ha!).
User avatar
vargaso
 
Posts: 453
Joined: Mon Sep 17, 2007 11:26 pm
Location: Lincoln, CA
Burning Since: 2007
Camp Name: Suburbanoya

Re: STEP has launched... but server crashed immediately

Postby vargaso » Fri Mar 02, 2012 2:28 pm

lemur wrote:
vargaso wrote:
lemur wrote:banks and healthcare providers also have billion $ budget.


That is true. Even so, this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.


how do you know ?


Because I work in this industry and have built the front-end aspect of contingency plans for server overload scenarios, wherein users get redirected to a landing page with explanations and instructions on what to do, so that people don't just sit there with a blank screen thinking their request is being processed.
User avatar
vargaso
 
Posts: 453
Joined: Mon Sep 17, 2007 11:26 pm
Location: Lincoln, CA
Burning Since: 2007
Camp Name: Suburbanoya

Re: STEP has launched... but server crashed immediately

Postby lemur » Fri Mar 02, 2012 2:31 pm

vargaso wrote:
lemur wrote:
vargaso wrote:
this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.


how do you know ?


Because I work in this industry and have built the front-end aspect of contingency plans for server overload scenarios, wherein users get redirected to a landing page with explanations and instructions on what to do, so that people don't just sit there with a blank screen thinking their request is being processed.



so... you dont know what resources burning man/inticketing have available.... and therefore you dont know if this process could have been handled any better.

experience elsewhere doesnt exactly translate to knowing what you could do if put in the shoes of the burningman/inticketing people today...

yeah, i can imagine a way it could be handled better too.... if i had an unlimited budget and as many resources as i could imagine.. but, i dont know what their budget is and i dont know what resources are available... perhaps this is the best that could have been done, perhaps it isnt... but none of us really know unless we are actually on the inside
Don't link to anything here!
User avatar
lemur
 
Posts: 3599
Joined: Fri Feb 13, 2009 3:07 pm
Location: Madagascar
Camp Name: Plug N Play Camp

Re: STEP has launched... but server crashed immediately

Postby RevDusty » Fri Mar 02, 2012 2:32 pm

vargaso wrote:
lemur wrote:
vargaso wrote:That is true. Even so, this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.


how do you know ?


Because I work in this industry and have built the front-end aspect of contingency plans for server overload scenarios, wherein users get redirected to a landing page with explanations and instructions on what to do, so that people don't just sit there with a blank screen thinking their request is being processed.


...and ultimately, if you do that right, you reduce the load on the servers from people who don't know what has happened to their requests.
Rev. Dusty Sportswood
RevDusty
 
Posts: 121
Joined: Thu Jul 14, 2005 9:37 pm
Location: Oakland, CA

Re: STEP has launched... but server crashed immediately

Postby JungoRoad » Fri Mar 02, 2012 2:33 pm

Dearest Troll-oh-bite,

"Moderator" is from the root "moderate".

Jungo Road
JungoRoad
 
Posts: 2
Joined: Thu Feb 02, 2012 10:23 am

Re: STEP has launched... but server crashed immediately

Postby beer » Fri Mar 02, 2012 2:36 pm

Look familiar ?

Image


I started clicking at 12:00 and did so at least 40-50 times. At least 5 of those times I got the page with the button to click to confirm, and got the same message after clicking it.
At 12:18 I finally got thru, and received the email a few minutes later that we're on the list (I requested 2 tickets).

It would be interesting to see how long it took to fill the queue. 20 minutes sound about right ?


k.
User avatar
beer
 
Posts: 42
Joined: Fri Jan 20, 2012 12:04 pm
Burning Since: 2012
Camp Name: Kidsville

Re: STEP has launched... but server crashed immediately

Postby vargaso » Fri Mar 02, 2012 2:36 pm

lemur wrote:
so... you dont know what resources burning man/inticketing have available.... and therefore you dont know if this process could have been handled any better.

experience elsewhere doesnt exactly translate to knowing what you could do if put in the shoes of the burningman/inticketing people today...

yeah, i can imagine a way it could be handled better too.... if i had an unlimited budget and as many resources as i could imagine.. but, i dont know what their budget is and i dont know what resources are available... perhaps this is the best that could have been done, perhaps it isnt... but none of us really know unless we are actually on the inside


The thing is, Burning Man tickets are about the most expensive tickets for any event on the planet. People pay an assload of dough for the whole thing. I give BM WAY more slack that I would any other entity because of what it is and how much it means to me. But at some point, they have to step (har!) up and face reality. Like I've said, this year is a learning experience for everyone and I don't fault them one bit. I'm hoping things improve next year.
User avatar
vargaso
 
Posts: 453
Joined: Mon Sep 17, 2007 11:26 pm
Location: Lincoln, CA
Burning Since: 2007
Camp Name: Suburbanoya

Re: STEP has launched... but server crashed immediately

Postby Sassy Britches » Fri Mar 02, 2012 2:38 pm

vargaso wrote:
Because I work in this industry and have built the front-end aspect of contingency plans for server overload scenarios, wherein users get redirected to a landing page with explanations and instructions on what to do, so that people don't just sit there with a blank screen thinking their request is being processed.


That was the frustrating part.

I don't know much at all about how technology works, and I also really don't think that the BMORG has ill intentions - I think their hearts are with the community - but that doesn't negate the feelings of frustration and sadness that are prevalent among our friends and so many others who want so much to be home on the Playa this year... Nor does it stop us from hoping that all who do want to be there will find their way to face value tickets.
User avatar
Sassy Britches
 
Posts: 89
Joined: Sat Nov 12, 2011 7:09 pm
Location: Phoenix

Re: STEP has launched... but server crashed immediately

Postby remi » Fri Mar 02, 2012 2:38 pm

lemur wrote:banks and healthcare providers also have billion $ budget.


BM makes a few bucks too.
Can you at least admit that nuclear explosions are awesome!?
User avatar
remi
 
Posts: 482
Joined: Fri Sep 23, 2011 12:58 pm
Location: Calgary, AB, Canada

Re: STEP has launched... but server crashed immediately

Postby theCryptofishist » Fri Mar 02, 2012 2:39 pm

vargaso wrote:The thing is, Burning Man tickets are about the most expensive tickets for any event on the planet.

I'm gonna have some fun with this one...
Simon's real sig line?

Embrace the Sock

Winners never quilt, quilters never win...
User avatar
theCryptofishist
 
Posts: 37425
Joined: Mon Feb 23, 2004 10:28 am
Location: In Exile
Burning Since: 2017

Re: STEP has launched... but server crashed immediately

Postby lemur » Fri Mar 02, 2012 2:39 pm

im guessing all of the people who suggest they know how to make it better have already filled out a volunteer questionnaire... because they really want to help make it better!

http://www2.burningman.com/people/ volunteer questionnaire

and! they probably have also volunteered their assistance with the 'tech team'


http://www.burningman.com/participate/techweb.html

and they might have already emailed techie-volunteers (at) burningman (dot) com and told them all about their expertise in the field and offered themselves up to make this process better..

volunteer!
Don't link to anything here!
User avatar
lemur
 
Posts: 3599
Joined: Fri Feb 13, 2009 3:07 pm
Location: Madagascar
Camp Name: Plug N Play Camp

Re: STEP has launched... but server crashed immediately

Postby theCryptofishist » Fri Mar 02, 2012 2:40 pm

remi wrote:
lemur wrote:banks and healthcare providers also have billion $ budget.


BM makes a few bucks too.

Billions to spend on IT compared to an event that has ~20 million for everything. Meaning that the IT budgets we're talking about are 500 times as large--at a minimum.
Simon's real sig line?

Embrace the Sock

Winners never quilt, quilters never win...
User avatar
theCryptofishist
 
Posts: 37425
Joined: Mon Feb 23, 2004 10:28 am
Location: In Exile
Burning Since: 2017

Re: STEP has launched... but server crashed immediately

Postby drutter » Fri Mar 02, 2012 2:41 pm

vargaso wrote:
lemur wrote:this year is a learning experience for everyone and I don't fault them one bit. I'm hoping things improve next year.

That's a lovely sentiment, but once you've been through 3 or 4 years, it becomes harder and harder to say genuinely. "It was a learning experience, next time will be better!" works well for some things, like riding a bike through a sand-trap on the playa. Not as well for things like parachuting onto the Esplanade. Some of us are getting tired of re-using that expression when it comes to the ticket process.
Heavy on preparations, light on expectations.
User avatar
drutter
 
Posts: 198
Joined: Mon Sep 29, 2008 6:19 am
Location: Vancouver, BC, Canada

Re: STEP has launched... but server crashed immediately

Postby vargaso » Fri Mar 02, 2012 2:42 pm

lemur wrote:im guessing all of the people who suggest they know how to make it better have already filled out a volunteer questionnaire... because they really want to help make it better!

http://www2.burningman.com/people/ volunteer questionnaire

and! they probably have also volunteered their assistance with the 'tech team'


http://www.burningman.com/participate/techweb.html

and they might have already emailed techie-volunteers (at) burningman (dot) com and told them all about their expertise in the field and offered themselves up to make this process better..

volunteer!


I volunteered for the Web Team in 2009, I sat in on a couple meetings via phone, but at that point, tickets had never sold out and most of the talk was around migrating burningman.com to a new content management system. I am absolutely volunteering for next year.

Edited to add:
Not that I think I'm a genius and/or that the current team isn't up to the job or that I can magically fix things. The major thing I could offer is some outside perspective on similar use cases and acting as an information architect, if they don't already have one. Someone like RevDusty, who sounds like a heavy duty software/server architect, would be great too.
Last edited by vargaso on Fri Mar 02, 2012 2:48 pm, edited 1 time in total.
User avatar
vargaso
 
Posts: 453
Joined: Mon Sep 17, 2007 11:26 pm
Location: Lincoln, CA
Burning Since: 2007
Camp Name: Suburbanoya

PreviousNext

Return to 2012 Tickets Discussion

Who is online

Users browsing this forum: No registered users and 1 guest