Web server demand is a solved problem

[Zakolantern]

Either I'm missing a very fundamental concept, or in the last few years, services have grown mature to completely solve the issue of huge spikes in website traffic / server load. Led by Amazon.com's EC2 ( http://aws.amazon.com/ec2/ ) , but including tons of other offerings, from companies like GoGrid, Rackspace, Joyent, etc, it's now very easy to rent very temporary, very inexpensive, dynamic server instances in what's become known as Infrastructure-as-a-Service (IaaS).

I see no major cost or other practical reason why Burning Man should have its servers go down even if it experiences 10,000x its normal website traffic when it releases tickets.

Do others agree with this, or see a problem with what I'm proposing?

Note that this is NOT meant to be a discussion on the lottery system; how to assign and distribute tickets doesn't have a "right" and "wrong" answer the same way this (technical) problem seems to. (Personally, I'm in favor of non- or limited-transferable tickets). Instead, use this thread to discuss if a hit on server load crashing the website is a reasonable excuse / problem to have in 2012. Thanks!

Note: additional reading, subscription required, follows; if someone actually working for the Burning Man Organization would like a PDF copy of these reports, please message me directly; I work for Forrester Research, who published these. http://www.forrester.com/rb/Research/ma ... /58925/t/2 http://www.forrester.com/rb/Research/ma ... /60978/t/2

[trilobyte]

You're missing a couple fundamental things.

1) The BMOrg doesn't handle the ticketing for the event.

2) Nobody asked you.

I'm kidding, though seriously nobody was asking about how to increase capacity. Stay tuned for the FAQ regarding the changes to the ticket policy, it's coming soon. If you've got technical advice or suggestions for InTicketing (assuming that's who will handle ticketing this year), you may want to reach out to them via their site.

[Rice]

You're missing a couple fundamental things.

1) The BMOrg doesn't handle the ticketing for the event.

2) Nobody asked you.

I'm kidding, though seriously nobody was asking about how to increase capacity. Stay tuned for the FAQ regarding the changes to the ticket policy, it's coming soon. If you've got technical advice or suggestions for InTicketing (assuming that's who will handle ticketing this year), you may want to reach out to them via their site.

Hehe!! Just went to the InTicketing web site (http://www.inticketing.com/) , their home page took over a minute to load.. So, yah, maybe they need to do something to improve their services. There are many technical solutions which would solve the bandwidth/capacity issues.

Here is hoping the ticket lottery thingy has some sort of sane plan

Rice

[gibson_]

You're making some major assumptions about how the ticketing system was designed.

Yeah, EC2 lets you spin up lots of instances really quickly, and this is good...
...if you designed your database system to accommodate this model.

[bradtem]

Actually, I think you can do it even without having designed well in advance. Sure, if all the ticket servers are not talking to one another (much) people can do things like buy more tickets than allowed per person. But you can roll that back after the fact, before you mail the tickets. The only thing the servers have to share is to know when a quota is reached, and the tier is sold out. That's not too hard to do. EC2 small instances are 9 cents/hour. You could literally have 10,000 instances at the peak hour for $900.

[gibson_]

Bradtem, not trying to step on toes, but are you a developer?

Unfortunately, I thing it's a bit more complex than that. Sure you could allocate 10,000 EC2 instances each with 5 tickets on it.

Okay, but then how do you route traffic to these? AWS has a load balancer, but what happens with 50% of the servers have 0 tickets available, and 50% have 2 available? Howabout when 1% has 1 ticket available and the rest have 0?

You need to build a mechanism for the servers to communicate to the load balancer how many tickets they have. This isn't difficult, but it certainly isn't trivial, and I'm *heavily* over simplifying how this process would need to work.

This is also assuming that you employ *no* ticket limits of any kind. Assuming 10,000 micro instances at 5 tickets each, what's to stop me from writing a bot to buy 1 ticket from each node? I can see the jobs coming to elance, guru, etc. now...

[bradtem]

Yes, I am a developer. And as I indicated, you can't do it entirely standalone, you would not allocate 3 tickets per server, you would need to be broadcasting counts of sold tickets to each server so they know when to stop. It doesn't have to be exact -- it's not the end of the world if you sell 5,012 tickets instead of 5,000 in your tier.

And I would not really do 10,000 instances, I just used that as an example of how cheap this has gotten. There are lots of load balancers out there, and standard ones are fine, though it might be a good place to put the ticket counter. I did not say it was trivial but it doesn't require heavy modification of older systems. You need to throw in a message passing system so that one central counter (or ideally more than one) gets simple messages about each pending ticket request (ie. hold 2 tickets) and the resolution (cancel or sold)


But all of this isn't how I would actually build this app. Instead I would do something much simpler. I would make a simple app that doesn't sell tickets, but instead hands out places in line. You go in, and fill out a basic web form with number of tickets and card number, perhaps email too. It hashes those together with a secret and your place in line, and gives it back to you. Save that file of course, and then you can buy your ticket at leisure on ordinary slow ticket servers, which accept the tokens at some rate. For example, the first hour they take tokens 1-100, next hour 1-200, 1-300, 100-400 and so on. As the number increases people who didn't use their token lose out and people get updated that while they got place 2001 out of 2000, now they are in.

Now a server to take a web form and hand out tokens is quite simple, and a high-end one can easily do a very high rate, and a small cluster of them with some message passing for the numbers could handle a thousand requests a second or more. Again, you don't need super fast sync as the sizes of the ticket pools need not be super-precise.

Of course there are people who have already built robust ticketing systems, but if I had to reinvent that wheel, this is the simplest way I would do it.

[BBadger]

Either I'm missing a very fundamental concept, or in the last few years, services have grown mature to completely solve the issue of huge spikes in website traffic / server load. Led by Amazon.com's EC2 ( http://aws.amazon.com/ec2/ ) , but including tons of other offerings, from companies like GoGrid, Rackspace, Joyent, etc, it's now very easy to rent very temporary, very inexpensive, dynamic server instances in what's become known as Infrastructure-as-a-Service (IaaS).

I see no major cost or other practical reason why Burning Man should have its servers go down even if it experiences 10,000x its normal website traffic when it releases tickets.

Do others agree with this, or see a problem with what I'm proposing?

It's not about server capability, but whether your software and database design is able to handle large loads. Even Microsoft's servers were brought to their knees because of SQL fragmentation. I believe the site was already on Amazon's EC2 server last year and we all remember what happened. I'm guessing that a high load, coupled with auto-refreshing and some poor-performing code were to blame and no amount of cloud computing could help that.

ticket_problems.png

[igor47]

There are lots of solutions to horizontally scaling a ticket-selling app, and most of them are pretty easy. InTicket is just kinda bad at technology, which is consistent with BMOrg being generally really bad at technology (exhibit 1: BM rideshare; exhibit 2: the forms site for theme camps/MVs; exhibit 3: the playa registration system; etc....).

I just checked inticketing's website. They want help with their site so much that their 'jobs' link is a redirect loop which just ends up giving you a browser error. I sent them a message anyway, but I'm not holding my breath.

[jkisha]

If you've got technical advice or suggestions for InTicketing (assuming that's who will handle ticketing this year), you may want to reach out to them via their site.

The only advice I have for anybody about InTicketing is that BMORG should have dumped them two years ago.

[MrMullen]

If you've got technical advice or suggestions for InTicketing (assuming that's who will handle ticketing this year), you may want to reach out to them via their site.

The only advice I have for anybody about InTicketing is that BMORG should have dumped them two years ago.

Wrong, they should have dumped them 6 years ago. This will be my 7th Burn out of 8 Years, and the ticket process has been fucked up for 7 of the last 8 years*.


* I do recall the ticket purchasing for 2010 being smooth, although you had to wait in a queue for a hour or so.

[illy dilly]

There are lots of solutions to horizontally scaling a ticket-selling app, and most of them are pretty easy. InTicket is just kinda bad at technology, which is consistent with BMOrg being generally really bad at technology (exhibit 1: BM rideshare; exhibit 2: the forms site for theme camps/MVs; exhibit 3: the playa registration system; etc....).

I just checked inticketing's website. They want help with their site so much that their 'jobs' link is a redirect loop which just ends up giving you a browser error. I sent them a message anyway, but I'm not holding my breath.

Hot diggidy damn!
Someone being proactive!

I wish I had any sort of skill that could help with the server issues, but unless you want blue print of it or an estimate on how much to build the building around it, I'm not much help.

[portaplaya]

I dont' think that Burning Man is saying that web server demand isn't solved.

I think they are saying that last year's first day of ticket sales indicates that all of the tickets will sell out in day-one. And without some system to fairly distribute the buying opportunities within that one day, the perception of hideous unfairness will be a dark taint.

A lottery allows them to do that more fairly, and also functions as a workaround from any mishaps that web technology might run into.

Burning Man said they were creating a new process because: "2011 provided two compelling reasons to change how tickets are sold: a challenging ticket launch day, and then, our first-ever sold-out event." I feel that this is different than saying "we can't handle web server demand."

[Rice]

I dont' think that Burning Man is saying that web server demand isn't solved.

I think they are saying that last year's first day of ticket sales indicates that all of the tickets will sell out in day-one. And without some system to fairly distribute the buying opportunities within that one day, the perception of hideous unfairness will be a dark taint.

A lottery allows them to do that more fairly, and also functions as a workaround from any mishaps that web technology might run into.

Burning Man said they were creating a new process because: "2011 provided two compelling reasons to change how tickets are sold: a challenging ticket launch day, and then, our first-ever sold-out event." I feel that this is different than saying "we can't handle web server demand."

Only 27,000ish tickets of the 50,000 sold in the first day and then it took over 5 months to sell the rest. I doubt they expect to have 50000+ people try and get tickets on the first day. What is more fair than standing in line, waiting your turn, and then purchasing your allotment of tickets?? There has to be something else going on. (Conspiracy theorists - warm up your keyboards)

The company that sold the tickets for the Burning Man Organization (http://www.inticketing.com/) had many technical challenges during the opening day. {fact} Politely, their servers got overloaded and stressed a few burners out. - That is certainly a problem that needs addressing. Historically, the BM Org has had a 5 year permit with the BLM. That permit allowed for some population growth over the same time period. I speculate that the new permit will also allow for a similar population growth. If this occurs there should not be any reason to sell out (since the probable growth should be less than the permit maximum).

Once we have been given the ticket Lottery details, we should be able to deduce what the plan is.

There is a problem that I do not see getting any better. Entry into BRC and Exodus from BRC. If large groups of people try to enter and/or leave the city at the same time, it is still going to take a long time. It is not to hard to figure out why it takes longer for 30K people to enter than it did for 20K. (I compare it to having a garden hose hooked up to a 20 gallon tank and it takes 60 minutes to empty, then hooking the same hose up to a 30 gallon tank and expecting it to take just as long to empty. ) There needs to be a system for reducing the massive influx/outflow of vehicles, something that balances things. Without that, ticket worries will be low on the list. Imagine taking 14-16 hours (or longer) to get into BRC from Gerlach?? {shudder} - Not something I would enjoy!!!!

Anyhow, not panicking. Starting to find other people's panic amusing though.

Love Rice

[portaplaya]

Only 27,000ish tickets of the 50,000 sold in the first day and then it took over 5 months to sell the rest. I doubt they expect to have 50000+ people try and get tickets on the first day.

Actually, it only takes 12,500 people ordering four tickets each to sell out in the first day.

If 7,000 people ordered four tickets each last year (for ~27,000) and 3,000 people did not get tickets at all last year and jump on the first day this year, we only need 2,500 people being prudent (or scalpers) and the event is sold out.