Web server demand is a solved problem

Want to talk about tickets? You've come to the right place

Web server demand is a solved problem

Postby Zakolantern » Mon Nov 14, 2011 5:47 pm

Either I'm missing a very fundamental concept, or in the last few years, services have grown mature to completely solve the issue of huge spikes in website traffic / server load. Led by Amazon.com's EC2 ( http://aws.amazon.com/ec2/ ) , but including tons of other offerings, from companies like GoGrid, Rackspace, Joyent, etc, it's now very easy to rent very temporary, very inexpensive, dynamic server instances in what's become known as Infrastructure-as-a-Service (IaaS).

I see no major cost or other practical reason why Burning Man should have its servers go down even if it experiences 10,000x its normal website traffic when it releases tickets.

Do others agree with this, or see a problem with what I'm proposing?

Note that this is NOT meant to be a discussion on the lottery system; how to assign and distribute tickets doesn't have a "right" and "wrong" answer the same way this (technical) problem seems to. (Personally, I'm in favor of non- or limited-transferable tickets). Instead, use this thread to discuss if a hit on server load crashing the website is a reasonable excuse / problem to have in 2012. Thanks!

Note: additional reading, subscription required, follows; if someone actually working for the Burning Man Organization would like a PDF copy of these reports, please message me directly; I work for Forrester Research, who published these. http://www.forrester.com/rb/Research/ma ... /58925/t/2 http://www.forrester.com/rb/Research/ma ... /60978/t/2
Zakolantern
 
Posts: 16
Joined: Thu Sep 22, 2011 10:43 pm

Re: Web server demand is a solved problem

Postby trilobyte » Mon Nov 14, 2011 6:00 pm

You're missing a couple fundamental things.

1) The BMOrg doesn't handle the ticketing for the event.

2) Nobody asked you. :mrgreen:

I'm kidding, though seriously nobody was asking about how to increase capacity. Stay tuned for the FAQ regarding the changes to the ticket policy, it's coming soon. If you've got technical advice or suggestions for InTicketing (assuming that's who will handle ticketing this year), you may want to reach out to them via their site.
User avatar
trilobyte
Site Admin
 
Posts: 10693
Joined: Tue Mar 02, 2004 11:54 pm
Location: San Francisco
Burning Since: 2004
Camp Name: Eridu Society

Re: Web server demand is a solved problem

Postby Rice » Mon Nov 14, 2011 6:04 pm

trilobyte wrote:You're missing a couple fundamental things.

1) The BMOrg doesn't handle the ticketing for the event.

2) Nobody asked you. :mrgreen:

I'm kidding, though seriously nobody was asking about how to increase capacity. Stay tuned for the FAQ regarding the changes to the ticket policy, it's coming soon. If you've got technical advice or suggestions for InTicketing (assuming that's who will handle ticketing this year), you may want to reach out to them via their site.


Hehe!! Just went to the InTicketing web site (http://www.inticketing.com/) , their home page took over a minute to load.. So, yah, maybe they need to do something to improve their services. There are many technical solutions which would solve the bandwidth/capacity issues.

Here is hoping the ticket lottery thingy has some sort of sane plan ;)

Rice
Love Rice

Roach: "I feel like in this day and age, every girl should know how to build a flamethrower."
User avatar
Rice
 
Posts: 1643
Joined: Fri Oct 20, 2006 8:33 pm
Location: Great White North, Saskatchewan, Canada
Burning Since: 2007

Re: Web server demand is a solved problem

Postby gibson_ » Mon Nov 14, 2011 6:30 pm

You're making some major assumptions about how the ticketing system was designed.

Yeah, EC2 lets you spin up lots of instances really quickly, and this is good...
...if you designed your database system to accommodate this model.
User avatar
gibson_
 
Posts: 107
Joined: Mon Jun 27, 2011 3:44 pm
Location: Phoenix

Re: Web server demand is a solved problem

Postby bradtem » Mon Nov 14, 2011 6:35 pm

Actually, I think you can do it even without having designed well in advance. Sure, if all the ticket servers are not talking to one another (much) people can do things like buy more tickets than allowed per person. But you can roll that back after the fact, before you mail the tickets. The only thing the servers have to share is to know when a quota is reached, and the tier is sold out. That's not too hard to do. EC2 small instances are 9 cents/hour. You could literally have 10,000 instances at the peak hour for $900.
See giant panoramas of BRC: http://www.templetons.com/brad/burn
User avatar
bradtem
 
Posts: 440
Joined: Fri Sep 05, 2003 12:27 pm
Location: Silicon Valley
Burning Since: 1998

Re: Web server demand is a solved problem

Postby gibson_ » Mon Nov 14, 2011 6:55 pm

Bradtem, not trying to step on toes, but are you a developer?

Unfortunately, I thing it's a bit more complex than that. Sure you could allocate 10,000 EC2 instances each with 5 tickets on it.

Okay, but then how do you route traffic to these? AWS has a load balancer, but what happens with 50% of the servers have 0 tickets available, and 50% have 2 available? Howabout when 1% has 1 ticket available and the rest have 0?

You need to build a mechanism for the servers to communicate to the load balancer how many tickets they have. This isn't difficult, but it certainly isn't trivial, and I'm *heavily* over simplifying how this process would need to work.

This is also assuming that you employ *no* ticket limits of any kind. Assuming 10,000 micro instances at 5 tickets each, what's to stop me from writing a bot to buy 1 ticket from each node? I can see the jobs coming to elance, guru, etc. now...
User avatar
gibson_
 
Posts: 107
Joined: Mon Jun 27, 2011 3:44 pm
Location: Phoenix

Re: Web server demand is a solved problem

Postby bradtem » Mon Nov 14, 2011 7:16 pm

Yes, I am a developer. And as I indicated, you can't do it entirely standalone, you would not allocate 3 tickets per server, you would need to be broadcasting counts of sold tickets to each server so they know when to stop. It doesn't have to be exact -- it's not the end of the world if you sell 5,012 tickets instead of 5,000 in your tier.

And I would not really do 10,000 instances, I just used that as an example of how cheap this has gotten. There are lots of load balancers out there, and standard ones are fine, though it might be a good place to put the ticket counter. I did not say it was trivial but it doesn't require heavy modification of older systems. You need to throw in a message passing system so that one central counter (or ideally more than one) gets simple messages about each pending ticket request (ie. hold 2 tickets) and the resolution (cancel or sold)


But all of this isn't how I would actually build this app. Instead I would do something much simpler. I would make a simple app that doesn't sell tickets, but instead hands out places in line. You go in, and fill out a basic web form with number of tickets and card number, perhaps email too. It hashes those together with a secret and your place in line, and gives it back to you. Save that file of course, and then you can buy your ticket at leisure on ordinary slow ticket servers, which accept the tokens at some rate. For example, the first hour they take tokens 1-100, next hour 1-200, 1-300, 100-400 and so on. As the number increases people who didn't use their token lose out and people get updated that while they got place 2001 out of 2000, now they are in.

Now a server to take a web form and hand out tokens is quite simple, and a high-end one can easily do a very high rate, and a small cluster of them with some message passing for the numbers could handle a thousand requests a second or more. Again, you don't need super fast sync as the sizes of the ticket pools need not be super-precise.

Of course there are people who have already built robust ticketing systems, but if I had to reinvent that wheel, this is the simplest way I would do it.
See giant panoramas of BRC: http://www.templetons.com/brad/burn
User avatar
bradtem
 
Posts: 440
Joined: Fri Sep 05, 2003 12:27 pm
Location: Silicon Valley
Burning Since: 1998

Re: Web server demand is a solved problem

Postby BBadger » Mon Nov 14, 2011 7:48 pm

Zakolantern wrote:Either I'm missing a very fundamental concept, or in the last few years, services have grown mature to completely solve the issue of huge spikes in website traffic / server load. Led by Amazon.com's EC2 ( http://aws.amazon.com/ec2/ ) , but including tons of other offerings, from companies like GoGrid, Rackspace, Joyent, etc, it's now very easy to rent very temporary, very inexpensive, dynamic server instances in what's become known as Infrastructure-as-a-Service (IaaS).

I see no major cost or other practical reason why Burning Man should have its servers go down even if it experiences 10,000x its normal website traffic when it releases tickets.

Do others agree with this, or see a problem with what I'm proposing?


It's not about server capability, but whether your software and database design is able to handle large loads. Even Microsoft's servers were brought to their knees because of SQL fragmentation. I believe the site was already on Amazon's EC2 server last year and we all remember what happened. I'm guessing that a high load, coupled with auto-refreshing and some poor-performing code were to blame and no amount of cloud computing could help that.

ticket_problems.png
You do not have the required permissions to view the files attached to this post.
"The essence of tyranny is not iron law. It is capricious law." -- Christopher Hitchens

Hate reading my replies? Click here to add me to your plonk (foe) list.
User avatar
BBadger
 
Posts: 3973
Joined: Wed Jan 19, 2011 11:37 am
Location: (near) Portland, OR, USA
Burning Since: I'm not sure

Re: Web server demand is a solved problem

Postby igor47 » Wed Nov 16, 2011 3:03 am

There are lots of solutions to horizontally scaling a ticket-selling app, and most of them are pretty easy. InTicket is just kinda bad at technology, which is consistent with BMOrg being generally really bad at technology (exhibit 1: BM rideshare; exhibit 2: the forms site for theme camps/MVs; exhibit 3: the playa registration system; etc....).

I just checked inticketing's website. They want help with their site so much that their 'jobs' link is a redirect loop which just ends up giving you a browser error. I sent them a message anyway, but I'm not holding my breath.
igor47
 
Posts: 45
Joined: Sun Aug 09, 2009 3:38 pm
Location: SF

Re: Web server demand is a solved problem

Postby jkisha » Wed Nov 16, 2011 8:56 am

trilobyte wrote: If you've got technical advice or suggestions for InTicketing (assuming that's who will handle ticketing this year), you may want to reach out to them via their site.

The only advice I have for anybody about InTicketing is that BMORG should have dumped them two years ago.
JK
Image
http://www.mudskippercafe.com
When I was a kid I used to pray every night for a new bicycle.
Then I realised that the Lord doesn't work that way so I stole one and asked Him to forgive me.
User avatar
jkisha
 
Posts: 11403
Joined: Tue Jul 17, 2007 10:00 pm
Location: Los Angeles
Burning Since: 2007
Camp Name: Mudskipper Cafe

Re: Web server demand is a solved problem

Postby MrMullen » Wed Nov 16, 2011 10:47 am

jkisha wrote:
trilobyte wrote: If you've got technical advice or suggestions for InTicketing (assuming that's who will handle ticketing this year), you may want to reach out to them via their site.

The only advice I have for anybody about InTicketing is that BMORG should have dumped them two years ago.


Wrong, they should have dumped them 6 years ago. This will be my 7th Burn out of 8 Years, and the ticket process has been fucked up for 7 of the last 8 years*.


* I do recall the ticket purchasing for 2010 being smooth, although you had to wait in a queue for a hour or so.
--
Mr Mullen
MrMullen
 
Posts: 314
Joined: Wed Jan 28, 2004 7:39 pm
Location: San Diego

Re: Web server demand is a solved problem

Postby illy dilly » Wed Nov 16, 2011 12:56 pm

igor47 wrote:There are lots of solutions to horizontally scaling a ticket-selling app, and most of them are pretty easy. InTicket is just kinda bad at technology, which is consistent with BMOrg being generally really bad at technology (exhibit 1: BM rideshare; exhibit 2: the forms site for theme camps/MVs; exhibit 3: the playa registration system; etc....).

I just checked inticketing's website. They want help with their site so much that their 'jobs' link is a redirect loop which just ends up giving you a browser error. I sent them a message anyway, but I'm not holding my breath.

Hot diggidy damn!
Someone being proactive!

I wish I had any sort of skill that could help with the server issues, but unless you want blue print of it or an estimate on how much to build the building around it, I'm not much help.
Why don't ya stick your head in that hole and find out? ~piehole
Plan for the worst, expect the best. Make the most out of it under any conditions. If you cannot do that you will never enjoy yourself. ~CrispyDave
User avatar
illy dilly
 
Posts: 4798
Joined: Wed Feb 17, 2010 12:02 pm
Location: Denver, CO
Burning Since: 2009
Camp Name: Gnome Dome

Re: Web server demand is a solved problem

Postby portaplaya » Wed Nov 16, 2011 4:42 pm

I dont' think that Burning Man is saying that web server demand isn't solved.

I think they are saying that last year's first day of ticket sales indicates that all of the tickets will sell out in day-one. And without some system to fairly distribute the buying opportunities within that one day, the perception of hideous unfairness will be a dark taint.

A lottery allows them to do that more fairly, and also functions as a workaround from any mishaps that web technology might run into.

Burning Man said they were creating a new process because: "2011 provided two compelling reasons to change how tickets are sold: a challenging ticket launch day, and then, our first-ever sold-out event." I feel that this is different than saying "we can't handle web server demand."
User avatar
portaplaya
 
Posts: 363
Joined: Sat Sep 10, 2005 6:49 pm
Location: Seattle area

Re: Web server demand is a solved problem

Postby Rice » Wed Nov 16, 2011 5:11 pm

portaplaya wrote:I dont' think that Burning Man is saying that web server demand isn't solved.

I think they are saying that last year's first day of ticket sales indicates that all of the tickets will sell out in day-one. And without some system to fairly distribute the buying opportunities within that one day, the perception of hideous unfairness will be a dark taint.

A lottery allows them to do that more fairly, and also functions as a workaround from any mishaps that web technology might run into.

Burning Man said they were creating a new process because: "2011 provided two compelling reasons to change how tickets are sold: a challenging ticket launch day, and then, our first-ever sold-out event." I feel that this is different than saying "we can't handle web server demand."


Only 27,000ish tickets of the 50,000 sold in the first day and then it took over 5 months to sell the rest. I doubt they expect to have 50000+ people try and get tickets on the first day. What is more fair than standing in line, waiting your turn, and then purchasing your allotment of tickets?? There has to be something else going on. (Conspiracy theorists - warm up your keyboards)

The company that sold the tickets for the Burning Man Organization (http://www.inticketing.com/) had many technical challenges during the opening day. {fact} Politely, their servers got overloaded and stressed a few burners out. - That is certainly a problem that needs addressing. Historically, the BM Org has had a 5 year permit with the BLM. That permit allowed for some population growth over the same time period. I speculate that the new permit will also allow for a similar population growth. If this occurs there should not be any reason to sell out (since the probable growth should be less than the permit maximum).

Once we have been given the ticket Lottery details, we should be able to deduce what the plan is.

There is a problem that I do not see getting any better. Entry into BRC and Exodus from BRC. If large groups of people try to enter and/or leave the city at the same time, it is still going to take a long time. It is not to hard to figure out why it takes longer for 30K people to enter than it did for 20K. (I compare it to having a garden hose hooked up to a 20 gallon tank and it takes 60 minutes to empty, then hooking the same hose up to a 30 gallon tank and expecting it to take just as long to empty. ) There needs to be a system for reducing the massive influx/outflow of vehicles, something that balances things. Without that, ticket worries will be low on the list. Imagine taking 14-16 hours (or longer) to get into BRC from Gerlach?? {shudder} - Not something I would enjoy!!!!

Anyhow, not panicking. Starting to find other people's panic amusing though.

Love Rice
Love Rice

Roach: "I feel like in this day and age, every girl should know how to build a flamethrower."
User avatar
Rice
 
Posts: 1643
Joined: Fri Oct 20, 2006 8:33 pm
Location: Great White North, Saskatchewan, Canada
Burning Since: 2007

Re: Web server demand is a solved problem

Postby portaplaya » Wed Nov 16, 2011 5:45 pm

stretch80 wrote:Only 27,000ish tickets of the 50,000 sold in the first day and then it took over 5 months to sell the rest. I doubt they expect to have 50000+ people try and get tickets on the first day.


Actually, it only takes 12,500 people ordering four tickets each to sell out in the first day.

If 7,000 people ordered four tickets each last year (for ~27,000) and 3,000 people did not get tickets at all last year and jump on the first day this year, we only need 2,500 people being prudent (or scalpers) and the event is sold out.
User avatar
portaplaya
 
Posts: 363
Joined: Sat Sep 10, 2005 6:49 pm
Location: Seattle area


Return to 2012 Tickets Discussion

Who is online

Users browsing this forum: No registered users and 1 guest