[ADMIN] So what have ILX been up to? And what next?

So: sorry baout the downtime this week.

What happened:
Basically a rare kind of server crash (caused by a bug in the operating system that only showed up after years) wrote a load of junk to key parts of the hard drives. That first made it look to the hosts like a hardware failure, then made us suspect there'd been an attack, and then finally meant some really low-level bits of the operating system had to be rebuilt before we could get back in. Our backups worked, so I restored from them, and here we are.

What now:
Everything should be back up and running. Search will take a few days to get back up to speed, and other indexes and caches will take a while to be normal. You might see slowness, and unread markers will be wrong etc. There's also probably going to be some more short bits of downtime, probably at the weekend, as I'm going to do some stuff to try and make the backups more reliable.

What next?
We (and I) got a bit lucky there, as the off-site backups were a good few weeks out of date. Part of the problem here is that the database is now getting really big -- more than 3gb in size, so backups take a long time to run, and nearly as long to move about. The correct thing to do in this situation would be to backup live to another server, which we don't have.

What's more, the server we do have is getting on -- it's 5 years old -- and if it fails we have nothing else to replace it with. The support contract has now expired, and Oracle (who bought Sun) want ludicrous money to renew it.

Meantime, I picked up a backup server for cheap some months back, but haven't got it set up yet. If we'd had to move to that, it would have taken a goodish number of weeks to get going, especially if I had to ship it to Canada. Putting it in a datacentre ahead of time would also be expensive -- would basically double our hosting bill. As an alternative, I looked at cloud-based server options (like Amazon's EC2, or VPS hosting) but for the amount of memory and database space we need, it's even more expensive to do.

Happily, the hosts think they can sort us out with a virtual server in their datacentre. This would in theory let us move ILX to a more modern setup, and potentially keep the existing server as a live backup.

So, that's my plan just now; I'm waiting on them getting back to me with details. If that doesn't work, I'll see if there are other options, potentially like reserving an Amazon server that only spins up once a day, downloads changes from the database, then goes back to sleep.

I guess one of the questions is how long a period of downtime are we prepared to take if this server goes down, and how many day's posts are we prepared to lose? I'd like that to be "hardly any" and "none at all", but that might cost us an ongoing chunk of change. What we do depends on exactly how much, I guess.

(Not to derail right away, but I've been wondering about how much we'd make putting ads on ilxor.com for non-registered users, hiding them from registereds; there is a lot of hassle connected with that, but if it gave us the money to run things properly I think it'd be worth considering.)

I recommend an ilx fund drive while you work out the details on spamming the unregistered

1) Er, I'm sure most of us could live without a couple of weeks on ILX (probably would help us somewhat in our normal lives). More than that? I guess I could get through that.

2) I have no issue with ads for non-registered users; there's potential for a goldmine of jokes out of it.

I've always thought ads for unregistered eyes was a smart way to go.

I recommend an ilx fund drive while you work out the details on spamming the unregistered

― the boy with the gorn at his side (Edward III), Wednesday, October 19, 2011 4:01 PM (2 minutes ago) Bookmark

what did unregistered ever do to you

he's a liar for one thing

well done Stet for sorting out these intractable-sounding issues.

Thanks stet!

Go for the fundraiser. I missed out on the last funding rush, and iirc quite a few others did too.

Still think we could make a go of ILX merch.

How many hits do we get from unregistereds?

Do we have the ability to do incremental backups rather than backing up the entire database?

we could just sign ILX up for iCloud

Restoring using incremental backups or transaction logs is a lot if work, iirc xpost

I'm all in favor of ads for unregistereds.

In the meantime, it seems like ILX Fund Raising always finishes before a lot of folks who want to contribute can - maybe run one for a set amount of time and just see how much gets donated rather than capping it at a hard number?
Could even do it through Kickstarter so that if it didn't reach an arbitrary goal (however much it takes to buy a new pimped-out server? I have no clue how much that is.) folks get their $$$ back.

but I'd be in favor of increased costs for more modern servers/backups. we max out the funding drive within hours each year!

Milo: if you collect too much you may be legally obliged to register as an organisation domewhere (where?) etc

domewhere? I can't even type anymore, so emotional

thunderdomewhere

I'm planning on winning the lottery and donating a server and hosting and liberate stet from his day job etc. - wish me luck!

We do, but on mysql they're taken from binary log files, which means you need to do pretty regular full backups anyway (and also restart the logging) or else you're just building on a giant binary of hope. xxp

On the funding thing: it's true that we raise what we need pretty quickly; this is also partly about thinking what the best thing to buy/rent is, so then we can work out how much it'll cost!

Good work getting things back in business, Stet.

And yeah, happy to throw into another funding drive, buy an ILX sweatshirt, support ads for nonreg's etc. Whatever it takes to keep this noble ship afloat.

thx stet; I'm not a sysadmin so I only know about this stuff in theory

i am also 100% in favor of ads for unregs

re: the fundraising etc yeah we would need to look into that, not that the site is prob going to be rolling in money, but i imagine there would need to be some sort of non-profit org set up or something to handle how the account would work. where my lawyer friends at?

That's an issue with ads, too, I guess.
ILX gets a ridiculous amount of traffic beyond just us registered folk, IIRC.

oh yeah i was talking about fundraising and ads if that wasnt clear - some entity has to hold the funds/document the income etc in order to keep anyone from being personally responsible/taxable/whatevs

my I suggest... Loutallica

I find the occasional disastrous downtime therapeutic tbh, as long as it isn't too disastrous.

would support ads for unregistered if all current mods resign

OCCUPY ILX

You might see slowness, and unread markers will be wrong etc

I lol'd

well there goes my productivity

Isn't there some way to set up links to sites like Amazon, so that whenever one of us wants to purchase something there, we could click through the link and ILX would automatically get a referral credit? I've heard of blogs raising money that way. Not sure if it would be much, but it seems like a fairly painless way to help out.

could we autoreplace chaki etc with ads

Let people who are banned buy their way back in?

Only ILX gold accounts are allowed to vote on this poll, sorry.

Let people who are banned buy their way back in?

OMG YES

that way lies metafilter

You might see slowness, and unread markers

http://i.imgur.com/OEwRG.png

How does SA do what they do? They charge $10 or something to be able to post, right? Does someone run it as a business? I mean I know theyre a shiiiiit ton bigger, but they seem to run on a model not too dissimilar to us in some ways.

SA charges $10 per post

i instinctively want to say hi to ppl whose posts i haven't read in nearly a week

hi ilx ppl

that's just awful

xp

hi nak

whiney :D

Salient points itt:

1. ilx needs money
2. unregistered is a liar
3. markers will be wrong

thx 4 new sn

Is a subscription model worth thinking about? Depending on implementation, the risk there is that you get tiers of users – the plebs who pay nothing and get kicked about, and the elites who do all the kicking and have more power than stet. Then again, you could always just have a tier called 'passionate' that doesn't net the user any advantage at all except a warm cosy feeling (3RRR does this.

)

we shd never ever do anything that makes us like SA or Metafilter imo

iirc fundraising drives have won out over ads in the past because they present less legal hassles? tbh personally i'd be against ads in any circumstances but i guess until stet's got a firm-ish estimate for how much cash we need this is a moot point

please god no tiers no subscriptions no inner/outer circles

How much would AWS really cost? We've never had a problem fundraising and it seems like it would be the most robust solution.

NO SUBSCRIPTION. That would be shit. I'd rather just go with more fundraisers than ads for unregistered users, but ads would be preferable to subscriptions.

I'm for fundraising - I think the last couple drives we had closed down before I even knew about them

please god no tiers no subscriptions no inner/outer circles

^^^also this

please god no tiers no subscriptions no inner/outer circles

Like 77 membership? (ok ignore the subscriptions bit)

yeah well I don't like the 77 board either

Well, I don't like the existence of 77, either. Would rather ilx be a level playing field.

xpost, ha

separate funding for 77, yo (since I wasn't allowed in)

think the exclusive club is people who refuse to read 77 tbh

if this turns into another 77 discussion i will lobby for ads for people that post in this thread fyi

Two key differences between ad hoc fundraising and subscriptions are (1) being a subscriber leads to a firmer sense of attachment to a community imo and (2) everyone who wants to contribute actually gets a chance. Having said that I'm easy either way, as long as stet gets money.

Even conservatively, it's around $140 a month, and that's just one instance. We gain in reliability etc, but it's still a bit all-eggs-one-basket. xxxp

staggering dangerously close to a "abolish SB" discussion in 5...4...3...

iirc ilx fundraising drives have been such a runaway success whenever implemented (at least in the recent past), that there should be no reason to suspect that simply having more of them, as needed, would not be enough to cover overhead.

I've always thought ads for unregistered eyes was a smart way to go.

I'm all in favor of ads for unregistereds.

i am also 100% in favor of ads for unregs

would support ads for unregistered

I hate you all.

(although I like the idea that I can starve ilx's main source of income simply by using Adblock)

charge to sb

i don't see why i shd pay for everything

what about a Poll tax??

Maybe we could have a swearbox? All those in favour say 'fuck yeah'.

SB tax

I would pay money to ban Geir

fundraise drives as and where neessary, poll the ten users with the lowest post/donation ratio, sb losers

enough to cover overhead.
there's overhead and then there's luxury. I am always impressed by how quickly ilxors stump up the cash for what's needed, but i don't want to spend 1p more of the hard-earned than necessary, and don't like idea of asking for stuff we don't absolutely *need* to keep the lights on. A big backup rig seems like a bit of a luxury, when we can get round it with, well, lots more manual labour.

If we have a free money spout from unregistered w/out affecting the rest of us otoh, we could hire a damn hell ass palace of a server set up and over-specify things a bit.

9 - 9 - 9 plan for ilx

ned gonna be paying for a lot

stet

Poll for 'bare minimum costs and stet slaving away' vs 'i trust stet to spend whatever i choose to donate towards running ilx'

ie don't be an ass

does advertising make ilx in some sense a "business"?

i don't think most people wd have qualms about chipping in for a luxury solution if it was clear what we were getting? can't imagine anybody (sane) thinking they were being ripped off.

don't even wanna think about what sort of ads will be appearing on iltmi

ned gonna be paying for a lot

Oh dear.

and the elites who do all the kicking and have more power than stet

How can anyone have more power than Man Who Make Boards Go?

Thanks for getting the boards back up, stet.

A big backup rig seems like a bit of a luxury, when we can get round it with, well, lots more manual labour.

Yes, but if that's lots more manual labour by one person who has a full-time job already, that's not really acceptable, is it?

we shd never ever do anything that makes us like SA or Metafilter imo

I just want to emphasize this important statement ^^^

Also, I would much rather donate more money (I've only donated once, having missed most of the fund drives) and thus create less work for stet. I mean, sheesh.

i will read and digest this thread more when i'm less drunk

(I've only donated once, having missed most of the fund drives)

u+k point, bears repeating every six posts

what, that the money got raised even tho lots of people didn't get a chance to throw in? nightmare!

stet

Poll for 'bare minimum costs and stet slaving away' vs 'i trust stet to spend whatever i choose to donate towards running ilx'

ie don't be an ass

― shite pele (darraghmac), Wednesday, October 19, 2011 5:39 PM (11 minutes ago)

What does this even mean?

That people probably wouldn't want stet to slave away and would instead be willing to donate money and trust stet to use it as necessary and that anyone who thinks otherwise is an ass, including stet.

sorry if that wasn't clear but jenny otm

hth

have the 1% pay for it

I don't see a problem with paying stet a stipend for all the work he does

I want to give someone money.

All my bookmarks are gone, btw. What are the odds they'll come back?

seriously though, just have another fundraiser, they go pretty fast. anything more than that is honestly too fucking complicated and more trouble than it's worth

^^^^^^^^^^^^^^

ya stet get to it ffs

ok, cool -- xps to Jenny and darragh

Je55e, maybe the loss of bookmarks has something to do with the fact that you somehow were able to have two different logins from the same email address? Just guessing at this point.

I'm like the a Jon Williams who only fucks up his own ILX and only unintentionally.

Thx guys this all makes sense. I'll get a shopping list together and see what we're talking about really.

Je55e: yr bookmarks shld be fine. Can you really sign in w/two accounts? How does that work?

I used to be Jesse and I was able to sign up as Jesse again under a different email address. It was weird. Now I'm Je55e.

stet, check MDF, there's a brief explanation

Oh, you think my bookmarks will come back someday? I hope so!

Je55e, maybe the loss of bookmarks has something to do with the fact that you somehow were able to have two different logins from the same email address? Just guessing at this point.

Boy did I misremember this!

I used to be Jesse and I was able to sign up as Jesse again under a different email address. It was weird. Now I'm Je55e.

― Je55e, Thursday, 20 October 2011 10:13 (7 minutes ago) Bookmark Suggest Ban Permalink

ilx needs a genius bar

curious as to why exactly people seem pretty hardcore opposed to ads for non-regs, esp if they could be accompanied by some text like "hey do u hate these ads? we do! reg and they go away. for free!"

"at the low low price of US$29.95"

(joek)

advertising is evil that's why

Who's hardcore opposed? It looked like everyone said it was a good idea, upthread.

like i enjoying boggling at how quickly the donation model seems to have worked but sustainable income seems to be easier to implement in the long run and a better idea.

esp if they could be accompanied by some text like "hey do u hate these ads? we do! reg and they go away. for free!"

we're looking to recruit then?

Who's hardcore opposed? It looked like everyone said it was a good idea, upthread.

― Occupy Kelly St (Trayce), Thursday, October 20, 2011 12:24 AM (6 minutes ago)

I'm opposed. Maybe not hardcore opposed, but opposed. But then I'm also opposed to the default ilx look not being old.css when I remember about that, so I guess I see unregistered/logged out ilx very very rarely. It's more just the principle of the thing. We should be a utopia.

We are, but the last 50 feet up the mountain is the act of registering.

WmC otm, if any unregistered user has the tenacity to work out what the hell ilx IS then registering is just the final stretch

idk i get that as a dude that fought tirelessly (and irritatingly!) against registration as a whole in the great reg battles of 2006 or whatever, but since people cant post unless they reg at this point, i dont see that throwing an ad up there as well is going to have any additional exclusionary vibe. xpost to e.mily

Surprised at the lack of this itt:

Advertising? In my vagina? It's ("more/less" poll result) likely than you think.

Did we lose anyone when we went reg or gtfo? I vaguely recall Anthony was so against it he said he'd disappear.

Never understood the problem, personally (with reging I mean not ads)

strong vote in favour of both ads for unregistered users AND having a 24-hour fundraiser that allows stet to spend whatever's leftover on DVDs, beer, or silk scarves.

we lost spice

~new zealand accent joke~

uss beed you for that

dead you rilly?

I used to be Jesse and I was able to sign up as Jesse again under a different email address. It was weird. Now I'm Je55e.

― Je55e, Thursday, 20 October 2011 10:13 (7 minutes ago) Bookmark Suggest Ban Permalink

ilx needs a genius bar

― Autumn Almanac (Schlafsack), Wednesday, October 19, 2011 6:21 PM (1 hour ago) Bookmark

Per my recent declaration of idgi pride, I must tell you: idgi.

I hope that in the event of a fund drive, there will be plenty of warning so that I don't miss it.

I'm also opposed to the default ilx look not being old.css

yessss

Actually haha yeah me too, whenever i'mnot logged in/somehow lose my prefs I'm always all "OMG WHAT HAPPENED EW".

Per my recent declaration of idgi pride, I must tell you: idgi.

just generally a place to take your ilx whenever it breaks

oh whoops, that could look like an attack if you don't know what a genius bar is

How much money is needed? I'd be more than happy to help contribute since I've missed passed fund drives. I've been on and off ilx for awhile now and with how much I've learn about music and whatnot it's the least I could do.

I need teh dunderhead bar :(

;___;

fwiw "genius bar" is what apple calls its service desks in shops

so, you know, you would take your ilx to the ilx genius bar and an ilx genius would fix it for you

I don't see any reason to tell unregs they can escape ads - that's just going to increase the number of people who register to lurk (or read one topic one time), further stressing the DB. If they decide to register, let the lack of ads be an Easter egg.

One simple tip that makes ILXors furious!
Click Here To Find Out More

yeah jesus christ don't encourage random googlers...

I wonder how many new eyes we got today just from MSN and NPR mentions.

During the downtime I searched for something completely unrelated to ilx and a gr8080 thread was on the first page of results.

welcome to your google bubble

Schlafsack, I didn't think you were being mean to me.

*\(^_^)/*

I would prefer that ILX remain unlike Metafilter, and I kinda like metafilter, for different things.

But I would also like to be able to send stet $5 (or "£5") at whatever arbitrary point in time it is convenient for me to do so.

As an N.B., mefi's revenue is dominated by ad views for googlers clicking on Ask Metafilter threads.

A friend of mines just posted that he's giving away a Sun E3000 - 6 x 250Mhz UltraSPARC-II w/ 6GB RAM and other accoutrements, pity it'd be too prohibitive to ship it overseas.

lets auction off the right to ban people/bring back banned people

But I would also like to be able to send stet $5 (or "£5") at whatever arbitrary point in time it is convenient for me to do so.

YES.

165,000₫ ready to post

would support ads for unregistered if all current mods resign

OCCUPY ILX

― shite pele (darraghmac), Wednesday, October 19, 2011 4:21 PM (6 hours ago)

otm

i will open the floodgates of my vast, untold riches in exchange for an sb veto

that was for you kk

i'm vehemently opposed to ads cos ads are evil. sorry.

esp if they could be accompanied by some text like "hey do u hate these ads? we do! reg and they go away. for free!"

What, we're Spotify now?

can we not move ILX to facebook?

no.

No, we can not not move ILX to facebook

Are we not Facebook

wdyll

imagine if u cd 51 people from facebook :)

if you cut us, do we not trend downwards

shylock thread

i never 51 people

https://www.youtube.com/watch?v=BGde6Tfp4qs

https://www.youtube.com/watch?v=nEtRaNSSbnQ

The Ultima Dragons moved to Facebook, but their group gets about 10 messages a week. We get that many an hour. No, I don't think it would work.

10 a minute, pretty sure.

i feel different since ilx went down/back up

anytime something or someone goes down on you, it changes the relationship

you guys know ilx with keyword-related porn ads would fucking slay

advertisements please, the time is now

i like the idea of google ads because of the aesthetic cohesion between them and ilx - i think i even mocked it up once on some thread somewhere

also old.css should be the default stylesheet

YES.

old.css is uuuugly imo

caek.css should be the default stylesheet you monsters.

reasonable compromise

old.css or get out

if you use anything besides old.css youre not actually browsing ilx

style sheet more like DENIAL sheet

style.css

judas.css

old.css makes big images tiny and difficult to see for old people, that is why it's called old.

I love ilxor. Let me know when it's time to donate. I believe I am B negative.

I can't remember which one I use. I wish there was a "you're currently using ________ style" on the prefs page.

click on "I want to change the way the site looks" and it will tell you.

style.css is the only one I can deal with at all.

I would poll the stylesheets but that's been done already

old for lyfe (ps: shut up)

oh shit, it does say what I'm currently using, doesn't it? It didn't used to, iirc!

ILX: B negative

you guys are like people who complain whenever facebook interface changes

stfu n00b

just reverted to old.css for like 5 seconds and shuddered

because it suxxxxxxx

i love the turquoise grey blue and occasional yellow rectangular blocks of standard.css

blueski-2 forever and ever.

otm

I hate when the way a message board's look changes. I tried out various styles and now I can't remember which one I'd been using. I'm trying not to panic but I can't fix this. This time I flew too high.

The exact same thing happened to me yesterday after I got curious reading this thread! Have no recollection of ever picking style.css but that's where I eventually got back to after a pretty rough 30 minutes.

blueski2 for all time.

https://www.youtube.com/watch?v=TNQdfHU7Vws&feature=fvst

OK phew. Back to comfy familiar ledge2.css. Glad I didn't forget.

Gah the whole point of ILX is it is completely plain text, yr all using it rong dammit. You might as well use a Mac with a Windows XP skin.

Sadly I don't seem to have a text-mode browser on this computer. I should remedy this.

While lynx compiles, I'd like to request the ability to browse ilx over the gopher protocol.

As usual, shit does not compile the first time on my mac, and I don't feel like trying to figure out what configure options I need to get this one to work. #rivetingupdates

If you're not using a teletype to surf ILX, you're a noob.

nokia 5230 ftw, no more annoying 'xp' notifications

I use a Herman Hollerith hPad.

My bookmarks never came back. As a result, I missed a whole week's worth of new Sandra Lee curiosities.