[ADMIN] So what have ILX been up to? And what next?

So: sorry baout the downtime this week.

What happened:
Basically a rare kind of server crash (caused by a bug in the operating system that only showed up after years) wrote a load of junk to key parts of the hard drives. That first made it look to the hosts like a hardware failure, then made us suspect there'd been an attack, and then finally meant some really low-level bits of the operating system had to be rebuilt before we could get back in. Our backups worked, so I restored from them, and here we are.

What now:
Everything should be back up and running. Search will take a few days to get back up to speed, and other indexes and caches will take a while to be normal. You might see slowness, and unread markers will be wrong etc. There's also probably going to be some more short bits of downtime, probably at the weekend, as I'm going to do some stuff to try and make the backups more reliable.

What next?
We (and I) got a bit lucky there, as the off-site backups were a good few weeks out of date. Part of the problem here is that the database is now getting really big -- more than 3gb in size, so backups take a long time to run, and nearly as long to move about. The correct thing to do in this situation would be to backup live to another server, which we don't have.

What's more, the server we do have is getting on -- it's 5 years old -- and if it fails we have nothing else to replace it with. The support contract has now expired, and Oracle (who bought Sun) want ludicrous money to renew it.

Meantime, I picked up a backup server for cheap some months back, but haven't got it set up yet. If we'd had to move to that, it would have taken a goodish number of weeks to get going, especially if I had to ship it to Canada. Putting it in a datacentre ahead of time would also be expensive -- would basically double our hosting bill. As an alternative, I looked at cloud-based server options (like Amazon's EC2, or VPS hosting) but for the amount of memory and database space we need, it's even more expensive to do.

Happily, the hosts think they can sort us out with a virtual server in their datacentre. This would in theory let us move ILX to a more modern setup, and potentially keep the existing server as a live backup.

So, that's my plan just now; I'm waiting on them getting back to me with details. If that doesn't work, I'll see if there are other options, potentially like reserving an Amazon server that only spins up once a day, downloads changes from the database, then goes back to sleep.

I guess one of the questions is how long a period of downtime are we prepared to take if this server goes down, and how many day's posts are we prepared to lose? I'd like that to be "hardly any" and "none at all", but that might cost us an ongoing chunk of change. What we do depends on exactly how much, I guess.

(Not to derail right away, but I've been wondering about how much we'd make putting ads on ilxor.com for non-registered users, hiding them from registereds; there is a lot of hassle connected with that, but if it gave us the money to run things properly I think it'd be worth considering.)

I recommend an ilx fund drive while you work out the details on spamming the unregistered

1) Er, I'm sure most of us could live without a couple of weeks on ILX (probably would help us somewhat in our normal lives). More than that? I guess I could get through that.

2) I have no issue with ads for non-registered users; there's potential for a goldmine of jokes out of it.

I've always thought ads for unregistered eyes was a smart way to go.

I recommend an ilx fund drive while you work out the details on spamming the unregistered

― the boy with the gorn at his side (Edward III), Wednesday, October 19, 2011 4:01 PM (2 minutes ago) Bookmark

what did unregistered ever do to you

he's a liar for one thing

well done Stet for sorting out these intractable-sounding issues.

Thanks stet!

Go for the fundraiser. I missed out on the last funding rush, and iirc quite a few others did too.

Still think we could make a go of ILX merch.

How many hits do we get from unregistereds?

Do we have the ability to do incremental backups rather than backing up the entire database?

we could just sign ILX up for iCloud

Restoring using incremental backups or transaction logs is a lot if work, iirc xpost

I'm all in favor of ads for unregistereds.

In the meantime, it seems like ILX Fund Raising always finishes before a lot of folks who want to contribute can - maybe run one for a set amount of time and just see how much gets donated rather than capping it at a hard number?
Could even do it through Kickstarter so that if it didn't reach an arbitrary goal (however much it takes to buy a new pimped-out server? I have no clue how much that is.) folks get their $$$ back.

but I'd be in favor of increased costs for more modern servers/backups. we max out the funding drive within hours each year!

Milo: if you collect too much you may be legally obliged to register as an organisation domewhere (where?) etc

domewhere? I can't even type anymore, so emotional

thunderdomewhere

I'm planning on winning the lottery and donating a server and hosting and liberate stet from his day job etc. - wish me luck!

We do, but on mysql they're taken from binary log files, which means you need to do pretty regular full backups anyway (and also restart the logging) or else you're just building on a giant binary of hope. xxp

On the funding thing: it's true that we raise what we need pretty quickly; this is also partly about thinking what the best thing to buy/rent is, so then we can work out how much it'll cost!

Good work getting things back in business, Stet.

And yeah, happy to throw into another funding drive, buy an ILX sweatshirt, support ads for nonreg's etc. Whatever it takes to keep this noble ship afloat.

thx stet; I'm not a sysadmin so I only know about this stuff in theory

i am also 100% in favor of ads for unregs

re: the fundraising etc yeah we would need to look into that, not that the site is prob going to be rolling in money, but i imagine there would need to be some sort of non-profit org set up or something to handle how the account would work. where my lawyer friends at?

That's an issue with ads, too, I guess.
ILX gets a ridiculous amount of traffic beyond just us registered folk, IIRC.

oh yeah i was talking about fundraising and ads if that wasnt clear - some entity has to hold the funds/document the income etc in order to keep anyone from being personally responsible/taxable/whatevs

my I suggest... Loutallica

I find the occasional disastrous downtime therapeutic tbh, as long as it isn't too disastrous.

would support ads for unregistered if all current mods resign

OCCUPY ILX

You might see slowness, and unread markers will be wrong etc

I lol'd

well there goes my productivity

Isn't there some way to set up links to sites like Amazon, so that whenever one of us wants to purchase something there, we could click through the link and ILX would automatically get a referral credit? I've heard of blogs raising money that way. Not sure if it would be much, but it seems like a fairly painless way to help out.

could we autoreplace chaki etc with ads

Let people who are banned buy their way back in?

Only ILX gold accounts are allowed to vote on this poll, sorry.

Let people who are banned buy their way back in?

OMG YES

that way lies metafilter

You might see slowness, and unread markers

http://i.imgur.com/OEwRG.png

How does SA do what they do? They charge $10 or something to be able to post, right? Does someone run it as a business? I mean I know theyre a shiiiiit ton bigger, but they seem to run on a model not too dissimilar to us in some ways.

SA charges $10 per post

i instinctively want to say hi to ppl whose posts i haven't read in nearly a week

hi ilx ppl

that's just awful

xp

hi nak

whiney :D

Salient points itt:

1. ilx needs money
2. unregistered is a liar
3. markers will be wrong

thx 4 new sn

Is a subscription model worth thinking about? Depending on implementation, the risk there is that you get tiers of users – the plebs who pay nothing and get kicked about, and the elites who do all the kicking and have more power than stet. Then again, you could always just have a tier called 'passionate' that doesn't net the user any advantage at all except a warm cosy feeling (3RRR does this.

)

we shd never ever do anything that makes us like SA or Metafilter imo

iirc fundraising drives have won out over ads in the past because they present less legal hassles? tbh personally i'd be against ads in any circumstances but i guess until stet's got a firm-ish estimate for how much cash we need this is a moot point

please god no tiers no subscriptions no inner/outer circles

How much would AWS really cost? We've never had a problem fundraising and it seems like it would be the most robust solution.

NO SUBSCRIPTION. That would be shit. I'd rather just go with more fundraisers than ads for unregistered users, but ads would be preferable to subscriptions.

wdyll

imagine if u cd 51 people from facebook :)

if you cut us, do we not trend downwards

shylock thread

i never 51 people

https://www.youtube.com/watch?v=BGde6Tfp4qs

https://www.youtube.com/watch?v=nEtRaNSSbnQ

The Ultima Dragons moved to Facebook, but their group gets about 10 messages a week. We get that many an hour. No, I don't think it would work.

10 a minute, pretty sure.

i feel different since ilx went down/back up

anytime something or someone goes down on you, it changes the relationship

you guys know ilx with keyword-related porn ads would fucking slay

advertisements please, the time is now

i like the idea of google ads because of the aesthetic cohesion between them and ilx - i think i even mocked it up once on some thread somewhere

also old.css should be the default stylesheet

YES.

old.css is uuuugly imo

caek.css should be the default stylesheet you monsters.

reasonable compromise

old.css or get out

if you use anything besides old.css youre not actually browsing ilx

style sheet more like DENIAL sheet

style.css

judas.css

old.css makes big images tiny and difficult to see for old people, that is why it's called old.

I love ilxor. Let me know when it's time to donate. I believe I am B negative.

I can't remember which one I use. I wish there was a "you're currently using ________ style" on the prefs page.

click on "I want to change the way the site looks" and it will tell you.

style.css is the only one I can deal with at all.

I would poll the stylesheets but that's been done already

old for lyfe (ps: shut up)

oh shit, it does say what I'm currently using, doesn't it? It didn't used to, iirc!

ILX: B negative

you guys are like people who complain whenever facebook interface changes

stfu n00b

just reverted to old.css for like 5 seconds and shuddered

because it suxxxxxxx

i love the turquoise grey blue and occasional yellow rectangular blocks of standard.css

blueski-2 forever and ever.

otm

I hate when the way a message board's look changes. I tried out various styles and now I can't remember which one I'd been using. I'm trying not to panic but I can't fix this. This time I flew too high.

The exact same thing happened to me yesterday after I got curious reading this thread! Have no recollection of ever picking style.css but that's where I eventually got back to after a pretty rough 30 minutes.

blueski2 for all time.

https://www.youtube.com/watch?v=TNQdfHU7Vws&feature=fvst

OK phew. Back to comfy familiar ledge2.css. Glad I didn't forget.

Gah the whole point of ILX is it is completely plain text, yr all using it rong dammit. You might as well use a Mac with a Windows XP skin.

Sadly I don't seem to have a text-mode browser on this computer. I should remedy this.

While lynx compiles, I'd like to request the ability to browse ilx over the gopher protocol.

As usual, shit does not compile the first time on my mac, and I don't feel like trying to figure out what configure options I need to get this one to work. #rivetingupdates

If you're not using a teletype to surf ILX, you're a noob.

nokia 5230 ftw, no more annoying 'xp' notifications

I use a Herman Hollerith hPad.

My bookmarks never came back. As a result, I missed a whole week's worth of new Sandra Lee curiosities.