[ADMIN] So what have ILX been up to? And what next?

Message Bookmarked
Bookmark Removed

So: sorry baout the downtime this week.

What happened:
Basically a rare kind of server crash (caused by a bug in the operating system that only showed up after years) wrote a load of junk to key parts of the hard drives. That first made it look to the hosts like a hardware failure, then made us suspect there'd been an attack, and then finally meant some really low-level bits of the operating system had to be rebuilt before we could get back in. Our backups worked, so I restored from them, and here we are.

What now:
Everything should be back up and running. Search will take a few days to get back up to speed, and other indexes and caches will take a while to be normal. You might see slowness, and unread markers will be wrong etc. There's also probably going to be some more short bits of downtime, probably at the weekend, as I'm going to do some stuff to try and make the backups more reliable.

What next?
We (and I) got a bit lucky there, as the off-site backups were a good few weeks out of date. Part of the problem here is that the database is now getting really big -- more than 3gb in size, so backups take a long time to run, and nearly as long to move about. The correct thing to do in this situation would be to backup live to another server, which we don't have.

What's more, the server we do have is getting on -- it's 5 years old -- and if it fails we have nothing else to replace it with. The support contract has now expired, and Oracle (who bought Sun) want ludicrous money to renew it.

Meantime, I picked up a backup server for cheap some months back, but haven't got it set up yet. If we'd had to move to that, it would have taken a goodish number of weeks to get going, especially if I had to ship it to Canada. Putting it in a datacentre ahead of time would also be expensive -- would basically double our hosting bill. As an alternative, I looked at cloud-based server options (like Amazon's EC2, or VPS hosting) but for the amount of memory and database space we need, it's even more expensive to do.

Happily, the hosts think they can sort us out with a virtual server in their datacentre. This would in theory let us move ILX to a more modern setup, and potentially keep the existing server as a live backup.

So, that's my plan just now; I'm waiting on them getting back to me with details. If that doesn't work, I'll see if there are other options, potentially like reserving an Amazon server that only spins up once a day, downloads changes from the database, then goes back to sleep.

I guess one of the questions is how long a period of downtime are we prepared to take if this server goes down, and how many day's posts are we prepared to lose? I'd like that to be "hardly any" and "none at all", but that might cost us an ongoing chunk of change. What we do depends on exactly how much, I guess.

stet, Wednesday, 19 October 2011 19:57 (thirteen years ago)

(Not to derail right away, but I've been wondering about how much we'd make putting ads on ilxor.com for non-registered users, hiding them from registereds; there is a lot of hassle connected with that, but if it gave us the money to run things properly I think it'd be worth considering.)

stet, Wednesday, 19 October 2011 19:59 (thirteen years ago)

I recommend an ilx fund drive while you work out the details on spamming the unregistered

the boy with the gorn at his side (Edward III), Wednesday, 19 October 2011 20:01 (thirteen years ago)

1) Er, I'm sure most of us could live without a couple of weeks on ILX (probably would help us somewhat in our normal lives). More than that? I guess I could get through that.

2) I have no issue with ads for non-registered users; there's potential for a goldmine of jokes out of it.

Miles "Tails" Davis (King Boy Pato), Wednesday, 19 October 2011 20:02 (thirteen years ago)

I've always thought ads for unregistered eyes was a smart way to go.

Martyr McFly (WmC), Wednesday, 19 October 2011 20:03 (thirteen years ago)

I recommend an ilx fund drive while you work out the details on spamming the unregistered

― the boy with the gorn at his side (Edward III), Wednesday, October 19, 2011 4:01 PM (2 minutes ago) Bookmark

what did unregistered ever do to you

2001: a based godyssey (dayo), Wednesday, 19 October 2011 20:03 (thirteen years ago)

he's a liar for one thing

the boy with the gorn at his side (Edward III), Wednesday, 19 October 2011 20:04 (thirteen years ago)

well done Stet for sorting out these intractable-sounding issues.

good luck in your pyramid (Neil S), Wednesday, 19 October 2011 20:05 (thirteen years ago)

Thanks stet!

Go for the fundraiser. I missed out on the last funding rush, and iirc quite a few others did too.

Still think we could make a go of ILX merch.

How many hits do we get from unregistereds?

Also unknown as Zora (Surfing At Work), Wednesday, 19 October 2011 20:06 (thirteen years ago)

Do we have the ability to do incremental backups rather than backing up the entire database?

do not wake the dragon (DJP), Wednesday, 19 October 2011 20:06 (thirteen years ago)

we could just sign ILX up for iCloud

2001: a based godyssey (dayo), Wednesday, 19 October 2011 20:08 (thirteen years ago)

Restoring using incremental backups or transaction logs is a lot if work, iirc xpost

StanM, Wednesday, 19 October 2011 20:08 (thirteen years ago)

I'm all in favor of ads for unregistereds.

In the meantime, it seems like ILX Fund Raising always finishes before a lot of folks who want to contribute can - maybe run one for a set amount of time and just see how much gets donated rather than capping it at a hard number?
Could even do it through Kickstarter so that if it didn't reach an arbitrary goal (however much it takes to buy a new pimped-out server? I have no clue how much that is.) folks get their $$$ back.

Kiarostami bag (milo z), Wednesday, 19 October 2011 20:09 (thirteen years ago)

but I'd be in favor of increased costs for more modern servers/backups. we max out the funding drive within hours each year!

2001: a based godyssey (dayo), Wednesday, 19 October 2011 20:09 (thirteen years ago)

Milo: if you collect too much you may be legally obliged to register as an organisation domewhere (where?) etc

StanM, Wednesday, 19 October 2011 20:11 (thirteen years ago)

domewhere? I can't even type anymore, so emotional

StanM, Wednesday, 19 October 2011 20:12 (thirteen years ago)

thunderdomewhere

Martyr McFly (WmC), Wednesday, 19 October 2011 20:13 (thirteen years ago)

I'm planning on winning the lottery and donating a server and hosting and liberate stet from his day job etc. - wish me luck!

StanM, Wednesday, 19 October 2011 20:14 (thirteen years ago)

We do, but on mysql they're taken from binary log files, which means you need to do pretty regular full backups anyway (and also restart the logging) or else you're just building on a giant binary of hope. xxp

On the funding thing: it's true that we raise what we need pretty quickly; this is also partly about thinking what the best thing to buy/rent is, so then we can work out how much it'll cost!

stet, Wednesday, 19 October 2011 20:14 (thirteen years ago)

Good work getting things back in business, Stet.

And yeah, happy to throw into another funding drive, buy an ILX sweatshirt, support ads for nonreg's etc. Whatever it takes to keep this noble ship afloat.

that mustardless plate (Bill A), Wednesday, 19 October 2011 20:15 (thirteen years ago)

thx stet; I'm not a sysadmin so I only know about this stuff in theory

do not wake the dragon (DJP), Wednesday, 19 October 2011 20:15 (thirteen years ago)

i am also 100% in favor of ads for unregs

re: the fundraising etc yeah we would need to look into that, not that the site is prob going to be rolling in money, but i imagine there would need to be some sort of non-profit org set up or something to handle how the account would work. where my lawyer friends at?

American Horror Sorry (jjjusten), Wednesday, 19 October 2011 20:16 (thirteen years ago)

That's an issue with ads, too, I guess.
ILX gets a ridiculous amount of traffic beyond just us registered folk, IIRC.

Kiarostami bag (milo z), Wednesday, 19 October 2011 20:17 (thirteen years ago)

oh yeah i was talking about fundraising and ads if that wasnt clear - some entity has to hold the funds/document the income etc in order to keep anyone from being personally responsible/taxable/whatevs

American Horror Sorry (jjjusten), Wednesday, 19 October 2011 20:18 (thirteen years ago)

my I suggest... Loutallica

do not wake the dragon (DJP), Wednesday, 19 October 2011 20:19 (thirteen years ago)

I find the occasional disastrous downtime therapeutic tbh, as long as it isn't too disastrous.

building a binary of hope (onimo), Wednesday, 19 October 2011 20:21 (thirteen years ago)

would support ads for unregistered if all current mods resign

OCCUPY ILX

shite pele (darraghmac), Wednesday, 19 October 2011 20:21 (thirteen years ago)

You might see slowness, and unread markers will be wrong etc

I lol'd

Y Kant Lou Reed (Le Bateau Ivre), Wednesday, 19 October 2011 20:21 (thirteen years ago)

well there goes my productivity

loads of personality, loved to chase chickens (forksclovetofu), Wednesday, 19 October 2011 20:21 (thirteen years ago)

Isn't there some way to set up links to sites like Amazon, so that whenever one of us wants to purchase something there, we could click through the link and ILX would automatically get a referral credit? I've heard of blogs raising money that way. Not sure if it would be much, but it seems like a fairly painless way to help out.

o. nate, Wednesday, 19 October 2011 20:22 (thirteen years ago)

could we autoreplace chaki etc with ads

cozen, Wednesday, 19 October 2011 20:23 (thirteen years ago)

Let people who are banned buy their way back in?

Only ILX gold accounts are allowed to vote on this poll, sorry.

StanM, Wednesday, 19 October 2011 20:26 (thirteen years ago)

Let people who are banned buy their way back in?

OMG YES

Martyr McFly (WmC), Wednesday, 19 October 2011 20:28 (thirteen years ago)

that way lies metafilter

stet, Wednesday, 19 October 2011 20:29 (thirteen years ago)

You might see slowness, and unread markers

http://i.imgur.com/OEwRG.png

Whiney G. Weingarten, Wednesday, 19 October 2011 20:38 (thirteen years ago)

How does SA do what they do? They charge $10 or something to be able to post, right? Does someone run it as a business? I mean I know theyre a shiiiiit ton bigger, but they seem to run on a model not too dissimilar to us in some ways.

Occupy Kelly St (Trayce), Wednesday, 19 October 2011 20:52 (thirteen years ago)

SA charges $10 per post

2001: a based godyssey (dayo), Wednesday, 19 October 2011 20:55 (thirteen years ago)

i instinctively want to say hi to ppl whose posts i haven't read in nearly a week

nakhchivan, Wednesday, 19 October 2011 20:58 (thirteen years ago)

hi ilx ppl

nakhchivan, Wednesday, 19 October 2011 20:58 (thirteen years ago)

that's just awful

xp

i love pinfold cricket (gbx), Wednesday, 19 October 2011 20:58 (thirteen years ago)

hi nak

loads of personality, loved to chase chickens (forksclovetofu), Wednesday, 19 October 2011 20:59 (thirteen years ago)

whiney :D

markers, Wednesday, 19 October 2011 21:05 (thirteen years ago)

Salient points itt:

1. ilx needs money
2. unregistered is a liar
3. markers will be wrong

Autumn Almanac (Schlafsack), Wednesday, 19 October 2011 22:07 (thirteen years ago)

thx 4 new sn

bomb.gif (dan m), Wednesday, 19 October 2011 22:09 (thirteen years ago)

Is a subscription model worth thinking about? Depending on implementation, the risk there is that you get tiers of users – the plebs who pay nothing and get kicked about, and the elites who do all the kicking and have more power than stet. Then again, you could always just have a tier called 'passionate' that doesn't net the user any advantage at all except a warm cosy feeling (3RRR does this.

Autumn Almanac (Schlafsack), Wednesday, 19 October 2011 22:11 (thirteen years ago)

)

Autumn Almanac (Schlafsack), Wednesday, 19 October 2011 22:11 (thirteen years ago)

we shd never ever do anything that makes us like SA or Metafilter imo

iirc fundraising drives have won out over ads in the past because they present less legal hassles? tbh personally i'd be against ads in any circumstances but i guess until stet's got a firm-ish estimate for how much cash we need this is a moot point

Two Noble Klinsmenn (Noodle Vague), Wednesday, 19 October 2011 22:16 (thirteen years ago)

please god no tiers no subscriptions no inner/outer circles

Two Noble Klinsmenn (Noodle Vague), Wednesday, 19 October 2011 22:17 (thirteen years ago)

How much would AWS really cost? We've never had a problem fundraising and it seems like it would be the most robust solution.

American Fear of Pranksterism (Ed), Wednesday, 19 October 2011 22:17 (thirteen years ago)

NO SUBSCRIPTION. That would be shit. I'd rather just go with more fundraisers than ads for unregistered users, but ads would be preferable to subscriptions.

emil.y, Wednesday, 19 October 2011 22:21 (thirteen years ago)

wdyll

shite pele (darraghmac), Thursday, 20 October 2011 15:43 (thirteen years ago)

imagine if u cd 51 people from facebook :)

Two Noble Klinsmenn (Noodle Vague), Thursday, 20 October 2011 15:45 (thirteen years ago)

if you cut us, do we not trend downwards

Threat Level: Panda (jjjusten), Thursday, 20 October 2011 15:45 (thirteen years ago)

shylock thread

Two Noble Klinsmenn (Noodle Vague), Thursday, 20 October 2011 15:47 (thirteen years ago)

i never 51 people

Gary Numan, or Gary Fletcher (ken c), Thursday, 20 October 2011 15:47 (thirteen years ago)

https://www.youtube.com/watch?v=BGde6Tfp4qs

Gary Numan, or Gary Fletcher (ken c), Thursday, 20 October 2011 15:56 (thirteen years ago)

https://www.youtube.com/watch?v=nEtRaNSSbnQ

Gary Numan, or Gary Fletcher (ken c), Thursday, 20 October 2011 15:57 (thirteen years ago)

The Ultima Dragons moved to Facebook, but their group gets about 10 messages a week. We get that many an hour. No, I don't think it would work.

Christine Green Leafy Dragon Indigo, Friday, 21 October 2011 07:33 (thirteen years ago)

10 a minute, pretty sure.

Gary Numan, or Gary Fletcher (ken c), Friday, 21 October 2011 09:06 (thirteen years ago)

i feel different since ilx went down/back up

flopson, Monday, 24 October 2011 01:48 (thirteen years ago)

anytime something or someone goes down on you, it changes the relationship

some dude, Monday, 24 October 2011 02:20 (thirteen years ago)

you guys know ilx with keyword-related porn ads would fucking slay

advertisements please, the time is now

pathos of the unwarranted encore (underrated aerosmith bootlegs I have owned), Monday, 24 October 2011 02:28 (thirteen years ago)

i like the idea of google ads because of the aesthetic cohesion between them and ilx - i think i even mocked it up once on some thread somewhere

ice cr?m, Monday, 24 October 2011 02:34 (thirteen years ago)

also old.css should be the default stylesheet

ice cr?m, Monday, 24 October 2011 02:35 (thirteen years ago)

YES.

Trayce, Monday, 24 October 2011 03:06 (thirteen years ago)

old.css is uuuugly imo

flopson, Monday, 24 October 2011 03:10 (thirteen years ago)

caek.css should be the default stylesheet you monsters.

caek, Monday, 24 October 2011 03:45 (thirteen years ago)

reasonable compromise

ice cr?m, Monday, 24 October 2011 03:45 (thirteen years ago)

old.css or get out

Millsner, Monday, 24 October 2011 04:03 (thirteen years ago)

if you use anything besides old.css youre not actually browsing ilx

max, Monday, 24 October 2011 13:19 (thirteen years ago)

style sheet more like DENIAL sheet

mid-song laughing elvis (schlump), Monday, 24 October 2011 13:20 (thirteen years ago)

style.css

Jeff, Monday, 24 October 2011 13:47 (thirteen years ago)

judas.css

mid-song laughing elvis (schlump), Monday, 24 October 2011 13:57 (thirteen years ago)

old.css makes big images tiny and difficult to see for old people, that is why it's called old.

caek, Monday, 24 October 2011 14:02 (thirteen years ago)

I love ilxor. Let me know when it's time to donate. I believe I am B negative.

Josh in Chicago, Monday, 24 October 2011 14:04 (thirteen years ago)

I can't remember which one I use. I wish there was a "you're currently using ________ style" on the prefs page.

Martyr McFly (WmC), Monday, 24 October 2011 16:55 (thirteen years ago)

click on "I want to change the way the site looks" and it will tell you.

style.css is the only one I can deal with at all.

sleeve, Monday, 24 October 2011 16:57 (thirteen years ago)

I would poll the stylesheets but that's been done already

old for lyfe (ps: shut up)

do not wake the dragon (DJP), Monday, 24 October 2011 16:58 (thirteen years ago)

oh shit, it does say what I'm currently using, doesn't it? It didn't used to, iirc!

Martyr McFly (WmC), Monday, 24 October 2011 17:09 (thirteen years ago)

ILX: B negative

mid-song laughing elvis (schlump), Monday, 24 October 2011 18:50 (thirteen years ago)

you guys are like people who complain whenever facebook interface changes

flopson, Monday, 24 October 2011 23:48 (thirteen years ago)

stfu n00b

ice cr?m, Monday, 24 October 2011 23:54 (thirteen years ago)

just reverted to old.css for like 5 seconds and shuddered

flopson, Monday, 24 October 2011 23:59 (thirteen years ago)

because it suxxxxxxx

flopson, Tuesday, 25 October 2011 00:00 (thirteen years ago)

i love the turquoise grey blue and occasional yellow rectangular blocks of standard.css

flopson, Tuesday, 25 October 2011 00:02 (thirteen years ago)

blueski-2 forever and ever.

They're coming to get you, (Jenny), Tuesday, 25 October 2011 00:06 (thirteen years ago)

otm

Autumn Almanac (Schlafsack), Tuesday, 25 October 2011 00:19 (thirteen years ago)

I hate when the way a message board's look changes. I tried out various styles and now I can't remember which one I'd been using. I'm trying not to panic but I can't fix this. This time I flew too high.

Je55e, Tuesday, 25 October 2011 02:54 (thirteen years ago)

The exact same thing happened to me yesterday after I got curious reading this thread! Have no recollection of ever picking style.css but that's where I eventually got back to after a pretty rough 30 minutes.

kinder, Tuesday, 25 October 2011 02:56 (thirteen years ago)

blueski2 for all time.

A Lip in the Blandscape (jaymc), Tuesday, 25 October 2011 02:58 (thirteen years ago)

https://www.youtube.com/watch?v=TNQdfHU7Vws&feature=fvst

Je55e, Tuesday, 25 October 2011 03:02 (thirteen years ago)

OK phew. Back to comfy familiar ledge2.css. Glad I didn't forget.

Je55e, Tuesday, 25 October 2011 03:04 (thirteen years ago)

Gah the whole point of ILX is it is completely plain text, yr all using it rong dammit. You might as well use a Mac with a Windows XP skin.

Trayce, Tuesday, 25 October 2011 03:24 (thirteen years ago)

Sadly I don't seem to have a text-mode browser on this computer. I should remedy this.

medium rear (silby), Tuesday, 25 October 2011 03:41 (thirteen years ago)

While lynx compiles, I'd like to request the ability to browse ilx over the gopher protocol.

medium rear (silby), Tuesday, 25 October 2011 03:41 (thirteen years ago)

As usual, shit does not compile the first time on my mac, and I don't feel like trying to figure out what configure options I need to get this one to work. #rivetingupdates

medium rear (silby), Tuesday, 25 October 2011 03:44 (thirteen years ago)

If you're not using a teletype to surf ILX, you're a noob.

cheque out my debit to building society (snoball), Tuesday, 25 October 2011 07:04 (thirteen years ago)

nokia 5230 ftw, no more annoying 'xp' notifications

generation lmbo (darraghmac), Tuesday, 25 October 2011 08:07 (thirteen years ago)

I use a Herman Hollerith hPad.

cheque out my debit to building society (snoball), Tuesday, 25 October 2011 08:19 (thirteen years ago)

My bookmarks never came back. As a result, I missed a whole week's worth of new Sandra Lee curiosities.

Je55e, Friday, 28 October 2011 19:59 (thirteen years ago)


You must be logged in to post. Please either login here, or if you are not registered, you may register here.