Threads on ILM missing on SNA

Some threads that are listed on ILM because they were updated this morning are not showing on up on (my) Site New Answers (which shows ILM). For example,

Transient Random-Noise Bursts w/ Polling Options [Started by Yngwie Hoosteen (Pillbox) in July 2009, last updated 1 hour ago by Jeff W] 23 new answers POLL closes: August 05 (in 1 week)

The Tres Hombres Poll: Which one is the Top? [Started by Euler in February 2008, last updated 1 hour ago by Sugar Slam (Euler)] 1 new answer POLL results

both show up on ILM but not on SNA.

Maybe there are other threads with this property as well. Is this a problem on my end only, somehow, or is something else going on?

i've noticed the same thing with threads from ILE a couple times

more censorship from our new regime of repressed jackbooted anti-masturbators

also fwiw the thread

'Like a Hurricane' vs. 'I'm the Ocean' - (which is the best Neil Young water metaphor song that is around 7 or 8 minutes long?) [Started by iatee in July 2009, last updated 5 hours ago by iatee] 27 new answers POLL closes: August 01 (in 4 days)

is missing from (my) SNA. It should be between the "pretty boy" and "Juggalo" threads (woot woot).

if you post in them they come right back

The Sonia Sotomayor Confirmation Thread

I think I'll wait on re-upping them myself, in case a coding mod can use this to pinpoint what's up.

The latest to disappear is

Mark Ronson? [Started by JoB (JoB) in August 2003, last updated 43 minutes ago by J4mi3 H4rl3y (Snowballing)] 5 new answers

which should be between the Misfits and rolling punk threads.

I'll stop checking for these after a little while, but I hope this is helpful.

I don't have time to look into this now, but thanks for letting me know - I'll take a look when I'm a little less busy.

But yes I can see a couple of threads missing. How strange. Don't think this is going to be all that easy to track down, so bear with me please.

Incidentally, do you mean that they do appear at the top of site new answers for a while and then disappear at some point after that? Or do they just never make it on to site new answers?

they definitely appear at first and then disappear, because i never look at just ILE new answers so i wouldn't know about the threads at all otherwise

Oh OK, cheers... How strange. This isn't obvious at all - as I say, might take some time. Thanks...

Great, Keith. I figured this would be hard to sort out (I used to program in a former life). If you want further instances of this as you're pinpointing the bug, just revive this thread and I'm sure someone can quickly find an example. I found several more off ILE today that I didn't post; it's happening fairly often.

Thanks, and in a strange way, that is good that it's happening reasonably frequently, as it ought to make it easier to spot.

Also - do you know if this is a recent thing, or it's been happening for ages?

I have thought this was happening before but whenever I do a search for the thread title I'm looking for, I can find it.

first i noticed was that july 15 one

I only noticed it today. xp

OK thanks, well it does seem to be happening. If anyone spots any common factors between the threads it's happening to, can they let me know, please?

i thought i was just imagining that there was a sotomayor thread, it was like a dream! then i found it in ILE new answers.

yep, same with my ZZ Top poll that I'd revived, only to see it gone (until I clicked over to ILM).

Right now this one is missing from my SNA but is on ILM.

Lykke Li - This summer's biggest hit? [Started by StanM in February 2008, last updated 1 hour ago by Mike Crandle, Financial Analyst, Bear Stearns, New York, NY 10185 (res)] 1 new answer

Yes, I've had a quick look and I'm beginning to suspect what's going on, but I can't quite figure it.

It's been years since I looked at this stuff, because no-one's ever noticed it not working, but I suspect it might never have been working quite right. There is a bug I can see, whether it's causing it or not, I don't know, but I'll take a look.

Basically, all boards are set with a specific number of entries on a new answers page, but site new answers defaults to 200.

There's a bunch of stuff here that when you put a new thread in, ensures that the cache hasn't got bigger than it's maximum size, but when it's site new answers that's being updated, it takes the board ID from the original board and gets the max new answers from that board, and if the cache size is greater than that, removes the entry at that point, but that's set to 100 for ILM and the same for ILE, so it looks like it'll knock something off of the middle of site new answers.

What I'll do for now is to set the max new answers on ILM only to 200, and then see if there are still problems. If there aren't, then that's probably it and I can think of the best way to fix.

OK I have reset ILM and restarted.

If I'm right, we should see all ILM threads OK on site new answers, but still see some dropping off from ILE (and probably other boards too).

At least 3 ILM threads missing now, from earlier in the list:

Best song on my current iTunes Genius playlist [Started by Four-TEEEEEEEEEEEEEEEEEEEEN! (HI DERE) in July 2009, last updated 34 minutes ago by the who cares (okamax)] 16 new answers POLL closes: July 31 (in 3 days)

Head Of David - 'SoulSpark' 10" [Started by aldo_cowpat (aldo_cowpat) in October 2003, last updated 35 minutes ago]

identify this record [Started by gareth in August 2002, last updated 37 minutes ago]

Do you mean right now, or before I restarted?

No you mean now, well bang goes that idea.

Correct, now; also some more further down (still ILM; will check ILE shortly):

The Church's new 2009 album/onslaught of releases [Started by Chris Barrus (Elvis Telecom) in January 2009, last updated 3 hours ago]

Esplendor Geometrico S/D [Started by Fetchboy (Felcher) in September 2006, last updated 4 hours ago]

Cassie: The new Ciara? [Started by The Brainwasher (Twilight) in May 2006, last updated 4 hours ago]

anticipate The Field - Yesterday and Today [Started by uncannydan in March 2009, last updated 4 hours ago]

'Like a Hurricane' vs. 'I'm the Ocean' - (which is the best Neil Young water metaphor song that is around 7 or 8 minutes long?) [Started by iatee in July 2009, last updated 4 hours ago] POLL closes: August 01 (in 4 days)

Though having said that, removing a given entry from site new answers would be from an arbitrary board, but still, I'd expect it to be further down than the first one you mention.

That lot are all in a row though, and not inconceivable they could be about a hundred down in the list. Maybe this makes some sense. I can't quite figure the iTunes one.

And several from ILE missing too, also from about the same time period (about 36 minutes ago):

UK Watercooler 36: ANATOLIAN BREAKFAST [Started by Bubble Withdrawal (Masonic Boom) in March 2009, last updated 36 minutes ago]
ILX running running thread [Started by Jeff in June 2007, last updated 37 minutes ago by blobfish russian (harbl)] 4 new answers
What do you look like? July 2009 [Started by wrong traditions are kept alive (country matters) in July 2009, last updated 38 minutes ago by Tuomas] 76 new answers
Can we have a new Flickr thread...(AKA Flickr.com 3) [Started by not_goodwin in October 2008, last updated 41 minutes ago by unban dictionary (blueski)] 1 new answer

All in a list, again.

Yes, I think this is what's going on (except the confusing iTunes one)... They're all in a bunch next to each other. There's more action on ILE, so the period where they're getting knocked off is closer to now. It makes some sense given that I can see there is actually a bug there.

And at about 4 hours, on ILE, same period as the other ILM one, also in a (big) list:

The official bored-at-work cryptic crossword pass it on thread. [Started by Gravel Puzzleworth (Gregory Henry) in December 2005, last updated 4 hours ago]
I am annoyed [Started by jel -- (jel) in December 2002, last updated 4 hours ago]
assassins slowly crumbling [Started by m. white btw (cozwn) in July 2009, last updated 4 hours ago]
Film noir: your favourites [Started by Jonathan Z. (Joanthan Z.) in April 2004, last updated 4 hours ago]
The League of Gentlemen - C/D? (now with added Psychoville) [Started by Matt DC (Matt DC) in November 2004, last updated 4 hours ago by grimly fiendish] 17 new answers
GOAL of the DECADE [Started by I can't make my face turn into a heart (Scik Mouthy) in July 2009, last updated 4 hours ago by Susan Tully Blanchard (MPx4A)] 48 new answers
HBO's True Blood -- the most preposterous sounding show since... [Started by Will M. in July 2008, last updated 4 hours ago]
The rolling RIP/obituary thread [Started by Elvis Telecom in July 2009, last updated 4 hours ago]
Mead in Pubs! [Started by MarkH (MarkH) in June 2004, last updated 4 hours ago]
Drug tripping with MSNBC and their list of 11 mind-altering vacations [Started by Carroll Shelby Downard (Elvis Telecom) in April 2009, last updated 5 hours ago] POLL results
goin to africa, brb [Started by i want to marry a pizza (gbx) in June 2009, last updated 5 hours ago]
Anticipating Michael Mann's 'Public Enemies', with Depp, Bale, et al. [Started by Eazy in March 2009, last updated 5 hours ago by julien schNAGL (s1ocki)] 7 new answers

OK, this looks at least vaguely conclusive. I'm going to have to run now, but I know what to do to sort it. I'll take a look tomorrow evening if I get the chance.

Thanks for the help.

Great. If you want any more examples of missing threads, we'll be happy to help.

OK, I think I have fixed this - I am about to deploy, but it's really tricky to test locally - kind of needs lots of threads being bumped by people. I'll upload it in a minute and start it and see what happens. If it's broken, I'll just revert to the old version and take my time about fixing it.

Well, so far so good, that looks to have sorted it. Hopefully it hasn't had any other nasty knock on effects.

Let me know if you see any problems.

Bloody annoying that this has been basically broken for probably 18 months without me or anyone else noticing!

It looks right this morning, Keith, on both ILM and ILE. I haven't checked subboards beyond those, but it sounds like the problem wasn't specific to any particular board. Thanks for your work on this, as always!