CEOP show Maddie is missing on 30th April 2007

Hi AndyB

If that's true, how do you explain http://web.archive.org/web/20080118235202/http://www.ceop.gov.uk/mccann.html ?

The most recent archive prior to that http://web.archive.org/web/20071012051850/http://www.ceop.gov.uk/ doesn't have any links to mccann.html that I can see.

It doesn't have to be the most recent archive prior, it just has to be a link at some point. If you go back to 14th May 2007 you will find the link there to mccann.html.

Sorry, I'm not being rude but what sort of background do you have that qualifies you to be so certain about how all crawlers work?

Please understand that it makes no difference to me whatsoever, whether people accept what I say. I have no agenda and no axe to grind, I happen to know a bit about crawlers so am just trying to help, and anything I say can be either verified or disproved by searching on Google.

If you don't think I'm right about something, just discount it, that's fine by me

As I said in an earlier post, there is software you can buy to run through a server that specifically seeks out orphaned pages. So clearly it is possible to find them.

That's for running on your own server though, you can't run it on anyone else's.

Yes Chirpy, it is. The first thing that leaps out at me, actually, is that none of these headlines are actually very specific! It reminds me of a "psychic" show that I went to once, with a group of friends. The guy stood up and shouted, "Is there anybody in the audience who has a birthday today, because I am getting a message for you!" Then later, it was "is anybody related to a grey haired lady called Ann who has passed over?" Another one was "has anybody got a dog, or had a dog, called Max". At least three people put their hands up (and I was one of them ).

Stay safe online - visit the café. Doesn't really tell us a lot, does it? There is a Google article, dated 27 October 2007, that pops up when you type that in, that's true. But it could also be argued that they knew in April that they were going to have a café page sometime around October. They were probably building it then.

Most wanted offender - back in police custody. You could write that at any time you liked, ever, and it would be true. Even a broken clock is right at least twice a day. Putting that phrase into Yahoo has returned 1 1/2 million hits, with all sorts of dates; if they had said "Wanted Offender Lord Everard Farquharson back in custody" then I would have been more impressed (with apologies to Lord Everard Farquharson, whoever he is).

Technophobic parents - it's time to catch up and protect your children online. A campaign launched by CEOP's sister site, police.CEOP.uk, that had been PLANNED to be launched on 1 October 2007.

So there might just be a little grain of truth when WLBTS claimed that Jim Gamble was psychic

I'm trying to remain impartial here, but none of these things really convince me. For all we know that could also have been a dummy page, a work in progress, a designer testing out future ideas, not the "real" CEOP homepage as it was intended to be seen. I honestly don't have a definitive answer to this one, sorry

What we would really need to see are the URLs where the red hyperlinks are beneath each "headline".

Nuala wrote:Hi AndyB

If that's true, how do you explain http://web.archive.org/web/20080118235202/http://www.ceop.gov.uk/mccann.html ?

The most recent archive prior to that http://web.archive.org/web/20071012051850/http://www.ceop.gov.uk/ doesn't have any links to mccann.html that I can see.

It doesn't have to be the most recent archive prior, it just has to be a link at some point. If you go back to 14th May 2007 you will find the link there to mccann.html.

Sorry, I'm not being rude but what sort of background do you have that qualifies you to be so certain about how all crawlers work?

Please understand that it makes no difference to me whatsoever, whether people accept what I say. I have no agenda and no axe to grind, I happen to know a bit about crawlers so am just trying to help, and anything I say can be either verified or disproved by searching on Google.

If you don't think I'm right about something, just discount it, that's fine by me

No, that misses the point. You've come on here and started telling people how all crawlers work. I just want to know what your experience is that allows you to authoritatively tell people how all crawlers work. What is your technical expertise please? That will allow me to make an informed decsion about whether I should attach weight to your opinion, ignore you completely, or something inbetween

Nuala wrote:As I said in an earlier post, there is software you can buy to run through a server that specifically seeks out orphaned pages. So clearly it is possible to find them.

That's for running on your own server though, you can't run it on anyone else's.

And you think that Government agencies and other bodies like that won't use it?

Resistor. Glad I didn`t imagine seeing that page. I found it not in my files but from Steve Marsden`s post of FB on the Abduction or Scam group. I have gone on to my lappy to send it as I am clueless on tablet. Maybe Steve will have saved more detailed grabs. If you are on FB you could message him. No more comments are allowed on the thread I posted from so would have to be a pm which he may not see. Don`t know if he is on Twitter.

Resistor wrote:
What we would really need to see are the URLs where the red hyperlinks are beneath each "headline".

The news articles appear to be genuine press releases. Here's two of them from a later archive
http://web.archive.org/web/20071012133130/http://ceop.gov.uk/news_items/article_20071002_ceop.htm

http://web.archive.org/web/20071012133059/http://ceop.gov.uk/news_items/article_20071001_ceop.htm

What would be really really interesting is what clicking those links brings up. If the page loads from 20070430115803 then that proves that the date of 30/04/2007 is wrong. If it loads from e.g. 20071012133130 then 30/04/2007 is correct and the retrieval/replay is wrong

Resistor wrote:

Nuala wrote:As I said in an earlier post, there is software you can buy to run through a server that specifically seeks out orphaned pages. So clearly it is possible to find them.

That's for running on your own server though, you can't run it on anyone else's.

And you think that Government agencies and other bodies like that won't use it?

Can you tell me how that relates to Wayback, I don't get the connection. Thanks.

chirpyinsect wrote:Resistor. Glad I didn`t imagine seeing that page. I found it not in my files but from Steve Marsden`s post of FB on the Abduction or Scam group. I have gone on to my lappy to send it as I am clueless on tablet. Maybe Steve will have saved more detailed grabs. If you are on FB you could message him. No more comments are allowed on the thread I posted from so would have to be a pm which he may not see. Don`t know if he is on Twitter.

Steve Marsden is not on twitter.. He is doing further research on the matter and will make a post when his work is complete hopefully later on today.. He will discuss it on our facebook page

https://www.facebook.com/groups/jkjoannekilcoyne/

Their outfit cannot be cheap to run and maintain. Who funds them? I can't see where it says it anywhere on their site. And their archives are used in both civil and criminal cases. I'm not so naivé to think that some Government, somewhere, doesn't have a vested interest.

AndyB wrote:
No, that misses the point. You've come on here and started telling people how all crawlers work. I just want to know what your experience is that allows you to authoritatively tell people how all crawlers work. What is your technical expertise please? That will allow me to make an informed decsion about whether I should attach weight to your opinion, ignore you completely, or something inbetween

Somebody on here seems to think that some crawlers can use referal logs.

http://webmasters.stackexchange.com/questions/21499/is-it-possible-for-web-crawlers-to-see-static-pages-without-following-a-link-to

Resistor wrote:Their outfit cannot be cheap to run and maintain.  Who funds them?  I can't see where it says it anywhere on their site.  And their archives are used in both civil and criminal cases.  I'm not so naivé to think that some Government, somewhere, doesn't have a vested interest.

As a complete aside, is GCHQ likely to run something similar to Wayback ?

Here's some of the source code from the first link that Andy B posted, the one about the wanted offender. The page has the date 2 October 2007 hard coded in, no javascript generating it dynamically, or anything like that.

Yet according to Wayback, it was also crawled on 30 April 2007! And it has also been removed from the archive.
It is interesting that it does not have a specific file name, like mccann.html, just a number - article_20071002_ceop.htm. Obviously the convention is just to name news articles by their date (what happens if two important things happen on the same day?)
This makes it possible to change the contents but keep the same file name.
I know it's a bit far fetched. I will have to have a think about this one.

Resistor wrote:

AndyB wrote:
No, that misses the point. You've come on here and started telling people how all crawlers work. I just want to know what your experience is that allows you to authoritatively tell people how all crawlers work. What is your technical expertise please? That will allow me to make an informed decsion about whether I should attach weight to your opinion, ignore you completely, or something inbetween

Somebody on here seems to think that some crawlers can use referal logs.

http://webmasters.stackexchange.com/questions/21499/is-it-possible-for-web-crawlers-to-see-static-pages-without-following-a-link-to

The person in question said this:

Some sites seem to provide access to those logs without any access restriction, thus crawlers can reach them too...

So that's something that has to be done pro-actively. Logs are generally private.

If what you're suggesting is that there happened to be a website on the Internet in April 2007 that had opened up its logs and had also happened to have a link from mccann.html in those logs, then honestly I'd have to say that it's more likely that mccann.html was momentarily linked to on the CEOP website the moment the Wayback crawler visited.

In other words, it starts to get to "it was captured by aliens" unless some common sense is applied.

I'm not suggesting anything. I'm just showing the opinion of somebody else. I genuinely want an answer to this, and nobody seems to have one.

And sorry, "that can't be right because it's just common sense" just doesn't cut it. Not now.

froggy wrote:

Resistor wrote:Their outfit cannot be cheap to run and maintain.  Who funds them?  I can't see where it says it anywhere on their site.  And their archives are used in both civil and criminal cases.  I'm not so naivé to think that some Government, somewhere, doesn't have a vested interest.

As a complete aside, is GCHQ likely to run something similar to Wayback ?

I'd be bloody astonished if they didn't.

Resistor wrote:I'm not suggesting anything.  I'm just showing the opinion of somebody else.  I genuinely want an answer to this, and nobody seems to have one.

And sorry, "that can't be right because it's just common sense" just doesn't cut it.  Not now.

Okey dokey I'll leave you to it  

I want an answer too, I want an answer that I can fully understand as why that CEOP page was archived with a date of 30th April 2007, but you will note that I haven't discussed that yet.

The reason I haven't discussed it is because it will go round in circles until people accept that the page had to be live to be indexed, otherwise the debate starts from a false premise and so therefore can't come to a correct conclusion.

So as I say, I'll leave you to it, and will go and do my own investigating as regards the date.

I don't see anyone on here arguing that the page wasn't live to be indexed.
And if you read from the very start of the thread (I know it's quite long now) you'll see plenty of arguments as to why the date is correct.

As a complete non-techie, I consider the default position to be that the date is correct, until there is undisputed evidence that it is wrong.

froggy wrote:As a complete non-techie, I  consider the default position to be that the date is correct, until there is undisputed evidence that it is wrong.

Me too Froggy given that the Wayback FAQs specifically answer the question of how come future stories are relayed onto previous dates.

The WB seemed to be working correctly at 12.09.52 on 30 Apr 2007 for a capture of Youtube.

http://web.archive.org/web/20070430120952/https://www.youtube.com/

Nuala wrote:until people accept that the page had to be live to be indexed, otherwise the debate starts from a false premise and so therefore can't come to a correct conclusion.

Its very difficult for people to accept a page has to be live to be indexed when there is evidence that that isn't necessarily the case and the person asserting it refuses to both engage with anyone who gainsays them or offer any evidence of any qualification or experience to justify their insistence that this is always the case for all crawlers everywhere.

Does this not answer our question, if not why not?

How did I end up on the live version of a site? or I clicked on X date, but now I am on Y date, how is that possible?

Not every date for every site archived is 100% complete. When you are surfing an incomplete archived site the Wayback Machine will grab the closest available date to the one you are in for the links that are missing. In the event that we do not have the link archived at all, the Wayback Machine will look for the link on the live web and grab it if available. Pay attention to the date code embedded in the archived url. This is the list of numbers in the middle; it translates as yyyymmddhhmmss. For example in this url http://web.archive.org/web/20000229123340/http://www.yahoo.com/ the date the site was crawled was Feb 29, 2000 at 12:33 and 40 seconds.

You can see a listing of the dates of the specific URL by replacing the date code with an asterisk (*), ie: http://web.archive.org/*/www.yoursite.com

Whatever archives we have are viewable in the Wayback Machine. Please note that there is a 6 - 14 month lag time between the date a site is crawled and the date it appears in the Wayback Machine.

chirpyinsect wrote:The WB seemed to be working correctly at 12.09.52 on 30 Apr 2007 for a capture of Youtube.

http://web.archive.org/web/20070430120952/https://www.youtube.com/

I think both sides of the debate accept that everything worked correctly on 30/04/2007. The issue is whether the CEOP archive really did get archived on that day or whether it was actually archived in late October and misfiled as 30/04/2007

AndyB wrote:

chirpyinsect wrote:The WB seemed to be working correctly at 12.09.52 on 30 Apr 2007 for a capture of Youtube.

http://web.archive.org/web/20070430120952/https://www.youtube.com/

I think both sides of the debate accept that everything worked correctly on 30/04/2007. The issue is whether the CEOP archive really did get archived on that day or whether it was actually archived in late October and misfiled as 30/04/2007

In which case there will need to be some proper evidence to that effect and not just an opinion that this is what must have happened.