search-engines on S Anand

Things I Learned - 23 Feb 2025

Sun, 23 Feb 2025 00:00:00 +0000

This week, I learned:

Remote Desktop may be the easiest way to have a Windows machine access files / screen from another Windows machine, even for home PCs.
Caddy sets up reverse proxies that get automatic SSL certificates from Let’s Encrypt!
The Nomic Embed v2 blog post has an excellent visualization for embedding quality. It takes all Wikipedia disambiguation articles and shows them on a Nomic Atlas, embedded via Nomic Embed v2. It lets you toggle to OpenAI text-ada-002 which moves the topics far away. Visually, this is very convincing.
Python 3.15 will enable UTF-8 mode by default. PEP 686
Python 3.13 supports sub-interpreters to bypass the GIL. It’s quite like web workers. PEP 554
The quickest way to change the fish prompt is function fish_prompt; echo '> '; end
At PyConf Hyderabad, about 3 people had read a PEP. 1 had used the match operator. But 80% knew what a Vector DB was. 20% had used a Gemini API. That’s how much traction LLM development is getting.
The productivity benefit people report from using LLms is about 3X. Ethan Mollick
Soon, you’ll be able to send an LLM to a virtual meeting on your behalf. It will talk like you. Ethan Mollick
Models tend to claim ignorance when you test them on topics they should avoid. But tend to answer when not being tested. Sneaky! Ethan Mollick
Mermaid has an Architecture Diagrams Syntax (in beta) that’s capable of creating elegant architecture diagrams with icons.
Blind is an app that allows users to post anonymously. It’s particularly useful to find honest negative feedback about (mostly US) companies.
Iconify.design is a single npm interface to most open source icon sets. It includes FontAwesome, Bootstrap, Material Design, and many others. icones.js.org is an alternate interface.
Self-pity may have evolved as a signal for social support and reducing conflict, while also encouraging self-reflection and behavioral adjustment. But in modern contexts it may be maladaptive and lead to depression. ChatGPT
Anecdotally, Grok 3 is very good for researching company information and latest news, particularly employee and customer sentiment. DeepSeek and Claude write more humanely than OpenAI. via Alberto Lopez Toledo, White Star Capital
There’s a YCombinator Founder Directory listing all founders of YC companies. At the moment, there are 8,628 founders. There’s also a co-founder matching tool.
LLMs are impacting not just data queries but geospatial queries as well. Here’s a good example of Natural Language Geocoding.
US companies typically pay employees every 2 weeks not every month.
What’s good about Snowflake? A few developers who explored it mentioned that:
- Its ability to scale up compute automatically makes queries run faster.
- “Time travel” allows you to see how data looked at any point in time and that is impressive and useful.
- Live data sharing with access control without the need for ETL pipelines is useful.
- Open-source competition: ClickHouse, Apache Druid, and Presto/Trino
- DataBricks is a lakehouse and less a data warehouse. It’s more about:
  - storing unstructured data (Snowflake prefers semi-structured: JSON, Avro, etc.)
  - running collaborative notebooks in Python, SQL, Scala, R (Snowflake encourages SQL)
I subscribed to ChatGPT Pro mainly for DeepResearch. Here are the first 50 reports I generated:
1. uv Package Manager Overview
2. DuckDB Analytics Comparison
3. Rust vs Python / JavaScript
4. Modern Data Engineering Course
5. LLM Code Migration Practices
6. Cloud Cost Optimization Strategies
7. LLM Coding Interview Tools Report (compare with Perplexity)
8. Text To Speech Engines
9. Customer Service in Indian Public Sector Banks
10. LLMs in Software Development
  - Old version 1: Gen AI in Software Development
  - Old version 2: Gen AI in Software Development
11. Leadership Training Content
12. Open-Source HTTP Servers. Caddy wins.
13. Deep Research Use Cases
14. Nagpur No-Parking Violations
15. Data Science in Food Services
16. Deep Research Disruption to Research Firms
17. LLMs in Design Thinking
18. EU Taxonomy Report Clarification
19. Shell Valuation Analysis Inquiry
20. LLMs in DSLs Research
21. Public API-Based Data Storage Options. Supabase wins.
22. Front-End JS Frameworks Analysis
23. Database Evaluation Guide
24. CSS Frameworks Evaluation Guide
25. CI/CD Tooling Ecosystem Report
26. Color Names Count
27. S Anand Biography. Meh, I know more about me, and it gets a few things wrong.
28. Cosmere Secrets Encyclopedia. This is the best. Deep Research is great if it’s stuff I actually want to read, rather than just learn about.
29. DBT course
30. Future of Coding AI
31. Claude Artifacts Use Cases. This is the only one that managed to get artifacts links correct. I used this for an article for The Hindu.
32. MCP Servers and Clients Research. Learnings:
  - Practically any “tool” can be an MCP server: file systems, APIs, codebases, browsers, collaboration platforms, memory, etc.
  - Most platforms have (or are) integrating MCP.
  - Clients: code editors, chat, and automation tools support MCP. GenAIScript is a good starting point.
  - Tester MCP Client is a browser-based test environment.
  - mcp-cli-client is a CLI-based client
  - mcp-chatbot is a chatbot client
33. Data Moats by Industry
34. Attorney Profile Research
35. Social Media Data APIs
36. Adobe Software Alternatives
37. LLM Hallucination Visualization Techniques
38. API vs Self-hosting Cost Analysis: Always use APIs, avoid self-hosting models.
39. AGI Preparation
  - AGI will emerge step by step. Knowing which step is next will help
  - AI native organisations will emerge in each of these areas. AI design agencies and AI creative Agencies being one example
  - Networking, empathy, leadership have more value now. So will human AI bridging roles (e.g. AI managers, AI consultants, ethics auditors)
  - What’s the value of a human when technology can do everything better? How did this play out in drama (decay) or sports (centralization) or music (globalization)?
40. Modern digital note taking
  - Voice note taking is the game changer
  - Automatically popping of notes based on context such as people places or conversations will be a thing
41. Local LLM Search Tools
42. Blog Post to research paper on copying - suggestions
43. Linux Dev Migration Guide
44. Raspberry Pi SIM options
45. Linux Dev migration guide
46. HTML to JATS conversion
47. LLM context splitting strategies
48. Strategy for AI services in Publishing
49. Gemini multi model editing use cases by industry
50. Pharma Conference Participation Guide
I learnt what a Memoji is for the first time. An avatar that follows your facial expressions. Cool!
Google shows US flight timings from FlightView. Emperically, based on one data point (my UA-2168 which was delayed by 4 hours), it gets updates faster than Flight Radar 24 or FlightAware or FlightStats.
When comparing Indian graduates with their western counterparts, the Indian ones are often seen as:
- 🟢 Theoretically sound
- 🟢 Analytical & technical
- 🟢 Academically disciplined
- 🟢 Resilient under pressure
- 🟢 Committed continuous learners
- 🔴 Rote-learning oriented
- 🔴 Limited independent inquiry
- 🔴 Limited creative innovation
- 🔴 Restricted practical exposure
- 🔴 Poor communicators
- 🔴 Low leadership / initiative
- 🔴 Need structured guidance
- 🔴 Struggle to network
HuggingFace has a “Model tree” against each model that shows the model’s ancestors and descendants. For example, as of now, Deepseek R1 has 75 adapters, 154 finetunes, and 23 quantizations.
Perplexity is now powered by Cerebras, which makes their inference as fast as Google. Source. The speed is a big factor, and I’ve switched my default search engine from Google to Perplexity, at least for now.
Interview Coder is a desktop app that offers live interview support for coding interviews. It’s a transparent window that reads your screen and answers questions for you. (Given this, I think we need an interviewer support system that tells interviewers what to ask!)

The Calvin and Hobbes search Takedown

Fri, 21 May 2010 10:53:27 +0000

Eight years ago, I started typing out each of the Calvin and Hobbes strips by hand. Four years ago, I set up a site that let people search for strips. Early this month, I was asked to take it down.

This is the story.

I can’t quite remember when I started reading Calvin & Hobbes. The earliest reference I can find in my blogs is in July 1999. I remember it didn’t take me long to become a fan. I’d read every strip on the newspaper; hunt them out at bookshops; and spend a fair bit of time searching for archives online.

At some point, I discovered a few archives of the complete Calvin & Hobbes images. These aren’t hard to find, and they’re still around in plenty. So that gave me a few more months of delight.

The trouble, though, was that I never could quite find a strip when I wanted to. A friend would refuse to accept something, and I’d want to pull out that strip where Calvin declares to reside in the state of “Denial”. Or if they said something fancy, I’d want to pull out the one where Hobbes says “I notice your oeuvre is monochromatic”. Or those strips where Calvin’s Dad explains how things work (“They build bigger and bigger trucks over the bridge until it breaks.”)

There were a few Calvin and Hobbes search engines around. None quite did what I wanted them to – which was to search the text, and show me the strip, with a nice scrollable interface.

So I set out to build one. I can’t remember when, exactly, but it was before Sep 11, 2002.

It took me many years. I’d spend several train rides and evenings typing this stuff out. My friends, employers and family were a bit puzzled, but just added it to my list of eccentricities and carried on. I was halfway there in 2005, pushed further in 2006, and with some help, I managed to finally complete it.

I was able to do a lot of cool stuff with this, like statistically improbable phrases and some amusing posts as well.

It also increased traffic to my site, which was a bit disconcerting. I didn’t want to attract attention. In 2007, I removed the page from Google’s indexes, which cut the number of hits a fair bit. Since then, the site was only visited by a few people that knew of it, and the occasional stumblers.

A month ago, I got reddit-ed and MetaFiltered.

It didn’t take me long to figure that a takedown notice would be on its way. It turned out to be quite a friendly mail, actually – scary only in parts. (A bit of a carrot-and-stick approach, perhaps.) Anyway, it took me all of 2 minutes to remove all of the pages and links.

Of course, the reason I went to all of this effort was because the original Calvin & Hobbes site does not have the search feature. I’ve reached out to United Media, offering my transcripts and code. Let’s see what happens.

Comments

Pradeep 29 Jul 2010 8:25 am: what a shame. how are u doing buddy? been some time…
Frank 23 Jun 2010 8:03 pm: Where there’s a will, there’s a way. Using two sites to do the work of one is not the ideal… but for now here is a way to find those strips you have in mind or are trying to search by topic or whatever. The first is a complete collection of all of the actual strips on the following site: http://www.marcellosendos.ch/comics/ch/ The cool thing about this site is that you can actually download the entire collection in a zipped file (105mb) if you prefer to have the strips stored on your computer, and it opens with your web browser. The following site will allow you to search for key words: http://www.reemst.com/calvin_and_hobbes/stripsearch It used to have the actual strips posted as well, but alas has had the same woes as just about everyone else. One reason I think that the scripts have been allowed to stay up is that they are no the actual verbatim texts from the strips… it is a commentary on each strip… so essentially it has all of the key words you might want to search for and extra, in the sense that it is a narrative of what is happening frame by frame, including the dialogue in the 3rd person… so the interesting thing is that you may be able to find topics through the narrative as well, not just the script/ speech bubbles. So anyways, what you do is pop in the topic or words you are looking for in the second link I posted… find what date it was published on… and then go along to the first link I posted - and you will be able to search by year and month. Each month that opens has 28-31 “batches” of strips for that particular month. Just count down to the day you are looking for as you scroll down. This may be a long-winded explanation, but it’s just 2-3 steps instead of one, which is what I’m going to make do with until something more convenient crops up. I have tried the above search method, and surprisingly gives me the same results… if not more options. Check it out for yourself and you’ll know what I’m talking about. A suggestion… since none of the material posted on the above sites is wholly property of the site owner, perhaps someone with the know-how could combine the commentaries and the strips (With a lot of manual copy-pasting however) to end up with a single downloadable PDF or Word document where anyone could easily do a word/strip search by simply using the Adobe reader/ Word search functions. Anyways, that’s my five cents for now. Abraços Frank
Ravi Atluri 13 Aug 2010 6:50 pm: http://michaelyingling.com/random/calvin_and_hobbes/ Why did they shut down yours?
Kannan 21 May 2010 12:12 pm: Moral of the story : If you do something good, just keep it to yourself or your elite group who will appreciate it’s usefulness. Over reaching can be bad in “free world”. That said, it was indeed a really really good page :)
Srihari 21 May 2010 1:23 pm: Been reading Calvin and Hobbes from your site for quite a long time. Time and again I would try and recollect a particular strip, and the search feature made it very comfortable. Sometimes I would just key in a particular idea, and search for it, and more often than not, Calvin would have something or the other to say about it! Shame it had to be taken down :( All the same, thanks to your efforts, we all were so many smiles richer! :)
Arvind 21 May 2010 4:40 pm: Hey Anand! First of all,hi-5! I’m a huge calvin n hobbes fan n I started late too, pretty much like u! But yea, left no stones unturned in devouring whatever C&H content I can set my eyes or lay my hands on!! :) Ohh this is nothing in relation to this post but I’m totally hooked onto ur blog for a while,now…So much so it’s hampering my productivity at work place(A tobacco firm…No prizes for guessing which 1)…I’m totally lagging behind schedules n deadlines but for the moment it just feels like there’s nothing which can eclipse the joy of readind your posts! :) Ohh n I coudln’t help noticing a few commonalities between us! Gives a mere mortal like me to identify and associate with the almighty Himself, u c…I’m a Tam bram too, from chennai…Name starts with A and ends with D…Teetotaler…Engineer + MBA…Though I cudnt make it to the IITs, guess i’m just a notch behind, having graduated from NIT, Trichy…Followed by a coupla years work-ex…And MBA from NITIE…Can relate to all ur pranks from undergrad n workplace cliches! No wonder I’m just addicted to the content on ur blog… Wish I cud meet u in person, sometime! Thanks for writing, Anand…Keep up the good(no, absoultely fantastic) work! :)
408wij 21 May 2010 11:34 pm: I suggest mapping your database to the comics at a legit site (gocomics.com). I.e., you host the search, but they host the comics.
Deepak 23 May 2010 7:18 am: Anand, Time to come out of hiding! I think I got to your site coz I used to work for your current company. Your CnH site was lovely - I pulled out some gems through that search myself. Sorry and sad you had to take it town. Hope all that work you put in transcripting goes to good use!
George 24 May 2010 1:39 am: I was one of those people that ‘stumbled’ across your C&H site a couple years ago. The searchability was brilliant. Good luck on transferring it to the other site!
evans 25 May 2010 4:18 am: I’m a teacher and would frequent your site to pull relevent comics to delight my kids. It takes way too much time for me to flip through my books to find what I am looking for, so your site was a godsend. I’m sad to see it down, and I hope that your work can live on somewhere else. Right now I’m trying to find a good luck strip to put on my final exam… really missing your site.
Andreas Schamanek 25 May 2010 10:02 am: First of all, thank you very much indeed for all the efforts, the typing, coding, and keeping it up for so long. It’s a pity it is gone. I was reading the strips of gocomics on a daily basis but just like you write when I was looking for a particular quote I had your search engine bookmarked. I had no idea you had typed all the strips yourself. This was utterly amazing work. Thanks again.
Bjarne 12 Jul 2010 3:14 pm: What a shame. For the last 6 years this site has helped me digging down in strips to find the one which did just fit in the situation. Hope United Media will take over the job :-) Thanks for the good work from Denmark.
» very sad RT @sanand0 The Calvi… Thej Live 15 Jun 2010 7:38 pm (pingback): […] sad RT @sanand0 The Calvin and Hobbes search Takedown http://www.s-anand.net/blog/the-calvin-and-hobbes-search-takedown/ […]
Graham 8 Jun 2010 12:51 pm: I can’t remember when i first stumbled apon your Calvin and Hobbes site but it was many moons ago. I’d just like to say a massive thanks really. I’d spent years laughing at the books while I was growing up and your efforts enabled me to instantly find all those excellent strips I’d remembered. All the best Graham:-)
Thejesh GN 15 Jun 2010 7:03 pm: sad :(
Naveen Arur 16 Jun 2010 1:02 pm: Really sad man. I am such a huge fan of C&H. Hope United Media comes to their senses and make use of your hard work.
Li 29 Oct 2010 6:43 pm: Anand, thanks for your great efforts. I stumbled across your Calvin & Hobbes site years ago and loved it, but I always knew it wouldn’t stay around forever. Alas, it was fun while it lasted. Ravi Atluri, thanks for posting on 8/13/10. I’m sure that’ll be taken down someday too. I’ll keep it to myself.
Hope 2 Dec 2010 10:19 am: I REALLY miss your website. :( I’d use it all the time, cuz i find myself in the exact situations as you, where I can come up with a calvin and hobbes strip for almost every situation in life and in most conversations.
x-f 7 Dec 2010 7:16 pm: Hi! I too used the search feature for a couple of years. It was sooo great I could instantly find the right strip I was looking for! Sad you had to remove it all. A very big thank you for your efforts writing it all down by hand! Hope it comes back one day.
Prashanth 14 Feb 2011 7:36 pm: It’s really sad that you had to take down the Calvin & Hobbes archives . I used to come to this site often to look up Calvin strips - for example the one where he goes into a dimension where time has no meaning! Came in today to look for strips where Calvin’s father tells him to eat his vegetables because it builds character and discovered this page. Anyway, just thought I would tell you - thank you for all your efforts!
BabyGodfather 15 Jan 2012 7:19 am: Man, this is really sad that you had to take it down. I had the exact same problems about searching the right Calvin’s strips at the right moment as you. Maybe only you know my annoyance. Is there anything that we could do ? I mean, I don’t think it needed to be taken down. They didn’t have any problems people posting Calvin strips in their blogs, did they ? Are they taking down all the pages that include Calvin Strips ? It’s ridiculous.
Visualising Text | Gramener blog 28 Jul 2012 5:01 am (pingback): […] to the joys that are lawyrs, @sanand0 was asked to take the Calvin & Hobbes text archive down. http://www.s-anand.net/blog/the-calvin-and-hobbes-search-takedown/ […]
From Calvin & Hobbes to Photo Tagging: Excel's Unexpected Image Capability - S Anand 26 Mar 2024 8:55 am (pingback): […] This is how I transcribed every Calvin & Hobbes. […]

The Bing effect

Fri, 12 Jun 2009 19:14:01 +0000

This graph is the number of referrals Microsoft’s new search engine, Bing, sent to my site over the last few days. Looks like the hype is dying out. Though Bing did leapfrog Yahoo briefly, that lasted just one day.

15 years of Dilbert searchable

Fri, 22 May 2009 07:17:47 +0000

The Dilbert search index now carries 15 years worth of Dilbert comics — over 5,500 strips typed out. This is mainly due to the contributions of BFMartin (over 6 years worth of strips) and Paul Dorman (over 3 years worth of strips), myself (over 3 years worth of strips) and a long tail of contributors.

You can search the strips here. While you can find strips as far back as 1989, you won’t see the images earlier than 2002 because geek.nl (whose images I’m shamelessly hotlinking without permission) only holds images that far back. But once you know the date of the comic (say 1991-02-03), you can visit the Dilbert official site at dilbert.com/1991-02-03/ and see the strip.

Dilbert started around 20 years ago. So we’ve covered 75% of all the strips, and this is in just 8 months after starting this collaborative effort. A couple of lessons I learnt from this:

Crowd sourcing beats going solo if you’re building content. It’s a no-brainer. There will always be only one or two people more passionate about something than you.
The long tail is not very big. There will only be one or two people more passionate than something about you. Don’t expect the long tail contribution to be the significant. The value comes from being able to attract “the big fish”.

Comments

Thejesh GN 27 May 2009 7:57 am: Thanks for the search engine.

Dilbert search statistics

Fri, 10 Oct 2008 12:00:00 +0000

It’s been three weeks since I initiated the effort to type in the Dilbert strips and the results are encouraging. About 2 years worth of strips have been typed out. So this Dilbert viewer now has a reasonably sized index for searching.

Many thanks are in order here. The first is due to geek.nl, whose images I have taken the liberty of hotlinking. Thanks also to those who’ve taken the time out to type strips:

granger95
bthangaraj
gdibyo
adrienbernard
sundar.ramakrishnan
pistohl
waywardone
balamurugan.cse
sruppenthal

… and several others.

When I initially planned to share the typing of the Dilbert strips, I anticipated that I would probably type in the most, and almost no one would pitch in. While I still have typed in the most, the contributions of the above have been of great help in more than the obvious way. When I typed out 10 years of Calvin & Hobbes, it took me 5 years. This is 2 years of Dilbert in 3 weeks. If nothing else, it’s pushing me to work harder on this. So thanks again for the motivation.

Here’s my request again to all you Dilbert fans.

Please go to dilbert-search.appspot.com
Log in using your Google account and type in as many strips as you like
Bookmark it for the future, whenever you’re bored

Here’s a Wordle cloud of all the strips typed out so far (with Dilbert and Pointy Haired Boss removed.)

(Seeing that there’s more “good” than “bad”, and more “like” than “dislike” or “hate”, you might even call Dilbert an optimistic strip.)

Comments

Dibyo 21 Oct 2008 8:10 am: You’ve posted this again? Each time you post this, I do atleast one strip!
S Anand 25 Oct 2008 6:18 am: Hmm… don’t know why that happened. I didn’t realise it appeared again on
Google Reader until yesterday. Maybe I just changed a word or so. Well, all
the better :-)
Anand
Elleana 25 Oct 2008 11:22 am: How should “Pointy Haired Boss” be capitalized?
S Anand 25 Oct 2008 11:25 am: I’m not sure, Elleana. I use “Pointy haired boss” consistently. I think
you’ve used “Pointy Haired Boss” consistently. Since the search is
ultimately case-insensitive, I guess it’s OK either way.
And thanks for all your effort!

Anand
ray 30 Oct 2008 8:06 am: Hi,

I was reading ur blog posts and found some of them to be very good.. u write well.. Why don’t you popularize it more.. ur posts on ur blog took my particular attention as some of them are interesting topics of mine too;

BTW I help out some ex-IIMA guys who with another batch mate run www.rambhai.com where you can post links to your most loved blog-posts. Rambhai was the chaiwala at IIMA and it is a site where users can themselves share links to blog posts etc and other can find and vote on them. The best make it to the homepage!

This way you can reach out to rambhai readers some of whom could become your ardent fans.. who knows.. :)

Cheers,

Dilbert search engine

Thu, 18 Sep 2008 12:00:00 +0000

UPDATE: 13 Jan 2026: Scott Adams passed away. RIP.

UPDATE: Mar 2023: Dilbert.com was closed but archives are accessible via the Wayback Machine (slow). Search does not work well. Dilbert viewer is an alternate interface via Reddit.

UPDATE: 2012: dilbert-search.appspot.com died, likely of old age.

–

Wouldn’t it be cool to be able to search through the Dilbert archives using text?

This used to be possible at Dilbert.com some years ago, as a paid service. In late 2003, I needed to find some Dilbert strips for a client, so I’d subscribed for a year. I could then search for the quotes (I happened to be looking for “outsourcing”, so you can guess the context).

But I can’t seem to find the feature any more, even as a paid service. The site looks a lot better, of course. But I can’t find strips.

Well, why not type them out? After all, I’d done that with Calvin and Hobbes.

This would be a much larger exercise, though. And I’m hoping to take your help. I’ve set up a site at dilbert-search.appspot.com. You can type in a comic randomly, starting from 2000. These will be made searchable on my Dilbert page. You can export the data and use it yourself, of course.

When typing in Calvin and Hobbes, I did have a few volunteers willing to pitch in, but collaboration tools weren’t easy to set up, and I ended up typing the whole thing myself. This time, I’d be delighted if even 10 people typed in just a strip each.

So, here’s my request, to all you Dilbert fans.

Please go to dilbert-search.appspot.com
Log in using your Google account and type in as many strips as you like
Bookmark it for the future, whenever you’re bored

As I said, the data is readily exportable from the page, so if you’re looking to do cool mash-ups with it, great! And if you want the data exported in other formats, please let me know.

Incidentally, I created the site using Google AppEngine. The source code is at dilbert-search.googlecode.com.

Comments

Dibyo 18 Sep 2008 11:50 am: Stud: I suggest that you add an option of putting tags on the comic strip apart from just the transcribed script. It might be useful while searching. I’ll contribute in the meanwhile.
S Anand 18 Sep 2008 5:10 pm: Thanks, Dibyo.

Regarding the tags, since full text search will be supported, you could just add any additional tags in brackets at the bottom. I don’t want to complicate the data structure just yet.
Vanaja Sarma 19 Sep 2008 12:47 am: Hey Anand

How about using these links…the first to search by keyword and the second for the actual strip using date

http://www.bfmartin.ca/finder/index.php?page=home
http://dribibu.xs4all.nl/dilbert.html
S Anand 19 Sep 2008 1:07 am: @Vanaja: I’ve used the Dilbert strip finder before, and it’s great. But it doesn’t seem to be updated, and it doesn’t have the actual words in the comic, which is what I’m hoping to correct.
Thangaraj 25 Sep 2008 2:42 pm: I found a list of characters with image in the URL below. Guess you could link it to this in the typing page since its easier to check which character it is.

http://www.unitedmedia.com/comics/dilbert/the_characters/html/character2.html
Elleana 25 Sep 2008 11:34 pm: I can’t wait until we’re done with the Dilbert database. I actually found your site a while ago because I was searching for a Calvin and Hobbes strip.
S Anand 26 Sep 2008 1:32 am: @Thangaraj: Thanks. Will link to it.
@Elleana: Me too! We’re done with about a year’s worth of work now. Hope to have everything done by end of 2008.
Marty 10 Feb 2009 11:13 am: I’ve been using Dilbert search engines to find a specific cartoon for a while now. I read it in one of the books. I recall it being a square book with four frames per page. A woman is telling a man about how her husband mooned a zoo animal and was killed by it. The man in the frame is trying not to laugh so hard that his heart blows out his back. Can anyone help me find this cartoon? I’d love to see it again.
CCurly 24 Feb 2009 9:39 pm: This list (http://john.ccac.rwth-aachen.de:8000/ftp/dilbert/dilbert.txt) is almost complete, but for years earlier than 1995 the index is not a date but a reference to a Dilbert book. I see some spelling errors also, and it’s missing the “who said it” part. Perhaps it’s usable as a way to get some content, and you could let people check the text in stead of having them type it. Love the idea !!
S Anand 24 Feb 2009 11:09 pm: Thanks CCurly! This looks to be a useful resource. A lot of strips are incomplete, but it will definitely help accelerate the effort. I’ll see if I can parse it and get the data in.
Sid Shniad 20 Mar 2009 4:57 pm: I’m looking for the following Dilbert cartoon, but had no luck on your site. Is there any chance you could find it and send it my way? Thanks. (It ran in 2003, possibly August 9.) Sid Shniad dilbert: i have some disturbing news. dilbert: we outsourced our customer-service function to india a few years ago. pointy haired boss: so? dilbert: apparently, they subcontracted the job to mexico. dilbert: then mexico sub-contracted to vietnam, who sub-contracted to the philippines… dilbert: …who subcontracted it to us. dilbert: it turns out that we’re the lowest cost provider, because we lie about our hold times. dilbert: in summary, we pay ourselves to hose ourselves. dilbert: are you thinking what i’m thinking? pointy haired boss: we should raise our prices?
S Anand 23 Mar 2009 7:46 am: You had the date bang-on. It’s at http://www.s-anand.net/dilbert.html#20030803
BlueBird 5 Apr 2009 1:39 am: I sent Scott Adams a suggestion and he made a strip from it, and like a dummy I put the paper in recycling before I saved the strip. Can you help me? It was about a co-worker who stood and talked to someone else, killing time at another co-worker’s expense, until it was time for him (1st guy) to either go to lunch or go home. Then he leaves and his victimn has to stay late to get his/her work finished. Since the boss doesn’t know from computers, the guy just stands and throws out computing terms in relative nonsense until he’s ready to clear out. I think it ran on a Thursday about a year, to two years ago. I’d love to find a copy of it. (He’s still doing it)I suggested to Mr. Adams that he make the character very fat, with suspenders, and I think he did. Thanks!
Paul 15 Apr 2009 6:40 pm: I have transcribed the text from 11-14-2003 to the present, including which character say it. You could probably reformat it to work for you. Also, on dilbert.com you can now view strips all the way back to the beginning. Let me know if you want me to email it to you or something.
adrian 7 Nov 2009 6:15 pm: Heya! I love the Dilbert search page. I made my own and typed a lot of strips, but deleted the database without a backup. Doh! Then I found yours. But, the page seems to have issues right now. It’s slow to view and usually crashes with an error. Please fix it so I can enter lots of strips!~ Also thanks for making the data easily downloadable. -a
Vanaja 20 Jul 2010 11:34 am: Looks like its updated… http://dilbert.com/blog/entry/strip_search/
TelDaMan 24 Mar 2012 10:43 pm: Hey Anand, this is really excellent work! About a year ago I started trawling for the text and strips and found some really good resources, including yours. Thanks!! My website http://dilbert.is-a-geek.com/ automatically updates every day with new strips. It also OCR’s the text on the strip and comes up a 80%+ guess on the words and then indexes them. Sadly I have to manually edit each strip to fix up the words to 100%. My next phase is to get visual recognisation going for the characters so it knows ‘who said what’. My final aim is to get “deep searching” setup. This should help find strips really easily (based on test, pictures, etc) as well as suggest other strips :-)

Attack of the bots

Sun, 31 Aug 2008 12:00:00 +0000

One out of every 5 hits to my site is from a bot.

I spent a fair bit of time this weekend analysing my log file for last month (which runs to gigabytes, and I ended up learning a few things about file system optimisation, but more on that later). 80% of the hits were from regular browsers. 20% were from robots. Here's a sample of the user-agents:

Mozilla/5.0 (compatible; Yahoo! Slurp; <a href="http://help.yahoo.com/help/us/ysearch/slurp)">http://help.yahoo.com/help/us/ysearch/slurp)</a>
Mozilla/5.0 (compatible; Googlebot/2.1; +<a href="http://www.google.com/bot.html)">http://www.google.com/bot.html)</a>
Mediapartners-Google
DotBot/1.0.1 (<a href="http://www.dotnetdotcom.org/#info">http://www.dotnetdotcom.org/#info</a>, crawler@dotnetdotcom.org)
Mozilla/5.0 (Twiceler-0.9 <a href="http://www.cuill.com/twiceler/robot.html)">http://www.cuill.com/twiceler/robot.html)</a>
msnbot/1.1 (+<a href="http://search.msn.com/msnbot.htm)">http://search.msn.com/msnbot.htm)</a>
FeedBurner/1.0 (<a href="http://www.FeedBurner.com)">http://www.FeedBurner.com)</a>
Mozilla/5.0 (compatible; attributor/1.13.2 +<a href="http://www.attributor.com)">http://www.attributor.com)</a>
WebAlta Crawler/2.0 (<a href="http://www.webalta.net/ru/about_webmaster.html)">http://www.webalta.net/ru/about_webmaster.html)</a> (Windows; U; Windows NT 5.1; ru-RU)
Yandex/1.01.001 (compatible; Win16; I)
...

You get the idea. The bulk of these are search engines. Over two-thirds of the bot requests were from Yahoo Slurp. Now, this struck me as weird. If I take the top 3 search engines that are sending traffic my way,

	Referral %	Crawl %
Google	90%	24%
Yahoo	6%	66%
Microsoft	3%	0.3%
Others	1%	9%

The search engine that sends me the most traffic is being reasonably conservative, while Yahoo is just eating up the bandwidth on my site. Actually, this shouldn't bother me too much. It's not taking up too much bandwidth, or even CPU usage, given that all the bots put together make up only 20% of my traffic. But somehow... it's sub-optimal. Inelegant, even.

So I decided to take a closer look. Just how often are they crawling my site?

Yahoo	*Every 5 seconds*
Google	Every 13 seconds
DotBot	Every 9 minutes
Cuill	Every 9 minutes
Microsoft	Every 18 minutes
Feedburner	Every 18 minutes
Attributor	Every 23 minutes
Yandex	Every 27 minutes

Look at those numbers. Yahoo is hitting my site once every 5 seconds. No wonder there's a help page at Yahoo titled How can I reduce the number of requests you make on my web site? I followed their advice and set the crawl-delay to 60, so at least it slows down to once a minute.

Just that one little line change should (hopefully) reduce the load on my site by around 15%.

As for the other engines, I don't mind that much in terms of load.

Google, for all that it crawls every 13 seconds, has faithfully reported that it has only 11% of my site under its index, so I've no idea what they're doing, but I'm not complaining about the traffic that's coming my way.
DotBot. Today was the first I'd heard of them. Visited the site, and smiled. These guys can do all the crawling of my site that they like, and I hope something interesting comes out of their work.
Cuill, sends me 0.2% of my traffic, but it's a new search engine, I'm happy to give it time.
Microsoft's OK, sends me a tiny stream of traffic.
Feedburner is just pinging my RSS feed every 18 minutes.
Attributor and Yandex I'm hearing of for the first time, again. Not too much load on a system, so that's OK.

What's amazing is the sheer number of bots out there. Last month, I counted over 600 distinct user-agent strings just representing bots. So it's true. The Web is no longer just for humans. We do need a Semantic Web.

Comments

Dhar 31 Aug 2008 9:09 pm: Hmmm, curious as to:
1. Why the bots should crawl your site every 5 seconds or so?
2. How you can find out how much of your site has been indexed by Google.
Cheers, D.
S Anand 1 Sep 2008 12:11 am:
1. I think Yahoo’s crawler is aggressive in any case. My site doesn’t seem to be an exception: there are a lot of threads discussing this problem.
2. Google’s webmaster tools tells you how many URLs have been indexed from your sitemap.

Top 10 bittorrent sites comparison

Mon, 25 Sep 2006 12:00:00 +0000

A comparison of the top 10 bittorrent sites. BTJunkie emerges the winner.

Comments

Dhar 27 Sep 2006 10:55 pm: Not convinced with the comparison. Looks like the chappie Renald is just using to blog to promote BTJunkie. Checked the features of BTJunkie, they are nowhere close to Torrentspy, yet it gets a rating of 9. No way anyone can confirm BTJunkies claims like Torrents Indexed, Torrents Added Daily etc. Neither does the author specify how he confirmed these points.

“At first I thought BTJunkie’s numbers must be fake, but I assure you it is real! I tested the number posted with the number in the actual directory for the day and they matched for a week straight!”

Yeah, sure!
S Anand 28 Sep 2006 6:47 am: Does sound like a plug, doesn’t it? :-) I’m planning to do a test myself. Why don’t you try one too?
Sumit Dhar 8 Oct 2006 12:54 pm: http://torrentfreak.com/the-largest-bittorrent-search-engine-on-the-web/
S Anand 8 Oct 2006 2:50 pm: A solid rebuttal. I tried BTJunkie last week. Wasn’t very impressed. And the few movies I couldn’t find anywhere else, I couldn’t find here either.

Krugle

Wed, 12 Jul 2006 12:00:00 +0000

Krugle is a code search engine.

Comments

sheikh 14 Dec 2006 3:08 pm: Thank you anand, It was very useful

Google meta search

Tue, 04 Jul 2006 12:00:00 +0000

What do you find when you search Google searches?

Comments

Sheikh 14 Dec 2006 3:09 pm: Thank You Anand. It was very useful

Bias among search engines

Tue, 16 May 2006 12:00:00 +0000

Search engines rank their own sites better. Yahoo Answers ranks higher on Yahoo, but not on MSN or Google. Google Answers ranks high on Google, but not on Yahoo or MSN.

This is bias, but not necessarily evil. They may just be fooling themselves.

Even scientists, the genuinely objective ones, do this. As Feynman points out in Cargo Cult Science:

Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It’s a little bit off because he had the incorrect value for the viscosity of air. It’s interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan’s, and the next one’s a little bit bigger than that, and the next one’s a little bit bigger than that, until finally they settle down to a number which is higher.

Why didn’t they discover the new number was higher right away? It’s a thing that scientists are ashamed of - this history - because it’s apparent that people did things like this: When they got a number that was too high above Millikan’s, they thought something must be wrong - and they would look for and find a reason why something might be wrong. When they got a number close to Millikan’s value they didn’t look so hard. And so they eliminated the numbers that were too far off, and did other things like that.

But, (and this is the important part):

We’ve learned those tricks nowadays, and now we don’t have that kind of a disease. But this long history of learning how to not fool ourselves–of having utter scientific integrity – is, I’m sorry to say, something that we haven’t specifically included in any particular course that I know of. We just hope you’ve caught on by osmosis.

It’s not easy to catch on, by osmosis or otherwise.

The Search

Sat, 15 Apr 2006 12:00:00 +0000

I was reading John Battelle’s The Search , and realised: We don’t sit down on the computer and say, “Let’s do a search”.

True. We want to get something done. We know it’s out there somewhere. We search.

So every search on a search engine is a commercial opportunity. Contrawise, every site must let people to do what they want to do on the site.

Think… What do people want to do when they’re on YOUR site?

Calvin and Hobbes Extensive Strip Search

Thu, 06 Apr 2006 12:00:00 +0000

Martijn’s Calvin and Hobbes Extensive Strip Search is back. It doesn’t let you search the quotes themselves, but a (pretty detailed) description of each cartoon instead. (Mine searches Calvin and Hobbes quotes).

Comments

Anonymous 21 Oct 2006 8:55 am: if you know calvin and hobbes u know how many are there. I have been looking for March 24, 1994 for an ages and this helped me to find it in a minute. great site I do recomend

New Google services

Wed, 22 Mar 2006 12:00:00 +0000

New from Google: Google Finance, and Google Mars.

Comments

Sathya 22 Mar 2006 8:30 am: Welcome Back !

Who is afraid of Google

Sat, 03 Dec 2005 12:00:00 +0000

Who’s afraid of Google? Everyone

Blind Search Engine Test

Mon, 28 Nov 2005 12:00:00 +0000

The Blind Search Engine Test. My vote turned out to be for Google.

MIT OpenCourseWare

Wed, 16 Nov 2005 12:00:00 +0000

MIT OpenCourseWare on Google Base.

Google Blogsearch

Wed, 14 Sep 2005 12:00:00 +0000

Google Blogsearch.

Comments

Ok 28 Sep 2005 7:29 am: Nice!

Technorati Blogfinder

Tue, 13 Sep 2005 12:00:00 +0000

Technorati Blogfinder tags blogs.

Google Sitemaps

Wed, 08 Jun 2005 12:00:00 +0000

Google Sitemaps.

…is an experiment in web crawling. Using Sitemaps to inform and direct our crawlers, we hope to expand our coverage of the web and improve the time to inclusion in our index. By placing a Sitemap-formatted file on your webserver, you enable our crawlers to find out what pages are present and which have recently changed, and to crawl your site accordingly.

Anil Dash links to alternate proposals. via Anders Jacobs