rss on S Anand

Things I Learned - 15 Dec 2024

Sun, 15 Dec 2024 00:00:00 +0000

This week, I learned:

**/*.md can search for all Markdown files. Julia Evans
Windows 11 2024 Update features: Ref
- Live captions (via the tray) can transcribe audio and microphone.
- Cocreator in Paint lets you draw crudely and enhances it with AI. The neat UI is a slider that lets you control how close it should be to your drawing.
- Voice Clarity automatically cancels echo, reduces background noise, and minimizes reverb.
- Studio Effects (via the tray) lets you apply camera effects on all apps. Eye contact feature is CLEVER!
- sudo lets you run commands with admin privileges from the command line. source
Roaming RAG is an alternative to RAG without the vector database.
- Applicable to well structured documents, e.g. technical books, manuals, etc.
- Create a hierarchical outline of the document. Code
  - Keep the top-level headings.
  - Preserve the first ~100 characters of opening text from each section.
  - Present the second-level headings, but without any subsidiary content.
  - Provide each section a unique 8 digit hex identifier.
  - Each section heading is followed by a guiding comment for the model: Section collapsed - expand with expand_section("{identifier}").
- Then read the relevant sections as context to answer the question. Code
Traffic to StackOverflow has fallen considerably. Especially from young and Indian developers. StackOverflow revenue is down. Via Prashanth. They’re exploring:
- Licensing their content. (Meta says high quality content improves LLM performance by 30% on HumanEval)
- Enterprise StackOverflow for system integration
- Fine-tuned versions of Enterprise Stackoverflow for enterprises
- Integrate StackOverflow within your IDE. Ask questions, post directly
I surveyed the Gramener QA team on how they were using LLMs.
- 7 used it for code generation (e.g. date extraction, regex generation)
- 4 used it for learning (e.g. Robot Framework, how to define test cases, API usage)
- 3 used it for formula generation (e.g. Excel)
- 2 used it for test scenario identification
- 2 used it for test data generation
- 2 used it for comparing expected vs actual datasets
- 1 used it for data type identification (e.g. given sample values, identify the data type).
- 1 used it for evaluating resulting (LLM as a judge)
I asked the Straive Digitalized Operations team what management techniques they would apply to manage LLMs. Here are the responses:
- Ask better questions. (Prompt engineering.)
- Create templates or step-by-step instructions. (Chain of Thought.)
- Ask for multiple options and pick from the best options. (Agentic approach?)
- Training. (Fine tuning.)
- Price weaker responses lower. (Stratified model pricing?)
“LLM hallucinations are a good thing. They are a sign of diversity, allowing us to improve the answer by exploring multiple paths.” – A colleague from Straive.
Hyperbrowser is a cloud based puppeteer service.
Bedrock Llama models can’t be directly called with their model names. You need to use their inference profile names, e.g. us.meta.llama3-2-11b-instruct-v1:0 if the model is in a US region.
Hacker News RSS is a good way to get RSS feeds from Hacker News. It’s also a good way to understand how to convert a news source into RSS feeds. BlueSky has RSS feeds too
When embedding using a SentenceTransformer.encode(docs) it’s best if we embed with smaller docs and call it multiple times (rather than embedding more at once). On Colab T4, for gte-base-en-v1.5, when embedding 1,000 docs of up to 8K chars each, here is the TOTAL time it took, based on batch sizes (lower is better)
- 1 doc per call: 10s
- 2 docs per call: 13s
- 4 docs per call: 19s
- 8 docs per call: 23s
- 16 docs per call: 32s
- 32 docs per call: 40s
Running embeddings without a GPU is extremely slow. It takes ~2.4 seconds per string.

Managing feed overload

Tue, 18 Dec 2007 12:00:00 +0000

I have only two problems with Google Reader.

The first is that it doesn’t support authenticated feeds. Ideally, I’d have liked to have a single reading list that combines my e-mail with newsfeeds. GMail offers RSS feeds of your e-mail. But the feeds require authentication (obviously) and Google Reader doesn’t support that right now. (So I usually don’t read e-mail :-)

The second is that it’s tough to manage large feeds. It’s a personal quirk, really. I like to read all entries. If there are 100, I read all 100. If there are 1000, I struggle but read all 1000. I’m too scared to “Mark all read” because there are some sources that I don’t want to miss.

The 80-20 rule is at work here. There are some prolific writers (like Scoble) who write many entries a day. There are some prolific sources (del.icio.us or digg). I can’t keep up with such writers / sources. I don’t particularly want to. If I missed one day of del.icio.us popular items, I’ll just read the next day’s.

With Google Reader, that makes me uneasy. I don’t like having 200 unread items. I don’t like to mark them all read.

In such cases, popurls’ approach is useful. It shows the top 15-30 entries of the popular sites as a snapshot. Any time you’re short of things to read, visit this. If you’re busy, don’t.

Using Google’s AJAX Feed API, it’s quite trivial to build your own feed reader. So I cloned popurls’ layout into my bookmarks page, and put in feeds that I like.

You can customise my bookmarks page to use your own feeds. Save the page, open it in Notepad, and look for existing feeds. They’ll look like this:

"hacker news" : {
    entries:15,
    url:"http://news.ycombinator.com/rss"
},

The first line (“hacker news”) is the title of the feed. You can call it what you want. Set entries to the number of feed entries you want to show. Set url to the RSS feed URL. Save it, and you have your own feed reader. (If you want to put it up on your site, you may want to change the Google API key.)

Try it! Just save this page and edit the feeds.

Here, I must point out three things about Google’s AJAX Feed API that make it extremely powerful.

The obvious is that is allows Javascript access to RSS in a very easy way. That makes it very easy to integrate with any web page.

The second is subtler – it includes historical entries. So even if an RSS feed had only 10 entries, I could pick up the last 100 or 1,000, as long as Google has known about the feed for long enough. This is what makes Google Reader more of a platform rather than a simple feed reader application. Google Reader is a feed archiver – not just a feed reader.

The third (I’m a bit crazy here) is that you can use it to schedule tasks. Google’s FeedFetcher refreshes feeds every 3 hours or so. If you want to do something every three hours (or some multiple thereof – say every 24 hours), you can write a program that does what you want, and subscribe to it’s output as a feed.

You may notice that I have a Referrers to s-anand.net on my bookmarks page. These are the sites that someone clicked on to visit my site. I have a PHP application that searches my access log for external referrers. Rather than visit that page every time, I just made an RSS feed out of it and subscribed to it. Every three hours or so, Google accesses the URL. I search my access.log and archives the latest results. So, even after my access.log is trimmed by the server, I have it all on Google Reader to catch up with later.

Since Google doesn’t forget to ping, I can schedule some fairly time-critical processes this way. For instance, if I wanted to download each Dilbert strip, every day as it arrives, I can create an application that downloads the file and returns a feed entry. Now, I don’t need to remember to run it every day! I just subscribe to the application on Google Reader, and Google will remind the application to run every 3 hours. (I know – I could use a crontab, but somehow, I like this.) Plus I would get the Dilbert strip on my feed reader as a bonus.

Update: Google has just released PartnerBar, which is a more flexible way of viewing a snapshot of feeds.

RSS feeds in Excel

Thu, 03 May 2007 12:00:00 +0000

The technique of Web lookups in Excel I described yesterday is very versatile. I will be running through some of the practical uses it can be put to over the next few days

TO generalise things beyond just getting the Amazon price, I created a user-defined function called XPATH. It takes two parameters:

URL of the XML feed to read
Search XPath list string (separated by spaces)

This function can be used to extract information out of any XML file on the Web and get it out as a table. For example, if you wanted to watch the Top 10 movies on the IMDb Top 250, and were looking for torrents, an RSS feed is available from mininova. The URL http://www.mininova.org/rss/movie_name/4 gives you an RSS file matching all movies with “movie_name”. From this, we need to extract the and <item><link> elements. That’s represented by “//item title link” on my search string.</p> <p><a href="https://www.s-anand.net/blog/assets/flickr-mininova-rss-feed-in-excel_482789516_o-png.webp"></a></p> <p>The formula becomes <code>XPath2( "http://www.mininova.org/rss/"&A2&"/4", "//item title link")</code>. The result is a 2-dimensional array returning individual items in rows, and the columns are title and link. Pulling it all together, you can get the sheet above.</p> <p>All of this could be done using a command-line program. Excel has one huge advantage though. It’s one of the most powerful user-interfaces. Increasingly, I’m beginning to rely on just two user interfaces for almost any task. One is the browser, and the other is Excel. With Excel, I could have a sheet that has my <a href="https://www.s-anand.net/blog/wishlist-for-movies/">movie wishlist</a> (which changes often), and add check to see if the torrent exists. Every time I add a bunch of movies to the wishlist, it’s just a matter of copying the formula down. No need to visit a torrent search engine and typing each movie in, one by one.</p> <p>Another example. Someone suggests 10 movies to watch. I’d rather watch the ones with a higher IMDb rating. Again, enter the Web lookup. Type in the movie names. Use a function like this to look up the rating on IMDb, and sort by the rating.</p> <p>The possibilities are endless.</p> <hr> <h2 id="comments">Comments</h2>  <ul> <li><strong>Isit?</strong> <em>3 May 2007 12:00 pm</em>: Is it possible to have the feeds update cumulative so the old feeds dont get replaced in excel?</li> <li><strong>Alex</strong> <em>3 May 2007 12:00 pm</em>: Hi, I have an Excel spreadsheet with URLs of move titles on IMDB, and I would like to retrieve the movie rating. I’ve read your tutorials, and this one comes closest to what I need. However I still cannot figure out what needs to be done. It looks like you have this problem solved. If so, can you possibly share your solution with me? Thank you, Alex</li> <li><strong>Kishan</strong> <em>3 May 2007 12:00 pm</em>: I came here searching for the other way round. That is to publish an RSS feed of what I have in Xcel! Thanks Anyways</li> <li><strong>venkat</strong> <em>15 Oct 2010 4:09 pm</em>: Anand Can you provide the xpath function?, I need to read a rss feed into excel, and i want only some columns of data, i would appreciate if you could send the xpath function to my email id Thanks venkat</li> <li><strong><a href="http://www.light-sensors.com">Light Sensors</a></strong> <em>2 Dec 2010 7:08 pm</em>: movie ratings really depend on how cool the movie is, i usually rate movies depending on the story line `-;</li> <li><strong>rt</strong> <em>27 Jun 2013 5:41 pm</em>: Hi, nice job ! can you share with me your xpath2 (i tried your other one without succes on amazon) ?Thanks, R</li> </ul>  </article> <article> <h1>Google Reader shared lists are tough to find</h1> <p>Tue, 16 May 2006 12:00:00 +0000</p> <p>Though <a href="http://googleblog.blogspot.com/2006/03/google-reader-learns-to-share.html">Google Reader learns to share</a>, I haven’t found it easy to see what others are reading. Even a <a href="http://www.google.com/search?q=-xyzzyx+site%3Awww.google.com%2Freader%2Fview">Google search for shared lists</a> reveals very few.</p> </article> <article> <h1>Share your OPML</h1> <p>Tue, 09 May 2006 12:00:00 +0000</p> <p><a href="http://share.opml.org/">Share your OPML</a>. Seems to be down, though.</p> <hr> <h2 id="comments">Comments</h2>  <ul> <li><strong>rajkumarc</strong> <em>11 May 2006 7:54 am</em>: 24/25 - Great work and kept me awake until I got it right. Couldn’t figure out what was song #23.</li> <li><strong>S Anand</strong> <em>11 May 2006 11:59 am</em>: #23: Kamal, Sridevi song.</li> <li><strong>rajkumarc</strong> <em>12 May 2006 5:00 am</em>: Thanks S Anand. I guessed it right the first time but typed in Sivappu instead of Sigappu and it was not accepted.</li> <li><strong>S Anand</strong> <em>14 May 2006 8:25 am</em>: Yeah, it’s a little difficult making tradeoffs on what variations to allow and what not to. ‘v’ maps to w, b, g… too many things.</li> </ul>  </article> <article> <h1>Why Google Reader</h1> <p>Thu, 20 Oct 2005 12:00:00 +0000</p> <p>I switched to <a href="http://reader.google.com/">Google Reader</a> as my blog reader (I was using Mozilla so far). The reason was simple: speed. Thanks to the Google site’s speed and keyboard navigation, I can read blog entries 10 times faster. Now <strong>there’s</strong> a unique proposition for Google that a lot of people are missing: that their site loads a whole lot faster than others. It makes a huge difference to the whole browsing experience.</p> <hr> <h2 id="comments">Comments</h2>  <ul> <li><strong>Dhar</strong> <em>24 Oct 2005 9:38 am</em>: Anand, check out <a href="http://www.groovy.com/">Groovy</a>.</li> <li><strong>Dhar</strong> <em>24 Oct 2005 9:38 am</em>: Sorry, it should be <a href="http://www.goowy.com">Goowy</a></li> <li><strong>S Anand</strong> <em>24 Oct 2005 2:26 pm</em>: Quite an app!</li> </ul>  </article> <article> <h1>Google Reader</h1> <p>Sun, 09 Oct 2005 12:00:00 +0000</p> <p><a href="http://reader.google.com/">Google Reader</a>. Gmail principles applied to RSS. You can subscribe to posts, label feeds and star RSS items. The reader suggests items from your subscriptions based on your interest.</p> </article> <article> <h1>Autoblog</h1> <p>Wed, 14 Sep 2005 12:00:00 +0000</p> <p>I have an automated (and lazy) way of finding interesting sites. This is what I do every day.</p> <ol> <li>I get the <a href="http://del.icio.us/">del.icio.us</a> tags of every URL I blog about. (It’s available at <a href="http://del.icio.us/rss/url/">http://del.icio.us/rss/url/</a> followed by the MD5 hex version of the URL).</li> <li>I pick the most popular tags (at least 50 links must have this tag), and use them as my “preferred tags”</li> <li>I scan the most <a href="http://del.icio.us/popular/">popular sites on del.icio.us</a>, and get each site’s tags</li> <li>If a site has my preferred tags, I give it points (the number of points is equal to the number of times I’ve blogged that tag)</li> <li>I pick the top 5 sites based on my points, and read them.</li> </ol> <p>There are two problems I have now. Firstly, I will find sites similar to those I have blogged about – not discover anything new. That’s fine to start with – I can search for those manually. The bigger problem is, this is restricted to del.icio.us. There are two ways I can extend this (lazily).</p> <ol> <li>By finding new sources of popular URLs (which requires a site with a list of popular URLs updated daily, which I will find interesting)</li> <li>By finding new sites that tag URLs (which ideally requires an API to get the tags for a given URL)</li> </ol> <p>There are lots of sources for popular URLs. But though many of sites, including notably <a href="http://www.technorati.com/">Technorati</a>, tag URLs, but none of them I know have APIs.</p> <hr> <h2 id="comments">Comments</h2>  <ul> <li><strong>Ravi</strong> <em>2 Dec 2023 5:08 pm</em>: I am a intensive user of ChatGPT . Asking it to ask clarifying questions was not successful in 3.5 . I gave up . I have not returned to it . May be I don’t think the sentence / context completion model is capable of saying, ‘ huh?’ I have not tried ‘ List the different ways will you will interpret the prompt , . But if I already know that there is room for ambiguity, why won’t I disambiguate the prompt myself ?</li> </ul>  </article> <article> <h1>NotCon Life Hacks</h1> <p>Mon, 29 Aug 2005 12:00:00 +0000</p> <p>Notes from <a href="http://www.craphound.com/lifehacks2.txt">Danny O’Brien’s NotCon Recap of Life Hacks</a>. Summary: People use shells. People use todo.txt. We’ll have private blogs. We’ll have private RSS feeds. Geeks use secret scripts.</p> </article> <article> <h1>RSS Clustering</h1> <p>Wed, 27 Jul 2005 12:00:00 +0000</p> <p><a href="http://monkey.org/~jose/wiki/doku.php?id=rssclustering&DokuWiki=2e25081184329c66f531c206f1069f9b">RSS clustering</a> is about doing what Gmail does on news sources, to RSS feeds of your choice.</p> <hr> <h2 id="comments">Comments</h2>  <ul> <li><strong>Arun</strong> <em>28 Jul 2005 4:01 pm</em>: How do you get to all these amazing links?</li> <li><strong>S Anand</strong> <em>29 Jul 2005 6:01 am</em>: Mainly using <a href="http://del.icio.us/">del.icio.us</a> these days.</li> <li><strong>Arun</strong> <em>29 Jul 2005 2:30 pm</em>: Ah, okie… But there is so much crap there…You are darn good at filtering stuff then! :-)</li> <li><strong>Arun</strong> <em>29 Jul 2005 2:36 pm</em>: btw, dunno if it’s a bug, but there is a problem with the cursor in this text book. If I leave the page and return to the text box after i start typing, am not able to get the cursor to refocus on the text box. It just disappears. In firefox.</li> <li><strong>RaM</strong> <em>29 Jul 2005 4:36 pm</em>: Update the placement section also</li> <li><strong>RaM</strong> <em>29 Jul 2005 4:44 pm</em>: Your Placement Section is static. Pls Update it. Your search tactics are very good</li> <li><strong>Dhar</strong> <em>30 Jul 2005 5:28 am</em>: Arun, I too have had that problem with the cursor.</li> <li><strong>Howie</strong> <em>30 Jul 2005 7:56 am</em>: few links are dead in Placement section</li> <li><strong>S Anand</strong> <em>31 Jul 2005 9:07 pm</em>: Could you give me a step-by-step regarding the cursor problem? I couldn’t reproduce it.</li> <li><strong>Dhar</strong> <em>2 Aug 2005 2:50 pm</em>: <a href="http://digg.com/spy">http://digg.com/spy</a> has an excellent collection of whacky/interesting links. And they use Ajax too. Another nice place to pick up good interesting posts.</li> </ul>  </article> <article> <h1>10 for 10</h1> <p>Fri, 22 Jul 2005 12:00:00 +0000</p> <p>A list of <a href="http://www.micropersuasion.com/2005/07/10_for_10.html">10 trends to keep an eye on over the next 10 years</a>. Includes the Long tail, Ajax, PDAs, Tagging and RSS.</p> </article> <article> <h1>Tagging</h1> <p>Fri, 22 Jul 2005 12:00:00 +0000</p> <p>Tagging is in full swing.</p> <ul> <li><a href="http://del.icio.us/">del.icio.us</a>, <a href="http://www.furl.com/">Furl</a>, <a href="http://myweb2.search.yahoo.com/">My Web 2.0</a>, <a href="http://www.spurl.net/">Spurl</a> etc. tags URLs</li> <li><a href="http://yummy.printfu.org/">Yummy</a> tags PDFs</li> <li><a href="http://www.rojo.com/">Rojo</a> tags RSS feeds</li> <li><a href="http://tagcloud.com/">Tagcloud</a>, <a href="http://www.24eyes.com/">24 eyes</a>, <a href="http://www.feedmarker.com/">Feedmarker</a>, etc. tag RSS posts</li> <li><a href="http://www.technorati.com/">Technorati</a> tags blog posts</li> <li><a href="http://tagsurf.com/">Tagsurf</a> tags discussions</li> <li><a href="http://www.jobazaar.com/">Job Bazaar</a> tags job postings</li> <li><a href="http://gmail.google.com/">Gmail</a> tags e-mails (but you can’t share tags)</li> <li><a href="http://www.connotea.org/">Connotea</a> and <a href="http://www.citeulike.org/">CiteULike</a> tag academic references</li> <li><a href="http://www.swik.net/">Swik</a> tags open-source projects</li> <li><a href="http://www.bigbold.com/snippets/">Snippets</a> tags source code</li> <li><a href="http://www.43things.com/">43 things</a> tags things you want to do</li> <li><a href="http://www.43places.com/">43 places</a> tags places you want to visit</li> <li><a href="http://digg.com/">Diggs</a> tags stories</li> <li><a href="http://www.librarything.com/">Library Thing</a> tags books</li> <li><a href="http://reader2.com/">Reader 2</a> tags books</li> <li><a href="http://upcoming.org/">Upcoming</a> tags events</li> <li><a href="http://www.dinnerbuzz.com/">Dinnerbuzz</a> tags restaurants</li> <li><a href="http://www.flickr.com/">Flickr</a> tags photos</li> <li><a href="http://www.tagzania.com/">Tagzania</a> tags locations</li> <li><a href="http://freesound.iua.upf.edu/">Freesound</a> tags sounds</li> <li><a href="http://www.podcast.com/">Podcast</a> tags podcasts</li> <li><a href="http://www.upto11.net/">Upto 11</a> tags P2P music</li> <li><a href="http://www.musicmobs.com/">Music mobs</a> and <a href="http://genielab.com/">Genie Lab</a> tag music</li> <li><a href="http://www.youtube.com/">You tube</a> tags videos</li> </ul> <p>In fact, <a href="http://supr.c.ilio.us/about">supr.c.ilio.us</a> tags tagging sites!</p> <hr> <h2 id="comments">Comments</h2>  <ul> <li><strong>REMO</strong> <em>23 Jul 2005 4:29 pm</em>: we expect something with a flavour of FINANCE from your page</li> <li><strong>REMO</strong> <em>23 Jul 2005 4:34 pm</em>: Enlighten our life with your financial Aura.</li> <li><strong>S Anand</strong> <em>9 Aug 2005 9:41 pm</em>: A tagged list of tagging sites at <a href="http://supr.c.ilio.us">supr.c.ilio.us</a>.</li> </ul>  </article> <article> <h1>15 things with RSS</h1> <p>Thu, 23 Jun 2005 12:00:00 +0000</p> <p><a href="http://timyang.com/comments.php?id=630_0_1_0_C">15 things you can do with RSS</a> – like <a href="http://rssmix.com/">mix RSS scripts</a>, <a href="http://www.wotzwot.com/rssxl.php">convert any page to RSS</a>, etc.</p> </article> <article> <h1>Open Search</h1> <p>Thu, 17 Mar 2005 12:00:00 +0000</p> <p><a href="http://opensearch.a9.com/">OpenSearch</a>. “We want OpenSearch to do for search what RSS has done for content.” It’s an RSS interface to search, and is an extremely powerful concept.</p> </article> <article> <h1>syndic8ed</h1> <p>Sun, 27 Oct 2002 12:00:00 +0000</p> <p><a href="http://www.syndic8.com/feedinfo.php?FeedID=16143">I’ve been syndic8ed</a>.</p> </article> <article> <h1>Professional news feeds</h1> <p>Thu, 05 Sep 2002 12:00:00 +0000</p> <p><a href="http://jrobb.userland.com/stories/2002/09/03/professionalNewsFeeds.html">Professional news feeds</a>. A treasure for RSS news junkies.</p> </article> <article> <h1>Essential Blogging</h1> <p>Fri, 24 May 2002 12:00:00 +0000</p> <p><a href="http://www.oreillynet.com/cs/weblog/view/wlg/1440">Essential Blogging</a>: a book on weblogging. Quite good. Technie note: This <a href="http://www.voidstar.com/rssfaq">RSS</a> thing looks interesting. <a href="http://www.danchan.com/weblog/daypop/22514">Daypop has started</a> offering RSS output. Need to see how I can use it.</p> </article> <article> <h1>News is free</h1> <p>Wed, 19 Dec 2001 12:00:00 +0000</p> <p><a href="http://www.newsisfree.com/">News is free</a>. Very good.</p> </article> </main></body></html>