pandas on S Anand

Things I Learned - 17 Mar 2024

Sun, 17 Mar 2024 00:00:00 +0000

This week, I learned:

DuckDB is 2-10 times faster than Pandas. ClickHouse is supposedly faster but doesn’t run on Windows.
Claude 3 Haiku input costs is $0.25/MTok. That’s half the GPT-3.5 cost. If it’s of comparable quality, it’s worth switching. But Claude 3 Opus is comparable to GPT-4 and twice the cost, so not worth it.
Tavily is a search API for LLMs
Interesting model garden models
There are sites you TRULY cannot scrape even in the browser because of the isTrusted read-only property of events that you can never set to true. Oracle Service Cloud checks for isTrusted in mouse actions.

A-Z of my browsing history

Fri, 09 May 2014 16:16:38 +0000

When you start typing in the address bar, Chrome suggests a link to visit, based on frecency. What do my recommendations look like?

A is for airtel.in/smartbyte-s/page.html – the page where you can check your bandwidth usage. I used to check it infrequently until I upgraded to a 125GB connection. Now I check it every few days and feel miserable that I’ve nowhere near used up my quota. This has coerced me to watch many Telugu movies, of which I don’t understand a word.
B is for blog.gramener.com – I blog there on data stories. The last month or so has been fairly active thanks to the elections.
C is for calendar.google.com – which has become primarily a shared calendar. It was always indispensible to manage my time. Now it helps my colleagues pick when to call me. Right now, my calendar has events booked about two months in advance.
D is for docs.google.com – for effectively one single purpose: shared spreadsheets. This is such a common and powerful use case, and I’m surprised it hasn’t become much easier to use.
E is for epaper.timesofindia.com – some of our content has been published by The Economic Times, and I keep doing ego-searches in the print edition. But close behind is eci.nic.in which I’ve been scraping a lot, and election-results.ibnlive.in.com which we created for CNN-IBN.
F is for flipkart.com – not facebook.com. I’m not often on Facebook.
G is for gramener.com. Naturally. (It’s not surprising that it’s not google.com: I search directly from the address bar.)
H is for handsontable.com – a library that I’ve been using a lot recently, followed by html5please.com that tells me which HTML5 features are ready for use.
I is for ibn.gramener.com – another property we created, but it only just beats irctc.co.in.
J is for join.me – a clean way to share your screen without the audience having to install anything (though you the sharer do have to install the software.)
K is for kraken.io – an amazingly efficient image compressor. As you might have guessed, I lead a strange life.
L is for learn.gramener.com – our Intranet. Sorry, you can’t access this one.
M is for mail.google.com. I’ll probably be moving away from gmail as a backend this weekend to Mail-in-a-box, though. Google’s pulling the plug on Google Reader has shaken my faith.
N is for news.ycombinator.com. When I’m bored and want to watch something while I have dinner, I don’t open YouTube. I open Hacker News.
O is for odc.datameet.org – the Open Data Camp. I’m quite into open data.
P is for pay.airtel.com, but if you ignore the number of bills I pay, it would be pandas.pydata.org, the home page of a remarkable data processing library.
Q is for quirksmode.org, PPK’s remarkable browser-compatibility guide
R is for reader.s-anand.net, my self-hosted RSS reader. It used to be reader.google.com, but Google let me down there.
S is for s-anand.net – this blog.
T is for twitter.com. Unlike Facebook, I don’t dislike Twitter so much.
U is for underscorejs.org. Clearly I need to get a life.
V is for visualizing.org. They have a number of interesting data visualisations.
W is for webpagetest.org – it helps measure the speed of web pages.
X is for xem.github.io. I’ve probably visited this page once, but it’s the only one in my recent history that starts with X
Y is for youtube.com. I lied. I spend an order of magnitude more time watching Telugu movies on YouTube than on Hacker News.
Z is for zoemob.com. Again, a page I visited only once, but there’s nothing else in Z at the moment.

Comments

Software I currently use | s-anand.net 9 May 2014 6:24 pm (pingback): […] course, some of my apps apps have moved online, and my earlier post on the A-Z of my browsing history covers that. But there are a few applications that I’ve hosted which I must talk about. […]
chandigarh 13 Oct 2015 7:27 pm: you can delete your web search history through link https://history.google.com/history

The most popular scientific Python modules

Tue, 11 Dec 2012 02:37:52 +0000

I just scraped the scientific packages on pypi. Here are the top 50 by downloads.

Name	Description	Size	Downloads
numpy	NumPy: array processing for numbers, strings, records, and objects.	2000000	133076
scipy	SciPy: Scientific Library for Python	7000000	33990
pygraphviz	Python interface to Graphviz	99000	22828
geopy	Python Geocoding Toolbox	32000	18617
googlemaps	Easy geocoding, reverse geocoding, driving directions, and local search in Python via Google.	69000	15135
Rtree	R-Tree spatial index for Python GIS	495000	14370
nltk	Natural Language Toolkit	1000000	12844
Shapely	Geometric objects, predicates, and operations	93000	12635
pyutilib.component.doc	Documentation for the PyUtilib Component Architecture.	372000	10181
geojson	Encoder/decoder for simple GIS features	12000	9407
GDAL	GDAL: Geospatial Data Abstraction Library	410000	8957
scikits.audiolab	A python module to make noise from numpy arrays	1000000	8856
pupynere	NetCDF file reader and writer.	16000	8809
scikits.statsmodels	Statistical computations and models for use with SciPy	3000000	8761
munkres	munkres algorithm for the Assignment Problem	42000	8409
scikit-learn	A set of python modules for machine learning and data mining	2000000	7735
networkx	Python package for creating and manipulating graphs and networks	1009000	7652
pyephem	Scientific-grade astronomy routines	927000	7644
PyBrain	PyBrain is the swiss army knife for neural networking.	255000	7313
scikits.learn	A set of python modules for machine learning and data mining	1000000	7088
obspy.seisan	SEISAN read support for ObsPy.	3000000	6990
obspy.wav	WAV(audio) read and write support for ObsPy.	241000	6985
obspy.seishub	SeisHub database client for ObsPy.	237000	6941
obspy.sh	Q and ASC (Seismic Handler) read and write support for ObsPy.	285000	6926
crcmod	CRC Generator	128000	6714
obspy.fissures	DHI/Fissures request client for ObsPy.	1000000	6339
stsci.distutils	distutils/packaging-related utilities used by some of STScI's packages	25000	6215
pyopencl	Python wrapper for OpenCL	1000000	6124
Kivy	A software library for rapid development of hardware-accelerated multitouch applications.	11000000	5879
speech	A clean interface to Windows speech recognition and text-to-speech capabilities.	17000	5809
patsy	A Python package for describing statistical models and for building design matrices.	276000	5517
periodictable	Extensible periodic table of the elements	775000	5498
pymorphy	Morphological analyzer (POS tagger + inflection engine) for Russian and English (+perhaps German) languages.	70000	5174
imposm.parser	Fast and easy OpenStreetMap XML/PBF parser.	31000	4940
hcluster	A hierarchical clustering package for Scipy.	442000	4761
obspy.core	ObsPy - a Python framework for seismological observatories.	487000	4608
Pyevolve	A complete python genetic algorithm framework	99000	4509
scikits.ann	Approximate Nearest Neighbor library wrapper for Numpy	82000	4368
obspy.imaging	Plotting routines for ObsPy.	324000	4356
obspy.xseed	Dataless SEED, RESP and XML-SEED read and write support for ObsPy.	2000000	4331
obspy.sac	SAC read and write support for ObsPy.	306000	4319
obspy.arclink	ArcLink/WebDC client for ObsPy.	247000	4164
obspy.iris	IRIS Web service client for ObsPy.	261000	4153
Orange	Machine learning and interactive data mining toolbox.	14000000	4099
obspy.neries	NERIES Web service client for ObsPy.	239000	4066
pandas	Powerful data structures for data analysis, time series,and statistics	2000000	4037
pycuda	Python wrapper for Nvidia CUDA	1000000	4030
GeoAlchemy	Using SQLAlchemy with Spatial Databases	159000	3881
pyfits	Reads FITS images and tables into numpy arrays and manipulates FITS headers	748000	3746
HTSeq	A framework to process and analyze data from high-throughput sequencing (HTS) assays	523000	3720
pyopencv	PyOpenCV - A Python wrapper for OpenCV 2.x using Boost.Python and NumPy	354000	3660
thredds	THREDDS catalog generator.	25000	3622
hachoir-subfile	Find subfile in any binary stream	16000	3540
fluid	Procedures to study geophysical fluids on Python.	210000	3520
pygeocoder	Python interface for Google Geocoding API V3. Can be used to easily geocode, reverse geocode, validate and format addresses.	7000	3514
csc-pysparse	A fast sparse matrix library for Python (Commonsense Computing version)	111000	3455
topex	A very simple library to interpret and load TOPEX/JASON altimetry data	7000	3378
arrayterator	Buffered iterator for big arrays.	7000	3320
python-igraph	High performance graph data structures and algorithms	3000000	3260
csvkit	A library of utilities for working with CSV, the king of tabular file formats.	29000	3236
PyVISA	Python VISA bindings for GPIB, RS232, and USB instruments	237000	3201
Quadtree	Quadtree spatial index for Python GIS	40000	3000
ProxyHTTPServer	ProxyHTTPServer -- from the creator of PyWebRun	3000	2991
mpmath	Python library for arbitrary-precision floating-point arithmetic	1000000	2901
bigfloat	Arbitrary precision correctly-rounded floating point arithmetic, via MPFR.	126000	2879
SimPy	Event discrete, process based simulation for Python.	5000000	2871
Delny	Delaunay triangulation	18000	2790
pymc	Markov Chain Monte Carlo sampling toolkit.	1000000	2727
PyBUFR	Pure Python library to encode and decode BUFR.	10000	2676
collective.geo.bundle	Plone Maps (collective.geo)	11000	2676
dap	DAP (Data Access Protocol) client and server for Python.	125000	2598
rq	RQ is a simple, lightweight, library for creating background jobs, and processing them.	29000	2590
pyinterval	Interval arithmetic in Python	397000	2558
StarCluster	StarCluster is a utility for creating and managing computing clusters hosted on Amazon's Elastic Compute Cloud (EC2).	2000000	2521
fisher	Fast Fisher's Exact Test	43000	2503
mathdom	MathDOM - Content MathML in Python	169000	2482
img2txt	superseded by asciiporn, http://pypi.python.org/pypi/asciiporn	443000	2436
DendroPy	A Python library for phylogenetics and phylogenetic computing: reading, writing, simulation, processing and manipulation of phylogenetic trees (phylogenies) and characters.	6000000	2349
geolocator	geolocator library: locate places and calculate distances between them	26000	2342
MyProxyClient	MyProxy Client	67000	2325
PyUblas	Seamless Numpy-UBlas interoperability	51000	2252
oroboros	Astrology software	1000000	2228
textmining	Python Text Mining Utilities	1000000	2198
scikits.talkbox	Talkbox, a set of python modules for speech/signal processing	147000	2188
asciitable	Extensible ASCII table reader and writer	312000	2160
scikits.samplerate	A python module for high quality audio resampling	368000	2151
tabular	Tabular data container and associated convenience routines in Python	52000	2114
pywcs	Python wrappers to WCSLIB	2000000	2081
DeliciousAPI	Unofficial Python API for retrieving data from Delicious.com	19000	2038
hachoir-regex	Manipulation of regular expressions (regex)	31000	2031
Kamaelia	Kamaelia - Multimedia & Server Development Kit	2000000	2007
seawater	Seawater Libray for Python	2000000	1985
descartes	Use geometric objects as matplotlib paths and patches	3000	1983
vectorformats	geographic data serialization/deserialization library	10000	1949
PyMT	A framework for making accelerated multitouch UI	18000000	1945
times	Times is a small, minimalistic, Python library for dealing with time conversions between universal time and arbitrary timezones.	4000	1929
CocoPy	Python implementation of the famous CoCo/R LL(k) compiler generator.	302000	1913
django-shapes	Upload and export shapefiles using GeoDjango.	9000	1901
sympy	Computer algebra system (CAS) in Python	5000000	1842
pyfasta	fast, memory-efficient, pythonic (and command-line) access to fasta sequence files	14000	1836

Comments

Ron Z 25 Jun 2013 10:19 pm: Another good one is simpleCV: http://www.simplecv.org/
Ravindranath M 8 Apr 2013 7:13 am: Nice list Anand. Very useful. – This is Ravi from Comviva, who attended your introduction.