Cobra's bits (Posts about mathematics)

Units

Cobra — Sun, 19 Jun 2022 12:17:54 GMT

As a physicist, it is very useful to be able to perform calculations quickly, whether it is by pure mental arithmetic or on the back of an envelope. I'm immensely grateful for the training I've received in school that allows me to perform such calculations with relative ease. When I see my students struggling with even the most trivial of these calculations and having to rely on their smartphones to get a result, I get the impression that the education system is dismissing these skills as obsolete in our digital age. A big mistake, if you ask me.

Don't get me wrong: when I need accurate results, I also resort to calculators or calculator apps on Android or Arch. And as an added benefit, some of them handle not only numbers, but also the units. This is really helpful, because converting numerical results from all sorts of obscure units to SI is often necessary when consulting older references, and it's as tedious as it is frustrating - just look at the Wikipedia page of the cgs system.

Apart from Mathematica, I know of three command-line programs (one of which, namely, qalc, is also available with a graphical user interface) for doing calculations with units: the veteran units, the modern qalc, and the new and hip insect. Here's an example of these three tools dealing with the conversion of the surface tension (in N/m) of water to surface energy (in eV/Å²), which for a liquid is one and the same despite the different units.

Functions with default values

Cobra — Sat, 25 May 2019 20:36:59 GMT

Suppose you would like to have a command generating a secure password for an online service at the command line. You would google for that and find 10 ways to generate a random password. At the end of his article, the author presents the ideal way to generate a secure password:

date | md5sum

The author of the article (Lowell Heddings, the founder and CEO of How-To Geek) states:

I’m sure that some people will complain that it’s not as random as some of the other options, but honestly, it’s random enough if you’re going to be using the whole thing.

Random enough? Sure, the 'whole thing' looks random enough:

9ec463af3db95e8e44de84417d9f408f

but the look is deceptive: this is in fact an extremely weak password. To understand why, let's look at the output of the 'date' command:

↪ date
Sun 12 May 2019 04:33:26 PM CEST

We see that without additional parameter (like +"%N"), 'date' gives us one password for each second of the year. How many passwords do we get in this way? Well,

 ↪ date +"%s"
1557666649

i.e., 1,557,666,649 seconds has passed since 00:00:00, Jan 1, 1970 (Unix epoch time), and that's how many passwords we get.

Now, the possibility to order Pizza online came much later, namely, at August 22nd, 1994.

↪ date -d 19940822 +"%s"
777506400

That leaves us with 780,160,249 passwords since this memorable day in 1994 or a complexity of 30 bits, corresponding to a 5-digit password with a character space of 62. Let's get one of these and see how difficult it is to crack:

↪ pwgen -s 5 -1
p9iCN

Now, even my ancient GTX650Ti with its modest MD5 hashing performance of 1.5 GH/s cracks this password in 5 s (note that an RTX2080 delivers 36 GH/s...):

○ → hashcat -O -a 3 -m 0 myhashes.hash ?a?a?a?a?a
hashcat (v5.1.0) starting...

OpenCL Platform #1: NVIDIA Corporation
======================================
- Device #1: GeForce GTX 650 Ti, 243/972 MB allocatable, 4MCU

0b91091d40a8623891367459d5b2a406:p9iCN

Session..........: hashcat
Status...........: Cracked
Hash.Type........: MD5
Hash.Target......: 0b91091d40a8623891367459d5b2a406
Time.Started.....: Mon May 13 12:48:58 2019 (5 secs)
Time.Estimated...: Mon May 13 12:49:03 2019 (0 secs)
Guess.Mask.......: ?a?a?a?a?a [5]
Guess.Queue......: 1/1 (100.00%)
Speed.#1.........: 514.0 MH/s (6.21ms) @ Accel:64 Loops:47 Thr:1024 Vec:2
Recovered........: 1/1 (100.00%) Digests, 1/1 (100.00%) Salts
Progress.........: 2328363008/7737809375 (30.09%)
Rejected.........: 0/2328363008 (0.00%)
Restore.Point....: 24379392/81450625 (29.93%)
Restore.Sub.#1...: Salt:0 Amplifier:0-47 Iteration:0-47
Candidates.#1....: s3v\, -> RPuJG
Hardware.Mon.#1..: Temp: 44c Fan: 33%

But actually, it's even worse: instead of cracking the hash one can easily precompute all possible values of the 'date | md5sum' command, and thus create a dictionary containing these “passwords”. I could start right away:

for (( time=777506400; time<=1557666649; time++ )); do date -d@$time | md5sum | tr -d "-"; done > lowell_heddings_passwords.txt

On my desktop with its Xeon E3 v2, this command computes one million passwords in about half an hour, i.e, I'd need about 17 days for computing all passwords back to 1994. Writing a corresponding program running on the GPU would cut this down to seconds. Note that the resulting list of “random enough” passwords is static, i.e., it is indeed a dictionary, and not even a particularly large one.

Lowell Heddings himself mentions several alternative ways to generate a password in his article before turning to the worst possible solution. But if we desire cryptographically secure solutions, even apparently innocuous commands are beset with difficulties, as pointed out by, for example, the carpetsmoker (better carpets than mattresses). In the end, it all boils down to the following three choices that are available on virtually any Linux installation. If we limit ourselves to a character space of 62:

cat /dev/urandom | base64 | tr -d /=+ | head -c 25; echo
openssl rand -base64 25 | tr -d /=+ | head -c 25; echo
gpg2 --armor --gen-random 1 25 | tr -d /=+ | head -c 25; echo

If we insist of having almost all printable characters (which often calls for trouble):

cat /dev/urandom | base91 | head -c 25
openssl rand 25 | base91 | head -c 25; echo
gpg2 --gen-random 1 25 | base91 | head -c 25; echo

One could, in principle, also utilize dedicated password generators, such as Theodore Tso's 'pwgen', Adel I. Mirzazhanov's 'apg', or haui's 'hpg':

pwgen -s 25
apg -a 1 -M ncl -m 25 -x 25
hpg --alphanum 25

All of these ways are cryptographically equivalent in the sense that the entropy of the passwords generated by either of them asymptotically approaches the theoretical value ($\log_2(62) \approx 5.954$ bits per character) when you average over many (10,000,000 or more). In the present context (functions with default values) the generators do not offer any advantage, but only add unnecessary complexity.

Now, whatever you chose as your favorite, you don't want to memorize the command or rely on the history of your favorite shell. One could define an alias with the password length as parameter, but I prefer to use a function for this case to have a default length of 25 characters with the option to change this value:

↪ pw62
mPcSU1c3lBTC7gChJ4MBw1sZW

↪ pw62 8
Yjs6NhYM

↪ pw62 32
cfn4KKugWHhOBF8qn6SO5Rj7uC2LksnK

Here's how to implement this functionality for the three major shells. Note the very elegant way with which a default value can be implemented within the bash. Update: haui reminded me that the zsh is a drop-in replacement for the bash and thus of course implements all bash variable substitution, particularly ${var:-default}. Hence, we can use the same syntax for the bash and the zsh, and only the fish needs the comparatively clumsy construct shown below. 😎

bash

function pw62
{
cat /dev/urandom | base64 | tr -d /=+ | head -c ${1:-25}; echo
}

fish

function pw62
if set -q $argv
set length 25
else
set length $argv
end

cat /dev/urandom | base64 | tr -d /=+ | head -c $length; echo

end

zsh (alternative to the bash syntax)

function pw62()
{
if [ "$1" != "" ]
then
integer length=$1
else
integer length=25
fi

cat /dev/urandom | base64 | tr -d /=+ | head -c $length; echo
}

Calculators

Cobra — Mon, 22 Apr 2019 16:03:35 GMT

For quick calculations, I prefer to use my HP handheld calculators whenever possible, simply because I'm much faster with them than with anything else thanks their responsive physical keypad and, of course, RPN. Alas, there are computational tasks that few, if any, handhelds are up to. Big numbers, in particular, usually result in an overflow rather than in the desired solution. Let's take factorials as example – they are faster growing than any ordinary functions (including exponential ones) and are thus perfectly suited for getting big numbers.

Here's how the factorial $n!$ looks in comparison to its little sister, the exponential $e^n$:

The dashed line shows the Stirling approximation $\sqrt{2 \pi n} \left(\frac{n}{e}\right)^n$, which reveals that the factorial essentially grows with $n^n$ and thus faster than any exponential whatever its base.

Now, the largest factorial my HP42s can handle is 253!, which amounts to 5.173460992e+499. For a handheld, this is more than respectable: the largest factorial one can compute on a Linux desktop with, for example, xcalc as the calculator application, is 170!, limited simply by the fact that numbers in xcalc are represented by double precision floats.

All right, xcalc is ancient. But as a matter of fact, most calculator applications running on Windows, MacOS, or Linux have difficulties with large numbers. The Windows calculator, for example, gives up at any numbers bigger than 1e+10,000, and hence can't calculate factorials larger than 3249!. And we didn't even talk about getting exact results, which demand arbitrary precision arithmetic already for much smaller numbers.

Let's have a look at some calculators for Linux that can do better than those above. I use Mathematica 11.2 as reference:

1,000,000! = 8.263932e+5,565,708, taking 1.1/0.2 s for an exact result/numerical approximation
100,000,000! = 1.617204e+756,570,556, taking 344/46 s for an exact result/numerical approximation
0verflows at $MaxNumber 1.605216761933662e+1,355,718,576,299,609

Note that a file storing the result of 100,000,000! has an uncompressed size of 0.757 GB. So be careful when writing even larger factorials to disk 😉 ($MaxNumber would be 1.35 PB!).

CLI Calculators

bc/dc

The Unix calculators. Offer arbitrary precision since 1970, and now you know what the 'bc' stands for in this blog's title! 😉 Neither of them supports factorials out of the box, but hey, these are programming languages, not plain calculators. Examples for scripts computing factorials can be found on Rosettacode and on Stackoverflow, but be aware that these examples are horribly inefficient — for fast algorithms see Peter Luschny's page. Here's the “script” for dc:

dc -e '?[q]sQ[d1=Qd1-lFx*]dsFxp'

After typing a number like 1000, we get an exact result. Overflows somewhat below 67,000!. bc does not, but it's too slow to be of much use for very much larger numbers.

wcalc

Approximate results with principally arbitrary precision defined by the command line parameter P (which accepts only integers and is thus useless for really large numbers).

wcalc -P 10 'fact(1000000)'

Takes 1.42 s, overflows somewhat below 44,500,000!

calc

My default calculator on PCs. Gives exact results.

calc 1000000!

Takes 330 s, and overflows somewhat below 2,200,000,000!

hypercalc

When firing up hypercalc, we are greeted with ”Go ahead -- just TRY to make me overflow!”. And indeed, that's not an easy task at first. Hypercalc gives approximate results only, but essentially instantaneous ones even for absolutely monstrous numbers. The factorials we have considered so far are kids play for this program. Instead of the factorial of a million, a billion, a trillion, why not ask for the factorial of a Googol! Hypercalc tells us this number amounts to 1e+(9.9565705518098e+101), and that agrees with the solution from Wolfram Alpha (see below), the only tool, which can at least partly follow hypercalc into the realm of big numbers.

But not when it comes to really big ones. Let's have a look, for example, at Pickover's superfactorial n$. What about, say, 10$? That's completely out of reach for any program I know, but not for hypercalc: 8pt8e+23804068 (PT stands for PowerTower). But even this is still a very very very small number: hypercalc overflows only at 1e+308pt1e+34 or, equivalently in Donald Knuth's up-arrow notation, 10↑↑1.7976e+308.

All of this is contained in a Perl script available for download (or for installation in the AUR for Archlinux users), and additionally in a Javascript powered web interface.

iPython

So far, we haven't been able to get exact results faster than with Mathematica. Let's see how python is doing in this regard. There are two possibilities, the first using plain python, the second scipy:

In [1]: import math
In [2]: %timeit math.factorial(1000000)
6.5 s ± 19.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [3]: from scipy.special import factorial
In [4]: %timeit factorial(1000000, exact=True)
6.53 s ± 17.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

It's disappointing that scipy doesn't outperform regular python, but instead seems to use exactly the same code. A special function should, IMHO, perform better than Mathematica (which needs only 1 s for the same task).

You can use python also directly from the terminal, and write the result to disk instead of displaying it directly:

echo 1000000 | python -c 'import sys; import math; print(math.factorial(int(sys.stdin.readline())))' > fac1M.dat

julia

Python turned out to be a disappointment, what about Julia, which is advertised to be suitable for high-performance numerical analysis and computational science?

julia> @time factorial(big(1000000));
0.162650 seconds (1.57 k allocations: 53.524 MiB, 2.00% gc time)
julia> @time factorial(big(100000000));
45.550577 seconds (432.60 k allocations: 11.586 GiB, 0.90% gc time)

Now we're talking!

You can write the results to disk in this way:

julia> using DelimitedFiles
julia> fac1M=factorial(big(1000000));
julia> writedlm("fac1M.dat", fac1M)

Graphical calculators

Most desktop calculators still attempt to imitate the look of handhelds, just like media players used to resemble stereo decks. Some of these reconstructions are historically accurate and appeal to our nostalgia, but in terms of usability, these relicts of the 1990s are among the most clumsy and inefficient user interfaces ever invented, particularly since people never use the keyboard to interact with these abominations, but the mouse. However, some graphical calculators give you the choice.

qalculate!

A great general purpose calculator packed with features. Includes an RPN mode and a plotting interface to gnuplot, as well as excellent conversion utilities that can be updated daily (important for currencies). We can get an approximate result for 1,000,000! in under 1 s, but 100,000,000! takes so long that I didn't wait. Overflows reportedly at 922,337,203,854,775,808!, which is impressive, but of little use because of the comparably poor performance.

speedcrunch

Easy to use and insanely fast. Even on slow hardware, the result is there as soon as you type it. Overflows at 72,306,961!

Web calculators

Young people tell me that installing local apps is so 1990ish, and of course, you can perform essentially all calculations you ever need in the interwebs.

Casio

When I look at my colleagues desks at the office, I frequently see Casio calculators from the 1980s. Even I have one, although I have no idea when and where I've acquired it (and I also don't remember ever using it). In any case, true to their roots, Casio offers a quite capable online calculator. It gives 1,000,000! in about 2 s and overflows only at 1e+100,000,000, just below 14,845,000!

Wolfram Alpha

More than a calculator: a knowledge engine. You may ask what's the weather in Berlin today, and get the interesting bit of information that on April 22nd, it was -4°C in 1997 and 31°C in 1968. But foremost, Wolfram Alpha is an arbitrary precision calculator. It gives exact results where appropriate (for reasonably sized outputs) and approximate ones when the output seems to large. Regardless the task, the answers take a few second, whether you calculate 2+2 or 1e+(9^9^9)! [1e+1e+(1.58274e+369693108)]. Overflows at (2e+1573347107)! or 1e+(1e+9.196824545990035)! That's much higher than Mathematica, but still nothing compared to hypercalc.

Representative surveys

Cobra — Sun, 25 Jun 2017 14:34:20 GMT

Statistical surveys are a standard tool of sociology, and have been the subject of extensive research. In the hands of professionals, the results of these surveys can be surprisingly accurate. As a result, surveys have attained the status of the oracle of Delphi, and people fervently believe in them. Naturally, this development has made surveys an attractive tool for manipulating public opinion. The standard way to do this is to load the questions with a moral obligation. Don't you agree that the internet should be regulated to deprive extremists of their safe spaces online? No? Really, what a kind of person are you? Don't you ever think of the children?

However, unexpected results of surveys do not always have sinister reasons, but may instead simply reflect the incompetence of the inquirer. In particular, the most elementary rule for designing a survey is frequently forgotten: namely, that those taking part in the survey have to understand the questions. Sounds obvious, doesn't it? But is it?

An example: the recent news of heise online that only(!) 16% percent of all Germans encrypt their emails (Umfrage: Nur 16 Prozent der Deutschen verschlüsseln ihre E-Mails). This survey was conducted by Convios Consulting on behalf of United Internet (UI), one of the largest internet and mail providers in Germany.

UI claims that about 750,000 of their users have generated PGP key pairs. That's a very impressive number, particularly since according to UI, only 4.7 million keys “exist” worldwide. The UI users would thus account for 16% of all PGP keys. Doesn't that demonstrate very nicely that UI's encryption initiative introduced in August 2016 is highly successful?

Well, the whole reason for the survey was to create exactly this impression. I have no doubts that the numbers quoted above are correct, but what do they mean?

First of all, the number reported for the existing keys worldwide only accounts for the keys that have been deposited on key servers. Nobody can estimate how many keys have actually been generated or are in use. That's quite different in the case of the UI encryption scheme, which is based on mailvelope, and the storage of the public key of the user in a database located on a UI server. The number given above is thus the total number of UI customers with a PGP key, unless they use a separate key in a stand-alone MUA (of which I know two 😉).

There are about 40 million email users in Germany. According to UI, about half of them use GMX or WEB.DE, which seems reasonable as UI is reported to have close to 20 million customers. Now, let's suppose that all UI customers who have generated a key are also actually using it to encrypt their mail. In this unlikely case, 3.75% of all UI users would encrypt their mail, much less than the “only 16%” of their survey. Obviously, that must mean that 28.25% of the other 20 million email users in Germany, who are mostly customers of the German Telekom, Google, and Microsoft, encrypt their mail. Right?

Of course not. Try to ask arbitrary Gmail users if they encrypt their mail. 84% will look at you with with blank eyes, but 16% will recognize the word and confirm that they do, YES! Ask them afterwards if they know the difference between transport and end-to-end encryption. I guarantee that you will soon get tired of asking people because even after several hundreds you won't find a single one who can answer your second question...

How many people do encrypt their mails? I don't think there exist any bona fide surveys on that topic. I can only provide anecdotal evidence with very limited statistical significance. On the other hand, I've been a serious advocate of end-to-end-encryption since about 15 years. I've written tutorials and motivated many of my personal contacts to use end-to-end encryption in email and messenging. Well, some would say I forced them at gunpoint. But that would be exaggerated...

I have currently 49 personal contacts with public PGP keys, and 16 business contacts. That doesn't sound too bad, does it? However, 17 and 4 of these keys are expired, leaving 32 and 12, respectively. Subtracting keys whose pass phrases have been forgotten by their users or were otherwise disposed of leaves 19 and 7. Some of my contacts have passed away, are retired, or I've simply lost touch, leaving in the end 4 and 5 with which I can, in principle, exchange end-to-end encrypted mails. Actually, however, there are only three persons with whom I regularly exchange encrypted mails: my patent attorney at work and my fellow PdeS (that's why they have that label) in actual life.

Three out of 65 with an actively used key, but of how many without any clue what that even means? I don't want to spent time on the question of how I could count the number of unique addresses in my mail folders over the past few years. But obviously, this number would be in the several hundreds. In other words, the total percentage of people employing end-to end-encryption in my emails is way lower than 1%. And if I wouldn't be interested in these kind of things, and if I wouldn't be a scientist, this percentage would be exactly zero. Not 16%. Not 0.16%. Zero.

Can we find out how many people really encrypt their mails by a survey? Not really. If less than 100 ppm of all people encrypt (which is the number I find most plausible), we would need a mega-survey over 50000 people to include at least 5 people who actually do encrypt, and that's never going to happen. And don't let them tell you that the rules of statistics somehow don't apply there and representative surveys can answer all of these questions as if by magic. That's bullshit. All of it.

Indices

Cobra — Sun, 22 May 2016 09:03:48 GMT

A ȷupyter notebook

Scientists move in mysterious ways, particularly when they try to measure their individual performance as a scientist. As I've explained in a previous post, the most popular and commonly accepted of these measures is the h index $\mathcal{H}$, which has been declared to be superfluous on both empirical and mathematical grounds. Either of these references relates $\mathcal{H}$ to the square root of the total number of citations $\mathcal{N}$, the first one approximately

$\mathcal{H} \approx 0.5 \sqrt{\mathcal{N}}$

and the second one exactly:

$\mathcal{H}=\sqrt{6}\log{2}\sqrt{\mathcal{N}}/\pi \approx 0.54 \sqrt{\mathcal{N}} $.

Since I anyway wanted to test pandas, seaborn and statsmodel, I gathered $\mathcal{H}$, $\mathcal{N}$, and the i10 index $\mathcal{I}$ from all my coauthors on Google Scholar. It turned out that not even a quarter of my coauthors have a Google Scholar account, but I figured that 71 data points would provide an acceptable statistics.

%matplotlib inline

import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import statsmodels.formula.api as sm
import seaborn as sns

# read data into a Pandas DataFrame
data = pd.read_table("/home/cobra/ownCloud/MyStuff/projects/python/publishing/hindex.dat")
# check the data head
data.head()

	Citations	Hindex	i10index
0	952	12	17
1	1913	20	35
2	5327	34	151
3	650	13	14
4	3855	25	68

Looks all right.

logdata = np.log10(data)

The correlation of the data is much more clear when displayed logarithmically:

vars = ["Citations", "Hindex", "i10index"]
sns.pairplot(logdata, vars=vars, size=3, kind="reg");

Now look at that! Two lines of code and seaborn visualizes all correlations in my data set. The diagonal elements of this 3x3 matrix plot show the distributions of $\mathcal{N}$, $\mathcal{H}$, and $\mathcal{I}$ (which seem to be close to normal distributions), and the off-diagonal elements visualize their correlations emphasized by a linear regression (kind="reg"). And how correlated they are! There's indeed no need for a definition of 'indices' if the number of citations is all what it boils down to.

Seaborn is great for visualization, as we have seen, but for quantitative statistical information, it's better to use statsmodel:

hc = sm.ols(formula='Hindex ~ Citations', data=logdata)
fithc = hc.fit()

ic = sm.ols(formula='i10index ~ Citations', data=logdata)
fitic = ic.fit()

hi = sm.ols(formula='Hindex ~ i10index', data=logdata)
fithi = hi.fit()

Let's compare the slope of our data with that predicted above:

fithc.params.Citations

0.45708354021378172

np.sqrt(6)*np.log(2)/np.pi

0.54044463946673071

Solid state phycisists have to work harder!

One can get also get more information, if desired:

fithi.summary()

OLS Regression Results
Dep. Variable:	Hindex	R-squared:	0.974
Model:	OLS	Adj. R-squared:	0.974
Method:	Least Squares	F-statistic:	2592.
Date:	Mon, 16 May 2016	Prob (F-statistic):	1.84e-56
Time:	14:42:47	Log-Likelihood:	114.08
No. Observations:	71	AIC:	-224.2
Df Residuals:	69	BIC:	-219.6
Df Model:	1
Covariance Type:	nonrobust

	coef	std err	t	P>\|t\|	[95.0% Conf. Int.]
Intercept	0.4498	0.019	23.723	0.000	0.412 0.488
i10index	0.5454	0.011	50.910	0.000	0.524 0.567

Omnibus:	3.438	Durbin-Watson:	2.391
Prob(Omnibus):	0.179	Jarque-Bera (JB):	2.600
Skew:	-0.408	Prob(JB):	0.273
Kurtosis:	3.462	Cond. No.	7.44

And of course, we can display these fits independent of seaborn:

xlist_cit = pd.DataFrame({'Citations': [logdata.Citations.min(), logdata.Citations.max()]})
xlist_i10 = pd.DataFrame({'i10index': [logdata.i10index.min(), logdata.i10index.max()]})

preds_hcit = fithc.predict(xlist_cit)
preds_hcit;

preds_i10cit = fitic.predict(xlist_cit)
preds_i10cit;

preds_hi10 = fithi.predict(xlist_i10)
preds_hi10;

logdata.plot(kind='scatter', x='Citations', y='Hindex')
plt.plot(xlist_cit, preds_hcit, c='red', linewidth=2);

logdata.plot(kind='scatter', x='Citations', y='i10index')
plt.plot(xlist_cit, preds_i10cit, c='red', linewidth=2);

logdata.plot(kind='scatter', x='i10index', y='Hindex')
plt.plot(xlist_i10, preds_hi10, c='red', linewidth=2);

Image quality

Cobra — Sat, 02 Apr 2016 10:45:19 GMT

A ȷupyter notebook

If I would be asked to nominate the one property which characterizes the modern world best, it would be its increasing focus on graphical representations. We are experiencing a transition from a textual to a pictorial society. People communicate with each other not by sending texts, but by photographs on social networks or emojis in chat applications. They learn how to do that not by reading a manual, but by video tutorials. Even highly technical articles on IT oriented websites are invariably illustrated, however unrelated or absurd the respective illustration may be.

I have no serious problem with that as long as the images are a pleasure to look at and are nice to my band width. For keeping this balance, however, a certain understanding of graphic formats is helpful. Only yesterday I received a two-page abstract for a conference which weighed a hefty 73 MB. Saving the images in an appropriate format reduced the size to 1.2 MB. That was not even the most extreme example I've experienced with excessively large files...

More frequently, however, the problem is not an excessive image size but a mediocre image quality. To avoid this problem requires either a certain perceptiveness, or an algorithm suitable for an objective ranking of image quality. The structural similarity index (SSIM) is such an algorithm, and is widely used as video quality metrics in the televison industry.

The present ȷupyter notebook calculates the SSIM for two bitmaps derived from a vector graphics. The first bitmap is obtained by

pdftocairo -png -r 1200 org.pdf org.png
pngquant org.png web.png

The resulting bitmaps measure 550×934 px, and the files 'org.png' and 'web.png' are 30519 and 14797 bytes in size, respectively — significantly larger than the original graphics 'org.pdf' (4214 bytes).

Can we reduce this size further by converting the image to the popular jpeg format? As was to be expected for a line graphics with hard contrasts, we can, but pay the prize of a much reduced image quality. Indeed, to obtain a jpeg file of similar size (to be precise, 15556 bytes) we have to use very low settings:

convert org.png -quality 20 lossy.jpg

As you see in the comparison below, the SSIM for 'web.png' is almost identical to the original, while that of 'lossy.jpg' is drastically lower. And many of you will probably wonder why, since you don't see a difference in the images below – right?

I do, but I have a trained eye. To convince you that the SSIM is meaningful, here are the two images for direct inspection: web.png lossy.jpg.

%matplotlib inline

import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt

from skimage import img_as_float
import skimage.io as io
from skimage.measure import compare_ssim as ssim

mpl.rcParams['font.size'] = 20
mpl.rcParams['figure.figsize'] = (16, 8)

org = io.imread('org.png', as_grey=False)
org_png = io.imread('web.png', as_grey=False)
org_jpg = io.imread('lossy.jpg', as_grey=False)

img_org = img_as_float(org)
img_png = img_as_float(org_png)
img_jpg = img_as_float(org_jpg)

fig, (ax0, ax1, ax2) = plt.subplots(nrows=1, ncols=3)
plt.tight_layout()

ssim_none = ssim(img_org, img_org, multichannel=True, 
                 dynamic_range=img_org.max() - img_org.min())
ssim_png = ssim(img_org, img_png, multichannel=True,
                  dynamic_range=img_png.max() - img_png.min())
ssim_jpg = ssim(img_org, img_jpg, multichannel=True, 
                  dynamic_range=img_jpg.max() - img_jpg.min())

label = 'SSIM: %.4f'

ax0.imshow(img_org, cmap=plt.cm.gray, vmin=0, vmax=1)
ax0.set_xlabel(label % (ssim_none))
ax0.set_title('Original image')

ax1.imshow(img_png, cmap=plt.cm.gray, vmin=0, vmax=1)
ax1.set_xlabel(label % (ssim_png))
ax1.set_title('PNG')

ax2.imshow(img_jpg, cmap=plt.cm.gray, vmin=0, vmax=1)
ax2.set_xlabel(label % (ssim_jpg))
ax2.set_title('JPEG')

plt.show()

Old habits

Cobra — Sun, 18 Oct 2015 12:09:14 GMT

Or: Recombination dynamics in a coupled two-level system with strong nonradiative contribution (an ipython notebook)

One of my students investigates the transient behavior of the photoluminescence emitted by (In,Ga)N quantum heterostructures after being irradiated by a short laser pulse. The characteristic feature of the transients he observes for these structures is a power-law decay of the photoluminescence intensity with time at low temperatures (10 K), which changes into an exponential decay at higher temperatures (150 K).

His results reminded me of ones I acquired myself ages ago, during my own time as a PhD student. I didn't have a sensible interpretation then, but I do have one now. Hence, to the surprise of my student, I nonchalantly wrote down the following two coupled differential equations as if they had just occurred to me:

${\dot n_b} = -n_b/\tau_{rel} - n_b/\tau_{nr} + n_w \exp(-\frac{E_b}{k_B T})/\tau_e$

${\dot n_w} = n_b/\tau_{rel} - t^{b-1} n_w/\tau_{w} - n_w \exp(-\frac{E_b}{k_B T})/\tau_e$

with the second term in the second equation ($t^{b-1} n_w/\tau_{w}$) being the experimental observable. The form of this term is giving rise to what is known as a stretched exponential, which for $b \rightarrow 0$ approaches a power law for long times.

Using Mathematica, it takes 7 lines of code to solve this system and to plot it for several temperatures $T$ (or, equivalently and as done here, for different energies $k_B T$):

from IPython.display import Image
Image(filename='/home/cobra/ownCloud/MyStuff/Documents/pdes-net.org/files/images/deqs.png')

As I had hoped, this simple model reproduces the behavior observed in the experiment fairly well. My student was also pleased, but only with the result, not with the method: he's familiar with Matlab, but not with Mathematica. Well, I suspect that he's also not too familiar with Matlab, since he could otherwise have easily solved the equations himself.

In any case, his admission reminded me that I actually wanted to migrate my computational activities to free software whenever possible. It's not easy to get rid of old habits, and as I'm using Mathematica since 23 years, the code above just came naturally, while the one below still required an explicit intellectual effort. But that's essentially the same lame excuse which I'm tired to hear from users of, for example, Microsoft Office when asked to prepare a document with LibreOffice.

So let's get moving. Here's the above differential equation system solved and plotted using numpy, scipy, and matplotlib in an ipython notebook. Note how the notebook integrates the actual code with comments, links, pictures and equations. Editing this notebook is a real treat thanks to the use of markdown and LaTeX syntax.

#Initialize
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint 

mpl.rcParams['figure.figsize'] = (6, 4)
mpl.rcParams['font.size'] = 16
mpl.rcParams['text.usetex'] = True
mpl.rcParams['font.family'] = 'Serif'
mpl.rcParams['lines.linewidth'] = 2
mpl.rcParams['xtick.major.pad'] = 7

# Parameters
taurel = 0.1       # capture time
taue = 0.1         # emission time
taunr = 0.01       # nonradiative lifetime
tauw = 1.65        # radiative lifetime
eb = 20            # activation energy (in meV)
b = 0              # stretching parameter (approaches power law for b -> 0)

# solve the system dn/dt = f(n,t) and plot the solution
fig = plt.figure()
#for kt in np.linspace(1,13,7):          # approximate temperatures
for T in [10,30,50,70,100,150]:          # exact temperatures

    kt = 0.086173324*T                   # in meV

    def f(n,t):
            nbt = n[0]
            nwt = n[1]
            # the model equations
            f0 = - nbt/taurel - nbt/taunr + nwt*np.exp(-eb/kt)/taue
            f1 = nbt/taurel - nwt*np.exp(-eb/kt)/taue - t**(b-1)*nwt/tauw
            return [f0, f1]

    # initial conditions
    nb0 = 1.                            # initial population in barrier
    nw0 = 0                             # initial population in well
    n0 = [nb0, nw0]                     # initial condition vector
    t  = np.logspace(-2,2,1000)         # logarithmic time grid

    # solve the DES
    soln = odeint(f, n0, t)
    nb = soln[:, 0]
    nw = soln[:, 1]

    # plot results
    plt.loglog(t, t**(b-1)*nw/tauw/max(t**(b-1)*nw/tauw), label=r'%.0f K' %T)
    plt.xlabel('Time (ns)')
    plt.ylabel('Intensity (arb. units)')
    plt.axis([7e-3,40,1e-5,2]);
    plt.legend(loc='lower left', frameon=False, prop={'size':15}, labelspacing=0.15)

fig.savefig('transients.pdf')

The above command saves this plot as a publication-ready figure in pdf format. There are many other available formats, including eps (for the traditional LaTeX/dvipdf toolchain), svg (for further editing with inkscape, or publishing in the web) and png (for insertion in a Powerpoint/Impress presentation).

Enough is not enough

Cobra — Mon, 25 May 2015 17:26:56 GMT

People like to tell me about their digital life. Recently, I hear a lot about laptops replacing desktops. To my surprise, many are willing to spend quite a bit for this transition: most more than €1000 and some €2000 and above. All of the former assure me that the performance of their desktop replacement is "more than enough". Several of the latter actually believe that the system's performance is directly related to its pricetag. One member of this group owns a Macbook 12 (sorry for the cliché, but what can I do) and tried to convince me in a particular insistent and tenacious way that his gadget would outperform even the most powerful desktops available.

As a matter of fact, it is quite far from this feat (look here). As all ultrabooks equipped with a Core M-5YXX processor, it performs slightly better (20–40%) than my €299 Fujitsu Lifebook which, however, is miles away from a decent desktop:

Right, the last one is not your usual desktop, but a compute server with—at the moment of the screenshot—a load of 24. The remaining 8 physical cores managed to outperform my desktop, if only by a slight margin. But I bet that my new office desktop (an i7 4790) will be able to complete the run under 100 s ... let's see tomorrow. 😉

Update: even below 80 s:

In any case, my lifebook has been thoroughly smashed and humiliated: instead of the 2 minutes required by the Xeons it needed a staggering 20 minutes for the same result. The lifebook is great for writing this entry, but for serious tasks I rather turn to a serious computer. And the same applies to your Core M driven ultrabooks.

At this point, the compassionate ones of the laptop owners are sure to secretly pity me. Just imagine that I have to stay all time in a kind of server room for being able to take advantage of the performance depicted above. ... how gruesome!

Well, as much as I like to sit in my study, I prefer to run my computations from where I want. How do I do that?

I have WiFi. 😄

Seriously, for computations I use an ipython server running on my desktop. The posts of Fillipo and Nikolaus have helped me to find the best (most robust and convenient) way to connect to this server.

For what follows, both the server and the client need to have mosh installed. For the server, tmux (which I also like to use for different reasons) is required in addition.

From the client, I connect to the server (blackvelvet) by issuing

mosh blackvelvet -- tmux new-session -s ipython

In this session, I then start an ipython notebook server on blackvelvet:

ipython notebook --no-browser --port=8889

and subsequently detach the session with Ctrl-a d.

I can attach again anytime by issuing

mosh blackvelvet -- tmux a

on the client. Isn't that neat?

To connect to the server, I start an ssh tunnel on the client with

ssh -N -L localhost:8888:localhost:8889 cobra@blackvelvet

and open the notebook at http://localhost:8888/:

Thanks to MathJax, the font rendering is way better than that of Mathematica.

The ssh tunnel can be stopped with a single Ctrl-C on the client, the ipython server needs a double one on the server (after attaching the session again).

Sculpting

Cobra — Sat, 02 May 2015 11:23:57 GMT

Suppose we have a crystal in the form of a rectangular block, given by a list of the absolute atomic coordinates with one atom per line

Ga   -9.278018850000   -9.642000000000   -7.870137870000
N    -7.422415050000   -9.642000000000   -8.518637230000
.
.
.

looking like that:

What we really want, though, is a hexagonal column which tapers down from the bottom to the top — just like a column of Doric order. How can we sculpt this column out of the block we have? Where do we get the digital chisel we need? A colleague of mine solved this quest in the most elegant fashion: with an awk one-liner.

awk '{x=sqrt($2^2);y=sqrt($3^2);d=x/2+sqrt(3)/2*y;z=$4;cutoff='$radius1'*('$zmax'-z)/'$zrange'+'$radius2'*(z-('$zmin'))/'$zrange'};d<=cutoff && x<cutoff' block.dat > column.dat

Voilà:

Having fun with classical ciphers

Cobra — Thu, 19 Dec 2013 16:11:23 GMT

19.12.2013
I've written most of what follows three years ago, but I've never found the time to finish and post it. As usual this close to Christmas, I have a few days off, and took the opportunity to delve once again into the fascinating world of classical ciphers. :cheesey:

27.12.2010
The millenia-old war between code-makers and code-breakers has essentially stopped in the last century: modern ciphers are so strong that they just can't be broken in any humanly accessible time-span. Diffie, Hellmann, and Merkle solved the key exchange problem, Phil Zimmermann gave encryption to the masses, and the GnuPG project transferred Zimmermanns 'Pretty Good Privacy' to the open source world.

End of story. Or is it?

Well, for most practical applications, that's basically it. We can trust that either of the following commands

gpg -ca --passphrase-repeat 0 --cipher-algo aes256 <file>
gpg -sea -r cobra <file>

results in an encrypted file whose content will be inaccessible to even the most nosy of our sniffing and snooping friends. Unless, of course, they revert to alternative ways of collecting information. :P

People had this particular problem since the beginning of time and invented steganography. Nowadays, one can can easily hide and encrypt information in one and the same command:

steghide embed -e rijndael-256 -z 9 -cf <coverfile> -ef <embeddedfile>; -sf <combinedfile>

All right, so where's the catch?

My itch with modern ciphers is not a practical one. I rather find their mathematical foundation abstract and dry, although I can digest most of it when I try really hard. However, you can't possibly utilize modern encryption without a computer.

Ancient ciphers, in contrast, are often intuitive and easy to master by paper and pencil. And what's more, ancient ciphers are surrounded by a mystical and glorious aura. Just think of the Enigma!

When I was a kid, Diffie and Hellman as well as Rivest, Shamir and Adleman still chased attractive female students across the campus. There was no modern cryptography, just an afterglow from the war. There were only a handful of computers, and hardly any of them in public possession. Being a nerdy kid, I became member of the local chess club with age eight. People there talked a lot about chess, but also about encryption, and the atmosphere turned out to be infectious. I adored Edgar Allan Poes Gold Bug as well as Arthur Conan Doyles Adventure of the Dancing Men. I was thrilled to death by Vigenère's Le chiffre indéchiffrable (French for 'the indecipherable cipher'). And I would have died for book ciphers. The unbreakable type, you know.

And now over Christmas, I thought, aw what the hell ... let's just play with all that's fun! Being fascinated by the Vigenère cipher, I sat down and implemented it using the only programming language I've ever learned: Pascal. And since we don't have to fill the _tabula recta _ourselves anymore, I've generalized the classical scheme to the entire 95 letter alphabet of the ASCII printable characters. That took me an entire day, but my Pascal courses took place almost 30 years ago. 😞

A Vigenère cipher can be as strong as desired. If the key is as long as the message, and truly random, it constitutes an example of a one-time pad (OTP), the only cipher which is information-theoretically proven to be unbreakable. Despite of this fact, OTPs have been rarely used in the past since key distribution proved to be an insurmountable problem.

Quantum key distribution has solved this problem in principle, but is not (yet) for everyone. Still, if the number of people on a communication channel is limited, the distribution of an OTP is easier than ever. Remember than one A4 page of text roughly corresponds to 3 kB. The 2 GB of an average USB stick would thus suffice for 700,000 of such pages, which should be enough for even the most communicative individuals. 😉

True randomness, however, is an requirement not easily obtained at all, since only natural phenomena such as atmospheric noise or radioactive decay are truly random. A computer, instead, is a deterministic beast and can only generate pseudo-random numbers. The "best" algorithms for doing that are called 'cryptographically secure pseudo-random number generators' (CSPRNG), but even these are predictable to a certain degree. I consider the one I'll use in the following to be a very solid contender, but not necessarily the best (for which Blum-Blum-Shub generators may qualify, though they are slow, and there's no readily available open-source implementation anyway).

How do we judge the quality of a CSPRNG? One of the most popular battery of tests are compiled in the dieharder suite, a successor of George Marsaglia's famous diehard tests. This suite interfaces to the continuous stream of random numbers of the generator, here csrng, like that:

csprng-generate | dieharder -g 200 -a

Two tests were indicated as "weak", but that turned out to be a result of the limited statistics with the default settings of dieharder.

We can also examine a finite list of random numbers created by such a generator. An infinite perfectly random sequence would exhibit an entropy per character equal to S=log2(k), where k is the character space. We will come back to that later.

A Vigenère cipher used in conjunction with a pad generated by a CSPRNG is an example for a running key cipher, but one could equally well call it a pseudo one-time pad (POTP). Depending on the cryptographic quality of the CSPRNG used, it may be trivial or essentially impossible to break. It will never be an information-theoretical secure system such as the OTP, but it may approach a level of security similar to that offered by modern cryptographic techniques.

My little Pascal routine in conjunction with the POTP created by csrng is perfectly capable of encrypting pure ASCII data, but can't handle anything beyond such as extended ASCII, UTF-8 or binary data. To improve on that, I preprocess the 'plaintext' (which may also be an image or an executable) with base91 to obtain a well-defined character set regardless of the file type.

Finally, I need to cut a section of the POTP serving as the key for each respective message, and to subsequently shred this section so it cannot be used again. All of that is easily done by a short shell script. Let's examine the essential ingredients of this script (in pseudo script code, for the actual one, see below):

csprng-generate -n 4G | tr -cd '[:graph:]' | fold -w 65 > POTP

Creates a stream of random numbers with a total volume of 4 GB. The binary stream is filtered through 'tr', letting pass only printable ASCII characters. These characters are then arranged in lines of 65 characters length and saved as our POTP. On my desktop, this command takes about 35 s to complete.

Let's now look at the encryption part (decryption is largely the same, just the order is different):

b91enc -w 65 PLAINTEXT -o ENCODEDTEXT

Encodes the PLAINTEXT with base91, and ensures that the ENCODEDTEXT has the same format as the POTP or the key.

tail -n NUMBER POPT > KEY

Generates the key (NUMBER is determined by the length of the ENCODEDTEXT, see the shell script for details).

vig -e ENCODEDTEXT KEY > ENCRYPTEDTEXT

Encrypts.

truncate -s KEY POTP

Removes the KEY from the POTP.

All of that is pretty straightforward except for the strange fact that I get the KEY from the bottom of the file rather than from the top, which would be the intuitive choice for a 'pad'. The reason for this choice is very simple: removing the key from the bottom takes 1 ms, but from the top several seconds depending on the speed of the hard disk. This asymmetry is an inevitable consequence of the file system's way to store data. With the present 'backward' scheme, the total time for the steps 1. — 4. amounts to 30 (450) ms for an A4 text page (a 5MB pdf).

Let's examine the files created by these steps using John Walker's entropy estimator ent. For the 95 characters used here, a perfectly random and infinite sequence of numbers is expected to have an entropy of 6.570 bit per character. For our POTP as well as for the plaintext encrypted with it, ent arrives at an entropy of 6.569 bits per character.

That's pretty satisfactory, but let's have a closer look. Deviations from randomness are often plainly evident in a visual inspection of the data in question. For example, we can examine the distribution of characters in our POTP using

ent -c POTP | head -97 | tail -94 | awk '{print $3}'> hist.dat

Plotting these values yields a histogram of the character distribution:

Looks almost entirely equally distributed, as we had hoped. However, in a finite random sequence, fluctuations do occur which can be made visible by plotting the deviation of each value from the mean of the distribution. Besides the fact that the values are small (the standard deviation is indeed only 0.02%), these deviations do not form any obvious pattern, but look indeed random.

We can perform the same analysis for the data after being encrypted by the Vigenère routine:

The fluctuations are larger (standard deviation 0.4%) since the sample size is significantly smaller, but once again the data do not exhibit any obvious deviations from a random distribution.

To analyze these data, by the way, I've used only the ipython console (ipython --pylab) and a few pylab commands:

a=loadtxt('hist.dat')
n=arange(94)
bar(n,a)
bar(n,a-mean(a))
std(a)/mean(a)

For sake of clarity and documentation, I've attached an archive containing the shell script hfwcc and the Pascal routine vig.pas. For the former, I've disabled truncation as this feature forfeits local testing (it should only be used with two independent POTPs). For convenience, I've included a 64 bit binary of the latter compiled with fpc 2.6.2. The encoding tool base91, the random number generator csnrg and the analysis program ent are all available in the AUR (Arch User Repository). I'd expect you'll also get them for Debian, but users of other distributions will probably have to compile them themselves.