February 18th, 2018
March 27th, 2016
In software-defined radio, there are well-established ways of visually representing the signal(s) in the entire bandwidth available from the hardware; we create a plot where the horizontal axis is frequency (using the Fourier transform to obtain the data). Then either the vertical axis is amplitude (creating an ordinary graph, sometimes called panorama) or the vertical axis is time and color is amplitude (creating a waterfall plot).
Here is an example of ShinySDR's spectrum display which includes both types (y=amplitude above and y=time below):
A further refinement is to display in the graph not just the most recent data but average or overlay many. In the above image, the blue fill color in the upper section is an overlay (both color and height correspond to amplitude), the green line is the average, and the red line is the peak amplitude over the same time interval.
We can see signals across an immensely wide spectrum (subject to hardware limitations), but is there a way to hear them meaningfully? Yes, there is, with caveats.
What's pictured above is a small portion of the band assigned to aviation use — they are used primarily for communication between aircraft in flight and air traffic control ground stations. The most significant thing about these communications is that there are a lot of different frequencies for different purposes, so if you're trying to hear “what's in the area”, you have to monitor all of them.
The conventional solution to this problem is a scanner, which is a radio receiver programmed to rapidly step through a range of frequencies and stop if a signal is detected. Scanners have disadvantages: they will miss the beginning of a signal, and they require a threshold set to trade off between missing weak signals and false-triggering on noise.
An alternative, specific to AM modulation (which is used by aircraft), is to make a receiver with very poor selectivity: the ability to receive only a specific channel and ignore other signals. (Historically, when RF electronic design was less well understood and components had worse characteristics, selectivity was a specification one would care about, but only if one lived in an area with closely-spaced radio stations — today, every receiver has good selectivity.)
I'm going to explain how to build an unselective receiver in software, and then refine this to create spatial audio — that is, the frequency of the signal shall correspond to the stereo panning of the output audio. This is the analogue of the spectrum display in audio.
Of course, this is an AM receiver and so it will only make intelligible sound for amplitude-modulated signals. However, many signals will produce some sound in an AM receiver. The exception is that a clean frequency-modulated (FM) or phase-modulated signal will produce silence, because its amplitude is theoretically constant, but this silence is still audibly distinct from background noise (if the signal is intermittent), and transmitted signals often do not have perfect constant amplitude.
A normal software AM demodulator has a structure like the following block diagram (some irrelevant details omitted). The RF signal is low-pass filtered to select the desired signal, then demodulated by taking the magnitude (which produces an audio signal with a DC offset corresponding to the carrier).
In order to produce an unselective receiver, we omit the RF filter step, and therefore also the downsampling — therefore demodulating at the RF sample rate. The resulting real signal must be low-pass filtered and downsampled to produce a usable audio sample rate (and because the high-frequency content is not interesting; see below), so we have now “just” swapped the two main components of the receiver.
This simple change works quite well. Two or more simultaneous AM signals can be received with clear stereo separation.
One interesting outcome is that, unlike the normal AM receiver, the audio noise when there is no signal is quieter (assuming AGC is present before the demodulator block in both cases) — this conveniently means that no squelch function is needed.
The reason for this is obvious-in-hindsight: loosely speaking, most of the noise power will be at RF frequencies and outside of the audio passband. In order to have a strong output signal, the input signal must contain a significant amount of power in a narrow band to serve as the AM carrier and sideband. (I haven't put any math to this theory, so it could be nonsense.)
In order to produce the spatial audio, we want the audio signal amplitude, in a single stereo channel, to vary with frequency. And that is simply a filter with a sawtooth frequency response. The signal path is split for the two stereo channels, with opposite-slope filters. (AGC must be applied before the split.)
An undesired effect is that near the band edges, since the filter has a steep but not perfectly sharp transition from full-left to full-right, there is a lot of slope detection (output from frequency-modulated signals) that does not occur anywhere else. Of course,
This design can of course be applied to more than two audio channels; using surround sound would avoid the need for steepness of the filter at the edges and map the inherently circular digitized spectrum to a circular space, so it's worth trying.
I've implemented this in ShinySDR (and it is perhaps the first novel DSP feature I've put in). Just click the “AM unselective” mode button.
Some “directions for future research”:
As I mentioned above, this is useless for listening to FM signals. Is some technique which can do the same for FM? Naïvely creating an “unselective FM receiver” seems like it would be a recipe for horrible noise, because to a FM demodulator, noise looks like a very loud signal (because the apparent frequency is jumping randomly within the band, and frequency maps to amplitude of the output).
If we declare that the output need not be intelligible at all, is there a way to make a receiver that will respond to localized signal power independent of modulation? Can we make an unmodulated carrier act like an AM signal? (CW receivers do this using the BFO but that is dependent on input frequency.)
September 6th, 2015
The usual definition of the decibel is of course that the dB value y is related to the proportion x by
y = 10 · log10(x).
It bothers me a bit that there's two operations in there. After all, if we expect that y can be manipulated as a logarithm is, shouldn't there be simply some log base we can use, since changing log base is also a multiplication (rather, division, but same difference) operation? With a small amount of algebra I found that there is:
y = log(100.1)(x).
Of course, this is not all that additionally useful in most cases. If you're using a calculator or a programming language, you usually have loge and maybe log10, and 10·log10 will have less floating-point error than involving the irrational value 100.1. If you're doing things by hand, you either have a table (or memorized approximations) of dB (or log10) and are done already, or you have a tedious job which carrying around 100.1 is not going to help.
June 17th, 2015
As vaguely promised before, another update on what I've been working on for the past couple of years:
Specifically, it is in the same space as Gqrx, SDR#, HDSDR, etc.: a program which runs on your computer (as opposed to embedded in a standalone radio) and uses a peripheral device (rtl-sdr, HackRF, USRP, etc.) for the RF interface. Given such a device, it can be used to listen to or otherwise decode a variety of radio transmissions (including the AM and FM broadcast bands everyone knows, but also shortwave, amateur radio, two-way radios, certain kinds of telemetry including aircraft positions, and more as I get around to it).
ShinySDR is basically my “I want my own one of these” project (the UI still shows signs of “I’ll just do what Gqrx did for now”), but it does have some unique features. I'll just quote myself from the README:
I (Kevin Reid) created ShinySDR out of dissatisfaction with the user interface of other SDR applications that were available to me. The overall goal is to make, not necessarily the most capable or efficient SDR application, but rather one which is, shall we say, not clunky.
Here’s some reasons for you to use ShinySDR:
Remote operation via browser-based UI: The receiver can be listened to and remotely controlled over a LAN or the Internet, as well as from the same machine the actual hardware is connected to. Required network bandwidth: 3 Mb/s to 8 Mb/s, depending on settings.
Phone/tablet compatible (though not pretty yet). Internet access is not required for local or LAN operation.
Persistent waterfall display: You can zoom, pan, and retune without losing any of the displayed history, whereas many other programs will discard anything which is temporarily offscreen, or the whole thing if the window is resized. If you zoom in to get a look at one signal, you can zoom out again.
Frequency database: Jump to favorite stations; catalog signals you hear; import published tables of band, channel, and station info; take notes. (Note: Saving changes to disk is not yet well-tested.)
Map: Plot station locations from the frequency database, position data from APRS and ADS-B, and mark your own location on the map. (Caveat: No basemap, i.e. streets and borders, is currently present.)
- Audio: AM, FM, WFM, SSB, CW.
- Other: APRS, Mode S/ADS-B, VOR.
If you’re a developer, here’s why you should consider working on ShinySDR (or: here’s why I wrote my own rather than contributing to another application):
All server code is Python, and has no mandatory build or install step.
Plugin system allows adding support for new modes (types of modulation) and hardware devices.
Demodulators prototyped in GNU Radio Companion can be turned into plugins with very little additional code. Control UI can be automatically generated or customized and is based on a generic networking layer.
On the other hand, you may find that the shiny thing is lacking substance: if you’re looking for functional features, we do not have the most modes, the best filters, or the lowest CPU usage. Many features are half-implemented (though I try not to have things that blatantly don’t work). There’s probably lots of code that will make a real DSP expert cringe.
Now that I've finally written this introduction post, I hope to get around to further posts related to the project.
At the moment, I'm working on adding the ability to transmit (given appropriate hardware), and secondarily improving the frequency database subsystem (particularly to have a useful collection of built-in databases and allow you to pick which ones you want to see).
Side note: ShinySDR may hold the current record for most popular program I've written by myself; at least, it's got 106 stars on GitHub. (Speaking of which: ShinySDR doesn't have a page anywhere on my own web site. Need to fix that — probably starting with a general
topics/radio. Eventually I hope to have a publicly accessible demo instance, but there’s a few things I want to do to make it more multiuser and robust first.)
January 20th, 2015
My interactive presentation on digital signal processing (previous post with video) is now available on the web, at visual-dsp.switchb.org! More details, source code, etc. at the site.
(P.S. I'll also be at the next meetup, which is tomorrow, January 21, but I don’t have another talk planned. (Why yes, I did procrastinate getting this site set up until a convenient semi-deadline.))
November 20th, 2014
I have really failed to get around to blogging what I've been doing lately, which is all software-defined radio. Let's start fixing that, in reverse order.
Yesterday, I went to a Bay Area SDR meetup, “Cyberspectrum” organized by Balint Seeber and gave a presentation of visual representations of digital signals and DSP operations. It was very well received. This video is a recording of the entire event, with my talk starting at 12:30.
October 13th, 2014
The theme of the game is “be consistent”. It's a minimalist-styled 2D platformer. The core mechanic is that whatever you do the first time, the game makes it so that that was the right action. Examples of how this could work:
At the start, you're standing at the center of a 2×2 checkerboard of background colors (plus appropriate greebles, not perfect squares). Say the top left and bottom right is darkish and the other quadrants are lightish. If you move left, then the darkish stuff is sky, the lightish stuff is ground, and the level extends to the left. If you move right, the darkish stuff is ground, and the level extends to the right.
The first time you need to jump, if you press W or up then that's the jump key, or if you press the space bar then that's the jump key. The other key does something else. (This might interact poorly with an initial “push all the keys to see what they do”, though.)
You meet a floaty pointy thing. If you walk into it, it turns out to be a pickup. If you shoot it or jump on it, it turns out to be an enemy.
If you jump in the little pool of water, the game has underwater sections or secrets. If you jump over the little pool, water is deadly.
October 5th, 2014
(I could say some meta-commentary about how I haven't been blogging much and I've made a resolution to get back to it and it'll be good for me and so on, but I think I've done that too many times already, so let's get right to the actual thing...)
When I wrote Cubes (a browser-based “Minecraft-like”), one of the components I built was a facility for key-bindings — that is, allowing the user to choose which keys (or mouse buttons, or gamepad buttons) to assign to which functions (move left, fly up, place block, etc.) and then generically handling calling the right functions when the event occurs.
Now, I want to use that in some other programs. But in order for it to exist as a separate library, it needs a name. I have failed to think of any good ones for months. Suggestions wanted.
Preferably, the name should hint at that it supports the gamepad API as well as keyboard and mouse. It should not end in “.js” because cliche. Also for reference, the other library that arose out of Cubes development I named Measviz (which I chose as a portmanteau and for having almost zero existing usage according to web searches).
(The working draft name is
web-input-mapper, which is fairly descriptive but also thoroughly clunky.)
February 25th, 2014
If you're feeling virtuous:
- Figure out what's going on.
- Figure out why what's going on wasn't immediately obvious.
- Make it so that such failures are caught and reported obviously.
- Make it so that the rest of the system recovers from such failures.
- Write a test for the bug, and a couple more while you're at it.
- Write the actual fix.
(I ought to do more actual concrete blogging, like what I've been doing lately. This crossed my mind as a light and easy piece — I actually followed part of this procedure yesterday after pushing a version (of what, I'll get to later) that was rather broken.)
July 23rd, 2013
One of the nice things about Common Lisp is the pervasive use of (its notion of) symbol objects for names. For those unfamiliar, I'll give a quick introduction to the relevant parts of their semantics before going on to my actual proposal for a “good parts version”.
A CL symbol is an object (value, if you prefer). A symbol has a name (which is a string). A CL package is a map from strings to symbols (and the string key is always equal to the symbol's name). A symbol may be in zero or more packages. (Note in particular that symbol names need not be unique except within a single package.)
Everywhere in CL that something is named — a variable, a function, a class, etc. — the name is a symbol object. (This is not impractical because the syntax makes it easy to write symbols; in fact, easier than writing strings, because they are unquoted.)
The significance of this is that the programmer need never give significance to characters within a string name in order to avoid collisions. Namespacing of explicitly written symbols is handled by packages; namespacing of programmatically generated symbols is handled by simply never putting them in any package (thus, they are accessible only by passing references); these are known as gensyms.
Now, I don't mean to say that CL is perfect; it fails by way of conflating too many different facilities on a single symbol (lexical variables, dynamic variables, global non-lexical definitions, ...), and some of the multiple purposes motivate programmers to use naming conventions. But I think that there is value in the symbol system because it discourages the mistake of providing an interface which requires inventing unique string names.
(One thinking along capability lines might ask — why use names rather than references at all? Narrowly, think about method names (selectors, for the Smalltalk/ObjC fans) and module exports; broadly, distribution and bootstrapping.)
So, here’s my current thought on a “good parts version”, specifically designed for an E-style language with deep equality/immutability and no global mutable state.
There is a notion of name, which includes three concrete types:
- A symbol is an object which has a string-valued name, and whose identity depends solely on that string.
- A gensym also has a name, but has an unique identity (selfish, in E terms). Some applications might reject gensyms since they are not data.
- A space-name holds two names and its identity depends solely on that combination. (That is, it is a “pair” or “cons” specifically of names.)
Note that these three kinds of objects are all immutable, and use no table structures, and yet can produce the same characteristics of names which I mentioned above. (For implementation, the identity of a name as above defined can be turned into pointer identity using hash consing, a generalization of interning.) Some particular examples and notes:
- A CL symbol in a package corresponds to a pair of two symbols, or perhaps a gensym and a symbol. This correspondence is not exact, of course. (In particular, there is no notion here of the set of exported symbols in a package. But that's the sort of thing you have to be willing to give up to obtain a system without global mutable state. And you can still imagine 'linting' for unexpected symbols.)
- The space-name type means that names can be arbitrary binary trees. If we consistently give the left side a “namespace” interpretation and the right side a “local name” one, then we have a system, I think, where people can carve out all sorts of namespaces without ever fearing collisions or conflicts, should it become necessary. Which probably means it's massively overdesigned (cf. "worse is better").
- Actual use case example: Suppose one wishes to define (for arbitrary use) a subtype of some well-known interface, which adds one method. There is a risk that your choice of name for that method conflicts with someone else's different subtype. Under this system, you can construct a space-name whose two components are a large random number (i.e. a unique ID) acting as the namespace, and a symbol which is your chosen simple name. One can imagine syntax and tools which make it easy to forget about the large random number and merely use the simple name.
- It's unclear to me how these names would be used inside the lexical variable syntax of a language, if they would at all; I suspect the answer is that they would not be, or mostly confined to machine-generated-code cases. The primary focus here is improving the default characteristics of a straightforwardly written program which uses a map from names to values in some way.
(This is all very half-baked — I'm just publishing it on the grounds described in my previous post: in the long run I'll have more ideas than I ever implement, and this is statistically likely to be one of them, so I might as well publish it and hope someone else finds some use for it; if nothing else, I can stop feeling any obligation to remember it in full detail.)
May 19th, 2013
I have come to realize that I have more ideas for programs than I'll ever have time to write. (This means they're not actually all that significant, on average — see all that's been said on ‘ideas vs. execution’.) But maybe I have the time to scribble a blog post about them, and that's stuff to blog about, if nothing else.
So, a video game idea I had today: reverse bullet-hell shooter.
A regular bullet-hell shooter is a game where you move in a 2D space dodging an immense number of mostly dumb instant-death projectiles launched in mostly predefined patterns, and trying to shoot back with dinkier, but better aimed, weapons. Instead, here you design the bullet pattern so as to trap and kill AI enemies doing the dodging.
The roles seem a bit similar to tower defense, but the space of strategies would be considerably more, ah, bumpy, since you're not doing a little bit of damage at a time and how it plays out depends strongly on the AI's choices.
That's probably the downfall of this idea: either the outcome is basically butterfly effect random due to enemy AI decisions and you mostly lose, or there are trivial ways to design undodgeable bullet patterns and you mostly win. I don't immediately see how to make the space of inputs and outcomes “smooth” enough.
April 20th, 2013
Tron was one of my childhood favorite movies.
Before I watched Tron: Legacy, I reminded myself to not critique it overly harshly compared to the original — after all, there had been the grid bug scene. (Which, if you don’t happen to recall, introduces grid bugs, without any relationship to the rest of the action, which then never appear again.)
Then they had to throw a bunch of philosophical blather in the middle, which I could only think of at the time as “let’s be all profound like The Matrix”.
But, I was just thinking about that, and the closing scene where the very last of the [SPOILER] is going out and about and [SPOILER], and the early scene in Tron in the laser lab (“Not disintegrating — digitizing!”), and:
Imagine an alternate universe (it can’t happen here, they went too long without) where a key scene goes like this:
MENTOR FIGURE: Remember, that which can be digitized, can be copied.
April 7th, 2013
Yesterday I completely rewrote my resource embedding test to cover more cases (especially ones new in HTML5) and be more usable. In the likely event you're not familiar with it, it's a HTML document which embeds many types of resources (images, audio, other HTML, etc.) using all of the possible containers (
<iframe>, etc.) to see how browsers react.
<img>, but will with
<object>. (In hindsight, this makes perfect sense given the fundamental design principle of Web security, namely “don't add anything which would obviously make existing sites' security assumptions false”, the assumption here being that it's safe to allow
<img>s as user-generated content.)
Specific new features:
- Audio, HTML, and plain text content. (Unfortunately, some combinations cause the audio to autoplay; I tried to make it quiet and plain to make up for that.)
- Scripts inside HTML and SVG content, which also attempt to modify
- Fixed-scrolling headers so you don't need a large window to make sense of the large table.
Let me know if you've thought of any additional useful cases.
March 26th, 2013
Let's say you have two or more independent Git branches, and you want to make sure the combination of them works correctly, but aren't ready to permanently merge or rebase them together. You can do a merge and discard it (either by resetting afterward or using a temporary branch), but that takes extra commands when you're done with the trial. Here's the script I put together to eliminate all unnecessary steps:
#!/bin/sh set -e set -x git checkout --detach HEAD git merge --no-edit -- "$@"
In a single command, this merges HEAD and any branches given as arguments and leaves you at the merge as a detached HEAD. This means that when you're done with it you can just switch back to your branch (git checkout - is a shortcut for that) and the merge is forgotten. If you committed changes on top of the merge, git checkout will tell you about them and you can transplant them to a real branch with git cherry-pick.
February 21st, 2013
When Larry Wall was designing Perl 6, he started with lots of community proposals, from which he made the following observation:
I also discovered Larry's First Law of Language Redesign: Everyone wants the colon.
When I was recently trying to redesign E, I found that this holds true even if only one person is involved in the process. One of the solutions considered was having “
: ” and “
:” be two different tokens…
February 11th, 2013
I really haven't been posting very much, have I? It's mostly the job occupying most of my “creative energy”, but I've also been doing a little bit of this and that and not ever finishing something to the point of feeling like writing it up.
On the programming-projects front, I'm attempting to extract two reusable libraries from Cubes for the benefit of other web-based games.
- Measviz takes performance-measurement data (frames per second and whatever else you want) and presents (in HTML) a compact widget with graphs; my excuse for not announcing it is that the API needs revision, and I haven't thought of a good toy example to put in the documentation-and-demo page I'm writing, but if you're willing to deal with later upgrades it's ready to use now.
- The other library, currently in need of a good name, is a generalized keybinding library (generalized in that it also handles gamepads/joysticks, which are completely different). You define the commands in your application, and it handles feeding events into them. Commands can be polled, or you can receive callbacks on press and release, with optional independent autorepeat. It's currently in need of a name, and also of API cleanup.
I've been making some sketches towards a redesign of E (list archive pointer: starting here), basically to take into account everything we've learned over the years without being constrained by compatibility, but it hasn't gotten very far, partly because language syntax is hard — all options are bad. (The current E syntax is pretty good for usability, but it has some particularly verbose/sea-of-punctuation corner cases, and I'd also like to see a simpler syntax, with more facilities moved into code libraries.)
January 18th, 2013
stdin, stdout, stderr, stdcpu, stdmem, stdfs
November 6th, 2012
Premise: Any attack on a password — whether online (login attempts) or offline (hash cracking) — will be designed so that the more likely a given password is, out of the space of all possible passwords, the less work is required to recover that password (unless a trivial amount of work is required to discover any possible password).
From (1), there exists a probability distribution of passwords.
Premise: There is a (practical) maximum length for passwords.
From (3), the set of possible passwords is finite.
From (2) and (4), there is a minimum probability in that distribution.
Use one of the passwords which has that minimum probability.
(There are at least two ways this doesn't work.)