Oliver Gmelin

The July 30 pressure wave / 100km Alpstein FAI

2026-07-30T00:00:00+02:00

Risk mitigation: A homage to modern (in-)flight information systems.
Technology is often considered unromantic in mountain sports, but some hassles are well worth it.

Tech Aversion in Mountain Sports

I am under the impression that there’s still a rift in the mountain sports and paragliding communities regarding the usage of technology. Devices and resources are sometimes waved aside as unnecessary, obtrusive or unromantic. After all, LCD screens distract us from true adventure and genuine experiences in nature, right? Mountaineers and pilots thereby miss out on safety enhancing, possibly life-saving technologies.

On July 30 2026, large parts of Europe were affected by a dry air pressure wave. Violent wind gusts originated from an array of thunderstorms cells over France and swept eastwards without any warning signs: No clouds, no steady increase in wind speeds, just life-threatening air conditions from one second to the next. This rare occurrence was predicted by the ICON models, albeit the exact timing remained a bit uncertain. It caused a stir on social media, most of the pertinent weather apps and services posted warnings about the following day.

Excellent depiction of the incoming threat in Burnair.

Takeaway 1: Do an exhaustive meteo briefing. Every single time. No exemptions.

Days like this prove that you will never outsmart modern weather models even with years of experience on your local site. There’s a nonzero chance that things turn bad in ways you won’t see coming. Depending on your planned activity, proper “state of the art” planning preparation likely requires a (paid) subscription to a specialized weather service. That’s a small price in comparison to the potential cost of an accident.

Thorough weather briefings are already a point of contention. People sometimes justify subpar preparation because their plans are allegedly a special case. “It’s just a quick hike and fly!” Sure, conditions the last 30 minutes might have been calm, but that could have led to a severe accident on July 30. Preparatory weather checks are mandatory whether you’re hiking, mountaineering, crosscountry- or speedflying. Only if you know exactly what’s coming, you can manage risks with adequate safety margins. This, on the other hand, opens up potentials on less than ideal days.

In-Flight Hassle vs Reward

While solid preparation is out of the question, I’d actually distinguish the recommended PG tech based on the planned activity. Short glides or even speedflying won’t give you much chance to check for weather updates. The measurement intervals are just too long, you’re busy flying and local indicators (trees, windsocks etc.) are just better scoped.

As soon as there’s a chance to thermal upwards or even dip into XCish terrain, I’d argue that in-flight weather information systems become sensible. Looking at the animation above, we can see that the measured wind speeds (isolated arrows) were actually increasing ahead of the predicted pressure wave. While modern weather predictions are pretty good, live weather updates still provide significant safety benefits. As depicted below, there were really no warning signs about the incoming danger.

About to pass by the Saentis.

Blue skies all around.

En route along Walensee.

Takeaway 2: Anything that could turn into a longer thermal or XC flight warrants in-flight weather data.

Collision Avoidance Systems

Collision avoidance systems based on FLARM and the newer ADS-L systems are not as widely adopted as one would expect in 2026. Granted, between paragliders they’re often useless: when thermalling together you just have to stay vigilant and collisions in low-visibility conditions (e.g. being sucked in a cloud) should not happen in the first place. But paragliders share the airspace with faster aircraft, whether it be sailplanes, motorized sporting planes, helicopters or even low-flying military jets.

A spectacular collision between a paraglider and Cessna in May shows that this is not a purely theoretical concern. If you’ve ever wondered whether that sail plane over there had actually seen you or when jets in the clouds above sounded too close for comfort you probably pondered on getting a vario with FLARM/ADS-L. Hopefully you acted on that thought. Caveat emptor: Currently there’s no guarantee other aircraft will actually use and see you on the necessary receivers. There’s rumors regarding mandatory electronical identification measures for paragliders in Switzerland. This has the potential to replace the unpopular stick-on labels. For U spaces that will be shared with autonomously flying drones, there will be no way around ADS-L anyways.

Takeaway 3: At least for thermalling/XC, get an FLARM/ADS-L transceiver.

Online Tracking and Distress Calls

The last puzzle piece of in-flight electronics/services that comes to mind is online tracking. If you’re using FLARM/ADS-L this is potentially covered through OGN receivers already, depending on your region. Alternative are GSM-based, e.g. the rega app or Burnair. These all rely (more or less) on a line of sight to a corresponding cellular transmitter. That’s already an enormous benefit: the search radius for a rescue operation can be narrowed down significantly if your last airborne position is known. There’s simply no excuse to take off without app-based live tracking enabled.

The last failure mode is rooted in the unavailability of terrestrial radio networks. In the air, cell coverage in the Alps is pretty good. In the event of a reserve deployment and a rough landing somewhere on a glacier you have a good chance to be found at some point if you had a GSM tracker running and someone notices that you’re out way longer than anticipated. The key advantage is time: being able to call for help immediately even without network coverage might be a life-saver and cold nights can turn threatening.

There’s multiple vendors for satellite communication that bridge the last remaining gap. The Iridium and Globalstar networks have global coverage, others are more region specific. Pick carefully, depending on your location. For adventures in remote areas these are a must-have.

Takeaway 4: Get a satellite communicator as soon as you start solo XC flying.

Common Counterarguments

I think a major point essentially boils down to not wanting to deal with yet another complicated piece of equipment, i.e. laziness. We should overcome this as a community. Another counterargument is the price point: Granted, a complete setup isn’t cheap. But considering the possible consequences of a paragliding accident I think it’s well invested and not paying off on a personal level, but also for the entire community.

The safer paragliding is, the less severe the consequences weather forecast misinterpretations and accidents become, the fewer ramifications and resultant regulations will we see as a community. Maybe we should adapt the “all the gear, all the time” credo from motorcycling.

The 100km FAI Triangle on July 30

Despite the pressure wave prediction I went flying on July 30. Absolutely fantastic day. Would have been nice to fly to Lucerne or Interlaken, but I obviously turned around and opted for the 100km triangle instead. The pressure wave was clearly visible on live weather in-flight.

Tracklog preview.

Full logs are available on DHV XC, xcontest and Burnair 3D

Computer Vision for Aquatic Biodiversity Exploratories

2026-06-20T00:00:00+02:00

Implementing imaging systems for automated long-term experiments on aquatic evolution.
A sporadically updated project log.

Intro

Biodiversity is declining and freshwater lakes are not exempt from this global trend. Modeling and understanding these huge systems is nontrivial. Classical lab setups, e.g. chemostats, do not provide sufficient volume and operational lifetime to observe the intricate interactions and derive robust underlying interaction patterns between species and environmental conditions.

Small-volume chemostats: insufficient models for entire lakes

Freshly initialized chemostats.

Algal growth.

Mesocosm Deployments

With project ABOVE, we aim for (mostly) automated long-term experiments on the evolution of large-scale aquatic model systems. Multiple sets of mesocosms, which are 600 liter tanks, are set up in groups of four. These share a multiparameter probe for chemical properties as well as flow cytometers for plankton imaging.

ABOVE mesocosm deployments

revised tank group installation

provisional control infrastructure

The depicted white tanks are already an improved, winter-hardened version. The tanks are not hermetically sealed, but allow for a reasonable environmental exchange (e.g. regarding oxygen) similar to that of a real lake. Internal flow pumps prevent the water from turning stale, a refill mechanism compensates for sample extraction and the temperatures are controlled within a viable range. Lights allow controlling the algal energy intake. The tanks themselves, the fluid handling and the control systems were implemented and manufactured by 4H Jena Engineering.

Test runs in the V1 tanks.

Tank delivery on the new platform.

Analysis Automation

With intended operational times in the ballpark of decades and many replicates, experiments of this extent would constitute a significant expenditure of human labor. This is where the automation aspect comes into play: flow cytometry instead of manual sampling and traditional microscopy, continuous 24/7 monitoring of relevant parameters and fusion of comprehensive time-series data is expected to generate insights into the evolution of aquatic systems.

Computer vision to the rescue

Flow cytometry image and instance segmentation.

Segmented objects at 10fps.

To this extent, we have provisioned several hundred terabytes of storage to keep raw data and derived descriptors for the foreseeable future. Results will follow. Probably as a paper, not so much a blog post.

LLMs on a GP102/Pascal 3-Way SLI Rig

2025-12-20T00:00:00+01:00

Hassle-free local inference through Vulkan.
Nvidia keeps nudging Pascal further out the door, but there’s an easier way to keep these cards doing useful work than fighting the CUDA toolchain on a rolling distro.

Three Cards, One Dead Standard

The rig is a leftover from when multi-GPU gaming was still a thing people did: three GP102-based GeForce cards on an X299 board, bridged for 3-way SLI, fed by a 1600W supply that has spent most of the last few years being wildly overqualified for the job.

Three-way GP102 setup.

SLI itself is thoroughly dead — no driver profile is coming to make three cards render one game any more. But that has nothing to do with whether the chips can still compute. Each GP102 is a perfectly capable device with a useful amount of VRAM attached, and three of them idling next to an oversized PSU is a waste when running models locally has become the normal way to try anything out. So: ollama, and how hard can it be.

Nvidia Shows Pascal the Door

Harder than it should be, because Pascal is now actively being removed from the stack — twice over, at two different layers.

The first wall is the driver. Nvidia’s 590 branch drops support for Pascal and everything older, as part of the same move that makes the open-source kernel modules the default. Pascal predates the GSP firmware those modules rely on, so it cannot follow. On Arch this is not a gentle deprecation: the nvidia, nvidia-dkms and nvidia-lts packages roll forward to 590, and if you let that happen on a Pascal box you get a broken graphical environment. The prescribed fix is to uninstall those and install nvidia-580xx-dkms from the AUR instead.

The good news is that 580 is an LTS branch, and Nvidia has committed to supporting pre-Turing hardware on it into mid-2028. So the card keeps working — you just pin it and stop upgrading that one thing.

CUDA 13 Can’t Build for It Either

The second wall is the toolkit, and it is the one that actually bites. CUDA 13.0 removed offline compilation for everything below compute capability 7.5 — which sweeps up Maxwell, Pascal and Volta together. GP102 is sm_61. nvcc from CUDA 13 will not emit device code for it at all; there is no flag to talk it round, because the target support is gone rather than merely deprecated.

Worth being precise about what this does and does not break. It is a build-time wall, not a runtime one: binaries compiled with CUDA 12.9 or earlier keep running fine on any hardware the installed driver supports. Nothing already working stops working. But the moment you want to compile something yourself — which is exactly what you are doing if you want an ollama build that targets your cards — the last toolkit that can do it is 12.9.

The Toolchain Rabbit Hole

Pinning an old toolkit sounds simple. On a rolling-release distro it starts a chain.

CUDA 12.9 supports GCC up to 14.x and no further. Arch has long since moved past that, so the system compiler is rejected outright — not with a subtle miscompile, but with a hard version check baked into CUDA’s host_config.h:

unsupported GNU version! gcc versions later than 14 are not supported!

So you install gcc14 alongside your real compiler and point nvcc at it with -ccbin, or wave the check away with -allow-unsupported-compiler and hope. Getting either of those threaded through a package build’s CMake configuration is its own small adventure.

Clear that, and the next layer surfaces: an old CUDA toolkit’s headers against a current libstdc++ don’t agree on exception specifications any more, and the build dies on mismatched declarations. Patching noexcept(true) onto the offending declarations in CUDA’s own headers gets you moving again.

Add it up and the “working” configuration is a pinned driver, a pinned toolkit, a second compiler installed solely to appease the first one, and hand-edited vendor headers — a stack of four things that each break independently on the next update. That is not a setup, that is a pet.

Vulkan Sidesteps All of It

None of it is necessary, because CUDA is not the only way to get compute out of these cards.

llama.cpp’s Vulkan backend — which ollama can be built against — talks to the GPU through the Vulkan compute API, and its shaders are compiled at runtime by the driver. There is no offline device-code generation step, so there is no nvcc, no architecture flag, no host compiler ceiling and no vendor headers to patch. The 580xx driver you are already pinned to for display purposes ships a perfectly good Vulkan implementation. What got cut for Pascal was the CUDA build toolchain, not the silicon’s ability to run compute shaders.

In practice this means swapping vulkan in for the cuda_v* entries in OLLAMA_LLAMA_BACKENDS when building the package, and that is essentially the whole intervention.

The trade-off is real but narrow. Vulkan and CUDA land in roughly the same place on token generation, which is the number you feel when a model is answering you. Prompt processing is where CUDA still has a clear lead, so long-context work and big prompt ingests are noticeably slower. For interactive use on hardware this old, that is a good trade.

Worth It?

For three cards that were otherwise going to sit in a cupboard, comfortably. The honest framing is that Pascal has entered its long tail: the driver is pinned until 2028, the CUDA path is closed to new builds, and nothing about that is going to improve. Vulkan is the option that doesn’t fight it — it asks nothing of the toolchain, so there is nothing in the setup to break the next time something upstream moves.

FFoveated: Spending Bits Where the Eye Is Looking

2020-06-11T00:00:00+02:00

Steering an H.264 encoder with live eye-tracking data.
My MSc thesis asked a simple question: if you know exactly where someone is looking, how many bits can you stop sending everywhere else?

Only a Tiny Part of the Frame Is Actually Seen

The human retina is wildly non-uniform. Cone density peaks in the fovea centralis, right on the visual axis, and falls off steeply from there — outside a narrow region of roughly 2.5° around the current fixation point, colour and detail simply are not resolved at full acuity. Every video codec on the planet nevertheless encodes the whole frame as if the viewer were scrutinising all of it at once.

Region-of-interest coding tries to exploit this, but conventionally has to guess which region matters, using saliency models or face detectors. Those guesses are speculative and frequently wrong, which caps how aggressively you can degrade the rest of the frame. If you guess wrong and the viewer looks straight at the part you threw away, they notice immediately.

The alternative is not to guess. Put an eye-tracker on the viewer, feed the fixation point back to the encoder, and adapt the frame while they watch it. That is foveation, and the idea is old — Girod discussed it back in the late eighties — but he considered it impractical because of the round-trip delay between an eye movement and the updated image. Thirty years of improvements to trackers, encoders and networks have quietly made it feasible.

FFoveated

I built FFoveated, a framework for prototyping foveated coding schemes, on top of FFmpeg’s libav* libraries (hence the ajar name). The goal was never a novel codec, but a rig where the whole loop — gaze capture, encoding, playback, and the user study around it — could be swapped and re-parameterised quickly.

A source (file or live) is decoded, re-encoded with foveation, and played back. The eye tracker closes the loop straight back into the encoder.

The awkward part is latency. A naive single-threaded decode-encode-decode cycle deadlocks itself on the very first frame, since all three codec instances want data nobody has produced yet. FFoveated instead splits reading, source decoding, foveated encoding, playback decoding and rendering across five threads joined by blocking FIFOs. The first two stages get generous 32-element buffers to absorb disk hiccups; the latency-critical stretch — from “encode this frame for where the eye is right now” to “show it” — gets buffers with a capacity of exactly one.

Fixation data reaches the encoder through FFmpeg’s side-data mechanism. I added a new AV_FRAME_DATA_FOVEATION_DESCRIPTOR type to AVFrameSideDataType carrying four floats: normalised x and y of the fixation point, the standard deviation σ, and the maximum quality offset δ. Nicely, this is the only codec-independent patch needed — and if you link against unpatched libav*, the wrappers just free the unknown side data and you get ordinary unfoveated encoding for free.

The Offset Function

The actual foveation happens in the x264 wrapper. H.264’s quantization parameter runs from 0 to 51, with higher values meaning coarser quantization and fewer bits. Rather than a hard region-of-interest rectangle, each macroblock gets a smooth offset added to its qp:

The offset is an inverted 2D Gaussian: zero penalty at the fixation point, saturating at δ in the periphery.

An inverted Gaussian is the natural shape given the circular fovea, and it has the nice property of inflicting no penalty at the fixation point itself. I fixed σ at 2.5° of visual angle, straight from the retinal characteristics — an educated approximation rather than a tuned optimum, since the mapping from qp to perceived quality depends heavily on content. That leaves δ, the peripheral penalty, as the single knob to push: how far can you crank it before someone notices?

For the codec configuration itself, real-time constraints dictate most choices: the ultrafast preset with zerolatency tuning (which, among other things, disables B-frames, since predicting from future frames is impossible on a causal source), a GOP limit of three frames to stop quantization error accumulating and to survive packet loss, and aq-mode 1 so there is a variance-based adaptive quantization pass to add the offset into. The output remains standard-conforming H.264 — any normal player will decode it.

Finding the Point Where People Notice

Rather than collecting mean opinion scores, the study hunted for each viewer’s just noticeable distortion threshold directly. Ten participants, ten source videos from the VQEG JEG Hybrid dataset (10 seconds each, 1080p at 25 fps), presented on a colour-calibrated UHD screen with an SMI RED250mobile tracker mounted below it.

The ten VQEG JEG sources — deliberately diverse: flat cartoon areas, fast action in small regions, high dynamic range, scene cuts and camera motion.

Each source was shown ten times in a row. Within a repetition, δ ratchets upward every few frames, so quality in the periphery decays continuously as you watch. The moment a participant sees an artefact they press a button; the repetition stops, a black screen flashes for a second, δ drops back by a set decrement, and the next repetition begins with a gentler ramp. Early repetitions climb fast and fall far, later ones creep — a repeated linear search converging on that person’s threshold for that particular clip. It also sidesteps the usual problem with subjective testing: nobody has to hold an abstract five-point quality scale in their head, they just have to say “there, I saw it.”

That produced 1000 video presentations and 734 interaction events.

63% Fewer Bits

The headline result: at the 10% JND — the strict definition, where only one in ten viewers reports visible distortion — foveated encoding used 62.76% fewer bits than the identical encoder with δ = 0. Averaged across the ten sources, 6980 kbit/s fell to 2652 kbit/s. At the more commonly used 25% JND the figure rises to 68.88%, though that number is flattered by the aggressive δ ramp early in each sequence.

Frame from participant #4, source #2. The red dot marks the fixation point; the court around it stays crisp while the ranks and the foreground crowd dissolve into macroblocks.

The frame above is the whole thesis in one image. All the action is confined to the left side of the court, the viewer is locked onto it, and the stadium ranks — which occupy most of the pixels — have been quantized into mush that nobody looked at closely enough to notice. Sports content like this is close to the ideal case; the savings are content-dependent, ranging from 54% on the busiest clip to 71% on the most forgiving one.

For context, the comparable numbers in the literature at the time were around 63% (Arndt and Antons) and around 42% (Illahi et al., for cloud-rendered game streaming). Direct comparison is genuinely difficult, since anchoring on the JND was novel and the field had no common benchmark — but the picture is consistent.

What the Eye-Tracker Told Me About the Participants

The most unexpectedly interesting data was not the bitrates but the gaze paths. Compare two participants watching the same clip — a cheetah pacing back and forth:

Participant #1: fixations track the animal.

Participant #6: fixations sprayed across the whole frame.

The first is exactly what you would hope for: a tight band of fixations following the region of interest. The second is a participant who worked out what the experiment was measuring and went hunting for artefacts in the periphery — large saccades terminating repeatedly in objectively uninteresting background regions. That is not natural viewing behaviour, and it systematically drags their reported threshold down.

Which suggests something reusable beyond this study: eye-tracking data could serve as a post-hoc filter on participants in quality assessment databases, separating people who watched from people who audited. As far as I know that had not been done in image or video quality assessment. I wrote that idea up separately, along with the observation that foveated coding gives you a strong prior on where a visible distortion must be — which in principle lets you infer noticeability from gaze irregularities alone, without asking anyone to press a button at all. The dataset here was too small to train anything on it, but the hook is there.

Where It Goes

Bigger screens and higher resolutions should push the savings further, since a larger share of every frame lands in the periphery — with encoding time as the likely bottleneck. Peripheral vision is disproportionately sensitive to abrupt contrast changes, so blurring the coarsely quantized blocks would probably buy additional headroom. And the lab dependency is the real barrier to scale: approximate webcam-based eye-tracking would be plenty precise for something this spatially forgiving, and would put the whole approach within reach of crowdsourced studies and, more to the point, of ordinary video calls.

The full thesis is available as a PDF. Parts of it were published as Foveated Video Coding for Real Time Streaming Applications at QoMEX 2020, and the participant-filtering ideas as Gaze Data for Quality Assessment of Foveated Video at the ET-MM workshop at ACM ETRA 2020.

Local NR-IQA using Convolutional Neural Networks

2018-05-05T00:00:00+02:00

Teaching a CNN to judge image quality one 64x64 patch at a time.
Perceptual quality has almost always been studied as a property of a whole image. My BSc thesis asked what happens if you zoom in instead.

Error Is Not Quality

The instinctive way to measure how badly an image has been degraded is to compare it to the original and average the squared differences. Mean squared error, and the peak signal-to-noise ratio derived from it, are still everywhere in the compression literature, largely because they are simple and have a tidy relationship to signal energy.

They are also close to useless as a model of what a person sees:

Every distorted version here has an identical MSE of 480 relative to the original. They are not remotely equal in perceived quality.

All three degraded birds sit at exactly the same distance from the original under MSE. Nobody would rank them equally. The metric is pixelwise and sign-independent, so it is blind to the structure that makes natural images legible — and, as the literature had already established, unreliable both across content and across distortion types.

Image Quality Assessment (IQA) is the field that tries to do better: predict the score a human panel would give, without convening one. The hardest and most useful flavour is no-reference IQA, where there is no pristine original to compare against — you get one image and have to judge it blind. That is the setting for anything real: photos uploaded to a platform, frames arriving over a stream, a compressor deciding how hard to squeeze.

One Number Per Image Isn’t Enough

Where I thought the field had left something on the table was locality. Essentially every method, classical or learned, collapses an image into a single scalar Mean Opinion Score — regardless of whether the photo is uniformly sharp, or a razor-focused subject sitting in a mushy out-of-focus background.

A global score cannot tell you where an image is good. Some methods operated on patches internally, but were only ever evaluated globally, and — more importantly — were trained on the assumption that an image’s global MOS is a valid label for every patch sampled from it. That assumption is obviously false for exactly the images where locality matters most.

So the thesis set out to do three things: build a dataset with genuinely local labels, train a no-reference predictor on it, and then check whether local predictions are still useful once you zoom back out to global tasks.

How Small Is a Quality Judgement?

First: what does “local” even mean? Two constraints pull against each other. A single pixel plainly carries no quality information, and small patches are hard for humans to assess at all. But a large patch starts smuggling in content — and once a rater can see what the picture is of, their judgement gets contaminated by whether they like the subject and how they think it ought to be depicted.

Content perceptibility against patch size. At 32x32 there is nothing to go on; at 224x224 — the standard ImageNet input — you are looking at a photo of a dragonfly and judging it as such.

I settled on 64x64 pixels, sampled from 1024x768 images: as small as possible while still assessable, and small enough to give away few content cues. It is an ad-hoc choice and I said so in the thesis — it is not defensible against every objection, but a decision had to be made.

Building KonPatch

No dataset of locally annotated patches existed, so the first contribution had to be one. KonPatch is 32,000 individually annotated 64x64 patches: 500 source images drawn from KonIQ-10k, 64 patches sampled at random locations from each. Those 500 images were then excluded from the rest of the database, so nothing downstream is ever evaluated on an image Patchnet partly trained on.

Annotation was deliberately cheap: a binary lab judgement per patch — does this look like it came from a high-quality image, or not? One vote each. That is a compromise, and it introduces a hard decision boundary into a phenomenon that is clearly not binary.

The trick that makes it workable is that every patch inherits a source image with a known global MOS. So the label becomes a continuous score: a patch flagged as high quality takes its parent image’s MOS, and a rejected patch takes zero. A binary study, scored continuously.

Flagged as indicative of a high-quality source.

Flagged as not indicative.

The distinction is intuitive even stripped of context: crisp edges, plausible focus and clean colour on one side; blur, noise and washed-out flat regions on the other.

Patchnet

Patchnet is a 14-layer CNN: seven convolutional layers with three maxpooling stages between them, the final feature maps vectorised into a fully connected predictor of 1024, 16, 8 and finally 1 neuron. ReLU throughout, except the output, where a sigmoid bounds the prediction to [0, 1].

Patchnet: 64x64x3 in, one quality score out.

Twenty percent of KonPatch was held out for testing, and the remaining 25,600 patches split five ways for cross-validation, with a separate training run from random initialisation per fold. Implementation was Keras on TensorFlow, Adam, batch size 512, trained on Nvidia K40s.

Validation loss bottoms out and turns upward after roughly 30 epochs. Rather than adding dropout to a network this small, I used early stopping, keeping the best-performing checkpoint per fold. Those five models land within a narrow band of each other on the held-out test set: MSE between 0.059 and 0.071, SROCC between 0.649 and 0.678.

Taken at face value, an MSE of 0.06 on labels spanning [0, 1] looks poor. It isn’t quite what it seems: because of how the scores were constructed, the label distribution is two well-separated clusters — rejected patches at zero, accepted patches up near their source MOS — with very little in between. The distance between the clusters is large compared to the error, so the model is not straddling the boundary the way that number suggests. The consistency of the early-stopped models transferring cleanly to unseen test data was the more meaningful signal.

From Patches to Maps

A patch scorer becomes something more interesting when you slide it. Run Patchnet across a whole image with stride δ and you get a quality map — a spatial readout of predicted quality instead of a single number.

Input (a), Patchnet’s quality map (b), the FISH wavelet sharpness baseline (c), and plain luminance (d). Patchnet separates the sharp animal from the softly blurred surroundings.

Do the Maps Match Humans?

Eyeballing a map and declaring it good is not evidence. But since no local quality benchmark existed, I had to build the yardstick too: 125 fresh images from KonIQ-10k, 30 participants each, asked to draw bounding boxes tightly around regions they considered high quality — or tick a box saying the image contained none.

Thirty participants’ bounding boxes (left) become a ground truth quality map (right) after normalising by participant count and smoothing.

Each pixel’s score is simply the fraction of participants whose boxes covered it, with repeat boxes from the same person counted once so overlapping selections cannot manufacture peaks. Because it is normalised by the number of participants rather than within the image, the resulting map expresses absolute quality, not just relative differences — an image where nobody found anything good stays dark everywhere. Rectangles are a crude segmentation primitive, so the raw maps get a Gaussian smoothing pass with σ at 10% of the image width.

Scoring predictions against this is not a straightforward MSE, since the two datasets are labelled differently. Instead I binarised the ground truth at a threshold, swept the prediction threshold, and measured the area under the resulting ROC curve — repeated across the whole range of ground truth thresholds. Median AUC sits around 0.9, with the lower quartile around 0.8 across the entire threshold range. There are genuine failures in the tail — individual images where the model does worse than chance — but the central tendency is clear: Patchnet’s local predictions track where people actually see quality.

Zooming Back Out

If local scores are meaningful, they should also say something about global quality. I ran the sliding window at stride 4 across the 9,500 KonIQ-10k images that KonPatch never touched, producing maps at about 5.5% of input resolution.

The crudest possible aggregation — take the mean of the map — already reaches SROCC 0.667 on KonIQ-10k, against 0.560 for FISH, the wavelet sharpness metric applied the same way. Neither is a strong correlation, which is expected when you flatten a whole map to its average, but the ordering is informative.

Doing it properly meant training a second model on the maps: a headless DenseNet-169 fed three stacked spatially-small inputs — the Patchnet map, a FISH sharpness map, and a downscaled grayscale image — with the classification head replaced by a 1x1 convolution and global maxpooling, so the model accepts any input resolution. With rotation and flip augmentation the 5,700 training images became 64,400 feature maps at 82 distinct resolutions.

That meta-model reaches SROCC 0.79 / PLCC 0.81 on KonIQ-10k, comfortably ahead of the classical no-reference metrics of the day — BRISQUE at 0.70/0.70, SSEQ at 0.59/0.61, BIQI at 0.54/0.61 — and ahead of the earlier patch-based CNNs, KangCNN (0.63/0.67) and BosICIP (0.65/0.67).

It does not, however, catch DeepBIQ (0.90/0.92) or DeepRN (0.92/0.95). That gap is the most interesting result in the thesis, because it falls along a clean line. Everything below it, mine included, builds a global score by aggregating predictions over spatially small regions. Everything above it sees large regions or the entire image at once, transfer-learned from ImageNet classification models. The obvious reading: perceptual quality is a mixture of local, technical properties — sharpness, noise, compression artefacts — and global, content-dependent ones to do with composition and aesthetics. No patch, however cleverly chosen, can see the second kind. Deliberately throwing away the big picture costs you exactly the part of the judgement that depends on it.

Application: Letting Quality Maps Steer JPEG

The last chapter puts the maps to work. My group had previously built a variable-quality JPEG coder that adjusts quantization per 8x8 block according to a saliency map — with the significant catch that the saliency maps came from eye-tracking or crowd studies, one per image. That does not scale.

The idea here was to substitute Patchnet’s quality maps for those human-generated saliency maps, on the hypothesis that high-quality regions and salient regions largely coincide: in a well-shot photo, the subject is in focus and the background is not.

VarJPEG: predict a map, prune to the strongest 10%, cluster to 8 centroids, blur, and modulate per-block quantization with the result.

The clustering step is not cosmetic. Storing a per-block quality map inside the file would eat the bit budget it is supposed to save, so the map is pruned to its strongest 10%, reduced to eight k-means centroids, and reconstructed at decode time by blurring those centroids back out — meaning the file carries eight coordinate pairs rather than a full map.

To find out whether it actually helps, I ran another crowd study on the same 125 images. For each source, a binary search over the global quality parameter produced 100 compressed versions with evenly spaced bitrates, and participants dragged a slider to degrade a test image beside the unmodified reference until they first noticed a difference — the just noticeable difference, 15 votes per image, with a standard JPEG run as the control.

It was a wash. No consistent bitrate win emerged, and at very low bitrates VarJPEG was systematically worse — the fixed centroid overhead is a larger share of a smaller file. There was no clear trend against overhead share, source bitrate, or MOS either.

The likely explanation is a good lesson about evaluation design. VarJPEG deliberately starves the background, so at equal average bitrate it puts more artefacts there than plain JPEG does. The interface then made that worse: dragging the slider makes the first distortions appear as a flicker, and they appear precisely in the regions the algorithm sacrificed. Participants were reporting distortions honestly — the task just pointed their attention at the background, which is exactly where the method trades quality away. The measurement was aimed at the wrong part of the frame.

Looking Back

The core claim held up. Quality is meaningfully local, a CNN can learn it from small patches with genuinely local labels, and those local predictions carry real information even when the eventual task is a global score. The parts that did not pan out were the more instructive ones: the gap to whole-image models pointing at what patches structurally cannot see, and the VarJPEG study showing how easily a subjective experiment can measure the wrong thing.

The full thesis — architectures, training curves and the complete correlation tables — is available as a PDF. Parts of this work were published at QoMEX 2018 as Disregarding the Big Picture: Towards Local Image Quality Assessment.

Graphcx

2017-05-08T00:00:00+02:00

A collection of graph analysis and drawing algorithms

Galaxy layout of tree82inc.

History

I began working on graphcx as a project on genetic algorithms at the University of Limerick in 2016. The mandatory implementation was largely extended with a flexible OpenGL based visualization module and bunch of interesting algorithms.

The probably most interesting part is an implementation of the Sampled Spectral Distance Embedding algorithm, which was presented by Çivril, Magdon-Ismail and Bocek-Rivele at the 14th International Symposium on Graph Drawing, Karlsruhe, 2006. http://dx.doi.org/10.1007/978-3-540-70904-6_5

The euclidean distances between vertices in a SSDE layout approximate the underlying graph-theoretical distances. An obvious application is to combine SSDE on a global scale with a further local optimizer (e.g. a force-directed algorithm) to speed up the drawing process and circumvent bad local energy minima.

About the Project

Written in Java, depends on la4j and jogl.
Build system / dependency management: gradle.
Source: GitHub

Functionality & Algorithms

Genetic optimization by simulated annealing:
- Grid or circular layout
- $\sum\limits_i ||e_i||_2$ or #EdgeIntersections as fitness criteria.
Galaxy: A straight-forward force directed implementation. The name is due to the typical shape this algorithm produces.
Fruchterman-Reingold algorithm.
Dijkstra shortest-path.
General graph handling, representation and conversion.
OpenGL based visualization which nicely depicts optimization progress.

Todo

Add rudimentary interface to avoid recompiling b/c of parameter changes.
Port OpenGL to Vulcan just for fun and out of interest.
Add more algorithms.

Sample Layouts:

Simulated annealing grid layout of tree82inc, using $\sum\limits_i ||e_i||_2$ as a fitness criterion.

SSDE layout of tree82inc.

Grid layout of matrix1inc, using $\sum\limits_i ||e_i||_2$ as a fitness criterion.

Galaxy layout of originalinc.

Traceu

2016-08-10T00:00:00+02:00

Piecewise-linear tracing utility for implicitly defined curves

A trace of $H^{-1}(0)$ for $n=10$.

Example: Fixpoints

Let $ g: \mathbb{R}^n \rightarrow \mathbb{R}^n, g_k(x) = \exp\Big( \cos\Big( k \cdot \sum\limits_{i=1}^n x_i \Big)\Big) $ for $k = 1,\ldots,n$

Considering the homotopy

$$H:[0,1] \times \mathbb{R}^n \rightarrow \mathbb{R}^n, (\lambda,x) \mapsto x - \lambda g(x)$$

it is obvious, that $H(0,x) = x$ and $H(1,x) = 0 \Leftrightarrow x = g(x)$ One approach to find fixpoints of $g$ is therefore to follow $H^{-1}(0)$ starting at $x=0$ until $\lambda = 1$ holds. traceu provides piecewise linear approximations of such curves with upper bounds for the error enabling further (e.g. newton-type) methods to be successfully applied on the results. The picture above depicts a trace of $H^{-1}(0)$ for $n=10$.

About the Project

Still beta, development is quite stale.
Written in C.
Quite unrestrictive regarding input requirements (e.g. no derivability needed).

Todo

Multithreading.
An input parser to get rid of recompiling for every problem.
A GUI would be nice, something written in ncurses would do perfectly fine.