This is Dan P.'s Typepad Profile.
Join Typepad and start following Dan P.'s activity
Join Now!
Already a member? Sign In
Dan P.
Recent Activity
I believe the image ascribed to me was from Chuck Yokota! Here's another one from the thread: N. pole is marked by a green circle 5 km in diameter.
Toggle Commented Aug 22, 2013 on Hole at Arctic Sea Ice
Wayne - having been staring at MODIS images around the pole for quite a while, it may be true that the pole is in water as little as 50% of the time right now. But that is because of a relatively small area of low concentration ice. Just away from the pole towards the western arctic there is obviously a solid 100% ice area. The high latitudes and scales of these low concentration areas seem to be unprecedented but I'd like to get my composites working well enough to make fair comparisons year-to-year. I agree that microwave seems to be being fooled at times by clouds right now, but I also take Peter Ellis's comments to heart that we're all pretty bad at judging concentrations by eye, as well as the fact that our eyes are all naturally drawn to the high contrast low concentration areas. So it's hard to make any quantitative statements about how much net area the microwave data might be mistakenly counting.
Toggle Commented Aug 22, 2013 on ASI 2013 update 7: cold and cloudy at Arctic Sea Ice
8-day composite through yesterday (day 225-233), via the new method. Longer runs reveal occasional glitches in the selection routine where an unusual cloudy square has managed to beat a clear one in my selection criteria, so I'll do some more debugging of that. But it's already quite useable at the 100% zoom although pretty ugly due to brightness/color variation from the blockiness. Because the underlying data I'm using is surface reflectance (corrected for sun angle, etc.), if I were able to choose entirely clear scenes, there should be no color or brightness variation when crossing mosaic squares. In practice the correction algorithms are never perfect, and of course unremoved clouds will always be a distraction. Channels 1+4+3, 12% scale: Channels 7+2+1: 1+4+3, 33% crop: 1+4+3, 100% crop:
Toggle Commented Aug 22, 2013 on ASI 2013 update 7: cold and cloudy at Arctic Sea Ice
There may be as many as 3 reasons these composites have better local detail than the other 8-day product I've been spitting out: 1. The previous 8-day averages are from tiles that had already been projected to a sinusoidal grid; though this is mathematically invertible, there are inevitable resolution losses in the areas with largest distortion, i.e. near the pole. 2. NASA's compositing algorithm *may* be averaging multiple swaths, which will inevitably smear movement detail. The documentation has left me somewhat confused on this point. In any case, my method is displaying actual unaveraged swath crops, which are merely chosen to be the clearest of the available ones. 3. NASA's composite is pixel-by-pixel, which means that neighboring pixels can be chosen heterogeneously across the time interval, which also effectively smears any motion. #2 and #3 don't matter for most of the planet, and in fact will generally enhance detail on vegetated areas than don't change quickly, by averaging down noise due to varying light conditions, etc. But they make a hash of the moving Arctic ice floe details.
Toggle Commented Aug 21, 2013 on ASI 2013 update 7: cold and cloudy at Arctic Sea Ice
Keeping the MODIS data downloads up-to-date (it's 18 GB a day for the whole Arctic!) is always the challenge but hopefully I'll toss out a few more images/movies. And as a teaser, here's a new product I'm making that's almost ready for primetime: This is a 2.5-day mosaic I constructed directly from the .hdf swaths by means of dividing into 105-pixel squares, and then choosing the observation swath for each square with the clearest view, as determined by a whole host of criteria such as (band1)/(band6 + band7) ratio for apparently icy pixels, min(band1 + band3) for water, altitude of sun, etc. etc. Full resolution is 500m/pixel. It is quite blocky for cloudy periods but it already looks much cleaner at a pixel level than my 8-day composites based on NASA's processed tiles, which smear out features that are changing in time. In clear periods and during higher-sun times of year it may be useful even as a daily composite. Right now I'm pulling down a few more days of data to see how good it looks with a few more days composited together.
Toggle Commented Aug 21, 2013 on ASI 2013 update 7: cold and cloudy at Arctic Sea Ice
From 2007 to 2012 the Eurozone population grew at a 0.3% rate according to Over the same period according to the Census the US grew at a 0.9% rate. So, a 0.6% rate difference, applied over 5.5 years of comparison, should give a 3.3% difference in GDP. In other words, half of the difference is explained by the fact that the US population is growing faster than the Eurozone's. This has been the case with pretty much every chart of this sort I've ever seen posted comparing the US and European/British/whatever recoveries. Why don't these comparisons over time ever correct for differing population growth rates?
1 reply
By the way, Brad's excerpting hasn't changed any of the vileness of the screed but it did make the last bit a little less comprehensible. If you go to the original, you'll see that the extra text fleshes out a bit the way in which this is a "ha ha only serious" speculative exercise.
1 reply
I thought the last aside (no really, I'm kidding ... or am I?) was pretty obvious, but clearly it wasn't so, or else I misunderstood it. The previous paragraphs were OSC ranting about Obama shutting down any dissenting views whatsoever, so I thought it was pretty obvious that the last paragraph was a "haha Big Brother, I don't really think that, of course not, wink wink!" that was not at all really meant to negate what he wrote beforehand. This is his blog, after all, not some kind of too-cute piece for the above-it-all intellectuals out there. Everyone who already is steeped in right-wing paranoia understands that particular convention. It's part of the left-wing paranoid phrasebook too for that matter.
1 reply
Chris: I doubt there's anything special that happened in Utah. A single state's unemployment statistics will naturally be noisier and occasionally have bigger jumps due to random fluctuations or measurement difficulties than the statistics for the whole country, where these fluctuations have more chance to average out. Even if the sudden change was real (there was after all a huge financial shock going into 2009), if it hit at slightly different times in different states, that would soften the quick increase in national statistics.
1 reply
That was a remarkable display of hippie-punching. If you're through, maybe we could work together to fight the policies we apparently agree are despicable.
1 reply
you know what I really hate? those shushers at orchestra concerts. After all, any modern composer today should be delighted if their music eventually makes it into being background music played in a mall somewhere like all the historical greats. So who cares if I'm listening quietly to my mp3 player during the concert, since that's totally less distracting than being in a mall anyway? Honestly, this is a really stupid argument for exactly the reasons that others have already stated. It shows amazing, and I've got to assume willful ignorance of the social space and unspoken (and even explicit) norms that have been created in the movie theater and how you're violating them by your behavior. The comments Ryan made show this perfectly: when a crowd manages to semi-democratically agree on different rules for the social space (cheers, snarky comments, whatever) it can work just as well as silence. Part of human intelligence is learning how to navigate these situations, and I understand that not everyone is great at it. But most don't go around telling the whole internet how bad they are at it.
1 reply
Kevin: if you need doggerel, I can provide. If it makes you feel better, I'm already stuck singing it. Look down, look down, From high up in the sky Look down, look down, And watch the sea ice die. [ice floe #1] The sun is strong it's hot as hell below [sea ice] Look down, look down, There's not ten years to go. [ice floe #2] The heads of state They'll know what they must do! [sea ice] Look down, look down, They've all forgotten you [ice floe #3] When I break free, you won't see me South to Fram! [sea ice] Look down, look down, Through roiling sea and wave, Look down, look down, You're melting in your grave. [Arctic Oil Driller Javert] Now bring me tile r04c04 The summer's up And your parole's begun You know what that means [ice floe Valjean] Yes - it means I'll freeze [Javert] No! It means you get one winter of reprieve you are a dangerous floe. [Valjean] I blocked one rig A polar bear was near death And we were melting [Javert] You'll melt again! Five years and we'll be back No feedbacks left in store r04c04 [sea ice] Look down, look down …etc.
Major breakups happening in the Parry channel over the last 2-3 days. 4-day animation ending at the end of day 207 (July 26), extracted from MODIS Terra swaths channels 1+4+3, 500m resolution.
Toggle Commented Jul 28, 2013 on Second storm at Arctic Sea Ice
Dan - it is a true color image made from red, green, and blue visible bands. I agree that it does show nicely the difference in albedo between the more and less fractured ice, but there are some cautions to reading the image. For example, the brightest white areas over the ice are generally contaminated by clouds; even though this is an 8-day mosaic and NASA's declouding algorithm attempts to choose low-cloud swaths, some areas were consistently cloudy. The clouds are better distinguished using IR band combinations like this one: (MODIS Terra 7-2-1) R: ch 7, 2100 nm IR G: ch 2, 857 nm near IR B: ch 1, 646 nm visible red
Toggle Commented Jul 25, 2013 on Second storm at Arctic Sea Ice
It's worth noting that all this discussion is about a third order effect: differences among regions in differences between parent's and children's circumstances. This is important and interesting research, but it's worth noting that even in better locations social mobility is still low, and there may still be plenty of poverty. Figure A4 of the whitepaper shows how unlikely you are to reach the top 1% if your parents were not in it. For those of us primarily interested in actually eliminating poverty, social mobility is important but not necessarily the first order of business. That doesn't mean it's not really valuable to know where intergenerational poverty seems most entrenched, and start figuring out its causes. But we shouldn't be complacent about absolute poverty in the light-colored areas on that map.
1 reply
For fun, here's a faster animation to complement A-team's longer one. This one lasts 36 hours; you can see from the motion and the timestamp that the frames aren't evenly timed since satellite passes are irregular. Channels 1+2+6 as RGB, processed from individual TERRA swaths. Also note the orientation is rotated a bit from DMI's version. George, you are right that in both our animations images north is close to straight up. Someone else might be better at explaining why the ice decided to rotate off in that direction, but from looking at the small pieces nearby it looks as though the currents near shore have a lot of local variation.
Toggle Commented Jul 20, 2013 on Ice pack in full at Arctic Sea Ice
Greenland sea has only recently truly cleared out of clouds, but here's an animation over the last 5 days or so where you can see the fast ice breaking apart along with the floes coming down through the Fram highway to oblivion. 500 km/pix resolution, MODIS 1+4+3
A-team: 1. The original images were all 16-bit, and in general made fair use of that dynamic range. A typical histogram had a minimum about 50, median around 2000-10000, and a long tail of brighter pixels. My linear rescaling to 8 bits doesn't always do justice to the contrast of features we might be interested in, but after a fair bit of experimentation I realized it would take a lot of effort to come up with a better generic solution for display. Often it is both the low and the high brightness features that need better contrast (obscured icy water in IR channels and cloud tops), which requires splitting the range into different functional forms. 2. For the apparently useless channels 11-18, as I said the problem is already there in the 16-bit file, although it is true a carefully optimized palette could extract a bit more contrast. But essentially to the extent the problem was information being discarded too early (rather than detector faults), it was by having the gain set too high for that sensor pipeline. 3. My colorspaces for all but the last one are direct RGB combination. The 1-2-6-20-26-32 was just an experiment that I expect to simplify and improve on, so I can only give you an approximate scheme: (ch_1)R + (ch_2)G + (ch_6)B + .37(ch_20)M + .44(ch_26)C + .44(ch_32)B I don't have a lot of expertise on color spaces, so the way I'm thinking about it is that I have a fundamental 3-D basis with some alternate conventional basis vectors. I can get away with adding extra channels above 3 because the eye can use spatial information to separate out coherent channels even if their color information overlaps, but it still seems like it would be better to keep the # of channels to no more than 4 or 5. Fufufunknknk: All of your points have some reason to them but I still lean towards "it would be a lot better if someone professional were doing this work and making it available rather than me". The answer to #2 is basically "haters gonna' hate", so you shouldn't expect policy to respond to whatever idiocies they spout. As for #1 and #3, they make a strong case for having the raw data available *in addition* to any higher level products, with careful descriptions of the methods used to get them. I believe NASA does well in this regard for many of the products they produce, but there is a disconnect that A-team has been stressing when it comes to easily-readable imagery. In many cases this imagery seems treated as a display product only (despite the fact that its universality generally means it will be an important input for a broader range of science than any other data product). I have a milder point of view than A-team about the persons involved in the production, even if I'm dissatisfied with the results of the overall system. I've run into comments in documentation several times with statements like "we have schemes for alternate methods of presentation and are seeking funding to implement them", written years ago, like a little cry for help from the data processing factory.
Toggle Commented Jul 9, 2013 on So, how slow was this start? at Arctic Sea Ice
I love the stacked time series, A-team. Here are all 36 MODIS channels in their glory, from a single exposure about a week ago in the much-looked-at area of broken ice in tile r04c04 about 400 km from the pole. I have slightly reordered the channels by wavelength order. Channels 1-2 have 250m resolution, channels 3-7 500m resolution, and 8-36 have 1km resolution. I reduced the animation to 1km pixel resolution to match all channels. There are important decisions in reducing from the 16-bit data to an 8-bit image as the dynamic range of every channel easily exceeds 8 bits. Most of the raw image histograms have a long tail of outlying bright pixels, so a simple rescaling/rebinning to 8 bits will result in a very dark image. Moreover the mean for each channel varies dramatically. I chose to linearly rescale, preserving all minimum data (subject to binning of course) but with a crude maximum channel-by-channel cutoff of mean + 2 standard deviations, which in practice clips to white significantly less than the 2.5% of pixels a gaussian would since the distribution is so skewed. Recall the usual visible RGB product is channels 143, with the combinations 367 and 127 of great use in distinguishing cloud from ice. Channel 5 and channels 8-36 are not used in any ordinary imagery product I've seen online though they are inputs to specialized higher-level products (e.g. vegetation maps). You can see that not every channel is equally usable. In fact a number of the visible channels (e.g. 10-15) are almost completely overexposed. This is not an artifact of my bit reduction as the original 16-bit images also have most pixels nearly 65535. Perhaps these channels are extremely sensitive and will be usable in lower light, or maybe there's something wrong with them. In any case for true night visible images Suomi's VIIRS instrument (already putting out uncalibrated data) will be superior. Anyone should please feel free to extract the channels as frames from the animation and suggest useful combination methods. Now for fun, here are a few spectral combinations I've tried: true color 143: 367 visible + IR: (note my channel 7 image has some hot pixel problems) 126: my equivalent to the 127 combination: 156: making use of an intermediate IR channel that isn't part of any of the usual image products 1-2-6-20-26-32: I took the 126 image and then added portions of the latter three channels in cyan, magenta, and blue. 26 is confusingly in near IR between 2 & 6, but 20 & 32 go much deeper into IR. The additional information helps distinguish cloud heights from each other (cirrus, etc.)
Toggle Commented Jul 9, 2013 on So, how slow was this start? at Arctic Sea Ice
I'm finally getting my MODIS swath pipeline sorted out as the melt is getting more exciting. Here's about 30 hours of swaths from covering the Arctic for Terra (Aqua has equal coverage on different passes). This is about 7 GB of swath data I downloaded, covering 00:00 UT July 6th - 07:00 UT July 7. The original resolution was 500 m/pix, and the animation does look beautiful at 4000 pixels wide. Adjacent swath images are 5 minutes apart in time. You can see that a small area around the north pole is covered on every pass, while areas further from the pole are still covered quite frequently compared with the 2x/day or so at the equator. Here is a subview at 500m resolution. The pole is marked, along with image timestamp. This isn't the most interesting part of the melt yet, but do note that the view is only 200 km wide, so that little rotating bunch of floes in the upper center is only 100 km from the pole. I can extract similar animations without much trouble, but the amount of upfront data download has remained cumbersome. Obviously NASA has done calibration beyond level 1B of the swath data to correct for solar angle, extinction, etc., which is why this mosaic will never look as pretty as earthview's. But I can get high temporal resolution, and use various cloud-free combinations more flexibly. I also have available the full range of wavelength bands into deep IR, which could give more useful cloud removal than yet seen (as well as better low-light images).
Toggle Commented Jul 8, 2013 on So, how slow was this start? at Arctic Sea Ice
First post from some new home brew image processing. As a polar orbiter nearly every one of MODIS's 17-18 daily orbits takes images the north pole: the poles are the most heavily-surveyed area of the planet for these high-resolution multi-band instruments! However the various public products are not geared towards the Arctic and woefully underrepresent this high rate of data, leaving us in the Arctic with just a poor daily mosaic of badly stitched pieces of various cloudy passes. Here's my first alternative: simply paste together every one of those swaths and let the eye make some sense of it. Here I have used channels 1-4-3 to approximately reproduce the MODIS "true color" composition. I chose a region centered around the pole; the width is 400 pixels and the pixel size is 500m, which is the top resolution of all but channels 1-2 of MODIS (I do intend to process those at 250m eventually). The animation is 16 frames from a single day (June 25th, chosen simply because I happened to have downloaded all 6 GB of the swaths covering the pole that day). Not every swath covers even this small frame, so there is a bit of persistence to portions of the animation. It was a formidable task getting up to speed on processing the .hdf files and particularly learning enough to get georeferencing around the pole to work properly for image alignment. But most of the pipeline is automated now, so I can start producing better products. I've had trouble producing a version of the 356 that looks as good as NASA's, but I've also produced some other band combinations that are intriguing (and I have access to all of them now if you have any requests!)
Thanks logicman! I had run across both of those links but hadn't spent enough time with the first one to see that it did actually have full resolution images from each swath. Sadly, the fact that they're JPGs in a random stretched projection (presumably oriented along the orbit) makes the problem of combining them hard enough that I'd rather just learn how to do the same with the original HDF files. Plus I think there's something a little buggy in how they projected the data, as you can see lots of ugly artifacts near the pole when you zoom in. However, if I'm trying to grab a few swath images or their corresponding data files, this is the best place I've seen to do it.
Does anyone know of a source for individual satellite swath images for the MODIS Terra/Aqua images? NASA composites these swaths into the mosaics we're used to staring at, complete with annoying stitching problems. These problems are especially severe at the poles. But in reality this is merely because of NASA's choice of stitching algorithm! Because the satellites are in polar orbits, the poles are actually the most densely sampled parts of the globe. Although the orbits vary off the poles, with 2300-km wide swaths they should capture the pole on every pass (every 90 minutes). The choice of stitching is sensible enough when nothing interesting is going on at the poles, but it is killing us right now. Imagine if we could combine the cloudless parts of 16 full daily passes in the polar region rather than being stuck with a single badly stitched mosaic! You can see the improvements when you do this yourself with a stack of a week or more of cloudy images (use e.g. "darken only" layer mode for a quick & dirty approach in gimp). NASA uses some version of this technique for some of its MODIS products, though I don't remember ones geared towards the Arctic. I waded around downloading the raw .hdf data files for passes, but they are huge (10-150 MB depending on which product) and the learning curve is steep for producing an image. As a side note, A-team is right to push us towards the 3-6-7 images, which I think are CMY channel composites. The infrared really helps distinguishing cloud from ice compared with the visible. But it seems like there's room for improvement, as the satellites have 36 total channels ranging well into infrared:
It was fun. Unsurprisingly there were no papers doubting AGW in my bunch. However there were 2 papers that were refuting half-baked alternatives and one examining the disconnect between science and policy, so AGW deniers can look on the bright side: their efforts are occupying a lot of time and space in the top journals!
Absolutely top notch Andy. You really brought your talents together exceptionally and the work clearly paid off. I shared it and likewise hope it goes viral.
Toggle Commented Apr 27, 2013 on Ice cube volume video at Arctic Sea Ice