This is Rob Dekker's Typepad Profile.
Join Typepad and start following Rob Dekker's activity
Join Now!
Already a member? Sign In
Rob Dekker
Recent Activity
Please remember that Fort McMurray is the capital of the environmentally so destructive Canadian Tar Sand Industry. Somehow I find it ironic, that the capital of Tar Sands is being destroyed by wild fires spurred by global warming... Sort of like Nature strikes back, sort of thing....
Toggle Commented yesterday on EGU2016, my impressions at Arctic Sea Ice
Finally had some time for a first implementation of a 'spatial' fill-in algorithm. This first one is pan-Arctic. Specifically, for a particular month and year, find a 'analog' from the same month, in the 1953-2013 period, that best matches. Best match is defined simply as : the lowest area of grid points that differ in ice extent for the grid points that have an "observational source". With that simple "pan-Arctic" algorithm, I ran some tests. I started with August 1935. We know that Walsh' newest reconstruction has a problem with the Kelly fields, and we can put that to the test. Turns out that if we consider the Kelly fields as an "observational source", then August 1935 best matches with August 1961. The match is not very good (1961 and 1935 still differ by 1.5 M km^2 (where one observed ice and the other did not, or the other way around) but interesting is that the extent of 1935 with Kelly fields (8.98) matches quite closely with 1961 extent of 9.03. The same test of August 1935, but now disregarding the Kelly fields source, obtains a best fit with August 1953 (which ended up with 8.7 M km^2). That result suggests that the Kelly fields over-estimate ice-extent in Aug 1935, which we know is the case, because Walsh placed the July Kelly fields in August. So that gives some confidence that we are on the right track. Incidentally, I also tested Sept 1935, which is a notoriously difficult one to "spatial-fill", since only the Russian side (AARI) has ice-observations. If I ignore Walsh's spatial filling, and instead use this first simple pan-Arctic-match spatial filling algorithm, the algorithm finds August 2007 (of all years) to best match with September 1935's (AARI) ice observations. That is a curious find, since obviously September 1935 did not remotely come close to 2007's 4.3 M km^2. I think the problem is that this first, super-simple, algorithm considers ANY difference in ice-concentration (ice or no ice) as a 'difference'. So if in the year under test there is a big ice-flow at pixel X,Y, but in the reference year there is a similar ice flow at X+1,Y, it gets counted as TWICE the difference in extent. So this simple algorithm is too sensitive to local ice dis-placement, and also it is "pan-Arctic", which means that it does not attach much value to localized ice observations (DMI or ACSYS) but instead attaches much more value to 'field' observations (such as AARI). Next I have to find some way to 'balance' the different nature of these observations.
Neven, with all due respect, I don't see the text that A-team provided in the references he linked too. It seems to apply only to cruises in the tropics. Maybe that text was flowed out of sarcasm after all.
Guys, I think you are over-doing this one. As long as they don't leave a mess, this cruise is infinitely better than the REAL problems facing the Arctic. The drilling for oil in the Arctic, or the shipping routes across the Arctic, where vessels use high-sulfur fuel oil, or the digging for Tar Sands in the Boreal, with not just the immediate pollution this causes, but also the vast amount of methane required to turn that goop into something useful. In that context, I think that some Arctic tourism like this cruise is comparable to well-organized Safari trips in the Serengeti. It does not hurt wildlife, and it may help create awareness of the uniqueness of the environment.
Found this latest NSIDC announcement about the F-17 trouble on your site, Jim :
Toggle Commented Apr 13, 2016 on Beaufort under early pressure at Arctic Sea Ice
With this satellite glitch, it is interesting to see which reports use F-17 as their source, on Neven's Arctic Sea Ice graph page : NSIDC obviously, but also Arctic Roos (NORSEX) and DMI and Cryosphere Today. Bremen is unaffected (since they use AMSR2 ?). Also the regional graphs are affected : although noticable only for the Barents, the Greenland Sea, and possibly the Okhotsk. Now let's hope that NSIDC can quickly calibrate the F-18 source to take over. But needless to say that with only one satellite working properly, SSMI records, which so dilligently provided a continuous record over the past 30 years are now in danger of becoming dis-continuous...
Toggle Commented Apr 12, 2016 on Beaufort under early pressure at Arctic Sea Ice
Diablo, Regarding the spacial filling algorithm that Walsh used, the documentation states : If the surrounding months do not have data with which to fill the missing value for a given point or points, the field is compared to the ice concentration fields between 1900 and 2000 for the same month. How well a potential analog field matches the gap field is measured by spatial correlation for the sub-domain mapped out by the points that have data in the gap field. And incidentally I am implementing an algorithm that is very similar. Not done yet, so stay tuned, but I do expect to show some first results this coming week (time permitting).
Bill, I just read that article you pointed to. What a weird story by Michael Lavine. As if he deliberately wants to misinterpret what Naome Oreskes is saying. Picking on minor definition differences and missing the big picture : If you KNOW that CO2 warms the planet, and you OBSERVE that the planet is warming, then do you really need more than 95 % confidence that the whole thing is happening by chance ? And then that patronizing last sentence : Yes, most scientists are skeptics. We do not accept claims lightly, we expect proof, and we try to understand our subject before we speak publicly and admonish others. No, Lavine. Scientists do not expect "proof". Proof is for mathematicians and alcoholics. Scientists deal with "evidence". And Naomi Oreskes explained the significance of the "evidence" a lot better than your article. Sorry. Just had to get that off my chest. I'm good now.
Before we move to the 1935-1952 period, here are the results of calibrating the Walsh&Johnson source to the 1972-1978 ESMR data, for the month of August. For starters, the Walsh&Johnson source in August determines a significantly smaller portion of the Arctic than for September over the 1972-1978 period. Where in September Walsh&Johnson determines 75% of ice cover and 50% of ice edge (the rest being AARI), in August other sources (such as ACSYS and the Dehn collection) take over, and Walsh&Johnson is just used for the Canadian Arctic and Baffin Bay area. Also, the mean of the complete Walsh series for August 1972-1978 is quite a bit higher than ESMR (486 k km^2 w.r.t. Walsh' satellite series, and a whopping 816 k km^2 w.r.t. Sea Ice Index satellite series). To bring the 'mean' ice extent in line with ESMR, I had to reduce the ice-concentration-cut-off for extent to 32% (to match with Walsh sat era) and 53% (to match SII sat era). Also the standard deviation for the remaining differences with ESMR are not that good (224 k km^2 for Walsh-adjusted, and 287 k km^2 for SII-adjusted). Interestingly enough, the best match with ESMR in August is obtained if we determine the ice-edge for extent to be at a concentration of 25% in the Walsh&Johnson source. Standard deviation from ESMR is then only 186 k km^2. This tells me that probably the Walsh&Johnson source should be interpreted as having an ice-edge(for extent) of some 25%. That removes its high-bias for both September and August. It also tells me that one or more of the other sources (Dehn collection, ACSYS, and/or AARI) still has a high-bias that is not yet accounted for, But enough about these calibration issues. Let us move on the the 1935-1952 period, and the 'spatial infilling' challenges that will bring. I am working on implementing a first version of my 'matching' algorithm which should be able to find 'analogs' where observations are sparse. I intend to try that algorithm out first on August 1935, and see if we can do a better job there than the misplaced Kelly fields in the Walsh reconstruction.
Diablo, in summary : If you would like to (maybe for a future publication) make a very accurate reconstruction of the 1953-1978 period, beyond what Meier et al did, then we should tease out the calibration issues of the various sources (AARI, Walsh&Johnson, and also NASA yearbook source which made 1963 the highest of them all). But is you would like to focus on the 1935-1952 period, then we should attempt to construct a statistically sound spatial and temporal in-filling algorithm and validate it using the real ice observations over that period. Your call which direction you want to take it.
Regarding calibration issues, Diablo said : (I'm thinking that maybe the best approach to adjust both Walsh&Johnson and AARI sources could be testing them against ESMR gridded data: Of course, it is a more difficult approach. If we do this, we should also check whether ESMR maps match Cavalieri numbers, and if that is the case we should use for the satellite era Cavalieri's numbers themselves.) I agree that an approach using gridded data will be more accurate. And yes, it is also more difficult. Maybe we should keep this issue (calibration with ESMR) aside for the moment, since the uncertainties facing us in pre-1953 reconstruction (spatial and temporal infilling) likely will be much larger than the (minor) uncertainties we face if we want to accurately reconstruct post-1952 ice extent time series.
Hi Bill, I would very much welcome a visit by a statistics expert. If not only because the next thing we will look into is a statistically sound 'spatial filling' algorithm. In its simplest form, the problem of spatial filling looks like (still informally) like this : Given N number of 2D images on a grid of ice extent in the Arctic (the calibration images), and given M number of observations (ice or no ice) at M grid pixels, what is the probability of finding ice in the grid pixels that do not have observations. And subsequently, what is the probability density distribution of ice extent (mean and standard deviation). Diablo, I get your point about 'importing' the 'ice concentration bias' from the Walsh&Johnson source into the infilling that Walsh did for 1935-1952. That effect may be real, but to know if it is, we would need to reverse-engineer Walsh' infilling algorithm (which may be needed any way to validate that part of his reconstruction) and even more important : we need to know if Walsh used the Walsh&Johnson source as an "ice observation" or not when he ran his 1935-1952 spatial infilling algorithm.
Regarding that EMSR data, do you have the extent numbers for 72-78 for August as well ? Homogenized on NSIDC sat era, and/or Walsh sat era. I'd like to calibrate Walsh&Johnson for August, and see if the concentration bias is the same or different than for September...
Before we move on, let me summarize what we have found in the new Walsh series, so far : 1) The "Kelly fields" source seems to be misplaced one month. This makes the August fields look like July, and the July fields look like June, and it significantly affects these month's 'extent' reconstruction for the periods that the Kelly fields are used (1924-1956?). 2) The "Walsh&Johnson" source that dominates the 1953-1978 reconstruction suffers from a "concentration bias", which moves the Walsh reconstruction above reconstructions such as by Meier et al over the same period. 3) If the concentration bias in the "Walsh&Johnson" source is calibrated over the EMSR 72-78 data, it shows a better correlation if the EMSR data is used that is homogenized with the NSIDC Sea Ice Index satellite era than the satellite era that is currently used in the Walsh dataset. It seems to me that it is time to contact Walsh and make him aware of our findings so far, for future updates of his dataset.
Thanks Diablo. To calibrate to the second ESMR series (7.82,...,7.52) I need to change the extent-cut-off concentration in the Walsh&Johnson source to 25%. And of course the corrected graph then moves a bit closer to the original Walsh series (which used 15% for all sources). Also the standard deviation of the remaining differences goes up (to 217 k km^2). But since we are now deep into detailed calibration issues, shouldn't be calibrate the AARI to the correct ESMR 72-78 series as well ?
Since the WalshJohnson is the dominant source in the Waslh series from 1953-1978 (determines some 75% of ice cover), it is important that we remove any 'bias' from this source. To get the WalshJohnson source 'calibrated' to the adjusted EMSR observations from 1972-1978, I looked at two methods : 1) Subtract a fixed amount : over 1972-1978 original Walsh series is 450 k km^2 higher than the adjusted EMSR numbers Diablo mentioned above. This works to get the 'mean' of the series alligned, but the resulting standard deviation of the remaining differences is 247 k km^2. 2) Adjust the WalshJohnson source ice concentration for 'extent' from 15% to 35% in the gridded product. This also brings the mean in line with EMSR, but the standard deviation of the remaining differences is lower : 187 k km^2. So adjusting WalshJohnson directly for 'ice-concentration' bias gives a more accurate match with EMSR numbers from 1972-1978 than simple scalar adjustment of the series. That is a GOOD thing, because it means that the WalshJohnson source appears to be better matching regional and local ice cover than a plain pan-Arctic scalar adjustment. Here is the result of the WalshJohnson adjustment for the entire Walsh series : I'm not sure how the adjusted Walsh series compares to Meier et al and Diablo, and my (AARI-only) series, but at first glance it looks to me that all series are now better aligned with each other over the 1953-1978 period. Which is a very encouraging find.
Question for Diablo : When you compiled this graph : did you compile "Walsh adjusted to match ESMR" ? Did you subtract a scalar that best matches the Walsh numbers with ESMR numbers over the training period (1972-1978) or did you do something 'gridded' ?
Also, I wonder if we could use the Canadian Ice Service charts as a way to 'calibrate' the Walsh and Johnson source and quantify its "concentration bias". After all, the Canadian charts (some of which Diablo posted above, for 1971) were part of the Walsh and Johnson source in the first place AFAIK. And they cover some unique areas such as Baffin Bay and the Canadian Arctic, which are not covered by other sources. Table 1 does list the Canadian Meteorological Service. Are these the same charts ?
I had only 30 minutes today, but I ran a quick experiment to 'calibrate' the Walsh and Johnson source in the new Walsh series with the 1972-1978 satellite observations. So as to quantify the "concentration bias" we suspect in that source, and adjust the series accordingly. Turns out the best fit is when the Walsh and Johnson ice concentration 'extent' concentration is set at 35 % instead of the normal 15%. I'm a bit cautious to publish the resulting 'adjusted' Walsh series, since the fit is not very good, and I'd like to run some statistics on other methods of calibration (such as a simple scalar subtraction or replacing the Walsh and Johnson source with a 'fixed climatology' and calibrating that one. Stay tuned.
I so, Mahoney's decline in summer sea ice in the Russian Arctic during the 30's and 40's is side-effect of his method, which discards AARI charts that show only open water at a particular latitude. And the decline may be an indication that the Russian Arctic in the 30's had more open water than in the 50's. With these caveats exposed, I put more trust in your (and my) method of using a climatology at the back-drop to AARI observations, and Walsh' method of spatial infilling. Mahoney's method is just confusing everything.
Diablo, indeed that 1937 - 1949 September difference is stunning, and puzzling how Mahoney could have concluded that 1949 was smaller than 1937. I looked into the numbers for an explanation : Here is the breakdown in months and seas for 1937 : 1937 : Jul Aug Sept Summer Barents 0.46 x x 0.46 Kara 1.03 1.04 0.73 0.93 Laptev 1.11 0.93 1.06 1.05 ESS 1.36 x x 1.36 W. Chukchi 0.48 x x 0.48 TOTAL 4.28 and for 1949 : 1949 : Jul Aug Sept Summer Barents 0.56 0.56 x 0.56 Kara 0.79 0.59 0.75 0.72 Laptev 1.19 1.06 1.00 1.10 ESS 1.40 1.39 1.32 1.38 W. Chukchi 0.42 0.31 0.28 0.35 TOTAL 4.11 The 'total' number for each year (4.28 and 4.11) does seem to match figure 6 in Mahoney's paper. That's re-assuring :o) Note that the difference between these two years is not that large. For example, year 1943 clocks in at around 3.4, which would be a starker contrast with 1937. But all that aside, let us break it down by month, to find out where the difference comes from. For July, 1937 clocks a total of 4.44 and 1949 comes in at 4.36. So Mahoney found that at least for July, 1949 is lower than 1937. Can we see that in the AARI maps ? For August and September, for 1937 Mahoney reports only the Kara and the Laptev, and for September specifically, 1949 and 1937 are not that different in the Kara and the Laptev. Which kind of (with a bit of imagination) matches your animated gif. So, it seems that the big difference between 1937 and 1949 was in the ESS and the Chukchi, and for these two seas, Mahoney does not report data in August and September. Thus, I take my previous comment back. It appears (at least for the 1937 to 1949 comparison) that the difference between Mahoney's report and what we see in the AARI charts comes from Mahoney's omission of charts that do not show an ice edge. That happens more often for low-extent and late (September) charts, and thus his "uptick" in the 30's may be more an indication of low ice extent than of high extent as Mahoney suggests. What do you think ?
About Mahoney et al, I agree with you that his early records are suspect. But I don't think that the AARI charts that he omitted are the issue (of his find of declining sea ice extent). After all, he states in his paper that he did a lot of purging himself, which did not affect the result. The issue has to be grounded in the actual AARI charts that he DID use for his estimates of the 30's and 40's. I first thought that possibly his early record was biased towards earlier (July and August) AARI charts, since there are few September records available in the 30's. That would bring the 'average' up a bit. I noticed some of that effect in his record of the Chukchi sea and the ESS, but to quantify it I need to run some numbers. Either way I would like to get to the bottom of Mahoney's mystery declining sea ice extent in the early record of the Russian Arctic.
Diablo, thank you for these comparisons. That is very nice of you. Moving forward, I think we have a reasonable shot at reconstructing the 1935-1978 Arctic sea ice extent beyond what has been done already (by Walsh, Meier and you), but we would need to do a couple of things : 1) For the 1953-1978 period, we need to correct that "concentration bias" in the "Walsh and Johnson" source. For that we need to understand the nature and origin of the bias, and find a calibration period to homogenize it with other observational sources over the same period. I read the "Walsh and Johnson" paper in more detail, and I think the nature of their concentration bias originates from their EOFs. It seems they created these 'smeared-out' ice concentrations we see in their source by possibly (implicitly) exchanging ice-concentration for ice-probability. If that is indeed the case, then we may interpret a 50% concentration in these Walsh and Johnson fields as "a 50% chance of finding the ice edge here". I will run some tests over the next week to see if that is solving the issue. At the same time, I can run some calibration tests to see which concentration in the "Walsh and Johnson" source gives the best correlation with the 1972-1978 overlapping period. For that, do you have the September (and/or August) extent numbers of passive microwave satellite observations over the period 1972-1978, as homogenized with the modern satellite era numbers ? These tests and calibration and consequent adjustments to the 'cut-off' ice concentration for the "Walsh and Johnson" source will result in an 'adjusted Walsh' series (for September and August) for the 1953-1978 period that should be much closer to the truth. 2) For the 1935-1952 period : for August we really need to fix these Kelly fields that are misplaced by a month. I have some idea on how to do that, and I will run some experiments over the next week. Once we have both this issues fixed we can provide a 1935-1978 record that is 'homogenized' to the best of our abilities (and observational sources). It would be interesting to analyse the trend over that August reconstruction. However, for September 1935 - 1952, we will have to resort to some form of spatial and temporal fill-in, or resort to a 'climatology', which is much less trustworthy in my opinion.
I noticed that Diablo previously commented on Mahoney et al, with the following applicable arguments : - They left out western Kara sea (just in that area, there is more ice during 1946-1951 than during 1935-1945). - Their methodology could lead to an extent overestimation when data are sparse (by assuming the same edge for a whole sea). I'm not sure how much the western Kara has influence, but it's woth checking out using the Walsh data set. I'll do that. The second argument is very interesting, especially since the AARI dataset is sparser in the early (30's) record. Not sure why that would cause an overestimation however. Again, I have some more work to do, but until then, my AARI-based reconstruction over 1935-1978 stands.
What I'm kind of interested in now, is why did I find a nearly flat 1935-1978 record based on AARI observations, while Mahoney et al finds actually a DECLINE in sea ice extent over the same period using the same observations : See figure 6, Russian Arctic, summer graph. It can't be that I restricted my analysis to September only, since the August reconstruction shows the same thing : nearly flat for the 1935-1978 period. I guess I have some more work to do :o)