Fast Fourier Analyses and Wavelet Analyses

[Chris Wolkersdorfer]

discussion moved from other Topic

Christopher Johnson

Postby Chris Wolkersdorfer » 2009-09-25, 22:47
Welcome Christopher,

what did you do with wavelet analyses and mine water?

Kind regards

Chris

PS: looking forward to seeing you next year!
Towards the “Zero Waste Mine”.

Re: Christopher Johnson

Postby CJ@RESPEC » 2009-09-26, 12:09
Hello Chris,

I have used wavelet analysis with mine water problems in a few different ways. I sometimes use it for cleaning up data sets and removing noise. In some cases, traditional filtering or cleaning of data (e.g., moving average) wipes out important detail that wavelet analysis sometimes can maintain while still cleaning the data. I also use it for decomposition of signals. For example, wavelet analysis allows me to extract earth tide responses from aquifer pressure data.

Regards,
CJ

CJ@RESPEC


Re: Christopher Johnson

Postby Chris Wolkersdorfer » 2009-09-26, 12:33
Dear Christopher,

why don't you use Fast Fourier Aanlyses instead? Which Software do you use?

Chris

PS: I am not questioning that wavelets are an excellent way to do things - I just wondered why you used the more complicated way to do things.
Towards the “Zero Waste Mine”.


Re: Christopher Johnson

Postby CJ@RESPEC » 2009-09-26, 13:59
Good question! Fast Fourier transform were tried, but wasn't as successful. Fast Fourier transforms doesn't "see" the data in time series and is the advantage of Wavelet Analysis (in my opinion). I use Matlab and its add in Wavelet Package.

Have you had good experience using Fast Fourier transforms and have you used Wavelets to do the same thing?

Cheers,
CJ

CJ@RESPEC


Re: Christopher Johnson

Postby Chris Wolkersdorfer » 2009-09-27, 19:18
Dear Christopher,

did you remove a trend in your time series? Did you normalize your data. Both procedures usually help. Wavelets analyse the time series time dependat, while FFT does not. In most cases it is not necessary to analyse fluctuations time dependen, therefore I didn't use wavelets so far - which does not mean that I analysed my time series by using them.

I use the Software Autosignal - excellent for all types of time series.

Glückauf

Christian
Towards the “Zero Waste Mine”.

Re: Christopher Johnson

Postby CJ@RESPEC » 2009-09-30, 09:45
Hello Chris,

I have tried removing trend, but haven't tried normalizing. When I have removed the trend, it has been on a data set with an apparent trend that is easy to remove. In some of the data sets I want to "clean", there are numerous different trends and slopes (formation pressure changes resulting from inflow increases and decreases) making it difficult to remove trend. This is where I use wavelet analysis to help pull out these complex trends. Usually, it is easy to see the general trend, but sometimes it is very important to know exactly when the trend changed which is where the difficulty comes in.

I will look into normalizing a data set.

Regards,
CJ

CJ@RESPEC

[romy]

Hi Christopher and Chris,

I have just been reading your discussion re time series analysis and I was wondering whether I could get your input on a problem I am facing at the moment.

I try to assess the trend of 5.5 years of load removal rates of a passive mine water treatment system (monthly measurements) by using SPSS. So far, exponential smoothing, seasonal decomposition and ARIMA have not helped because the data are highly heteroscedastic and non-stationary. Differencing wouldnt help and I am now stuck with a bad model with highly autocorrelated residuals.

any comments would be more than welcome,

very many thanks,

romy

[Chris Wolkersdorfer]

Hello Romy,

let's start with a number of questions. First thing: your data needs to be evenly distributed. That means the time between each data point must be exactly the same. In your case you would start with month 1, month 2 etc.

If your data is not evenly distributed you must start with curve fitting (e.g. spline, but there are better procedures available). Then you can start with the time series analyses. Of course, there are algorythms which don't need the even distribution - but by theory you need it. This also means that you have to find a model for missing data.

Let's assume you already made sure that the above is true for your data.

First of all: can you SEE a seasonality in your data? If not: why do you think you need a time series analyses? Of course, there could be some hidden patterns, but do you expect hidden patterns? If yes, what could they be?

You write you have heteroscedastic behavoiuror. What do you mean in your case? I couldn't think of a case where a time series with monthly data would show such a behaviour. If it does it would mean there is no seasonality. Did you plot histogramms? How do they look like? How is your data distribued? Log normal, normal, sqare?

Did you conduct auto-correlation of your data? Did you normalize your data before you tried to analyse it? Did you remove potential trends in your data?

All the above should be done before starting your analyses. Otherwise you might also deal with numerical problems and some of your "relicts" might be due to numerical issues.

Did you conduct wavelet analyses? Because you have 5 years this might be possible (in either case: normalization and even distribution is mandatory).

Kind regards

Chris

[romy]

Hi Chris,

thanks for getting back to me that quickly. I attach the removal loads of iron as an example to make the whole discussion a bit more authentic.

Re your comments:

1) yes, data are evenly distributed
2) I dont need the time series to discuss seasonality but rather to discuss long term trends. I.e. is the passive treatment system performing consistently throughout the years ? We saw strong seasonality in the influents, but because the system consists of 2 influents and 1 effluent and water is dispersed due to overflow (short-circuiting) and formation of pockets the patterns of the removal loads in the systems are hard to identify (see example figure).
3) The data is not normally distributed. thats why I normalized them by Blom's transformation (other transformations log, ln, etc wouldnt work)
4) The data is autocorrelated, yes. But differencing (1,2,3, ...order) to remove this autocorrelation wasnt helpful, but rather increased the autocorrelation.
5) Basically because the data is autocorrelated it doesnt make sense to continue with any further data analysis until this issue is resolved. I am stuck !
6) I have never heard of wavelet analysis before reading the convers. between you and Christopher.

thanks again for your comments !

Romy

[Chris Wolkersdorfer]