Fastest way to sum over rows of sparse matrix

I have a big csr_matrix(1M*1K) and I want to add over rows and obtain a new csr_matrix with the same number of columns but reduced number of rows. Actually my problem is exactly same as this Sum over rows in scipy.sparse.csr_matrix. The only thing is...
more »

2017-09-28 20:09 (1) Answers

Piecewise linear fit with n breakpoints

I have used some code found in the question How to apply piecewise linear fit in Python?, to perform segmented linear approximation with a single breakpoint. The code is as follows: from scipy import optimize import matplotlib.pyplot as plt import ...
more »

2017-09-14 14:09 (1) Answers

Trying to understand masking

I've been trying to understand masking and how it works with image filters. I'm using the following code to try to develop my understanding. import scipy.ndimage as ndi import matplotlib.pyplot as plt import numpy as np # Generate a random binary...
more »

2017-08-26 02:08 (1) Answers

Scipy Sparse Cumsum

Suppose I have a scipy.sparse.csr_matrix representing the values below [[0 0 1 2 0 3 0 4] [1 0 0 2 0 3 4 0]] I want to calculate the cumulative sum of non-zero values in-place, which would change the array to: [[0 0 1 3 0 6 0 10] [1 0 0 3 0 6 1...
more »

2017-08-03 21:08 (0) Answers

Large Numpy Scipy CSR Matrix, row wise operation

I want to iterate over the rows of a CSR Matrix and divide each element by the sum of the row, similar to this here: numpy divide row by row sum My problem is that I'm dealing with a large matrix: (96582, 350138) And when applying the operation fr...
more »

2017-07-28 14:07 (2) Answers

Out-of-core processing of sparse CSR arrays

How can one apply some function in parallel on chunks of a sparse CSR array saved on disk using Python? Sequentially this could be done e.g. by saving the CSR array with joblib.dump opening it with joblib.load(.., mmap_mode="r") and processing the ch...
more »

2017-07-17 15:07 (0) Answers

How can I fit a gaussian curve in python?

I'm given an array and when I plot it I get a gaussian shape with some noise. I want to fit the gaussian. This is what I already have but when I plot this I do not get a fitted gaussian, instead I just get a straight line. Fairly new to python, but I...
more »

2017-06-11 05:06 (3) Answers

Fast K-step discounting in numpy/scipy/python

x has shape [batch_size, n_time] where the batches are independent If k=3, d=discount_rate. Pseudocode: x[:,i] = x[:,i] + x[:,i+1]*(d**1) + x[:,i+2]*(d**2) + x[:,i+3]*(d**3) Here's working code, but it's very slow. I'll be executing this function...
more »

2017-05-02 21:05 (1) Answers

Build developer's version of scipy

I am new and trying to contribute to scipy development. I forked scipy in my github and tried to build it based on this documentation. However, while trying to import scipy in Python, I get the following error: >>> import scipy Traceback (m...
more »

2017-04-29 17:04 (1) Answers save/load Matlab file performance

Many questions here have been asked about the numpy save/load performance, particularly in comparison to alternatives like pickling or using hdf5. In the context of exploratory scientific computing, my issue is that it's difficult to save/load arbitr...
more »

2017-04-21 00:04 (0) Answers

Producing spectrogram from microphone

Below I have code that will take input from a microphone, and if the average of the audio block passes a certain threshold it will produce a spectrogram of the audio block (which is 30 ms long). Here is what a generated spectrogram looks like in the...
more »

2017-04-11 20:04 (1) Answers

Interpolating a 3d array in Python expanded

My question expands on the code response seen here: Interpolating a 3d array in Python. How to avoid for loops?. Relevant original solution code is below: import numpy as np from scipy.interpolate import interp1d array = np.random.randint(0, 9, size...
more »

2017-04-03 13:04 (0) Answers

Geometric mean applied on row

I have this data frame as example: Col1 Col2 Col3 Col4 1 2 3 2.2 I would like to to add a 4th column called 'Gmean' that calculate the geometric mean of the first 3 columns on each row. How can get it...
more »

2017-02-24 11:02 (2) Answers