## Different output in sparse representation as compared to numpy array

import numpy as np import scipy as sc from sklearn.preprocessing import normalize import scipy.sparse as sp import numpy import numpy as np import scipy.sparse as sp def func1(A,c,eps,maxiter): c=0.8 eps=1e-4 maxiter=20 n=sc.shape...
more »

## Numerically stable evaluation of log of integral of a function with extremely small values

If I have a random number Z that is defined as the sum of two other random numbers, X and Y, then the probability distribution of Z is the convolution of the probability distributions for X and Y. Convolution is basically to an integral of the produc...
more »

## Using numpy/scipy to identify slope changes in digital signals?

I am trying to come up with a generalised way in Python to identify pitch rotations occurring during a set of planned spacecraft manoeuvres. You could think of it as a particular case of a shift detection problem. Let's consider the solar_elevation_...
more »

## Fastest method for determining if 2 (vertically or horizontally) adjacent elements of a numpy array have the same value

I am looking for the fastest way of determining if 2 (vertically or horizontally) adjacent elements have the same value. Let's say I have a numpy array of size 4x4. array([ [8, 7, 4, 3], [8, 4, 0, 4], [3, 2, 2, 1], ...
more »

## How to create Numpy array using fancy indexing

How do use numpy's fancy indexing to create this, I would like the fastest performance: array([[ 1, 2, 3, 4, 16, 31], [ 2, 3, 4, 5, 17, 32], [ 3, 4, 5, 6, 18, 33], [ 4, 5, 6, 7, 19, 34], [ 5, 6, 7, 8, 20, 3...
more »

## Fastest way to sum over rows of sparse matrix

I have a big csr_matrix(1M*1K) and I want to add over rows and obtain a new csr_matrix with the same number of columns but reduced number of rows. Actually my problem is exactly same as this Sum over rows in scipy.sparse.csr_matrix. The only thing is...
more »

## Efficient pairwise computation of identical elements in a large 2D numpy array

I have a 2D numpy array with hundreds of thousands of rows and a thousand or so columns (let's say it's a N x P array with N = 200,000, P = 1000). The goal here is to compute the number of identical elements between each pair of row vectors, ideally ...
more »

## Piecewise linear fit with n breakpoints

I have used some code found in the question How to apply piecewise linear fit in Python?, to perform segmented linear approximation with a single breakpoint. The code is as follows: from scipy import optimize import matplotlib.pyplot as plt import ...
more »

I've been trying to understand masking and how it works with image filters. I'm using the following code to try to develop my understanding. import scipy.ndimage as ndi import matplotlib.pyplot as plt import numpy as np # Generate a random binary...
more »

## Scipy Sparse Cumsum

Suppose I have a scipy.sparse.csr_matrix representing the values below [[0 0 1 2 0 3 0 4] [1 0 0 2 0 3 4 0]] I want to calculate the cumulative sum of non-zero values in-place, which would change the array to: [[0 0 1 3 0 6 0 10] [1 0 0 3 0 6 1...
more »

## Large Numpy Scipy CSR Matrix, row wise operation

I want to iterate over the rows of a CSR Matrix and divide each element by the sum of the row, similar to this here: numpy divide row by row sum My problem is that I'm dealing with a large matrix: (96582, 350138) And when applying the operation fr...
more »

## Out-of-core processing of sparse CSR arrays

How can one apply some function in parallel on chunks of a sparse CSR array saved on disk using Python? Sequentially this could be done e.g. by saving the CSR array with joblib.dump opening it with joblib.load(.., mmap_mode="r") and processing the ch...
more »

## Calculating pairwise spatial distances in periodic 2D lattice

I've been searching in vain for a solution to compute spatial distances between discrete, equidistant square sites on a lattice of periodic/wrapped edges. For example, setting adjacent sites to be 2 units apart on 9x9 lattice: m = 9 lattice = np.zer...
more »

## How to determine the uncertainty of fit parameters with Python?

I have the following data for x and y: x y 1.71 0.0 1.76 5.0 1.81 10.0 1.86 15.0 1.93 20.0 2.01 25.0 2.09 30.0 2.20 35.0 2.32 40.0 2.47 45.0 2.65 50.0 2.87 55.0 3.16 60.0 3.53 65.0 4.02 70.0 4.69 75....
more »

## What python function is this SciPy tutorial referring to?

SciPy's website has a tutorial that references a python function source that lists the source code of functions written in python, but I cannot use it in python or find documentation for it online. The reference is at the bottom of this page. I see t...
more »

## What is a faster way to get the location of unique rows in numpy

I have a list of unique rows and another larger array of data (called test_rows in example). I was wondering if there was a faster way to get the location of each unique row in the data. The fastest way that I could come up with is... import numpy...
more »

## Scipy sparse matrix exponentiation: a**16 is slower than a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a?

I'm doing a simple sparse matrix exponentiation, a**16, using scipy-0.17. (Note, not element-wise multiplication). However, on my machines (running Debian stable and Ubuntu LTS) this is ten times slower than using a for loop or doing something silly ...
more »

## How can I fit a gaussian curve in python?

I'm given an array and when I plot it I get a gaussian shape with some noise. I want to fit the gaussian. This is what I already have but when I plot this I do not get a fitted gaussian, instead I just get a straight line. Fairly new to python, but I...
more »

## What is the error in this sample code via scipy docs that attempts to solve an ode?

I am trying to learn about solving ODEs in python and scipy seemed like a good starting point. I've become comfortable with using odeint and am now trying to learn how to use ode. I tried running the sample code in the scipy docs, but it is returning...
more »

## Scipy.optimize.curve_fit won't fit cosine power law

For several hours now, I have been trying to fit a model to a (generated) dataset as a casus for a problem I've been struggling with. I generated datapoints for the function f(x) = A*cos^n(x)+b, and added some noise. When I try to fit the dataset wit...
more »

## Portfolio Selection in Python with constraints from a fixed set

I am working on a project where I am trying to select the optimal subset of players from a set of 125 players (example below) The constraints are: a) Number of players = 3 b) Sum of prices <= 30 The optimization function is Max(Sum of Votes)...
more »

## Wrong result from scipy.optimize.minimize used on Fortran function using ctypes

I currently have a Fortran function that I want to optimize using SciPy wrapping it using Ctypes. Is this possible? Perhaps I've done something wrong in my implementation. For example, assume I have: cost.f90 module cost_fn use iso_c_binding, ...
more »

## Apply function on a rolling slice of a pandas DataFrame

I'd like to calculate the determinant of 2x2 matrices which are taken by rolling a window of size 2 on a Nx2 matrix. I'm just using the determinant as an example function. In general, I'd like to apply a function to a dataframe which is taken by wind...
more »

## Fast K-step discounting in numpy/scipy/python

x has shape [batch_size, n_time] where the batches are independent If k=3, d=discount_rate. Pseudocode: x[:,i] = x[:,i] + x[:,i+1]*(d**1) + x[:,i+2]*(d**2) + x[:,i+3]*(d**3) Here's working code, but it's very slow. I'll be executing this function...
more »

## Build developer's version of scipy

I am new and trying to contribute to scipy development. I forked scipy in my github and tried to build it based on this documentation. However, while trying to import scipy in Python, I get the following error: >>> import scipy Traceback (m...
more »

## Fastest way to create strictly increasing lists in Python

I would like to find out what is the most efficient way to achieve the following in Python: Suppose we have two lists a and b which are of equal length and contain up to 1e7 elements. However, for the ease of illustration we may consider the followi...
more »

## Find most frequent row or mode of a matrix of vectors - Python / NumPy

I have a numpy array of shape (?,n) that represents a vector of n-dimensional vectors. I want to find the most frequent row. So far it seems that the best way is to just iterate over all the entries and store a count, but it seems obscene that num...
more »

## scipy.io save/load Matlab file performance

Many questions here have been asked about the numpy save/load performance, particularly in comparison to alternatives like pickling or using hdf5. In the context of exploratory scientific computing, my issue is that it's difficult to save/load arbitr...
more »

## Producing spectrogram from microphone

Below I have code that will take input from a microphone, and if the average of the audio block passes a certain threshold it will produce a spectrogram of the audio block (which is 30 ms long). Here is what a generated spectrogram looks like in the...
more »

## Fitting a Lognormal Distribution in Python using CURVE_FIT

I have a hypothetical y function of x and trying to find/fit a lognormal distribution curve that would shape over the data best. I am using curve_fit function and was able to fit normal distribution, but the curve does not look optimized. Below are...
more »

## Interpolating a 3d array in Python expanded

My question expands on the code response seen here: Interpolating a 3d array in Python. How to avoid for loops?. Relevant original solution code is below: import numpy as np from scipy.interpolate import interp1d array = np.random.randint(0, 9, size...
more »

## Why is using tanh definition of logistic sigmoid faster than scipy's expit?

I'm using a logistic sigmoid for an application. I compared the times using the scipy.special function, expit, versus using the hyperbolic tangent definition of the sigmoidal. I found that the hyperbolic tangent was 3 times as fast. What is going on...
more »

## Geometric mean applied on row

I have this data frame as example: Col1 Col2 Col3 Col4 1 2 3 2.2 I would like to to add a 4th column called 'Gmean' that calculate the geometric mean of the first 3 columns on each row. How can get it...
more »

## Suggestions for fitting noisy exponentials with scipy curve_fit?

I'm trying to fit data that would be generally modeled by something along these lines: def fit_eq(x, a, b, c, d, e): return a*(1-np.exp(-x/b))*(c*np.exp(-x/d)) + e x = np.arange(0, 100, 0.001) y = fit_eq(x, 1, 1, -1, 10, 0) plt.plot(x, y, 'b')...
more »