## Numpy: vectorizing a function that integrate 2D array

I need to perform this following integration for a 2D array: That is, each point in the grid get the value RC, which is integration over 2D of the difference between the whole field and the value of the field U at certain point (x,y), multiplying the...
more »

## Python: How to increase the dimensions of a matrix without expanding it?

I am quite new in Python, and am currently working on a code in which I want to store previous iterations of an 3-dimensional matrix, a version of which is created in each single step of a for-loop. The way I want to go about this is by concatenating...
more »

## Addition of array in python with list in a list

How to sum up the below to list? a=[[1,2,3],[4,5,6],[7,8,9]] b=[[1,2,3],[4,5,6],[7,8,9]] I apply this code: Total=[x + y for x, y in zip(a, b)] So the output will be: Total=[[1,1,2,2,3,3],[4,4,5,5,6,6],[7,7,8,8,9,9]] but I wish to get Tot...
more »

## What additional work is done by np.power?

I realised that np.power(a, b) is slower than np.exp(b * np.log(a)): import numpy as np a, b = np.random.random((2, 100000)) %timeit np.power(a, b) # best of 3: 4.16 ms per loop %timeit np.exp(b * np.log(a)) # best of 3: 1.74 ms per loop The resul...
more »

## Pythonic way to extract 2D array from list

Say I have a list which contains 16 elements: lst=['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P'] This list represents a 4 x 4 array where all elements have been put into a 1D list. In its array form it has this form: 'A', 'B', ...
more »

## Quickly convert a very large number of lists of strings into ndarray

My file looks as follows, where the first 3 numbers per line denote a triangle/triplet of things, and the 4th number is a marker for each triangle: 1 2 3 1 5 6 7 0 300 10 11 5 0 14 15 9 I currently read it as follows: import numpy as np file = op...
more »

## Scipy Sparse Cumsum

Suppose I have a scipy.sparse.csr_matrix representing the values below [[0 0 1 2 0 3 0 4] [1 0 0 2 0 3 4 0]] I want to calculate the cumulative sum of non-zero values in-place, which would change the array to: [[0 0 1 3 0 6 0 10] [1 0 0 3 0 6 1...
more »

## What does `x[False]` do in numpy?

Say I have an array x = np.arange(6).reshape(3, 2). What is the meaning of x[False], or x[np.asanyarray(False)]? Both result in array([], shape=(0, 3, 2), dtype=int64), which is unexpected. I expected to get an IndexError because of an improperly s...
more »

## Numpy: Multiplying large arrays with dtype=int8 is SLOW

Consider the following piece of code, which generates some (potentially) huge, multi-dimensional array and performs numpy.tensordot with it (whether we multiply the same or two different arrays here, does not really matter). import time import numpy...
more »

## How to efficiently fill an incomplete pandas dataframe consisting of pairwise combinations of values in a list?

Let's say I have a list of values, lst=['orange','apple','banana', 'grape', 'lemon'] I also have a pandas dataframe of the form, df: Source Destination Weight orange apple 0.4 banana orange 0.67 grape lemon...
more »

## Why does np.ndarray.__deepcopy__ require a superfluous argument?

The duplicate will answer what, but will not adequately answer why or why it is not mentioned in the docs even though it is required. Read on for more... There are two ways of making a deep copy of a numpy array. One way is using copy.deepcopy fr...
more »

## numpy dot product for tensors (3d times 2d)

Currently I use Na = (3, 2, 4) Nb = Na[1:] A = np.arange(np.prod(Na)).reshape(Na) b = np.arange(np.prod(Nb)).reshape(Nb) I want to calculate: r = np.empty((A.shape[0], A.shape[2]) for i in range(A.shape[2]): r[:, i] = np.dot(A[:, :, i], b[...
more »

## Add n zeros to the end of an array

I want to add n zeros to an array. When your array is x, and you want to add 3 zeros at the and of an array without creating 2 arrays: x = np.array([1.0, 2.0, 1.0, 2.0, 7.0, 9.0, 1.0, 1.0, 3.0, 4.0, 10.0]) I thought this command would be helpful...
more »

## Should I use np.absolute or np.abs?

Numpy provides both np.absolute and the alias np.abs defined via from .numeric import absolute as abs which seems to be in obvious violation of the zen of python: There should be one-- and preferably only one --obvious way to do it. So I'm g...
more »

## Speeding up calculation of nearby groups?

I have a data frame that contains a group ID, two distance measures (longitude/latitude type measure), and a value. For a given set of distances, I want to find the number of other groups nearby, and the average values of those other groups nearby. ...
more »

## Large Numpy Scipy CSR Matrix, row wise operation

I want to iterate over the rows of a CSR Matrix and divide each element by the sum of the row, similar to this here: numpy divide row by row sum My problem is that I'm dealing with a large matrix: (96582, 350138) And when applying the operation fr...
more »

## Calling function on valid values of masked arrays

I have two numpy masked arrays: >>> x masked_array(data = [1 2 -- 4], mask = [False False True False], fill_value = 999999) >>> y masked_array(data = [4 -- 0 4], mask = [False True False False], ...
more »

## Find nearest indices for one array against all values in another array - Python / NumPy

I have a list of complex numbers for which I want to find the closest value in another list of complex numbers. My current approach with numpy: import numpy as np refArray = np.random.random(16); myArray = np.random.random(1000); def find_neares...
more »

## Interweave two dataframes

Suppose I have two dataframes d1 and d2 d1 = pd.DataFrame(np.ones((3, 3), dtype=int), list('abc'), [0, 1, 2]) d2 = pd.DataFrame(np.zeros((3, 2), dtype=int), list('abc'), [3, 4]) d1 0 1 2 a 1 1 1 b 1 1 1 c 1 1 1 d2 3 4 a ...
more »

## Creating Pandas dataframe from numpy array row

I have a numpy array looking like this: a = np.array([35,2,160,56,120,80,1,1,0,0,1]) Then I'm trying to transform that array into pandas dataframe with logic "one column-one value" like this: columns=['age','gender','height', 'weight','ap_hi...
more »

## Numpy match indexing dimensions

Problem I have two numpy arrays, A and indices. A has dimensions m x n x 10000. indices has dimensions m x n x 5 (output from argpartition(A, 5)[:,:,:5]). I would like to get a m x n x 5 array containing the elements of A corresponding to indice...
more »

## How can I create a list of records groupedby index in Pandas?

I have a CSV of records: name,credits,email bob,,test1@foo.com bob,6.0,test@foo.com bill,3.0,something_else@a.com bill,4.0,something@a.com tammy,5.0,hello@gmail.org where name is the index. Because there are multiple records with the same name, I'...
more »

## Fill in numbers in a numpy array in sets of 1024

I have a large numpy array with increasing elements as follows A = [512,2560,3584,5632,....] Elements are always spaced at least 1024 apart Let's say I need to transform that above array to the one below, where for each element of the original m...
more »

## How to find memory leak with pandas

I have a program which repeatedly loops over a pandas data frame like below: monts = [some months] for month in months: df = original_df[original_df.month == month].copy() result = some_function(df) print(result) However, the memory which i...
more »

## Numpy integer division sometimes yields wrong results when casted

What is the reason those two divisions give different results? I am very confused because with some numbers it gives the same results and with some it doesn't. >>> import numpy as np >>> a, b = np.array([844]), np.array([8186]) >...
more »

## (Python) Find the indices of the values in two arrays that are equal to the values in two other arrays

I have the following 4 arrays and I want to get the indices of the values that are equal in the arrays A and X corresponding to the values in the same position in B and Y. So for the following example, import numpy as np A = np.asarray([400...
more »

## NumPy sum one array based on values in another array for each matching element in 3rd array

I have two numpy arrays, one containing values and one containing each values category. values=np.array([1,2,3,4,5,6,7,8,9,10]) valcats=np.array([101,301,201,201,102,302,302,202,102,301]) I have another array containing the unique categories I'd l...
more »

## Numpy: creating batch of numpy arrays within another numpy array (reshaping)

I have a numpy array batch of shape (32,5). Each element of the batch consists of a numpy array batch_elem = [s,_,_,_,_] where s = [img,val1,val2] is a 3-dimensional numpy array and _ are simply scalar values. img is an image (numpy array) with dim...
more »

## Python: Stretch 2D array into 3D based on corresponding array with 3rd-dimension index

Lets say I have some 2D array a = np.ones((3,3)) I want to stretch this array into 3 dimensions. I have array b, the same size as a, that provides the index in the 3rd dimension that each corresponding element in a needs to go too. I also have 3D a...
more »

## Index Matrix Rows and Columns on MATLAB, is this code ok?

I have some code on MATLAB, which I have to translate it to Python to work with it. Going through the code I found this line: filtered_proj (new_len, :, 1) = 0; So after reading here MATLAB Documentation, I understood and tried to apply on my Pyth...
more »

## Is there a fast way to create a numpy array that reduces unique values to their lowest form?

Sorry if the question is worded confusingly. I have an array similar to the following: [[3,7,9,5],[3,3,7,5]], though much larger. How can I convert this into a form such as[[0,2,3,1],[0,0,2,1]] where each unique value starting with the lowest is giv...
more »

## Initialize transposed numpy array

I want to use the Singular-Value-Decomposition of matrix A. If possible I would write: V, S, W.T = np.linalg.svd(A) But I can't initialise an array with its transposed. Now I have two questions: As far as I understand the python internals ther...
more »

## How to implement a smooth clamp function in python?

The clamp function is clamp(x, min, max) = min if x < min, max if x > max, else x I need a function that behaves like the clamp function, but is smooth (i.e. has a continuous derivative). ...
more »

## Passing C++ vector to Numpy through Cython without copying and taking care of memory management atuomatically

Dealing with processing large matrices (NxM with 1K <= N <= 20K & 10K <= M <= 200K), I often need to pass Numpy matrices to C++ through Cython to get the job done and this works as expected & without copying. However, there are ...
more »