Measuring peak disk use of a process

I am trying to benchmark a tool I'm developing in terms of time, memory, and disk use. I know /usr/bin/time gives me basically what I want for the first two, but for disk use I came to the conclusion I would have to roll my own bash script that perio...
more »

2017-01-08 13:01 (2) Answers

Command output mangled on redirection

Given a directory with a few million files in it we want to extract some data from those files. find /dir/ -type f | awk -F"|" '$2 ~ /string/{ print $3"|"$7 }' > the_good_stuff.txt That will never scale so we introduce xargs. find /dir/ -type f...
more »

2016-12-28 23:12 (2) Answers

readChar from stdin

I would like to read in one character at a time and convert it to a string command to execute once a space is entered. I tried con <- file("stdin") open(con, blocking=TRUE) while(q!=" "){ #q=scan("",what="", nmax=1) q=readChar(con,1) cmd[...
more »

2016-12-18 08:12 (1) Answers

c stdout to stdin realtime

I have make two programs one of them named generator which print some text every second int main(int argc, char** argv) { for(int i = 0; i < 10 ; i++) { printf("something\n"); sleep(1); } return (EXIT_SUCCESS); } ...
more »

2016-11-04 19:11 (5) Answers

mysql i/o usage is very high

I have mysql running on RedHat EL 6.5 and mysql version - MySql Enterprise 5.6 Server configurations are as below - 1 processor / 4 cores Memory - 8GB Disc space - 256 GB Am experiencing very high Disk IO usage. Please share some advice/tuning...
more »

2016-10-17 14:10 (0) Answers

Close file after a kill command

I have an infinite loop in my program where it will write in a file defined as stdout until I kill it. Therefore, the file will never be closed and won't contain what was wrote. Is there a way to close the file after a kill command ? I could use s...
more »

2016-10-10 12:10 (2) Answers

Looking for a way to force a short read in linux

I am looking for a method of producing short reads in linux so I can unit test the handling code around them. I have a number of methods which at the lower levels call pread / pread64 to read from a file within the file system. These are designed to...
more »

2016-09-20 05:09 (2) Answers

args python parser, a whitespace and Spark

I have this code in foo.py: from argparse import ArgumentParser parser = ArgumentParser() parser.add_argument('--label', dest='label', type=str, default=None, required=True, help='label') args = parser.parse_args() and when I execute: spark-su...
more »

2016-09-07 21:09 (1) Answers

How to open and read file without buffering in C?

I want to benchmark an algorithm which operates on a file. The algorithm iterates N rounds and in each round, it reads one data block, do some magic stuff and set the offset of the next block. Here's the pseudo code: int run_algorithm(int offset) ...
more »

2016-09-03 05:09 (1) Answers

Print editable to console in Ruby

Let's say I have the following code in Ruby: print("Enter a filename:") editableprint("untitled.txt") filename = gets.chomp! What would be the function "editableprint" so that "untitled.txt" is part of the input of the user for the gets function? ...
more »

2016-08-20 08:08 (2) Answers

Optimizing disk IO

I have a piece of code that analyzes streams of data from very large (10-100GB) binary files. It works well, so it's time to start optimizing, and currently disk IO is the biggest bottleneck. There are two types of files in use. The first type of fi...
more »

2016-08-19 18:08 (1) Answers

Python - Pandas Output Limits Columns

When dealing with Pandas, I'm attempting to print analysis of an objects Kinematic and Angular states. My code for doing so is as follows: def displayData(tList, xList, zList, dxList, dzList, thetaList, dthetaList, Q_sList): states = pd.DataFram...
more »

2016-08-11 20:08 (1) Answers

How to read all bytes of a stream but the last 8

I have the following code: using (var fs = new FileStream(@"C:\dump.bin", FileMode.Create)) { income.CopyTo(fs); } income is a stream that I need to save to disk, the problem is that I want to ignore the last 8 bytes and save everything before...
more »

2016-08-01 06:08 (1) Answers

Write user input string to file in C

My program needs to take a user's input and save it to an external file for future reference. Here is the basic outline of the code. void newActivity(FILE *foutput) { char name[31]; char description[141]; finput = fopen("activities.txt"...
more »

2016-07-31 16:07 (1) Answers

How to use ioctl with FS_IOC_FIEMAP

My problem is to deal with sparse file reads and understand where the extents of the file are to perform some logic around it. Since, there is no direct API call to figure these stuff out, I decided to use ioctl api to do this. I got the idea from h...
more »

2016-07-30 03:07 (2) Answers

What does #tty? on STDIN mean / do in ruby?

Reading the ruby docs isn't overly helpful here: Returns true if ios is associated with a terminal device (tty), false otherwise. I was hoping to get some additional resources or explanation to help me understand this better. For context, I'...
more »

2016-06-25 20:06 (2) Answers

How to deal with a huge, one-line file in Java

I need to read a huge file (15+GB) and perform some minor modifications (add some newlines so a different parser can actually work with it). You might think that there are already answers for doing this normally: Reading a very huge file in java Ho...
more »

2016-06-03 20:06 (1) Answers

Piping result of command as an argument

I want to get the result of rpm -qa | grep something and then run the result I got rpm -ql result-i-got, and all in one line. I have found this to work: rpm -ql $(rpm -qa | grep something) I wonder, is there any better or even different way of pi...
more »

2016-05-24 11:05 (2) Answers