Gah!

[vlion]

Said yes to a request to help do some data analysis for someone, for free.

Turned out the data was 17K files, 77GB of data, and the data is fairly 'dirty' and in a number of different structures of delimited separated data - tab, commas, etc.

Bloody mistake on my part. If I don't know the reliability of a dataset, I am not going to help someone out without looking into it first. I've burnt near 20 hours on it already in the last 5-6 weeks.

I didn't really have time to deal with this amount of data, and the requestor has deadlines.

Advice for the reader: don't volunteer without understanding the task ahead of you.

Comments

[foxfirefey]

Oh lordy. That was very unfair of them! I'm sorry you ended up in such a pickle.

[vlion]

They are pretty ignorant of programming, I think they took some C++ class back in the 90s....

I strongly suggested studying SQL and learning how to handle data.

[foxfirefey]

SQLite is very handy for these kinds of things, assuming it can scale that much.

[vlion]

I assure you, if this was a long-running project, it would start by spawning either a sqlite database or a mysql one. Unfortunately for the next poor shlub, it's a one-off.

[exor674]

I am actually amazed at SQLite, my dataset is not even close to that large, but I use SQLite for the accounting datastore ( we've fetched these, we've got to fetch these ) when I do an edges fetch, and I have been *SO* waiting for SQLite to just fall over -- it hasn't.