jimlongley wrote:Even if they are, that is a mind boggling amount of data that they already, according to his testimony, don't have time to go through. By the time they get the ability to sort through it with any level of efficiency we will all be beyond caring.
A bunch of years ago I was the engineer of a network control center. The center monitored the condition (not the content) of a large (195 nodes) T1 network for the State of NY. We were required, per contract, to record and monitor ALL alarms and events on the network, and investigate each an every one, as well as keeping a database of those events, a raw record of those events, and a separate database of the trouble tickets generated and solutions. The amount of data gathered rapidly became so unmanageable that it was considered a joke to threaten someone on the staff with having to go look for a specific incident in the raw record. On top of that we kept a backup copy at a "geographically diverse location."
As the head engineer, I was charged with the responsibility for ensuring that all of the data was stored and accessible, and I hired a database "expert" to program the access to the raw data as well as sorting and storing it in a useable manner. It quickly became obvious that the state of the art was not up to the task, and the programmer was even behind that. (At one point, shortly before he was moved to another job we noticed that the trouble ticket database was taking huge amounts of time to load, and it turned out that what he had done to handle completed trouble tickets was set a "delete flag" on those tickets, so that when you searched for open tickets they got ignored, but every time you accessed the database ALL of it got loaded. One day my lead computer operator, who was not a programmer herself but had some programming ability, decided to see if she could improve the speed by "packing" the database, and the command she issued deleted all of the data with delete flags set. That was when it was decided that our programmer would be better off in a different job and that it was a good thing to have a separate copy of the database off site. Forgot to mention, one salient thing on his resume was his experience with NSA.
We got a new programmer who understood the troubleshooting process and trouble tickets and things got a little better.
All of this with the State of New York looking over our shoulders and nitpicking.
I eventually quit the job and went back to something more comfortable.
I don't think NSA has the ability, now or in the near future, to process that data, and the amount will continue to grow as they sit on it, so I don't much care what they are keeping of mine.
I wish they'd sort out my Junk/Spam folder, I get tired of responding to all these E-mails promising to make me two inches taller, I do not need to grow two inches, I'm 6'0" tall already and that's plenty until the osteoporosis gets worse. I'm not looking for an online pharmacy either ...collecting and storing all emails for the last 10 years (so far)
(My sweet dad, before he passed on, at first used to reply to each and every Junk E-mail, telling them no thank you, not interested at this time etc)
