The Blog of Justin Loutsch

I don't know the question, but the answer is 42!

About

I'm Justin, and I live in Boston. I'm a huge geek into process automation and work reduction, and am also an editor at Eat Your Serial. Thanks for dropping by!

This was done in CentOS 6.4 but PolicyKit is part of gnome so it should work across different distriubtions.

I’ve run into this problem before, but it finally today became something I needed to solve after I moved the server to a new office with no monitor or inputs.

I would VNC in but the external drives could not be mounted due to gnome policykit controls (for security’s sake).

So, here’s how to enable disk mounting in a VNC session in policykit.

CD into /etc/polkit-1/localauthority and you’ll find a bunch of files:

net.reactivated.fprint.device.policy org.fedoraproject.config.firewall.policy
org.fedoraproject.config.services.policy
org.fedoraproject.systemconfig.kdump.policy
org.freedesktop.NetworkManager.policy
org.freedesktop.RealtimeKit1.policy
org.freedesktop.consolekit.policy
org.freedesktop.devicekit.power.policy
org.freedesktop.devicekit.power.qos.policy
org.freedesktop.modem-manager.policy
org.freedesktop.network-manager-settings.system.policy
org.freedesktop.packagekit.policy
org.freedesktop.policykit.policy
org.freedesktop.udisks.policy
org.gnome.clockapplet.mechanism.policy
org.gnome.control-center.defaultbackground.policy
org.gnome.cpufreqselector.policy
org.gnome.gconf.defaults.policy
org.gnome.settings-daemon.plugins.wacom.policy
org.gnome.system-monitor.policy
org.opensuse.cupspkhelper.mechanism.policy

The one we’ll want to edit is org.freedesktop.udisks.policy.

First, backup this file:

sudo cp /usr/share/polkit-1/actions/org.freedesktop.udisks.policy /usr/share/polkit-1/actions/org.freedesktop.udisks.policy.bak (or any other location you prefer).

In the terminal, run the following command: vi /usr/share/polkit-1/actions/org.freedesktop.udisks.policy

Look for this section:

<action id=”org.freedesktop.udisks.filesystem-mount”>
<description>Mount a device</description>
<description xml:lang=”da”>Montér en enhed</description>
<message>Authentication is required to mount the device</message>
<message xml:lang=”da”>Autorisering er påkrævet for at montere et fil system</message>
<defaults>
<allow_any>yes</allow_any>
      <allow_inactive>yes</allow_inactive>
      <allow_active>yes</allow_active>
</defaults>
</action>

Change the fields in red so that all say “yes” instead of no.

Save the file (in vi, this is down by hitting the escape key, then typing a colon, W, Q!, and will look like this: :wq!). Restart the VNC session. You should now be able to mount your external drives in a VNC session.

Recently a hard drive failed in our RAID 5 QNAP NAS array. The CTO suggested a low level format would make the drive usable again, and he was right. In my case it was a Seagate drive, so I used Seatools and it’s full erase feature. I found that Seatools wouldn’t run via USB, nor would it run on a spare box I had laying around, so it was easiest to burn Seatools to a CD and just plug the drive into my desktop. This took about 14 hours total so I let it run overnight.

Once I plugged the drive back into the QNAP, I expected that the RAID array would begin rebuilding. This was not the case, however, and is apparently a shortcoming of the QNAP system. Their software RAID can’t recover from dead disks apparently, but only ones incorrectly flagged as faulty.

I did a lot of googling to figure out how to fix this, and with a lot of help from this site figured out step 1.

The first step is clone the partition structure of the drives over to the newly formatted drive. To do this, first ssh into the QNAP server as the administrator.

Run fdisk -l (that’s a lowercase L) to get a listing of all disks in the system. My QNAP server has 4 drives, each labeled with a number over the drive bay. These numbers correspond to sda, sdb, sdc, and sdd. In my case, sdc failed, and after replacing it was still recognized as sdc.

Find one that’s complete, as this is what you’ll want to copy over to the new drive. The new drive will be the missing label in the fdisk output (sdc was missing in my case).

Next command: fdisk /dev/drive label (for me: fdisk/dev/sdc)

Type n and tap enter for a new partition. For partition type, enter p for primary. Enter 1 for the partition number, and 66 for the last cylinder size. Stick with the default value for first cylinder, but enter different values for the last cylinder of each partition.
Type n and tap enter for a new partition. For partition type, enter p for primary. Enter 2 for the partition number, and 132 for the last cylinder size.
Type n and tap enter for a new partition. For partition type, enter p for primary. Enter 3 for the partition number, and 243138 for the last cylinder size.
Type n and tap enter for a new partition. For partition type, enter p for primary. Enter 4 for the partition number, and 243200 for the last cylinder size.

Next mark the first partition of your drive as bootable:

Command (m for help): a Partition number (1-4): 1
Then change partition 2 to ‘Linux Swap / Solaris’ format Command (m for help): t Partition number (1-4): 2 Hex code (type L to list codes): 82 Changed system type of partition 2 to 82 (Linux swap / Solaris)

Finally, save the new partition table Command (m for help): w The partition table has been altered!

Ctrl-C to exit fdisk

Eject the new disk and reinsert

The site I linked to then says that all you have to do after this is eject the disk and reinsert it, and the rebuilding will begin. However this is not what happened for me.

The disk structure was cloned just fine, but the array was still running in degraded mode even after I restarted the NAS.

I had to begin the rebuild process manually, and actually ended up calling QNAP to find out exactly how to do this. They wanted to remote into my machine using teamviewer, but I declined and ask them to walk me through the commands (since I’m comfortable with the terminal, this was fine, but YMMV).

This command must also be run while SSH’ed into the QNAP server: mdadm -a /dev/md0 /dev/sdc3

sdc3 is the largest partition in the QNAP layout and the one that is part of the array, while the other smaller ones are for the OS this one contains the data stored in the array.

The rebuild process could take a few hours (I allowed mine to run overnight). To follow the progress of the rebuild, use this command: cat /proc/mdstat (note: this will not work directly from the command prompt upon SSHing into the QNAP. I had to run “cd ..” to go back to the root directory before it would work.

Once the rebuild is complete, reboot the NAS. Once it’s online again and you can log into the web GUI, you’ll see in the RAID management page that the status has gone from “degraded mode” to “ready” and you will see all 4 drives listed instead of just the 3 healthy ones.

My year in review

December 23rd, 2012

I’ve been quite busy since my last blog post.  I’m currently writing this home from home, where I’ll be until some time next week for the Christmas holiday, but I figured now was a good time to fill in my readers on what I’ve been up to.

I was unemployed for 3 months, from March to May. This was a tough time for me, surviving off of severance and unemployment benefits, but I made out just fine. I went to the gym every day, and applied for jobs all week long. An ex came to visit when I started my new job, which didn’t really go so well mostly on my part due to the stress of having to find a place to live in 2 months, saving the $1000+ deposit and first/last month rent needed to move in, and paying the bills.  Not much happened during this 3 month chunk, which has made my year feel very compressed.  March was a great month to be unemployed in Boston though, 2 weeks of 70-80 degree weather, enjoying it on my friend’s deck.
I got creative towards the end of my unemployment and decided to crash some job fairs at local universities. Luckily Boston has a little over a dozen of these. I went to a career fair at Boston University, just slipping in past the sign-up table wearing my suit and began talking to recruiters. Ran into one former recruiter from my old company so we chatted for a bit and exchanged info, and then I moved on and found another recruiter who told me his cousin was looking for an IT person.  I gave him my resume, got his card, and emailed him a copy of my resume a day later.  Soon, I had an interview with a small biomed firm in downtown Boston.  That interview was the Thursday before mother’s day, I had 2 interviews in all and then started on June 18th.
I’m proud to say I’ve been employed full time since September (I started on a 3 month contract in June) and started with an excellent raise as a network administrator. My familiarity with linux was a great advantage to getting this role, as we use CentOS all the time, but my willingness to play around with new software and find better/faster ways to complete a task played a role as well.  I do a large chunk of software testing and IT support, and it’s safe to say that I’ve roled all of my skills into this one position. I even work with a German, and get to speak German with her every day. I will be starting a second job in January doing IT support at a local German school’s evening programs, where her father works. This will give me the chance to attend cultural functions I normally wouldn’t attend on my own, and expand my skillset while meeting authors, filmmakers, and other guests of the Institute.
As to finding a place to live, I moved from East Somerville, MA to Revere, MA. East Somerville was not very convenient and made me heavily reliant upon the bus system which got old very quickly if you had to go anywhere when it wasn’t rush hour.  Revere has a few downsides (Beachmont specifically is very sleepy) but I live just a 10 minute walk from the train station, which takes 20 minutes to get me to work. I live in a 2 story house on the top of the hill and I can see the ocean from my window. Many of my friends remark that I live far away, but in reality it’s only far when you factor in where they live. From the downtown area it’s often much closer than the other transit lines, plus I always get a seat because I live so close to the end. Bonus points for the house because it has a bar and a woodstove. I live with 2 other people, one of whom is the landlord, and he takes care of everything. Both guys have cars and I can occasionally borrow one if needed but with Target and the grocery store a mile away I generally try to walk there when able. The best part about this place is the rent is only $600/month rather than the $975 I paid for a studio, and while I sometimes do miss the privacy a studio affords, I’m really enjoying saving the extra money I used to be paying on rent. It feels great to have a safety net savings account for the first time in my life.
Other notable events of the year include meeting Kat and Mouse, if for just a few short hours. Kat came to town to check out hotel accomodations for a conference she was organizing, and Mouse came down from Maine so the 3 of us got dinner on the hotel and shot the shit for a while.  I put together a server for the first time ever which was a lot of fun, and learned a lot about RAID and VLM, and took this one step further to set up a RAID array with LVM in a remote server using only a text file and some googled walkthroughs tailored to my needs (post forthcoming). I’ve also been learning a lot about software testing and automating, using tools such as sikuli, selenium, and TestLink, and hope to do some writeups on those in the future. Oh, and I turned 27.
For now, I’m enjoying being home, spending time with the family and all the animals, and occasionally shoveling some snow. Off to bed with me now.

Most ebooks these days have some of DRM scheme in use to protect the rights of copyright holders. EPUB files from Sony and Google use Adept while Barnes and Noble has yet another scheme, and Amazon has several schemes depending on the file format.

These schemes effectively lock the buyer into buying from only one company, but there are several drawbacks to this: Amazon has removed books from devices remotely (1984 incident), but the bigger problem is if one of these companies goes out of business you will lose all of those ebooks that you paid for. Another important item to consider is availability of titles. Perhaps Barnes & Noble has the newest book in a series you love, while Amazon doesn’t, but you have a kindle. I guess you’ll just have to wait.

This is where we have some good news: all the major ebook DRM schemes have been broken (with the possible exception of iBooks, not 100% sure on that). Not only have they been broken, but over the last 2 years the process has become extremely simple. It began as several sets of python scripts, one that acquired the encryption key (to verify you own the file) and then the second to use that key to remove encryption. They had to be run in a terminal and were somewhat complicated if you’ve never done it before. Fast forward to now and there is a wonder ebook library program called Calibre which has plugins available for DRM removal which make it a simple case of adding in your kindle serial number or B&N credit card number (used as the encryption key with your account) and then once the device is plugged in and connected to Calibre all you need to do is right click the file on the device and choose add to library and not only is it added instantly but the DRM is removed at the same time, leaving you free to convert the file to any format needed and own it as long as you wish.

So, on to Pottermore. They went a different route, and instead of choosing DRM they applied a digital watermark to these files to identify anyone who might have shared them after purchase. The files are completely unencrypted but still identifiable. I for one was excited to check out how this watermark was implemented, and bought the first Harry Potter book for $7.99 the minute I got the invitation from Amazon. Upon purchasing one book, you are given multiple downloads (I think 7) which allows you to download the books in multiple formats.

I downloaded the epub and mobi formats of the books and unpacked them to look at the file contents. EPUB files are just zip files and can be renamed to .zip and opened with any archiving utility. Then it’s just a matter of looking through the various files searching for a watermark tag and removing it, as well as the HTML file with the text of the book itself.

For MOBI it’s slightly more complicated as the mobiunpack script is required to unpack the contents of the file, but the same process applies. Just edit the HTML to remove the identifying watermark line and save it, then use Amazon’s kindlegen to create a new mobi file from the edited files. Add it to Calibre and you’ve got a brand new unsecured, unidentifiable ebook.

This was done for educational purposes and I haven’t uploaded the ebook anywhere (they existed in eformat ever since publication anyway thanks to HP fans) but altogether I expected something a lot more complicated when hearing about a watermark but I think this was just the easiest way to explain the identifying line with an analog that we all understand today.

I’ve lived in East Somerville for 8 months now, and have noticed a few things about it that differentiate it from Somerville proper. First, there’s the pejorative nickname “Slumerville” use to denote what looks like a poorer area of the city. This area of Somerville is also underserved by the subway (as compared to Davis Square) and relies chiefly on buses and cars for transit needs.

In my neighborhood alone I can take the 89 or 101 to Sullivan Square subway station or the 80 close to Davis station or right to Lechmere, within a 5 minute walk. 15-20 minute walk would bring me to the 88 bus line which also runs to Davis but up and down Highland. This is convenient to that entire neighborhood as well as anyone going to the Somerville public library.

I have many friends who either have lived in this area, or live close by and consider it “far” from Davis, the most hopping/trendy part of Somerville (with Union Square a close second, in my opinion).  In actuality it’s a 20 minute walk, or 5-10 minute busride if you can stomach the long waits during off-peak hours. So essentially, this area is most convenient to live in during rush hour when the buses stop every 10-15 minutes rather than once every hour and a half or so. For all other times, various bus tracking apps allow you to see how many minutes til the next bus will arrive, which makes that waiting slightly more variable.

I believe another reason this area is referred to as “Slumerville” is because, the further down Winter Hill you go, in the direction of Sullivan Station (southeast) the poorer the area begins to look. I think this is largely attributed to the various immigrant groups that live in this part of East Somerville, an observation based on a few things. First, every morning at the local park there is what appears to be a large number of immigrants standing around waiting to be picked up for work (I’ve even seen painting vans picking them up on my way to work once in a while).

Another good indication of this demographic is the amount of ethnic restaurants and shops in the area. Mexican, Brazilian, Creole, Italian, Ethiopian, and more. I for one think that the inhabitants of this area benefit from having such a wide selection of food choices in a very small area.  Another benefit is that immigrants make better neighbors, leaving the areas they live safer in a bid to succeed and live safely. What does Davis have? Joshua Tree and upscale bars, JP Licks, McDonalds, barber shops and cafes, a homestyle restaurant with a bit of everything, and a burger joint. Oh and some Indian restaurants, Japanese restaurants, and a BBQ place.  All definitely a bit more upscale than you can find in East Somerville, even if not far away.

I think what this mainly comes down to is gentrification. The areas close to subway stops are obviously coveted and therefore more valuable, causing rents to go up. When I looked at my apartment the realtor told me it was much cheaper because it’s not convenient to a subway station. What did he mean by cheaper? Oh, $975. The people who live closer to the subway pay a premium for the privilege, while those live further away pay less for the same or better accommodations, but make up for it by having to take the bus before getting on a train.

I hear rumors of gentrification coming east Somerville fairly often, starting with nice French restaurant a few blocks away from Sullivan Station, and I worry that at some point this will push out the local businesses already there and provide the residents with less choice but I think that time is slow to come. Perhaps as the green line construction progresses more businesses will choose to move further into Somerville, but some have already tried this banking on construction starting only to be burned when the MBTA delayed the project due to funding.

So what does that live us with in East Somerville vs the rest of Somerville?

East Somerville has a Super Stop and Shop grocery store and Sullivan Station T stop, Rite Aid/Walgreens/CVS, bars, liquor stores, Dunkin Donuts, myriad restaurants and 2 convenient libraries.  Davis has hipsters and more well-to-do folks, and Union Square has homeowners who complain about a public housing building project that will supposedly block their marvelous view of Boston even though it’s blocked by a tree 6 months out of the year anyway despite the fact that it will house firefighters, police, and other low income earners.

Hi Chris

December 21st, 2011

Feel free to comment! Miss you buddy.

So southie taxi is pretty lame. They have spiders that look for any blog posts with the word Boston in it, and then steal that content and add it to their own website. I mean I know I wrote that I’m in Boston, but that’s no reason for Southie Taxi to go and steal content, is it Boston? Boston does not appreciate this.

Irene

August 28th, 2011

So looks like I survived Irene just fine, there was worse damage further inland than on the coast and on Boston. Afterwards I went down to a liquor store and bought myself a celebratory pack of rolling rock and also discovered that half of my neighborhood is without power. I thankfully am not part of that group.

I’ve been very busy since the last time I posted, sorry about that. Mostly I’ve been busy with work, or meeting up with friends, or with a girlfriend (we broke up when she went home to Korea in June, so that’s no longer an issue.

I’ve not only been working at my normal job for CSNstores.com, but have also begun one-on-one tutoring sessions by answering an ad I found in a restaurant.  This has netted me about $550 in 2 months, and that was with only 2 students. I’m currently down to one since the other left, but I’m going to focus on Korean students in Boston who need help understanding their English class readings, as their parents seem to be willing to pay anything for their children to succeed.  I’ve witnessed this first hand with the girl I dated and am also seeing it with the current people I’m tutoring, and it seems there’s lots of demand for this kind of service so that’s one thing I’ll be focusing on in the coming months.

I also just recently accepted a new position at work: processing.  This has been keeping me very busy. Basically we have spreadsheets full of product data concatenated into SQL insert scripts, and it is my job to run them to enter everything into the system (one of the main tasks).  I also gather project materials, put them together, and send the work out to be done.  I have just now started to get comfortable with the position after a week, but I know there’s a lot more I can do.  Many nights I’ve had to stay late to get the work done on time, but yesterday I was shown some organizational steps that will help out a great deal.

I was asked to take this position because my new boss would like to improve the process, make it more efficient, reduce steps, etc, and that is exactly what I was doing in my previous position.  So far I’ve really impressed my new boss, which is awesome since it’s been only 1 week since I started the position.

In my next post I’m going to write more about how exactly I do this, but I’d like to wrap this one by letting everyone know that I moved into my new apartment on August 1st. I left the previous place a month early to get out of the September 1st lease cycle, which is just awful with all the students moving in on the same day.  I now live in Somerville, MA, just a few miles north of Boston and it so much nicer.  I’m very happy with it so far.

Finally, I’m all prepared for hurricane Irene to hit tomorrow so we’ll see how that goes. Good luck to all my other friends on the east coast!

So now I’d like to take some time and illustrate how you can use excel and firefox extensions to fill in data for you, create scripts, and generally reduce the amount of work you have to do.

First, if you’ve never used CONCATENATE, make it your friend. It’s super easy.  Type in =CONCATENATE to any cell and then follow it with (“your input goes here”).  Remember to use quotations around the text just like that there.  This is useful for SQL scripting since you have defined columns which can be pasted directly from the database output into your spreadsheet.  Ex: Product Name (Cell C1), Product ID (Cell B1), NewProductName (Cell C1).

You can write a script to update this information by typing: =CONCATENATE(“Update <ExampleTbl> set ProductName ‘”,C1,”‘ where ProductID = ‘”,B1,”‘”)

So the ‘ on either side of the cell info is to tell SQL that you are using variable characters, and the quotes and commas are how Excel knows what cell you’re referencing.  The very last quotation mark is simply to close the quotes from the beginning of the concatenate formula. You can then drag that down and automatically fill in new names for all the products that need updating.

Another useful formula is the IF statement, which can be combined with many others and is very versatile.  One of my favorite uses is when you have to changed a command at the very end of a list of scripts.  For example, every single query needs to have “union all” at the end save the last one.  Let’s say there’s a list of product IDs from A1 to A20. The IF(ISBLANK) formula needs to be used to do this, and you would tell it to see if the cell below the first one in the list is empty or not, which defines if you’ve reached the end of your list.

So now you’ve updated those product names and you want to make sure they’re succeeded.

Formula: =IF(ISBLANK(A2), CONCATENATE(“select from ExampleTbl with (nolock) where ProductId = ‘”,A1,”‘”), CONCATENATE(“select from ExampleTbl with (nolock) where ProductId = ‘”,A1,”‘ union all”).

This formula essentially looks to see if A2 is blank. If it is, the if commands tells Excel to use the first statement if the given criteria are met. But if not, the second command will be used. This will automatically fill in union all to all queries but the last one.  If statements are also useful to make sure data matches:

=IF(A1=C3, “Names Match”,”Names Do Not Match”) which can make comparing data entered into the system and the original product info as easy as dragging and seeing if anything doesn’t match. Useful when you have lots of data entry to check.

My new position at work involves a lot of file manipulation and folder creation/moving.  To make this as easy as possible, the best way is to start with VLOOKUPs.  VLookup stands for vertical lookup, and will look up information from a table arranged vertically. A typical formula looks like this: =VLOOKUP(C2,A2:B20, 2, False).  This is telling the vlookup to look for the information from cell C2 in columsn A2:B20 (your table, the first cell on top left and last on bottom right), and to return the corresponding information for whatever is in cell C2 from the 2nd column of the table.  False tells the command to return only exact matches, while true will look for any matching words. For every category we may have, there is a standard directory (ex: Pet Items\Cats) and so you build out a list of all category names and their corresponding directories. In one cell of your template you will always have to paste the category name(C2), and then the next cell would have a vlookup like this: =VLOOKUP(C2,A2:B20, 2, False). A, the 1st column of the table, would be the names of your categories, and B (2nd column) would be the directory of each category.

I also make brand new folders, and you can concatenate the directory name to add the new names like this. D2 is directory lookup cell, E2 is new folder name: CONCATENATE(“”,D2,”",E2,”") which would get you Pet Items\Cats\NewFolderName in cell F2.

Say you want to create that new folder. Again you can use concatenate, this time with windows command line commands: =CONCATENATE(“CD “”Pet Items\Cats & MKDIR “”,E2,”"”) which would essentially create a new folder  in that category for every new name you need.  This drastically reduces the time spent creating new folders and is much simpler than right click, new folder, copy name, paste name, hit enter, do it again. It typically takes me about 30 seconds to create 80 folders in this way, if that long.

You can also use the windows command line and concatenate to copy new files to these folders. Again, this requires standardized file locations for the files in question.  I recommend making one if you don’t have it already, and then adding a new column for the filename itself. The easiest way to get the file names is click start, click run, then type cmd and hit enter. In the command prompt box type this: CD “DirectoryName” and hit enter to change to the needed directory. then type dir > FileList.txt. Dir lists all the files in a directory, and > tells it to make a file out the output (in this case, a text file). Next just open the text file, and paste it into excel. Using text to columns, choose standard width, and move the line between the filesize and the file name, then next and finish. It will create a new column with all names which you can then add into cell F2

Then simply concatenate together the directories with the file names and you’re all set: =CONCATENATE(“XCOPY “”Pets\”,FilenameCell,” “”Pets\Cats\NewFolder”" /s /i”) this command will add in all needed info to copy any files you may need, and you can then select and paste all commands at once to run multiple copy actions at one time. I actually recommend doing 3-5 at a time as you cannot view all error messages after a certain amount.

Moving on to firefox, some of the extensions I use most at work are iMacros, AutoCopy, and Clippings. iMacros basically records all mouse clicks and input into text boxes, which you can save and run again over and over. Think of it like this: If you are constantly pasting in data, then clicking a button, and then typing in something else, that can be recorded once and then when you double click a macro the whole thing is run before your eyes, kind of like watching a ghost use your computer.

What makes this even more powerful is that you can edit the scripts, simply by right clicking on them. The most useful edit I’ve found is telling it to paste from your clipboard, instead of whatever value was used when you first made the macro.  To do this, make a macro and save it.  Then right click it, choose edit, and find the line with the copied information. Highlight the copied info and replace it with this: {{!CLIPBOARD}}  This will tell it to paste whatever’s on the clipboard.  This works really well unless you happen to have copied info your system won’t recognize, like your girlfriend’s phone number or an essay :D

Other times, you can find out how webpages and dropdown menus were built by editing the macro. For example, at work when we choose a name from a dropdown, we just type it in. But each of those names actually has an ID associated with it, and when viewing the macro after saving it I found that it was not a name that showed up, but actually the matching ID.  Knowing this, you can again replace that ID with {{!CLIPBOARD}} and then quickly get to the info you need just by copying an ID.

Autocopy will automatically copy highlighted text after a user-determined amount of time, and clippings allows you to save your most used responses, like “Hey , here’s the thing you wanted” where you can just copy and paste in the person’s name in the greeting.  It may not be personal but it sure is quick and means you don’t have to do any more typing of repetitive forms and letters.

The uses are endless but your time is not, so try out these commands and make excel and firefox do all the hard work so you don’t have to!